Driver state online identification system and method based on virtual reality

Abstract

This invention discloses an online driver state recognition system and method based on virtual reality, comprising the following steps: Step 1, building a virtual reality driving system using the Unity 3D engine and virtual reality equipment; Step 2, collecting raw EEG signals and driving behavior data of the driver in the virtual reality environment using an EEG acquisition device and the Unity 3D engine; Step 3, preprocessing the collected data; Step 4, classifying different alertness states based on the preprocessed driving behavior data using the K-means clustering algorithm, extracting time-frequency features and coefficient of variation features of EEG signals under different alertness states, and obtaining EEG data with extracted features; Step 5, training a classification model with high classification accuracy using a machine learning algorithm; Step 6, analyzing and processing the real-time collected EEG signal data and inputting it into the classification model to obtain the online driver alertness state. This invention can achieve online recognition with high accuracy.

Description

Technical Field

[0001] This invention relates to the field of online driver alertness recognition technology, and in particular to an online driver alertness recognition system and method based on virtual reality. Background Technology

[0002] The development prospects of online driver alertness recognition systems in traffic accident prevention have attracted great attention from various countries. Researchers have conducted multifaceted studies on the physiological and operational characteristics of drivers when fatigued. The detection methods for driver alertness can be roughly divided into detection methods based on driver physiological signals, driver driving behavior characteristics, driver reaction time, and facial expression characteristics.

[0003] Identifying driver states based on physiological signal characteristics primarily involves analyzing collected signals from the driver's electroencephalogram (EEG), electrocardiogram (ECG), and electrooculogram (EOG). EEG signals have long been considered the "gold standard" for detecting changes in alertness. Driver alertness is assessed by analyzing changes in delta (δ), theta (θ), alpha (α), and beta (β) waves within the EEG signal. Studies have shown that when alertness declines, corresponding low-frequency signal activity in the EEG increases, and the amplitude of event-related potentials (ERPs) decreases accordingly.

[0004] Identifying driver state based on driving behavior characteristics primarily involves analyzing changes in vehicle parameters during driving. When a driver's alertness decreases, their perception of the external environment diminishes. When the vehicle deviates from its course, the driver's steering wheel input becomes sluggish, failing to adjust to the deviation in a timely manner. Consequently, the driver's control precision decreases, leading to significant changes in vehicle state parameters. Studying these changes can help infer the driver's state.

[0005] Domestic and international researchers have studied drivers' reaction times based on the decline in their perceptual abilities under fatigue. Each subject's reaction time is different; reaction time is inherent in each individual, and different people respond differently to the same event. Therefore, it is necessary to study the relationship between each driver's reaction time and their electroencephalogram (EEG) signals in order to make an overall assessment of the driver's alertness state.

[0006] Research on driver status recognition based on facial expression features often uses image processing techniques for judgment. This involves using an in-vehicle camera to capture video of the driver's facial expressions during driving and analyzing the data offline to determine the driver's status at that moment. However, offline analysis cannot provide real-time alerts and has a lag effect. Summary of the Invention

[0007] The purpose of this invention is to address the technical deficiencies in the existing technology by providing a method for online driver status recognition based on virtual reality.

[0008] Another objective of this invention is to provide an online driver status recognition system based on the aforementioned recognition method.

[0009] The technical solution adopted to achieve the purpose of this invention is:

[0010] A method for online driver status recognition based on virtual reality includes the following steps:

[0011] Step 1: Build a virtual reality driving system using the Unity 3D engine and virtual reality equipment;

[0012] Step 2: Use EEG acquisition equipment and Unity 3D engine to acquire the driver's raw EEG signals and driving behavior data in the virtual reality environment of Step 1;

[0013] Step 3: Preprocess the raw EEG signals and driving behavior data collected in Step 2 to improve the reliability of the data;

[0014] Step 4: Based on the preprocessed driving behavior data, the K-means clustering algorithm is used to divide different alertness states, and the time-frequency features and coefficient of variation features of the EEG signals under different alertness states are extracted to obtain the EEG data after feature extraction.

[0015] Step 5: Use machine learning algorithms to train a classification model with high classification accuracy on the EEG data after feature extraction;

[0016] Step 6: After analyzing and processing the real-time collected EEG signal data, the data is put into the classification model to obtain the driver's alertness status identified online.

[0017] In the above technical solution, step 1, the method for building a virtual reality driving system, includes the following steps:

[0018] S111 uses the Unity 3D engine to build a complete car driving environment;

[0019] S112, use the Inspector panel to render and stitch together the road model to form a straight road model, add straight signs and pedestrian crossing lines to the road, and select and place the environmental scenery on both sides of the road.

[0020] S113, using Unity 3D skybox to create the weather system;

[0021] S114, The experimental scenario is a virtual reality simulated driving environment. The subjects wear EEG acquisition devices to perform driving tasks. The subject's alertness status is displayed in real time in the upper right corner of the simulated driving system's GUI display interface.

[0022] S115, the steering wheel in the driving simulator is connected to the computer via a USB interface, and the steering wheel is used to control the direction of the simulated vehicle.

[0023] S116: The virtual reality device is connected to the computer via a streaming box. Two base stations capture the movement of the head-mounted display and transmit the movement information of the head-mounted display to Unity 3D. Steam VR then switches the real-time view of the human eye based on the head rotation information and outputs it to the head-mounted display to achieve 3D stereoscopic display of the scene.

[0024] In the above technical solution, in step 2, the EEG acquisition device acquires the raw EEG signals at four leads: P8, T7, T8 and P7. Preferably, the sampling frequency of the raw EEG signals is 500Hz. The driving behavior data includes real-time time, vehicle left and right offset, and steering wheel angle. Preferably, the driving behavior data sampling rate is 30Hz.

[0025] In the above technical solution, the preprocessing method of the original EEG signal in step 3 is as follows: bandpass filtering is performed in the range of 0.5Hz to 40Hz to reduce the sampling rate from 500Hz to 256Hz, and EEG data of a fixed duration are selected for subsequent analysis. Preferably, the fixed duration is 60min. After data sliding window processing with a fixed window length and step size, EEG data of m samples are obtained. Preferably, m = 1200.

[0026] In the above technical solution, the preprocessing method for driving behavior data in step 3 is as follows: select driving behavior data of a fixed duration for subsequent analysis, and each data segment of the driving behavior data corresponds one-to-one with the EEG data. The window length and step size of the sliding window for the driving behavior data are the same as those for the EEG data. Calculate the mean values ​​of the vehicle's left and right offset and the steering wheel angle in each window of the driving behavior data as statistical indicators. Finally, obtain m samples of driving behavior variables as the preferred ones, where m = 1200.

[0027] In the above technical solution, in step 4, the K-means algorithm clusters the driving behavior data, dividing the driver's alertness into three categories: high alertness, semi-alertness, and low alertness. The clustering process is as follows:

[0028]

[0029] Where, p x For each data point in the driving behavior data, n is the number of data points, and k is the number of cluster centers. The cluster centers are the cluster centers after j iterations.

[0030] In the above technical solution, in step 4, the d3 sub-band is selected for extracting the time-frequency features of the EEG signal; when extracting the coefficient of variation features, Where σ represents the standard deviation of the d3 subband of the sample; μ represents the mean of the d3 subband of the sample; and CV is the EEG data after feature extraction.

[0031] In the above technical solution, in step 5, the support vector machine algorithm is used to train the classification model using the EEG data after feature extraction:

[0032] y i [(w·x i )+b]≥1i=1,2,…,l

[0033] Among them, (x i ,y i ) represents the training samples, l represents the number of samples, and w and b represent the normal vector and intercept of the hyperplane constructed by the support vector machine algorithm, respectively.

[0034] In the above technical solution, the analysis and processing method in step 6 is as follows: bandpass filtering is performed in the range of 0.5Hz to 40Hz to reduce the sampling rate from 500Hz to 256Hz, then discrete wavelet transform is performed to select the d3 sub-band, and then CV features are extracted through the coefficient of variation.

[0035] In another aspect of the present invention, a virtual reality-based online driver state recognition system includes an EEG acquisition device, an analysis module, and a classification model obtained using the above method. The EEG acquisition device acquires raw EEG signals and transmits the signals wirelessly in real time via a local area network. After receiving the raw EEG signals, the analysis module performs preprocessing, filtering, and downsampling to extract alertness-related features. The classification model then analyzes these features to obtain the driver's alertness state.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] 1. The driver status online recognition method based on virtual reality provided by this invention can monitor the driver's status in real time, obtain the alertness status result, and give the driver a timely warning, which is of great significance for ensuring the driver's personal safety.

[0038] 2. The driver status online recognition system based on virtual reality provided by the present invention can process and analyze driver brain signals online in real time. The monitoring and analysis system is integrated into the virtual reality simulation system, which is more convenient to use than the previous system that required a separate monitoring device.

[0039] 3. The online identification system of this invention uses wireless methods such as local area networks to exchange data information, making it easy to use. At the same time, the device is highly portable, wirelessly connected, and does not restrict the driver's freedom of movement, making it easy to use. Attached Figure Description

[0040] Figure 1 This is a flowchart of the online driver status recognition method based on virtual reality as described in this invention.

[0041] Figure 2 This is a flowchart of the method for building a virtual reality-based simulated driving system according to the present invention.

[0042] Figure 3 This is a flowchart of the driver status-based online recognition system described in this invention.

[0043] Figure 4 This is a schematic diagram of a driving simulator system.

[0044] Figure 5 It means turning the steering wheel to correct straight-line driving.

[0045] Figure 6 This is the HTC Vive system display process.

[0046] Figure 7 This is a diagram illustrating the lateral displacement of the vehicle.

[0047] Figure 8 It is the result of clustering the driving behavior of the test subjects.

[0048] Figure 9 Wavelet transform subband EEG signals of drivers in different states.

[0049] Figure 10 These are the characteristics of EEG signals under different states. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0051] Example 1

[0052] like Figure 1 As shown, a method for online driver status recognition based on virtual reality includes the following steps:

[0053] Step 1 (S110): Build a driving simulation system using the Unity 3D engine and HTC Vive virtual reality device;

[0054] Conducting driving experiments in a real-world environment presents significant challenges in ensuring driver safety, and data acquisition is susceptible to interference from various external factors. This invention utilizes Unity 3D and HTC Vive to build a virtual reality driving simulation platform, enabling research to be conducted in a laboratory setting. This ensures driver safety while minimizing the impact of external factors on data collection. The simulation system recreates realistic road, building, and weather scenarios, providing researchers with an authentic driving experience.

[0055] Step 2 (S120): Use the Boruikang EEG acquisition device and Unity 3D to acquire the driver's original EEG signals and driving behavior data in the virtual reality environment in Step 1;

[0056] The data collected in the virtual reality driving simulation experiment mainly consists of two parts: driver's electroencephalogram (EEG) data and driving behavior data. Driving behavior data includes real-time vehicle movement time, lateral deviation, and steering wheel angle.

[0057] Step 3 (S130) preprocesses the collected raw data to improve the reliability of the data;

[0058] During experimental data acquisition, EEG signals are relatively weak and easily interfered with by other noises, making it difficult to acquire clean signal data and leading to inaccurate results in subsequent data analysis and identification. Common interfering signals include electrooculography (EOG), electromyography (EMG), and electromagnetic interference. Therefore, it is necessary to preprocess the acquired EEG signals to remove artifacts. This invention uses a bandpass filter in the range of 0.5Hz-40Hz, which can effectively remove high-frequency noise and low-frequency drift. Due to the large amount of data acquired, the sampling rate was reduced from 1000Hz to 256Hz to improve the efficiency of subsequent data analysis and processing.

[0059] Step 4 (S140): Classify different alertness states based on driving behavior, and extract the time-frequency features and coefficient of variation features of EEG signals under different alertness levels;

[0060] Alertness levels are categorized based on drivers' highly relevant driving behaviors during driving tasks, classifying alertness states into three categories: high alertness, semi-alertness, and low alertness. EEG signals are non-stationary signals, with their frequency components constantly changing over time. Wavelet transform utilizes scaling and translation factors, employing multi-scale analysis to better represent the time-frequency information of the signal. The coefficient of variation is a normalized measure of the dispersion of a probability distribution; effective feature extraction from EEG signals can improve the accuracy of data classification.

[0061] Step 5 (S150): Use machine learning algorithms to train a classification model with high classification accuracy on the EEG data after feature extraction;

[0062] Electroencephalogram (EEG) signals exhibit significant variability between individuals and between different states within a single individual. This study employed machine learning algorithms to analyze the classification differences among three alertness states. Support Vector Machine (SVM) classification algorithms, based on a limited set of samples, achieve the best generalization ability of the trained model by improving the learning accuracy and capacity of the training samples. Compared to other machine learning methods, SVM effectively addresses classification and recognition problems with small sample sizes and limited feature dimensions, and can handle both linear and nonlinear data.

[0063] Step 6 (S160): After analyzing and processing the real-time collected EEG signal data, it is put into the classification model to obtain the driver's alertness status identified online.

[0064] During driving, real-time data on the driver's driving behavior and EEG signals are collected and processed online (bandpass filtering is performed in the 0.5Hz–40Hz range, the sampling rate is reduced from 500Hz to 256Hz, then discrete wavelet transform is performed, the d3 sub-band is selected, and CV features are extracted using the coefficient of variation). The subject's alertness level is displayed in real-time in the upper right corner of the simulated driving system's GUI interface, allowing passengers and drivers to easily observe changes in alertness. A reference state value is provided to both the driver and passengers, enabling intervention when the driver's alertness level is low.

[0065] Example 2

[0066] Step 1 uses the Unity 3D engine and HTC Vive virtual reality device to build a simulated driving system, such as... Figure 4 As shown, the specific steps include, as follows: Figure 2 As shown:

[0067] The S111 driving simulation system first uses the Unity 3D engine to build a complete car driving environment. The Unity 3D engine is the preferred development tool for large-scale game production and the creation of 3D animations.

[0068] S112, Road Simulation Environment. This is a model of the Island Highway Race, downloaded from the Unity 3D asset store. Considering the driving alertness state recognition experiment, to increase the monotony of the driving road, the road model was rendered and stitched together using the Inspector panel to create a straight-ahead road model. Straight-ahead signs and pedestrian crossing markings were added to the road, and environmental scenery such as trees, buildings, mountains, and beaches along the roadside was selected and placed. The road uses a single-lane, five-lane highway model, with each lane 3.75m wide, to facilitate the collection of driving behavior information.

[0069] S113, Weather Simulation Environment. The weather system is created using a Unity 3D skybox, with materials and textures configured, and a sunny effect selected. The car model uses the HQ Racing Car Model No. 1203, with a width of 1.8m. Two cameras, Camera1 and Camera2, are added; by adjusting the camera angles, the first-person and third-person perspectives during driving can be switched.

[0070] S114, Driver Status. The experimental scenario was a virtual reality simulated driving environment. The subject wore an EEG acquisition device to perform a driving task at a speed of approximately 70 km / h. The subject's alertness level was displayed in real time in the upper right corner of the simulation driving system's GUI interface, allowing passengers and the driver to observe changes in alertness.

[0071] S115, the steering and control system. It uses the steering wheel from Logitech's driving simulator. The Logitech G29 steering wheel is similar to a real steering wheel, capable of rotating 900 degrees and highly sensitive to rotation angles, providing precise measurements. The Logitech G29 steering wheel connects to a computer via USB. The driver manipulates the steering wheel to control the simulated vehicle's direction, such as... Figure 5 As shown.

[0072] S116, Visual Driving Feedback. For example... Figure 6 As shown, after setting up the Unity 3D driving simulation system, the driving simulation interface on the computer needs to be displayed on the HTC Vive virtual reality device. Two base stations are powered on and placed in suitable locations, forming a closed space between them. The positioning space created by the two base stations is used to integrate and process the signals received by the laser sensors on the head-up display. The infrared laser locators on the base stations send signals, and the streaming box connects the head-up display and the computer. The head-up display has many laser sensors that can determine the position and angle of objects and scenes within the display, thus performing a positioning function. The driver SDK and Steam VR software need to be installed on the computer, and the HTC Vive virtual reality device and the computer need to be correctly connected. Upon launching the Steam VR software, a clear virtual interface should appear. The two HTC Vive base stations capture the movement of the head-up display. The PC is connected to the head-up display via an HDMI cable, transmitting the head-up display's motion information to Unity 3D. The Steam VR software then switches the real-time view based on head rotation information and outputs it to the head-up display, achieving a 3D stereoscopic display of the scene.

[0073] Step 2, which involves using the Boruikang EEG acquisition device and Unity 3D to acquire the driver's raw EEG signals and driving behavior data in the virtual reality environment of Step 1, specifically includes the following steps:

[0074] The Boruikang 8-channel portable wireless EEG acquisition system is a new generation of NeuSen series wireless EEG devices independently developed by Boruikang. The electrode positions are arranged according to the international 10-20 system, with a sampling frequency of 500Hz. It can be used in multiple fields such as brain-computer interfaces, psychology, neuromarketing, and rehabilitation engineering. The system features excellent portability, stable signal, and good shielding, allowing for free movement in natural environments. It is equipped with a nine-axis motion sensor to eliminate signal interference during subject movement.

[0075] Different leads in the brain produce different EEG signals, and the selection of leads is crucial for analyzing the characteristics of EEG signals. The more leads selected, the more comprehensive the EEG signal acquisition. However, not all EEG signals are relevant to the driver's state detected in this paper. Therefore, this invention only uses the raw EEG signals from four leads in the temporal region—P8, T7, T8, and P7—which have a strong correlation with alertness and high classification accuracy.

[0076] During the experiment, real-time data such as vehicle travel time, lateral deviation, and steering wheel angle were collected. Figure 7 As shown, the vehicle's left and right offset is the distance between the vehicle's position and the lane centerline. This information is saved in txt format in designated folders on the desktop using the GetFolderPath statement in Unity3D software. Steering wheel and pedal data are collected using the Input command in Unity3D; the steering wheel data refers to the steering wheel angle. Real-time vehicle displacement information is collected using the local command in Unity3D, and the vehicle's left and right offset is obtained from this real-time displacement information.

[0077] Step 3 involves preprocessing the collected raw EEG signals and driving behavior data to improve the reliability of the data. Specifically,

[0078] The preprocessing of the raw EEG signals is as follows:

[0079] This invention uses a bandpass filter within the 0.5Hz-40Hz range to effectively remove high-frequency noise and low-frequency drift. Due to the large amount of data collected, the sampling rate was reduced from 1000Hz to 256Hz to improve the efficiency of subsequent data analysis and processing. The experiment lasted approximately 60 minutes. To ensure data length consistency and avoid data overlap, the data from the first 60 minutes was uniformly selected for subsequent analysis. Therefore, after processing the data through a sliding window with a window length and a step size of 3 seconds, a total of 1200 EEG samples were obtained.

[0080] The preprocessing of driving behavior data is as follows:

[0081] Driving behavior data was primarily collected using the Unity 3D engine, with a raw data sampling rate of 30Hz. The collected driving behavior data mainly included vehicle lateral deviation and steering wheel angle. Since this study selected data from the first 60 minutes for analysis, to ensure a one-to-one correspondence with each segment of the EEG data, the sliding window size and step size for the driving behavior data were both set to 3 seconds, with zero overlap points. The mean values ​​(means) of vehicle deviation and steering wheel angle data within each window were calculated as statistical indicators, resulting in 1200 driving behavior variables.

[0082] Step 4 involves classifying different alertness states based on driving behavior data and extracting the time-frequency features and coefficient of variation features of EEG signals under different alertness levels; this includes the following steps:

[0083] This invention starts with driving behavior data collected during driving tasks, finds the correspondence between driving behavior data and changes in alertness, and thus, based on time synchronization, marks the alertness of EEG signals.

[0084] Driving behavior data is a discrete data signal. When performing discrete labeling on driving behavior data, the K-means clustering algorithm is mainly used. It utilizes the spatial distribution relationship of driving behavior data and combines the temporal dependence of alertness changes to perform discrete alertness state labeling.

[0085] The K-means algorithm clusters driving behavior data. During the clustering process, because the data sizes vary greatly among different driving behaviors, the clustering results can easily highlight high-value indicators. Therefore, it is necessary to perform max-min standardization on the clustered data to ensure that the results fall within the [0,1] interval. Driver alertness is divided into three categories: high alertness, semi-alertness, and low alertness.

[0086] The clustering process is as follows:

[0087]

[0088] Where, p x For each data point in the driving behavior data, n is the number of data points, and k is the number of cluster centers. The cluster centers are the cluster centers after j iterations.

[0089] Clustering yielded three alertness levels: high alertness, semi-alertness, and low alertness. The results are as follows: Figure 8 As shown, during the clustering process, the horizontal axis represents the left and right offset of the vehicle, and the vertical axis represents the steering wheel angle. Figure 8 The horizontal and vertical axis data are obtained by normalizing the original data to eliminate differences between data.

[0090] Electroencephalogram (EEG) signals are non-stationary signals, and their frequency components change continuously over time. Wavelet transform is a time-frequency analysis method that uses scaling and translation factors to better represent the time-frequency information of the signal through multi-scale analysis. In the analysis of EEG signals in this invention, discrete wavelet transform is used to analyze EEG signals at different frequency bands. Discrete wavelet transform uses scaling functions and wavelet functions to perform high-pass and low-pass filtering on the time-domain varying EEG signals, respectively.

[0091] Discrete wavelet transform is used to perform multi-resolution decomposition on the EEG signal x[n] to obtain its time-frequency features. The signal is first decomposed into different frequency bands {d1, d2, d3, d4, d5, d6, d7}. d3 covers most of the β segment, such as... Figure 9 As shown, the d3 sub-band is selected in the time-frequency feature extraction part; then, further feature extraction is performed. This invention extracts features from the sub-band signal obtained through wavelet transform, and the feature selection is the coefficient of variation. The coefficient of variation (COP), also known as the coefficient of dispersion, is a normalized measure of the dispersion of a probability distribution. It is defined as the ratio between the standard deviation and the mean deviation, and the formula is:

[0092]

[0093] Where σ represents the standard deviation of the d3 subband of the sample; μ represents the mean of the d3 subband of the sample; and CV represents the EEG data after feature extraction. The results are as follows: Figure 10 As shown, in this embodiment, the above-mentioned feature extraction is performed on 1200 EEG signal samples after data sliding window processing with a window length and a step length of 3s.

[0094] Step 5 involves using a machine learning algorithm to train a high-accuracy classification model on the extracted EEG data. This specifically includes the following steps: EEG signals exhibit significant differences between individuals and between different states within a single individual. In this study, a machine learning algorithm was chosen to analyze the classification differences among the three alertness states. The Support Vector Machine (SVM) classification algorithm, based on a limited number of samples, achieves the best generalization ability of the trained model by improving the learning accuracy and learning capacity of the training samples. Compared to other machine learning methods, SVM effectively solves classification and recognition problems with small sample data and limited feature dimensions, and can handle both linear and nonlinear data.

[0095] A classification model was trained on the feature-extracted EEG data using the support vector machine algorithm.

[0096] y i [(w·x i )+b]≥1i=1,2,…,l

[0097] Among them, (x i,y i ) represents the training samples, l represents the number of samples, and w and b represent the normal vector and intercept of the hyperplane constructed by the support vector machine algorithm, respectively.

[0098] Step 6 involves analyzing and processing the real-time collected EEG signal data and then inputting it into a classification model to obtain the driver's alertness state through online identification. This step specifically includes the following steps:

[0099] The experimental scenario was a virtual reality driving simulation environment. The driver wore an EEG acquisition device to perform a driving task at approximately 70 km / h. The driver's alertness level was displayed in real-time in the upper right corner of the simulation system's GUI interface, allowing both passengers and the driver to observe changes in alertness. To objectively and accurately reflect the driver's state, the driver's driving behavior was used as a control indicator for the online EEG signal alertness detection results. The study achieved an online recognition accuracy of 86.7655%, based on the driver state online recognition system... Figure 3 As shown.

[0100] In order to achieve real-time transmission of EEG data and state recognition results in online system recognition, the EEG acquisition system module S162, Matlab data analysis module S163, and Unity 3D simulation driving module S164 all need to be connected to the same local area network created by the S161 router. This allows the acquired EEG signals to be transmitted to Matlab for analysis and processing in a timely manner, and the processed state results from Matlab to be transmitted to Unity 3D for display in the simulation driving interface.

[0101] In this embodiment, when the driver first activates the online EEG recognition system while driving, a message indicates that monitoring has begun; when the monitoring time ends, a message indicates that monitoring has ended, and the background Matlab EEG analysis and processing program automatically closes. During the monitoring process, the system outputs monitoring results every minute. This invention only performs a three-category study on the driver's state; therefore, based on the monitoring results of the driver's alertness, it outputs three states: high alertness, semi-alertness, and low alertness.

[0102] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for online driver status recognition based on virtual reality, characterized in that, Includes the following steps: Step 1: Build a virtual reality driving system using the Unity 3D engine and virtual reality equipment; Step 2: Use EEG acquisition equipment and Unity 3D engine to acquire the driver's raw EEG signals and driving behavior data in the virtual reality environment of Step 1; Step 3: Preprocess the raw EEG signals and driving behavior data collected in Step 2 to improve the reliability of the data; Step 4: Based on the preprocessed driving behavior data, the K-means clustering algorithm is used to divide different alertness states, and the time-frequency features and coefficient of variation features of the EEG signals under different alertness states are extracted to obtain the EEG data after feature extraction. Step 5: Use machine learning algorithms to train a classification model with high classification accuracy on the EEG data after feature extraction; Step 6: After analyzing and processing the real-time collected EEG signal data, the data is put into the classification model to obtain the driver's alertness status identified online. Step 1, the method for building the virtual reality driving system, includes the following steps: S111 uses the Unity 3D engine to build a complete car driving environment; S112, use the Inspector panel to render and stitch together the road model to form a straight road model, add straight signs and pedestrian crossing lines to the road, and select and place the environmental scenery on both sides of the road. S113, using Unity 3D skybox to create the weather system; S114, The experimental scenario is a virtual reality simulated driving environment. The subjects wear EEG acquisition devices to perform driving tasks. The subject's alertness status is displayed in real time in the upper right corner of the simulated driving system's GUI display interface. S115, the steering wheel in the driving simulator is connected to the computer via a USB interface, and the steering wheel is used to control the direction of the simulated vehicle. S116, the virtual reality device is connected to the computer through a streaming box. Two base stations capture the movement of the head-mounted display device and transmit the movement information of the head-mounted display device to Unity 3D. Steam VR then switches the real-time view of the human eye according to the head rotation information and outputs it to the head-mounted display device to realize the 3D stereoscopic display of the scene. In step 4, the K-means algorithm clusters the driving behavior data, classifying driver alertness into three categories: high alertness, semi-alertness, and low alertness. The clustering process is as follows: in, For each data point in the driving behavior data, The number of data points The number of cluster centers. for Cluster centers after the next iteration; In step 4, the extraction of time-frequency features of the EEG signal is selected from... d 3 sub-bands; during coefficient of variation feature extraction, in Representing the sample d The standard deviation of the 3 sub-bands; Representing the sample d The average value of the 3 sub-bands CV The EEG data after feature extraction; In step 5, the support vector machine algorithm is used to train a classification model using the EEG data after feature extraction. in, As training samples, For the number of samples, w and b The normal vector and intercept of the hyperplane are constructed for the Support Vector Machine algorithm, respectively.

2. The online driver status recognition method based on virtual reality as described in claim 1, characterized in that, In step 2, the EEG acquisition device collects raw EEG signals from four leads: P8, T7, T8, and P7; the driving behavior data includes real-time time, vehicle lateral deviation, and steering wheel angle.

3. The online driver status recognition method based on virtual reality as described in claim 2, characterized in that, The sampling frequency of the raw EEG signal was 500 Hz.

4. The online driver status recognition method based on virtual reality as described in claim 2, characterized in that, The driving behavior data sampling rate is 30Hz.

5. The online driver status recognition method based on virtual reality as described in claim 1, characterized in that, In step 3, the preprocessing method of the original EEG signal is as follows: bandpass filtering is performed in the range of 0.5Hz to 40Hz to reduce the sampling rate from 500Hz to 256Hz. EEG data of a fixed duration are selected for subsequent analysis. After data sliding window processing with fixed window length and step size, EEG data of m samples are obtained.

6. The online driver status recognition method based on virtual reality as described in claim 5, characterized in that, The fixed duration is 60 minutes.

7. The online driver status recognition method based on virtual reality as described in claim 5, characterized in that, m=1200。 8. The online driver status recognition method based on virtual reality as described in claim 1, characterized in that, In step 3, the preprocessing method for driving behavior data is as follows: select driving behavior data of a fixed duration for subsequent analysis, and each data segment of the driving behavior data corresponds one-to-one with that of the EEG data. The window length and step size of the sliding window for the driving behavior data are the same as those for the EEG data. Calculate the mean values ​​of the vehicle's left and right offset and the steering wheel angle in each window of the driving behavior data as statistical indicators, and finally obtain M samples of driving behavior variables.

9. The online driver status recognition method based on virtual reality as described in claim 8, characterized in that, M=1200。 10. The online driver status recognition method based on virtual reality as described in claim 1, characterized in that, In step 6, the analysis and processing method is as follows: bandpass filtering is performed in the range of 0.5Hz to 40Hz to reduce the sampling rate from 500Hz to 256Hz, and then discrete wavelet transform is performed before selection. d Three sub-bands are used to extract CV features through the coefficient of variation. Driver alertness levels include high alertness, semi-alertness, and low alertness.

11. A driver status online recognition system based on virtual reality, characterized in that, The system includes an EEG acquisition device, an analysis module, and a classification model obtained using the method described in any one of claims 1-10. The EEG acquisition device acquires raw EEG signals and transmits the signals wirelessly in real time via a local area network. The analysis module receives the raw EEG signals, performs preprocessing filtering and downsampling, extracts alertness-related features, and analyzes them using the classification model to obtain the driver's alertness state.

Images