Driving mental load prediction method and system based on non-contact eye movement signal

Abstract

The invention discloses a driving mental load prediction method and system based on a non-contact eye movement signal, and relates to the technical field of auxiliary driving. Comprising the steps of 1, collecting eye movement time sequence data of a driver; 2, preprocessing and time sequence signal feature extraction are carried out on the collected eye movement data, and time sequence information is extracted as a feature variable by considering the time-varying feature of mental workload; 3, constructing a mental load regression prediction model; according to the method, long-time driving mental load dynamic prediction is carried out by adopting a single mode of an eye movement signal, and the mental load can be accurately predicted by an established mental load regression prediction model; the problems that in an existing mental load prediction method, data acquisition is difficult, calculation difficulty is large, the mental load of the driver is evaluated based on a static view angle, time-varying features of the mental load in the long-time driving process are not considered, and evaluation precision is not high are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of assisted driving, and in particular relates to a method and system for predicting driving mental workload based on non-contact eye movement signals. Background Art

[0002] The vehicle driving system is a highly coupled human-computer interaction system consisting of the driver, the vehicle and the traffic environment. The driver needs to pay a lot of cognitive resources to maintain good driving performance during long-term driving. Continuous cognitive resource expenditure is prone to mental load; and as the driving time increases, the mental load can easily reach an overload level, which in turn causes brain fatigue and distraction, which can easily increase driving safety risks and induce traffic accidents. Therefore, how to evaluate and predict the changes in the driver's mental load during long-term driving is important for ensuring driving safety. Since mental load can be characterized by physiological signals of multiple modalities, the most mainstream method for evaluating driver mental load is to integrate the physiological data of the driver during driving and model the machine learning evaluation model. These methods based on experimental data have good reliability, credibility and robustness, and provide a theoretical and methodological basis for driving safety and traffic safety research. However, these methods and technologies still generally have the following important shortcomings.

[0003] First, the existing technology generally uses multiple physiological modalities for modeling and evaluation, and mostly collects data through wearable sensors, which makes data acquisition difficult and computationally difficult. For example, in the Chinese invention patent with patent number "CN200910093545.0" and titled "Method for measuring driving workload of automobile drivers", the scheme uses the driver's physiology, psychology and driving behavior to characterize the driver's driving workload. This invention requires the collection of physiological signals such as the driver's EEG, ECG, respiration, and skin, and uses them to build a mental workload evaluation model. In the Chinese invention patent with patent number "CN113627740A" and titled "A system and method for building a driving load evaluation model", the scheme collects the driver's physiological signals, vehicle operation signals, and driving behavior signals, and builds a driver's mental workload evaluation model based on these signal indicators. Although adding all the physiological modal data that can characterize mental workload to the model for training can improve the evaluation accuracy, various physiological signals need to be obtained through complicated wearable sensors, and data processing is also very complicated. Among them, data acquisition requires drivers to wear more physiological signal acquisition devices, which will cause contact intrusion. These devices are bulky when put together, which will affect driving operation and comfort during actual driving, causing safety risks, so they cannot be effectively applied. For example, when drivers wear EEG signal acquisition devices, multi-electrode EEG signal acquisition may interfere with the driver's comfort and freedom of movement, causing safety problems; secondly, interference from the external environment and head movement will reduce the clarity and reliability of the signal.

[0004] Secondly, the existing technologies basically evaluate the mental workload of drivers based on a static perspective, without considering the time-varying characteristics of mental workload during long-term driving (i.e., mental workload will change dynamically over time due to changes in personnel status, tasks and environmental status), and the relationship between physiological indicators characterizing mental workload and mental workload will also change, which leads to the mental workload evaluation model established under a static perspective having problems such as insufficient reliability and reduced accuracy in the actual dynamic driving application process. For example, the existing technologies mostly use a shorter period of driving data to train related models to predict the mental workload of drivers. However, in real life, drivers often perform long-term driving tasks. After a long driving task, the corresponding relationship between mental workload and physiological signals may have changed to a certain extent compared to the early stage. However, if the model trained with early data is still used for mental workload evaluation, it will obviously reduce the accuracy of the evaluation results. Therefore, it is necessary to consider the time factor as a correction factor when constructing the model.

[0005] Finally, the existing technology usually uses a classification model to evaluate the driver's mental workload. This classification model has the problem of low evaluation accuracy and is difficult to meet the actual needs of more refined evaluation of mental workload. The classification level in the classification model only indicates that the mental workload is within a certain range, and the classification model outputs the same result for the lowest and highest values ​​in the range, but in fact, there is a certain difference in the mental workload represented by these two values, which can easily mislead drivers in real driving applications, causing drivers to make inappropriate judgments on whether they are suitable to continue driving.

[0006] Therefore, the present invention proposes a dynamic prediction method for long-term driving mental workload based on non-contact eye movement signals. The method adopts a regression prediction method and considers the time-varying characteristics of mental workload by introducing time series information. It can construct a more sophisticated mental workload regression prediction model, and can achieve more accurate mental workload assessment. It has better sensitivity and accuracy in practical applications, thereby solving the problems existing in the above-mentioned prior art. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] In view of the shortcomings of the prior art, the present invention provides a method and system for predicting driving mental workload based on non-contact eye movement signals. The method uses eye movement signals as a single mode to dynamically predict the mental workload during long-term driving. The established mental workload regression prediction model can accurately predict the mental workload, overcoming the shortcomings of the existing mental workload assessment methods and systems in terms of large requirements for equipment and computing resources, and solving the problems existing in the existing mental workload prediction methods, such as difficulty in data acquisition, high computational difficulty, assessment of the driver's mental workload based on a static perspective, failure to consider the time-varying characteristics of mental workload during long-term driving, and low assessment accuracy.

[0009] (II) Technical solution

[0010] In order to achieve the above-mentioned purpose of accurately predicting mental workload, the present invention provides the following technical solutions:

[0011] The present invention provides a first solution: a method for predicting driving mental workload based on non-contact eye movement signals, comprising:

[0012] Step 1: Collect the driver's eye movement time series data;

[0013] Step 2: Preprocess the collected eye movement time series data and extract the time series signal features, and consider the time-varying characteristics of mental workload to extract the time series information as the feature variable;

[0014] Wherein: the eye movement timing signal includes eye movement point, fixation point, pupil and blink index;

[0015] Step 3: Construct a regression prediction model for mental workload

[0016] Based on the eye movement signal features and time series information features, the driver's mental workload is predicted using linear regression, neural network and Gaussian process regression machine learning regression algorithms, and the Gaussian process regression model is determined as the mental workload regression prediction model;

[0017] Step 4: Use the mental workload regression prediction model to calculate the mental workload prediction value, and use the driving assistance decision module to remind the driver of the mental workload level and make corresponding driving suggestions.

[0018] As one of the preferred embodiments of the present invention, the eye movement point processing process includes:

[0019] (1) Data interpolation: Assume that the maximum interval length is 75 ms. The signal loss less than the maximum interval is compensated by linear interpolation. The linear interpolation formula is:

[0020]

[0021] Among them, the coordinates (x 1 ,y 1 ) and (x 0 ,y 0 ) are the values ​​of the two end points of the interpolation interval, and the coordinates (x, y) represent the interpolation points of the required interpolation interval;

[0022] (2) Data denoising: Use the sliding mean filtering method to perform data denoising.

[0023] As one of the preferred implementation modes of the present invention, the gaze point extraction process includes:

[0024] (1) Set the angular velocity calculation window length to 20ms and the angular velocity threshold to 30° / s. When the angular velocity is greater than 30° / s, it is classified as a saccade, and when it is less than or equal to 30° / s, it is classified as a fixation point;

[0025] (2) Set the maximum time threshold between fixations to 75 ms and the maximum angular velocity threshold between fixations to 0.5°; when the interval between two adjacent fixations is less than 75 ms and the interval angle is less than 0.5°, merge the two adjacent fixations into one;

[0026] (3) The minimum fixation time threshold is set to 60 ms, and fixation points with a fixation time below the threshold are deleted.

[0027] As one of the preferred embodiments of the present invention, the pupil processing process includes:

[0028] (1) Data interpolation: The minimum pupil diameter is set to 2 mm, and the data smaller than the minimum pupil diameter is defined as an outlier and removed. After the outliers are removed, linear interpolation is performed using the linear interpolation method;

[0029] (2) Data denoising: Use the sliding mean filtering method to perform data denoising.

[0030] As one of the preferred embodiments of the present invention, the process of blink processing includes:

[0031] (1) Set the maximum time threshold to 350ms and the minimum time threshold to 75ms; when the left and right eyes are closed at the same time and the closing time is between 75ms and 350ms, it is classified as a blink;

[0032] (2) After completing the preprocessing of the eye movement data, the correlation coefficient method is used to obtain the eye movement features that are highly correlated with mental workload;

[0033] In the process of using the correlation coefficient method to obtain eye movement features that are highly correlated with mental workload: considering the time-varying characteristics of mental workload, the extracted eye movement signals are time-series marked, and the time-series information is used as a supplementary feature variable; the time-series information usually defaults to the driving time when the eye movement data is collected.

[0034] As one of the preferred embodiments of the present invention, in step 3, when using the machine learning regression algorithms of linear regression, neural network and Gaussian process regression to predict the driver's mental workload, the determination coefficients R of the linear regression model, the neural network model and the Gaussian process regression model are compared respectively. 2 , root mean square error RMSE and mean absolute error MAE to determine the optimal model, where:

[0035]

[0036] in, is the predicted value of mental workload, y is the observed value of mental workload, is the average value of mental workload observations, and n is the number of samples.

[0037] As one of the preferred implementations of the present invention, before establishing the linear regression model, the neural network model and the Gaussian process regression model, it is necessary to perform feature conversion on the eye movement input features extracted in step 2. The feature conversion method is specifically as follows:

[0038]

[0039] Among them: x is the input feature; x mean is the mean of the feature; x stddev is the standard deviation of the feature; x std The results are standardized.

[0040] As one of the preferred implementations of the present invention, the construction process of the Gaussian process regression model includes:

[0041] Step 3.4.1: Assume that the function f(x) of the input training set X and the output Y has a Gaussian distributed random process;

[0042] Here, the Gaussian distribution is specified by a mean function m(x) and a covariance function K(x,x');

[0043] Step 3.4.2: Obtain x by calculating the posterior distribution of the acquired training data * , the posterior distribution is:

[0044] P(f(x * )|X,Y,x * )=N(f(x * ) * ,∑ * )

[0045] Among them, μ * and∑ * are the mean and covariance of the posterior distribution, respectively, and are given by:

[0046]

[0047] Among them, K(x * ,X)、K(X,X)、K(x * ,x * ) represent x * and X, X and X, x * With x * The covariance matrix between is the observation noise variance;

[0048] Step 3.4.3: Use the covariance calculation method. The covariance calculation method uses the Matern5 / 2 function. The mathematical expression of the Matern5 / 2 function is:

[0049]

[0050] In the formula, Represents x i With x j The Euclidean distance between l is the characteristic length dimension, σ f is the standard deviation of the signal.

[0051] The present invention provides a second solution: a driving mental workload prediction system based on non-contact eye movement signals, comprising:

[0052] A data acquisition module, used for collecting the driver's eye movement timing data;

[0053] A data processing module, used for performing data preprocessing and eye movement timing signal feature extraction on the eye movement signals collected by the data collection module;

[0054] The mental workload regression prediction model module is used to dynamically predict mental workload in a long-term dynamic environment;

[0055] The decision support module is used to remind the driver of the mental workload level and provide corresponding driving suggestions.

[0056] Among them, the data acquisition module, data processing module and mental workload regression prediction model module are used to implement a driving mental workload prediction method based on non-contact eye movement signals.

[0057] (III) Beneficial effects

[0058] Compared with the prior art, the present invention provides a method and system for predicting driving mental workload based on non-contact eye movement signals, which has the following beneficial effects:

[0059] 1. This method only collects eye movement signals in a non-contact manner and does not interfere with the driving process. The relevant collection equipment is lightweight and has good installation compatibility with the vehicle; it solves the problem that the existing solutions need to collect complex physiological signals, which requires the driver to wear multiple physiological signal collection devices, which are bulky when combined and affect driving operations;

[0060] 2. This method takes the time-varying characteristics of mental workload into consideration and improves the modeling process by introducing time series factors, thereby improving the model's compatibility with time-varying characteristics and prediction accuracy. It also solves the problem that the existing scheme does not consider the possible changes in the mental workload characterization indicators during long-term driving, so the model built only based on short-term data will have a lower accuracy in predicting mental workload during long-term driving tasks.

[0061] 3. This method proposes a sophisticated mental workload regression prediction model, which can obtain a mental workload prediction value between 0 and 100 points in a long-term dynamic environment, achieve a more accurate mental workload assessment, and have better sensitivity and accuracy in practical applications; it solves the problem that in the existing technical solutions, the mental workload assessment of drivers mostly relies on classification models. This method may result in the same output results when the mental workload varies greatly, which can easily mislead drivers in actual driving applications and make them make inappropriate judgments on whether they are suitable to continue driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The flowchart of establishing the mental workload regression prediction model of the present invention;

[0063] Figure 2 It is a flow chart of the method for predicting driving mental workload based on non-contact eye movement signals of the present invention;

[0064] Figure 3 It is a structural diagram of the neural network model of the present invention;

[0065] Figure 4 This is a system structure diagram of the driving mental workload prediction system based on non-contact eye movement signals of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] See also Figure 1-Figure 2 The present invention provides a technical solution: a method for predicting driving mental workload based on non-contact eye movement signals. The method uses the single mode of eye movement signals to dynamically predict the mental workload of long-term driving, including:

[0068] Step 1: Collect the driver’s eye movement timing data

[0069] The simulated driving system used in this embodiment includes a driving simulator, simulated driving scenarios and simulated driving tasks. Among them: the high-simulation driving simulator includes 3 AOC display screens (visual simulation units), 1 set of Logitech force feedback steering wheel pedal cover control system (motion simulation unit), 1 set of audio external speaker equipment (audio simulation unit) and a high-performance computer.

[0070] The experiment adopts a two-factor (3×6) within-group design. The independent variables are task difficulty (low, medium, and high difficulty) and time (6 time points, 10, 20, 30, 40, 50, and 60 minutes after the start of the task). This experiment uses the N-Back subtask to adjust the task difficulty, which is divided into low difficulty (driving task only), medium difficulty (driving task plus 1-Back subtask), and high difficulty (driving task plus 2-Back subtask). The N-Back subtask is carried out synchronously with the main driving task in the form of voice broadcast. The subjects need to respond according to the rules of the N-Back subtask, that is, quickly press the button installed on the steering wheel. The N-Back task rules in this embodiment are: the electronic voice randomly broadcasts 0 to 9 in Mandarin, and the subjects need to judge whether the current number heard is the same as the first number (1-Back) or the second number (2-Back) heard before. If they are the same, they need to immediately press the response button installed on the steering wheel.

[0071] In order to eliminate the influence of irrelevant variables, this study arranged the subjects to perform a certain task difficulty experiment in the same time period for three consecutive days, and the order of task difficulty was randomly arranged on three days. Before the first experiment began, the main experimenter introduced the basic process and precautions of the experiment to the subjects. After filling in the basic information and signing the informed consent form, the subjects needed to complete a 15-minute simulation experiment. The simulation training includes driving training, N-Back training and combination training. The formal experiment began after the simulation training. In the experiment, the subjects needed to complete a total of 1 hour of simulated driving tasks, including quickly completing a 100-point NASA-TLX subjective mental workload score every 10 minutes (NASA-TLX is the most widely used mental workload assessment tool in the world and is generally considered to be the gold standard for mental workload assessment).

[0072] The Tobii Pro X3-120 telemetry eye tracker was installed at the bottom center of the display screen of the simulated driving system to collect eye movement signals. The acquisition frequency was 120Hz, the accuracy was 0.24°, the precision was 0.4°, and the allowed head movement distance was 50-90 cm. In order to compare and verify the effectiveness of this solution, this experiment also collected EEG, ECG and skin conduction data during the task. The experiment finally obtained driving data of 24 subjects.

[0073] Step 2: Preprocess the eye movement data collected in step 1 and extract the time series signal features, consider the time-varying characteristics of mental workload, and extract the time series information as the feature variable.

[0074] The eye movement signals extracted in this step include eye movement points, fixation points, pupils, and blink indicators. The processing and calculation methods are briefly described as follows:

[0075] (1) Eye movement point processing: including interpolation and noise reduction

[0076] The maximum interval length is set to 75ms, and the signal loss shorter than this interval length is compensated by linear interpolation.

[0077] The linear interpolation formula is as follows:

[0078] Among them, the coordinates (x 1 ,y 1 ) and (x 0 ,y 0 ) is the value of the two end points of the interpolation interval, (x, y) represents the interpolation points of the interval to be interpolated; it should be noted that the meaning of linear interpolation is to use the straight line formed by the two end points of the interval to be interpolated to approximately replace the points to be interpolated.

[0079] Use the sliding mean filtering method to complete data noise reduction;

[0080] The principle of sliding mean filtering is: suppose the number of input sampling points is N, and use k times sliding mean filtering, then the first data output by the filter is after k sampling cycles, that is, the k input data are averaged first, and the average result is x 0 =(x 1 +x 2 +...+x k ) / 8, after 8 cycles, the sliding input of data can be realized, and the output of the filter is x i =((7*x i -1)+x i +7) / 8, i=1,2......k.

[0081] (2) Gaze point extraction: Set the angular velocity calculation window length to 20ms and the angular velocity threshold to 30° / s. When the angular velocity is greater than 30° / s, it is classified as a saccade, and when it is less than or equal to 30° / s, it is classified as a gaze point.

[0082] Set the maximum time threshold between fixations to 75ms and the maximum angular velocity threshold between fixations to 0.5°; when the interval between two adjacent fixations is less than 75ms and the interval angle is less than 0.5°, the two adjacent fixations are merged into one. Set the minimum fixation time threshold to 60ms, and delete fixations with a fixation time shorter than this threshold.

[0083] (3) Pupil processing: including data interpolation and signal noise reduction. The minimum pupil diameter is set to 2 mm, and the data smaller than the minimum pupil diameter is defined as an outlier and removed. After the outliers are removed, linear interpolation is used for linear interpolation processing, and then the sliding mean filter method is used to complete the signal noise reduction. The principles and calculation methods of linear interpolation and sliding mean filtering are as described above.

[0084] (4) Blink processing: Set the maximum time threshold to 350ms and the minimum time threshold to 75ms. When the left and right eyes are closed at the same time and the closing time is between 75ms and 350ms, it is classified as blinking. After completing the preprocessing of the eye movement data, extract the features of the eye movement signal, and use the correlation coefficient method to obtain the eye movement features with high correlation with mental workload. The eye movement features selected in this embodiment include pupil diameter, distance between fixation points, blink frequency, and eye saccade frequency.

[0085] Step 3: Construct a regression prediction model for mental workload

[0086] This example uses three typical machine learning regression algorithms, namely linear regression, neural network and Gaussian process regression, to predict the driver's mental workload, and compares the prediction effects of eye movement features on the driver's mental workload under these three algorithms. 2 ), mean absolute error (MAE) and root mean square error (RMSE) are used to measure the estimation accuracy of the model (see Formula 2-Formula 4). The three models trained with the collected data are comprehensively compared with R 2 , MAE, and RMSE, the Gaussian process regression model has the best prediction effect.

[0087] In the following formula, is the predicted value of mental workload, y is its observed value, is the average value of the observed value of mental workload, and n is the number of samples. 2 Used to measure the overall prediction effect of the model; MAE can reflect the actual situation of the error between the predicted value and the true value; RMSE can measure the deviation between the observed value and the true value. The smaller the value, the higher the accuracy of the prediction model.

[0088]

[0089] Step 3.1: Perform feature standardization

[0090] Before building the mental workload regression prediction model, the input features need to be transformed. Feature transformation is an operation before the data is input into the machine learning algorithm. It transforms a set of features into a new set of features while retaining as much information as possible about the original features. In distance-based algorithms, features with larger values ​​get more weight because distance is calculated using the values ​​of data points. In linear models and algorithms based on gradient descent optimization, feature scaling is crucial because it will be difficult to converge to the global if data of different sizes are input. Using values ​​of the same range will reduce the burden of algorithm learning. Common feature transformation methods include standardization, normalization, and logarithmic transformation. In machine learning, the most widely used feature transformation method is standardization. The calculation formula is:

[0091]

[0092] Among them: x is the input feature; x mean is the mean of the feature; x stddev is the standard deviation of the feature; x std is the standardized result;

[0093] Step 3.2: Build a linear regression model

[0094] The processing and acquisition of experimental data were implemented according to the above process. The experimental results showed that the determination coefficient (R 2 ) is 0.27, the root mean square error (RMSE) is 14.81, and the mean absolute error (MAE) is 11.70.

[0095] The linear regression model is constructed as follows:

[0096] Step 3.2.1: Obtain the eye movement features and subjective mental workload of the subjects required for model construction according to the above process, and introduce the timing points of eye movement signal collection as the timing information input features of the model. It is assumed that there is a linear relationship between the input features and the output subjective mental workload score.

[0097] Step 3.2.2: For a given dataset D{X, Y}, X is the input feature [x 1 , x 2 , ..., x n ], Y is the output variable, then the mathematical model of linear regression is:

[0098]

[0099] In this embodiment, the actual meaning of X is the eye movement signal features and the introduced timing information features, and Y is the subjective mental workload score corresponding to the input features.

[0100] The vector form is expressed as:

[0101]

[0102] Where w = [w 1 ,w 2 ,...,w n ] T .

[0103] Step 3.2.3: After obtaining the linear regression model of the input features and the output mental workload score, w and b are obtained by minimizing the loss function, and the regression model can be obtained. The loss function used to obtain w and b is the mean square error (MSE) minimization method in this embodiment. The mathematical expression of the process is:

[0104]

[0105] Step 3.3: Build a neural network model

[0106] The processing and acquisition of experimental data were implemented according to the above process. The experimental results showed that the determination coefficient (R 2 ) is 0.34, the root mean square error (RMSE) is 11.06, and the mean absolute error (MAE) is 14.09.

[0107] The neural network model is constructed as follows:

[0108] Step 3.3.1: Build the input layer, hidden layer, and output layer of the neural network, and initialize the weights of each neuron. The activation function of the hidden layer node uses the Sigmoid function, which is:

[0109]

[0110] The constructed neural network model structure is as follows Figure 4 As shown;

[0111] Step 3.3.2: For a given dataset D = {X, Y}, X is the input feature [x 1 , x 2 ,...x n ], Y is the output variable, where the actual meaning of X is the eye movement signal characteristics and the introduced time series information characteristics, and Y is the subjective mental workload score corresponding to the input characteristics. The model is iteratively trained through the data set, and the network parameters are updated according to the gradient descent method. The neural network adjusts the weight vector in a way that minimizes the error, and finally controls the error within the required range. The mathematical expression of the error is:

[0112]

[0113] Among them, Y is the expected output, O is the network output, and m is the number of output nodes.

[0114] Step 3.4: Construction of Gaussian process regression model

[0115] The processing and acquisition of experimental data were implemented according to the above process. The experimental results showed that the determination coefficient (R 2 ) is 0.71, the root mean square error (RMSE) is 6.90, and the mean absolute error (MAE) is 9.40.

[0116] The Gaussian process regression model is constructed as follows:

[0117] Step 3.4.1: Assume that the function f(x) of the input training set X and the output Y has a Gaussian distribution. The Gaussian distribution is specified by a mean function m(x) and a covariance function K(x,x').

[0118] Step 3.4.2: Obtain x by calculating the posterior distribution of the acquired training data * , the posterior distribution is:

[0119] P(f(x * )|X,Y,x * )=N(f(x * ) * ,∑ * ) (Formula 11)

[0120] Among them, μ * and∑ * are the mean and covariance of the posterior distribution, respectively. The mean and covariance are given by:

[0121]

[0122] In the formula, K(x * ,X)、K(X,X)、K(x * ,x * ) represent x * and X, X and X, x * With x * The covariance matrix between is the observation noise variance.

[0123] Step 3.4.3: Select the covariance calculation method. The covariance calculation method uses the Matern5 / 2 function, and its specific mathematical expression is as follows:

[0124]

[0125] In the formula, Represents xi With x j The Euclidean distance between l is the characteristic length dimension, σ f is the standard deviation of the signal.

[0126] Step 3.5: Select the model

[0127] By comparing the determination coefficients (R 2 ), root mean square error (RMSE), mean absolute error (MAE), and Gaussian process regression model are all optimal, so the Gaussian process regression model is selected as the model for regression prediction of drivers' mental workload in long-term driving tasks.

[0128] Step 4: Use the mental workload regression prediction model to calculate the mental workload prediction value, and use the auxiliary decision-making module to remind the driver of the mental workload level and make corresponding driving suggestions.

[0129] This embodiment adopts a non-contact eye tracker fixed in front of the driver to obtain long-term real-time data, and uses a Gaussian process regression model to build a regression prediction model. After iterative training, R 2 The maximum value of is 0.71. The prediction accuracy of the Gaussian process regression model under different modes is shown in Table 1.

[0130] Table 1: Prediction accuracy of Gaussian process regression model under different modes

[0131]

[0132]

[0133] In the Gaussian process regression model with different modalities without introducing temporal information features, the R 2 is 0.60, the R of eye movement + ECG 2 is 0.70, the R of eye movement + EEG + skin conduction 2 The R of eye movement + EEG + ECG is 0.71. 2 The R of eye movement + skin electricity + heart rate is 0.68. 2 The R of eye movement + EEG + skin electricity + heart electricity is 0.71. 2 It is 0.73.

[0134] With the increase of input modes, the model accuracy of Gaussian process regression model has been improved to a certain extent. After the introduction of time series information features, all types of models R 2 All of them have been improved, especially the model R constructed using the single modality of eye movement signals. 2The improvement is significant. Among them, the single eye movement modal model is improved from 0.60 to 0.71, the eye movement + ECG modal model is improved from 0.70 to 0.72, the eye movement + EEG + EGV modal model is improved from 0.71 to 0.73, the eye movement + EEG + ECG modal model is improved from 0.68 to 0.71, the eye movement + EGV + ECG modal model is improved from 0.71 to 0.74, and the eye movement + EEG + EGV + ECG modal model is improved from 0.73 to 0.75. It can be concluded that the introduction of timing information features can improve the accuracy of the model, and the single modal input using only eye movement signals can achieve the same prediction effect of multi-modal input. In addition, the present embodiment only needs to collect eye movement signals, and the equipment used is a non-contact eye tracker fixed in front of the driver. Therefore, the driver does not need to wear additional physiological acquisition equipment, thereby avoiding interference with normal driving operations. Compared with the traditional classification model, the regression prediction model of mental workload constructed by the present embodiment can obtain an accurate score of the driver's mental workload, avoiding the defect that the classification model is not sensitive enough to the change of mental workload, and to a large extent ensuring that the driver has a more appropriate understanding of his own mental workload.

[0135] From the above comparison we can see that:

[0136] (1) The existing scheme needs to collect complex physiological signals, which requires the driver to wear multiple physiological signal collection devices. These devices are bulky when combined, which affects driving operations. The scheme of this embodiment only collects eye movement signals and obtains them in a non-contact manner, which will not interfere with the driving process. The relevant collection equipment is lightweight and has good installation compatibility with the vehicle.

[0137] (2) Existing solutions usually build models through test data of a shorter period of time, while in actual driving tasks, drivers often need to perform long-term driving tasks. Existing solutions do not take into account the possible changes in indicators characterizing mental workload during long-term driving. Therefore, the accuracy of predicting mental workload in long-term driving tasks will be reduced for models built only based on short-term data. To address this issue, the present embodiment takes the time-varying characteristics of mental workload into consideration and improves the modeling process by introducing time series factors (such as driving time), thereby improving the model's compatibility with time-varying characteristics and prediction accuracy.

[0138] (3) In the prior art solutions, the evaluation of the driver's mental workload mostly relies on the classification model. This method may result in the same output results when the mental workload varies greatly, which is easy to mislead the driver in actual driving applications and make him make inappropriate judgments on whether he is suitable to continue driving. This embodiment scheme proposes a more sophisticated mental workload regression prediction model (which can obtain a mental workload prediction value between 0-100 points (which is more accurate than the previous classification models (such as low, medium and high categories))), which can achieve more accurate mental workload evaluation in a long-term dynamic environment and has better sensitivity and accuracy in practical applications.

[0139] As a preferred embodiment of the present invention, the feature vector data format constructed by the method includes pupil diameter, fixation point distance, blink frequency, eye saccade frequency and data timing information. Other ways of changing or adding variables are all within the feature data vector format of the method. The device used by the method to collect eye movement signals can be a non-contact eye tracker, or a camera with an eye movement signal acquisition function.

[0140] As a preferred embodiment of the present invention, as an alternative technical solution, a method for predicting mental workload based on radar waves is provided. Compared with eye movement signals, radar waves also have the characteristics of non-contact and convenient measurement.

[0141] As a preferred embodiment of the present invention, when in use, data processing and data cleaning are performed based on the background server to remove abnormal data. After data preprocessing, the eye movement characteristics and driving time are input into the pre-trained Gaussian process regression model to obtain the real-time mental workload score of the driver.

[0142] A prediction system based on the above-mentioned driving mental workload prediction method based on non-contact eye movement signals comprises a data acquisition module, a data processing module, a mental workload regression prediction model module and a decision-making support module; wherein:

[0143] The data acquisition module is used to collect eye movement signals, with a collection frequency of 120 Hz, a precision of 0.24°, an accuracy of 0.4°, and an allowable head movement distance of 50-90 cm;

[0144] The data processing module is used to perform data preprocessing and eye movement signal feature extraction on the collected eye movement signals, and the extracted eye movement signals include eye movement points, fixation points, pupils, and blink indicators;

[0145] The mental workload regression prediction model module includes a linear regression model, a neural network model and a Gaussian process regression model, which are used to evaluate mental workload in a long-term dynamic environment;

[0146] The auxiliary decision-making module is used to inform the driver of relevant information and make driving behavior suggestions after the mental workload regression prediction model module obtains the mental workload prediction value, so as to assist the driver in making judgments and ensure driving safety;

[0147] Specifically, after obtaining the predicted level of mental load, the auxiliary decision-making module can output corresponding suggestions according to the mental load level. Specifically, when starting to drive, the system-predicted mental load value is directly output at regular intervals (such as 10 minutes); as time goes by, if the system-predicted mental load value increases relative to the initial load value, the system-predicted mental load value and the "increased driving load" reminder are output at the same time; if the system-predicted mental load value increases by more than 30% relative to the initial load value, the system-predicted mental load value is output as a specific percentage of increased driving load, and the driver is advised to select a nearby place to take a proper rest before driving to ensure driving safety; if the system-predicted mental load value increases by more than 50% relative to the initial load value, the system-predicted mental load value is output as a specific percentage of increased driving load, and the driver is advised that the driving load is too high, and it is recommended to stop the car safely as soon as possible and take a full rest before driving to ensure driving safety.

[0148] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0149] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for predicting driving mental workload based on non-contact eye movement signals, characterized by: include: Step 1: Collect the driver's eye movement time series data; Step 2: Preprocess the collected eye movement time series data and extract the time series signal features. Considering the time-varying characteristics of mental workload, the time series information is extracted as the feature variable. Among them: eye movement timing signals include eye movement points, fixation points, pupils and blink indicators; Step 3: Build a mental workload regression prediction model Based on the features of eye movement signals and time series information, linear regression, neural network and Gaussian process regression machine learning regression algorithms are used to predict the driver's mental workload respectively, and the Gaussian process regression model is determined as the mental workload regression prediction model; Step 4: Use the mental workload regression prediction model to calculate the mental workload prediction value, and use the driving assistance decision module to remind the driver of the mental workload level and provide corresponding driving suggestions.

2. The method for predicting driving mental workload based on non-contact eye movement signals according to claim 1, wherein: The eye movement processing process includes: (1) Data interpolation: Assume that the maximum interval length is 75ms. The signal loss less than the maximum interval is compensated by linear interpolation. The linear interpolation formula is: Among them, the coordinates (x1, y1) and (x0, y0) are the values ​​of the two end points of the interpolation interval, and the coordinates (x, y represent the interpolation points of the required interpolation interval; (2) Data denoising: Use the sliding mean filtering method to perform data denoising.

3. The method for predicting driving mental workload based on non-contact eye movement signals according to claim 1, wherein: The fixation point extraction process includes: (1) Set the angular velocity calculation window length to 20ms and the angular velocity threshold to 30° / s. Angular velocities greater than 30° / s are classified as saccades, and angular velocities less than or equal to 30° / s are classified as fixations. (2) Set the maximum time threshold between fixations to 75ms and the maximum angular velocity threshold between fixations to 0.5°; when the interval between two adjacent fixations is less than 75ms and the interval angle is less than 0.5°, merge the two adjacent fixations into one; (3) Set the minimum fixation time threshold to 60ms and delete the fixation points with a fixation time lower than the threshold.

4. The method for predicting driving mental workload based on non-contact eye movement signals according to claim 1, wherein: Pupil processing includes: (1) Data interpolation: The minimum pupil diameter is set to 2 mm, and the data smaller than the minimum pupil diameter is defined as an outlier and removed. After the outliers are removed, linear interpolation is performed using the linear interpolation method; (2) Data denoising: Use the sliding mean filtering method to perform data denoising.

5. The method for predicting driving mental workload based on non-contact eye movement signals according to claim 1, wherein: The blink processing process includes: (1) Set the maximum time threshold to 350ms and the minimum time threshold to 75ms; when both eyes are closed at the same time and the closing time is between 75ms and 350ms, it is classified as a blink; (2) After completing the preprocessing of the eye movement data, the correlation coefficient method was used to obtain the eye movement features that were highly correlated with mental workload; In the process of using the correlation coefficient method to obtain eye movement features that are highly correlated with mental workload: considering the time-varying characteristics of mental workload, the extracted eye movement signals are time-series marked, and the time-series information is used as a supplementary feature variable; this time-series information usually defaults to the driving time when the eye movement data is collected.

6. The method for predicting driving mental workload based on non-contact eye movement signals according to claim 1, wherein: In step 3, when using the machine learning regression algorithms of linear regression, neural network and Gaussian process regression to predict the driver's mental workload, the determination coefficients R of the linear regression model, neural network model and Gaussian process regression model are compared. 2 , root mean square error RMSE and mean absolute error MAE to determine the optimal model, where: in, is the predicted value of mental workload, y is the observed value of mental workload, is the average value of the mental workload observations, and n is the number of samples.

7. The method for predicting driving mental workload based on non-contact eye movement signals according to claim 6, wherein: Before establishing the linear regression model, neural network model, and Gaussian process regression model, it is necessary to perform feature conversion on the eye movement input features extracted in step 2. The feature conversion method is as follows: Where: x is the input feature; x mean is the mean of the feature; x stddev is the standard deviation of the feature; x std The results are normalized.

8. The method for predicting driving mental workload based on non-contact eye movement signals according to claim 6, wherein: The construction process of the Gaussian process regression model includes: Step 3.4.1: Assume that the function f(x) between the input training set X and the output Y has a Gaussian distributed random process; Here, the Gaussian distribution is specified by a mean function m(x) and a covariance function K(x,x'); Step 3.4.2: Obtain x by calculating the posterior distribution of the acquired training data * , the posterior distribution is: P(f(x * )|X,Y,x * )=N(f(x * )u * ,∑ * ) Among them, μ * and ∑ * are the mean and covariance of the posterior distribution, respectively, and are given by: Among them, K(x * ,X)、K(X,X)、K(x * ,x * ) represent x * with X, X and X, x * with x * The covariance matrix between is the observation noise variance; Step 3.4.3: Use the covariance calculation method. The covariance calculation method uses the Matern5 / 2 function. The mathematical expression of the Matern5 / 2 function is: Where, Represents x i with x j The Euclidean distance between l is the characteristic length dimension, σ f is the standard deviation of the signal.

9. A driving mental workload prediction system based on non-contact eye movement signals, characterized by: include: A data acquisition module, used to collect the driver's eye movement time series data; A data processing module is used to perform data preprocessing and eye movement timing signal feature extraction on the eye movement signals collected by the data acquisition module; Mental workload regression prediction model module, used for dynamic prediction of mental workload in a long-term dynamic environment; The decision-making support module is used to remind the driver of the mental workload level and provide corresponding driving suggestions. The data acquisition module, data processing module and mental workload regression prediction model module are used to implement the method according to any one of claims 1 to 8.

Images