Fatigue detection model construction method, driver fatigue detection method, device, equipment, vehicle and storage medium

Abstract

The invention discloses a fatigue detection model construction method, a driver fatigue detection method, a driver fatigue detection device, driver fatigue detection equipment, a vehicle and a storage medium. The method comprises the following steps: acquiring a plurality of groups of sets of electroencephalogram signal data at different preset positions; extracting a plurality of statistical features from each group of electroencephalogram signal data at different preset positions and forming a statistical feature matrix; each group of electroencephalogram signal data at different preset positions is decomposed into corresponding wavebands according to different preset frequency ranges, differential entropy features and detrending fluctuation analysis features are extracted from the wavebands respectively, and then a differential entropy feature matrix and a detrending fluctuation analysis feature matrix are formed respectively; forming a fusion feature matrix by using the statistical feature matrix, the differential entropy feature matrix and the de-trending fluctuation analysis feature matrix of each group; establishing a fatigue detection model; and respectively inputting the multiple groups of fusion feature matrixes into a fatigue detection model for training, verification and testing. According to the invention, the accuracy of the fatigue state detection result can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of safe driving, and in particular, to a method for constructing a fatigue detection model, a driver fatigue detection method, device, equipment, vehicle, and storage medium. Background Art

[0002] Drowsy driving is one of the main factors leading to traffic accidents, posing a great threat to traffic safety and personal safety. Therefore, it is very necessary to effectively identify the fatigue state of drivers to avoid traffic accidents. Fatigue detection has become an important topic in the field of driving safety. In the prior art, the driver fatigue state detection methods are mainly divided into three categories. The first category is the detection method based on vehicle motion characteristics, that is, the driver fatigue state is identified by using parameters such as vehicle driving speed, steering wheel angle, and throttle and brake pedal openings. The second category is the detection method based on driver behavior characteristics, that is, the driver fatigue state is judged by the driver's facial features, head posture, and driving posture. The third category is the detection method based on driver physiological signals. The commonly used driver physiological signals include electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, electrooculogram (EOG) signals, and electromyogram (EMG) signals, etc. The driver fatigue state is judged based on one or several of these driver physiological signals.

[0003] Fatigue is a physiological reaction. Therefore, the fatigue detection method based on physiological signals has higher accuracy and reliability. Among them, EEG signals are closely related to brain activities and can accurately reflect the changes in physiological states, and are considered to be one of the most reliable criteria for detecting driving fatigue states. With the development of brain-computer interfaces and the neural field, fatigue detection based on EEG signals has received more and more attention. In recent years, with the rapid development of artificial intelligence, deep learning, as the core technology of artificial intelligence, has been widely applied in various fields, making breakthrough progress in various fields. Therefore, the driver fatigue detection method based on deep learning to identify EEG signals has also received more and more attention from scholars. In the prior art, the driver fatigue detection method based on deep learning to identify EEG signals is mainly divided into two steps: First, a signal processing method is used to pre-extract EEG signal features, and then the obtained EEG signal feature representation is used as the input of the deep learning model to further extract deep features, so as to realize the identification of EEG signals in different fatigue states. However, in this method, most of the power spectral density (PSD) features of EEG signals, and / or entropy value features such as sample entropy and fuzzy entropy are extracted. The features extracted are relatively single, and the EEG signal features cannot be comprehensively mined from multiple dimensions, resulting in the result of detecting the driver's fatigue state by the deep learning model trained with the extracted EEG signal features not being accurate enough in practice. Summary of the Invention

[0004] To solve at least one aspect of the above problems, the present invention provides a method for constructing a fatigue detection model, a driver fatigue detection method, device, equipment, vehicle, and storage medium.

[0005] In a first aspect, the present application provides a method for constructing a fatigue detection model, including the following steps: Step 1: Obtain a set of electroencephalogram (EEG) signal data at several different preset positions of different drivers, and divide it into data subsets under different fatigue states according to a preset fatigue state judgment rule, where the fatigue states include a wake state, a moderate fatigue state, and a severe fatigue state, and the set is a publicly available dataset or a dataset collected through a driving simulation experiment; Step 2: Randomly extract several groups of data from the data subsets under different fatigue states, and divide the several groups of data into a training set, a validation set, and a test set according to a preset ratio; Step 3: Standardize the EEG signal data at several different preset positions in each group of data; Step 4: Extract multiple statistical features from the EEG signal data at several different preset positions in each group of standardized data, and jointly form an m×n statistical feature matrix with the multiple statistical features corresponding to the EEG signal data at several different preset positions in each group, where the multiple statistical features include standard deviation, skewness, kurtosis, root mean square, root amplitude, absolute mean, and peak value, m is the number of positions, and n is the number of items of statistical features; Step 5: Decompose the EEG signal data at several different preset positions in each group of standardized data into corresponding frequency bands according to different preset frequency ranges; Step 6: Extract differential entropy features from the multiple frequency bands obtained by decomposing the EEG signal data at several different preset positions in each group, and jointly form an m×p differential entropy feature matrix with the multiple differential entropy features corresponding to the EEG signal data at several different preset positions in each group, where p is the number of preset frequency ranges; Step 7: Extract detrended fluctuation analysis features from the multiple frequency bands obtained by decomposing the EEG signal data at several different preset positions in each group, and jointly form an m×p detrended fluctuation analysis feature matrix with the multiple detrended fluctuation analysis features corresponding to the EEG signal data at several different preset positions in each group; Step 8: Standardize the m×n statistical feature matrix, the m×p differential entropy feature matrix, and the m×p detrended fluctuation analysis feature matrix in each group; Step 9: Perform feature fusion on the standardized m×n statistical feature matrix, the m×p differential entropy feature matrix, and the m×p detrended fluctuation analysis feature matrix in each group in a matrix splicing manner to form an m×(n + 2p) fusion feature matrix; Step 10: Establish a fatigue detection model based on a convolutional neural network model; Step 11: Input several groups of fusion feature matrices in the training set into the fatigue detection model for training, and input several groups of fusion feature matrices in the validation set into the trained fatigue detection model for validation, and repeat multiple cycles to obtain a trained fatigue detection model; Step 12: Input several groups of fusion feature matrices in the test set into the trained fatigue detection model for testing, and when the test result is qualified, obtain a usable fatigue detection model.

[0006] Preferably, step A is provided between step 2 and step 3: several groups of data are generated by linear synthesis in the training sets in the divided wakeful state, moderate fatigue state, and severe fatigue state.

[0007] Preferably, the fatigue detection model established based on the convolutional neural network model is an input layer, multiple groups of cascaded residual modules and residual downsampling modules, a global average pooling layer, a fully connected layer, and an output layer connected in series in sequence; the main route of the residual module consists of a 3×3 convolutional layer, a Dropout layer, a BN layer, and a Leaky ReLU activation function layer, and the bypass is a skip connection with an identity mapping; the main route of the residual downsampling module consists of a zero-padding layer, a 3×3 convolutional layer with a stride of 2, a Dropout layer, a BN layer, and a Leaky ReLU layer, and the bypass consists of a 1×1 convolutional layer with a stride of 2, a BN layer, and a Leaky ReLU activation function layer.

[0008] Preferably, the normalization process adopts the Min-Max normalization method.

[0009] Preferably, p is 5, the first preset frequency range is 0.5 - 4 Hz, the second preset frequency range is 4 - 8 Hz, the third preset frequency range is 8 - 13 Hz, the fourth preset frequency range is 13 - 32 Hz, and the fifth preset frequency range is 32 - 50 Hz.

[0010] Second aspect, the present application provides a driver fatigue detection method, including the following steps: Step 1: Obtain the electroencephalogram signal data at m different preset positions of the driver to be measured; Step 2: Perform normalization processing on the electroencephalogram signal data at m different preset positions respectively; Step 3: Extract n statistical features from the electroencephalogram signal data at m different preset positions after normalization processing respectively, and jointly form an m×n statistical feature matrix with the n statistical features corresponding to the electroencephalogram signal data at m different preset positions respectively; Step 4: Decompose the electroencephalogram signal data at m different preset positions into corresponding frequency bands according to p different preset frequency ranges respectively; Step 5: Extract differential entropy features from the multiple frequency bands obtained by decomposing the electroencephalogram signal data at m different preset positions respectively, and jointly form an m×p differential entropy feature matrix with the multiple differential entropy features corresponding to the electroencephalogram signal data at m different preset positions respectively; Step 6: Extract detrended fluctuation analysis features from the multiple frequency bands obtained by decomposing the electroencephalogram signal data at m different preset positions respectively, and jointly form an m×p detrended fluctuation analysis feature matrix with the multiple detrended fluctuation analysis features corresponding to the electroencephalogram signal data at m different preset positions respectively; Step 7: Perform normalization processing on the m×n statistical feature matrix, the m×p differential entropy feature matrix, and the m×p detrended fluctuation analysis feature matrix respectively; Step 8: Perform feature fusion on the normalized m×n statistical feature matrix, the m×p differential entropy feature matrix, and the m×p detrended fluctuation analysis feature matrix in a matrix splicing manner to form an m×(n + 2p) fusion feature matrix; Step 9: Input the fusion feature matrix into the fatigue detection model to obtain the fatigue degree result of the driver, where the fatigue detection model is constructed by any one of the above-mentioned fatigue detection model construction methods.

[0011] In a third aspect, the present application provides a driver fatigue detection device, the device comprising: a receiving module configured to acquire electroencephalogram signal data at m different preset positions of a driver to be measured; a first data processing module configured to perform normalization processing on the electroencephalogram signal data at the m different preset positions respectively; a second data processing module configured to extract n statistical features from the electroencephalogram signal data at the m different preset positions after normalization processing respectively, and jointly form an m×n statistical feature matrix with the n statistical features corresponding to the electroencephalogram signal data at the m different preset positions respectively; a third data processing module configured to decompose the electroencephalogram signal data at the m different preset positions into corresponding frequency bands respectively according to p different preset frequency ranges; a fourth data processing module configured to extract differential entropy features from the multiple frequency bands into which the electroencephalogram signal data at the m different preset positions are decomposed respectively, and jointly form an m×p differential entropy feature matrix with the multiple differential entropy features corresponding to the electroencephalogram signal data at the m different preset positions respectively; a fifth data processing module configured to extract detrended fluctuation analysis features from the multiple frequency bands into which the electroencephalogram signal data at the m different preset positions are decomposed respectively, and jointly form an m×p detrended fluctuation analysis feature matrix with the multiple detrended fluctuation analysis features corresponding to the electroencephalogram signal data at the m different preset positions respectively; a sixth data processing module configured to perform normalization processing on the m×n statistical feature matrix, the m×p differential entropy feature matrix and the m×p detrended fluctuation analysis feature matrix respectively; a seventh data processing module configured to perform feature fusion on the normalized m×n statistical feature matrix, the m×p differential entropy feature matrix and the m×p detrended fluctuation analysis feature matrix in a matrix splicing manner to form an m×(n + 2p) fusion feature matrix; a detection module configured to input the fusion feature matrix into a fatigue detection model to obtain a fatigue degree result of the driver, wherein the fatigue detection model is constructed by the method for constructing a fatigue detection model described in any one of the above.

[0012] In a third aspect, the present application provides a driver fatigue detection device, the device comprising a memory and a processor, and a computer program is stored on the memory, and when the computer program is executed by the processor, the above-described driver fatigue detection method is implemented.

[0013] In a fourth aspect, the present application provides a vehicle comprising the above-described driver fatigue detection device.

[0014] In a fifth aspect, the present application provides a storage medium storing computer-readable instructions, and when the instructions are run by a processor, the above-described method is executed.

[0015] A method for constructing a fatigue detection model, a driver fatigue detection method, device, equipment, vehicle, and storage medium according to the present invention have the following beneficial effects:

[0016] (1) In this application, EEG signal features are extracted from multiple dimensions, including seven statistical features and two non-linear dynamics features, and different feature matrices are fused by matrix splicing. Compared with extracting a single feature, the fused features express the EEG signal features more comprehensively, which helps to improve the accuracy and reliability of the fatigue detection model trained by the EEG signal features extracted by the method of this application in detecting the fatigue state of drivers in practice.

[0017] (2) When establishing the fatigue detection model in this application, a form of staggered series connection of residual modules and downsampling residual modules is adopted. The residual structure can inhibit network degradation and accelerate network convergence. At the same time, the skip connection has the ability of feature fusion and enhanced information flow. And using the residual downsampling module to replace the pooling layer to complete the downsampling operation can avoid the information loss caused by the pooling layer.

[0018] (3) In this application, a linear synthesis method is used to synthesize new EEG signal samples in the training set to expand the training set, which helps to inhibit model overfitting and improve the robustness of the model. Description of the Drawings

[0019] In order to better understand the above and other objects, features, advantages, and functions of the present invention, reference may be made to the embodiments shown in the drawings. The same reference numerals in the drawings refer to the same components. Those skilled in the art should understand that the drawings are intended to schematically illustrate the preferred embodiments of the present invention and have no restrictive effect on the scope of the present invention. Each component in the drawings is not drawn to scale.

[0020] Figure 1 Shows a flowchart of a method for constructing a fatigue detection model according to an embodiment of the present invention;

[0021] Figure 2 Shows a preset position distribution diagram of a method for constructing a fatigue detection model according to an embodiment of the present invention;

[0022] Figure 3 Shows a structural schematic diagram of a fatigue detection model of a method for constructing a fatigue detection model according to an embodiment of the present invention;

[0023] Figure 4 Shows a structural schematic diagram of a residual module of a method for constructing a fatigue detection model according to an embodiment of the present invention;

[0024] Figure 5Shows a schematic structural diagram of a residual downsampling module of a fatigue detection model construction method according to an embodiment of the present invention;

[0025] Figure 6 Shows a flowchart of a driver fatigue detection method according to an embodiment of the present invention;

[0026] Figure 7 Shows a framework diagram of a driver fatigue detection device according to an embodiment of the present invention. Detailed implementation manners

[0027] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist in understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0028] As used herein, the term "including" and its variations mean open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "an exemplary embodiment" and "an embodiment" mean "at least one exemplary embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. There may also be other explicit and implicit definitions below.

[0029] To at least partially solve one or more of the above problems and other potential problems, embodiments of the present disclosure propose a fatigue detection model construction method, as Figure 1 shown, including the following steps:

[0030] Step 1: Obtain a set of electroencephalogram (EEG) signal data at several different preset positions of different drivers, and divide it into data subsets in different fatigue states according to a preset fatigue state judgment rule, where the fatigue states include a wake state, a moderate fatigue state, and a severe fatigue state, and the set is a publicly available dataset in the prior art or a dataset collected through a driving simulation experiment.

[0031] Specifically, each group of EEG signal data has corresponding electrooculogram (EOG) signal data. The preset fatigue state judgment rule is to divide the driver fatigue state level according to the PERCLOS (Percent of Eye Closure) index. The PERCLOS index represents the percentage of the eye closure time within a certain time, and is calculated by the following two formulas:

[0032] ,

[0033] ,

[0034] Among them, represents the total time, represents the blink time, represents the eye-closed time, represents the fixation time, represents the saccade time. When the PERCLOS value is in the interval [0, 0.35), it indicates a wakeful state; when the PERCLOS value is in the interval [0.35, 0.7), it indicates a moderate fatigue state; when the PERCLOS value is in the interval [0.7, 1], it indicates a severe fatigue state.

[0035] As Figure 2 shown, there are 17 preset positions. During acquisition, the electrode positions of the 17 channels of the acquisition device are CP1, CP2, P1, PZ, P2, P03, P0Z, P04, 01, 0Z, 02, FT7, FT8, T7, T8, TP7, TP8 respectively, and CPZ is the reference electrode. The electroencephalogram signals collected by each electrode , or x t and t = 1, 2... i, denoted as , then a set of data is .

[0036] Step 2: Randomly select several groups of data from the data subsets in different fatigue states, and divide the several groups of data into a training set, a validation set, and a test set according to a preset ratio. Specifically, the preset ratio is 8:1:1. In a preferred embodiment, it further includes step A: generating several groups of data in the training sets in the wakeful state, moderate fatigue state, and severe fatigue state by linear synthesis; specifically, the specific process of generating several groups of data by linear synthesis is: randomly select two groups of data and simultaneously and randomly from any one of the training sets in the three states, and randomly take a number a in the interval (0, 1) to synthesize a set of data The expression of is:

[0037] ,

[0038] Step 3: Standardize the electroencephalogram signal data at several different preset positions in each group of data; preferably, the standardization method used is the Min-Max standardization method, so that each electroencephalogram signal in each group of data Transform to between [0, 1]. Specifically, the expression for processing the EEG signal through Min-Max normalization is:

[0039] ,

[0040] The EEG signal collected by each electrode After being processed by the Min-Max normalization method, it changes to , then each normalized EEG signal is , or expressed as and t = 1, 2... i.

[0041] Step 4: Extract multiple statistical features from each group of EEG signal data at several different preset positions after the normalization process, and jointly form an m×n statistical feature matrix with the multiple statistical features corresponding to each group of EEG signal data at several different preset positions. Among them, the multiple statistical features include standard deviation, skewness, kurtosis, root mean square, root amplitude, absolute average value, peak value, etc. m is the number of positions, and n is the number of items of statistical features; specifically, as Figure 2 shown, if the electrical signals at 17 preset positions are obtained, then m is 17, and since the multiple statistical features include 7 items: standard deviation, skewness, kurtosis, root mean square, root amplitude, absolute average value, and peak value, then n is 7. The expressions for calculating the standard deviation, skewness, kurtosis, root mean square, root amplitude, absolute average value, and peak value of each normalized EEG signal are respectively:

[0042] , where, ;

[0043] ,

[0044] ,

[0045] ,

[0046] ,

[0047] ,

[0048] ,

[0049] Then the 7 statistical features corresponding to each group of 17 EEG signal data at different preset positions jointly form a 17×7 statistical feature matrix as:

[0050] .

[0051] Step 5: Decompose the EEG signal data at several different preset positions in each group after standardization into corresponding frequency bands according to different preset frequency ranges; preferably, the number of preset frequency ranges is 5, the first preset frequency range is 0.5 - 4 Hz, the second preset frequency range is 4 - 8 Hz, the third preset frequency range is 8 - 13 Hz, the fourth preset frequency range is 13 - 32 Hz, and the fifth preset frequency range is 32 - 50 Hz, so that the EEG signal data at several different preset positions in each group are decomposed into corresponding first, second, third, fourth, and fifth frequency bands according to the first, second, third, fourth, and fifth preset frequency ranges respectively.

[0052] Step 6: Extract differential entropy features from the multiple frequency bands into which the EEG signal data at several different preset positions in each group are decomposed respectively, and jointly form an m×p differential entropy feature matrix with the multiple differential entropy features corresponding to the EEG signal data at several different preset positions in each group, where p is the number of preset frequency ranges; in a preferred embodiment, p is 5, and the 17 signals in each group of data are decomposed into corresponding first, second, third, fourth, and fifth frequency bands according to the first, second, third, fourth, and fifth preset frequency ranges respectively, and then differential entropy features are extracted from each frequency band. The expression for extracting differential entropy is:

[0053] ,

[0054] Then the 5 differential entropy features corresponding to the EEG signal data at 17 different preset positions in each group jointly form a 17×5 differential entropy feature matrix as follows:

[0055] .

[0056] Step 7: Extract detrended fluctuation analysis features from the multiple frequency bands into which the EEG signal data at several different preset positions in each group are decomposed respectively, and jointly form an m×p detrended fluctuation analysis feature matrix with the multiple detrended fluctuation analysis features corresponding to the EEG signal data at several different preset positions in each group; in a preferred embodiment, p is 5, and the 17 signals in each group of data are decomposed into corresponding first, second, third, fourth, and fifth frequency bands according to the first, second, third, fourth, and fifth preset frequency ranges respectively, and then detrended fluctuation analysis features are extracted from each frequency band. The method for extracting detrended fluctuation analysis features includes the following steps:

[0057] Step a: Calculate the cumulative deviation of the time series to obtain a new series :

[0058] ,

[0059] wherein, is the mean value of the time series .

[0060] Step b: Divide into subsequences of equal length s, and in each subsequence, fit a linear trend using the least squares method , subtract the linear trend from each subsequence to obtain a detrended sequence, and calculate the root mean square of the detrended sequence :

[0061] .

[0062] Step c: Take different lengths s, repeat Step b, and calculate the corresponding .

[0063] Step d: Plot the relationship curve between s and on a double logarithmic coordinate system, and calculate the slope by the least squares method, which is the DFA index.

[0064] Then, the 5 detrended fluctuation analysis features corresponding to the EEG signal data at 17 different preset positions in each group together form a 17×5 detrended fluctuation analysis feature matrix as follows:

[0065] .

[0066] Step 8: Standardize each of the m×n statistical feature matrix, m×p differential entropy feature matrix, and m×p detrended fluctuation analysis feature matrix for each group. In the preferred embodiment, the 17×7 statistical feature matrix, 17×5 differential entropy feature matrix, and 17×5 detrended fluctuation analysis feature matrix for each group of data are standardized respectively. Preferably, the standardization method used is the Min - Max standardization method, which converts each feature to the range [0,1], helping to eliminate the differences in numerical magnitudes between different features; specifically, the formula for converting each matrix table is:

[0067] ,

[0068] wherein, represents the eigenvalue at the i - th row and j - th column in a matrix; represents the eigenvalue at the i - th row and j - th column after standardizing the matrix; m is the total number of rows of the matrix, n is the total number of columns of the matrix; is the maximum eigenvalue in the matrix; is the minimum eigenvalue in the matrix.

[0069] Step 9: Perform feature fusion on the standardized m×n statistical feature matrix, m×p differential entropy feature matrix, and m×p detrended fluctuation analysis feature matrix of each group in a matrix splicing manner to form a fused feature matrix of m×(n + 2p); in a preferred embodiment, perform feature fusion on the standardized 17×7 statistical feature matrix, 17×5 differential entropy feature matrix, and 17×5 detrended fluctuation analysis feature matrix of each group in a matrix splicing manner to form a 17×17 fused feature matrix.

[0070] Step 10: Establish a fatigue detection model based on a convolutional neural network model; in a preferred embodiment, as Figures 3 to 5 shown, the fatigue detection model established based on the convolutional neural network model is an input layer, multiple groups of cascaded residual modules and residual downsampling modules, a global average pooling layer, a fully connected layer with 3 neurons, and an output layer connected in series; preferably, the number of groups of cascaded residual modules and residual downsampling modules is 2; the main route of the residual module consists of a 3×3 convolutional layer, a Dropout layer, a BN layer, and a Leaky ReLU activation function layer, and the bypass is a skip connection with an identity mapping; the main route of the residual downsampling module consists of a zero-padding layer, a 3×3 convolutional layer with a stride of 2, a Dropout layer, a BN layer, and a Leaky ReLU layer, and the bypass consists of a 1×1 convolutional layer with a stride of 2, a BN layer, and a Leaky ReLU activation function layer; the output layer is a Softmax activation function layer, which is used to output the classification confidence of the three-state electroencephalogram signals.

[0071] Step 11: Input several groups of fused feature matrices in the training set into the fatigue detection model for training, and input several groups of fused feature matrices in the validation set into the trained fatigue detection model for validation, and repeat multiple cycles to obtain a trained fatigue detection model. Preferably, before training, select a suitable optimization algorithm and loss function, and set a suitable batchsize (the number of data provided to the model at one time) and learning rate; more preferably, select the Adam optimization algorithm for the optimization algorithm, select the cross-entropy loss function for the loss function, set the batchsize to 50, and the learning rate to 0.001.

[0072] Step 12: Input several groups of fused feature matrices in the test set into the trained fatigue detection model for testing. When the test results are qualified, an available fatigue detection model is obtained, that is, input the data groups in the waking state in the test set into the fatigue detection model to obtain the results representing the waking state, input the data groups in the moderate fatigue state into the fatigue detection model to obtain the results representing the moderate fatigue state, and input the data groups in the severe fatigue state into the fatigue detection model to obtain the results representing the severe fatigue state.

[0073] In a specific embodiment:

[0074] Step 1: Use the SEED-VIG driver fatigue detection public dataset provided by Shanghai Jiao Tong University. This dataset contains EEG signals collected from 21 subjects participating in 23 simulated driving experiments. Each experiment collected 885 EEG signals of 17 channels, and the signal sampling frequency was 200 Hz. The 17 electrode positions are as Figure 2 shown as CP1, CP2, P1, PZ, P2, P03, P0Z, P04, 01, 0Z, 02, FT7, FT8, T7, T8, TP7, TP8. According to the preset fatigue state judgment rule, the dataset is divided into a sub-dataset in the waking state, which includes 7405 groups of EEG signals; a sub-dataset in the moderate fatigue state, which includes 8901 groups of EEG signals; and a sub-dataset in the severe fatigue state, which includes 4049 groups of EEG signals.

[0075] Step 2: Randomly extract 4000 groups of data from the data sub-datasets in different fatigue states, and divide the 4000 groups of data into a training set, a validation set, and a test set according to 8:1:1. That is, each fatigue state in the training set contains 3200 groups of EEG signals, and each fatigue state in the validation set and the test set contains 400 groups of EEG signals respectively.

[0076] Step A: Use the linear synthesis method to generate 6800 groups of EEG signals for each fatigue state in the training set to expand the training set.

[0077] Step 3: Normalize the EEG signal data at 17 different preset positions in each group of data using the Min-Max normalization method.

[0078] Step 4: Extract 7 statistical features from the EEG signal data at 17 different preset positions in each group after normalization, and jointly form a 17×7 statistical feature matrix with the 7 statistical features corresponding to the EEG signal data at 17 different preset positions in each group. Among them, multiple statistical features include standard deviation, skewness, kurtosis, root mean square, root amplitude, absolute mean, and peak value.

[0079] Step 5: Decompose the EEG signal data at 17 different preset positions in each group after normalization into corresponding first, second, third, fourth, and fifth bands according to the first preset frequency range of 0.5 - 4 Hz, the second preset frequency range of 4 - 8 Hz, the third preset frequency range of 8 - 13 Hz, the fourth preset frequency range of 13 - 32 Hz, and the fifth preset frequency range of 32 - 50 Hz.

[0080] Step 6: The differential entropy features are extracted from the five bands into which the EEG signal data at each of the 17 different preset positions are decomposed respectively, and the five differential entropy features corresponding to the EEG signal data at each of the 17 different preset positions are jointly formed into a 17×5 differential entropy feature matrix.

[0081] Step 7: The detrended fluctuation analysis features are extracted from the five bands into which the EEG signal data at each of the 17 different preset positions are decomposed respectively, and the five detrended fluctuation analysis features corresponding to the EEG signal data at each of the 17 different preset positions are jointly formed into a 17×5 detrended fluctuation analysis feature matrix.

[0082] Step 8: The 17×7 statistical feature matrix, 17×5 differential entropy feature matrix, and 17×5 detrended fluctuation analysis feature matrix of each group are respectively standardized by the Min-Max normalization method.

[0083] Step 9: The standardized 17×7 statistical feature matrix, 17×5 differential entropy feature matrix, and 17×5 detrended fluctuation analysis feature matrix of each group are subjected to feature fusion in a matrix splicing manner to form a 17×17 fused feature matrix.

[0084] Step 10: A fatigue detection model is established based on the convolutional neural network model, as Figures 3 to 5 shown. The fatigue detection model is an input layer, multiple groups of cascaded residual modules and residual downsampling modules, a global average pooling layer, a fully connected layer with 3 neurons, and an output layer connected in series in sequence; the number of groups of cascaded residual modules and residual downsampling modules is 2; the main route of the residual module consists of a 3×3 convolutional layer, a Dropout layer, a BN layer, and a Leaky ReLU activation function layer, and the bypass is a skip connection with an identity mapping; the main route of the residual downsampling module consists of a zero-padding layer, a 3×3 convolutional layer with a stride of 2, a Dropout layer, a BN layer, and a Leaky ReLU layer, and the bypass consists of a 1×1 convolutional layer with a stride of 2, a BN layer, and a Leaky ReLU activation function layer; the output layer is a Softmax activation function layer, which is used to output the classification confidence degrees of the three-state EEG signals.

[0085] Step 11: The Adam optimization algorithm and the cross-entropy loss function are selected, the batchsize is set to 50, and the learning rate is set to 0.001. 30,000 groups of fused feature matrices in the training set are input into the fatigue detection model for training, and 1,200 groups of fused feature matrices in the validation set are input into the trained fatigue detection model for validation. The trained fatigue detection model is obtained by repeating multiple loops.

[0086] Step 12: Input the fusion feature matrices of 1200 groups in the test set into the trained fatigue detection model for testing. When the test results are qualified, an available fatigue detection model can be obtained.

[0087] This application also provides a driver fatigue detection method, as Figure 6 shown, including the following steps:

[0088] Step 1: Obtain the electroencephalogram (EEG) signal data at m different preset positions of the driver to be tested; specifically, use an EEG signal acquisition device to obtain the EEG signals at Figure 2 CP1, CP2, P1, PZ, P2, P03, P0Z, P04, 01, 0Z, 02, FT7, FT8, T7, T8, TP7, TP8 as shown in , and record them as respectively. Then the EEG signal data group Y obtained by the driver to be tested is .

[0089] Step 2: Standardize the EEG signal data at m different preset positions respectively; preferably, use the Min-Max standardization method for standardization processing. Specifically, the expression of using the Min-Max standardization method for processing is:

[0090] ,

[0091] After standardizing the EEG signals collected at each preset position in the EEG signal data group obtained by the driver to be tested it changes to .

[0092] Step 3: Extract n statistical features from the EEG signal data at m different preset positions after standardization processing, and jointly form an m×n statistical feature matrix with the n statistical features corresponding to the EEG signal data at m different preset positions respectively; specifically, for the EEG signals collected at each preset position in the EEG signal data group obtained by the driver to be tested after standardization extract the standard deviation, skewness, kurtosis, root mean square, root amplitude, absolute average value and peak value respectively. The expressions for calculating the standard deviation, skewness, kurtosis, root mean square, root amplitude, absolute average value and peak value are as follows:

[0093] , where ;

[0094] ,

[0095] ,

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] Then, the 7 statistical features corresponding to the EEG signals collected at 17 preset positions in the standardized EEG signal data set obtained by the driver to be measured jointly form a 17×7 statistical feature matrix as follows:

[0101] .

[0102] Step 4: Decompose the EEG signal data at m different preset positions after the standardization process into corresponding frequency bands according to p different preset frequency ranges; specifically, the number of preset frequency ranges is 5, the first preset frequency range is 0.5 - 4 Hz, the second preset frequency range is 4 - 8 Hz, the third preset frequency range is 8 - 13 Hz, the fourth preset frequency range is 13 - 32 Hz, and the fifth preset frequency range is 32 - 50 Hz, so that the EEG signal data at different preset positions in the EEG signal data set obtained by the driver to be measured after the standardization process are decomposed into corresponding first, second, third, fourth, and fifth frequency bands according to the first, second, third, fourth, and fifth preset frequency ranges respectively.

[0103] Step 5: Extract differential entropy features from the multiple frequency bands into which the EEG signal data at m different preset positions are decomposed, and jointly form an m×p differential entropy feature matrix with the multiple differential entropy features corresponding to the EEG signal data at m different preset positions respectively; specifically, extract differential entropy features from the first, second, third, fourth, and fifth frequency bands into which the EEG signals collected at 17 preset positions in the EEG signal data set obtained by the driver to be measured are decomposed respectively, and the expression for extracting differential entropy is:

[0104] ,

[0105] Then, the 5 differential entropy features corresponding to the EEG signal data at 17 different preset positions in the EEG signal data set obtained by the driver to be measured jointly form a 17×5 differential entropy feature matrix as follows:

[0106] .

[0107] Step 6: Extract the detrended fluctuation analysis features for each of the multiple frequency bands into which the EEG signal data at m different preset positions are decomposed, and jointly form an m×p detrended fluctuation analysis feature matrix with the multiple detrended fluctuation analysis features corresponding to the EEG signal data at m different preset positions respectively; specifically, extract the detrended fluctuation analysis features for the first frequency band, the second frequency band, the third frequency band, the fourth frequency band, and the fifth frequency band into which the EEG signals collected at 17 preset positions in the EEG signal data set obtained by the driver to be tested are decomposed respectively. The method for extracting the detrended fluctuation analysis features includes the following steps:

[0108] Step a: Calculate the cumulative deviation of the time series to obtain a new series :

[0109] ,

[0110] where is the mean of the time series .

[0111] Step b: Divide into subsequences of equal length s, and use the least squares method to fit the linear trend in each subsequence. Subtract the linear trend from each subsequence to obtain a detrended sequence, and calculate the root mean square of the detrended sequence:

[0112] ;

[0113] Step c: Take different lengths s, repeat Step b, and calculate the corresponding ;

[0114] Step d: Plot the relationship curve between s and on a double logarithmic coordinate system, and calculate the slope by the least squares method, which is the DFA index.

[0115] Then, the 5 detrended fluctuation analysis features corresponding to the EEG signal data at 17 different preset positions in the EEG signal data set obtained by the driver to be tested jointly form a 17×5 detrended fluctuation analysis feature matrix as follows:

[0116] .

[0117] Step 7: Standardize the m×n statistical feature matrix, the m×p differential entropy feature matrix, and the m×p detrended fluctuation analysis feature matrix respectively; preferably, use the Min - Max standardization method for standardization. Specifically, the expression for using the Min - Max standardization method is:

[0118] ,

[0119] Among them, represents the eigenvalue of the i-th row and j-th column in a matrix; represents the eigenvalue of the i-th row and j-th column after the matrix is standardized; m is the total number of rows of the matrix, and n is the total number of columns of the matrix; is the largest eigenvalue in the matrix; is the smallest eigenvalue in the matrix.

[0120] Then, the m×n statistical feature matrix, the m×p differential entropy feature matrix, and the m×p detrended fluctuation analysis feature matrix are converted into the corresponding standardized m×n statistical feature matrix, m×p differential entropy feature matrix, and m×p detrended fluctuation analysis feature matrix.

[0121] Step 8: Perform feature fusion on the standardized m×n statistical feature matrix, m×p differential entropy feature matrix, and m×p detrended fluctuation analysis feature matrix in a matrix splicing manner to form a fused feature matrix of m×(n + 2p).

[0122] Step 9: Input the fused feature matrix into the fatigue detection model to obtain the fatigue degree result of the driver. The fatigue detection model is constructed by any one of the above-mentioned fatigue detection model construction methods. Specifically, the fused feature matrix obtained from the electroencephalogram signal data group acquired for the driver to be measured is input into the fatigue detection model to obtain the classification confidence, and the fatigue degree result of the driver is obtained based on the classification confidence.

[0123] This application also provides a driver fatigue detection device, as Figure 7As shown, the device includes: a receiving module configured to obtain electroencephalogram signal data at m different preset positions of a driver to be measured; a first data processing module configured to perform normalization processing on the electroencephalogram signal data at m different preset positions respectively; a second data processing module configured to extract n statistical features from the electroencephalogram signal data at m different preset positions after normalization processing respectively, and jointly form an m×n statistical feature matrix with the n statistical features corresponding to the electroencephalogram signal data at m different preset positions respectively; a third data processing module configured to decompose the electroencephalogram signal data at m different preset positions into corresponding frequency bands respectively according to p different preset frequency ranges; a fourth data processing module configured to extract differential entropy features from the multiple frequency bands into which the electroencephalogram signal data at m different preset positions are decomposed respectively, and jointly form an m×p differential entropy feature matrix with the multiple differential entropy features corresponding to the electroencephalogram signal data at m different preset positions respectively; a fifth data processing module configured to extract detrended fluctuation analysis features from the multiple frequency bands into which the electroencephalogram signal data at m different preset positions are decomposed respectively, and jointly form an m×p detrended fluctuation analysis feature matrix with the multiple detrended fluctuation analysis features corresponding to the electroencephalogram signal data at m different preset positions respectively; a sixth data processing module configured to perform normalization processing on the m×n statistical feature matrix, the m×p differential entropy feature matrix and the m×p detrended fluctuation analysis feature matrix respectively; a seventh data processing module configured to perform feature fusion on the normalized m×n statistical feature matrix, the m×p differential entropy feature matrix and the m×p detrended fluctuation analysis feature matrix in a matrix splicing manner to form an m×(n + 2p) fusion feature matrix; a detection module configured to input the fusion feature matrix into a fatigue detection model to obtain the fatigue degree result of the driver, where the fatigue detection model is constructed by any one of the above-mentioned fatigue detection model construction methods.

[0124] The present application also provides a driver fatigue detection device, the device includes a memory and a processor, and a computer program is stored on the memory, and when the computer program is executed by the processor, the above-mentioned driver fatigue detection method is implemented.

[0125] The present application also provides a vehicle, including the above-mentioned driver fatigue detection device.

[0126] The present application also provides a storage medium, storing computer-readable instructions, and when the instructions are run by a processor, the above-mentioned method is executed.

[0127] The embodiments of the present disclosure have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary technicians in the art to understand the present disclosure.

Claims

1. A fatigue detection model construction method, characterized in that: The following steps are involved: Step 1: Obtain a set of EEG signal data of multiple groups of different preset positions of different drivers, and divide it into data subsets under different fatigue states according to preset fatigue state judgment rules, where fatigue states include awake state, moderate fatigue state and severe fatigue state. The set is a public data set or a data set collected through a driving simulation experiment; Step 2: Randomly extract several groups of data from the data subsets under different fatigue states, and divide the several groups of data into training sets, validation sets, and test sets according to preset proportions; Step 3: Standardize the EEG signal data of several different preset positions in each set of data respectively; Step 4: extract multiple statistical features from each group of EEG signal data at different preset positions after standardization, and use the multiple statistical features corresponding to each group of EEG signal data at different preset positions to form an m×n statistical feature matrix, where the multiple statistical features include standard deviation, skewness, kurtosis, root mean square, root amplitude, absolute mean and peak value, m is the number of positions, and n is the number of statistical features; Step 5: Decomposing each group of EEG signal data at different preset positions after standardization into corresponding bands according to different preset frequency ranges; Step 6: extract differential entropy features from the multiple bands decomposed from each group of EEG signal data at different preset positions, and combine the multiple differential entropy features corresponding to each group of EEG signal data at different preset positions into an m×p differential entropy feature matrix, where p is the number of preset frequency ranges; Step 7: extract detrended fluctuation analysis features from the multiple bands decomposed from each group of EEG signal data at different preset positions, and use the multiple detrended fluctuation analysis features corresponding to each group of EEG signal data at different preset positions to form an m×p detrended fluctuation analysis feature matrix; Step 8: Standardize the m×n statistical feature matrix, m×p differential entropy feature matrix and m×p detrended fluctuation analysis feature matrix of each group respectively; Step 9: The standardized m×n statistical feature matrix, m×p differential entropy feature matrix and m×p detrended fluctuation analysis feature matrix of each group are fused by matrix concatenation to form an m×(n+2p) fused feature matrix; Step 10: Establish a fatigue detection model based on the convolutional neural network model; Step 11: using several groups of fused feature matrices in the training set to input into the fatigue detection model for training, and using several groups of fused feature matrices in the verification set to input into the trained fatigue detection model for verification, repeating the cycle multiple times to obtain the trained fatigue detection model; Step 12: Use several groups of fusion feature matrices in the test set to input into the trained fatigue detection model for testing. When the test results are qualified, a usable fatigue detection model is obtained.

2. A fatigue detection model construction method according to claim 1, characterized in that: Step A is provided between step 2 and step 3: a plurality of groups of data are generated by using a linear synthesis method in the divided training sets of the awake state, the moderate fatigue state and the severe fatigue state.

3. A fatigue detection model construction method according to claim 1, characterized in that: The fatigue detection model based on the convolutional neural network model is composed of an input layer connected in series, multiple groups of residual modules and residual downsampling modules connected in series, a global average pooling layer, a fully connected layer and an output layer; the main route of the residual module is composed of a 3×3 convolution layer, a Dropout layer, a BN layer, and a Leaky ReLU activation function layer, and the bypass is a jump connection with an identity mapping; the main route of the residual downsampling module is composed of a zero padding layer, a 3×3 convolution layer with a step size of 2, a Dropout layer, a BN layer, and a Leaky ReLU layer, and the bypass is composed of a 1×1 convolution layer with a step size of 2, a BN layer, and a Leaky ReLU activation function layer.

4. A fatigue detection model construction method according to claim 1, characterized in that: The standardization process adopts the Min-Max standardization method.

5. A fatigue detection model construction method according to claim 1, characterized in that: The p is 5, the first preset frequency range is 0.5-4 Hz, the second preset frequency range is 4-8 Hz, the third preset frequency range is 8-13 Hz, the fourth preset frequency range is 13-32 Hz, and the fifth preset frequency range is 32-50 Hz.

6. A driver fatigue detection method, characterized in that: The following steps are involved: Step 1: Obtaining EEG signal data of the driver to be tested at m different preset positions; Step 2: Standardize the EEG signal data of m different preset positions respectively; Step 3: extracting n statistical features from the EEG signal data of m different preset positions after the standardized processing, and forming an m×n statistical feature matrix with the n statistical features corresponding to the EEG signal data of m different preset positions; Step 4: Decomposing the standardized EEG signal data at m different preset positions into corresponding bands according to p different preset frequency ranges; Step 5: extract differential entropy features from the multiple bands decomposed from the EEG signal data at m different preset positions, and combine the multiple differential entropy features corresponding to the EEG signal data at m different preset positions into an m×p differential entropy feature matrix; Step 6: extract detrended fluctuation analysis features from the multiple bands decomposed from the EEG signal data at m different preset positions, and combine the multiple detrended fluctuation analysis features corresponding to the EEG signal data at m different preset positions into an m×p detrended fluctuation analysis feature matrix; Step 7: Standardize the m×n statistical feature matrix, the m×p differential entropy feature matrix, and the m×p detrended fluctuation analysis feature matrix respectively; Step 8: The standardized m×n statistical feature matrix, the m×p differential entropy feature matrix and the m×p detrended fluctuation analysis feature matrix are fused by matrix concatenation to form an m×(n+2p) fused feature matrix; Step 9: Input the fused feature matrix into a fatigue detection model to obtain the driver's fatigue level result, wherein the fatigue detection model is constructed by a fatigue detection model construction method described in any one of claims 1 to 5.

7. A driver fatigue detection device, characterized in that: The device comprises: A receiving module configured to obtain brain electrical signal data of m different preset positions of the driver to be tested; A first data processing module is configured to perform standardization processing on the EEG signal data of m different preset positions respectively; The second data processing module is configured to extract n statistical features from the EEG signal data at m different preset positions after the standardization process, and to form an m×n statistical feature matrix with the n statistical features corresponding to the EEG signal data at m different preset positions; A third data processing module is configured to decompose the EEG signal data at m different preset positions after standardization into corresponding bands according to p different preset frequency ranges; The fourth data processing module is configured to extract differential entropy features from the multiple bands decomposed from the EEG signal data at m different preset positions, and to form an m×p differential entropy feature matrix from the multiple differential entropy features corresponding to the EEG signal data at m different preset positions; A fifth data processing module is configured to extract detrended fluctuation analysis features from the multiple bands decomposed into the EEG signal data at m different preset positions, and to form an m×p detrended fluctuation analysis feature matrix with the multiple detrended fluctuation analysis features corresponding to the EEG signal data at m different preset positions; a sixth data processing module, configured to perform standardization processing on the m×n statistical feature matrix, the m×p differential entropy feature matrix, and the m×p detrended fluctuation analysis feature matrix, respectively; A seventh data processing module is configured to perform feature fusion on the standardized m×n statistical feature matrix, the m×p differential entropy feature matrix and the m×p detrended fluctuation analysis feature matrix in a matrix splicing manner to form an m×(n+2p) fused feature matrix; The detection module is configured to input the fused feature matrix into a fatigue detection model to obtain the fatigue level result of the driver, wherein the fatigue detection model is constructed by a fatigue detection model construction method described in any one of claims 1 to 5.

8. A driver fatigue detection device, characterized in that: The device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, a driver fatigue detection method as described in claim 6 is implemented.

9. A vehicle, characterized in that: Including a driver fatigue detection device as described in claim 8.

10. A storage medium, characterized in that: Computer readable instructions are stored, and when the instructions are executed by a processor, the method according to claim 6 is performed.

Images