A method and system for identifying and intervening in driver mental fatigue based on electroencephalogram signals

Abstract

The present invention discloses a method and system for identifying and intervening in driver mental fatigue based on electroencephalogram (EEG) signals, which relates to the field of intelligent transportation technology, and solves the problem that the fatigue recognition algorithm is easily interfered by the external environment, resulting in low robustness of fatigue recognition. The present invention includes: S1: preprocessing the physiological signals of the driver; S2: constructing a brain functional network according to the preprocessed physiological signals; S3: extracting EEG topological features from the brain functional network; S4: inputting the EEG topological features into an identification model, and the identification model identifies the mental fatigue state of the driver according to the EEG topological features; S5: determining fatigue intervention measures according to the mental fatigue state of the driver. By analyzing the EEG signals, the system of the present invention can detect the fatigue condition of the driver in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent transportation, and particularly relates to a method and system for identifying and intervening in driver mental fatigue based on electroencephalogram signals. Background Art

[0002] Mental fatigue is a serious problem affecting driving safety, especially for long-term or repetitive driving tasks. Driver mental fatigue can lead to a decline in attention, a slower reaction speed, impaired decision-making ability, and significantly increase the risk of accidents. Currently, the detection of fatigue driving is mostly based on image recognition and is easily interfered by external environmental factors, such as light, eye occluders, etc. In addition, image recognition relies on external visual information, cannot truly reflect the mental state inside the driver's brain, and image recognition requires a large amount of computing resources to process images, resulting in the urgent need to improve the robustness and accuracy of image recognition algorithms.

[0003] Brain-computer interface systems can use electroencephalogram (EEG) signals to detect changes in brain activity. By analyzing these patterns, brain-computer interfaces can detect early signs of mental fatigue and evaluate the driver's cognitive state in real time. The application of brain-computer interface technology in fatigue detection mainly includes two key links: data acquisition and system identification and intervention. In terms of data acquisition, electroencephalogram (EEG) signals are collected in real time through non-invasive electrode sensors. These signals can reflect the working state of the driver's brain and provide important information about the degree of fatigue. In terms of system identification and intervention, with the rapid development of artificial intelligence technology, machine learning and deep learning are widely used in the analysis and interpretation of EEG signals. Machine learning methods can extract features from historical data through training models to help identify fatigue states. Deep learning technology, with its powerful self-learning and automatic feature extraction capabilities, can process more complex signal features and achieve higher accuracy and robustness. In recent years, deep learning algorithms such as convolutional neural networks (CNNs) and long short-term memory networks (LSTMs) have become mainstream technologies in EEG signal analysis, capable of automatically identifying fatigue patterns from a large amount of complex EEG data, greatly improving the efficiency and accuracy of fatigue detection.

[0004] Although existing fatigue detection methods based on EEG signals are real-time and relatively accurate, they still face some challenges, such as noise interference, individual differences, low computational efficiency, difficulty in obtaining labeled data, and limited signal spatial resolution. EEG signals are easily affected by the external environment, electromagnetic interference, and user movements, resulting in significant noise. Although existing filtering techniques can reduce some noise, they may also weaken the integrity of the signals. At the same time, the high computational resource requirements of deep learning algorithms may affect real-time performance, especially in applications on embedded systems. The dependence of the model training process on high-quality fatigue labels further increases the complexity and subjectivity of data acquisition. Moreover, due to the limited number of electrodes in non-invasive EEG devices, the collected spatial resolution is insufficient to comprehensively reflect brain activities, thus limiting the ability to analyze complex brain activities. Summary of the Invention

[0005] To solve the problems existing in the above-mentioned prior art, the present invention provides a method and system for identifying and intervening in driver mental fatigue based on electroencephalogram signals, which solves the problem that existing fatigue recognition algorithms are easily interfered by the external environment, resulting in low robustness of fatigue recognition.

[0006] A method for identifying and intervening in driver mental fatigue based on electroencephalogram signals includes:

[0007] S1: Preprocess the physiological signals of the driver;

[0008] S2: Construct a brain functional network based on the preprocessed physiological signals;

[0009] S3: Extract EEG topological features from the brain functional network;

[0010] S4: Input the EEG topological features into an identification model. The identification model identifies the driver's mental fatigue state according to the EEG topological features. The identification model includes an asynchronous graph diffusion network, a time-aware convolutional network, and a semi-autoregressive prediction network. The asynchronous graph diffusion network models the spatial relationship between EEG channels, the time-aware convolutional network models the temporal relationship between EEG channels, and the semi-autoregressive prediction network synthesizes the spatio-temporal features learned by the asynchronous graph diffusion network and the time-aware convolutional network to provide accurate fatigue state prediction;

[0011] S5: Determine fatigue intervention measures according to the driver's mental fatigue state.

[0012] The driver's mental fatigue state includes awake, sleepy, and drowsy.

[0013] Further, the S2 includes:

[0014] S21: Process the EEG signals using non-linear canonical polyadic decomposition;

[0015] S22: Calculate the components of the EEG signal after decomposition according to different frequency ranges;

[0016] S23: Separate the frequency components within the frequency range corresponding to the frequency band to be extracted;

[0017] S24: Convert different frequency bands into corresponding frequency components, calculate the functional connectivity between different brain regions under different frequency components, and obtain the correlation matrix between different component signals;

[0018] S25: Construct a brain functional network based on the correlation matrix.

[0019] Furthermore, S2 further includes S25: After constructing the brain functional network, calculate network metrics to analyze the connection characteristics of the brain. The specific calculation metrics include node strength, clustering coefficient, eigenvector centrality, local efficiency, global efficiency, small-world property, and shortest path.

[0020] Furthermore, S4 includes:

[0021] S41: Asynchronous graph convolution is to input the EEG feature dataset into the Chebyshev polynomial graph convolution to learn the dependency relationships existing between feature sets. The calculation formula is:

[0022]

[0023] where, T k (A) is the k-th order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the input matrix reflecting the EEG features, which describes the feature information of each EEG channel or electrode at a certain time point t, and K is the order of the Chebyshev polynomial;

[0024] S42: Define the recursion for higher-order Chebyshev polynomials:

[0025] T0(A) = I

[0026] T1(A) = A

[0027] T k (A) = 2A·T k-1 (A) - T k-2 (A) for k ≥ 2

[0028] where, I is the identity matrix, A is the adjacency matrix, T0(A) and T1(A) are the identity matrix and the adjacency matrix respectively, and T k (A) continuously captures the information of the k-th order neighborhood through the recursive relationship; through this recursive relationship, higher-order Chebyshev polynomials are calculated;

[0029] S43: Convolve the Chebyshev polynomials and aggregate the output to obtain the sum of information propagation between all electrode channels; finally, add the bias term b to the convolution result to adjust the offset in the feature space. The specific calculation formula is as follows:

[0030]

[0031] where, T k (A) is the k-th order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the feature matrix at time step t, and b is the learned bias term;

[0032] S44: To learn θ k and b, compare Output(t) with the actual t e and minimize the error Z through the loss function. The specific calculation formula is as follows:

[0033]

[0034] where, I is the total number of data sets or samples, n i is the number of elements in the i-th data subset, t e (i, j) is the actual value or observed value of the j-th element in the i-th data set or sample, and t a (i, j) is the predicted value or expected value of the j-th element in the i-th data set or sample;

[0035] S45: The convertible time-aware convolutional network is that the aggregation result uses time encoding to generate time features, and converts the time information into a vector with a fixed dimension. The specific calculation formula is as follows:

[0036] TimeFeature = W · t

[0037] where, W is the learned weight, t is the matrix of time steps, and TimeFeature is the time feature generated after time encoding;

[0038] S46: Combine the time feature with the convolutional kernel, and dynamically generate the weights of the convolutional kernel by inputting the time feature into a fully connected layer. The specific calculation formula is as follows:

[0039] Kernel(t) = MetaLayer(TimeFeature)

[0040] where, MetaLayer is a fully connected layer responsible for generating the convolutional kernel from the time feature, and Kernel(t) is the generated convolutional kernel;

[0041] S47: Perform convolution calculation on the EEG feature data by taking the dynamically generated convolution sum in a sliding window manner. For each time step i, the sliding window selects consecutive K time steps of data, and convolve the selected data using the generated convolution kernel. The specific calculation formula is as follows:

[0042]

[0043] Among them, Y i is the convolution output result, W k is the convolution kernel weight, N i+k is the data of the (i + k)-th time step in the EEG feature data;

[0044] S48: Use the semi-autoregressive prediction network to predict the output value. For each time step t, the calculation formula of the model is as follows:

[0045]

[0046] Among them, y t is the fatigue state prediction value at time step t, X 1:t is all historical data from time step 1 to t, is all fatigue state prediction values from time step 1 to t - 1, and f(·) is the prediction function.

[0047] Furthermore, the physiological signal described in S1 includes the EEG signal. The preprocessing of the EEG signal includes:

[0048] Perform band-pass filtering and power frequency notch filtering on the i electrode channel signal in the collected EEG signal to retain the EEG signal within the required frequency band range;

[0049] Perform principal component analysis on the filtered EEG signal data to retain the EEG signal data with the cumulative variance contribution reaching the standard, and decompose the obtained EEG signal data into independent components;

[0050] Use the electroencephalogram data created by the synthetic electrooculogram (EOG) channel to detect blinks, horizontal and vertical movements. Components with high temporal or spatial correlation (ρ > 0.8) are marked as artifacts. The specific calculation formula is as follows:

[0051]

[0052] Among them, ρ is the correlation coefficient calculated between the signal S i and the electrooculogram (EOG) signal, S i is the i-th component of the electroencephalogram (EEG) signal, EOG is the electrooculogram (EOG) signal, which is usually used to record the horizontal and vertical movements of the eyes, mainly reflecting the eye blinks and eye movements, is Si The standard deviation of, representing the electroencephalogram (EEG) signal component S i The degree of dispersion of the data distribution of, σ EOG Is the standard deviation of the EOG signal, representing the degree of dispersion of the data distribution of the electrooculogram (EOG) signal.

[0053] Remove the artifact components with high temporal or spatial correlations to obtain the corresponding EEG data;

[0054] Segment the continuous EEG signal into smaller time periods for analysis and calculation, and use a time window for calculation;

[0055] Baseline correction removes the pre-stimulus fluctuations in the EEG signal data and normalizes the EEG signal within the time window.

[0056] Furthermore, the physiological signal in S1 further includes eye movement data, and the preprocessing of the eye movement data includes: using cubic spline interpolation to fit a spline function to interpolate and complete the eye movement data:

[0057] S i (x) = a i (x - x i ) 3 + b i (x - x i ) 2 + c i (x - x i ) + d i

[0058] Wherein, S i (x) is the vertical coordinate of the gaze, x is the horizontal coordinate of the interpolation completion point, x i Is the known horizontal coordinate of the gaze, a i , b i , c i And d i Are determined by the boundary conditions and continuity requirements.

[0059] Furthermore, the S21 includes:

[0060] Regard the EEG signal as a multi-dimensional tensor E, each dimension corresponding to a specific aspect of the EEG signal, including time, space, and frequency. Given a third-order tensor The regular multi-way decomposition calculation formula is as follows:

[0061]

[0062] Wherein, Represents the outer product, represents the modal factor vector, and ΔE represents the error tensor. The tensor norm includes time (temporal component), frequency (spectral component), and electrode (spatial component). To separate the above components into rank-1 tensors, the calculation formula is as follows:

[0063] T = a1°b1°c1 + a2°b2°c2 + E

[0064] where a r represents the temporal component, b r represents the frequency component, c r represents the spatial component, r = 1, 2.

[0065] Using non-linear least squares (NLS) to solve the decomposition and minimize the reconstruction error, the specific calculation formula is as follows:

[0066]

[0067] where ‖·‖ F is the Frobenius norm.

[0068] The specific formula for calculating the relative error of each factor matrix U (n) is as follows:

[0069]

[0070] where is the estimated factor matrix, U (n) is the actual factor matrix, P is the permutation matrix, and D is the scaling matrix.

[0071] Furthermore, the calculation formula of S22 is:

[0072]

[0073] where X(f) represents the signal spectrum, x(t) is the component after the decomposition of the electroencephalogram signal, f is the acquisition frequency of the electroencephalogram signal, and j is the imaginary unit.

[0074] Furthermore, the above S24 includes: representing the power distribution of the signal at different frequencies through the power spectral density P(f):

[0075] P(f) = |X(f)| 2

[0076] Converting the different frequency bands in the above electroencephalogram signal into corresponding frequency components, calculating the functional connections between different brain regions under different frequency components, and constructing a connection matrix based on the correlation relationship between each electrode signal to obtain:

[0077]

[0078] where x i represents the component of the i-th electroencephalogram (EEG) signal, and C ij is the functional connectivity strength between the i-th and j-th components calculated using the phase-locking value. COV(x i , x j ) is the covariance between the EEG signal data of the i-th and j-th components, and Var(x i ) and Var(x j ) are the variances between the EEG signal data of the i-th and j-th components. Through the calculation of the above formula, the correlation matrix C between different component signals is obtained, where each element C ij represents the functional connectivity strength between components i and j in a given frequency band;

[0079] Furthermore, S25 includes: using the correlation matrix C to construct a brain functional network G. The thresholding formula for the brain functional network is as follows:

[0080] G = (V, E)

[0081] where V is the set of electrode channels, and E is the set of edges. The edges are defined according to the correlation values to describe the connectivity between brain regions;

[0082] Applying a threshold T to the correlation matrix determines the importance. Specifically, connections below the threshold are considered invalid and need to be discarded, while connections above the threshold are considered valid and retained. The adjacency matrix A is defined as:

[0083]

[0084] The above binary adjacency matrix A represents the functional connections in the network.

[0085] A driver mental fatigue recognition and intervention system based on electroencephalogram (EEG) signals includes a physiological information collection device, an algorithm calculation unit, and an in-vehicle intervention unit;

[0086] Physiological information collection device: used to collect the physiological information of the driver, including the driver's EEG signals, eye movement data, etc.;

[0087] Algorithm calculation unit: as the management and calculation center of the brain-computer interface, predicts according to the received driver's EEG signals according to the designed detection and recognition algorithm, determines the mental fatigue state of the driver, and selects a suitable intervention plan to intervene in the driver's fatigue based on the driver's future mental fatigue state to ensure road safety;

[0088] In-vehicle intervention unit: provides reasonable solutions and suggestions for the driver's fatigue intervention according to the selected intervention plan to achieve the elimination of the driver's fatigue and ensure road safety.

[0089] The beneficial effects of the present invention include:

[0090] (1) By analyzing the electroencephalogram (EEG) signals, the system of the present invention can detect the fatigue status of the driver in real time. When it detects that the driver's mental state is abnormal, the system will automatically issue an alarm to remind the driver to rest or stop driving, thus avoiding accidents caused by fatigue driving.

[0091] (2) The present invention takes into account the driving experience of the driver. According to the EEG signal characteristics of each driver and combined with their driving habits, it automatically pushes customized rest suggestions. In addition, the system can also combine in-vehicle devices and environmental information to automatically adjust the in-vehicle environment to help the driver restore situational awareness when it identifies that the driver is fatigued. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a flowchart of a method for identifying and intervening in driver mental fatigue based on EEG signals according to an embodiment of the present application.

[0093] Figure 2 It is a schematic diagram of the recognition model algorithm according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0094] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all of them. Therefore, the detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.

[0095] A method for identifying and intervening in driver mental fatigue based on EEG signals, as Figure 1 shown, includes:

[0096] S1: Preprocess the physiological signals of the driver;

[0097] S2: Construct a brain functional network according to the preprocessed physiological signals;

[0098] S3: Extract EEG topological features from the brain functional network;

[0099] S4: Input the EEG topological features into the recognition model, and the recognition model identifies the driver's mental fatigue status according to the EEG topological features;

[0100] S5: Determine fatigue intervention measures according to the driver's mental fatigue status.

[0101] The driver's mental fatigue state includes being awake, sleepy, and drowsy.

[0102] Once it is detected that the driver is in a fatigued state, the system needs to quickly take intervention measures to reduce the accident risk. The intervention methods are divided into two categories: one is automatic intervention based on physiological responses, and the other is reminder intervention based on external cues. The former stimulates the driver to recover from the fatigued state by adjusting the driver's physiological state, such as through mild vibration, audio signals, or visual stimuli. The latter reminds the driver to take a rest or take appropriate rest measures through an intelligent reminder system, such as a sound alarm, seat vibration, or dashboard warning light.

[0103] In another embodiment, the S2 includes:

[0104] S21: Process the electroencephalogram (EEG) signals using non - linear canonical polyadic decomposition;

[0105] S22: Calculate the components of the EEG signals after decomposition according to different frequency ranges;

[0106] S23: Separate the frequency components within the frequency range corresponding to the frequency band to be extracted;

[0107] S24: Convert different frequency bands into corresponding frequency components, calculate the functional connectivity between different brain regions under different frequency components to obtain the correlation matrix between different component signals;

[0108] S25: Construct a brain functional network based on the correlation matrix.

[0109] In another embodiment, the S2 further includes S25: After constructing the brain functional network, calculate network metrics to analyze the connection characteristics of the brain. The specific calculation metrics include node strength, clustering coefficient, eigenvector centrality, local efficiency, global efficiency, small - world property, and shortest path.

[0110] In another embodiment, the S4 includes:

[0111] S41: Asynchronous graph convolution is to input the EEG feature dataset into the Chebyshev polynomial graph convolution to learn the dependency relationships existing between the feature sets. The calculation formula is:

[0112]

[0113] where, T k (A) is the k - th order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k - th order polynomial, N(t) is the input matrix reflecting the EEG features, which describes the feature information of each EEG channel or electrode at a certain time point t, and K is the order of the Chebyshev polynomial;

[0114] S42: Define the recurrence of high-order Chebyshev polynomials as follows:

[0115] T0(A) = I

[0116] T1(A) = A

[0117] T k (A) = 2A·T k-1 (A) - T k-2 (A) for k ≥ 2

[0118] where I is the identity matrix, A is the adjacency matrix, T0(A) and T1(A) are the identity matrix and the adjacency matrix respectively, and T k (A) continuously captures the information of the k-th order neighborhood through the recurrence relation; through this recurrence relation, high-order Chebyshev polynomials are calculated;

[0119] S43: Convolve the Chebyshev polynomials and aggregate the outputs to obtain the sum of information propagation between all electrode channels; finally, add the bias term b to the convolution result to adjust the offset in the feature space. The specific calculation formula is as follows:

[0120]

[0121] where T k (A) is the k-th order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the feature matrix at time step t, and b is the learned bias term;

[0122] S44: To learn θ k and b, compare Output(t) with the actual t e and minimize the error Z through the loss function. The specific calculation formula is as follows:

[0123]

[0124] where I is the total number of data sets or samples, n i is the number of elements in the i-th data subset, t e (i,j) is the actual value or observed value of the j-th element in the i-th data set or sample, and t a (i,j) is the predicted value or expected value of the j-th element in the i-th data set or sample;

[0125] S45: The convertible time-aware convolutional network is that the aggregation result uses time encoding to generate time features and converts the time information into a vector with a fixed dimension. The specific calculation formula is as follows:

[0126] TimeFeature = W·t

[0127] Among them, W is the learned weight, t is the matrix of time steps, and TimeFeature is the time feature generated after time encoding;

[0128] S46: Combine the time feature with the convolutional kernel, and dynamically generate the weight of the convolutional kernel by inputting the time feature into a fully connected layer. The specific calculation formula is as follows:

[0129] Kernel(t) = MetaLayer(TimeFeature)

[0130] Among them, MetaLayer is a fully connected layer responsible for generating the convolutional kernel from the time feature, and Kernel(t) is the generated convolutional kernel;

[0131] S47: Perform convolution calculation on the EEG feature data with the dynamically generated convolution sum in the form of a sliding window. For each time step i, the sliding window selects the data of continuous K time steps, and convolves the selected data with the generated convolutional kernel. The specific calculation formula is as follows:

[0132]

[0133] Among them, Y i is the convolution output result, W k is the convolutional kernel weight, and N i+k is the data of the (i + k)-th time step in the EEG feature data;

[0134] S48: Use a semi-autoregressive prediction network to predict the output value. For each time step t, the calculation formula of the model is as follows:

[0135]

[0136] Among them, y t is the fatigue state prediction value at time step t, X 1:t is all historical data from time step 1 to t, is all fatigue state prediction values from time step 1 to t - 1, and f(·) is the prediction function.

[0137] In another embodiment, the physiological signal described in S1 includes an EEG signal, and the preprocessing of the EEG signal includes:

[0138] Perform band-pass filtering and power frequency notch filtering on the i electrode channel signal in the collected EEG signal to retain the EEG signal within the required frequency band range;

[0139] Perform principal component analysis on the filtered EEG signal data, retain the EEG signal data with the cumulative variance contribution reaching the standard, and decompose the obtained EEG signal data into independent components;

[0140] Use the electroencephalogram data created by the synthetic electrooculogram (EOG) channel to detect blinks, horizontal and vertical movements. Components with high temporal or spatial correlation (ρ > 0.8) are marked as artifacts. The specific calculation formula is as follows:

[0141]

[0142] where ρ is the correlation coefficient calculated between the signal S i and the electrooculogram (EOG) signal, S i is the i-th component of the electroencephalogram (EEG) signal, EOG is the electrooculogram (EOG) signal, which is usually used to record the horizontal and vertical movements of the eyes, mainly reflecting the eye blinks and the eye movement conditions, is S i 's standard deviation, representing the degree of dispersion of the data distribution of the EEG signal component S i , σ EOG is the standard deviation of the EOG signal, representing the degree of dispersion of the data distribution of the electrooculogram signal EOG.

[0143] Remove the artifact components with high temporal or spatial correlation to obtain the corresponding EEG data;

[0144] Segment the continuous EEG signal into smaller time periods for analysis and calculation, and use a time window for calculation;

[0145] Baseline correction removes the pre-stimulus fluctuations in the EEG signal data and normalizes the EEG signal within the time window.

[0146] In another embodiment, the physiological signal in S1 further includes eye movement data, and the preprocessing of the eye movement data includes: using cubic spline interpolation to fit the spline function to interpolate and complete the eye movement data:

[0147] S i (x) = a i (x - x i ) 3 + b i (x - x i ) 2 + c i (x - x i ) + d i

[0148] where S i (x) is the vertical coordinate of the gaze, x i is the horizontal coordinate of the gaze, ai , b i , c i and d i are determined by boundary conditions and continuity requirements.

[0149] In another embodiment, S21 includes:

[0150] Regarding the EEG signal as a multi-dimensional tensor E, each dimension corresponding to a specific aspect of the EEG signal, including time, space, and frequency, given a third-order tensor The regular multi-way decomposition calculation formula is as follows:

[0151]

[0152] where ° represents the outer product, represents the modal factor vector, and ΔE represents the error tensor. The tensor norm includes time (temporal component), frequency (spectral component), and electrodes (spatial component). To separate the above components into rank-1 tensors, the calculation formula is as follows:

[0153] T = a1°b1°c1 + a2°b2°c2 + E

[0154] where a r represents the temporal component, b r represents the frequency component, and c r represents the spatial component.

[0155] Using non-linear least squares (NLS) to solve the decomposition and minimize the reconstruction error, the specific calculation formula is as follows:

[0156]

[0157] where ||·|| F is the Frobenius norm.

[0158] The specific formula for calculating the relative error of each factor matrix U (n) is as follows:

[0159]

[0160] where is the estimated factor matrix, U (n) is the actual factor matrix, P is the permutation matrix, and D is the scaling matrix.

[0161] In another embodiment, the calculation formula of S22 is:

[0162]

[0163] Among them, X(f) represents the signal spectrum, x(t) is the component after the decomposition of the electroencephalogram (EEG) signal, f is the acquisition frequency of the EEG signal, and j is the imaginary unit.

[0164] In another embodiment, S24 includes: representing the power distribution of the signal at different frequencies through the power spectral density P(f):

[0165] P(f) = |X(f)| 2

[0166] Convert different frequency bands in the above EEG signal into corresponding frequency components, calculate the functional connectivity between different brain regions under different frequency components, and construct a connection matrix based on the correlation relationship between each electrode signal to obtain:

[0167]

[0168] where x i represents the component of the i-th EEG signal, C ij is the functional connectivity strength calculated using the phase locking value between the i-th and j-th components, COV(x i , x j ) is the covariance between the EEG signal data of the i-th and j-th components, Var(x i ) and Var(x j ) are the variances between the EEG signal data of the i-th and j-th components. Through the calculation of the above formula, the correlation matrix C between different component signals is obtained, where each element C ij represents the functional connectivity strength between components i and j in a given frequency band;

[0169] In another embodiment, S25 includes: using the correlation matrix C to construct a brain functional network G, and the thresholding formula of the brain functional network is as follows:

[0170] G = (V, E)

[0171] where V is the set of electrode channels, E is the set of edges, and the edges are defined according to the correlation value to describe the connectivity between brain regions;

[0172] Apply a threshold T to the correlation matrix to determine the importance. Specifically, connections below the threshold are considered invalid and need to be discarded, while connections above the threshold are considered valid and retained. Define the adjacency matrix A as:

[0173]

[0174] The above binary adjacency matrix A represents the functional connections in the network.

[0175] In another embodiment, a method for identifying and intervening in driver mental fatigue based on EEG signals includes:

[0176] S1. Divide the fatigue level of the driver during driving into three stages: awake, sleepy, and drowsy, and collect the physiological signals of the driver in the state of mental fatigue. The physiological signals include the electroencephalogram (EEG) signals and eye movement data of the driver.

[0177] Preprocess the collected physiological signals, including filtering, rereferencing, segmenting, baseline correction, and independent component analysis of EEG signals, and interpolation and completion of eye movement data. Among them, the baseline correction includes: in each segment of the signal, select the baseline interval (for example, -200 ms to 0 ms before the stimulus) to calculate the baseline potential. Subtract the baseline potential from each segment of the signal to ensure that the corrected signal has the baseline as zero.

[0178] The specific scheme is as follows:

[0179] S11. Perform band-pass filtering and power frequency notch filtering on the i-electrode channel signals in the collected EEG signals, and retain the required frequency band range (0.1 Hz - 50 Hz).

[0180] S12. Perform principal component analysis on the filtered EEG signal data, and retain the data with a cumulative variance contribution of 99%.

[0181] S13. Decompose the preprocessed EEG data into independent components (S).

[0182] S14. Use the electroencephalogram data created by the synthetic electrooculogram (EOG) channel to detect blinks, horizontal and vertical movements. The components with high temporal or spatial correlation (ρ > 0.8) are marked as artifacts. The specific calculation formula is as follows:

[0183]

[0184] S15. Remove the artifact components with high temporal or spatial correlation to obtain clean EEG data.

[0185] S16. Segment the continuous EEG signals into smaller and manageable time periods for analysis and calculation, and use an 8-second time window for calculation. Baseline correction is used to remove the pre-stimulus fluctuations in the EEG signal data and standardize the EEG signals within the 8-second time window.

[0186] S17. Use cubic spline interpolation to fit the spline function to interpolate and complete the eye movement data. The specific formula is:

[0187] S i (x) = a i (x - x i ) 3 + b i (x - x i ) 2 + ci (x - x i ) + d i

[0188] where S i (x) is the vertical coordinate of the gaze, x is the horizontal coordinate of the interpolation point to be completed, and x i is the horizontal coordinate of the known gaze. a i , b i , c i and d i are determined by the boundary conditions and continuity requirements.

[0189] S2. Extract different frequency bands (Delta, Theta, Alpha, Beta, and Gamma) of the EEG signal and construct a brain functional network, and extract the corresponding topological features. The specific scheme is as follows:

[0190] To perform temporal reconstruction of the EEG signal with a relatively low loss, the nonlinear canonical polyadic decomposition (CPD) is used to process the EEG signal. The EEG signal can be regarded as a multi-dimensional tensor E, and each dimension corresponds to a specific aspect of the EEG signal, including time, space, and frequency. Given a third-order tensor The calculation formula for canonical polyadic decomposition is as follows:

[0191]

[0192] where represents the outer product, represents the modal factor vector, and ΔE represents the error tensor. The tensor norm includes time (temporal component), frequency (spectral component), and electrodes (spatial component). To separate the above components into rank-1 tensors, the calculation formula is as follows:

[0193]

[0194] where a r represents the temporal component, b r represents the frequency component, and c r represents the spatial component.

[0195] Use nonlinear least squares (NLS) to solve the decomposition to minimize the reconstruction error. The specific calculation formula is as follows:

[0196]

[0197] where ‖·‖ F is the Frobenius norm.

[0198] Each factor matrix U (n)The specific formula for calculating the relative error is as follows:

[0199]

[0200] Among them, is the estimated factor matrix, and U (n) is the actual factor matrix, P is the permutation matrix, and D is the scaling matrix.

[0201] Subsequently, according to the different frequency ranges, the calculation methods are as follows:

[0202]

[0203] Among them, x(t) is the component after the decomposition of the electroencephalogram signal, f is the acquisition frequency of the electroencephalogram signal, and j is the imaginary unit.

[0204] In order to extract specific frequency bands and analyze signals within a specific frequency range, it is necessary to isolate the corresponding frequency components. The frequency ranges of different frequency bands are Delta band (0.1 - 4 Hz), Theta band (4 - 8 Hz), Alpha band (8 - 12 Hz), Beta band (12 - 30 Hz), and Gamma band (30 - 40 Hz). The power distribution of the signal at different frequencies is represented by the power spectral density P(f), and the specific formula is as follows:

[0205] P(f) = |X(f)| 2

[0206] Convert the different frequency bands in the above electroencephalogram signal into corresponding frequency components, calculate the functional connectivity between different brain regions under different frequency components, and construct a connection matrix based on the correlation relationship between each electrode signal. The specific formula is as follows:

[0207]

[0208] Among them, x i represents the component of the i-th electroencephalogram signal, and C ij is the functional connectivity strength calculated using the phase-locking value between the i-th and j-th components. COV(x i , x j ) is the covariance between the electroencephalogram signal data of the i-th and j-th components, and Var(x i ) and Var(x j ) are the variances between the electroencephalogram signal data of the i-th and j-th components. Through the calculation of the above formula, the correlation matrix C between different component signals is obtained, where each element C ij represents the functional connectivity strength between components i and j in a given frequency band.

[0209] Then, the correlation matrix C is used to construct the brain functional network G. The thresholding of the brain functional network can be expressed by the following formula:

[0210] G = (V, E)

[0211] where V is the set of electrode channels, and E is the set of edges. The edges are defined according to the correlation values to describe the connectivity between brain regions.

[0212] To construct a more efficient brain functional connectivity network, a threshold T is applied to the correlation matrix to determine which connections are considered important. In this method, if the threshold is lower than 15%, it is regarded as an invalid connection that needs to be discarded; if it is higher than 15%, it is regarded as a valid connection and retained. The adjacency matrix A is defined as:

[0213]

[0214] The above binary adjacency matrix A represents the functional connections in the network.

[0215] Based on the construction of the brain functional network, various network metrics are calculated to analyze the connection characteristics of the brain. The specific metrics include node strength, clustering coefficient, eigenvector centrality, local efficiency, global efficiency, small-world property, and shortest path, as follows:

[0216] The node strength represents the total functional connectivity of the brain region (electrode) corresponding to the node. The specific calculation formula is as follows:

[0217]

[0218] where A ij represents the weight of the connection between components i and j of the adjacency matrix, N represents the number of components, and S i represents the i-th electrode.

[0219] The clustering coefficient C of electrode channel i i quantifies the tightness of the clustering of the neighbors of electrode channel i. It measures the tendency of a component to form a cluster. The specific calculation formula is as follows:

[0220]

[0221] where E i represents the number of connection edges between adjacent components, and k i represents the degree of the component.

[0222] Eigenvector centrality is a measure of the influence of components in the network. If a component is connected to other components that are highly central themselves, then the centrality of this component is relatively high. The specific calculation formula is as follows:

[0223] A·v = λ·v

[0224] Among them, A is the adjacency matrix of the network, v is the centrality vector of the eigenvector (the centrality score of each node), and λ is the eigenvalue corresponding to the eigenvector.

[0225] The local efficiency measures the efficiency of information transfer between the neighbors of a component, only considering the neighbors of the component and the edges directly connecting them. The specific calculation formula is as follows:

[0226]

[0227] Among them, d jk is the shortest path distance between components i and j, and k i represents the degree of the component.

[0228] The global efficiency reflects the efficiency of information transmission in the entire brain network. It is the average inverse shortest path length between all pairs of components and is related to the global ability of the network to transfer information. The specific calculation formula is as follows:

[0229]

[0230] Among them, N represents the number of components, and d jk is the shortest path distance between components i and j.

[0231] The small-world property indicates that the network has a high clustering coefficient and a short average path length between components, reflecting the efficiency of the human brain in processing and transmitting information. The specific calculation formula is as follows:

[0232]

[0233] Among them, C observed is the clustering coefficient of the observed network, C random is the clustering coefficient of a random network with the same number of components and edges, L observed is the average path length of the observed network, and L random is the average path length of the random network.

[0234] The shortest path length d between two components i and j ij is the minimum number of edges that must be passed from i to j. The specific calculation formula is as follows:

[0235]

[0236] Among them, N represents the number of components, and d jk is the shortest path distance between components i and j.

[0237] S4. Construct an identification algorithm for driver mental fatigue. Input the EEG signal features of the driver, identify the mental fatigue state of the driver according to the features, and classify the level of its mental fatigue state. For example Figure 2 as shown

[0238] S41. Obtain the EEG feature dataset X(t), perform normalization processing on the collected EEG feature dataset X(t) to obtain the normalized feature set N(t);

[0239] S42. Asynchronous graph convolution is to input the normalized feature set N(t) into the Chebyshev polynomial graph convolution to learn the dependence relationship existing between the feature sets. The calculation formula is:

[0240]

[0241] where T k (A) is the k-th order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, and K is the order of the Chebyshev polynomial.

[0242]

[0243] S43. To efficiently calculate the high-order Chebyshev polynomial, its recursive definition is as follows:

[0244] T0(A) = I

[0245] T1(A) = A

[0246] T k (A) = 2A·T k-1 (A) - T k-2 (A) for k ≥ 2

[0247] where I is the identity matrix and A is the adjacency matrix. Through this recursive relationship, the high-order Chebyshev polynomial can be efficiently calculated.

[0248] S43. Convolve the Chebyshev polynomial and aggregate the output to obtain the sum of information propagation between all electrode channels. Finally, add the bias term b to the convolution result to adjust the offset in the feature space. The specific calculation formula is as follows:

[0249]

[0250] where T k (A) is the k-th order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the feature matrix at time step t, and b is the learned bias term.

[0251] S44: The convertible time-aware convolutional network generates time features by using time encoding for the results, converting the time information into a vector with a fixed dimension. The specific calculation formula is as follows:

[0252] TimeFeature = W·t

[0253] Among them, W is the learned weight, t is the matrix of time steps, and TimeFeature is the time feature generated after time encoding.

[0254] S45: Combine the time feature with the convolutional kernel, and dynamically generate the weight of the convolutional kernel by inputting the time feature into a fully connected layer. The specific calculation formula is as follows:

[0255] Kernel(t) = MetaLayer(TimeFeature)

[0256] Among them, MetaLayer is a fully connected layer responsible for generating the convolutional kernel from the time feature, and Kernel(t) is the generated convolutional kernel.

[0257] S46: Perform convolution calculation on the EEG feature data with the dynamically generated convolution in a sliding window manner. For each time step i, the sliding window selects continuous K time step data, and convolves the selected data with the generated convolutional kernel. The specific calculation formula is as follows:

[0258]

[0259] Among them, Y i is the convolution output result, W k is the convolutional kernel weight, N i+k is the data of the (i + k)-th time step in the EEG feature data.

[0260] S47: Use the semi-autoregressive prediction network to predict the output value. For each time step t, the calculation formula of the model is as follows:

[0261]

[0262] Among them, y t is the fatigue state prediction value at time step t, X 1:t is all historical data from time step 1 to t, is all fatigue state prediction values from time step 1 to t - 1, and f(·) is the prediction function.

[0263] S5: Provide reasonable fatigue intervention measures for drivers in a state of mental fatigue to reduce the risk of road accidents.

[0264] If the predicted mental state of the driver is awake, no intervention measures are required;

[0265] If the predicted mental state of the driver is fatigued, appropriate interventions need to be taken according to the driver's current state, including playing music, vibrating the seat (steering wheel), and visual guidance on the HUD;

[0266] If the predicted mental state of the driver is drowsy, strong intervention measures will be taken to ensure that the driver can safely reach the rest area for rest.

[0267] In the specific implementation process, the EEG data of 21 subjects were used for verification, and it was written and tested using Python. To prevent overfitting, a dropout layer was added, and the dropout rate was set to 0.5. In addition, to make greater use of the data and reduce the data bias caused by randomly dividing the data, the dataset was divided into a ratio of 4:1 for training and verification. The evaluation metrics of the recognition algorithm were obtained, including accuracy, precision, recall, and F1 score.

[0268] In the task of identifying driver mental fatigue using this method, the Delta and Alpha frequency bands performed excellently individually, with accuracies reaching 94.10% and 93.94% respectively, and the F1 scores were also close to 93%. The Gamma and Beta frequency bands were relatively stable, with accuracies of 92.88% and 91.46% respectively, and the F1 scores remained at about 91%, showing good recognition ability. The performance of the Theta frequency band was good, but the standard deviation was large (13.33%), and the stability was relatively low. When all frequency band information was fused, the performance of the model reached the best, the accuracy was increased to 95.71%, the F1 score reached 94.00%, and the standard deviation was significantly reduced, showing stronger robustness and generalization ability. This result proves that the method proposed in this application can effectively identify the fatigue state of the driver, as shown in Table 1.

[0269] Table 1 Data table for effect verification

[0270]

[0271] A driver mental fatigue recognition and intervention system based on EEG signals, including a physiological information acquisition device, an algorithm calculation unit, and an in-vehicle intervention unit;

[0272] Physiological information acquisition device: used to collect the physiological information of the driver, including the EEG signals and eye movement data of the driver, etc.;

[0273] Algorithm calculation unit: As the management calculation center of the brain-computer interface, according to the received electroencephalogram signals of the driver, it makes predictions according to the designed detection and recognition algorithm, determines the mental fatigue state of the driver, and based on the future mental fatigue state of the driver, selects an appropriate intervention plan to intervene in the driver's fatigue to ensure road safety;

[0274] Vehicle-mounted intervention unit: According to the selected intervention plan, it provides reasonable solutions for the driver's fatigue intervention to achieve the elimination of the driver's fatigue and ensure road safety.

[0275] The above embodiments only represent the specific implementation manners of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the protection scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the technical solution of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application.

Claims

1. A method for identifying and intervening in driver mental fatigue based on electroencephalogram signals, characterized in that, Including: S1: Preprocess the driver's physiological signals; S2: Construct a brain functional network based on the preprocessed physiological signals; S3: Extract electroencephalogram (EEG) topological features from the brain functional network; S4: Input the EEG topological features into an identification model, and the identification model identifies the driver's mental fatigue state according to the spatio-temporal features of the EEG topology; S5: Determine fatigue intervention measures according to the driver's mental fatigue state; The said S4 includes: S41: Input the EEG feature dataset into Chebyshev polynomial graph convolution to learn the dependency relationships existing between feature sets, and the calculation formula is: Among them, T k (A) is the k-th order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the input matrix reflecting the EEG characteristics, which describes the characteristic information of each EEG channel or electrode at a certain time point t, and K is the order of the Chebyshev polynomial; S42: Define the recursion of higher-order Chebyshev polynomials: T0(A) = I T1(A) = A T k (A) = 2A·T k-1 (A) - T k-2 (A) for k≥2 where I is the identity matrix, A is the adjacency matrix, T0(A) and T1(A) are the identity matrix and the adjacency matrix respectively, and T k (A) continuously captures the information of the k-th order neighborhood through the recurrence relation; through this recurrence relation, the high-order Chebyshev polynomials are calculated; S43: Convolve the Chebyshev polynomials and aggregate the outputs to obtain the sum of information propagation between all electrode channels; finally, add a bias term b to the convolution result to adjust the offset in the feature space, and the specific calculation formula is as follows: where, T k (A) is the k-th order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the feature matrix at time step t, and b is the learned bias term; S44: To learn θ k and b, compare Output(t) with the actual t e and minimize the error Z through a loss function. The specific calculation formula is as follows: where I is the total number of data sets or samples, n i is the number of elements in the i-th data subset, t e (i,j) is the actual value or observed value of the j-th element in the i-th data set or sample, t a (i,j) is the predicted value or expected value of the j-th element in the i-th data set or sample; S45: The aggregation result generates time features using time encoding, and converts the time information into a vector with a fixed dimension, and the specific calculation formula is as follows: TimeFeature = W·t where W is the learned weight, t is the matrix of time steps, and TimeFeature is the time feature generated after time encoding; S46: Combine the time feature with the convolution kernel, and dynamically generate the weights of the convolution kernel by inputting the time feature into a fully connected layer, and the specific calculation formula is as follows: Kernel(t) = MetaLayer(TimeFeature) where MetaLayer is a fully connected layer responsible for generating the convolution kernel from the time feature, and Kernel(t) is the generated convolution kernel; S47: Convolve the dynamically generated convolution with the EEG feature data in a sliding window manner. For each time step i, the sliding window selects continuous K time step data, and convolve the selected data using the generated convolution kernel. The specific calculation formula is as follows: Among them, Y i is the convolution output result, W k is the convolution kernel weight, N i+k is the data of the (i + k)-th time step in the EEG feature data; S48: Use a semi-regression prediction model to predict the output value. For each time step t, the calculation formula of the model is as follows: Among them, y t is the predicted fatigue state value at time step t, X 1:t is all historical data from time step 1 to t, is all predicted fatigue state values from time step 1 to t - 1, and f(·) is the prediction function.

2. The method for identifying and intervening driver mental fatigue based on electroencephalogram signals according to claim 1, characterized in that The said S2 includes: S21: Process the EEG signals using non-linear canonical polyadic decomposition; S22: Calculate the components of the EEG signals after decomposition according to different frequency ranges; S23: Separate the frequency components within the frequency range corresponding to the frequency band to be extracted; S24: Convert different frequency bands into corresponding frequency components, calculate the functional connections between different brain regions under different frequency components to obtain the correlation matrix between different component signals; S25: Construct a brain functional network based on the correlation matrix.

3. The driver mental fatigue recognition and intervention method based on electroencephalogram signals according to claim 1, wherein, The said S2 also includes calculating network metrics to analyze the connection characteristics of the brain after constructing the brain functional network. The specific calculation metrics include node strength, clustering coefficient, eigenvector centrality, local efficiency, global efficiency, small-world property, and shortest path.

4. A method for identifying and intervening in driver mental fatigue based on electroencephalogram signals according to claim 1, characterized in that, The physiological signals in S1 include EEG signals, and the preprocessing of EEG signals includes: Perform band-pass filtering and power frequency notch filtering on the i electrode channel signals in the collected EEG signals to retain the EEG signals within the required frequency band range; Perform principal component analysis on the filtered EEG signal data, retain the EEG signal data with the cumulative variance contribution reaching the standard, and decompose the obtained EEG signal data into independent components; Use the EEG data created by the synthetic electrooculogram (EOG) channels to detect blinks, horizontal and vertical movements; components with high temporal or spatial correlation are marked as artifacts; the specific calculation formula is as follows: where ρ is the correlation coefficient calculated between the signal S i and the electrooculogram (EOG) signal, S i is the i-th component of the electroencephalogram (EEG) EOG signal, EOG is the electrooculogram EOG signal, is the standard deviation of S i indicating the degree of dispersion of the data distribution of the EEG signal component S i , and σ EOG is the standard deviation of the EOG signal, indicating the degree of dispersion of the data distribution of the electrooculogram signal EOG; Remove the artifact components with high temporal or spatial correlation to obtain the corresponding EEG data; Segment the continuous EEG signal into smaller time periods for analysis and calculation, and use a time window for calculation; Use baseline correction to remove the pre-stimulus fluctuations in the EEG signal data and normalize the EEG signal within the time window.

5. The driver mental fatigue recognition and intervention method based on EEG signals according to claim 1, characterized in that, The physiological signal described in S1 also includes eye movement data. The preprocessing of the eye movement data includes: using cubic spline interpolation to fit the spline function to interpolate and complete the eye movement data: S i f(x) = i(x - x i ) 3 + b i (x - x i ) 2 + c i (x - x i ) + d i Among them, S i (x) is the vertical coordinate of the gaze, x is the horizontal coordinate of the complemented interpolation point, x i is the horizontal coordinate of the known gaze, a i , b i , c i and d i are determined by the boundary conditions and continuity requirements.

6. The driver mental fatigue recognition and intervention method based on electroencephalogram signals according to claim 2, characterized in that Extraction of the signal spectrum. The calculation formula of S22 is: Among them, X(f) represents the signal spectrum, x(t) is the component after the decomposition of the EEG signal, f is the acquisition frequency of the EEG signal, and j is the imaginary unit.

7. A method for identifying and intervening in driver mental fatigue based on electroencephalogram signals according to claim 2, characterized in that, The said S24 includes: representing the power distribution of the signal at different frequencies through the power spectral density P(f): P(f) = |X(f)| 2 Convert the different frequency bands in the EEG signal into corresponding frequency components, calculate the functional connectivity between different brain regions under different frequency components, and construct a connection matrix based on the correlation relationship between each electrode signal to obtain: where x i represents the component of the i-th EEG signal, and C ij is the functional connectivity strength between the i-th and j-th components calculated using the phase-locking value, COV(x i , x j ) is the covariance between the EEG signal data of the i-th and j-th components, and Var(x i ) and Var(x j ) are the variances between the EEG signal data of the i-th and j-th components; the correlation matrix C between different component signals is obtained through the calculation of the above formula, where each element C ij represents the functional connectivity strength between components i and j in a given frequency band.

8. A method for identifying and intervening in driver mental fatigue based on electroencephalogram signals according to claim 2, characterized in that, S25 includes: using the correlation matrix C to construct the brain functional network G. The threshold formula of the brain functional network is as follows: G=(V,E) Among them, V is the set of electrode channels, E is the set of edges, and the edges are defined according to the correlation value to describe the connectivity between brain regions; Apply the threshold T to the correlation matrix to determine the importance. Specifically, those below the threshold are regarded as invalid connections and need to be discarded, and those above the threshold are regarded as valid connections and retained. Define the adjacency matrix A as: The adjacency matrix A represents the functional connectivity in the brain functional network.

9. A driver mental fatigue recognition and intervention system based on electroencephalogram signals, characterized in that, Apply a method for identifying and intervening in driver mental fatigue based on EEG signals according to any one of claims 1-8, including a physiological information acquisition device, an algorithm calculation unit, and an in-vehicle intervention unit; Physiological information acquisition device: used to collect the physiological information of the driver, including the driver's EEG signal and eye movement data, etc.; Algorithm calculation unit: According to the received EEG signal of the driver, perform prediction according to the designed detection and recognition algorithm to determine the mental fatigue state of the driver. Based on the future mental fatigue state of the driver, select an intervention plan to intervene in the driver's fatigue; In-vehicle intervention unit: According to the selected intervention plan, provide solutions and suggestions for the driver's fatigue intervention.

Images