Electroencephalogram signal-based driver mental fatigue recognition intervention method and system

Abstract

The invention discloses a driver mental fatigue recognition intervention method and system based on electroencephalogram signals, relates to the technical field of intelligent traffic, and solves the problem that fatigue recognition robustness is not high due to the fact that a fatigue recognition algorithm is easily interfered by the external environment. The method comprises the following steps: S1, preprocessing a physiological signal of a driver; s2, constructing a brain function network according to the preprocessed physiological signals; s3, electroencephalogram topological features are extracted from the brain function network; s4, inputting the electroencephalogram topological features into a recognition model, and recognizing the mental fatigue state of the driver by the recognition model according to the electroencephalogram topological features; s5, determining fatigue intervention measures according to the mental fatigue state of the driver; by analyzing the electroencephalogram signals, the system can detect the fatigue condition of the driver in real time.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method and system for identifying and intervening in driver mental fatigue based on electroencephalogram signals. Background Art

[0002] Mental fatigue is a serious problem that affects driving safety, especially for long or repetitive driving tasks. Driver mental fatigue can lead to decreased attention, slower reaction speed, impaired decision-making ability, and significantly increase the risk of accidents. At present, the detection of fatigue driving is mostly based on image recognition and is easily interfered by external environmental factors, such as lighting, eye occlusions, etc. In addition, image recognition relies on external visual information and cannot truly reflect the mental state of the driver's brain. Image recognition also requires a large amount of computing resources to process images, resulting in the urgent need to improve the robustness and accuracy of image recognition algorithms.

[0003] Brain-computer interface systems can use electroencephalogram (EEG) signals to detect changes in brain activity. By analyzing these patterns, brain-computer interfaces can detect early signs of mental fatigue and assess the driver's cognitive state in real time. The application of brain-computer interface technology in fatigue detection mainly includes two key links: data acquisition and system identification intervention. In terms of data acquisition, electroencephalogram (EEG) signals are collected in real time through non-invasive electrode sensors. These signals can reflect the working state of the driver's brain and provide important information about the degree of fatigue. In terms of system identification intervention, with the rapid development of artificial intelligence technology, machine learning and deep learning are widely used in the analysis and interpretation of EEG signals. Machine learning methods can extract features from historical data by training models to help identify fatigue states. Deep learning technology, with its powerful self-learning and automatic feature extraction capabilities, can handle more complex signal features and achieve higher accuracy and robustness. In recent years, deep learning algorithms such as convolutional neural networks (CNN) and long short-term memory networks (LSTM) have become mainstream technologies in EEG signal analysis, which can automatically identify fatigue patterns from a large amount of complex EEG data, greatly improving the efficiency and accuracy of fatigue detection.

[0004] Although existing fatigue detection methods based on EEG signals are real-time and highly accurate, they still face some challenges, such as noise interference, individual differences, low computational efficiency, difficulty in obtaining labeled data, and limited signal spatial resolution. EEG signals are easily affected by the external environment, electromagnetic interference, and user movements, resulting in high noise. Although existing filtering techniques can reduce some noise, they may also weaken the integrity of the signal. At the same time, the high requirements of deep learning algorithms for computing resources may affect real-time performance, especially in embedded systems. The reliance of the model training process on high-quality fatigue labels also further increases the complexity and subjectivity of data acquisition. Furthermore, due to the limited number of electrodes and insufficient spatial resolution of the acquisition, non-invasive EEG devices are difficult to fully reflect brain activity, thereby limiting the ability to analyze complex brain activity. Summary of the invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a driver mental fatigue identification intervention method and system based on EEG signals, which solves the problem that the existing fatigue recognition algorithm is easily interfered by the external environment, resulting in low robustness of fatigue recognition.

[0006] A driver mental fatigue identification and intervention method based on EEG signals, comprising:

[0007] S1: pre-processing the driver's physiological signals;

[0008] S2: Construct brain functional network based on preprocessed physiological signals;

[0009] S3: Extracting EEG topological features from brain functional networks;

[0010] S4: Input the EEG topological features into the recognition model, and the recognition model recognizes the driver's mental fatigue state according to the EEG topological features, wherein the recognition model includes an asynchronous graph diffusion network, a time-aware convolutional network, and a semi-autoregressive prediction network; the asynchronous graph diffusion network models the spatial relationship between EEG channels, the time-aware convolutional network models the temporal relationship between EEG channels, and the semi-autoregressive prediction network integrates the spatiotemporal features learned by the asynchronous graph diffusion network and the time-aware convolutional network to provide accurate fatigue state prediction;

[0011] S5: Determine fatigue intervention measures based on the driver’s mental fatigue status.

[0012] The driver's mental fatigue state includes wakefulness, sleepiness and drowsiness.

[0013] Further, the S2 includes:

[0014] S21: Processing EEG signals using nonlinear normative multivariate decomposition;

[0015] S22: Calculate the decomposed components of the EEG signal according to the different frequency ranges;

[0016] S23: separating the frequency components within the frequency range corresponding to the frequency band to be extracted;

[0017] S24: Convert different frequency bands into corresponding frequency components, calculate the functional connections between different brain regions under different frequency components, and obtain the correlation matrix between different component signals;

[0018] S25: Constructing brain functional network based on correlation matrix.

[0019] Furthermore, S2 also includes S25: after constructing the brain functional network, calculating the network metric to analyze the connection characteristics of the brain, and specific calculation indicators include node strength, clustering coefficient, eigenvector center, local efficiency, global efficiency, small world property and shortest path.

[0020] Further, the S4 includes:

[0021] S41: Asynchronous graph convolution is to input the EEG feature data set into the Chebyshev polynomial graph convolution to learn the dependencies between feature sets. The calculation formula is:

[0022]

[0023] Among them, T k (A) is the kth order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the input matrix reflecting the EEG characteristics, describing the characteristic information of each EEG channel or electrode at a certain time point t, and K is the order of the Chebyshev polynomial;

[0024] S42: Recursively define higher-order Chebyshev polynomials:

[0025] T0(A)=I

[0026] T1(A)=A

[0027] T k (A) = 2A·T k-1 (A)-T k-2 (A) for k ≥ 2

[0028] Where I is the identity matrix, A is the adjacency matrix, T0(A) and T1(A) are the identity matrix and adjacency matrix respectively, T k (A) The information of the k-order neighborhood is continuously captured through the recursive relationship; through this recursive relationship, the high-order Chebyshev polynomial is calculated;

[0029] S43: Convolve the Chebyshev polynomials and aggregate the outputs to obtain the sum of information propagation between all electrode channels; finally, add a bias term b to the convolution result to adjust the offset in the feature space. The specific calculation formula is as follows:

[0030]

[0031] Among them, T k (A) is the kth order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the feature matrix at time step t, and b is the learned bias term;

[0032] S44: To learn θ k and b, compare Output(t) with the actual t e For comparison, the error Z is minimized through the loss function. The specific calculation formula is as follows:

[0033]

[0034] Where I is the total number of data sets or samples, n i is the number of elements in the ith data subset, t e (i,j) is the actual or observed value of the jth element in the i-th data set or sample, t a (i,j) is the predicted value or expected value of the jth element in the i-th data set or sample;

[0035] S45: Convertible time-aware convolutional network is the aggregation result that uses time encoding to generate time features and converts time information into a vector with a fixed dimension. The specific calculation formula is as follows:

[0036] TimeFeature=W·t

[0037] Among them, W is the learned weight, t is the matrix of time steps, and TimeFeature is the time feature generated after time encoding;

[0038] S46: Combine the time feature with the convolution kernel and dynamically generate the weight of the convolution kernel by inputting the time feature into a fully connected layer. The specific calculation formula is as follows:

[0039] Kernel(t)=MetaLayer(TimeFeature)

[0040] Among them, MetaLayer is a fully connected layer responsible for generating convolution kernels from temporal features, and Kernel(t) is the generated convolution kernel;

[0041] S47: The dynamically generated convolution is used to perform convolution calculation on the EEG feature data in a sliding window manner. For each time step i, the sliding window selects data of K consecutive time steps, and the selected data is convolved with the generated convolution kernel. The specific calculation formula is as follows:

[0042]

[0043] Among them, Y i is the convolution output, W k is the convolution kernel weight, N i+k is the data of the i+kth time step in the EEG feature data;

[0044] S48: The output value is predicted using a semi-autoregressive prediction network. For each time step t, the calculation formula of the model is as follows:

[0045]

[0046] Among them, y t is the fatigue state prediction value at time step t, X 1:t is all the historical data from time step 1 to t, are all fatigue state predictions from time step 1 to t-1, and f(·) is the prediction function.

[0047] Furthermore, the physiological signal in S1 includes an electroencephalogram signal, and the preprocessing of the electroencephalogram signal includes:

[0048] Perform bandpass filtering and power frequency notch processing on the i-electrode channel signal in the collected EEG signal to retain the EEG signal within the required frequency band;

[0049] Perform principal component analysis on the filtered EEG signal data to retain the EEG signal data whose cumulative variance contribution reaches the standard, and decompose the obtained EEG signal data into opposing components;

[0050] EEG data created using synthetic electrooculogram (EOG) channels are used to detect eye blinks, horizontal and vertical movements. Components with high temporal or spatial correlation (ρ>0.8) are marked as artifacts. The specific calculation formula is as follows:

[0051]

[0052] Where ρ is the calculated signal S i The correlation coefficient between the electrooculogram (EOG) signal, S i is the i-th component of the electroencephalogram (EEG) signal, and EOG is the electrooculogram (EOG) signal, which is usually used to record the horizontal and vertical movements of the eyes, mainly reflecting the blinking and eye movement of the eyes. YesSi The standard deviation of the EEG signal component S i The degree of dispersion of the data distribution, σ EOG It is the standard deviation of the EOG signal, which indicates the degree of dispersion of the data distribution of the electrooculogram signal EOG.

[0053] Remove artifact components with high temporal or spatial correlation to obtain corresponding EEG data;

[0054] Split the continuous EEG signal into smaller time periods for analysis and calculation, using time windows for calculation;

[0055] Baseline correction removes pre-stimulus fluctuations in the EEG data and normalizes the EEG signals within the time window.

[0056] Furthermore, the physiological signal in S1 also includes eye movement data, and the preprocessing of the eye movement data includes: using cubic spline interpolation fitting spline function to interpolate and complete the eye movement data:

[0057] S i (x) = a i (xx i ) 3 +b i (xx i ) 2 +c i (xx i )+d i

[0058] Among them, S i (x) is the vertical coordinate of the gaze, x is the horizontal coordinate of the interpolation point, and x i is the horizontal coordinate of the known gaze, a i 、b i 、c i and d i It is determined by boundary conditions and continuity requirements.

[0059] Further, the S21 includes:

[0060] The EEG signal is considered as a multidimensional tensor E, where each dimension corresponds to a specific aspect of the EEG signal, including time, space, and frequency. Given a 3rd-order tensor The regular multivariate decomposition calculation formula is as follows:

[0061]

[0062] in, represents the outer product, represents the modal factor vector, and ΔE represents the error tensor. The tensor modulus includes time (time component), frequency (spectral component), and electrode (spatial component). In order to separate the above components into a tensor of rank 1, the calculation formula is as follows:

[0063] T=a1°b1°c1+a2°b2°c2+E

[0064] Among them, a r represents the time component, b r represents the frequency component, c r Represents spatial component, r=1,2.

[0065] The nonlinear least squares (NLS) is used to solve the decomposition and minimize the reconstruction error. The specific calculation formula is as follows:

[0066]

[0067] in, ‖·‖ F It is the Frobenius specification.

[0068] Each factor matrix U (n) The specific formula for calculating the relative error is as follows:

[0069]

[0070] in, is the estimated factor matrix, U (n) is the actual factor matrix, P is the permutation matrix, and D is the scaling matrix.

[0071] Furthermore, the calculation formula of S22 is:

[0072]

[0073] Among them, X(f) represents the signal spectrum, x(t) is the component of the EEG signal after decomposition, f is the acquisition frequency of the EEG signal, and j is the imaginary unit.

[0074] Further, the S24 includes: representing the power distribution of the signal at different frequencies by a power spectrum density P(f):

[0075] P(f)=|X(f)| 2

[0076] The different frequency bands in the above EEG signals are converted into corresponding frequency components, the functional connections between different brain regions under different frequency components are calculated, and the connection matrix is ​​constructed based on the correlation between each electrode signal to obtain:

[0077]

[0078] Among them, x i represents the component of the i-th EEG signal, C ij The functional connectivity strength between the i-th and j-th components is calculated using the phase-locking value, COV(x i ,x j ) is the covariance between the EEG signal data of the i-th and j-th components, Var(x i ) and Var(x j ) is the variance between the EEG signal data of the i-th and j-th components. The correlation matrix C between different component signals is obtained by calculating the above formula, where each element C ij represents the strength of functional connectivity between components i and j in a given frequency band;

[0079] Further, S25 includes: using the correlation matrix C to construct a brain function network G, and the thresholding formula of the brain function network is as follows:

[0080] G=(V,E)

[0081] Where V is the set of electrode channels, E is the set of edges, and the edges are defined as the connectivity between brain regions according to the correlation value;

[0082] A threshold T is applied to the correlation matrix to determine the importance. Specifically, if the value is below the threshold, it is considered as an invalid connection and needs to be discarded, and if it is above the threshold, it is considered as a valid connection and retained. The adjacency matrix A is defined as:

[0083]

[0084] The above binary adjacency matrix A represents the functional connections in the network.

[0085] A driver mental fatigue identification and intervention system based on EEG signals, including a physiological information acquisition device, an algorithm calculation unit and an on-vehicle intervention unit;

[0086] Physiological information collection equipment: used to collect the driver's physiological information, including the driver's EEG signals and eye movement data;

[0087] Algorithm computing unit: As the management computing center of the brain-computer interface, it predicts the driver's mental fatigue state based on the received EEG signals and the designed detection and recognition algorithm. Based on the driver's future mental fatigue state, it selects the appropriate intervention plan to intervene in the driver's fatigue and ensure road safety.

[0088] On-vehicle intervention unit: Based on the selected intervention plan, it provides reasonable solutions for driver fatigue intervention, eliminates driver fatigue and ensures road safety.

[0089] The beneficial effects of the present invention include:

[0090] (1) The present invention can detect the driver's fatigue status in real time by analyzing EEG signals. When the driver's mental state is detected to be abnormal, the system will automatically sound an alarm to remind the driver to rest or stop driving, thereby avoiding accidents caused by fatigue driving.

[0091] (2) The present invention takes into account the driver's driving experience and automatically pushes customized rest suggestions based on the EEG signal characteristics of each driver and their driving habits. In addition, the system can also combine the vehicle-mounted equipment and environmental information to automatically adjust the in-vehicle environment to help restore the driver's situational awareness when the driver is identified to be fatigued. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 This is a flow chart of a method for identifying and intervening in driver mental fatigue based on EEG signals according to an embodiment of the present application.

[0093] Figure 2 Schematic diagram of the recognition model algorithm involved in the embodiment of the present application. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0095] A driver mental fatigue identification and intervention method based on EEG signals, such as Figure 1 As shown, including:

[0096] S1: pre-processing the driver's physiological signals;

[0097] S2: Construct brain functional network based on preprocessed physiological signals;

[0098] S3: Extracting EEG topological features from brain functional networks;

[0099] S4: inputting the EEG topological features into the recognition model, and the recognition model recognizes the driver's mental fatigue state according to the EEG topological features;

[0100] S5: Determine fatigue intervention measures based on the driver’s mental fatigue status.

[0101] The driver's mental fatigue state includes wakefulness, sleepiness and drowsiness.

[0102] Once the driver is detected to be in a state of fatigue, the system needs to take intervention measures quickly to reduce the risk of accidents. Intervention methods are divided into two categories: one is automatic intervention based on physiological reactions, and the other is reminder intervention based on external prompts. The former stimulates the driver to recover from fatigue by adjusting the driver's physiological state, such as through mild vibration, audio signals or visual stimulation. The latter reminds the driver to take a rest or take appropriate rest measures through intelligent reminder systems, such as sound alarms, seat vibrations or dashboard warning lights.

[0103] In another embodiment, the S2 includes:

[0104] S21: Processing EEG signals using nonlinear normative multivariate decomposition;

[0105] S22: Calculate the decomposed components of the EEG signal according to the different frequency ranges;

[0106] S23: separating the frequency components within the frequency range corresponding to the frequency band to be extracted;

[0107] S24: Convert different frequency bands into corresponding frequency components, calculate the functional connections between different brain regions under different frequency components, and obtain the correlation matrix between different component signals;

[0108] S25: Constructing brain functional network based on correlation matrix.

[0109] In another embodiment, S2 also includes S25: after constructing the brain functional network, calculating the network metric to analyze the connection characteristics of the brain, and the specific calculation indicators include node strength, clustering coefficient, eigenvector center, local efficiency, global efficiency, small world property and shortest path.

[0110] In another embodiment, the S4 includes:

[0111] S41: Asynchronous graph convolution is to input the EEG feature data set into the Chebyshev polynomial graph convolution to learn the dependencies between feature sets. The calculation formula is:

[0112]

[0113] Among them, T k (A) is the kth order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the input matrix reflecting the EEG characteristics, describing the characteristic information of each EEG channel or electrode at a certain time point t, and K is the order of the Chebyshev polynomial;

[0114] S42: Recursively define higher-order Chebyshev polynomials:

[0115] T0(A)=I

[0116] T1(A)=A

[0117] T k (A) = 2A·T k-1 (A)-T k-2 (A) for k ≥ 2

[0118] Where I is the identity matrix, A is the adjacency matrix, T0(A) and T1(A) are the identity matrix and adjacency matrix respectively, T k (A) The information of the k-order neighborhood is continuously captured through the recursive relationship; through this recursive relationship, the high-order Chebyshev polynomial is calculated;

[0119] S43: Convolve the Chebyshev polynomials and aggregate the outputs to obtain the sum of information propagation between all electrode channels; finally, add a bias term b to the convolution result to adjust the offset in the feature space. The specific calculation formula is as follows:

[0120]

[0121] Among them, T k (A) is the kth order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the feature matrix at time step t, and b is the learned bias term;

[0122] S44: To learn θ k and b, compare Output(t) with the actual t e For comparison, the error Z is minimized through the loss function. The specific calculation formula is as follows:

[0123]

[0124] Where I is the total number of data sets or samples, n i is the number of elements in the ith data subset, t e (i,j) is the actual or observed value of the jth element in the i-th data set or sample, t a (i,j) is the predicted value or expected value of the jth element in the i-th data set or sample;

[0125] S45: Convertible time-aware convolutional network is the aggregation result that uses time encoding to generate time features and converts time information into a vector with a fixed dimension. The specific calculation formula is as follows:

[0126] TimeFeature=W·t

[0127] Among them, W is the learned weight, t is the matrix of time steps, and TimeFeature is the time feature generated after time encoding;

[0128] S46: Combine the time feature with the convolution kernel and dynamically generate the weight of the convolution kernel by inputting the time feature into a fully connected layer. The specific calculation formula is as follows:

[0129] Kernel(t)=MetaLayer(TimeFeature)

[0130] Among them, MetaLayer is a fully connected layer responsible for generating convolution kernels from temporal features, and Kernel(t) is the generated convolution kernel;

[0131] S47: The dynamically generated convolution is used to perform convolution calculation on the EEG feature data in a sliding window manner. For each time step i, the sliding window selects data of K consecutive time steps, and the selected data is convolved with the generated convolution kernel. The specific calculation formula is as follows:

[0132]

[0133] Among them, Y i is the convolution output, W k is the convolution kernel weight, N i+k is the data of the i+kth time step in the EEG feature data;

[0134] S48: The output value is predicted using a semi-autoregressive prediction network. For each time step t, the calculation formula of the model is as follows:

[0135]

[0136] Among them, y t is the fatigue state prediction value at time step t, X 1:t is all the historical data from time step 1 to t, are all fatigue state predictions from time step 1 to t-1, and f(·) is the prediction function.

[0137] In another embodiment, the physiological signal in S1 includes an electroencephalogram signal, and the preprocessing of the electroencephalogram signal includes:

[0138] Perform bandpass filtering and power frequency notch processing on the i-electrode channel signal in the collected EEG signal to retain the EEG signal within the required frequency band;

[0139] Perform principal component analysis on the filtered EEG signal data to retain the EEG signal data whose cumulative variance contribution reaches the standard, and decompose the obtained EEG signal data into opposing components;

[0140] EEG data created using synthetic electrooculogram (EOG) channels are used to detect eye blinks, horizontal and vertical movements. Components with high temporal or spatial correlation (ρ>0.8) are marked as artifacts. The specific calculation formula is as follows:

[0141]

[0142] Where ρ is the calculated signal S i The correlation coefficient between the electrooculogram (EOG) signal, S i is the i-th component of the electroencephalogram (EEG) signal, and EOG is the electrooculogram (EOG) signal, which is usually used to record the horizontal and vertical movements of the eyes, mainly reflecting the blinking and eye movement of the eyes. YesS i The standard deviation of the EEG signal component S i The degree of dispersion of the data distribution, σ EOG It is the standard deviation of the EOG signal, which indicates the degree of dispersion of the data distribution of the electrooculogram signal EOG.

[0143] Remove artifact components with high temporal or spatial correlation to obtain corresponding EEG data;

[0144] Split the continuous EEG signal into smaller time periods for analysis and calculation, using time windows for calculation;

[0145] Baseline correction removes pre-stimulus fluctuations in the EEG data and normalizes the EEG signals within the time window.

[0146] In another embodiment, the physiological signal in S1 further includes eye movement data, and the preprocessing of the eye movement data includes: interpolating and completing the eye movement data using a cubic spline interpolation fitting spline function:

[0147] S i (x) = a i (xx i ) 3 +b i (xx i ) 2 +c i (xx i )+d i

[0148] Among them, S i (x) is the vertical coordinate of gaze, x i is the horizontal coordinate of gaze, ai 、b i 、c i and d i It is determined by boundary conditions and continuity requirements.

[0149] In another embodiment, the S21 includes:

[0150] The EEG signal is considered as a multidimensional tensor E, where each dimension corresponds to a specific aspect of the EEG signal, including time, space, and frequency. Given a 3rd-order tensor The regular multivariate decomposition calculation formula is as follows:

[0151]

[0152] Among them, ° represents the outer product, represents the modal factor vector, and ΔE represents the error tensor. The tensor modulus includes time (time component), frequency (spectral component), and electrode (spatial component). In order to separate the above components into a tensor of rank 1, the calculation formula is as follows:

[0153] T=a1°b1°c1+a2°b2°c2+E

[0154] Among them, a r represents the time component, b r represents the frequency component, c r Represents the spatial component.

[0155] The nonlinear least squares (NLS) is used to solve the decomposition and minimize the reconstruction error. The specific calculation formula is as follows:

[0156]

[0157] in, ||·|| F is the Frobenius specification.

[0158] Each factor matrix U (n) The specific formula for calculating the relative error is as follows:

[0159]

[0160] in, is the estimated factor matrix, U (n) is the actual factor matrix, P is the permutation matrix, and D is the scaling matrix.

[0161] In another embodiment, the calculation formula of S22 is:

[0162]

[0163] Among them, X(f) represents the signal spectrum, x(t) is the component of the EEG signal after decomposition, f is the acquisition frequency of the EEG signal, and j is the imaginary unit.

[0164] In another embodiment, the S24 includes: representing the power distribution of the signal at different frequencies by a power spectrum density P(f):

[0165] P(f)=|X(f)| 2

[0166] The different frequency bands in the above EEG signals are converted into corresponding frequency components, the functional connections between different brain regions under different frequency components are calculated, and the connection matrix is ​​constructed based on the correlation between each electrode signal to obtain:

[0167]

[0168] Among them, x i represents the component of the i-th EEG signal, C ij The functional connectivity strength between the i-th and j-th components is calculated using the phase-locking value, COV(x i ,x j ) is the covariance between the EEG signal data of the i-th and j-th components, Var(x i ) and Var(x j ) is the variance between the EEG signal data of the i-th and j-th components. The correlation matrix C between different component signals is obtained by calculating the above formula, where each element C ij represents the strength of functional connectivity between components i and j in a given frequency band;

[0169] In another embodiment, S25 includes: using the correlation matrix C to construct a brain function network G, and the thresholding formula of the brain function network is as follows:

[0170] G=(V,E)

[0171] Where V is the set of electrode channels, E is the set of edges, and the edges are defined as the connectivity between brain regions according to the correlation value;

[0172] A threshold T is applied to the correlation matrix to determine the importance. Specifically, if the value is below the threshold, it is considered as an invalid connection and needs to be discarded, and if it is above the threshold, it is considered as a valid connection and retained. The adjacency matrix A is defined as:

[0173]

[0174] The above binary adjacency matrix A represents the functional connections in the network.

[0175] In another embodiment, a method for identifying and intervening in driver mental fatigue based on EEG signals includes:

[0176] S1. Divide the driver's fatigue level during driving into three stages: awake, sleepy and drowsy, and collect the driver's physiological signals in the state of mental fatigue, including the driver's EEG signals and eye movement data;

[0177] The collected physiological signals are preprocessed, including filtering, re-referencing, segmentation, baseline correction, independent component analysis, and interpolation of eye movement data. The baseline correction includes: in each signal segment, select the baseline interval (e.g., -200ms to 000ms before stimulation) to calculate the baseline potential. Subtract the baseline potential from each signal segment to ensure that the corrected signal takes the baseline as the zero point.

[0178] The specific plan is as follows:

[0179] S11, performing bandpass filtering and power frequency notch processing on the i electrode channel signal in the collected EEG signal to retain the required frequency band range (0.1 Hz-50 Hz);

[0180] S12. Perform principal component analysis on the filtered EEG signal data, and retain data with a cumulative variance contribution of 99%.

[0181] S13, decomposing the preprocessed EEG data into independent components (S);

[0182] S14. Use EEG data created using synthetic electrooculogram (EOG) channels to detect eye blinks, horizontal and vertical movements. Components with high temporal or spatial correlation (ρ>0.8) are marked as artifacts. The specific calculation formula is as follows:

[0183]

[0184] S15, remove artifact components with high temporal or spatial correlation to obtain clean EEG data;

[0185] S16, segmentation divides the continuous EEG signal into smaller, manageable time periods for analysis and calculation, using an 8-second time window for calculation. Baseline correction is used to remove pre-stimulus fluctuations in the EEG signal data and standardize the EEG signal within an 8-second time window;

[0186] S17, using cubic spline interpolation fitting spline function to interpolate and complete the eye movement data, the specific formula is:

[0187] S i (x) = a i (xx i ) 3 +b i (xx i ) 2 +ci (xx i )+d i

[0188] Among them, S i (x) is the vertical coordinate of the gaze, x is the horizontal coordinate of the interpolation point, and x i is the horizontal coordinate of the known gaze, a i 、b i 、c i and d i It is determined by boundary conditions and continuity requirements.

[0189] S2. Extract different frequency bands of EEG signals (Delta, Theta, Alpha, Beta and Gamma) and construct brain functional networks to extract corresponding topological features. The specific scheme is as follows:

[0190] In order to reconstruct the EEG signal in time with low loss, nonlinear canonical multivariate decomposition (CPD) is used to process the EEG signal. The EEG signal can be regarded as a multidimensional tensor E, where each dimension corresponds to a specific aspect of the EEG signal, including time, space, and frequency. Given a 3rd-order tensor The regular multivariate decomposition calculation formula is as follows:

[0191]

[0192] in, represents the outer product, represents the modal factor vector, and ΔE represents the error tensor. The tensor modulus includes time (time component), frequency (spectral component), and electrode (spatial component). In order to separate the above components into a tensor of rank 1, the calculation formula is as follows:

[0193]

[0194] Among them, a r represents the time component, b r represents the frequency component, c r Represents the spatial component.

[0195] The nonlinear least squares (NLS) is used to solve the decomposition and minimize the reconstruction error. The specific calculation formula is as follows:

[0196]

[0197] in, ‖·‖ F It is the Frobenius specification.

[0198] Each factor matrix U (n)The specific formula for calculating the relative error is as follows:

[0199]

[0200] in, is the estimated factor matrix, U (n) is the actual factor matrix, P is the permutation matrix, and D is the scaling matrix.

[0201] Then, depending on the frequency range, the calculation method used is as follows:

[0202]

[0203] Among them, x(t) is the component of the EEG signal after decomposition, f is the acquisition frequency of the EEG signal, and j is the imaginary unit.

[0204] In order to extract a specific frequency band and analyze the signal within a specific frequency range, the corresponding frequency components need to be isolated. The frequency ranges of different bands are Delta band (0.1-4Hz), Theta band (4-8Hz), Alpha band (8-12Hz), Beta band (12-30Hz) and Gamma band (30-40Hz). The power spectrum density P(f) is used to represent the power distribution of the signal at different frequencies. The specific formula is as follows:

[0205] P(f)=|X(f)| 2

[0206] The different frequency bands in the above EEG signals are converted into corresponding frequency components, and the functional connections between different brain regions under different frequency components are calculated. The connection matrix is ​​constructed based on the correlation between each electrode signal. The specific formula is as follows:

[0207]

[0208] Among them, x i represents the component of the i-th EEG signal, C ij The functional connectivity strength between the i-th and j-th components is calculated using the phase-locking value, COV(x i ,x j ) is the covariance between the EEG signal data of the i-th and j-th components, Var(x i ) and Var(x j ) is the variance between the EEG signal data of the i-th and j-th components. The correlation matrix C between different component signals is obtained by calculating the above formula, where each element C ij Represents the strength of functional connectivity between components i and j in a given frequency band.

[0209] Then the correlation matrix C is used to construct the brain function network G. The thresholding of the brain function network can be expressed using the following formula:

[0210] G=(V,E)

[0211] Among them, V is the set of electrode channels, E is the set of edges, and the edges are defined according to the correlation values ​​to describe the connectivity between brain regions.

[0212] In order to construct a more efficient brain function connectivity network, a threshold T is applied to the correlation matrix to determine which connections are considered important. In this method, if the threshold is lower than 15%, it is considered an invalid connection and needs to be discarded. If it is higher than 15%, it is considered a valid connection and retained. The adjacency matrix A is defined as:

[0213]

[0214] The above binary adjacency matrix A represents the functional connections in the network.

[0215] On the basis of constructing the brain functional network, various network metrics are calculated to analyze the brain's connection characteristics. Specific indicators include node strength, clustering coefficient, eigenvector center, local efficiency, global efficiency, small world property and shortest path.

[0216] The node strength represents the total functional connectivity of the brain area (electrode) corresponding to the node. The specific calculation formula is as follows:

[0217]

[0218] Among them, A ij represents the weight of the connection between components i and j of the adjacency matrix, N represents the number of components, S i It represents the i-th electrode.

[0219] Clustering coefficient C of electrode channel i i Quantifies the tightness of the neighbor clustering of electrode channel i. It measures the tendency of a component to form a cluster, and the specific calculation formula is as follows:

[0220]

[0221] Among them, E i represents the number of connecting edges between adjacent components, k i Represents the degree of quantity.

[0222] Eigenvector centrality is a measure of the influence of a component in a network. If a component is connected to other components that are themselves highly central, then the centrality of the component is high. The specific calculation formula is as follows:

[0223] A·v=λ·v

[0224] Among them, A is the adjacency matrix of the network, v is the centrality vector of the eigenvector (the centrality score of each node), and λ is the eigenvalue corresponding to the eigenvector.

[0225] Local efficiency measures the efficiency of information transmission between the neighbors of a component, considering only the neighbors of the component and the edges directly connecting them. The specific calculation formula is as follows:

[0226]

[0227] Among them, d jk is the shortest path distance between components i and j, k i Represents the degree of quantity.

[0228] Global efficiency reflects the efficiency of information transmission in the entire brain network. It is the average inverse shortest path length between all component pairs and is related to the global ability of the network to transmit information. The specific calculation formula is as follows:

[0229]

[0230] Where N represents the number of components, d jk is the shortest path distance between components i and j.

[0231] The small-world property shows that the network has a higher clustering coefficient and a shorter average path length between components, reflecting the efficiency of the human brain in processing and transmitting information. The specific calculation formula is as follows:

[0232]

[0233] Among them, C observed is the clustering coefficient of the observation network, C random is the clustering coefficient of a random network with the same number of components and edges, L observed is the average path length of the observed network, L random is the average path length of a random network.

[0234] The shortest path length d between two components i and j ij is the minimum number of edges that must be passed from i to j. The specific calculation formula is as follows:

[0235]

[0236] Where N represents the number of components, d jk is the shortest path distance between components i and j.

[0237] S4. Construct a recognition algorithm for driver mental fatigue, input the characteristics of the driver's EEG signal, identify the driver's mental fatigue state according to the characteristics, and classify the mental fatigue state into grades, such as Figure 2 As shown:

[0238] S41, obtaining an EEG feature data set X(t), and normalizing the collected EEG feature data set X(t) to obtain a normalized feature set N(t);

[0239] S42. Asynchronous graph convolution is to input the normalized feature set N(t) into the Chebyshev polynomial graph convolution to learn the dependency between feature sets. The calculation formula is:

[0240]

[0241] Among them, T k (A) is the kth order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, and K is the order of the Chebyshev polynomial.

[0242]

[0243] S43. In order to efficiently calculate high-order Chebyshev polynomials, their recursive definition is as follows:

[0244] T0(A)=I

[0245] T1(A)=A

[0246] T k (A) = 2A·T k-1 (A)-T k-2 (A) for k ≥ 2

[0247] Where I is the identity matrix and A is the adjacency matrix. Through this recursive relationship, high-order Chebyshev polynomials can be efficiently calculated.

[0248] S43, convolve the Chebyshev polynomials and aggregate the outputs to obtain the sum of information propagation between all electrode channels. Finally, add the bias term b to the convolution result to adjust the offset in the feature space. The specific calculation formula is as follows:

[0249]

[0250] Among them, T k (A) is the kth order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the feature matrix at time step t, and b is the learned bias term.

[0251] S44: The convertible time-aware convolutional network generates time features using time encoding for the results, converting the time information into a vector with a fixed dimension. The specific calculation formula is as follows:

[0252] TimeFeature=W·t

[0253] Among them, W is the learned weight, t is the matrix of time steps, and TimeFeature is the time feature generated after time encoding.

[0254] S45. Combine the time feature with the convolution kernel and dynamically generate the weight of the convolution kernel by inputting the time feature into a fully connected layer. The specific calculation formula is as follows:

[0255] Kernel(t)=MetaLayer(TimeFeature)

[0256] Among them, MetaLayer is a fully connected layer responsible for generating convolution kernels from temporal features, and Kernel(t) is the generated convolution kernel.

[0257] S46, convolution calculation is performed on the EEG feature data using the dynamically generated convolution and sliding window. For each time step i, the sliding window selects data of K consecutive time steps, and convolution is performed on the selected data using the generated convolution kernel. The specific calculation formula is as follows:

[0258]

[0259] Among them, Y i is the convolution output, W k is the convolution kernel weight, N i+k It is the data of the i+kth time step in the EEG feature data.

[0260] S47. The output value is predicted using a semi-autoregressive prediction network. For each time step t, the calculation formula of the model is as follows:

[0261]

[0262] Among them, y t is the fatigue state prediction value at time step t, X 1:t is all the historical data from time step 1 to t, are all fatigue state predictions from time step 1 to t-1, and f(·) is the prediction function.

[0263] S5. Provide reasonable fatigue intervention measures for drivers in a state of mental fatigue to reduce the risk of road accidents.

[0264] If the driver's mental state is predicted to be sober, no intervention is required;

[0265] If the driver's mental state is predicted to be fatigue, appropriate interventions should be taken based on the driver's current state, including playing music, vibrating the seat (steering wheel), and visual guidance from the HUD;

[0266] If the driver's mental state is predicted to be drowsy, strong intervention measures will be taken to ensure that the driver safely reaches the rest area to rest.

[0267] In the specific implementation process, the EEG data of 21 subjects were used for verification, and Python was used for writing and testing. In order to prevent overfitting, a loss layer was added, and the loss rate was set to 0.5. In addition, in order to maximize the use of data and reduce the data deviation caused by random data division, the data set was divided into 4:1 for training and verification. The evaluation indicators of the recognition algorithm were obtained, including accuracy, precision, recall rate and F1 score.

[0268] In the task of identifying driver mental fatigue using this method, the Delta and Alpha frequency bands performed excellently individually, with accuracy rates of 94.10% and 93.94% respectively, and the F1 score was close to 93%. The Gamma and Beta frequency bands performed relatively stably, with accuracy rates of 92.88% and 91.46% respectively, and the F1 score remained at around 91%, showing good recognition ability. Although the Theta frequency band performed well, the standard deviation was large (13.33%) and the stability was relatively low. After fusing all the frequency band information, the model's performance reached the best, with the accuracy rate increased to 95.71%, the F1 score reached 94.00%, and the standard deviation was significantly reduced, showing stronger robustness and generalization ability. The results prove that the method proposed in this application can effectively identify the driver's fatigue state, as shown in Table 1.

[0269] Table 1 Effect verification data table

[0270]

[0271] A driver mental fatigue identification and intervention system based on EEG signals, including a physiological information acquisition device, an algorithm calculation unit and an on-vehicle intervention unit;

[0272] Physiological information collection equipment: used to collect the driver's physiological information, including the driver's EEG signals and eye movement data;

[0273] Algorithm computing unit: As the management computing center of the brain-computer interface, it predicts the driver's mental fatigue state based on the received EEG signals and the designed detection and recognition algorithm. Based on the driver's future mental fatigue state, it selects the appropriate intervention plan to intervene in the driver's fatigue and ensure road safety.

[0274] On-vehicle intervention unit: Based on the selected intervention plan, it provides reasonable solutions for driver fatigue intervention, eliminates driver fatigue and ensures road safety.

[0275] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A driver mental fatigue identification and intervention method based on EEG signals, characterized in that: include: S1: pre-processing the driver's physiological signals; S2: Construct brain functional network based on preprocessed physiological signals; S3: Extracting EEG topological features from brain functional networks; S4: input the EEG topological features into the recognition model, and the recognition model recognizes the driver's mental fatigue state according to the spatiotemporal features of the EEG topology; S5: Determine fatigue intervention measures based on the driver’s mental fatigue status.

2. The method for identifying and intervening in driver mental fatigue based on EEG signals according to claim 1 is characterized in that: The S2 includes: S21: Processing EEG signals using nonlinear normative multivariate decomposition; S22: Calculate the decomposed components of the EEG signal according to the different frequency ranges; S23: separating the frequency components within the frequency range corresponding to the frequency band to be extracted; S24: Convert different frequency bands into corresponding frequency components, calculate the functional connections between different brain regions under different frequency components, and obtain the correlation matrix between different component signals; S25: Constructing brain functional network based on correlation matrix.

3. The method for identifying and intervening in driver mental fatigue based on EEG signals according to claim 1 is characterized in that: The S2 also includes calculating network metrics to analyze brain connection characteristics after constructing the brain functional network. Specific calculation indicators include node strength, clustering coefficient, eigenvector center, local efficiency, global efficiency, small-world property and shortest path.

4. The method for identifying and intervening in driver mental fatigue based on EEG signals according to claim 1 is characterized in that: The S4 includes: S41: Input the EEG feature data set into the Chebyshev polynomial graph convolution to learn the dependencies between feature sets. The calculation formula is: Among them, T k (A) is the kth order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the input matrix reflecting the EEG characteristics, describing the characteristic information of each EEG channel or electrode at a certain time point t, and K is the order of the Chebyshev polynomial; S42: Recursively define higher-order Chebyshev polynomials: T0(A)=I T1(A)=A T k (A)=2A·T k-1 (A)-T k-2 (A)for k≥2 Where I is the identity matrix, A is the adjacency matrix, T0(A) and T1(A) are the identity matrix and adjacency matrix respectively, T k (A) The information of the k-order neighborhood is continuously captured through the recursive relationship; through this recursive relationship, the high-order Chebyshev polynomial is calculated; S43: Convolve the Chebyshev polynomials and aggregate the outputs to obtain the sum of information propagation between all electrode channels; finally, add a bias term b to the convolution result to adjust the offset in the feature space. The specific calculation formula is as follows: Among them, T k (A) is the kth order Chebyshev polynomial of the adjacency matrix A, θ k is the learning weight corresponding to the k-th order polynomial, N(t) is the feature matrix at time step t, and b is the learned bias term; S44: To learn θ k and b, compare Output(t) with the actual t e For comparison, the error Z is minimized through the loss function. The specific calculation formula is as follows: Where I is the total number of data sets or samples, n i is the number of elements in the ith data subset, t e (i,j) is the actual or observed value of the jth element in the i-th data set or sample, t a (i,j) is the predicted value or expected value of the jth element in the i-th data set or sample; S45: Aggregation results use time encoding to generate time features and convert time information into a vector with a fixed dimension. The specific calculation formula is as follows: TimeFeature=W·t Among them, W is the learned weight, t is the matrix of time steps, and TimeFeature is the time feature generated after time encoding; S46: Combine the time feature with the convolution kernel and dynamically generate the weight of the convolution kernel by inputting the time feature into a fully connected layer. The specific calculation formula is as follows: Kernel(t)=MetaLayer(TimeFeature) Among them, MetaLayer is a fully connected layer responsible for generating convolution kernels from temporal features, and Kernel(t) is the generated convolution kernel; S47: The dynamically generated convolution is used to perform convolution calculation on the EEG feature data in a sliding window manner. For each time step i, the sliding window selects data of K consecutive time steps, and the selected data is convolved with the generated convolution kernel. The specific calculation formula is as follows: Among them, Y i is the convolution output, W k is the convolution kernel weight, N i+k is the data of the i+kth time step in the EEG feature data; S48: The output value is predicted using the semi-regression prediction model. For each time step t, the calculation formula of the model is as follows: Among them, y t is the fatigue state prediction value at time step t, X 1:t is all the historical data from time step 1 to t, are all fatigue state predictions from time step 1 to t-1, and f(·) is the prediction function.

5. The method for identifying and intervening in driver mental fatigue based on EEG signals according to claim 1 is characterized in that: The physiological signal in S1 includes an electroencephalogram signal, and the preprocessing of the electroencephalogram signal includes: Perform bandpass filtering and power frequency notch processing on the i-electrode channel signal in the collected EEG signal to retain the EEG signal within the required frequency band; Perform principal component analysis on the filtered EEG signal data to retain the EEG signal data whose cumulative variance contribution reaches the standard, and decompose the obtained EEG signal data into opposing components; EEG data created using synthetic EOG channels are used to detect eye blinks, horizontal and vertical movements; components with high temporal or spatial correlation are marked as artifacts; the specific calculation formula is as follows: Where ρ is the calculated signal S i The correlation coefficient between the EOG signal and the i is the i-th component of the EEG signal, EOG is the electrooculogram signal, YesS i The standard deviation of the EEG signal component S i The degree of dispersion of the data distribution, σ EOG It is the standard deviation of the EOG signal, which indicates the degree of discreteness of the data distribution of the electrooculogram signal EOG; Remove artifact components with high temporal or spatial correlation to obtain corresponding EEG data; Split the continuous EEG signal into smaller time periods for analysis and calculation, using time windows for calculation; Baseline correction was used to remove pre-stimulus fluctuations in the EEG data and to normalize the EEG signals within the time window.

6. The method for identifying and intervening in driver mental fatigue based on EEG signals according to claim 1 is characterized in that: The physiological signal in S1 also includes eye movement data. The preprocessing of the eye movement data includes: using cubic spline interpolation fitting spline function to interpolate and complete the eye movement data: S i (x)=a i (x-x i ) 3 +b i (x-x i ) 2 +c i (x-x i )+d i Among them, S i (x) is the vertical coordinate of the gaze, x is the horizontal coordinate of the interpolation point, and x i is the horizontal coordinate of the known gaze, a i 、b i 、c i and d i It is determined by boundary conditions and continuity requirements.

7. The method for identifying and intervening in driver mental fatigue based on EEG signals according to claim 2 is characterized in that Extraction of signal spectrum, calculation formula of S22 is: Among them, X(f) represents the signal spectrum, x(t) is the component of the EEG signal after decomposition, f is the acquisition frequency of the EEG signal, and j is the imaginary unit.

8. The method for identifying and intervening in driver mental fatigue based on EEG signals according to claim 2 is characterized in that: The S24 includes: representing the power distribution of the signal at different frequencies by the power spectrum density P(f): P(f)=|X(f)| 2 The different frequency bands in the EEG signal are converted into corresponding frequency components, the functional connections between different brain regions under different frequency components are calculated, and the connection matrix is ​​constructed based on the correlation between each electrode signal to obtain: Among them, x i represents the component of the i-th EEG signal, C ij The functional connectivity strength between the i-th and j-th components is calculated using the phase-locking value, COV(x i ,x j ) is the covariance between the EEG signal data of the i-th and j-th components, Var(x i ) and Var(x j ) is the variance between the EEG signal data of the i-th and j-th components; the correlation matrix C between the different component signals is obtained by calculating the above formula, where each element C ij Represents the strength of functional connectivity between components i and j in a given frequency band.

9. The method for identifying and intervening in driver mental fatigue based on EEG signals according to claim 2 is characterized in that: S25 includes: using the correlation matrix C to construct a brain function network G, and the thresholding formula of the brain function network is as follows: G=(V,E) Where V is the set of electrode channels, E is the set of edges, and the edges are defined as the connectivity between brain regions according to the correlation value; A threshold T is applied to the correlation matrix to determine the importance. Specifically, if the value is below the threshold, it is considered as an invalid connection and needs to be discarded, and if it is above the threshold, it is considered as a valid connection and retained. The adjacency matrix A is defined as: The above binary adjacency matrix A represents the functional connections in the brain functional network.

10. A driver mental fatigue identification and intervention system based on EEG signals, characterized in that: A method for identifying and intervening in driver mental fatigue based on EEG signals as described in any one of claims 1 to 9 is applied, comprising a physiological information acquisition device, an algorithm calculation unit and an on-board intervention unit; Physiological information collection equipment: used to collect the driver's physiological information, including the driver's EEG signals and eye movement data; Algorithm calculation unit: based on the received EEG signal of the driver, the driver is predicted according to the designed detection and recognition algorithm to determine the mental fatigue state of the driver, and based on the driver's future mental fatigue state, an intervention plan is selected to intervene in the driver's fatigue; On-board intervention unit: Provides solution suggestions for driver fatigue intervention based on the selected intervention plan.

Images