A three-flow state space driving fatigue detection system and detection method

Abstract

The application discloses a three-flow state space driving fatigue detection system, comprising an electroencephalogram signal acquisition and feature extraction module, an electrooculogram signal acquisition and feature extraction module, a three-flow state space modeling module, a neurophysiological interaction module and a multi-modal fusion and fatigue regression module; the detection method is that electroencephalogram and electrooculogram signals of a driver are collected and pretreated first, features are extracted and divided into at least two electroencephalogram feature flows of different brain areas and an electrooculogram feature flow, then each feature flow is input into a corresponding state space model for parallel modeling, subsequently, the neurophysiological interaction module is used to simulate the functional connection between brain areas and the electrooculogram features are used to correct the electroencephalogram features, finally, the corrected features are weighted and fused, and a continuous driving fatigue evaluation result is output. The application introduces state space modeling and neurophysiological interaction, reduces the calculation complexity, improves the accuracy and stability of fatigue detection, and enhances the physiological interpretability of the model.

Description

Technical Field

[0001] This invention relates to the field of driving safety monitoring and EEG signal processing technology, specifically to a three-flow state space driving fatigue detection system and method based on EEG and electrooculogram signals. Background Technology

[0002] Driver fatigue is a significant contributing factor to road traffic accidents. During prolonged driving or nighttime driving, drivers' alertness tends to decline to varying degrees, significantly increasing the risk of accidents. Existing research has confirmed that when drivers are fatigued, the spectral distribution of their electroencephalogram (EEG) signals and the eye movement characteristics reflected by their electrooculogram (EOG) signals exhibit regular changes. Therefore, multimodal fatigue detection methods based on EEG and EOG signals have gradually become an important technical means for objectively assessing driver fatigue.

[0003] However, current multimodal driver fatigue detection methods still have some shortcomings in practical applications. Firstly, some methods employ deep neural network models based on attention mechanisms. These models require extensive parameter training and computation, resulting in high computational complexity and making real-time detection impossible on hardware platforms with limited computing power and storage resources, such as in-vehicle embedded devices. Secondly, most multimodal fusion methods simply splice together EEG and EEG features without considering human neurophysiological mechanisms. This fails to reflect the functional connections between different brain regions and the physiological regulatory relationship between EEG and EEG, thus hindering the rationality and robustness of multimodal information fusion. Thirdly, most existing detection models lack effective constraints on the continuous evolution of fatigue states, leading to abrupt changes in fatigue prediction results that do not conform to physiological laws, thereby affecting the reliability of the detection results in practical applications.

[0004] Therefore, developing a driver fatigue detection system with high computational efficiency, good physiological interpretability, and adaptability to in-vehicle real-time monitoring scenarios has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a three-stream state-space driving fatigue detection system and method. This invention, by introducing state-space modeling and neurophysiological interaction, improves the accuracy and stability of fatigue detection while reducing computational complexity, and enhances the physiological interpretability of the model.

[0006] The technical solution of the present invention: a three-flow state space driving fatigue detection system, comprising: The EEG signal acquisition and feature extraction module is used to acquire the driver's EEG signals and extract EEG features from at least two different brain regions to form an EEG feature stream; The electrooculogram (EOG) signal acquisition and feature extraction module is used to acquire the driver's EOG signals and extract eye movement behavior features to form an EOG feature stream; The three-stream state space modeling module is connected to the EEG signal acquisition and feature extraction module and the EEG signal acquisition and feature extraction module. It is used to perform parallel state space modeling of the EEG feature stream and the EEG feature stream of at least two brain regions to obtain the latent state variables that characterize the driver's fatigue state. The neurophysiological interaction module is connected to the three-stream state space modeling module to simulate the functional connectivity between the EEG feature streams of different brain regions, and to guide and correct the EEG feature streams using the electrooculogram feature streams to obtain the corrected multi-path state features. The multimodal fusion and fatigue regression module is connected to the neurophysiological interaction module and is used to fuse the corrected multi-path state features and output continuous driving fatigue assessment results.

[0007] In the aforementioned three-flow state space driving fatigue detection system, the EEG signal acquisition and feature extraction module includes an EEG signal acquisition device. The EEG signal acquisition device is used to acquire multi-channel EEG signals from the driver's head and extract EEG state features of multiple frequency bands through filtering, noise reduction and standardization processing. The EEG state features are divided into EEG feature streams of at least two different brain regions according to the location of the brain region.

[0008] In the aforementioned three-flow state-space driving fatigue detection system, the three-flow state-space modeling module includes at least three independent state-space models, which respectively model the feature flows of different brain regions and the electrooculogram feature flows. Each state-space model models the input features based on the following state update relationship: ; in, Indicates time The hidden state variables are used to characterize the driver's fatigue state; This represents the feature vector input at the corresponding time point; This is the state update function.

[0009] In the aforementioned three-stream state-space driving fatigue detection system, the three-stream state-space modeling module further includes parallel state-space models for EEG feature flows of different brain regions, and the state update relationship of the state-space model is expressed as follows: ; in, This represents the characteristic flow of brain electrical signals corresponding to different brain regions. This represents the characteristic flow of electrooculography (EOG).

[0010] In the aforementioned three-flow state space driving fatigue detection system, the neurophysiological interaction module models the functional connectivity between EEG state characteristics of different brain regions as follows: ; in, The regulation coefficient represents the correlation between the electroencephalographic state characteristics of different brain regions.

[0011] In the aforementioned three-flow state space driving fatigue detection system, the neurophysiological interaction module guides and corrects the electroencephalogram (EEG) state characteristics based on electrooculogram (EOG) state characteristics, as shown below: ; in, Indicates the characteristics of brain electrical states, Indicates the characteristics of electrooculography (EOG) state. This represents the guiding function.

[0012] In the aforementioned three-stream state-space driving fatigue detection system, the multimodal fusion and fatigue regression module performs weighted fusion of multi-path state features after neurophysiological interaction modeling, and outputs fatigue assessment results that satisfy the following time continuity constraints: ; in, Indicates time The fatigue assessment results This is a preset threshold.

[0013] The aforementioned detection method for the three-flow state space driving fatigue detection system includes the following steps: Step 1: Collect the driver's electroencephalogram (EEG) and electrooculogram (EOG) signals, and perform filtering, noise reduction, and standardization preprocessing on the signals; Step 2: Extract multi-band EEG state features from the preprocessed EEG signals and divide them into EEG feature streams of at least two different brain regions. At the same time, extract EOG state features from the EOG signals to form EOG feature streams. Step 3: Input the at least two brain region feature streams and the electrooculography feature streams into the corresponding state space models for modeling to obtain multi-path state features; Step 4: Based on the neurophysiological interaction mechanism, the functional connectivity between the EEG state features of different brain regions in the multi-path state features is modeled, and the EEG state features are guided and corrected using the electrooculogram state features to obtain the corrected multi-path state features. Step 5: Fuse the corrected multi-path state features to output continuous driving fatigue assessment results.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention achieves effective characterization of the continuous evolution of fatigue states by performing parallel state-space modeling of EEG and EEG state features from different brain regions in a three-stream state-space modeling module. Furthermore, the multimodal fusion and fatigue regression modules are configured with temporal continuity constraints, making the output fatigue assessment results smoother and more consistent with human physiological patterns, significantly improving the reliability of the detection results. This invention employs a three-stream structure for state-space modeling and simulates the functional connectivity between EEG state features from different brain regions in the multi-path state features through a neurophysiological interaction module, replacing the traditional simple feature splicing method and significantly improving the rationality and robustness of multimodal information fusion. This invention abandons the computationally complex deep neural network model, achieving fatigue detection through state-space modeling and simple weighted fusion. The overall computational complexity of the model is low, enabling real-time operation in in-vehicle embedded devices, adapting to the hardware resources and real-time monitoring requirements of the in-vehicle environment. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall system structure of the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention; Figure 3 A schematic diagram illustrating the three-flow state space modeling and neurophysiological interaction; Figure 4 This is a schematic diagram of fatigue detection output. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0017] Example: A three-flow state space driving fatigue detection system, such as Figure 1 As shown, this system, applied to driver fatigue monitoring scenarios, includes: an EEG signal acquisition and feature extraction module, an EEG signal acquisition and feature extraction module, a three-flow state space modeling module, a neurophysiological interaction module, and a multimodal fusion and fatigue regression module.

[0018] The EEG signal acquisition and feature extraction module is used to acquire multi-channel EEG signals from the driver and extract EEG state features related to fatigue. The module includes an EEG signal acquisition device, which acquires multi-channel EEG signals from the driver's head and extracts features through filtering, noise reduction, and standardization. , , and The system collects EEG state characteristics across multiple frequency bands, categorizing these characteristics into at least two distinct brain regions based on their location within the prefrontal, parietal, and occipital lobes. The EEG signal acquisition device utilizes a portable EEG acquisition cap conforming to the international 10-20 standard electrode distribution system. To balance ease of use in in-vehicle scenarios with effectiveness in fatigue monitoring, this invention selects 17 key EEG channels highly correlated with alertness and cognitive function, and algorithmically divides them into two major anatomical functional brain regions: 1. Temporal Region: Contains 6 electrodes (FT7, FT8, T7, T8, TP7, TP8), mainly responsible for auditory processing, memory encoding, and perception of subtle changes in the early stages of fatigue.

[0019] 2. Parieto-Occipital Region: Contains 11 electrodes, including CP1, CP2, P1, Pz, P2, PO3, POz, PO4, O1, Oz, and O2, which are mainly responsible for visual information processing, spatial attention, and higher cognitive regulation.

[0020] In this embodiment, the total dimension of a single frame of global EEG features is 17 (number of electrodes) × 5 (number of frequency bands) × 5 (feature types) = 425 dimensions. For the input three-stream state-space model (NeuroSSM), physical stripping of the spatial dimensions was implemented early in the network's development. 1. Temporal Stream: Extract features corresponding to the 6 temporal lobe electrodes to form a 6×5×5=150-dimensional temporal feature vector (corresponding to network structure parameters). ).

[0021] 2. Parietal Stream: Extract features corresponding to the 11 apical and occipital leaf electrodes to form a 11×5×5=275-dimensional temporal feature vector (corresponding to network structure parameters). ).

[0022] The electrooculogram (EOG) signal acquisition and feature extraction module is used to acquire the driver's EOG signals and extract EOG state features. This module includes an EOG signal acquisition device for acquiring the driver's EOG signals and extracting EOG state features reflecting changes in eye movement behavior (blink frequency, saccade amplitude, gaze duration, and duration of eye closure, etc.) to form an EOG feature stream. The EOG signal acquisition device uses patch-type Ag / AgCl surface electrodes, deploying vertical EOG (VEO) acquisition electrodes at the upper and lower eye sockets and horizontal EOG (HEO) acquisition electrodes at the left and right outer canthi, with the sampling rate simultaneously set to 200Hz. EOG signals contain a large amount of involuntary movement information that directly reflects visual fatigue. This invention further extracts various spatiotemporal statistical features from the EOG component matrix separated based on independent component analysis (ICA). These specifically include three main categories of indicators: 1. Blinking metrics: These include blinking frequency per unit time, average duration of eye closure, maximum duration of eye closure, and PER-CLOS (percentage of total time spent with eyes closed, used as the gold standard for objective fatigue).

[0023] 2. Saccade index: includes the frequency of horizontal and vertical saccades, average saccade amplitude, maximum saccade velocity, and average saccade velocity.

[0024] 3. Fixation metrics: These include the variance and mean of fixation frequency and continuous fixation duration.

[0025] In this embodiment, the three categories of indicators mentioned above are numerically calculated and cascaded within a predefined time window (such as an 8-second or 16-second sliding window), ultimately quantified into a 36-dimensional comprehensive eye-tracking feature vector. This 36-dimensional eye-tracking sequence serves not only as an independent alertness regression feature branch in the model but is also input into the "artifact decoupling gate (AD-Gate)" as a priori noise reference to eliminate eye-tracking artifact leakage in the EEG signal.

[0026] Specifically, before being input into the core network, the raw electroencephalogram (EEG) and electrooculogram (EOG) signals acquired in this invention undergo multi-stage signal preprocessing and feature sequence construction to eliminate environmental and physiological noise interference and extract temporal features characterizing the evolution trend of driving fatigue. First, the acquired multi-channel raw signals are bandpass filtered using a Butterworth bandpass filter with a cutoff frequency of 0.5Hz to 75Hz to remove DC baseline drift and high-frequency environmental noise. Simultaneously, a 50Hz notch filter is used to eliminate power grid frequency interference. The filtered signals are then uniformly downsampled to 200Hz to reduce subsequent computational overhead. Considering the contamination of EEG signals by eye movements and electromyography in real driving scenarios, Independent Component Analysis (ICA) is used to perform blind source separation on the filtered signals. The multi-channel signal was decomposed into statistically independent source components. The cross-correlation coefficient between each independent component and the reference electrooculography (EOG) channel was calculated. EOG components containing high-amplitude artifacts such as blinking and saccades were automatically identified and removed. The remaining components were then reconstructed into a pure EEG signal. Short-time Fourier transform (STFT) was then used to perform time-frequency analysis on the pure EEG signal, with the non-overlapping time window size set to 1 second (or 2 seconds). Five classic frequency bands highly correlated with fatigue states were extracted. (1-4Hz) (4-8Hz) (8-14Hz) (14-31Hz) and (31-50Hz). To comprehensively capture neural dynamics, five linear and nonlinear acoustic indices were extracted for each frequency band: power spectral density (PSD), differential entropy (DE), differential asymmetry (DASM), rational asymmetry (RASM), and differential tail asymmetry (DCAU). For a specific frequency band EEG signal that approximately follows a Gaussian distribution, the formula for calculating its differential entropy is: ; Differential entropy (DE) characteristics are calculated for each frequency band to measure the complexity of signals in a specific frequency band. Among these, This represents the time series variance of the EEG signal in this specific frequency band.

[0027] The transition of human fatigue state is a slow physiological evolution process, and the extracted DE features are easily affected by local sensor noise, resulting in abrupt changes. Therefore, this invention employs a linear dynamic system (LDS) for smoothing. The LDS algorithm establishes a hidden Markov evolution model and uses Kalman smoothing to infer the expected value of the hidden state, thereby filtering out outlier high-frequency fluctuations that do not conform to the slow evolution of fatigue. To address the significant individual differences (DomainShift) in physiological signals among different subjects, the Z-Score normalization algorithm is used to scale the smoothed EEG and EOG features. Let the feature flow matrix be... Calculate the mean of the training set data on each feature dimension. and standard deviation Then, the following formula is used for conversion: ; It should be emphasized that the test set directly reuses the data calculated from the training set. and Scaling is performed to strictly avoid data leakage issues.

[0028] To adapt to the three-stream state-space model (NeuroSSM) with its ability to capture long-term temporal dependencies, a sliding window segmentation is applied to the normalized feature stream. The sequence length is set to 16, and the stride is set to 1. That is, at any given time, intervals are extracted... The historical feature fragments within are combined into a sample input matrix. Through this operation, the original two-dimensional feature matrix is ​​converted into a three-dimensional tensor with the shape of... This enables the model to perceive the physiological state evolution trajectory over the past 16 time steps in a single forward propagation.

[0029] The three-stream state space modeling module is connected to the EEG signal acquisition and feature extraction module and the EEG signal acquisition and feature extraction module. It is used to perform parallel state space modeling of EEG and EEG state features from different brain regions, outputting multi-path state features. The three-stream state space modeling module includes at least three independent state space models, each modeling the feature streams from different brain regions and the EEG feature streams. Each state space model models the input features based on the following state update relationship: ; in, Indicates time The hidden state variables are used to characterize the driver's fatigue state; This represents the feature vector input at the corresponding time point; This is the state update function.

[0030] Preferably, the three-stream state-space modeling module further includes parallel state-space models for EEG feature streams of different brain regions, and the state update relationship of the state-space model is expressed as: ; in, This represents the characteristic flow of brain electrical signals corresponding to different brain regions. These three elements constitute a three-flow characteristic flow system, representing the electrooculography characteristic flow.

[0031] In this embodiment, addressing the processing needs of long-term multimodal physiological signals, the present invention innovatively abandons the quadratic computational complexity of traditional self-attention mechanisms (Transformers) and the gradient vanishing problem of recurrent neural networks (RNNs), constructing a three-stream parallel coding architecture based on the Selective State Space Model (Selective SSM or Mamba). This architecture uses a bidirectional selective state space model as its core foundation, with each unit containing independent forward and backward SSMUnit branches. By performing bidirectional scanning and feature fusion on the time series, the model can both perceive historical fatigue accumulation states and incorporate future evolutionary trends for contextual constraints, while maintaining a strictly linear O(L) computational complexity. In terms of macroscopic structure, the network is instantiated as a three-stream parallel coding stream: Temporal Stream, Parietal Stream, and EOG Stream, each stream stacked with multiple layers of bidirectional SSM encoders. Microscopic parameters are optimized as follows: hidden state dimension ( The model features 16 times the length of the memory for long-range information. The unified interface is set to 64, and the expansion factor is 2, making the internal feedforward network 128-dimensional. Enhance nonlinear representation, one-dimensional convolution kernel size ( The time step rank is 4 to capture local mutations. )Pick Adaptive step size control is implemented. The AdamW optimizer (initial learning rate 1×10⁻⁶) is used during the training phase. -3 Weight decay 1×10 -2 ), in conjunction with cosine annealing learning rate scheduling (period) =60, minimum learning rate 1×10 -5The batch size is set to 64, and the maximum training epochs are set to 60. Overfitting prevention measures include: applying a Dropout rate of 0.4 to both fully connected layers and attention layers; implementing an early stopping mechanism (monitoring the validation set RMSE; terminating if there is no improvement after 15 consecutive epochs); and using L2 norm gradient pruning (threshold). =1.0) to prevent gradient explosion. The model was validated on the SEED-VIG public simulated driving fatigue dataset, using continuous 16-second physiological feature segments (shape) The predicted PERCLOS value at the end of the 16th second (a continuous floating-point label of 0-1, with a value greater than 0.7 indicating extreme fatigue) was used as the regression target. To test the generalization ability across subjects, a rigorous leave-one-out cross-validation (LOSO) method was used: the model was trained with data from N-1 subjects each time, and tested with the remaining subject's complete data. This process was repeated until all subjects were tested, and the global average performance was finally reported, effectively demonstrating the model's robustness to unknown drivers.

[0032] The neurophysiological interaction module is connected to the three-stream state space modeling module to simulate the functional connectivity between EEG state features of different brain regions in multi-channel state features. It also uses electrooculography (EOG) features to guide and correct the EEG features, obtaining corrected multi-channel state features. The neurophysiological interaction module models the functional connectivity between EEG state features of different brain regions as follows: ; in, The regulation coefficient representing the correlation of EEG state characteristics in different brain regions is a constant between 0 and 1, determined through training using functional connectivity analysis of EEG signals (such as Pearson correlation and mutual information).

[0033] The neurophysiological interaction module guides and corrects EEG state characteristics based on electrooculography (EOG) state characteristics, as shown below: ; in, Indicates the characteristics of brain electrical states, Indicates the characteristics of electrooculography (EOG) state. This refers to the guiding function, the guiding function It is a linear or nonlinear mapping function, obtained through supervised learning training based on the physiological correlation between driver's EEG and EEG signals.

[0034] This invention, in the feature encoding and fusion process of the three-stream state-space model (NeuroSSM), breaks through the traditional simple late-stage fusion method of "feature cascading" and innovatively designs a deep neurophysiological interaction module. Based on prior physiological knowledge of changes in brain functional connectivity under driving fatigue, it achieves high-order nonlinear information interaction between EEG and EEG, and between different brain regions, through four core sub-components: spectral feature recalibration unit (SRU), artifact decoupling gate (AD-Gate), functional connectivity bridge (FC-Bridge), and eye-guided attention (OGA). Different frequency bands (such as...) Waves and The contribution of the wave (high alertness / fatigue sensitivity) to fatigue evolution dynamically changes with individual state. Traditional feature input methods indiscriminately process all frequency bands, causing key neural biomarkers to be overwhelmed by noise. This invention designs an SRU module that adaptively weights the input differential entropy features through a channel attention mechanism, thereby achieving dynamic recalibration of the "high alertness / fatigue sensitivity" frequency band during the feature embedding stage. In specific implementation, the input feature matrix is ​​first processed... One-dimensional adaptive average pooling is performed in the time dimension. Compression is performed to obtain the global spectral statistics for each channel. The specific formula is as follows: ; in, Representing the Global spectral statistics for each channel. Representing the Spectral characteristics of each channel This refers to the number of feature channels (e.g., 150-dimensional features corresponding to temporal lobe flow).

[0035] Subsequently, a nonlinear bottleneck transformation is performed through a "bottleneck structure" (reduction rate of 4) composed of two layers of linear mapping, i.e. This process captures complex nonlinear dependencies between channels; then, the recalibration coefficients are restricted to the range [0,1] using the Sigmoid function to generate channel weight vectors. Finally, the weight vector is applied to the original input features to achieve feature recalibration. Through SRU recalibration, the model can automatically select more discriminative combinations of EEG frequency bands for each time segment, providing input with a higher signal-to-noise ratio for subsequent SSM state modeling. Artifact decoupling gating (AD-Gate) addresses the strong high-frequency artifact interference generated in the frontal and temporal EEG signals by subjects' blinking and saccade movements in the vehicle-mounted acquisition environment. To extract pure neural representations, this invention introduces AD-Gate immediately after early feature mapping, using EEG features as a priori noise reference to denoise the temporal lobe EEG features. Specifically, the EEG features are first... Mapped to potential artifact leakage through a linear projection layer. Subsequently, the temporal lobe EEG characteristics were analyzed. and The weights are concatenated and dynamically decoupled gating weights are generated using a multilayer perceptron (MLP) and sigmoid activation. Finally, noise reduction is achieved through residual subtraction, specifically by utilizing the generated gating weights. Adjusting the amount of artifact leakage The degree of influence, and through scaling factors (The value was empirically set to 0.1 in the experiment) Interference was removed from the original EEG features in reverse order, and the pure temporal lobe EEG features were obtained after removing the interference. The specific formula is as follows: .

[0036] The Functional Connectivity Bridge (FC-Bridge) simulates the dynamics of brain region-level functional connectivity during fatigue evolution, where inhibitory or compensatory signals are transmitted from the parieto-occipital lobe to the temporal lobe, achieving cross-brain region feature modulation. Specifically, it first modulates the parieto-occipital lobe features encoded by the SSM (Simultaneous Transmission Scale). Temporal lobe features Based on the characteristics of the parietal and occipital lobes As a control variable, a single-layer linear mapping and Sigmoid activation are performed to generate a cross-brain region attention gating matrix. The specific formula is as follows: ; Then the gating matrix and temporal lobe features were combined. Element-wise multiplication is performed, and the original temporal lobe information is preserved through residual connections to achieve adaptive gain or suppression of the temporal lobe by the parietal region. The specific formula is as follows: .

[0037] Eye-guided attention (OGA) uses a cross-modal cross-attention mechanism to force EEG feature encoding to focus on temporal keyframes highly correlated with eye movement behavior. Specifically, it uses EEG features (such as parietal-occipital lobe features) ) as query vector ( Using electrooculogram (EOG) features as key vectors ( ) and value vector ( The spatiotemporal dependence of EEG sequences on EEG sequences was calculated using a multi-head attention mechanism (4 heads, inactivation rate of 0.25). The specific formula is as follows: ; Cross-modal attention output was then compared with raw EEG characteristics. The residuals are accumulated and normalized using LayerNorm, then fed into a two-layer feedforward neural network (FFN) with GELU activation function, where the hidden layer dimension is increased to 2×. The final output is the enhanced EEG characteristics after eye-tracking guidance. This enhances the robustness of multimodal joint characterization.

[0038] The multimodal fusion and fatigue regression module is connected to the neurophysiological interaction module to fuse the corrected multi-path state features and output continuous driving fatigue assessment results. The multimodal fusion and fatigue regression module performs weighted fusion of the multi-path state features after neurophysiological interaction modeling and outputs fatigue assessment results that satisfy the following time continuity constraints: ; in, Indicates time The fatigue assessment results The preset threshold is used. These are empirical or training values, set according to the actual scenario of vehicle-mounted driver fatigue monitoring, with a range of 0.01-0.1 (fatigue assessment results are normalized to the 0-1 range).

[0039] In this embodiment, after the aforementioned neurophysiological interaction module and SSM deep temporal encoding, this invention abandons the traditional, rudimentary approach of directly splicing the output of fully connected layers, and designs a complete multimodal fusion and regression quantization system based on "coherence extraction - dynamic gating weighting - trend - numerical dual constraints". The cross-modal coherence extractor calculates the parietal-occipital lobe EEG features through bilinear pooling mapping. With electrooculogram characteristics The second-order interaction features are mapped to a reduced-dimensional space. The specific formula is as follows: ; Then, one-dimensional adaptive average pooling (AdaptiveAvgPool1d) is used to perform global temporal aggregation on the sequence length (e.g., a 16-second sliding window). After removing the time dimension, it is further compressed through a single-layer linear network. Global coherence eigenvectors of dimension First, temporal lobe features enhanced by eye-guided attention (OGA) were analyzed. Characteristics of the occipital lobe and primitive electrooculography characteristics Global mean pooling is performed along the time dimension to obtain a single-frame feature vector of length 64. , and Combine it with the aforementioned 16 dimensions. Channel concatenation is performed to construct a multimodal global joint feature matrix with a total dimension of 208.

[0040] Input the 208-dimensional cascaded features into the... A gated generative network is constructed, and four normalized dynamic weight coefficients are output using the Softmax activation function. , , , (satisfy ).

[0041] Based on the driver's current physiological performance, an adaptive scalar multiplication weighting is applied to the four feature branches. When eye movement signals are lost due to strong light, the weighting is automatically reduced. Increase brainwave weight and The weighted features are re-concatenated to form the final 208-dimensional representation vector used for regression.

[0042] The final features, after applying Dropout (experimentally set to 0.4 to prevent performance degradation due to overfitting of multimodal features), are passed through the last linear regressor layer. It outputs a one-dimensional continuous scalar prediction value, namely the predicted instantaneous PERCLOS fatigue index.

[0043] To address the dual characteristics of fatigue evolution—namely, "global slow drift (trend)" and "local small oscillations (numerical values)"—this invention designs a joint optimization objective function. Specifically, a smooth L1 loss is used to optimize the predicted values. Compared with the true value The absolute error is constrained, making it more resistant to outliers in data annotation compared to standard MSE loss. A Pearson correlation coefficient loss is constructed, calculating the covariance and standard deviation of the predicted and true sequences, using the following formula: ; in, , , To prevent overflow during division by zero.

[0044] The final mixture loss is determined by the scaling hyperparameter. To control the balance, this embodiment uses cross-validation to... The optimal value is set to 0.6. ; The numerical error constraint uses smoothing L1 loss (resisting outliers), and the trend correlation constraint uses Pearson correlation coefficient loss.

[0045] This invention employs a Selective State-Space Modeling (SSM) mechanism, using a hardware-aware parallel scanning algorithm to reduce computational complexity to linear O(L), eliminating the computational bottleneck of long-term processing. Since human fatigue is a gradual physiological process, to eliminate non-physiological prediction jumps caused by instantaneous sensor noise, this invention designs a moving average filter at the vehicle-mounted inference output. Specifically, a sliding window size of 5 is set to smooth the original predicted values ​​output by the model in real time, significantly improving the stability and practicality of the driver warning system.

[0046] Detection methods based on a three-flow state-space driving fatigue detection system, such as Figure 2 As shown, it includes the following steps: Step 1: Collect the driver's electroencephalogram (EEG) and electrooculogram (EOG) signals, and perform filtering, noise reduction, and standardization preprocessing on the signals; In this step, the system first acquires multi-channel EEG signals from the driver using an EEG signal acquisition device, and then acquires the driver's electrooculogram (EOG) signals using an EOG signal acquisition device. The acquired EEG and EOG signals are then filtered, artifact removed, and normalized by a preprocessing module to eliminate the influence of environmental noise and individual differences. Filtering uses a 0.5-30Hz bandpass filter to eliminate power frequency interference; noise reduction uses independent component analysis (ICA) to remove EOG and EMG artifacts; and normalization uses Z-score normalization to eliminate individual physiological signal differences.

[0047] Step 2: Extract multi-band EEG state features from the preprocessed EEG signals and divide them into EEG feature streams of at least two different brain regions. At the same time, extract EOG state features from the EOG signals to form EOG feature streams. Step 3: Input at least two brain region feature streams and electrooculography (EOG) feature streams into the corresponding state space models for modeling, to obtain multi-path state features, such as... Figure 3 As shown; Step 4: Based on the neurophysiological interaction mechanism, the functional connectivity between the EEG state features of different brain regions in the multi-path state features is modeled, and the EEG state features are guided and corrected using the electrooculogram state features to obtain the corrected multi-path state features. Step 5: Fuse the corrected multi-path state features to output continuous driving fatigue assessment results, such as... Figure 4 As shown.

[0048] In this step, the corrected multi-path state features are weighted and fused. The weight values ​​are determined based on the importance of each feature flow in representing the fatigue state and are obtained through training with a random forest or gradient boosting tree model. The weights of the EEG feature flow are allocated according to the correlation between brain regions and fatigue, and the EEG feature flow is used as a guiding weight for the correction feature allocation adaptation.

[0049] Experimental verification shows that, under NVIDIA RTX 4000 hardware conditions, the model of this invention has only 253K parameters and a storage size of only 0.97MB. Compared to the current state-of-the-art (SOTA) Transformer baseline model (such as E2CF, with a size of 1.13MB), the size of this invention is reduced by 14.1%. More importantly, the number of floating-point operations (FLOPs) of this invention is only 112.1M, which is only 5% of that of the Transformer model (2.062GFLOPs). When processing single-frame samples (BatchSize=1), the average inference time of this invention is only 3.63 milliseconds (i.e., it can process 275.1 samples per second), while the Transformer baseline model requires 29.74 milliseconds. This invention achieves an 8.19x inference speedup, fully meeting the real-time early warning requirements of high-frequency sampling in automotive equipment. Furthermore, this invention employs the SEED-VIG dataset (containing 2 hours of simulated driving data from 23 subjects) and performs extremely rigorous leave-one-subject-out (LOSO) cross-validation to ensure that the test set data is completely invisible during the training phase, thus accurately reflecting the model's generalization ability to unknown drivers. Experimental results show that the average Pearson correlation coefficient (COR) of this invention on the 23 subjects reaches 0.8738±0.10, and the root mean square error (RMSE) is as low as 0.1294±0.06. Compared with traditional methods such as Support Vector Regression (SVR), Deep Belief Network (DBM), and Dynamic Graph Convolutional Network (DGCNN), the COR is improved by 3%-27.2%, and the RMSE is reduced by 4.4%-30.9%.

[0050] To verify the necessity of the four unique neurophysiological interaction modules in this invention, rigorous ablation studies were conducted to assess the effectiveness of the neurophysiological modules. Data shows that removing any one module leads to a significant decrease in predictive performance. 1. Removing the functional connecting bridge (FC-Bridge) reduced COR by 5.20%; 2. Removing eye-guided attention (OGA) resulted in a 5.17% decrease in COR; 3. Removing the Spectral Feature Recalibration Unit (SRU) resulted in a 4.00% decrease in COR. 4. Removing the artifact decoupling gate (AD-Gate) reduced COR by 3.54%.

[0051] The experimental data above fully demonstrate that simply relying on mathematical stacking cannot effectively process physiological signals. The strategy of explicitly transforming prior neuroscience knowledge (such as parietal-temporal regulation and eye-brain coupling mechanism) into network computing modules in this invention not only greatly improves accuracy but also endows the model with reliable biological interpretability.

[0052] This invention achieves effective characterization of the continuous evolution of fatigue states by performing parallel state-space modeling of EEG and EEG state features from different brain regions in a three-stream state-space modeling module. Furthermore, the multimodal fusion and fatigue regression modules are configured with temporal continuity constraints, making the output fatigue assessment results smoother and more consistent with human physiological patterns, significantly improving the reliability of the detection results. This invention employs a three-stream structure for state-space modeling and simulates the functional connectivity between EEG state features from different brain regions in the multi-path state features through a neurophysiological interaction module, replacing the traditional simple feature splicing method and significantly improving the rationality and robustness of multimodal information fusion. This invention abandons the computationally complex deep neural network model, achieving fatigue detection through state-space modeling and simple weighted fusion. The overall computational complexity of the model is low, enabling real-time operation in in-vehicle embedded devices, adapting to the hardware resources and real-time monitoring requirements of the in-vehicle environment.

[0053] In summary, this invention, by introducing state-space modeling and neurophysiological interaction, improves the accuracy and stability of fatigue detection while reducing computational complexity, and enhances the physiological interpretability of the model.

Claims

1. A three-flow state space driving fatigue detection system, characterized in that, include: The EEG signal acquisition and feature extraction module is used to acquire the driver's EEG signals and extract EEG features from at least two different brain regions to form an EEG feature stream; The electrooculogram (EOG) signal acquisition and feature extraction module is used to acquire the driver's EOG signals and extract eye movement behavior features to form an EOG feature stream; The three-stream state space modeling module is connected to the EEG signal acquisition and feature extraction module and the EEG signal acquisition and feature extraction module. It is used to perform parallel state space modeling of the EEG feature stream and the EEG feature stream of at least two brain regions to obtain the latent state variables that characterize the driver's fatigue state. The neurophysiological interaction module is connected to the three-stream state space modeling module to simulate the functional connectivity between the EEG feature streams of different brain regions, and to guide and correct the EEG feature streams using the electrooculogram feature streams to obtain the corrected multi-path state features. The multimodal fusion and fatigue regression module is connected to the neurophysiological interaction module and is used to fuse the corrected multi-path state features and output continuous driving fatigue assessment results.

2. The three-flow state space driving fatigue detection system according to claim 1, characterized in that: The EEG signal acquisition and feature extraction module includes an EEG signal acquisition device, which is used to acquire multi-channel EEG signals from the driver's head and extract EEG state features of multiple frequency bands through filtering, noise reduction and standardization processing. The EEG state features are divided into EEG feature streams of at least two different brain regions according to the location of the brain regions.

3. The three-flow state space driving fatigue detection system according to claim 1, characterized in that: The three-flow state space modeling module includes at least three independent state space models, which respectively model the feature flows of different brain regions and the electrooculogram feature flows. Each state space model models the input features based on the following state update relationship: ； in, Indicates time The hidden state variables are used to characterize the driver's fatigue state; This represents the feature vector input at the corresponding time point; This is the state update function.

4. The three-flow state space driving fatigue detection system according to claim 3, characterized in that: The three-stream state-space modeling module further includes parallel state-space models for EEG feature streams of different brain regions, and the state update relationship of the state-space model is expressed as follows: ； in, This represents the characteristic flow of brain electrical signals corresponding to different brain regions. This represents the characteristic flow of electrooculography (EOG).

5. The three-flow state space driving fatigue detection system according to claim 4, characterized in that: The neurophysiological interaction module models the functional connectivity relationships between EEG state characteristics of different brain regions as follows: ； in, The regulation coefficient represents the correlation between the electroencephalographic state characteristics of different brain regions.

6. The three-flow state space driving fatigue detection system according to claim 1, characterized in that: The neurophysiological interaction module guides and corrects EEG state characteristics based on electrooculography (EOG) state characteristics, as shown below: ； in, Indicates the characteristics of brain electrical states, Indicates the characteristics of electrooculography (EOG) state. This represents the guiding function.

7. The three-flow state space driving fatigue detection system according to claim 1, characterized in that: The multimodal fusion and fatigue regression module performs weighted fusion of multipath state features after neurophysiological interaction modeling and outputs fatigue assessment results that satisfy the following time continuity constraints: ； in, Indicates time The fatigue assessment results This is a preset threshold.

8. The detection method of the three-flow state space driving fatigue detection system according to any one of claims 1-7, characterized in that, Includes the following steps: Step 1: Collect the driver's electroencephalogram (EEG) and electrooculogram (EOG) signals, and perform filtering, noise reduction, and standardization preprocessing on the signals; Step 2: Extract multi-band EEG state features from the preprocessed EEG signals and divide them into EEG feature streams of at least two different brain regions. At the same time, extract EOG state features from the EOG signals to form EOG feature streams. Step 3: Input the at least two brain region feature streams and the electrooculography feature streams into the corresponding state space models for modeling to obtain multi-path state features; Step 4: Based on the neurophysiological interaction mechanism, the functional connectivity between the EEG state features of different brain regions in the multi-path state features is modeled, and the EEG state features are guided and corrected using the electrooculogram state features to obtain the corrected multi-path state features. Step 5: Fuse the corrected multi-path state features to output continuous driving fatigue assessment results.

Images