A Driver Vigilance Estimation Method Based on Hierarchical Multi-Scale Topology Augmentation Network

Abstract

This invention discloses a driver vigilance estimation method based on a hierarchical multi-scale topology enhancement network, comprising: acquiring differential entropy features of different frequency bands of EEG signals and 36-dimensional features of EOG signals; constructing a driver vigilance neural network model; sequentially inputting the differential entropy features of different frequency bands of EEG signals into the topology self-aware attention module, frequency band feature fusion module, hierarchical multi-scale module, and residual network module of the driver vigilance neural network model to output driver fatigue classification results and vigilance regression prediction results. This invention not only optimizes the feature information integration process but also improves the model's sensitivity to potential individual differences in the input data, thereby enhancing overall performance.

Description

Technical Field

[0001] This invention belongs to the technical field of driver alertness, specifically relating to a driver alertness estimation method based on a hierarchical multi-scale topology enhancement network. Background Technology

[0002] In driving, alertness typically refers to a driver's sensitivity to their surroundings, their ability to anticipate potential dangers, and their capacity to react quickly to unexpected situations. Studies have shown that prolonged driving gradually diminishes an individual's cognitive abilities. Furthermore, in recent years, an increasing number of traffic accidents have been caused by reduced or completely lost driver alertness.

[0003] Currently, there are two main methods for detecting alertness levels. One is behavioral, such as analyzing eye state, facial expressions, and body posture collected via cameras. The other is physiological, such as electroencephalography (EEG), electrooculography (EOG), electrocardiography (ECG), and electromyography (EMG). Among the biosignals mentioned above, EEG is considered a reliable physiological indicator because it is affected by factors such as fatigue and alertness during different activities, such as driving. EOG collects the potentials between the front and back of the eyes (especially the cornea and retina), recorded from the forehead, containing information about alertness and eye movements (such as blinking and eye movement), and has the advantages of easy setup and high signal-to-noise ratio (SNR).

[0004] Various fusion methods have been used in previous multimodal models of EEG and EOG to maximize the complementary advantages of these two signals in alertness estimation. Traditional methods typically project the data from both modalities into a new feature space. For example, features such as differential entropy and power spectral density of each channel in EEG and EOG are extracted and concatenated along the channel dimension. Alternatively, EEG and EOG features are concatenated along the time dimension. However, this traditional fusion method is difficult to implement when there are differences in dimensionality and sampling rate between the modalities, and directly concatenating all feature information along a single dimension also results in a loss of information richness for each modality.

[0005] While the use of fused features fully utilizes information from both EEG and EOG, it faces challenges in analyzing specific channel or relational features compared to single-modality approaches. Previous studies have often enhanced the representational capabilities of single-modality approaches before fusion. For example, multi-head attention techniques have been used to enhance the representational capabilities of EEG single-modality approaches. Temporal channel attention modules have been used to study key temporal information and important channels in driver fatigue detection, refining the input EEG signals. However, these attention methods often directly calculate the corresponding attention weights based on features, failing to consider the complex topological structures or relationships between features. Summary of the Invention

[0006] The purpose of this invention is to address the aforementioned shortcomings in the prior art by providing a driver vigilance estimation method based on hierarchical multi-scale topology enhancement networks. This method solves the problem that in existing EEG and EOG multimodal models, attention methods often directly calculate the corresponding attention weights through features, failing to consider the complex topological structure or relationships between features.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A driver vigilance estimation method based on hierarchical multi-scale topology enhancement networks includes the following steps:

[0009] S1. Acquire EEG and EOG signals, and preprocess them to obtain the differential entropy features of different frequency bands of the EEG signal and the 36-dimensional features of the EOG signal.

[0010] S2. Construct a driver alert neural network model;

[0011] S3. Input the differential entropy features of different frequency bands of the EEG signal into the topological self-aware attention module in the driver warning neural network model, and output the EEG features combined with attention weights.

[0012] S4. Input the EEG features combined with attention weights and the 36-dimensional features of the EOG signal into the frequency band feature fusion module in the driver warning neural network model, and output the fused features of EEG and EOG.

[0013] S5. Input the fused features of EEG and EOG into the hierarchical multi-scale module in the driver warning neural network model, and output multi-scale features.

[0014] S6. Input the multi-scale features into the residual network module of the driver alertness neural network model, and output the driver fatigue classification results and alertness regression prediction results.

[0015] Furthermore, S1 specifically includes:

[0016] The EEG signal is decomposed into five frequency bands, namely δ, θ, α, β and γ, using a Butterworth filter. The α, β and γ frequency bands are selected and their differential entropy features are extracted.

[0017] The 36-dimensional features of the EOG signal were extracted using the features_table_ica, features_table_minus, and features_table_icav_minh separation methods, respectively.

[0018] Furthermore, S3 specifically includes:

[0019] In the topological self-aware attention module, the PCC method is used to calculate the differential entropy feature X of the EEG signal. E ∈R B×C×T Connectivity matrices for different frequency bands:

[0020]

[0021] Where, ρ ij For x i and x j The connectivity values ​​are calculated from two vectors, and multiple connectivity values ​​form a connectivity matrix. Three connectivity matrices A1∈R are calculated based on different frequency bands. C×T A2∈R C×T A3∈R C×T A1, A2, and A3 represent the connectivity matrices for the α, β, and γ frequency bands, respectively; x i ∈R T and x j ∈R T These represent two EEG channels; cov(·) represents the covariance calculated from the two EEG channel vectors; std is the variance of the calculated channel vectors; B is the number of frequency bands; C is the number of channels; T is the time point; R is the real number field; i,j are the EEG channel identifiers;

[0022] Calculate the attention weight for each frequency band based on the connectivity matrices A1, A2, and A3 of the three frequency bands:

[0023]

[0024] Where b represents different frequency bands, b = 1, 2 or 3; and q is a linearly variable parameter; b ,k b For the calculated key value, is the attention weight matrix; Att() is used to assign attention; d represents the weight factor, and softmax represents normalization;

[0025] Differential entropy characteristics of EEG signals in different frequency bands X E ∈R B×C×T By combining the corresponding attention weight matrix, we obtain the EEG features incorporating the attention weights:

[0026]

[0027] Where Concat(·) is the join function; and These are the differential entropy characteristics of the three frequency bands α, β, and γ, respectively. and These are the attention weight matrices for the three frequency bands α, β, and γ, respectively. EEG features that incorporate attention weights.

[0028] Furthermore, the S4 outputs the fused features of EEG and EOG, specifically as follows:

[0029] EEG features that incorporate attention weights 36-dimensional features X of EOG signals O ∈R 3×36 The input is fed into the frequency division band fusion module, and the output is the fused feature X of EEG and EOG. Fusion :

[0030]

[0031] Here, pad(·) is the padding method, using 0-value pairs. Fill in, represents the unpadded EEG features with attention weights; Concat(·) is the concatenation function.

[0032] Furthermore, the hierarchical multi-scale modules in S5 include pyramid pooling structures and optional convolutional structures;

[0033] The fused features of EEG and EOG are input into a pyramid pooling structure, and the expanded fused features are output.

[0034] The expanded fused features are input into an optional convolutional structure, and the weighted multi-scale features are output.

[0035] Furthermore, the fused features of EEG and EOG are input into the hierarchical multi-scale module, and the expanded fused features are output, including:

[0036] For fusion feature X Fusion Perform pooling operations:

[0037] P i =AdaptiveAvgPool2d(X Fusion ,(s i ,s i ))

[0038] Among them, P i The first intermediate fusion feature; AdaptiveAvgPool2d represents pooling operations at different scales, s i Let i be the pooling size, i = 1, 2, ..., n;

[0039] The first intermediate fusion feature P obtained i Perform bilinear interpolation adjustment and convolution processing:

[0040] R i =Conv(Interpolate(P i ,(H,W)))

[0041] Among them, R i ∈R C×H×W For feature maps; Conv is a 1x1 convolution; Interpolate represents bidirectional interpolation adjustment to incorporate the first intermediate fused feature P. i The integer is of size D×H×W, where D is the height, H and W are the height and width of a certain frequency band feature matrix;

[0042] Feature map R i With fusion feature X Fusion Perform cascading:

[0043] F = Concat(R1,R2,…,R) n ,X Fusion )

[0044] Where F is the second intermediate fusion feature, and Concat is the connection function;

[0045] The second intermediate fusion feature F is pooled and convolved to output the expanded fusion feature Z∈R. D′×H×W :

[0046] Z = ReLU(Conv(F,C′,1))

[0047] Where ReLU is the activation operation and C′ is the number of output channels.

[0048] Furthermore, the expanded fused features are input into an optional convolutional structure, outputting weighted multi-scale features, including:

[0049] The expanded fused feature Z is grouped and convolved using three convolutional kernels of different sizes, and then batch normalization and ReLU processing are performed separately to obtain feature values ​​of three different scales after segmentation. Where D' is the number of input channels;

[0050] Eigenvalues ​​at three different scales Integrate by summing the elements:

[0051]

[0052] Where, U∈R C‘×H×W C' represents the fused feature after integration; C' represents the number of channels after integration.

[0053] Calculate the average eigenvalue s for each channel:

[0054]

[0055] Among them, F gp (.) Calculates the average value of the feature values ​​for each channel, s i For the value in the i-th channel, the feature value will pass through a fully connected layer to obtain z∈R. r′ ;

[0056] The average feature value s is input into the fully connected layer to obtain the overall feature value z;

[0057]

[0058] Where, z∈R r′ ;F fc (.) Activates and standardizes the eigenvalues; δ is the ReLU activation function, and β is batch normalization. denoted as the feature weights, r′ as the number of features obtained after passing through the fully connected layer, and r as the number of features without passing through the linear layer;

[0059] A soft attention mechanism is used to adaptively select channel connectivity information at different scales for the overall eigenvalue z:

[0060]

[0061] in, For the feature weights at three different scales, a d b d c d These are the weights at three different scales;

[0062] The weights a at three different scales d b d c d Eigenvalues ​​at three different scales and

[0063]

[0064] in, These are the weighted multi-scale features.

[0065] Furthermore, S6 outputs the driver fatigue classification results as follows:

[0066] C class =SoftMax(WX′+b)

[0067] X′=ResNet(U)

[0068] Among them, C classX′ represents the fatigue category of the output, X′ represents the output result after passing through the residual ResNet module, and W represents the feature weights of X′.

[0069] Furthermore, S6 outputs the alertness regression prediction results, specifically as follows:

[0070] C reg =Sigmoid(WX′+b)

[0071] Among them, C reg This represents the regression prediction results.

[0072] The driver vigilance estimation method based on hierarchical multi-scale topology enhancement network provided by this invention has the following beneficial effects:

[0073] This invention not only preserves information about brain activity in different frequency bands of EEG but also effectively integrates relevant information from EOG. Building upon this, it further proposes a hierarchical multi-scale topology augmentation network (HMS-TENet). This network achieves multi-scale feature fusion and adaptive receptive field selection through pyramid pooling (PPS) and optional convolutional (SCS) structures, enabling the extraction of more effective features from small-sized features. Simultaneously, it enhances the representational power of EEG through the topological relationships between EEG channels. Furthermore, the model's output can be selected for both regression and classification tasks, providing greater flexibility and adaptability. This invention not only optimizes the feature information integration process but also improves the model's sensitivity to potential individual differences in the input data, thereby enhancing overall performance. Attached Figure Description

[0074] Figure 1 This is a flowchart of the driver vigilance estimation method based on hierarchical multi-scale topology enhancement network of the present invention.

[0075] Figure 2 This invention relates to a hierarchical multi-scale network.

[0076] Figure 3 Ablation experiments of the modules were conducted on different subjects for this invention.

[0077] Figure 4 This invention provides a comparison of the performance characteristics of fusion features and single-modal features.

[0078] Figure 5 The three subjects with better and worse performance in this invention demonstrate the fitting effect on the regression task.

[0079] Figure 6 For this invention, subject 2 and subject 11 were selected for the classification task, and the results were displayed using a confusion matrix.

[0080] Figure 7This invention provides classification and regression results for each subject.

[0081] Figure 8 This is the fitting situation across subjects in this invention.

[0082] Figure 9 The present invention visualized the EEG channel attention weights of subjects 1, 6, and 9 at different frequency bands.

[0083] Figure 10 The original features of this invention are classified into those using traditional direct splicing and frequency band fusion methods respectively. Detailed Implementation

[0084] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0085] Example 1

[0086] This embodiment provides a driver vigilance estimation method based on a hierarchical multi-scale topology augmentation network. It utilizes sampling devices to collect EOG, EEG, and eye opening / closing ratio data. Based on this, a deep learning model combined with a frequency banding method is used to fuse multimodal features from the preprocessed data. Finally, the model's output can be adapted to classification and regression tasks as needed. (Refer to...) Figure 1 Specifically, it includes the following:

[0087] Step S1: Obtain EEG and EOG signals, and preprocess the EEG and EOG signals to obtain the differential entropy features of different frequency bands of the EEG signal and the 36-dimensional features of the EOG signal.

[0088] The EEG signal was decomposed into five frequency bands using a Butterworth filter: δ (1-4 Hz), θ (4-8 Hz), α (8-14 Hz), β (14-31 Hz), and γ (31-51 Hz). The α, β, and γ frequency bands with larger weights were selected, and their differential entropy features were extracted.

[0089] The 36-dimensional features of the EOG signal were extracted using the features_table_ica, features_table_minus, and features_table_icav_minh separation methods, respectively.

[0090] Step S2: Construct a driver alert neural network model;

[0091] refer to Figure 2 The model in this embodiment consists of four parts: a topology self-aware attention module, a frequency band fusion module, a hierarchical multi-scale module, and a residual network module.

[0092] The topology-aware attention module calculates the correlation of EEG channels and then generates an attention weight matrix.

[0093] The frequency division fusion module fuses EEG and EOG features;

[0094] The hierarchical multi-scale module includes a pyramid pooling structure and an optional convolutional structure. The former uses average pooling layers of different sizes to capture global and local contextual information of the fused feature maps from different levels, while the latter captures multi-scale features at different levels by selectively applying convolutional kernels of different scales.

[0095] The residual network module obtains the final features through ResNet and then outputs the results through a fully connected layer. The final output of this network can be selected from two different tasks: regression and classification. The effectiveness of the fusion method and the network can be fully demonstrated through evaluation on these two tasks.

[0096] Step S3: Input the differential entropy features of different frequency bands of the EEG signal into the topological self-aware attention module in the driver warning neural network model, and output the EEG features combined with attention weights, which specifically include the following:

[0097] In the topological self-aware attention module, the PCC method is used to calculate the differential entropy feature X of the EEG signal. E ∈R B×C×T Connectivity matrices for different frequency bands:

[0098]

[0099] Where, ρ ij For x i and x j The connectivity values ​​are calculated from two vectors, and multiple connectivity values ​​form a connectivity matrix. Three connectivity matrices A1∈R are calculated based on different frequency bands. C×T A2∈R C×T A3∈R C×T A1, A2, and A3 represent the connectivity matrices for the α, β, and γ frequency bands, respectively; x i ∈R T and x j ∈R TThese represent two EEG channels respectively; cov(·) represents the covariance calculated from the two EEG channel vectors; std is the variance of the calculated channel vectors; B is the number of frequency bands; C is the number of channels; T is the time point; R is the real number field; i is the i-th EEG channel, representing the channel identifier; j is the j-th EEG channel, representing the channel identifier.

[0100] Calculate the attention weight for each frequency band based on the connectivity matrices A1, A2, and A3 of the three frequency bands:

[0101]

[0102] Where b represents different frequency bands, b = 1, 2 or 3; and q is a linearly variable parameter; b ,k b For the calculated key value, is the attention weight matrix; Att() is used to assign attention; d represents the weight factor, and softmax represents normalization;

[0103] Differential entropy characteristics of EEG signals in different frequency bands X E ∈R B×C×T By combining the corresponding attention weight matrix, we obtain the EEG features incorporating the attention weights:

[0104]

[0105] Where Concat(·) is the join function; and These are the differential entropy characteristics of the three frequency bands α, β, and γ, respectively. and These are the attention weight matrices for the three frequency bands α, β, and γ, respectively. EEG features that incorporate attention weights.

[0106] Step S4: Input the EEG features combined with attention weights and the 36-dimensional features of the EOG signal into the frequency band feature fusion module in the driver warning neural network model, and output the fused features of EEG and EOG, which specifically include the following:

[0107] EEG features that incorporate attention weights 36-dimensional features X of EOG signals O ∈R 3×36 The input is fed into the frequency division band fusion module, and the output is the fused feature X of EEG and EOG. Fusion :

[0108]

[0109] Among them, pad(· ) As a fill method, use 0 value pairs. Fill in, represents the unpadded EEG features with attention weights; Concat(·) is the concatenation function.

[0110] Step S5: Input the fused features of EEG and EOG into the hierarchical multi-scale module in the driver warning neural network model and output multi-scale features; the hierarchical multi-scale module includes a pyramid pooling structure and an optional convolutional structure;

[0111] Pyramid pooling structure;

[0112] The fused features of EEG and EOG are fed into this pyramid pooling structure. The pooling layer operations are similar to those of traditional pooling layers, but the difference lies in the fact that the pyramid pooling layer can extract features at multiple scales, thereby capturing features of different sizes and proportions. Specifically, the pyramid pooling layer generates feature maps at multiple scales by performing pooling operations on windows of different sizes, thus expanding the original fused features.

[0113] Specifically, first, a pooling size set (s1, s2, ..., s) is given. n In this embodiment, the selected set in the model is (1, 2, 3, 6), and then the fusion feature X is... Fusion Perform pooling operations:

[0114] P i =AdaptiveAvgPool2d(X Fusion ,(s i ,s i ))

[0115] Among them, P i The first intermediate fusion feature; AdaptiveAvgPool2d represents pooling operations at different scales, s i Let i be the pooling size, i = 1, 2, ..., n;

[0116] The first intermediate fusion feature P obtained i Perform bilinear interpolation adjustment and convolution processing:

[0117] R i =Conv(Interpolate(P i ,(H,W)))

[0118] Among them, R i ∈R C×H×W For feature maps; Conv is a 1x1 convolution; Interpolate represents bidirectional interpolation adjustment to incorporate the first intermediate fused feature P. iThe integer is of size D×H×W, where D is the height, H and W are the height and width of a certain frequency band feature matrix;

[0119] Feature map R i With fusion feature X Fusion Perform cascading:

[0120] F = Concat(R1,R2,…,R) n ,X Fusion )

[0121] Where F is the second intermediate fusion feature, and Concat is the connection function;

[0122] The features are concatenated using the Concat method to obtain the features F∈R after the pooling layer. (C·(n+1))×H×W The pooling size set selected by this model yields a feature of size F (15×36×36). Then, a 1x1 convolution is used to increase the number of channels in the fused feature to the number of output channels C′. The model is set to 16, and the output is the enlarged fused feature Z∈R. D ′×H×W :

[0123] Z = ReLU(Conv(F,C′,1))

[0124] Where ReLU is the activation operation and C′ is the number of output channels.

[0125] Optional convolutional structures;

[0126] The features output from the pyramid pooling structure are fed into a selective convolutional kernel structure. In traditional convolution operations, the size and shape of the convolutional kernel are fixed, meaning it can only capture features at a fixed scale. However, in the real world, a single-scale convolutional kernel may not be able to fully capture all features. Selective convolutional structures introduce a selective mechanism, allowing the network to dynamically select appropriate convolutional kernels based on the input data to process features at different scales. The selective convolutional structure includes three processes: segmentation, fusion, and selection.

[0127] segmentation;

[0128] The expanded fused feature Z is grouped and convolved using three convolutional kernels of different sizes, and then batch normalization and ReLU processing are performed separately to obtain feature values ​​of three different scales after segmentation. Where D' is the number of input channels;

[0129] Integration;

[0130] Eigenvalues ​​at three different scales Integrate by summing the elements:

[0131]

[0132] Where, U∈R C‘×H×W C' represents the fused feature after integration; C' represents the number of channels after integration.

[0133] Calculate the average eigenvalue s for each channel:

[0134]

[0135] Among them, F gp (.) Calculates the average value of the feature values ​​for each channel, s i For the value in the i-th channel, the feature value will pass through a fully connected layer to obtain z∈R. r′ ;

[0136] The average feature value s is input into the fully connected layer to obtain the overall feature value z;

[0137]

[0138] Where, z∈R r′ ;F fc (.) Activates and standardizes the eigenvalues; δ is the ReLU activation function, and β is batch normalization. denoted as the feature weights, r′ as the number of features obtained after passing through the fully connected layer, and r as the number of features without passing through the linear layer;

[0139] choose;

[0140] A soft attention mechanism is employed to adaptively select channel connectivity information at different scales for the overall feature value z, with attention weights...

[0141]

[0142] in, For the feature weights at three different scales, a d b d c d These are the weights at three different scales;

[0143] The weights a at three different scales d b d c d Eigenvalues ​​at three different scales and

[0144]

[0145] in, These are the weighted multi-scale features.

[0146] Step S6: Input the multi-scale features into the residual network module of the driver alertness neural network model, and output the driver fatigue classification results and alertness regression prediction results, which specifically include the following:

[0147] The first half of the residual network module mainly uses a ResNet network for downsampling. The advantage of ResNet is that it introduces residual connections:

[0148] X′=ResNet(U)

[0149] Different output modules are selected depending on the model's task. For classification tasks, the final output module is passed through a fully connected layer to obtain the feature dimension of the number of categories, and then SoftMax is used to calculate the categories. Specifically:

[0150] C class =SoftMax(WX′+b)

[0151] Among them, C class X′ represents the fatigue category of the output, X′ represents the output result after passing through the residual ResNet module, and W represents the feature weights of X′.

[0152] If the model is used for a regression task, then the feature dimension of the final output is set to 1, and then the Sigmoid function is used for calculation, specifically:

[0153] C reg =Sigmoid(WX′+b)

[0154] Among them, C reg This represents the regression prediction results.

[0155] Example 2

[0156] This embodiment uses experiments to evaluate and verify the method and the constructed driver warning neural network model in Embodiment 1, specifically including the following:

[0157] Dataset Introduction

[0158] SEED (SJTU Emotion EEG Dataset) is a publicly available dataset provided by the Center for Brain-Inspired Computing and Machine Intelligence at Shanghai Jiao Tong University. SEED-VIG (SJTU Emotion EEG Dataset-VigilanceEstimation) is a subset of SEED. It simulates a driving environment with straight and monotonous roads, making it easier to induce fatigue in participants. Electroencephalogram (EEG) and electrooculogram (EOG) signals were recorded using a neural scanning system. The experiment involved 23 participants, most of whom participated after lunch, and lasted approximately two hours. The driver's psychological fatigue state was recorded using the percentage of time their eyes were closed within eight seconds (PERCLOS), categorized into three types: alert, fatigued, and drowsy, with thresholds of 0.35 and 0.7, respectively.

[0159] Previous studies have primarily used this dataset for classification and regression tasks. In classification, a threshold was used to set the categories. This invention uses a threshold of 0.35 to define two categories: fatigue and alertness, for binary classification. For regression, the model output is used to fit the PERCLOS values. To evaluate the model's performance and applicability to these tasks, mixed experiments were conducted, including cross-subject and within-subject experiments. Five-fold cross-validation was used to reduce the volatility and randomness of the results.

[0160] Experiment Introduction

[0161] The regression task in this embodiment uses the root mean square error (RMSE) and the average correlation coefficient (COR) as the metrics:

[0162]

[0163] Where x and y are the observed value and the predicted value, respectively. and Calculate the average values ​​of x and y respectively. In summary, the lower the RMSE value and the higher the COR value, the higher the accuracy of the regression prediction.

[0164] The evaluation metrics for classification tasks are five commonly used classification metrics: accuracy, recall, precision, F1 score, and Kappa.

[0165]

[0166] In this context, TP represents a Truepositive case, TN represents a Truenegative case, FP represents a Falsepositive case, and FN represents a Falsenegative case.

[0167] Model evaluation uses five metrics: accuracy, recall, precision, F1 score, and Kappa. Accuracy is the proportion of correctly classified samples out of the total number of samples; recall is the proportion of correctly classified positive samples out of the actual number of positive samples; precision is the proportion of correctly classified positive samples out of the total number of samples predicted as positive; the F1 score is the harmonic mean of recall and precision; and Kappa is a statistical measure of consistency, used in classification problems to measure whether the model's predictions match the actual classification results.

[0168] Table 1. Performance of hierarchical multiscale modules with different parameters on classification and regression tasks.

[0169]

[0170] Table 1 presents the comparative results of setting different parameters for pyramid pooling and optional convolutional structures in the hierarchical multi-scale module, and validates their performance in classification and regression tasks. Experimental results show that at several larger scales, pyramid pooling and optional convolutional structures perform slightly worse than at smaller scales. This indicates that when fusing features, using small-scale sampling can more effectively capture feature information.

[0171] Ablation experiment;

[0172] Model ablation;

[0173] To clearly verify the advantages of each module in HMS-TENet of this invention, detailed ablation experiments were conducted, and the results are shown in Table 2. The experimental data clearly reveal that each module significantly improves performance, which fully demonstrates the effectiveness and value of the module design.

[0174] Meanwhile, the experimental results showed that removing PPS alone was even worse than not using HMM. However, when PPS and SCS were used together, the model performance was significantly improved. The experiments demonstrate the synergistic effect between PPS and SCS; they complement each other and jointly enhance the model's ability to capture features, thereby improving overall performance.

[0175] Table 2. Ablation experiments of different modules in the classification task. W / O refers to no...

[0176]

[0177] This embodiment conducted an ablation experiment to analyze the role of different modules in individual differences, and obtained the following results: Figure 3The results are shown. Experimental results indicate that, for a specific group of participants, removing certain modules has a significant impact on performance. Specifically, in the cases of participants 21 and 15, performance significantly decreased after removing the HMM module. Particularly for participant 21, the accuracy significantly improved from 86% to 97% after introducing the HMM module, an increase of 9%. This suggests that hierarchical multi-scale modules have significant performance improvement potential in certain individuals and exhibit superior feature extraction capabilities.

[0178] In the experiments with subjects 2, 6, and 23, the accuracy also decreased significantly after removing the topologically self-aware attention module. For subject 23, the accuracy dropped from 93% to 89% after removing the module. This result confirms that the attention weights calculated based on the topological relationships between EEG channels can significantly improve model performance on specific individuals.

[0179] Feature ablation

[0180] To verify that the model can indeed learn the fusion features between EEG and EOG, this embodiment conducted experimental analyses on features containing only EEG, features containing only EOG, and the fusion features of the two. The results are as follows: Figure 4 Demonstration. Experimental results show that the model using fused features performs significantly better in the network compared to models based solely on EEG or EOG features. In particular, it was observed that both the fused features and the EOG features, used individually, significantly outperformed the case of using only EEG features, a finding that strongly demonstrates our model's superior ability to capture EOG features.

[0181] Within-subjects experiment;

[0182] In in-subject experiments, the proposed method was evaluated on classification and regression tasks, and the results are shown in Table 3. The results show that in the regression task, the method of this invention exhibits significant performance advantages, with most participants achieving a correlation coefficient (COR) of approximately 0.98, demonstrating a high linear correlation between the model's predicted values ​​and the PERCLOS labels. Simultaneously, in the classification task, the method of this invention performed excellently across all participants, with most achieving classification accuracy exceeding 90%. Particularly noteworthy is that several participants achieved a perfect 100% accuracy, demonstrating the significant advantages and robustness of this method in handling classification tasks.

[0183] Table 3 shows the results of performing regression and classification tasks on each subject.

[0184]

[0185]

[0186] refer to Figure 5 This study demonstrates a comparison of the label fit on the test set between three poorly performing subjects (subjects 20, 2, and 10) and three well-performing subjects (subjects 8, 13, and 9) in a regression task. Overall, the model of this invention exhibits good fitting ability and significant correlation in the regression task. Figure 6 This demonstrates the confusion matrices of subjects 2 and 11 in the classification task. The results show that the model of this invention exhibits significant accuracy in classifying true negative samples.

[0187] Table 4. Regression tasks within the participants' tasks

[0188]

[0189] Table 5. Classification tasks within the participants' tasks

[0190]

[0191] Table 6 compares the models with other models over time.

[0192]

[0193] Tables 4 and 5 show the performance comparison between the proposed model and other methods on the SEED-VIG dataset. Experimental results demonstrate that the proposed method exhibits significant performance advantages in both regression and classification tasks. Compared to existing models and methods, the proposed method demonstrates superior performance and applicability in multiple aspects. In the regression task, the proposed method improves the COR to 0.98, a significant improvement over other models. In the classification task, the accuracy reaches a high level of approximately 97%. These experimental results fully demonstrate the significant advantages of the proposed model in feature extraction tasks.

[0194] Table 6 shows the superiority of our model over other methods in terms of computational speed. Specifically, our model exhibits a significant speed advantage during runtime, demonstrating higher computational efficiency compared to competing methods.

[0195] Cross-subject experiment

[0196] In addition, this invention also conducted cross-subject experiments to verify the generalization effect of the model. In the experiments, data from 23 subjects were used as the training set and the test set, respectively, for evaluation of classification and regression tasks.

[0197] Figure 7A dual y-axis plot is shown to illustrate the cross-subject performance in classification and regression tasks. In the classification task, the model of this invention demonstrated excellent classification performance on subjects 4, 8, 12, and 22, with an average accuracy of 85.2%. In the regression task, the model exhibited good fitting ability, with an average correlation coefficient (COR) and root mean square error (RMSE) of 0.852 and 0.146 respectively across subjects, demonstrating strong correlation.

[0198] refer to Figure 8 The results show the fit of subject 8 to PERCLOS in the cross-subject experiment. The results demonstrate a strong correlation between the model's predicted values ​​and PERCLOS, clearly demonstrating the powerful performance of the model proposed in this invention across subjects.

[0199] Visualization and Explanation

[0200] Frequency band fusion verification

[0201] refer to Figure 9 This embodiment explores the performance differences between frequency band fusion and traditional direct stitching methods in feature extraction and regression tasks. First, a simple ResNet was used to train the features obtained from both fusion methods. Then, t-SNE technology was used to visualize the high-dimensional feature space to intuitively observe the feature distribution.

[0202] from Figure 10 In (a), the original features were observed to be mixed and difficult to distinguish in t-SNE visualization, while Figure 10 (b) and Figure 10 (c) shows that both direct splicing and frequency-band fusion can improve feature discriminability, but frequency-band fusion is more significantly better at distinguishing feature distributions. To more intuitively demonstrate the advantage of frequency-band fusion, the COR of visualized samples from both fusion methods was calculated, and the results show that frequency-band fusion is more relevant.

[0203] The proposed frequency-band fusion method demonstrates excellent performance in feature extraction and classification tasks. This research provides important guidance and inspiration for subsequent exploration and optimization of feature fusion methods. Furthermore, in practical applications, the frequency-band fusion method not only effectively solves the problem of feature dimension mismatch between EEG and EOG, but also preserves the richness of feature information.

[0204] Attention weight matrix

[0205] To further investigate the connectivity between EEG channels, this invention employs attention matrix visualization technology to explore the correlation between channels for subjects with different prediction results (poor, average, and good) (e.g., subjects 2, 7, and 10). Figure 9 The study revealed a strong connection between channels in the frontal and occipital lobes, both of which have been shown to be related to mental attention. Furthermore, the research indicated that during distracted driving experiments, the frontal lobe was allocated more attentional resources for cognitive processing, while the occipital lobe was associated with sensory processing or stimulus selection. Additionally, we found significant activation values ​​in the CP1, CP2, P1, P2, and PZ channels across different frequency bands of the attention matrix, consistent with other studies.

[0206] Significant differences in attentional weights were observed among different participants within the same frequency band, particularly in the β band, where participant 10's attentional weights were almost entirely concentrated in the PZ channel. In contrast, participants 2 and 7 exhibited similar activation patterns, while participant 10's neural activation was primarily concentrated in the occipital lobe region. Comparing the attentional weights of these three participants, participant 10 showed a more pronounced difference in channel connectivity compared to the other two, yet demonstrated superior performance in fitting neural activity models. This suggests that attentional weights generated through topological relationships can more accurately reflect participants' specific neural responses and reveal correlations between channels.

[0207] Although specific embodiments of the invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of this patent. Various modifications and variations that can be made by a person skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of this patent.

Claims

1. A driver vigilance estimation method based on hierarchical multi-scale topology augmentation networks, characterized in that, Includes the following steps: S1. Acquire EEG and EOG signals, and preprocess them to obtain the differential entropy features of different frequency bands of the EEG signal and the 36-dimensional features of the EOG signal. S2. Construct a driver alert neural network model; S3. Input the differential entropy features of different frequency bands of the EEG signal into the topological self-aware attention module in the driver warning neural network model, and output the EEG features combined with attention weights, which specifically include: In the topological self-aware attention module, the PCC method is used to calculate the differential entropy features of the EEG signal. Connectivity matrices for different frequency bands: in, For x i and x j The connectivity values ​​are calculated from two vectors, and multiple connectivity values ​​are combined to form a connectivity matrix. Three connectivity matrices are calculated based on different frequency bands. A1, A2 and A3 represent respectively The connectivity matrix of the three frequency bands; and These represent two EEG channels; This represents the covariance calculated using two EEG channel vectors; std is the variance of the calculated channel vectors; B is the number of frequency bands; C is the number of channels; T is the time point; R is the real number field. i , j EEG channel identifier; Calculate the attention weight for each frequency band based on the connectivity matrices A1, A2, and A3 of the three frequency bands: in, To indicate different frequency bands, b = 1, 2, or 3; and , where is a linearly variable parameter; For the calculated key value, This is the attention weight matrix; Att() is used to assign attention. d represents the weighting factor, and softmax represents normalization; Differential entropy characteristics of EEG signals in different frequency bands By combining the corresponding attention weight matrix, we obtain the EEG features incorporating the attention weights: in, For connection functions; , and They are respectively Differential entropy characteristics of the three frequency bands; They are respectively Attention weight matrices for three frequency bands; EEG features incorporating attention weights; S4. Input the EEG features combined with attention weights and the 36-dimensional features of the EOG signal into the frequency band feature fusion module in the driver warning neural network model, and output the fused features of EEG and EOG. S5. Input the fused features of EEG and EOG into the hierarchical multi-scale module in the driver warning neural network model, and output multi-scale features. S6. Input the multi-scale features into the residual network module of the driver alertness neural network model, and output the driver fatigue classification results and alertness regression prediction results.

2. The driver vigilance estimation method based on hierarchical multi-scale topology augmentation network according to claim 1, characterized in that, S1 specifically includes: The EEG signal is decomposed into five frequency bands, namely δ, θ, α, β and γ, using a Butterworth filter. The α, β and γ frequency bands are selected and their differential entropy features are extracted. The 36-dimensional features of the EOG signal were extracted using the features_table_ica, features_table_minus, and features_table_icav_minh separation methods, respectively.

3. The driver vigilance estimation method based on hierarchical multi-scale topology augmentation network according to claim 1, characterized in that, The fusion feature of EEG and EOG output in S4 is specifically as follows: EEG features that incorporate attention weights 36-dimensional features of EOG signals The input is fed into the frequency band feature fusion module, which outputs the fused features of EEG and EOG. : in, As a fill method, use 0 value pairs. Fill in, EEG features with attention weights that have not been padded; This is a join function.

4. The driver vigilance estimation method based on hierarchical multi-scale topology augmentation network according to claim 3, characterized in that: The hierarchical multi-scale module in S5 includes a pyramid pooling structure and an optional convolutional structure; The fused features of EEG and EOG are input into a pyramid pooling structure, and the expanded fused features are output. The expanded fused features are input into an optional convolutional structure, and the weighted multi-scale features are output.

5. The driver vigilance estimation method based on hierarchical multi-scale topology augmentation network according to claim 4, characterized in that, The fused features of EEG and EOG are input into the hierarchical multi-scale module, and the expanded fused features are output, including: Fusion features Perform pooling operations: in, This is the first intermediate fusion feature; Pooling operations of different scales. For pooling size, ; The first intermediate fusion feature obtained Perform bilinear interpolation adjustment and convolution processing: in, For feature maps; It is a 1x1 convolution; This indicates a bidirectional interpolation adjustment to incorporate the first intermediate fusion feature. Complete to size D is the height and width of a certain frequency band feature matrix; feature map With fusion features Perform cascading: Where F represents the second intermediate fusion feature. For connection functions; The second intermediate fusion feature F is pooled and convolutionally processed to output the expanded fusion feature. : in, For activation, This represents the number of output channels.

6. The driver vigilance estimation method based on hierarchical multi-scale topology augmentation network according to claim 5, characterized in that, The expanded fused features are input into an optional convolutional structure, which outputs weighted multi-scale features, including: Three convolutional kernels of different sizes are used to apply the expanded fused features. Perform grouped convolutions, and then batch normalization and... The process yields feature values ​​at three different scales after segmentation. , , ;in, Input the number of channels; Eigenvalues ​​at three different scales , , Integrate by summing the elements: in, The integrated fusion features; This refers to the number of channels after integration. Calculate the average eigenvalue on each channel : in, The process of averaging the eigenvalues ​​on each channel is as follows: i For the first i The values ​​on each channel, and the feature values, are obtained by passing through a fully connected layer. ; average eigenvalues The input is fed into a fully connected layer to obtain the overall feature value z; in, ; Activation and standardization processes are applied to the eigenvalues; Here, β is the ReLU activation function, and β is the batch normalization. For feature weights, denoted as the number of features obtained after passing through the fully connected layer, and r is the number of features without passing through the linear layer. A soft attention mechanism is used to adaptively select channel connectivity information at different scales for the overall eigenvalue z: in, , , These are the feature weights at three different scales. , , These are the weights at three different scales; Weights at three different scales , , Eigenvalues ​​at three different scales , and : in, These are the weighted multi-scale features.

7. The driver vigilance estimation method based on hierarchical multi-scale topology augmentation network according to claim 6, characterized in that, The output of the driver fatigue classification result in S6 is as follows: in, For the fatigue category of the output, The output of the residual ResNet module is W. The characteristic weights.

8. The driver vigilance estimation method based on hierarchical multi-scale topology augmentation network according to claim 7, characterized in that, The output of the alertness regression prediction result in S6 is as follows: in, This represents the regression prediction results.

Images