Driver alert estimation method based on hierarchical multi-scale topology enhancement network

Abstract

The invention discloses a driver alert estimation method based on a hierarchical multi-scale topology enhancement network. The method comprises the following steps: acquiring differential entropy features of different frequency bands of an EEG signal and 36-dimensional features of an EOG signal; constructing a driver warning neural network model; and the differential entropy features of different frequency bands of the EEG signals are sequentially input into a topology self-perception attention module, a frequency band division special fusion module, a hierarchical multi-scale module and a residual network module in the driver alert neural network model, and a driving fatigue classification result and an alertness regression prediction result are output. The method not only optimizes the integration process of the feature information, but also improves the sensitivity of the model to the potential individual difference in the input data, thereby improving the overall performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of driver alertness, and in particular relates to a driver alertness estimation method based on a hierarchical multi-scale topology enhancement network. Background Art

[0002] In driving tasks, alertness often refers to the driver's sensitivity to the surrounding environment, the ability to predict potential dangers, and the ability to respond quickly to emergencies during driving. Studies have shown that long-term driving tasks will gradually weaken the individual's cognitive ability. In recent years, more and more traffic accidents have been caused by the reduction or complete loss of driver alertness.

[0003] There are currently two mainstream ways to detect alertness levels. One is a behavioral approach, such as collecting eye states, facial expressions, and body postures through a camera for analysis. The other method is a physiological method, such as electroencephalogram (EEG), electrooculogram (EOG), electrocardiogram (ECG), and electromyogram (EMG). Among the biological signals mentioned above, the EEG is affected by factors such as fatigue and alertness in different activities such as driving, so it is considered to be a reliable physiological feature. EOG collects the potential between the front and back of the human eye (especially the cornea and retina), recorded from the forehead, contains information about vigilance and eye movements (such as blinking and eye movement), and has the advantages of easy setup and high signal-to-noise ratio (SNR).

[0004] In previous EEG and EOG multimodal models, various fusion methods have been used to maximize the complementary advantages of these two signals in alertness estimation. The traditional method usually projects the data of the two modalities into a new feature space. For example, the differential entropy, power spectral density and other features of each channel of EEG and EOG are extracted to splice the features of different modalities in the channel dimension. Or the EEG and EOG features are spliced ​​in the time dimension. However, this traditional fusion method is difficult to fuse if there are differences in dimension and sampling rate between the modalities, and directly splicing all feature information in a certain dimension will also lose the information richness of each modality.

[0005] Although the use of fusion features makes full use of the information of EEG and EOG, it has difficulties in analyzing specific channel or relationship features compared with a single modality. In previous studies, the representation ability of a single modality is often enhanced before fusion. For example, on the single modality of EEG, multi-head attention is used to enhance its representation ability. The time channel attention module is used to study the key time information and important channels of driving fatigue detection and refine the input EEG signals. However, these attention methods often directly calculate the corresponding attention weights through features, and fail to take into account the complex topological structure or relationship between features. Summary of the invention

[0006] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art and to provide a driver alertness estimation method based on a hierarchical multi-scale topological enhancement network, so as to solve the problem that in the existing EEG and EOG multimodal models, the attention method often directly calculates the corresponding attention weights through features, but fails to take into account the complex topological structure or relationship between features.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A driver alertness estimation method based on a hierarchical multi-scale topology enhancement network comprises the following steps:

[0009] S1, obtaining EEG signals and EOG signals, and preprocessing the EEG signals and EOG signals to obtain differential entropy features of different frequency bands of the EEG signal and 36-dimensional features of the EOG signal;

[0010] S2, constructing a driver alert neural network model;

[0011] S3, inputting the differential entropy features of different frequency bands of the EEG signal into the topological self-perception attention module in the driver alertness neural network model, and outputting the EEG features combined with the attention weight;

[0012] S4, inputting the EEG features combined with the attention weight and the 36-dimensional features of the EOG signal into the frequency band feature fusion module in the driver alert neural network model, and outputting the fusion features of the EEG and EOG;

[0013] S5, inputting the fusion features of EEG and EOG into the hierarchical multi-scale module in the driver alertness neural network model, and outputting the multi-scale features;

[0014] S6. Input the multi-scale features into the residual network module in the driver alertness neural network model, and output the driving fatigue classification results and alertness regression prediction results.

[0015] Furthermore, S1 specifically includes:

[0016] The Butterworth filter was used to decompose the EEG signal into five frequency bands, namely, δ, θ, α, β and γ bands, and the α, β and γ bands were selected, and the differential entropy features of the α, β and γ bands were extracted;

[0017] The features_table_ica, features_table_minus and features_table_icav_minh separation methods are used to extract the 36-dimensional features of the EOG signal.

[0018] Furthermore, S3 specifically includes:

[0019] In the topological self-aware attention module, the PCC method is used to calculate the EEG signal differential entropy feature X E ∈R B×C×T Connectivity matrix for different frequency bands:

[0020]

[0021] Among them, ρ ij For x i and x j The connectivity values ​​calculated by the two vectors, multiple connectivity values ​​form a connectivity matrix, and three connectivity matrices A1∈R are calculated according to different frequency bands C×T ,A2∈R C×T ,A3∈R C×T , A1, A2 and A3 represent the connectivity matrices of the three frequency bands α, β and γ respectively; x i ∈R T and x j ∈R T Represent two EEG channels respectively; cov(·) represents the covariance calculated by two EEG channel vectors, std is the variance of the calculated channel vector; B is the number of frequency bands; C is the number of channels; T is the time point; R is the real number domain; i, j are EEG channel identifiers;

[0022] According to the connectivity matrices A1, A2 and A3 of the three frequency bands, the attention weight of each frequency band is calculated:

[0023]

[0024] Wherein, b represents different frequency bands, b = 1, 2 or 3; and is a linearly variable parameter; q b ,k b is the calculated key value, is the attention weight matrix; Att() is the allocated attention; d represents the weight factor, and softmax represents normalization;

[0025] Differentiate entropy features X of EEG signals in different frequency bands E ∈R B×C×T Combined with the corresponding attention weight matrix, the EEG features combined with the attention weight are obtained:

[0026]

[0027] Among them, Concat(·) is the concatenation function; and They are the differential entropy characteristics of the three frequency bands α, β, and γ; and They are the attention weight matrices for the three frequency bands α, β, and γ; is the EEG feature combined with attention weight.

[0028] Furthermore, S4 outputs the fusion features of EEG and EOG, specifically:

[0029] EEG features combined with attention weights and the 36-dimensional feature X of the EOG signal O ∈R 3×36 Input to the frequency band feature fusion module, output EEG and EOG fusion feature X Fusion :

[0030]

[0031] Among them, pad(·) is a filling method, using 0 value to To fill, is the unfilled EEG feature combined with the attention weight; Concat(·) is the connection function.

[0032] Furthermore, the hierarchical multi-scale module in S5 includes a pyramid pooling structure and an optional convolutional structure;

[0033] Input the fused features of EEG and EOG into the pyramid pooling structure and output the expanded fused features;

[0034] The enlarged fusion features are input into the optional convolution structure and the weighted multi-scale features are output.

[0035] Furthermore, the fused features of EEG and EOG are input into the hierarchical multi-scale module, and the expanded fused features are output, including:

[0036] For the fusion feature X Fusion Perform pooling operation:

[0037] P i =AdaptiveAvgPool2d(X Fusion ,(s i ,s i ))

[0038] Among them, P i is the first intermediate fusion feature; AdaptiveAvgPool2d is the pooling operation of different scales, s i is the pooling size, i=1,2,…,n;

[0039] The first intermediate fusion feature P i Perform bilinear interpolation adjustment and convolution processing:

[0040] R i =Conv(Interpolate(P i ,(H,W)))

[0041] Among them, R i ∈R C×H×W is a feature map; Conv is a 1x1 convolution; Interpolate represents a bidirectional interpolation adjustment to convert the first intermediate fusion feature P i Resize to D×H×W, where D is, H and W are the height and width of a frequency band feature matrix;

[0042] The feature map R i With the fusion feature X Fusion To cascade:

[0043] F=Concat(R1,R2,…,R n ,X Fusion )

[0044] Among them, F is the second intermediate fusion feature, and Concat is the connection function;

[0045] Perform pooling and convolution on the second intermediate fusion feature F, and output the expanded fusion feature Z∈R D′×H×W :

[0046] Z = ReLU(Conv(F,C′,1))

[0047] Among them, ReLU is the activation operation and C′ is the number of output channels.

[0048] Furthermore, the expanded fusion features are input into the optional convolution structure to output weighted multi-scale features, including:

[0049] Three convolution kernels of different sizes are used to perform group convolution on the enlarged fusion feature Z, and then batch normalization and ReLU processing are performed respectively to obtain the feature values ​​of three different scales after segmentation. Where D' is the number of input channels;

[0050] The three eigenvalues ​​of different scales Integration is done by summing the elements:

[0051]

[0052] Among them, U∈R C‘×H×W is the integrated fusion feature; C' is the number of channels after integration;

[0053] Calculate the average eigenvalue s on each channel:

[0054]

[0055] Among them, F gp (.) Calculate the average value of the feature value on each channel, s i is the value on the i-th channel, and the feature value will be obtained through the fully connected layer z∈R r′ ;

[0056] Input the average eigenvalue s into the fully connected layer to obtain the overall eigenvalue z;

[0057]

[0058] Where z∈R r′ ; F fc (.) activates and normalizes the feature values; δ is the RELU activation function, β is batch normalization, is the feature weight, r′ is the number of features obtained after the fully connected layer, and r is the number of features without passing through the linear layer;

[0059] The soft attention mechanism is used to adaptively select channel connectivity information of different scales for the overall eigenvalue z:

[0060]

[0061] in, are the feature weights at three different scales, a d , b d , c d They are weights at three different scales;

[0062] The weights a of the three different scales d , b d , c d Eigenvalues ​​for three different scales and

[0063]

[0064] in, is the weighted multi-scale feature.

[0065] Furthermore, S6 outputs the driving fatigue classification result, which is specifically:

[0066] C class =SoftMax(WX′+b)

[0067] X′=ResNet(U)

[0068] Among them, C classis the output fatigue category, X′ is the output result after the residual ResNet module, and W is the feature weight of X′.

[0069] Furthermore, S6 outputs the alertness regression prediction result, which is:

[0070] C reg =Sigmoid(WX′+b)

[0071] Among them, C reg The regression prediction results.

[0072] The driver alertness estimation method based on the hierarchical multi-scale topology enhancement network provided by the present invention has the following beneficial effects:

[0073] The present invention not only retains the information about brain activity in different frequency bands of EEG, but also effectively integrates the relevant information of EOG. On this basis, a hierarchical multi-scale topological enhancement network (HMS-TENet) is further proposed. The network realizes multi-scale feature fusion and adaptive receptive field selection through a pyramid pooling structure (PPS) and a selectable convolution structure (SCS), so that more effective features can be mined from small-size features. At the same time, the representation ability of EEG is enhanced through the topological relationship between EEG channels. In addition, the output of the model can be selected as two task types, regression and classification, with higher flexibility and adaptability. The method of the present invention not only optimizes the integration process of feature information, but also improves the sensitivity of the model to potential individual differences in the input data, thereby improving the overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 The present invention is a flowchart of a driver alertness estimation method based on a hierarchical multi-scale topology enhancement network.

[0075] Figure 2 This is a hierarchical multi-scale network of the present invention.

[0076] Figure 3 The ablation experiment of the module was carried out on different subjects of the present invention.

[0077] Figure 4 The figure shows the performance comparison between the fusion features of the present invention and the features of a single modality.

[0078] Figure 5 This is the fitting effect of the three subjects with better and worse effects in the regression task of the present invention.

[0079] Figure 6 For the present invention, subjects 2 and 11 were selected for the classification task and the results were displayed using a confusion matrix.

[0080] Figure 7This is the classification and regression effect of the present invention across each subject.

[0081] Figure 8 This is the cross-subject fitting situation of the present invention.

[0082] Fig. 9 The EEG channel weights of attention weights in different frequency bands of subjects 1, 6, and 9 were visualized for the present invention.

[0083] Fig.10 The original features of the present invention and the feature classification using the traditional direct splicing and sub-band fusion methods respectively. DETAILED DESCRIPTION

[0084] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0085] Example 1

[0086] This embodiment provides a driver alertness estimation method based on a hierarchical multi-scale topological enhancement network. The sampling device is used to collect EOG, EEG and eye opening and closing ratio data respectively. On this basis, the pre-processed data is fused with multimodal features through a deep learning model combined with a frequency band division method. Finally, the output of the model can be applied to classification tasks and regression tasks according to the needs. Figure 1 , which specifically includes the following:

[0087] Step S1, acquiring EEG signals and EOG signals, and preprocessing the EEG signals and EOG signals to obtain differential entropy features of different frequency bands of the EEG signals and 36-dimensional features of the EOG signals;

[0088] The Butterworth filter was used to decompose the EEG signal into five frequency bands, namely, δ (1-4 Hz), θ (4-8 Hz), α (8-14 Hz), β (14-31 Hz) and γ (31-51 Hz). The α, β and γ bands with larger weights were selected, and the differential entropy features of the α, β and γ bands were extracted.

[0089] The features_table_ica, features_table_minus and features_table_icav_minh separation methods are used to extract the 36-dimensional features of the EOG signal.

[0090] Step S2, constructing a driver alert neural network model;

[0091] refer to Figure 2 ,The model of this embodiment includes four parts: topological self-aware attention module, frequency band fusion module, hierarchical multi-scale module, and residual network module.

[0092] The topological self-aware attention module calculates the correlation of EEG channels and then generates an attention weight matrix;

[0093] The frequency band feature fusion module fuses EEG and EOG features;

[0094] The hierarchical multi-scale module includes a pyramid pooling structure and an optional convolutional structure. The former uses average pooling layers of different sizes to capture the global and local contextual information of the fused feature map from different levels, while the latter captures multi-scale features at different levels by selectively applying convolution kernels of different scales.

[0095] The residual network module obtains the final features through ResNet and then outputs the results through the fully connected layer. The final output of the network can be selected into two different outputs: regression task and classification task, and the effectiveness of the fusion method and network is fully demonstrated through the evaluation of these two tasks.

[0096] Step S3: Input the differential entropy features of different frequency bands of the EEG signal into the topological self-perception attention module in the driver alert neural network model, and output the EEG features combined with the attention weights, which specifically includes the following contents:

[0097] In the topological self-aware attention module, the PCC method is used to calculate the EEG signal differential entropy feature X E ∈R B×C×T Connectivity matrix for different frequency bands:

[0098]

[0099] Among them, ρ ij For x i and x j The connectivity values ​​calculated by the two vectors, multiple connectivity values ​​form a connectivity matrix, and three connectivity matrices A1∈R are calculated according to different frequency bands C×T ,A2∈R C×T ,A3∈R C×T , A1, A2 and A3 represent the connectivity matrices of the three frequency bands α, β and γ respectively; x i ∈R T and x j ∈R TRepresent two EEG channels respectively; cov(·) represents the covariance calculated by two EEG channel vectors, std is the variance of the calculated channel vector; B is the number of frequency bands; C is the number of channels; T is the time point; R is the real number domain; i is the i-th EEG channel, indicating the channel identifier; j is the j-th EEG channel, indicating the channel identifier;

[0100] According to the connectivity matrices A1, A2 and A3 of the three frequency bands, the attention weight of each frequency band is calculated:

[0101]

[0102] Wherein, b represents different frequency bands, b = 1, 2 or 3; and is a linearly variable parameter; q b ,k b is the calculated key value, is the attention weight matrix; Att() is the allocated attention; d represents the weight factor, and softmax represents normalization;

[0103] Differentiate entropy features X of EEG signals in different frequency bands E ∈R B×C×T Combined with the corresponding attention weight matrix, the EEG features combined with the attention weight are obtained:

[0104]

[0105] Among them, Concat(·) is the concatenation function; and They are the differential entropy characteristics of the three frequency bands α, β, and γ; and They are the attention weight matrices for the three frequency bands α, β, and γ; is the EEG feature combined with attention weight.

[0106] Step S4: input the EEG features combined with the attention weight and the 36-dimensional features of the EOG signal into the frequency band feature fusion module in the driver alert neural network model, and output the fusion features of EEG and EOG, which specifically include the following contents:

[0107] EEG features combined with attention weights and the 36-dimensional feature X of the EOG signal O ∈R 3×36 Input to the frequency band feature fusion module, output EEG and EOG fusion feature X Fusion :

[0108]

[0109] Among them, pad( ) For the filling method, use 0 value To fill, is the unfilled EEG feature combined with the attention weight; Concat(·) is the connection function.

[0110] Step S5, inputting the fusion features of EEG and EOG into the hierarchical multi-scale module in the driver alert neural network model, and outputting the multi-scale features; the hierarchical multi-scale module includes a pyramid pooling structure and an optional convolution structure;

[0111] Pyramid pooling structure;

[0112] The fused features of EEG and EOG are passed into the pyramid pooling structure, where the pooling layer operation is similar to the traditional pooling layer, but the difference is that the pyramid pooling layer can extract features at multiple scales, thereby capturing features of different sizes and proportions. Specifically, the pyramid pooling layer generates feature maps of multiple scales by performing pooling operations on windows of different sizes, thereby expanding the original fused features.

[0113] Specifically, we first give a pooling size set (s1, s2, ..., s n ), the set selected in the model of this embodiment is (1, 2, 3, 6), and then the fusion feature X Fusion Perform pooling operation:

[0114] P i =AdaptiveAvgPool2d(X Fusion ,(s i ,s i ))

[0115] Among them, P i is the first intermediate fusion feature; AdaptiveAvgPool2d is the pooling operation of different scales, s i is the pooling size, i=1,2,…,n;

[0116] The first intermediate fusion feature P i Perform bilinear interpolation adjustment and convolution processing:

[0117] R i =Conv(Interpolate(P i ,(H,W)))

[0118] Among them, R i ∈R C×H×W is a feature map; Conv is a 1x1 convolution; Interpolate represents a bidirectional interpolation adjustment to convert the first intermediate fusion feature P iResize to D×H×W, where D is, H and W are the height and width of a frequency band feature matrix;

[0119] The feature map R i With the fusion feature X Fusion To cascade:

[0120] F=Concat(R1,R2,…,R n ,X Fusion )

[0121] Among them, F is the second intermediate fusion feature, and Concat is the connection function;

[0122] The Concat method is used to connect the features and obtain the feature F∈R after the pooling layer. (C·(n+1))×H×W , and the size of the pooling size set selected by this model is (15×36×36) feature, and then the number of channels of the fused feature is increased to the output channel number C′ using 1x1 convolution, and the model is set to 16, and the expanded fused feature Z∈R is output D ′×H×W :

[0123] Z = ReLU(Conv(F,C′,1))

[0124] Among them, ReLU is the activation operation and C′ is the number of output channels.

[0125] Optional convolutional structure;

[0126] The features output from the pyramid pooling structure are passed into the selective convolution kernel structure; in traditional convolution operations, the size and shape of the convolution kernel are fixed, which means that it can only capture features of a fixed scale. But in the real world, a convolution kernel of a single scale may not be able to fully capture all features. The selective convolution structure allows the network to dynamically select appropriate convolution kernels to process features of different scales based on the input data by introducing a selective mechanism. The selective convolution structure includes three processes: segmentation, fusion, and selection;

[0127] segmentation;

[0128] Three convolution kernels of different sizes are used to perform group convolution on the enlarged fusion feature Z, and then batch normalization and ReLU processing are performed respectively to obtain the feature values ​​of three different scales after segmentation. Where D' is the number of input channels;

[0129] Fusion;

[0130] The three eigenvalues ​​of different scales Integration is done by summing the elements:

[0131]

[0132] Among them, U∈R C‘×H×W is the integrated fusion feature; C' is the number of channels after integration;

[0133] Calculate the average eigenvalue s on each channel:

[0134]

[0135] Among them, F gp (.) Calculate the average value of the feature value on each channel, s i is the value on the i-th channel, and the feature value will be obtained through the fully connected layer z∈R r′ ;

[0136] Input the average eigenvalue s into the fully connected layer to obtain the overall eigenvalue z;

[0137]

[0138] Where z∈R r′ ; F fc (.) activates and normalizes the feature values; δ is the RELU activation function, β is batch normalization, is the feature weight, r′ is the number of features obtained after the fully connected layer, and r is the number of features without passing through the linear layer;

[0139] choose;

[0140] The soft attention mechanism is used to adaptively select channel connectivity information of different scales for the overall eigenvalue z, and the attention weight

[0141]

[0142] in, are the feature weights at three different scales, a d , b d , c d They are weights at three different scales;

[0143] The weights a of the three different scales d , b d , c d Eigenvalues ​​for three different scales and

[0144]

[0145] in, is the weighted multi-scale feature.

[0146] Step S6: input the multi-scale features into the residual network module in the driver alertness neural network model, and output the driving fatigue classification results and alertness regression prediction results, which specifically include the following contents:

[0147] The first half of the residual network module mainly uses the ResNet network for down sampling. The advantage of ResNet is that it introduces residual connections:

[0148] X′=ResNet(U)

[0149] Different output modules are selected according to the tasks of the model. If it is a classification task, the feature dimension of the number of classification categories is obtained by the fully connected layer, and then the categories are calculated using SoftMax, specifically:

[0150] C class =SoftMax(WX′+b)

[0151] Among them, C class is the output fatigue category, X′ is the output result after the residual ResNet module, and W is the feature weight of X′.

[0152] If the model is used for regression tasks, the feature dimension of the final output is set to 1, and then Sigmoid is used for calculation, specifically:

[0153] C reg =Sigmoid(WX′+b)

[0154] Among them, C reg The regression prediction results.

[0155] Example 2

[0156] This embodiment uses experiments to evaluate and verify the method and the constructed driver alert neural network model in Embodiment 1, which specifically includes the following contents:

[0157] Dataset Introduction

[0158] SEED (SJTU Emotion EEG Dataset) is a public dataset provided by the laboratory of the Brain-like Computing and Machine Intelligence Research Center of Shanghai Jiao Tong University. SEED-VIG (SJTU Emotion EEG Dataset-VigilanceEstimation) is a sub-dataset of SEED, which developed a simulated driving environment with straight and monotonous roads, which are more likely to induce fatigue in subjects, and used a neural scanning system to record EEG signals and electrooculogram signals. There were 23 subjects in the experiment, most of whom were tested after lunch, and the experiment lasted about 2 hours. The driver's psychological fatigue state was recorded by the percentage of eye closure time within 8 seconds (PERCLOS), which was divided into three types: wakefulness, fatigue and drowsiness, with thresholds of 0.35 and 0.7 respectively.

[0159] In previous studies, this dataset was mainly used for classification and regression tasks. In the classification task, the classification category was mainly set according to the threshold, while the present invention mainly uses a threshold of 0.35 to set the two categories of fatigue and wakefulness for binary classification. For the regression task, the value output by the model is used to fit the value of PERCLOS to perform regression. In order to evaluate the performance of the model and its applicability in these two tasks. Mixed experiments, cross-subject experiments and within-subject experiments were conducted, and in order to reduce the volatility and contingency of the experimental results, five-fold cross validation was used for the experiment.

[0160] Experiment Introduction

[0161] The indicators used in the regression task of this embodiment are the average root mean square error (RMSE) and the average correlation coefficient (COR):

[0162]

[0163] Among them, x and y are the observed value and the predicted value respectively. and The average values ​​of x and y are marked respectively. In short, the lower the RMSE value, the higher the COR value, and the higher the accuracy of the prediction regression.

[0164] The evaluation indicators of the classification task use five commonly used classification indicators, namely accuracy, recall, precision, F1 score and Kappa;

[0165]

[0166] Among them, TP represents Truepositive, TN represents Truenegtive, FP represents Falsepositive, and FN represents Falsenegtive.

[0167] Model evaluation uses five evaluation indicators: accuracy, recall, precision, F1 score and Kappa. Accuracy refers to the ratio of the total number of correctly classified samples to the total number of samples; Recall refers to the ratio of the number of correctly classified positive samples to the total number of actual positive samples; Precision refers to the ratio of the number of correctly classified positive samples to the total number of samples predicted as positive samples; F1 score refers to the harmonic mean of recall and precision; Kappa is an indicator of consistency in statistics, and is used to measure whether the model prediction results are consistent with the actual classification results in classification problems.

[0168] Table 1 Effects of hierarchical multi-scale modules with different parameters on classification and regression tasks

[0169]

[0170] Table 1 shows the comparison results of setting different parameters for pyramid pooling structure and optional convolution structure in the hierarchical multi-scale module, and verifies their effects in classification and regression tasks. The experimental results show that at multiple larger scales, the performance of pyramid pooling structure and optional convolution structure is slightly worse than that of smaller scales. This shows that when fusing features, using small-scale sampling can more effectively capture feature information.

[0171] Ablation experiments;

[0172] Model ablation;

[0173] In order to clearly verify the advantages of each module in the HMS-TENet of the present invention, the present invention implements a detailed ablation experiment, and the results are shown in Table 2. The experimental data clearly reveals that each module has a significant improvement in performance, which fully proves the effectiveness and value of the design of each module.

[0174] At the same time, it is noted in the experimental results that the effect of removing PPS alone is worse than when HMM is not used. However, when PPS and SCS are used together, the model performance is significantly improved. The experiment proves the synergy between PPS and SCS. They complement each other and jointly enhance the model's ability to capture features, thereby improving the overall performance.

[0175] Table 2 Ablation experiments of different modules in classification tasks. W / O means no

[0176]

[0177] This example conducts an ablation experiment to analyze the role of different modules in individual differences, and obtains the following results: Figure 3Results shown. The experimental results show that for certain groups of subjects, removing certain modules has a significant impact on performance. Specifically, in the cases of subjects 21 and 15, the performance dropped significantly after removing the HMM module. In particular, for subject 21, the accuracy increased significantly from 86% to 97% after the introduction of the HMM module, an increase of 9%. This shows that the hierarchical multi-scale module has significant performance improvement potential in certain individuals and exhibits excellent feature extraction capabilities.

[0178] In the experiments of subjects 2, 6, and 23, the accuracy also dropped significantly after removing the topological self-sensing attention module. According to the results of subject 23, after removing this module, the accuracy dropped from 93% to 89%. This result confirms that the attention weights calculated based on the topological relationship between EEG channels can significantly improve the model performance on specific individuals.

[0179] Feature Ablation

[0180] In order to verify that the model can indeed learn the fusion features between EEG and EOG, this embodiment conducts experimental analysis on the features containing only EEG, only EOG, and the fusion features of the two. The results are as follows: Figure 4 The experimental results show that the model using fusion features performs better in the network than the model based on EEG or EOG single features. In particular, it is observed that the effect of fusion features and EOG features when used alone is significantly better than that of using only EEG features. This finding strongly proves that our model has excellent ability to capture EOG features.

[0181] within-subjects experiment;

[0182] In the within-subjects experiment, the proposed method was evaluated for classification and regression tasks, and the results are shown in Table 3. The results show that in the regression task, the method of the present invention exhibits a significant performance advantage, and the correlation coefficient (COR) of most subjects can reach about 0.98, showing the characteristics of a high linear correlation between the model prediction value and the PERCLOS label. At the same time, in the classification task, the method of the present invention performs well in all subjects, and the classification accuracy of most subjects exceeds 90%. It is particularly noteworthy that the classification results of several subjects achieved a perfect 100% accuracy, demonstrating the significant advantages and robustness of the method in handling classification tasks.

[0183] Table 3 Effects of performing regression and classification tasks on each

[0184]

[0185]

[0186] refer to Figure 5 , which shows the comparison of the fitting of labels on the test set by three subjects with poor performance (subjects 20, 2, 10) and three subjects with good performance (subjects 8, 13, 9) in the regression task. In general, the model of the present invention shows good fitting ability and significant correlation in the regression task. Figure 6 The confusion matrices of subjects 2 and 11 are respectively examined in the classification task. From the results, the model of the present invention shows significant accuracy in the classification effect of true negative samples.

[0187] Table 4 Regression tasks within the subject task

[0188]

[0189] Table 5 Classification tasks within the subject task

[0190]

[0191] Table 6 compares the time with other models

[0192]

[0193] Tables 4 and 5 show the comparative effects of the model proposed in the present invention and other methods on the SEED-VIG dataset. The experimental results show that the method of the present invention shows significant performance advantages in both regression and classification tasks. Compared with other existing models and methods, the method of the present invention exhibits superior performance and applicability in many aspects. In the regression task, the present invention improves the COR to 0.98, which is a significant improvement compared with other models. In the classification task, the accuracy rate reaches a high accuracy rate of about 97%. These experimental results fully prove that the model proposed in the present invention has significant advantages and in feature extraction tasks.

[0194] Table 6 shows the superiority of our model over other methods in terms of computational speed. Specifically, our model exhibits significant speed advantages at runtime and is more computationally efficient than competing methods.

[0195] Cross-subject experiment

[0196] At the same time, the present invention also conducted cross-subject experiments to verify the generalization effect of the model. In the experiment, the data of 23 subjects were used as training sets and test sets respectively to perform classification and regression task evaluation.

[0197] Figure 7A dual y-axis graph is shown to show the cross-subject effects in the classification and regression tasks. In the classification task, the model of the present invention showed excellent classification results on subjects 4, 8, 12 and 22, with an average accuracy of 85.2%. In the regression task, the present invention showed good fitting ability, and the average correlation coefficient (COR) and root mean square error (RMSE) across the experiment reached 0.852 and 0.146 respectively, showing a strong correlation.

[0198] refer to Figure 8 , showing the fitting of subject 8 to PERCLOS in the cross-subject experiment. The results show that there is a strong correlation between the predicted value of the model and PERCLOS, which clearly shows the powerful performance of the model proposed in the present invention in the cross-subject experiment.

[0199] Visualization and interpretation

[0200] Frequency division fusion verification

[0201] refer to Fig. 9 This example explores the performance differences between frequency band fusion and traditional direct concatenation in feature extraction and regression tasks. First, the features obtained by the two fusion methods are trained using a simple ResNet. Then, the high-dimensional feature space is visualized using t-SNE technology to intuitively observe the distribution of features.

[0202] from Fig.10 In (a), it is observed that the original features appear mixed and difficult to distinguish in t-SNE visualization, while Fig.10 (b) and Fig.10 (c) shows that both direct concatenation and frequency band fusion can improve the distinguishability of features, but the frequency band fusion method is more significant in distinguishing the feature distribution. In order to more intuitively prove that the frequency band fusion method is more advantageous, the COR of the visualized samples of the two fusion methods is calculated, and the results show that the frequency band fusion method is more relevant.

[0203] The proposed frequency band fusion method shows excellent performance in feature extraction and classification tasks. This research result provides important guidance and inspiration for the subsequent exploration and optimization of feature fusion methods. In addition, in practical applications, the frequency band fusion method not only effectively solves the problem of mismatch between EEG and EOG feature dimensions, but also retains the richness of feature information.

[0204] Attention weight matrix

[0205] In order to further study the connectivity between EEG channels, the present invention uses attention matrix visualization technology to explore the correlation between channels for subjects with different prediction effects (poor, average, and good) (such as subjects 2, 7, and 10). Fig. 9 It shows that the channels in the frontal lobe area are strongly connected to the channels in the occipital lobe area, and these two areas have been shown to be related to mental attention functions. And studies have shown that in distracted driving experiments, the frontal lobe area is allocated more attention resources for cognitive processing, while the occipital lobe area is related to sensory processing or stimulus selection. Further, we found that there are larger activation values ​​on the CP1, CP2, P1, P2, and PZ channels in the attention matrix of different frequency bands, which is also consistent with other studies.

[0206] It was observed that there were significant differences in the attention weights of different subjects in the same frequency band, especially in the β frequency band, where the attention weight of subject 10 was almost entirely concentrated on the PZ channel. Relatively speaking, subjects 2 and 7 showed similar activation characteristics in neural activation patterns, while the neural activation of subject 10 was mainly concentrated in the occipital lobe area. Comparing the attention weights of these three subjects, it was found that the channel connectivity of subject 10 showed more obvious differences compared with the other two subjects, but showed better results when fitting the neural activity model. This shows that the attention weights generated by topological relationships can more accurately reflect the specific neural responses of the subjects and reveal the correlation between channels.

[0207] Although the specific implementation of the invention is described in detail in conjunction with the drawings, it should not be understood as limiting the scope of protection of this patent. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.

Claims

1. A driver alertness estimation method based on a hierarchical multi-scale topology enhancement network, characterized in that: The following steps are involved: S1, obtaining EEG signals and EOG signals, and preprocessing the EEG signals and EOG signals to obtain differential entropy features of different frequency bands of the EEG signal and 36-dimensional features of the EOG signal; S2, constructing a driver alert neural network model; S3, inputting the differential entropy features of different frequency bands of the EEG signal into the topological self-perception attention module in the driver alertness neural network model, and outputting the EEG features combined with the attention weight; S4, inputting the EEG features combined with the attention weight and the 36-dimensional features of the EOG signal into the frequency band feature fusion module in the driver alert neural network model, and outputting the fusion features of the EEG and EOG; S5, inputting the fusion features of EEG and EOG into the hierarchical multi-scale module in the driver alertness neural network model, and outputting the multi-scale features; S6. Input the multi-scale features into the residual network module in the driver alertness neural network model, and output the driving fatigue classification results and alertness regression prediction results.

2. The driver alertness estimation method based on hierarchical multi-scale topology enhancement network according to claim 1 is characterized in that: The S1 specifically includes: The Butterworth filter was used to decompose the EEG signal into five frequency bands, namely, δ, θ, α, β and γ bands, and the α, β and γ bands were selected, and the differential entropy features of the α, β and γ bands were extracted; The features_table_ica, features_table_minus and features_table_icav_minh separation methods are used to extract the 36-dimensional features of the EOG signal.

3. The driver alertness estimation method based on hierarchical multi-scale topology enhancement network according to claim 1 is characterized in that: The S3 specifically includes: In the topological self-aware attention module, the PCC method is used to calculate the EEG signal differential entropy feature X E ∈R B×C×T Connectivity matrix for different frequency bands: Among them, ρ ij For x i and x j The connectivity values ​​calculated by the two vectors, multiple connectivity values ​​form a connectivity matrix, and three connectivity matrices A1∈R are calculated according to different frequency bands C×T ,A2∈R C×T ,A3∈R C×T , A1, A2 and A3 represent the connectivity matrices of the three frequency bands α, β and γ respectively; x i ∈R T and x j ∈R T Represent two EEG channels respectively; cov(·) represents the covariance calculated by two EEG channel vectors, std is the variance of the calculated channel vector; B is the number of frequency bands; C is the number of channels; T is the time point; R is the real number domain; i, j are EEG channel identifiers; According to the connectivity matrices A1, A2 and A3 of the three frequency bands, the attention weight of each frequency band is calculated: Wherein, b represents different frequency bands, b = 1, 2 or 3; and is a linearly variable parameter; q b ,k b is the calculated key value, is the attention weight matrix; Att() is the allocated attention; d represents the weight factor, and softmax represents normalization; Differentiate entropy features X of EEG signals in different frequency bands E ∈R B×C×T Combined with the corresponding attention weight matrix, the EEG features combined with the attention weight are obtained: Among them, Concat(·) is the concatenation function; and These are the differential entropy characteristics of the three frequency bands α, β, and γ; and They are the attention weight matrices for the three frequency bands α, β, and γ; is the EEG feature combined with attention weight.

4. The driver alertness estimation method based on hierarchical multi-scale topology enhancement network according to claim 3 is characterized in that: The fusion features of EEG and EOG output in S4 are specifically: EEG features combined with attention weights and the 36-dimensional feature X of the EOG signal O ∈R 3×36 Input to the frequency band feature fusion module, output EEG and EOG fusion feature X Fusion : Among them, pad( ) For the filling method, use 0 value Fill it up. is the unfilled EEG feature combined with the attention weight; Concat(·) is the connection function.

5. The driver alertness estimation method based on hierarchical multi-scale topology enhancement network according to claim 4 is characterized in that: The hierarchical multi-scale module in S5 includes a pyramid pooling structure and an optional convolution structure; Input the fused features of EEG and EOG into the pyramid pooling structure and output the expanded fused features; The enlarged fusion features are input into the optional convolution structure and the weighted multi-scale features are output.

6. The driver alertness estimation method based on hierarchical multi-scale topology enhancement network according to claim 5 is characterized in that: The fused features of EEG and EOG are input into the hierarchical multi-scale module, and the expanded fused features are output, including: For the fusion feature X Fusion Perform pooling operation: P i =AdaptiveAvgPool2d(X Fusion ,(s i ,s i )) Among them, P i is the first intermediate fusion feature; AdaptiveAvgPool2d is the pooling operation of different scales, s i is the pooling size, i=1,2,…,n; The first intermediate fusion feature P i Perform bilinear interpolation adjustment and convolution processing: R i =Conv(Interpolate(P i ,(H,W))) Among them, R i ∈R C×H×W is a feature map; Conv is a 1x1 convolution; Interpolate represents a bidirectional interpolation adjustment to convert the first intermediate fusion feature P i Resize to D×H×W, where D is, H and W are the height and width of a frequency band feature matrix; The feature map R i With the fusion feature X Fusion To cascade: F=Concat(R1,R2,…,R n ,X Fusion ) Among them, F is the second intermediate fusion feature, and Concat is the connection function; Perform pooling and convolution on the second intermediate fusion feature F, and output the expanded fusion feature Z∈R D′×H×W : Z = ReLU(Conv(F,C′,1)) Among them, ReLU is the activation operation and C′ is the number of output channels.

7. The driver alertness estimation method based on hierarchical multi-scale topology enhancement network according to claim 6 is characterized in that: The expanded fusion features are input into the optional convolution structure to output weighted multi-scale features, including: Three convolution kernels of different sizes are used to perform group convolution on the enlarged fusion feature Z, and then batch normalization and ReLU processing are performed respectively to obtain the feature values ​​of three different scales after segmentation. Where D' is the number of input channels; The three eigenvalues ​​of different scales Integration is done by summing the elements: Among them, U∈R C‘×H×W is the integrated fusion feature; C' is the number of channels after integration; Calculate the average eigenvalue s on each channel: Among them, F gp (.) Calculate the average value of the feature value on each channel, s i is the value on the i-th channel, and the feature value will be obtained through the fully connected layer z∈R r′ ; Input the average eigenvalue s into the fully connected layer to obtain the overall eigenvalue z; Where z∈R r′ ; F fc (.) activates and normalizes the feature values; δ is the RELU activation function, β is batch normalization, is the feature weight, r′ is the number of features obtained after the fully connected layer, and r is the number of features without passing through the linear layer; The soft attention mechanism is used to adaptively select channel connectivity information of different scales for the overall eigenvalue z: in, are the feature weights at three different scales, a d , b d , c d They are weights at three different scales; The weights a of the three different scales d , b d , c d Eigenvalues ​​for three different scales and in, is the weighted multi-scale feature.

8. The driver alertness estimation method based on hierarchical multi-scale topology enhancement network according to claim 7 is characterized in that: The driving fatigue classification result is output in S6, specifically: C class =SoftMax(WX′+b) X′=ResNet(U) Among them, C class is the output fatigue category, X′ is the output result after the residual ResNet module, and W is the feature weight of X′.

9. The driver alertness estimation method based on hierarchical multi-scale topology enhancement network according to claim 8 is characterized in that: The alertness regression prediction result is output in S6, specifically: C reg =Sigmoid(WX′+b) Among them, C reg The regression prediction results.

Images