Fatigue driving detection method based on electroencephalogram and myoelectricity multi-mode graph structure

Abstract

The invention discloses a fatigue driving detection method based on an electroencephalogram and myoelectricity multi-mode graph structure. According to the method, electroencephalogram signals and electromyographic signals of a driver are synchronously collected, psychological fatigue and physiological fatigue states are represented respectively, an electroencephalogram and electromyographic multi-mode graph structure is constructed from multiple perspectives of time domain, frequency domain and phase domain, and features under different perspectives are extracted by adopting a graph convolutional neural network. Meanwhile, a cross-view attention fusion mechanism and a shared subspace alignment learning strategy are introduced, and the fatigue driving state of the driver is predicted. According to the method, fatigue characteristics of psychological and physiological levels can be comprehensively captured, the recognition precision of recessive fatigue and early-stage fatigue is remarkably improved, and a more reliable technical support is provided for safety early warning in a complex driving environment. And by effectively modeling a dynamic interaction relationship between psychological fatigue and physiological fatigue, the accuracy and reliability of fatigue state recognition in a complex driving environment are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electro-digital data processing, and further relates to a fatigue driving detection method based on the structure of electroencephalogram (EEG) and electromyogram (EMG) multimodal graphs in the technical field of electro-digital data detection. The present invention constructs multi-perspective graph data in the time domain, frequency domain, and phase domain through the EEG signal and EMG signal of the driver, and combines graph neural networks (GNN) and cross-perspective fusion mechanisms to deeply explore the interaction relationship between mental fatigue and physical fatigue, so as to achieve high-precision recognition and analysis of the driver's fatigue state. Background Art

[0002] Fatigue driving has become one of the important causes of traffic accidents, posing a serious threat to social and personal safety. Especially for large passenger and freight vehicle drivers who drive continuously for a long time, or due to factors such as stress and lack of sleep in daily life, it is easy for drivers to experience mental and physical fatigue. Research has shown that mental fatigue is usually manifested as a decrease in attention, slow reaction, and weakened judgment. Physical fatigue is manifested as physical tiredness and muscle tension. These two types of fatigue driving not only increase the risk of traffic accidents but also seriously affect road traffic and the safe driving of vehicles.

[0003] Ming Shang Technology Co., Ltd. disclosed a fatigue driving detection technology based on physiological signals in its patent document "Intelligent Early Warning System for Fatigue Driving Based on Physiological Signal Monitoring Technology" (Patent Application No. CN202411390548.1, Publication No. CN118894119A). The implementation solution of this technology is to obtain the EEG signal, electrocardiogram signal, eye movement signal, and skin conductance signal of the driver through physiological signal sensors; preprocess and extract features from the collected multimodal physiological signals to obtain their respective physiological characteristic parameters; fuse the multimodal physiological characteristics through a weighted fusion algorithm, construct a personalized physiological model, and judge the driver's fatigue state; according to the fatigue recognition result, link the intelligent driving system to execute intervention measures such as early warning prompts and vehicle speed adjustment. The deficiencies of this technology are as follows: It only focuses on the driver's mental fatigue state, that is, it evaluates the decrease in attention and slow reaction through indicators such as EEG changes, electrocardiogram changes, eye movement characteristics, and skin conductance characteristics, lacking the detection of physical fatigue, which leads to missed detection of muscle fatigue-related risks (such as brake delay and decreased steering wheel control), thereby increasing the probability of traffic accidents caused by operational errors, especially in the scenario of long-term continuous driving.

[0004] Xidian University discloses a fatigue driving detection method in its patent document "Fatigue Driving Detection Method Based on Multimodal Information Fusion" (Patent Application No. CN202011408091.4, Publication No. CN112528815A). The method includes the following steps: (1) Synchronously collect the electroencephalogram signal, facial image and vehicle lane image of the driver; (2) Construct and train four convolutional neural networks for electroencephalogram signal classification, eye and mouth region localization, eye and mouth opening and closing state judgment, and lane line position localization respectively; (3) Use the trained network to preliminarily predict the fatigue state of the electroencephalogram signal, facial behavior characteristics and lane deviation; (4) Integrate the prediction results of the three modalities. If at least two are determined to be fatigued, it is finally determined that the driver is in a fatigued state. The deficiencies of this method are as follows: This method only focuses on the impact of physiological fatigue on driving actions. In fact, there is a complex dynamic interaction relationship between mental fatigue and physiological fatigue, and the two often affect and promote each other. However, the essential feature of the evolution of the fatigue state is that muscle fatigue will lead to a slowdown in driving action responses, thus exacerbating mental fatigue; while mental fatigue may cause improper operations, further accelerating the accumulation of physiological fatigue. The interaction between the two will greatly increase the probability of dangerous situations occurring during driving. Summary of the Invention

[0005] The purpose of the present invention is to propose a fatigue driving detection method based on the electroencephalomyogram multimodal graph structure for the deficiencies of the above-mentioned prior art, so as to solve two key problems existing in the prior art: one is to only detect mental fatigue and ignore physiological fatigue, resulting in the missed detection of operation risks caused by muscle fatigue; the other is to ignore the dynamic interaction between mental and physiological fatigue, making it difficult to accurately evaluate the composite fatigue state of the driver.

[0006] The idea to achieve the object of the present invention is as follows: The present invention simultaneously collects physiological data of two modalities, namely electroencephalogram (EEG) signals and electromyogram (EMG) signals of the driver, to achieve the collaborative detection of mental fatigue and physiological fatigue states, avoid the operation risks caused by missing the detection of physiological fatigue due to only focusing on mental fatigue, and thus improve the accuracy and robustness of the fatigue driving detection method. The present invention respectively adopts mutual information (MI), partial directed coherence (PDC) and phase locking value (PLV) analysis methods to construct a multi-view graph structure of EEG and EMG (Multi-view Graph Generation) from multiple perspectives of time domain, frequency domain and phase domain, effectively modeling the complex dynamic interaction relationship between mental fatigue and physiological fatigue, and avoiding the problem of decreased detection accuracy caused by ignoring the interaction. The present invention uses the training set generated by the multi-view graph structure of EEG and EMG to train a graph convolutional network (GCN). The GCN extracts features from each perspective of time domain, frequency domain and phase domain. After feature extraction, a multi-head attention mechanism is introduced to fuse the features from different perspectives; at the same time, in the multi-view feature fusion stage, a shared subspace alignment strategy is introduced to minimize the distribution difference of features from each perspective in the shared subspace, improve the consistency and comparability of cross-perspective features, enhance the adaptability of the system to modal changes in different driving scenarios, and ensure stable detection performance in complex driving environments.

[0007] The specific steps of the present invention are as follows:

[0008] Step 1: Synchronously collect the EEG signals and EMG signals of the driver, and preprocess the EEG signals;

[0009] Step 2: Construct multi-view graphs of EEG and EMG signals in the time domain, frequency domain and phase domain;

[0010] Step 3: Iteratively train the EEG-EMG fusion fatigue driving detection network model through the multi-view graph until the total loss function of the network model converges;

[0011] Step 4: Obtain the multi-view graph of the signal to be detected in the same way as in Step 1 and Step 2, and use the trained multi-view graph neural network of EEG-EMG fusion to detect fatigue driving. If the output is 0, it is determined that the driver is in an alert state; if the output is 1, it is determined that the driver is in a fatigued state.

[0012] Further, the driver's electroencephalogram (EEG) signal and electromyogram (EMG) signal refer to synchronously collecting at least 2,000 EEG data and EMG data to form a data set. The ratio of data in the alert state to that in the fatigue state in the data set is 1:1. The alert state is labeled as "0", and the fatigue state is labeled as "1". The number of channels of the EEG signal is 18, the number of channels of the EMG signal is 4, the sampling rate of both is 200 Hz, and the sampling time is 8 s.

[0013] Further, the preprocessing refers to using a band-pass filter with a passband of [0.5 Hz, 40 Hz] to filter the EEG signal, and then using independent component analysis to separate and remove the artifact components in the signal to suppress the noise and artifacts in the EEG signal.

[0014] Further, the steps of constructing multi-view graphs in the time domain, frequency domain, and phase domain of EEG and EMG signals are as follows:

[0015] In the first step, construct a time-domain graph. The nodes of the time-domain graph are the electrode channels of EEG and EMG. The node features are extracted based on the time-domain statistical features of EEG and EMG signals, including the mean, standard deviation, and maximum and minimum values. The mutual information of the connection strength between two adjacent nodes in the time-domain graph is used as the edge of the time-domain graph.

[0016] In the second step, construct a frequency-domain graph. The nodes of the frequency-domain graph are the electrode channels of EEG and EMG. The energies of the δ (1 - 4 Hz) band, θ (4 - 8 Hz) band, α (8 - 14 Hz) band, and β (14 - 31 Hz) band of EEG and EMG signals are used as node features. The partial directed coherence of the connection strength between two adjacent nodes in the frequency-domain graph is used as the edge of the frequency-domain graph.

[0017] In the third step, construct a phase graph. The nodes of the phase graph are the electrode channels of EEG and EMG. The node features are phase statistics, specifically including the phase standard deviation and the components of the average phase vector. The phase-locking value of the connection strength between two adjacent nodes in the phase graph is used as the edge of the phase graph.

[0018] The mutual information of the connection strength between two adjacent nodes is obtained by the following formula:

[0019]

[0020] where MI(X i , X j ) represents the connection strength calculated based on mutual information between the i-th node and the j-th node in the time-domain graph, and X i , X j represent the signal channels corresponding to the i-th node and the j-th node in the time-domain graph respectively, and p(x), p(y) represent the signal channels X i , X jThe marginal distribution probability, and p(x,y) represents the signal channel X i 、X j The joint distribution probability, and log represents the logarithmic operation with base 10.

[0021] The partial directed coherence of the connection strength between the pairwise adjacent nodes is obtained by the following formula:

[0022]

[0023] where PDC ij (f) represents the connection strength calculated based on the partial directed coherence between the i-th node and the j-th node at the f-th frequency in the frequency domain graph. The value range of f is [1Hz, 31Hz], and S ij (f) is the regression coefficient of the Multivariate Autoregressive (MVAR) model calculated based on the least squares method, representing the magnitude of the influence degree of the i-th node and the j-th node on the f-th frequency. N represents the total number of nodes in the graph, represents the sum of the influence degrees received from all nodes by the i-th node and the j-th node at the f-th frequency.

[0024] The phase locking value of the connection strength between the pairwise adjacent nodes is obtained by the following formula:

[0025]

[0026] where PLV ij represents the connection strength calculated based on the phase locking value between the i-th node and the j-th node in the time series of the phase diagram. |·| represents the modulus operation, respectively represent the phases of the i-th node and the j-th node at the t-th sampling moment. L represents the total length of the sampling time series, and e (·) represents the exponential function with the natural constant e as the base, and the superscript j represents the imaginary unit symbol.

[0027] The structure of the multi-view graph neural network for electroencephalogram and electromyogram fusion is, in sequence, an input layer, a multi-view layer, a fusion layer, and a fully connected layer; among them, the fusion layer uses a multi-head attention mechanism to calculate the weighted average between different views to obtain the fusion features between different views; the multi-view layer is composed of three graph convolutional neural networks with the same structure in parallel; each graph convolutional neural network is composed of a first graph convolutional layer, a first activation layer, a second graph convolutional layer, a second activation layer, and a pooling layer; both the first and second activation layers are implemented using the LeakyReLU activation function, and the Dropout value is 0.2 for both.

[0028] The calculation of the weighted average between different views refers to the graph-level time-domain feature F output by the multi-view layer in the multi-view graph neural network time , the graph-level frequency-domain feature F freq , the graph-level phase feature F phase , and the fused feature F is output after view fusion by the fusion layer fusion ; This fused feature F fusion is obtained by the following formula:

[0029] F fusion = concat(head1,head2,...,head n )W o

[0030]

[0031] where concat(·) represents the concatenation operation, and W o represents the projection matrix output by the multi-view graph neural network, which is a learnable weight matrix, and respectively represent learnable weight matrices, d k represents the feature dimension of each head, n represents the total number of heads of attention, the value range of n is [3,8], and the superscript T represents the transpose operation.

[0032] Furthermore, the total loss function Loss total of the network model is as follows:

[0033] Loss total = Loss cls + λLoss align

[0034]

[0035] where Loss cls represents the binary cross-entropy loss function between the true performance y and the fatigue driving prediction label , Loss align represents the alignment loss function, which is used to enhance the feature consistency between different views, λ represents the weight parameter, and its value range is [0,1), which is used to balance the contributions of the classification loss and the alignment loss in the total loss. S = {(time,freq),(freq,phase),(time,phase)} represents the pairwise combinations in the time domain, frequency domain, and phase domain, ‖‖·‖‖ * represents the nuclear norm operation, Z m represents the graph-level feature of the m-th view, and μ m represents the mean vector of the m-th view.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] First, by synchronously collecting the electroencephalogram (EEG) signals and electromyogram (EMG) signals of the driver, which respectively characterize the mental fatigue and physiological fatigue states, the present invention overcomes the limitation of the prior art that only relies on mental fatigue indicators for detection. This enables the present invention to comprehensively capture the fatigue characteristics at the mental and physiological levels, significantly improving the recognition accuracy of latent fatigue and early fatigue, and providing more reliable technical support for safety warning in complex driving environments.

[0038] Second, since the present invention constructs a multi-perspective graph structure in the time domain, frequency domain, and phase domain based on EEG and EMG signals, and uses a graph convolutional neural network to extract features from different perspectives, while introducing a cross-perspective attention fusion mechanism and a shared subspace alignment learning strategy, it overcomes the problems in the prior art of ignoring the complex interaction between mental fatigue and physiological fatigue, as well as the simple multi-modal fusion structure, resulting in insufficient feature representation ability and inaccurate capture of dynamic evolution laws. Therefore, by effectively modeling the dynamic interaction relationship between mental and physiological fatigue, the present invention significantly improves the accuracy and reliability of fatigue state recognition in complex driving environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of an embodiment of the present invention;

[0040] Figure 2 is a schematic structural diagram of a multi-perspective graph neural network for brain-muscle electrofusion of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0042] Referring to Figure 1 , the implementation steps of the embodiment of the present invention will be further described in detail.

[0043] Step 1, synchronously collect the EEG signals and EMG signals of the driver.

[0044] Synchronously collect the M EEG data and EMG data of multiple drivers to form a data set. The data ratio of the alert state and the fatigue state in the data set is 1:1. Label the alert state as "0" and the fatigue state as "1"; the number of channels C eeg of the EEG signal is 18, the number of channels C emg of the EMG signal is 4, the sampling rate is 200 Hz, and the sampling time is 8 s. The data shape of the EEG data is (M, C eeg , T), the data shape of the EMG data is (M, C emg , T), and each group of data contains 1600 sampling points. In the embodiment of the present invention, M = 5313, C eeg= 18, C emg = 4, T = 1600.

[0045] Step 2, preprocess the collected electroencephalogram (EEG) signals of the driver.

[0046] The main steps of preprocessing are band-pass filtering and artifact removal. According to the useful frequency band of the EEG signals, after filtering with a band-pass filter with a passband of [0.5 Hz, 40 Hz], the independent component analysis (ICA) is then used to separate and remove the artifact components. After preprocessing, the noise and artifacts in the originally collected EEG signals of the driver are significantly suppressed, and the useful EEG information is extracted.

[0047] Step 3, construct a multi-view graph structure.

[0048] In the multi-view graph, the EEG signals are used to represent mental fatigue, and the electromyogram (EMG) signals are used to represent physical fatigue. Therefore, the constructed multi-view graph structure can be regarded as modeling the interaction relationship between mental fatigue and physical fatigue.

[0049] Step 3.1, construct the time-domain graph G time .

[0050] Each node in the time-domain graph represents an EEG electrode channel or an EMG electrode channel. The node features are extracted based on the time-domain statistical features of EEG and EMG signals, including the mean, standard deviation, and maximum and minimum values. The mutual information of the connection strength between two adjacent nodes in the time-domain graph is used as the edge of the time-domain graph. In the embodiment of the present invention, there are 18 electrode channels for the EEG signals and 4 electrode channels for the EMG signals. Therefore, there are a total of 22 nodes in the time-domain graph.

[0051] The mutual information of the connection strength between two adjacent nodes is obtained by the following formula:

[0052]

[0053] where MI(X i , X j ) represents the connection strength calculated based on mutual information between the i-th node and the j-th node in the time-domain graph, X i , X j represent the signal channels corresponding to the i-th node and the j-th node in the time-domain graph respectively, p(x), p(y) represent the marginal distribution probabilities of the signal channels X i , X j , and p(x,y) represents the joint distribution probability of the signal channels X i , X j , and log represents the logarithmic operation with base 10.

[0054] Step 3.2, construct the frequency domain graph G freq .

[0055] Each node of the frequency domain graph represents an EEG electrode channel or an EMG electrode channel. The energy of the δ (1 - 4 Hz) frequency band, θ (4 - 8 Hz) frequency band, α (8 - 14 Hz) frequency band, and β (14 - 31 Hz) frequency band of the EEG and EMG signals is used as the node feature, and the partial directed coherence of the connection strength between two adjacent nodes in the frequency domain graph is used as the edge of the frequency domain graph. In the embodiment of the present invention, the EEG signal has 18 electrode channels and the EMG signal has 4 electrode channels, so there are 22 nodes in the frequency domain graph.

[0056] The mutual information of the connection strength between two adjacent nodes is obtained by the following formula:

[0057]

[0058] where, MI(X i , X j ) represents the connection strength calculated based on mutual information between the i-th node and the j-th node in the time domain graph, X i , X j respectively represent the signal channels corresponding to the i-th node and the j-th node in the time domain graph, p(x), p(y) represent the marginal distribution probabilities of the signal channels X i , X j , p(x,y) represents the joint distribution probability of the signal channels X i , X j , and log represents the logarithm operation with base 10.

[0059] Step 3.3, construct the phase graph G phase .

[0060] Each node of the phase graph represents an EEG electrode channel or an EMG electrode channel. The node feature is the phase statistic, specifically including the phase standard deviation and the components of the average phase vector, and the phase locking value of the connection strength between two adjacent nodes in the phase graph is used as the edge of the phase graph. In the embodiment of the present invention, the EEG signal has 18 electrode channels and the EMG signal has 4 electrode channels, so there are 22 nodes in the phase graph.

[0061] The phase locking value of the connection strength between two adjacent nodes is obtained by the following formula:

[0062]

[0063] where, PLV ij represents the connection strength calculated based on the phase locking value between the i-th node and the j-th node in the time series, |·| represents the modulus operation, respectively represent the phases of the $i$-th node and the $j$-th node at the $t$-th sampling moment, $L$ represents the total length of the sampling time series, and $e$ (·) represents the exponential function with the natural constant $e$ as the base, and the superscript $j$ represents the imaginary unit symbol.

[0064] Step 4, generate the training set and the test set.

[0065] Randomly divide the multi-view graph into the training set and the test set according to the ratio of 8:2. Use the time-domain graph constructed by EEG data and EMG data as the time-domain graph training set, use the frequency-domain graph constructed by EEG data and EMG data as the frequency-domain graph training set, and use the phase graph constructed by EEG data and EMG data as the phase graph training set.

[0066] Step 5, construct a multi-view graph neural network (Multi-view GCN) for brain-computer electroencephalogram fusion.

[0067] Refer to Figure 2 for a further description of the structure of the multi-view graph neural network for brain-computer electroencephalogram fusion.

[0068] Build a multi-view graph neural network for brain-computer electroencephalogram fusion. The structure of this network is, in sequence, the input layer, the multi-view layer, the fusion layer, and the fully connected layer.

[0069] To ensure the distribution consistency of the features of the three views in the shared subspace, project the features of each view and map them into a unified shared subspace. This strategy can unify the feature representations in the time domain, frequency domain, and phase domain, making the descriptions of the same data from different views more consistent.

[0070] The fusion layer uses the multi-head attention mechanism to calculate the weighted average between different views and obtain the fusion features between different views.

[0071] Calculating the weighted average between different views means that the graph-level time-domain feature $F$ time output by the multi-view layer in the multi-view graph neural network, the graph-level frequency-domain feature $F$ freq , the graph-level phase feature $F$ phase , and the fusion feature $F$ fusion output after view fusion by the fusion layer.

[0072] The said fusion feature $F$ fusion is obtained by the following formula:

[0073] $F$ fusion = concat(head1,head2,...,head n )$W$ o

[0074]

[0075] Among them, concat(·) represents the concatenation operation, and W o represents the projection matrix output by the multi-view graph neural network, which is a learnable weight matrix. and respectively represent learnable weight matrices, and d k represents the feature dimension of each head, n represents the total number of heads of the attention, n ∈ [3, 8]. In the embodiments of the present invention, n = 3.

[0076] Since the multi-view layer is composed of three graph convolutional neural networks with the same structure in parallel, therefore, the time-domain graph training set is input into the time-domain graph convolutional neural network to obtain time-domain graph-level features, the frequency-domain graph training set is input into the frequency-domain graph convolutional neural network to obtain frequency-domain graph-level features, and the phase graph training set is input into the phase graph convolutional neural network to obtain phase graph-level features.

[0077] Each graph convolutional neural network is composed of a first graph convolutional layer, a first activation layer, a second graph convolutional layer, a second activation layer, and a pooling layer; both the first and second activation layers are implemented using the LeakyReLU activation function, and the Dropout value is 0.2 for both.

[0078] Step 6, train the brain-computer electroencephalogram fusion fatigue driving detection network model.

[0079] Step 6.1, set the training parameters. Initialize the number of iterations as p, the maximum number of iterations as P, P ≥ 100, and the weight parameters of the current brain-computer electroencephalogram fusion fatigue driving detection network model as θ t , in the embodiments of the present invention, P = 100.

[0080] Step 6.2, input each perspective graph structure in the training set into the multi-view graph neural network of the brain-computer electroencephalogram fusion. Obtain graph-level features of three perspectives through the multi-view layer, fuse the graph-level features of the three perspectives, and then send the fused features into the fully connected network to obtain the fatigue driving prediction label

[0081] Step 6.3, calculate the loss Loss of the brain-computer electroencephalogram fusion fatigue driving detection network model through the true label y of each training sample and the fatigue driving prediction label total .

[0082] Step 6.4, use the stochastic gradient descent method to iteratively update the weight parameters θ of the network t until the maximum number of iterations P is reached or the total loss function Loss of the network model total converges, and obtain the trained brain-computer electroencephalogram fusion fatigue driving detection network model. ​

[0083] To constrain the distribution consistency of different views in the shared subspace, an MLP is used for projection. Assume h v represents the features of the v-th view, and after being transformed by the MLP, the representation z in the shared subspace is obtained v : z v = MLP(h v ).

[0084] The total loss function Loss of the network model total is as follows:

[0085] Loss total = Loss cls + λLoss align

[0086]

[0087] where Loss cls represents the binary cross-entropy loss function between the true performance y and the fatigue driving prediction label , Loss align represents the alignment loss function, whose role is to enhance the feature consistency between different views, λ represents the weight parameter, and its value range is [0, 1), which is used to balance the contributions of the classification loss and the alignment loss in the total loss. S = {(time, freq), (freq, phase), (time, phase)} represents the pairwise combinations in the time domain, frequency domain, and phase domain, ‖‖·‖| * represents the nuclear norm operation, Z m represents the graph-level features of the m-th perspective, μ m represents the mean vector of the m-th perspective. In the embodiment of the present invention, λ = 0.2.

[0088] Step 7, input the test set into the trained brain-muscle electrical fusion fatigue driving detection network model for forward propagation to obtain the fatigue driving classification results of each test sample. If the output is 0, it is determined that the driver is in an alert state. If the output is 1, it is determined that the driver is in a fatigued state.

[0089] To verify the effectiveness of the method of the present invention, experimental tests were carried out using the actually collected EEG data and EMG data. The data collection was based on a simulated driving experiment with 16 subjects, including 6 females, and the average age was 22.3 years. A total of 18 channels of EEG signals were recorded with a sampling rate of 200 Hz, and the sampling time for each segment was 8 seconds. The EEG channels included 18 electrodes, namely CP1, CPZ, CP2, P1, PZ, P2, PO3, POZ, PO4, O1, OZ, O2, FT7, FT8, T7, T8, TP7, and TP8, arranged according to the international 10–20 system. A total of 4 electrodes on the arms and calves were collected for EMG signals, and finally 5313 data samples were collected. Based on the proposed multi-view graph neural network model, the collected dataset was trained and tested to evaluate the performance of the model in the fatigue driving detection task. The accuracy and loss rate of the model on the test set were recorded in the experiment. Among them, the accuracy represents the matching degree between the model's predicted fatigue state label and the true label, reflecting the overall classification performance of the model. The loss rate represents the average error level between the model output and the true label, usually calculated by the loss function, and is used to measure the convergence degree and fitting effect of the model. The experimental results show that the accuracy of the method of the present invention on the test set is 88.43%, and the loss rate is 0.28, verifying the reliability and effectiveness of the proposed method in a complex driving environment.

Claims

1. A fatigue driving detection method based on the multimodal graph structure of electroencephalomyogram, characterized in that, Construct a multi-modal graph structure of brain-muscle electroencephalogram (EEG) from multiple perspectives in the time domain, frequency domain, and phase domain, a brain-muscle EEG fusion fatigue driving detection network model, and a cross-perspective fusion mechanism to deeply explore the interaction relationship between mental fatigue and physiological fatigue. The steps of this detection method are as follows: Step 1: Synchronously collect the driver's EEG signals and electromyogram (EMG) signals, and preprocess the EEG signals. Step 2: Construct multi-perspective graphs of EEG and EMG signals in the time domain, frequency domain, and phase domain. Step 3: Iteratively train the brain-muscle EEG fusion fatigue driving detection network model through the multi-perspective graphs until the total loss function of the network model converges. Step 4: Obtain the multi-perspective graphs of the signals to be detected in the same way as in Step 1 and Step 2, and use the trained multi-perspective graph neural network of brain-muscle EEG fusion to detect fatigue driving. If the output is 0, it is determined that the driver is in an alert state; if the output is 1, it is determined that the driver is in a fatigued state.

2. The fatigue driving detection method based on the brain-muscle multi-modal graph structure according to claim 1, wherein, The driver's EEG signals and EMG signals in Step 1 refer to synchronously collecting at least 2000 EEG data and EMG data to form a data set, where the data ratio of the alert state to the fatigued state in the data set is 1:

1. The alert state is labeled as "0", and the fatigued state is labeled as "1". The number of channels of the EEG signals is 18, the number of channels of the EMG signals is 4, the sampling rate of both is 200 Hz, and the sampling time is 8 s.

3. The fatigue driving detection method based on the brain-muscle multi-modal graph structure according to claim 1, wherein The preprocessing in Step 1 refers to filtering with a band-pass filter with a passband of [0.5 Hz, 40 Hz], and then using independent component analysis to separate and remove the artifact components in the signals to suppress the noise and artifacts in the EEG signals and EMG signals.

4. The fatigue driving detection method based on the brain-muscle electro-multimodal graph structure according to claim 1, characterized in that, The steps of constructing multi-perspective graphs of EEG and EMG signals in the time domain, frequency domain, and phase domain in Step 2 are as follows: The first step: Construct a time-domain graph. The nodes of the time-domain graph are the electrode channels of EEG and EMG. The node features are extracted based on the time-domain statistical features of EEG and EMG signals, including the mean, standard deviation, and maximum and minimum values. The mutual information of the connection strength between two adjacent nodes in the time-domain graph is used as the edge of the time-domain graph. The second step: Construct a frequency-domain graph. The nodes of the frequency-domain graph are the electrode channels of EEG and EMG. The energies of the δ (1 - 4 Hz) frequency band, θ (4 - 8 Hz) frequency band, α (8 - 14 Hz) frequency band, and β (14 - 31 Hz) frequency band of EEG and EMG signals are used as node features. The partial directed coherence of the connection strength between two adjacent nodes in the frequency-domain graph is used as the edge of the frequency-domain graph. The third step: Construct a phase graph. The nodes of the phase graph are the electrode channels of EEG and EMG. The node features are phase statistics, specifically including the phase standard deviation and the components of the average phase vector. The phase-locking value of the connection strength between two adjacent nodes in the phase graph is used as the edge of the phase graph.

5. The fatigue driving detection method based on the brain-muscle multi-modal graph structure according to claim 4, wherein The mutual information of the connection strength between two adjacent nodes is obtained by the following formula: where MI(X i , X j ) represents the connection strength calculated based on mutual information between the i-th node and the j-th node in the time-domain graph, X i , X j respectively represent the signal channels corresponding to the i-th node and the j-th node in the time-domain graph, p(x) and p(y) represent the marginal distribution probabilities of the signal channels X i , X j , and p(x, y) represents the joint distribution probability of the signal channels X i , X j . log represents the logarithmic operation with base 10.

6. The fatigue driving detection method based on the brain-muscle multi-modal graph structure according to claim 5, characterized in that, The partial directed coherence of the connection strength between two adjacent nodes is obtained by the following formula: Among them, PDC ij (f) represents the connection strength calculated based on partial directed coherence between the i-th node and the j-th node at the f-th frequency in the frequency domain graph. The value range of f is [1Hz, 31Hz], S ij (f) is the regression coefficient of the Multivariate Autoregressive (MVAR) model calculated based on the least squares method, representing the magnitude of the influence of the i-th node and the j-th node on the f-th frequency. N represents the total number of nodes in the graph, represents the sum of the influence degrees received by the i-th node and the j-th node from all nodes at the f-th frequency.

7. The fatigue driving detection method based on the brain-muscle multi-modal graph structure according to claim 6, wherein The phase-locking value of the connection strength between two adjacent nodes is obtained by the following formula: Among them, PLV ij represents the connection strength calculated based on the phase locking value between the \(i\)-th node and the \(j\)-th node in the phase diagram over the time series, and \(|\cdot|\) represents the modulus operation. respectively represent the phases of the \(i\)-th node and the \(j\)-th node at the \(t\)-th sampling moment, \(L\) represents the total length of the sampling time series, and \(e\) () represents the exponential function with the natural constant \(e\) as the base, and the superscript \(j\) represents the imaginary unit symbol.

8. The fatigue driving detection method based on the brain-muscle multi-modal graph structure according to claim 7, characterized in that, The structure of the multi-view graph neural network for brain-computer electroencephalogram fusion described in step 2 is, in sequence, an input layer, a multi-view graph layer, a fusion layer, and a fully connected layer; among them, the fusion layer uses a multi-head attention mechanism to calculate the weighted average between different views to obtain the fusion features between different views; the multi-view graph layer is composed of three graph convolutional neural networks with the same structure in parallel; each graph convolutional neural network is composed of a first graph convolutional layer, a first activation layer, a second graph convolutional layer, a second activation layer, and a pooling layer; both the first and second activation layers are implemented using the LeakyReLU activation function, and the Dropout value is 0.2 for both.

9. The fatigue driving detection method based on the brain-muscle multi-modal graph structure according to claim 8, characterized in that, The calculation of the weighted average between different views refers to the graph-level time-domain feature F output by the multi-view layer in the multi-view graph neural network time , the graph-level frequency-domain feature F freq , the graph-level phase feature F phase , and the fused feature F output after view fusion by the fusion layer fusion ; This fused feature F fusion is obtained by the following formula: F fusion = concat(head1, head2,..., head n )W o Among them, concat(·) represents the concatenation operation, and W o represents the projection matrix output by the multi-view graph neural network, which is a learnable weight matrix, and respectively represent learnable weight matrices, d k represents the feature dimension of each head, n represents the total number of heads of attention, the value range of n is [3, 8], and the superscript T represents the transpose operation.

10. The fatigue driving detection method based on the brain-muscle multi-modal graph structure according to claim 9, characterized in that, The total loss function Loss of the network model described in step 3 total is as follows: Loss total = Loss cls + λLoss align Among them, Loss cls represents the binary cross-entropy loss function between the true performance y and the fatigue driving prediction label . Loss align represents the alignment loss function, which is used to enhance the feature consistency between different views. λ represents the weight parameter, and its value range is [0, 1), which is used to balance the contributions of the classification loss and the alignment loss in the total loss. S = {(time, freq), (freq, phase), (time, phase)} represents the pairwise combinations in the time domain, frequency domain, and phase domain. ‖‖·‖‖ * represents the nuclear norm operation, and Z m represents the graph-level feature of the m-th perspective, and μ m represents the mean vector of the m-th perspective.

Images