Driving fatigue EEG signal detection method based on graph neural network

Abstract

The invention provides a driving fatigue EEG signal detection method based on a graph neural network, and the method comprises the steps: extracting an electroencephalogram signal adjacent graph, and employing a strategy for achieving data enhancement at a graph level to expand data on the basis of additionally introducing an adjacent connection coefficient K to reduce the time complexity, so as to reduce the interference on the graph structure features; then, a lightweight graph dual neural network is used for feature extraction and classification; then, reducing the scale of the model through a self-adaptive pruning strategy considering neurons and channels at the same time, so as to deploy the model to the Raspberry Pi;

Description

Technical Field

[0001] The present invention relates to technical fields such as drowsiness recognition, and in particular to a method for detecting driving fatigue EEG signals based on a graph neural network. Background Art

[0002] As the pace of modern society accelerates, drivers' driving time is also gradually increasing. However, long-term driving tasks are prone to driving fatigue, which causes drivers to lose the ability to make correct judgments and respond quickly and accurately. This is particularly dangerous in emergency situations. Studies have shown that fatigue driving is the cause of 20%-30% of traffic accidents. Therefore, it is crucial to prevent driver fatigue driving. Drowsiness recognition technology is a type of technology that recognizes biological characteristics based on physiological signal analysis.

[0003] Electroencephalogram (EEG) is widely used in non-invasive driver fatigue identification to reduce the risk of traffic accidents due to its high resolution and low cost. Most existing methods use traditional machine learning methods and deep neural network methods to achieve the recognition task based on the driver's sleepy EEG. However, traditional machine learning methods are tedious and time-consuming, and still have limitations in accuracy. Although deep neural network methods have been proven to be effective in feature learning in the field of time series data by a large number of studies, the methods proposed by some researchers to improve EEG processing capabilities have higher requirements in terms of network structure, which will lead to a significant increase in training time and memory usage. Summary of the invention

[0004] Therefore, in view of the defects and shortcomings of the prior art, the present invention proposes a lightweight enhanced graph convolutional neural network (LAGNN) for a single-channel EEG driver drowsiness recognition method. Specifically, the present invention considers the impact of neurophysiological knowledge on the drowsiness state for the first time, and uses wavelet transform to perform time-frequency conversion on EEG, and constructs a Baseline Drowsiness Status Adjacency Graph with interpretability and reliability. In addition, the time complexity is reduced by setting adjacency connectivity coefficients, and the Augmented Graph-level Module is used to enhance the expression ability of effective features. In order to overcome the computing limitations of edge devices, a lightweight enhanced graph convolutional neural network (LAGNN) is designed, which can identify drowsiness states on different drivers in real time. Finally, by adaptively pruning and optimizing the channels and neurons of the model, the delay is reduced in the inference stage while the recognition accuracy is improved.

[0005] The purpose of the present invention is to design a driving fatigue EEG signal detection method based on graph neural network and its deployment on the Raspberry Pi. The method can accurately perform time-frequency conversion on EEG signals, construct a Baseline Drowsiness StatusAdjacency Graph with interpretability and reliability, and adopt a lightweight enhanced graph convolutional neural network (LAGNN) and Augmented Graph-level Module, which is suitable for real-time classification of the driver's drowsiness state. By using lightweight networks and model pruning and optimization techniques, the present invention can efficiently identify the driver's drowsiness state on edge devices, providing support for real-time monitoring of the driver's mental state and preventing traffic accidents.

[0006] The present invention specifically adopts the following technical solutions:

[0007] A method for detecting driving fatigue EEG signals based on graph neural networks extracts the adjacency graph of EEG signals and expands the data by implementing data enhancement at the graph level based on the additional introduction of the adjacency connection coefficient K to reduce the time complexity, so as to reduce the interference on the graph structure features; then, a lightweight graph dual neural network is used for feature extraction and classification; then, the model size is reduced by an adaptive pruning strategy that considers both neurons and channels so that it can be deployed on the Raspberry Pi.

[0008] Furthermore, the specific implementation process of extracting the adjacency graph of the EEG signal and adopting the strategy of implementing data enhancement at the graph level on the basis of additionally introducing the adjacency connection coefficient K to reduce the time complexity is as follows:

[0009] After filtering and separating the EEG signal in the frequency domain, the power average of the Theta and Alpha bands is selected as the BDST to retain the key frequency domain features of the driver's drowsiness:

[0010]

[0011] In the process of constructing the adjacency graph BDSAG, the adjacency coefficient K is introduced to reduce the time complexity;

[0012] After obtaining the extracted interpretable adjacency graph, a graph-level enhancement module is used to perform data enhancement. Graph-level enhancement is achieved by using subgraphs to represent the global view or local view.

[0013] Among them, the global view is expressed as:

[0014]

[0015] Where f(·) represents the global view random sampling method, and R represents the ratio of retained graph-level features;

[0016] The partial view is represented as:

[0017]

[0018] Where r represents the ratio of retained graph-level features; BDSAG is combined with the global view and the local view, expressed as:

[0019]

[0020] in, It is represented as an enhanced graph; it is mapped to the latent space by introducing a nonlinear change function g(·) to improve data stability.

[0021] Furthermore, the EEG signal is converted into time-frequency by wavelet transform, and the EEG signal is filtered and separated in the frequency domain, including the δ, θ, α and β frequency bands.

[0022] Furthermore, in the process of constructing the adjacency graph BDSAG, the adjacency coefficient K is introduced to reduce the time complexity as follows:

[0023] When the BDST that retains the key frequency domain features of the driver's drowsiness state is calculated, it is necessary to find the feature adjacency relationship between different sampling points as much as possible and construct BDSAG: r Approaching BDST c When the value of is greater than , it indicates that the probability of the driver being drowsy is high; otherwise, the driver is awake; r and BDST c The weights between them are the same, and the constructed graph is undirected weighted symmetric. The calculation method is:

[0024]

[0025] Where r represents the index of the row, r∈[1,2,3,…,n], c represents the index of the column, c∈[1,2,3,…,n]; E r,c ,E c,r Represents x i and BDST c The weight relationship between them; K is the adjacency connection coefficient, which is used to specify the range of column calculation.

[0026] Furthermore, a single-channel EEG is used to construct an adjacency graph to express the information in non-Euclidean space, and a graph convolutional network is used to perform driver drowsiness recognition on edge devices.

[0027] Furthermore, the specific implementation process of extracting and classifying features using a lightweight graph dual neural network is as follows:

[0028] After enhancing the data at the graph level, the processed data is input into NodeAggregation for aggregation operation to extract the features of adjacent nodes and achieve feature dimensionality reduction. Next, depthwise separable convolution is used to perform pointwise convolution and depthwise convolution in sequence, and a two-stream convolution structure is used to transmit and interact information from multiple angles or different levels.

[0029] Then, a fusion layer is set to control the flow of features, selectively passing part of the input features to the next layer and keeping the other part of the features unchanged; then a Relu activation layer and a BatchNorm layer are used to prevent the gradient vanishing problem caused by the convolution operation; then a global average pooling layer is used; and finally, a fully connected layer is used.

[0030] Furthermore, the specific implementation process of reducing the model size by considering both neurons and channels for deployment on the Raspberry Pi is as follows:

[0031] Channel pruning is to filter the weights between channels. For unimportant channels, P cindex Indicates that the screening is calculated as follows:

[0032]

[0033] Among them, argminsort means sorting the importance of channels from small to large, w i It is expressed as the weight within the channel, channel represents the total number of channels at this level, and PR represents the pruning ratio; channel PR represents the number of pruned channels;

[0034] Neuron pruning is to screen the weights of neurons. For unimportant neurons, P nindex Indicates that the screening is calculated as follows:

[0035]

[0036] Among them, argminsort means sorting the importance of neurons according to the median method, using median sorting as the sorting method, neuron represents the total number of neurons in a channel, and pr represents the pruning ratio;

[0037] Next, the unimportant channel P cindex and unimportant neurons P nindex The weights are all set to 0, while other weights remain unchanged;

[0038] Then, the final pruning results are deployed to the edge Raspberry Pi.

[0039] And, a driving fatigue EEG signal detection system based on graph neural network is characterized in that: based on the Raspberry Pi end, the model obtained according to the method described above is deployed, and the driver's EEG data is collected and the driver's drowsiness recognition result is output after data analysis.

[0040] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the driving fatigue EEG signal detection method based on a graph neural network as described above are implemented.

[0041] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the driving fatigue EEG signal detection method based on a graph neural network as described above.

[0042] Compared with the prior art, the present invention and its preferred solution first propose an interpretable Baseline Drowsiness Status Adjacency Graph (BDSAG) by combining neurophysiological knowledge, which can effectively express the characteristics of driver drowsiness. Then, a lightweight graph dual-stream convolutional network (LGDCN) is designed using Augmented Graph-level Module, which enables it to include more effective features under limited data sets while improving recognition accuracy. Finally, the model trained under the PyTorch framework is converted into a C language embedded format and successfully deployed to the STM32 MCU. At the same time, the present invention also applies adaptive pruning optimization, which reduces the inference delay by about half, and successfully deploys the model to the edge of the fourth-generation Raspberry Pi. Experiments on benchmark data sets show that LGDCN has better trade-offs in performance and resources than the existing state-of-the-art algorithms. The present invention can be widely used in vehicle-mounted anti-fatigue systems, smart bracelets, smart clothing and other equipment, which is of great significance to ensuring driving safety and preventing cardiovascular diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0044] Figure 1 It is a conceptual diagram of an embodiment of the present invention.

[0045] Figure 2 The figure is a comparison diagram of the process framework of the method according to the embodiment of the present invention and the traditional method.

[0046] Figure 3Schematic diagram of the calculation process of the mathematical variables max and BDST in the embodiment of the present invention. Wherein: (a) without taking into account the adjacent connectivity coefficient K, the time complexity is O(n 2 ). (b) When the adjacency connectivity coefficient K = 8, the time complexity is reduced to O(n).

[0047] Figure 4 This is a block diagram of the adaptive pruning optimization of an embodiment of the present invention. Adaptive pruning is performed from two aspects, channels and neurons of the model feature graph, to find the optimal answer for model optimization.

[0048] Figure 5 A block diagram of edge deployment according to an embodiment of the present invention.

[0049] Figure 6 To extract the number of drowsiness and wakefulness samples from different subjects using the data set in the embodiment of the present invention.

[0050] Figure 7 The drowsiness recognition accuracy, precision and specificity of 11 different drivers selected for the embodiment of the present invention.

[0051] Figure 8 Comparison of the proposed model with the state-of-the-art methods in terms of performance and resources.

[0052] Fig. 9 Performance of different data enhancement modules in classification.

[0053] Fig.10 Comparison of different pruning methods.

[0054] Fig.11 Schematic diagram of ablation study of key components of BDSAG according to an embodiment of the present invention.

[0055] Fig.12 Ablation study of key components of LGDCN.

[0056] Fig.13 Schematic diagram of optimization of convolution channel pruning and neuron pruning respectively according to an embodiment of the present invention.

[0057] Fig.14 A schematic diagram of combined optimization of channel pruning and neuron pruning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail with the accompanying drawings as follows:

[0059] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0060] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0061] Before describing, the relevant parameters of the present invention are conceptually defined:

[0062] Definition 1: (Time domain of EEG signal): Single channel EEG signal z = {z1,z2,z3,…,z n}, where z i is the i-th sampling point of the EEG signal, and n is the total number of sampling points contained in the EEG signal.

[0063] Definition 2: (frequency domain of EEG signal): For EEG signal x = {x1, x2, x3, ..., x n}, where x i is the frequency domain signal of the i-th element corresponding to z. The frequency domain signal x is calculated as:

[0064]

[0065] Definition 3: (Graph representation of EEG signals): The graph is defined as G = (V, ε). Where V represents the set of nodes in the graph, V = {v1, v2, v3, …, v n}, each node v i Corresponding to the EEG sampling point x i In addition, ε represents the edge set of the graph, and the expression is as follows:

[0066]

[0067] Each edge E i,j The characteristic adjacency relationship between corresponding nodes is also emphasized by the present invention.

[0068] Figure 1 An adjacency graph is constructed for the single-channel EEG (Oz) used in the present invention to express its non-Euclidean space information, and a graph convolutional network is used to perform driver drowsiness recognition on edge devices.

[0069] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0070] like Figure 2 As shown, first, the EEG signal is filtered and separated in the frequency domain, including the δ, θ, α and β frequency bands. Among them, the power of the Theta band and the Alpha band increases significantly when the driver is in a drowsy state. Therefore, the present invention selects the power average of the Theta and Alpha bands as the Baseline Drowsiness Status Tensor (BDST):

[0071]

[0072] When calculating the BDST that retains the key frequency domain features of the driver's drowsiness state, it is necessary to find the feature adjacency relationship between different sampling points as much as possible and construct the Baseline Drowsiness Status Adjacency Graph (BDSAG). Existing methods usually have great deficiencies in the interpretability of weights and high computational complexity. The present invention focuses on the differences in sampling points and drowsiness features: when the frequency domain signal x of the EEG is r Approaching BDST C When the value of is greater than , it indicates that the probability of the driver being drowsy is high; otherwise, the driver may be awake. r and BDST C The weights between them are the same, so the constructed graph is undirected weighted symmetric, and the calculation method is as follows:

[0073]

[0074] r represents the index of the row, r∈[1,2,3,…,n], c represents the index of the column, c∈[1,2,3,…,n]. In addition, E r,c ,E c,r Represents x i and BDST C In addition, the present invention introduces an additional adjacency connection coefficient K to specify the range of column calculation, which is no longer limited to the total number of sampling points n, so as to reduce the time complexity. In the research of the present invention, through repeated experiments, it is determined that the range of K is between [5-10], which can reduce the time complexity to O(n). Figure 3 The above formula is modified as follows:

[0075]

[0076] In the case of limited electroencephalogram (EEG) datasets, many methods usually augment the data by implementing data augmentation on the time series (one-dimensional direction) of EEG signals. However, this approach may cause serious interference to the graph structure characteristics of the proposed method. In addition, existing studies have shown that sampling a subset of graphs can benefit downstream tasks in different categories of graph datasets. [1] Therefore, the present invention pays more attention to the adjacency graph itself and adopts a strategy to achieve data enhancement at the graph level. By considering the frequency domain characteristics of the EEG signal, while constructing an interpretable Baseline DrowsinessStatusAdjacency Graph (BDSAG), the adjacency connection coefficient K is introduced, which effectively reduces the time complexity of graph construction and provides an efficient data foundation for subsequent graph convolutional network learning.

[0077] The present invention adopts a strategy of realizing data enhancement at the graph level, using a larger subgraph to represent the global view and a smaller subgraph to represent the local view to realize graph-level enhancement. The global view is represented as:

[0078]

[0079] Among them, f(·) represents the random sampling method of the global view, and R represents the ratio of the retained graph-level features. Through a similar process, the representation of the local view can be obtained as:

[0080]

[0081] Among them, r represents the ratio of the retained graph-level features. Therefore, BDSAG can be combined with the global view and the local view, which can be expressed as:

[0082]

[0083] in, Represented as an enhancement graph. The introduction of nonlinear variation function g(·) mapped to the latent space can improve data stability and enhance the model’s ability to express complex EEG features.

[0084] Secondly, the present invention takes into account that the features contained in the one-dimensional information graph structure are richer and more complete, which is obvious. Therefore, in the process of feature learning, it must be recognized that the computational complexity of the graph convolutional network (GCN) is an important factor that cannot be ignored. Especially in resource-constrained edge devices, the computational complexity of the network needs to be reduced as much as possible to ensure the efficiency and deployability of the model. Therefore, the present invention proposes a Lightweight Graph Dual Convolution Network (LGDCN), see Figure 2(c). First, for the input enhanced graph-level module, an aggregation operation is performed on each node to extract the features of adjacent nodes and achieve feature dimensionality reduction. This step is considered to be one of the key steps of graph convolution. Then, according to Fran et al. [2] It shows that the depthwise separable convolution is superior to the traditional convolution in computational complexity. More importantly, the depthwise separable convolution shows obvious advantages in ensuring the integrity of features when they interact, which has a natural advantage in efficiently extracting graph structure features. Specifically, the depthwise separable convolution consists of two key steps: pointwise convolution and depthwise convolution. Pointwise convolution does not involve the mixing of spatial information when performing linear transformations between channels. Therefore, although the relationship between channels will change, it will not affect the spatial structure within the channel. The depthwise convolution is processed in an independent spatial region, and the positional relationship of elements within the channel remains unchanged, thus maintaining the spatial structure within the channel. This enables the network to learn a more accurate expression of graph features. Importantly, the order of operations of the present invention is different from that of the classic depthwise separable convolution. [2] . Since point-wise convolution is performed first, the convolution operation on the channel dimension can be decomposed into separate convolutions on multiple channels, thereby reducing the computational cost of deep convolution, which is very beneficial for edge terminals or resource-constrained scenarios. In addition, the model of the present invention adopts a dual-stream convolution structure to transmit and interact information from multiple angles or different levels, which enhances the robustness of the model to a certain extent. Subsequently, a fusion layer is set. This embodiment refers to a method called partial convolution proposed in existing literature. [3] . Partial convolution allows the flow of features to be controlled, and a portion of the input features can be selectively passed to the next layer while keeping the other portion unchanged. It not only reduces the propagation of irrelevant information, but also makes the network more compact. The use of Relu activation layer and BatchNorm layer is to prevent the gradient vanishing problem caused by convolution operations. Next, the use of global average pooling layer greatly reduces the parameters. Finally, in terms of the setting of the fully connected layer, this helps to enhance the nonlinear ability of the model, enabling the model to better capture the semantic features of the data, thereby improving the accuracy of the classification task.

[0085] In addition, in order to efficiently deploy on the edge, the present invention proposes an adaptive pruning optimization strategy. This optimization not only improves the accuracy of the model, but also significantly reduces the inference latency. Therefore, this adaptive pruning optimization strategy will be introduced in detail below, and the technical solution for edge device deployment will be discussed.

[0086] At present, model pruning has become one of the main optimization strategies to solve the problem of over-parameterization of network models, and the parameter scale selection of the network model directly affects the computational complexity and has a significant impact on the final effect of the model. The present invention observes a phenomenon: when the number of output channels of the convolutional layer is adjusted to 4 or 8, or the number of output neurons of the fully connected layer is set to 128 or 256, the final recognition accuracy fluctuates. Therefore, the present invention speculates that there may be an optimal parameter configuration between the range of the number of channels [4-8] and the range of the number of neurons [128-256], which can both reduce the inference time and maintain a high prediction accuracy. Fig.10 This proves that the idea is correct. Therefore, the adaptive pruning optimization proposed in this invention solves this problem from two aspects. The first is channel pruning. Since the full connection only needs one channel, it only targets the convolutional layer. Specifically, the weights between channels are screened, and P is used for unimportant channels. cindex Indicates that the screening is calculated as follows:

[0087]

[0088] argminsort means sorting the channels from small to large in importance, w i It is expressed as the weight within the channel, channel represents the total number of channels in this layer, and PR represents the pruning ratio. Therefore, channel PR represents the number of pruned channels. The second is neuron pruning. Similarly, the weights of neurons are still screened, and P is used for unimportant neurons. nindex Indicates that the screening is calculated as follows:

[0089]

[0090] Argminsort means sorting the importance of neurons by median. Considering that there is a lot of redundant information in the fully connected layer, median sorting is selected as the sorting method to discard unimportant features to the greatest extent, thereby improving the expressiveness of the features. Therefore, for unimportant channels P cindex and unimportant neurons P nindex , reset their weights to 0, while other weights remain unchanged. This optimization method makes the model sparse, thereby reducing the computational delay time. Among them, neuron represents the total number of neurons in a channel, and pr represents the pruning ratio. Figure 4 This is the adaptive pruning optimization block diagram. Adaptive pruning is performed on the channels and neurons of the model feature graph to find the optimal answer for model optimization.

[0091] like Figure 5As shown in the figure, the model is deployed in the inference phase. Specifically, there are two types of inputs. One is to perform adaptive pruning optimization on the trained model to reduce the model running delay. The other is the driver's EEG test data set. They are stored in the SD Card through the reading and writing device and transmitted to the edge through the USB wired device for data analysis. Finally, the display is connected through the HDMI interface to output the driver's drowsiness recognition results. It is worth noting that Figure 5 The D in the letter "A" stands for Drowsiness, and the A stands for Alert.

[0092] The present invention is evaluated on a public dataset of different drivers’ drowsiness recognition [4] , the participating drivers performed a continuous driving task in a virtual reality simulator, and the EEG signals were sampled at a sampling frequency of 500Hz using 30 electrodes. The dataset contains 2022 samples from 11 independent subjects, including 1011 sleepy state samples and 1011 awake state samples. The number of samples for each subject can be found in Figure 6 The size of each sample is 30 (channels) × 384 (sample points). Previous studies have shown that a single channel 'Oz', which is most related to the sleepy state, can be selected for sleepiness recognition. [4][5] .

[0093] Next, various aspects of the evaluation, description of the training and deployment details, and comparison of the method of the present invention with the method of the most advanced model are respectively made below:

[0094] First, the following two points include details of the model's evaluation in terms of performance and resources:

[0095] 1. Performance evaluation: Five performance evaluation indicators are used: Accuracy (Acc), Precision (Pre.), Specificity (Spe.), Recall (Rec.) and F1 Score (F1.) [6] . For the task of the present invention, accuracy reflects the ability of the model to correctly judge the driver's state, including sleepiness and wakefulness. Precision measures the accuracy of the model in judging the driver as sleepy, and recall evaluates the model's ability to recognize actual sleepy drivers. Specificity measures the degree to which the model correctly recognizes actual awake drivers. The F1 score takes into account both precision and recall, and is an indicator for comprehensively evaluating model performance.

[0096] 2. Resource evaluation: Five resource evaluation indicators are used: Footprint, MFLOPs, Latency, Power Dissipation and Energy Consumption. Among them, Footprint (occupied space) measures the size of data or programs in memory, MFLOPs (million floating point operations per second) reflects the floating point operation speed of the system, Latency (latency) refers to the time required to complete the task, Power Dissipation (power consumption) indicates the amount of electrical energy consumed by the system, and Energy Consumption (energy consumption) considers the total energy consumption of the system in a specific time period. These indicators are used to comprehensively evaluate the performance of the model in the driver drowsiness and wakefulness recognition tasks, and then guide the trade-off between performance and resources.

[0097] Next, more specific implementation details are described:

[0098] 1. Training details: The model of the present invention is trained on a single GPU, Nvidia GTX 3060 12GB, and compiled using the PyTorch framework. Specifically, the following training configuration is used: the maximum epoch of the model is set to 30 times, the model is trained using the Adam optimizer, the learning rate is 0.0015, the batch size is 32, and the Negative Log-Likelihood (NLL) is used as the loss function.

[0099] 2. Deployment details: The purpose of this invention is to perform real-time driver drowsiness recognition on edge devices, and Raspberry Pi 4Model B (the fourth generation Raspberry Pi) can well meet the requirements of this invention. The fourth generation Raspberry Pi (4GB RAM, 16GB Flash) has a large and active community, which is very helpful for researchers to replicate the code of this method. In addition, it is a popular choice for edge devices and is often used for local offline development tasks. In order to achieve edge deployment, the Linux-based Raspberry Pi OS and necessary dependencies were installed, and the inference phase of the model was successfully run on the fourth generation Raspberry Pi.

[0100] Finally, the comparison between the proposed method and the method of the most advanced model is as follows:

[0101] The proposed method is compared with state-of-the-art models, including EEG-ARNN [7] 、FBMSNet [8] LGGNet [9]SSGCNet

[10] 、CAGNN

[11] and TSception

[12] . Figure 7 The accuracy, precision and specificity of drowsiness recognition on 11 different drivers are shown. These data intuitively illustrate that the model of the present invention is robust among different individuals and does not fluctuate significantly due to individual differences. It is encouraging that the present invention performs well in terms of performance and resources, see Figure 8 . Compared with the SSGCNet model, it shows significant advantages in average performance, which is specifically reflected in an increase of 5.79% in accuracy, 3.37% in precision, and 3.14% in specificity. Compared with the TSception model, the method of the present invention improves the recall rate by 3.87% and the F1Score by 9.69%. In terms of resource utilization, compared with the second-ranked model, the Mflops is reduced by 1.45, the latency is reduced by 4.68ms, the power is reduced by 0.20W, and the energy is reduced by 58.80mJ. This clearly demonstrates the effectiveness of the lightweight model of the present invention. Although the performance in terms of footprint is not ideal, this is mainly due to the amount of parameters contained in the graph features of the present invention. However, with the continuous advancement of storage capacity technology, 81.24kb is much smaller than 1Mb, and this impact can be ignored.

[0102] In addition, the performance of different data augmentation modules in classification is evaluated. Fig. 9 As shown in Figure 1, the enhanced graph-level module of the present invention is effective on the basis of the original module. However, other common data enhancement methods are not suitable for the BDSAG constructed by the present invention because they damage the feature information, resulting in large errors in feature interaction, which generally reduces the performance indicators. In order to further verify the contribution of the pruning method in the present invention, a comparison is made with other popular pruning methods, see Fig.10 . It can be found that other pruning methods have a certain degree of decline in performance indicators. However, compared with the original module, the pruning method of the present invention has achieved improvements in accuracy, precision and specificity. It is worth noting that the pruning method of the present invention also significantly speeds up the testing time of the model, reducing the time by half (about 44.44 milliseconds). Compared with the ADMIM method, APO improved the accuracy by 0.9%, the precision by 2.49%, and the specificity by 3.45%. Not only are there more pruning parameters, but the delay is also reduced by 5.31ms. This shows that the Adaptive Pruning Optimization of the present invention can achieve lossless pruning and effectively reduce the consumption of test delays.

[0103] Finally, an ablation study was performed to prove that the proposed BDSAG is not only interpretable but also has a significant impact on the performance. Fig.11 The results of the ablation study are presented in . By extracting EEG features in different frequency bands and using these features to construct a new BDSAG, it is easy to find that the BDSAG constructed using frequency bands unrelated to sleepiness (such as δ and β) shows a decrease in all five performance indicators. This is because the δ and β frequency bands appear less frequently when the driver is in a sleepy state, resulting in insufficient feature expression of the constructed BDSAG, which affects the analysis results of the LGDCN model. Therefore, this result once again verifies the results of Lal et al.

[13] 's point of view and highlights the reliability of the BDSAG constructed by the present invention.

[0104] exist Fig.12 The ablation results are shown in the figure to measure the effectiveness of each layer in the LDGCN model. (1) In method '1-2', the impact of only a single layer of graph convolution on the performance is evaluated, including a single layer of pointwise convolution and depthwise convolution (PDwise), and a single layer of fully connected layer (FC). It can be observed that the accuracy is reduced by 2.17% and 3.64% respectively compared with the complete method. This shows that the two-layer convolution structure is valuable in feature learning. (2) In method '3-4', the impact of pointwise convolution and depthwise convolution on monitoring is evaluated respectively. From the performance index results, it can be seen that the learning ability of pointwise convolution between channels and depthwise convolution within channels are both effective, indicating that the graph features of the present invention still have strong integrity after the convolution operation, which is conducive to the expression ability of the features. (3) In method '5', the effectiveness of PartialConv in the fusion layer is evaluated. Partial Conv can not only selectively control the flow of information to make the network more compact, but also show the effect of feature transfer and interaction. In addition, channels and neurons are pruned and optimized using different ratios, see Fig.13 . The observation results show that the model's accuracy is highest when only 10% of the channels are pruned and 30% of the neurons are pruned. Therefore, it is speculated that the accuracy may be further improved by 10% of the channels and 30% of the neurons at the same time. Therefore, the following combined pruning experiments are added. For specific results, see Fig.14 It is worth noting that the experimental results fully prove that the conjecture is correct. When 10% channel pruning and 30% neuron pruning are applied simultaneously, the accuracy of the model reaches the highest level, which is better than other combined pruning methods. This result shows the effectiveness of the adaptive pruning optimization strategy proposed in this invention.

[0105] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

[0106] References:

[0107] [1]Yi Ding et al. "Lggnet: Learning from local-global graphrepresentations for brain–computer interface". In: IEEE Transactions on NeuralNetworks and Learning Systems (2023).

[0108] [2] Chollet. "Xception: Deep learning with depthwise separableconvolutions". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 1251–1258.

[0109] [3]Jierun Chen et al. "Run, Don't walk: Chasing higher FLOPS for faster neural networks". In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 12021–12031.

[0110] [4]Jian Cui et al. "A compact and interpretable convolutional neural network for cross-subject driver drowsiness detection from single-channelEEG".

[0111] In: Methods 202(2022), pp.173–184.

[0112] [5]Nikhil R Pal et al.“EEG-based subject-and session-independentdrowsiness detection:an unsupervised approach”.In:EURASIP Journal on Advancesin Signal Processing 2008(2008),pp.1–11.

[0113] [6]Jian Cui et al.“EEG-based cross-subject driver drowsinessrecognition with an interpretable convolutional neural network”.In:IEEETransactions on Neural Networks and Learning Systems(2022).

[0114] [7]Biao Sun et al.“Graph convolution neural network based end-to-endchannel selection and classification for motor imagerybrain-computerinterfaces”.

[0115] In:IEEE transactions on industrial informatics(2022).

[0116] [8]Ke Liu et al.“FBMSNet:A filter-bank multi-scale convolutionalneural network for EEG-based motor imagery decoding”.In:IEEE Transactions onBiomedical Engineering 70.2(2022),pp.436–445.

[0117] [9]Yi Ding et al.“Lggnet:Learning from local-global-graphrepresentations for brain–computer interface”.In:IEEE Transactions on NeuralNetworks and Learning Systems(2023).

[0118]

[10] Jialin Wang et al.“Ssgcnet:a sparse spectra graph convolutionalnetwork for epileptic eeg signal classification”.In:IEEE Transactions onNeural Networks and Learning Systems(2023).

[0119]

[11] Zhuoli Zhuang et al.“A Connectivity-Aware Graph Neural Networkfor Real-Time Drowsiness Classification”.In:IEEE Transactions on NeuralSystems and Rehabilitation Engineering(2023).

[0120]

[12] Yi Ding et al.“Tsception:Capturing temporal dynamics and spatialasymmetry from eeg for emotion recognition”.In:IEEE Transactions on AffectiveComputing(2022).

[0121]

[13] Saroj KL Lal and Ashley Craig.“Driver fatigue:

[0122] electroencephalography and psychological assessment”.In:Psychophysiology

[0123] 39.3(2002),pp.313–321.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of driving fatigue EEG signal detection methods based on graph neural networks under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A method for detecting driving fatigue EEG signals based on graph neural network, characterized in that: By extracting the adjacency graph of EEG signals and introducing an additional adjacency connection coefficient K to reduce the time complexity, a data augmentation strategy is adopted at the graph level to expand the data in order to reduce the interference on the graph structure features. Then, a lightweight graph dual neural network is used for feature extraction and classification. Then, an adaptive pruning strategy that considers both neurons and channels is used to reduce the size of the model so that it can be deployed on the Raspberry Pi.

2. The method for detecting driving fatigue EEG signals based on graph neural network according to claim 1, characterized in that: The specific implementation process of extracting the adjacency graph of the EEG signal and expanding the data by adopting the strategy of data enhancement at the graph level based on the additional introduction of the adjacency connection coefficient K to reduce the time complexity is as follows: After filtering and separating the EEG signal in the frequency domain, the power average of the Theta and Alpha bands is selected as the BDST to retain the key frequency domain features of the driver's drowsiness: In the process of constructing the adjacency graph BDSAG, the adjacency coefficient K is introduced to reduce the time complexity; After obtaining the extracted interpretable adjacency graph, a graph-level enhancement module is used to perform data enhancement. Graph-level enhancement is achieved by using subgraphs to represent the global view or local view. Among them, the global view is expressed as: Where f(·) represents the global view random sampling method, and R represents the ratio of retained graph-level features; The partial view is represented as: Where r represents the ratio of retained graph-level features; BDSAG is combined with the global view and the local view, expressed as: in, It is represented as an enhanced graph; it is mapped to the latent space by introducing a nonlinear change function g(·) to improve data stability.

3. The method for detecting driving fatigue EEG signals based on graph neural network according to claim 2, characterized in that: Wavelet transform is used to perform time-frequency conversion on the EEG signal, and the EEG signal is filtered and separated in the frequency domain, including the δ, θ, α and β frequency bands.

4. The method for detecting driving fatigue EEG signals based on graph neural network according to claim 2, characterized in that: In the process of constructing the adjacency graph BDSAG, the adjacency coefficient K is introduced to reduce the time complexity as follows: When the BDST that retains the key frequency domain features of the driver's drowsiness state is calculated, it is necessary to find the feature adjacency relationship between different sampling points as much as possible and construct BDSAG: r Approaching BDST c When the value of is greater than , it indicates that the probability of the driver being drowsy is high; otherwise, the driver is awake; r and BDST c The weights between them are the same, and the constructed graph is undirected weighted symmetric. The calculation method is: Where r represents the index of the row, r∈[1,2,3,…,n], c represents the index of the column, c∈[1,2,3,…,n]; E r,c ,E c,r Represents x i and BDST c The weight relationship between them; K is the adjacency connection coefficient, which is used to specify the range of column calculation.

5. The method for detecting driving fatigue EEG signals based on graph neural network according to claim 3 is characterized in that: An adjacency graph is constructed using single-channel EEG to express the information of non-Euclidean space, and a graph convolutional network is used to perform driver drowsiness recognition on edge devices.

6. The method for detecting driving fatigue EEG signals based on graph neural network according to claim 1, characterized in that: The specific implementation process of extracting and classifying features using a lightweight graph dual neural network is as follows: After enhancing the data at the graph level, the processed data is input into NodeAggregation for aggregation operation to extract the features of adjacent nodes and achieve feature dimensionality reduction. Next, depthwise separable convolution is used to perform pointwise convolution and depthwise convolution in sequence, and a two-stream convolution structure is used to transmit and interact information from multiple angles or different levels. Then, a fusion layer is set to control the flow of features, selectively passing part of the input features to the next layer and keeping the other part of the features unchanged; then a Relu activation layer and a BatchNorm layer are used to prevent the gradient vanishing problem caused by the convolution operation; then a global average pooling layer is used; and finally, a fully connected layer is used.

7. The method for detecting driving fatigue EEG signals based on graph neural network according to claim 1, characterized in that: The specific implementation process of reducing the model size by taking into account both neurons and channels so as to deploy it on the Raspberry Pi is as follows: Channel pruning is to filter the weights between channels. For unimportant channels, P cindex Indicates that the screening is calculated as follows: Among them, argminsort means sorting the importance of channels from small to large, w i It is expressed as the weight within the channel, channel represents the total number of channels at this level, and PR represents the pruning ratio; channel PR represents the number of pruned channels; Neuron pruning is to screen the weights of neurons. For unimportant neurons, P nindex Indicates that the screening is calculated as follows: Among them, argminsort means sorting the importance of neurons according to the median method, using median sorting as the sorting method, neuron represents the total number of neurons in a channel, and pr represents the pruning ratio; Next, the unimportant channel P cindex and unimportant neurons P nindex The weights are all set to 0, while other weights remain unchanged; Then, the final pruning results are deployed to the edge Raspberry Pi.

8. A driving fatigue EEG signal detection system based on graph neural network, characterized by: Based on the Raspberry Pi, the model obtained according to any of the methods described in claims 1-7 is deployed to collect the driver's EEG data and output the driver's drowsiness recognition result after data analysis.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the driving fatigue EEG signal detection method based on graph neural network as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the driving fatigue EEG signal detection method based on graph neural network as described in any one of claims 1 to 7 are implemented.

Images