Multi-modal lightweight epilepsy electroencephalogram signal classification model training method, establishment method and classification method
Through the deep learning model combining adaptive decoupling knowledge distillation and multimodal feature fusion technology, EEG signals are processed, and the problems of insufficient accuracy of epilepsy detection and high model complexity in the existing technology are solved, and the resource-limited environment is realized Medium and efficient multimodal epilepsy detection.
Patent Information
- Application Number
- CN202410500883.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-24
- Publication Date
- 2024-11-19
AI Technical Summary
The existing epilepsy detection technology is insufficient in processing EEG signals and has high model complexity, making it difficult to effectively apply in resource-limited environments, and single modal analysis cannot fully utilize multimodal information, limiting detection performance.
Using a multimodal lightweight epilepsy detection model based on deep learning, combining adaptive decoupled knowledge distillation (ADKD) and multimodal feature fusion technology, time-domain and frequency-domain feature analysis is used using short-time Fourier transform (STFT) , through one-dimensional and two-dimensional feature extraction modules, global consistency modules and local co-occurrence modules, to improve the accuracy and efficiency of the model.
It realizes the accuracy and efficiency of epilepsy detection in resource-limited environments, and can better utilize multimodal EEG signal information, reduce computing resource requirements, and improve the generalization performance and robustness of the model.
Smart Images

Figure QLYQS_1 
Figure BDA0004808638060000041 
Figure BDA0004808638060000042
Abstract
Description
Technical Field
[0001] The present invention relates to a lightweight multimodal epilepsy detection model based on deep learning and multimodal analysis. In particular, it relates to a method for analyzing the time domain and frequency domain features of electroencephalogram (EEG) signals using short-time Fourier transform (STFT), and combining adaptive decoupled knowledge distillation (ADKD) and multimodal feature fusion technology to improve the accuracy and efficiency of epilepsy detection. Background Art
[0002] Epilepsy is one of the most common neurological diseases in the world. According to the World Health Organization, about 60 million people are affected by epilepsy worldwide. The main feature of epilepsy is repeated and unexpected seizures, which are caused by sudden abnormal electrical activity in the brain. Due to the suddenness and unpredictability of epileptic seizures, this has a significant impact on patients' daily life and mental health.
[0003] Over the past few decades, electroencephalography (EEG) has become an important tool for diagnosing and monitoring epilepsy. EEG signals reflect complex patterns of electrical activity in the brain and can help identify specific patterns associated with epileptic seizures. However, due to the high nonlinearity and non-stationarity of EEG signals, traditional epilepsy detection methods, such as those based on manually extracted features and simple threshold decisions, often have difficulty achieving high accuracy.
[0004] With the development of artificial intelligence technology, especially deep learning technology, significant progress has been made in the field of epilepsy detection. Deep learning models, such as convolutional neural networks (CNN) and long short-term memory networks (LSTM), can automatically learn and extract meaningful features from complex EEG signals, significantly improving the accuracy of epilepsy detection. However, these models usually require a lot of computing resources and data, which is impractical in resource-limited environments.
[0005] In addition, most existing epilepsy detection methods rely on single-modality signal processing, that is, using only the time domain or frequency domain information of EEG signals. However, brain activity is a multimodal process that involves multiple dimensions such as time and frequency. Single-modality analysis cannot fully utilize this information and may overlook potentially valuable patterns, thus limiting the performance of the detection method.
[0006] Therefore, there is an urgent need to develop a new epilepsy detection method that can comprehensively utilize the rich information of multimodal EEG signals while keeping the model lightweight and efficient to adapt to the resource constraints in practical application scenarios. Summary of the invention
[0007] The present invention aims to solve the problems existing in the existing epilepsy detection technology, especially the problems of insufficient accuracy and high model complexity when processing epileptic electroencephalogram (EEG) signals, thereby improving the accuracy and efficiency of epilepsy detection while reducing the demand for computing resources. In view of the above problems, the present invention proposes an epilepsy detection method based on an improved deep learning model and a corresponding system.
[0008] The present invention is implemented through the following technical solutions. On the one hand, the present invention provides a training method for a multimodal lightweight epileptic EEG signal classification model, and the training method includes: adaptive decoupling knowledge distillation, a teacher model and a student model.
[0009] The adaptive decoupled knowledge distillation includes target class loss, non-target class loss, label loss and adaptive loss weight;
[0010] The target class loss is used to calculate the distance between the output of the teacher model and the output of the student model on the target class;
[0011] The non-target class loss is used to calculate the distance between the output of the teacher model and the output of the student model on the non-target class;
[0012] The label loss is used to calculate the difference between the output of the student model and the data label;
[0013] The adaptive loss weight is used to adaptively adjust the weights of the three losses during the model training process, so that the model can achieve better training effect;
[0014] The teacher model includes one-dimensional RepVGG-A2 and two-dimensional RepVGG-B2SE;
[0015] The student model includes a feature extraction module of a lightweight multimodal epilepsy detection model provided by the present invention.
[0016] In a second aspect, the present invention provides a method for establishing a lightweight multimodal epilepsy detection model, wherein the lightweight multimodal epilepsy detection model comprises: a feature extraction module, a multimodal feature fusion module and a classification module.
[0017] The feature extraction module includes a one-dimensional feature extraction module and a two-dimensional feature extraction module;
[0018] The one-dimensional feature extraction module includes an expansion reparameterization module and an SE expansion reparameterization module;
[0019] The dilated reparameterization module includes dilated convolution and structural reparameterization blocks, which are used to extract features of different scales in one-dimensional EEG signals and control the receptive field of the model;
[0020] The structure reparameterization block includes three convolution channels in the training stage, wherein the first channel includes a 3×3 convolution and a batch normalization layer, the second channel includes a 1×1 convolution and a batch normalization layer, and the third channel includes a batch normalization layer;
[0021] The SE expansion reparameterization block includes an SE module and an expansion reparameterization block, which is used to increase the weight of the one-dimensional EEG signal feature;
[0022] The two-dimensional feature extraction module includes a two-dimensional expansion reparameterization module and a two-dimensional SE expansion reparameterization module, which are used to extract features in the two-dimensional image and increase the weight of the two-dimensional image features;
[0023] The multimodal feature fusion module includes a global consistency module and a local co-occurrence module;
[0024] The global consistency module includes a global maximum pooling, a convolution layer and a projection layer, which is used to fuse the one-dimensional features and the two-dimensional features in the form of a jump connection, thereby playing a role in characterizing the complementarity of the two modalities in global representation;
[0025] The projection layer includes a dimension transformation operation and a channel adjustment operation, which is used to map the one-dimensional feature to the same dimension and number of channels as the two-dimensional feature;
[0026] The local co-occurrence module includes a cross-attention mechanism and a global average pooling to analyze the long-term interactions between samples in the two modalities and construct a feature vector from the two modalities;
[0027] The classification module includes global average pooling and 1×1 convolution for integrating the outputs of the global consistency module and the local co-occurrence module and obtaining a score for each category.
[0028] In a third aspect, the present invention provides a classification method for a multimodal lightweight epileptic EEG signal classification model, the method comprising:
[0029] Step 1, obtaining a two-dimensional time-frequency diagram of epileptic EEG signals;
[0030] Step 2: input the one-dimensional EEG signal and the two-dimensional time-frequency graph into the lightweight multimodal epilepsy classification model as described above to obtain classification results of the one-dimensional EEG signal and the two-dimensional time-frequency graph. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0032] Figure 1 The overall process of adaptive decoupled knowledge distillation for model training proposed by the present invention;
[0033] Figure 2 It is a structural schematic diagram of a one-dimensional feature extraction module of the model of the present invention;
[0034] Figure 3 The structure of the structural reparameterization module in the model of the present invention and the schematic diagram of the reparameterization process;
[0035] Figure 4 It is a structural schematic diagram of a two-dimensional feature extraction module of the model of the present invention;
[0036] Figure 5 The overall structure of MMECNet for epileptic EEG signal classification proposed in the present invention;
[0037] Figure 6 It is a schematic diagram of the structure of the cross attention mechanism in the model of the present invention;
[0038] Figure 7 Visualization image of Bonn epilepsy EEG dataset;
[0039] Figure 8 Schematic diagram of the process of obtaining a two-dimensional time-frequency map for a one-dimensional EEG signal;
[0040] Fig. 9 The comparison results between the MMECNet model of the present invention and other networks. DETAILED DESCRIPTION
[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0042] Embodiment 1: A training method for a multimodal lightweight epileptic EEG signal classification model, the training method comprising: adaptive decoupling knowledge distillation, a teacher model and a student model.
[0043] The adaptive decoupled knowledge distillation includes target class loss, non-target class loss, label loss and adaptive loss weight;
[0044] The target class loss is used to calculate the distance between the output of the teacher model and the output of the student model on the target class;
[0045] The non-target class loss is used to calculate the distance between the output of the teacher model and the output of the student model on the non-target class;
[0046] The label loss is used to calculate the difference between the output of the student model and the data label;
[0047] The adaptive loss weight is used to adaptively adjust the weights of the three losses during the model training process, so that the model can achieve better training effect;
[0048] The teacher model includes one-dimensional RepVGG-A2 and two-dimensional RepVGG-B2SE;
[0049] The student model includes a feature extraction module of a lightweight multimodal epilepsy detection model provided by the present invention.
[0050] In this embodiment, if Figure 1 As shown in the figure, first, considering that the classic knowledge distillation loss is a highly coupled formulation, the non-target class loss term is weighted by a coefficient that is negatively correlated with the teacher's prediction confidence for the target class. Therefore, a larger prediction score leads to a smaller weight. This coupling significantly suppresses the effect of non-target classes on well-predicted training samples. Secondly, the weights of the target class and the non-target class are also coupled, that is, the target class and the non-target class cannot be weighted separately. Since their contributions come from different aspects, the target class loss and the non-target class loss are considered separately.
[0051] Furthermore, the data label loss branch will also have a positive impact on improving model performance and distillation effect. Therefore, an adaptive loss weight is introduced to optimize the impact of the three losses on model performance during training.
[0052] Embodiment 2: This embodiment further limits the training method of a multimodal lightweight epilepsy EEG signal classification model described in embodiment 1. In this embodiment, the target class loss and non-target class loss are further limited, specifically including:
[0053] The target class loss and non-target class loss are determined by KL divergence.
[0054] In this implementation, KL divergence is used to determine the difference in the distribution of the target class and non-target class between the output of the teacher model and the student model. The specific calculation formula is:
[0055]
[0056]
[0057] Where: Represents the KL divergence of the binary probabilities output by the teacher model and the student model in the target class, written as: and Represents the KL divergence between the teacher model and the student model in non-target categories, written as:
[0058] in is the binary probability of the target category, is the binary probability of the non-target class. α and β are the weights of the target class loss and the non-target class loss, respectively.
[0059] Embodiment 3: This embodiment further limits the training method of a multimodal lightweight epilepsy EEG signal classification model described in Embodiment 1. In this embodiment, the label loss is further limited, specifically including:
[0060] The label loss is determined by the cross entropy loss between the student model output and the data label. The specific formula is:
[0061]
[0062] Where C is the total number of categories, y i,c is an indicator (0 or 1) indicating whether the i-th sample belongs to category c, y i,c is the probability that the model predicts that the i-th sample belongs to category c.
[0063] Embodiment 4: This embodiment further limits the training method of a multimodal lightweight epilepsy EEG signal classification model described in Embodiment 1. In this embodiment, the adaptive loss weight is further limited, specifically including:
[0064] By using adaptive loss weights, the three losses can be used more effectively and flexibly to enhance the generalization performance of the model, while avoiding the problems that may be caused by manually setting weights. In the initial stage of training, the weights of the three loss functions are initialized to 1. During the training process, the AdamW optimizer is used to optimize the three weights using loss gradients. The expression of the adaptive loss weight is specifically:
[0065]
[0066] Among them, L sum is the total loss of the target class and the non-target class, W i is the weight of the i-th loss, L i is the i-th loss, is a regularization term to avoid negative values and zero solutions.
[0067] Embodiment 5: A method for establishing a lightweight multimodal epilepsy detection model, wherein the lightweight multimodal epilepsy detection model comprises: a feature extraction module, a multimodal feature fusion module and a classification module.
[0068] The feature extraction module includes a one-dimensional feature extraction module and a two-dimensional feature extraction module;
[0069] The one-dimensional feature extraction module includes an expansion reparameterization module and an SE expansion reparameterization module;
[0070] The dilated reparameterization module includes dilated convolution and structural reparameterization blocks, which are used to extract features of different scales in one-dimensional EEG signals and control the receptive field of the model;
[0071] The structure reparameterization block includes three convolution channels in the training stage, wherein the first channel includes a 3×3 convolution and a batch normalization layer, the second channel includes a 1×1 convolution and a batch normalization layer, and the third channel includes a batch normalization layer;
[0072] The SE expansion reparameterization block includes an SE module and an expansion reparameterization block, which is used to increase the weight of the one-dimensional EEG signal feature;
[0073] The two-dimensional feature extraction module includes a two-dimensional expansion reparameterization module and a two-dimensional SE expansion reparameterization module, which are used to extract features in the two-dimensional image and increase the weight of the two-dimensional image features;
[0074] The multimodal feature fusion module includes a global consistency module and a local co-occurrence module;
[0075] The global consistency module includes a global maximum pooling, a convolution layer and a projection layer, which is used to fuse the one-dimensional features and the two-dimensional features in the form of a jump connection, thereby playing a role in characterizing the complementarity of the two modalities in global representation;
[0076] The projection layer includes a dimension transformation operation and a channel adjustment operation, which is used to map the one-dimensional feature to the same dimension and number of channels as the two-dimensional feature;
[0077] The local co-occurrence module includes a cross-attention mechanism and a global average pooling to analyze the long-term interactions between samples in the two modalities and construct a feature vector from the two modalities;
[0078] The classification module includes global average pooling and 1×1 convolution for integrating the outputs of the global consistency module and the local co-occurrence module and obtaining a score for each category;
[0079] In this embodiment, firstly, considering that the feature representation ability of the single modality of time domain is limited, the model cannot learn sufficient features, which increases the possibility of misclassification. Therefore, the one-dimensional EEG signal is transformed to obtain its time-frequency diagram, which enriches the feature expression of epileptic EEG signals, enables the model to learn the frequency domain features of the signal, and improves the classification effect of the model;
[0080] Secondly, in order to better integrate one-dimensional features and two-dimensional features, a global consistency module and a local co-occurrence module are added after the feature extraction backbone. The global consistency module maps the two modal features to the same dimension and then fuses the two modal features in the form of jump connections. The local co-occurrence module obtains the local correlation features of the two modalities through the cross-attention mechanism.
[0081] Embodiment 6: This embodiment further limits the method for constructing a multimodal lightweight epilepsy EEG signal classification model described in Embodiment 5. In this embodiment, the feature extraction module is further limited, specifically including:
[0082] The one-dimensional feature extraction module and the two-dimensional feature extraction module;
[0083] The two-dimensional feature is a time-frequency diagram obtained by transforming the EEG signal using short-time Fourier transform;
[0084] In this embodiment, a one-dimensional feature extraction module and a two-dimensional feature extraction module are used to extract features from a one-dimensional signal and a two-dimensional image, respectively. The one-dimensional signal is the original epileptic EEG signal, and the two-dimensional image is a spectrum obtained by transforming the original epileptic EEG signal using a short-time Fourier transform (STFT) with a Hanning window. Epileptic seizures are usually accompanied by significant changes in brain activity, which are more obvious in the time-frequency diagram, which helps to identify signal patterns unique to epilepsy;
[0085] Furthermore, the absolute values of these spectrograms (i.e., the amplitude of the STFT) are calculated. Next, the amplitude of the STFT is converted into a spectrogram in decibel (dB) scale. In addition, the first-order difference (delta) and second-order difference (delta-delta) of the spectrogram are calculated, which represent the first-order and second-order rates of change of the spectrogram over time, respectively. Finally, the dB-scaled spectrogram, delta, and delta-delta are combined to form a time-frequency graph containing three channels. The values of the time-frequency graph corresponding to each EEG signal are normalized to between 0 and 1, and the image length and width are scaled to 224×224.
[0086] Embodiment 7: This embodiment further limits the feature extraction module described in Embodiment 6. In this embodiment, the one-dimensional feature extraction module is further limited, specifically including:
[0087] The one-dimensional feature extraction module includes three dilation reparameterization blocks and one SE reparameterization module;
[0088] The one-dimensional feature extraction module is as follows: Figure 2As shown in the figure, the stem layer includes normalization and standardization operations on the data. The normalization and batch standardization operations are to solve the problem of inconsistent numerical scales of data in different dimensions. This problem may cause the model to encounter numerical stability problems during training, such as gradient disappearance or gradient explosion. The specific expression is:
[0089]
[0090] Among them, x norm is the result after normalization and standardization, x min is the minimum value of the input signal, x max is the maximum value of the input signal. γ and β are learnable scaling and shifting parameters that can be optimized through the training process. ò is a small number used to avoid the problem of dividing by zero;
[0091] Furthermore, each expansion reparameterization block contains 1-2 structure reparameterization blocks, and each structure reparameterization block has a different step size, so as to control the size of the receptive field and extract multi-scale features;
[0092] The structure reparameterization block is as follows Figure 3 As shown in the figure, in the model training stage, it is a three-branch structure, while in the model reasoning stage, it can be losslessly converted into a single-branch structure, thereby reducing the number of model parameters and improving the model reasoning speed;
[0093] Furthermore, only the first two dilated reparameterization blocks contain dilated convolutions, while the third dilated reparameterization block consists of two structural reparameterization blocks, so that the receptive field in the shallow layer of the model is not too small to lose some global features, and the receptive field in the deep layer of the model is not too large to ignore some small features;
[0094] Furthermore, the SE reparameterization module includes a structural reparameterization module and a SE module. The SE module is the last layer of the one-dimensional feature extraction module, which is used to further refine the deepest features with the richest semantic information, suppress the expression of irrelevant features, and improve the representation ability of key effective features, thereby improving the classification performance.
[0095] Embodiment 8: This embodiment further limits the feature extraction module described in Embodiment 6. In this embodiment, the two-dimensional feature extraction module is further limited, specifically including:
[0096] The two-dimensional feature extraction module includes three two-dimensional structure reparameterization modules and one two-dimensional SE reparameterization module;
[0097] The two-dimensional feature extraction module is as follows: Figure 4As shown, the Stem layer includes batch normalization and 3×3 convolution operations, which are used to normalize the two-dimensional time-frequency map and perform preliminary feature extraction;
[0098] Furthermore, in the three structural reparameterization modules, the number of output channels is 16, 64 and 256 respectively, which is used to gradually extract more detailed features to improve the robustness of the model;
[0099] Furthermore, in the SE reparameterization module shown, the number of output channels is the same as the number of output channels of the previous layer structural reparameterization module, which is used to further condense the deep features rich in semantic information output by the previous layer using the SE module, suppress the expression of irrelevant features, and improve the representation ability of key effective features, thereby improving the classification performance.
[0100] Embodiment 9: This embodiment further limits the method for constructing a multimodal lightweight epilepsy EEG signal classification model described in Embodiment 5. In this embodiment, the global consistency module is further limited, specifically including:
[0101] The global consistency module is as follows Figure 5 As shown, it includes two global maximum pooling layers (GMP) for extracting the most significant features of one-dimensional modal features and two-dimensional modal features respectively. For one-dimensional features, after GMP, it is necessary to pass through a projection layer, which includes a dimension expansion operation and a channel adjustment operation, respectively, to map the dimension of the one-dimensional modal feature to the same dimension as the two-dimensional modal feature, which is convenient for subsequent feature fusion operations. The channel adjustment operation is implemented by 1×1 convolution. For the fusion operation between the two modalities, the feature values of the two modalities are added by using a jump connection.
[0102] By extracting the most significant features, the complexity of the data can be reduced, and the model's attention to key information can be increased, thereby enhancing the classification performance; secondly, the effective fusion of modal features can more comprehensively characterize the characteristics of EEG signals and enhance the model's adaptability to the diversity of EEG signals; finally, through channel adjustment and feature fusion operations, efficient integration of data of different dimensions can be achieved, and the generalization ability of the model can be improved, which plays an important role in improving the accuracy and reliability of epilepsy detection.
[0103] Embodiment 10: This embodiment further limits the method for constructing a multimodal lightweight epilepsy EEG signal classification model described in Embodiment 5. In this embodiment, the local co-occurrence module is further limited, specifically including:
[0104] The local co-occurrence module includes a cross-attention mechanism and a global average pooling layer (GAP);
[0105] The local co-occurrence module is as follows Figure 5 As shown in the figure, the local co-occurrence module aims to construct a feature vector from the two modalities by analyzing the long-term interaction between samples in the two modalities. After the time-frequency features are combined with the length and width dimensions of the feature map to reach the same dimension as the time domain features, the cross-attention mechanism is used to match the relevant parts of the two modal features, and the local co-occurrence features of the two modal features are obtained by global average pooling (GAP);
[0106] The cross attention mechanism is as follows Figure 6 As shown, the calculation of cross attention can be divided into the following steps:
[0107] Step 1: Calculate the query, key, and value. For each feature in I, calculate a query, and for each feature in S, calculate a key and value. These can be obtained through a learnable linear transformation, that is,
[0108] Q=IW Q ,K=SW K ,V=SW V
[0109] Where: W Q , W K and W V is a trainable weight matrix.
[0110] Step 2: Calculate the attention score. Calculate the similarity between the query of each feature in I and the key of each feature in S, usually using the dot product. This attention score indicates which features in S are most relevant to the current feature in I. The formula is:
[0111]
[0112] Where: d k is the dimension of the Key vector, which is used to scale the dot product to prevent the gradient vanishing problem.
[0113] Step 3: Apply softmax and get the weighted value. The attention score of each query is normalized using the softmax function, and then the value is weighted summed using these normalized scores. This ensures that the model focuses on the most relevant parts of S.
[0114] Step 4: Output the result. Finally, the weighted sum is output, which represents the weighted representation of the features in I to the features in S.
[0115] Embodiment 11: This embodiment further limits the method for constructing a multimodal lightweight epilepsy EEG signal classification model described in Embodiment 5. In this embodiment, the classification module is further limited, specifically including:
[0116] The classification module includes feature concatenation operation, GAP and 1×1 convolution;
[0117] The classification module is as follows Figure 5 As shown in the figure, after concatenating the features of the output of the global consistency module and the local co-occurrence module, in order to reduce the number of model parameters and enable MMECNet to adaptively process inputs of different sizes, GAP combined with 1×1 convolution is used to replace the fully connected layer with a large number of parameters to complete the classification of epileptic EEG signals.
[0118] Embodiment 12: A classification method for a lightweight multimodal epilepsy detection model, the method comprising:
[0119] Step 1, obtaining a two-dimensional time-frequency diagram of epileptic EEG signals;
[0120] Step 2: input the one-dimensional EEG signal and the two-dimensional time-frequency graph into the lightweight multimodal epilepsy classification model as described above to obtain classification results of the one-dimensional EEG signal and the two-dimensional time-frequency graph.
[0121] The classification method in this embodiment includes: (1) The feature extraction module pre-trained by adaptive decoupling knowledge distillation can have a strong feature extraction ability when the model parameters are small, and combine the multi-scale idea to fully extract the effective features of the one-dimensional modality and the two-dimensional modality; (2) Here, the SE module is a plug-and-play module applied to the tail end of the feature extraction module. The multimodal information of the EEG signal can be represented by deep features; (3) The global consistency module can obtain a deeper level of multimodal semantic expression, making the model more robust. (4) The local co-occurrence module can extract the correlation score between the two modalities, highlight the features with strong correlation, and make the model more capable of learning and utilizing features.
[0122] Next, the classification results of a lightweight multimodal epilepsy detection model based on the present invention are tested through experiments.
[0123] 1 Experimental set and data preprocessing
[0124] This experiment uses the epilepsy EEG dataset from the University of Bonn, Germany, which has been widely used in the study of epilepsy detection. Each record in the dataset is about 23.6 seconds, with a sampling frequency of 173.61Hz. The complete dataset contains five groups (A to E), each containing 100 single-channel EEG signals; these signals are selected from continuous multi-channel EEGs by manual inspection to remove some EEG data that are greatly affected by other electrophysiological activities. In addition, the signal segments in the dataset must also meet the stationary discrimination conditions. A total of 4097 samples were collected for each EEG signal. Groups A and B are records of five healthy volunteers in a waking state with their eyes open (Group A) and closed (Group B), respectively. Groups C, D, and E are intracranial records of epileptic patients. The EEG signals of Group D come from the area that causes epileptic seizures, and the records of Group C come from the hippocampal structure on the contralateral side of the brain. Groups C and D are EEG signals in the pre-epilepsy period (without epileptic seizures), while Group E is EEG signals during epileptic seizures. The sampling frequency of the signal is 173.61 Hz, and it has been filtered using a 0.53-40 Hz bandpass filter. The graphical representation of the EEG signal in the data is as follows Figure 7 shown.
[0125] In this experiment, the EEG dataset used is relatively small, and each set contains only 100 continuous EEG signals. Considering the deep learning method used in this paper, the model parameters are relatively large, so it is necessary to artificially divide the data in order to obtain more training data and test data, in order to improve the training effect and generalization ability of the model. The specific approach is: apply a time window with a window size of 1024 and a step size of 128 to intercept each single-channel signal. In this way, each complete signal is divided into 25 sub-signals, each of which has a length of 1024. This sliding window strategy effectively expands the original 400 data to 10,000, significantly increasing the size of the data set. This data amplification strategy not only increases the amount of data available for training and testing, but also helps to improve the generalization ability of the model under different time windows, so that the model has better stability and accuracy. Through this division and amplification strategy, the limited original data can be fully utilized, providing richer and more diverse data resources for the training and verification of deep learning models. For the acquisition of the above-mentioned two-dimensional time-frequency graph, such as Figure 8 shown.
[0126] The cross entropy loss function is used as the loss function for back propagation, and AdamW is used as the optimizer for model parameters to train the model. In order to make the model have better performance in the initial training iteration and avoid gradient explosion, the warm-up learning rate and cosine learning rate adjustment methods are used. Table 1 shows the detailed hyperparameters in the model training process.
[0127] The experimental environment is: the deep learning framework is Pytorch 1.9.0, the Linux system is equipped with Intel(R) Xeon(R) Silver 4216CPU@2.10GHz and NVIDIA GeForce RTX 3070 (8GB, 1.73GHz), and python3.8 is used as the programming language.
[0128] Table 1 Model training hyperparameters
[0129] Hyperparameters Parameter value (2 categories) Parameter value (3 categories) Number of training rounds 200 500 Optimizer AdamW AdamW Batch size 128 128 Input Dimensions 128×1×1024 128×1×1024 Learning Rate 2e-3 1e-3 Preheating rounds 5 10 Warm up the initial learning rate 1e-3 1e-3 Learning rate at the end of training 1e-6 1e-6
[0130] 2 Evaluation Metrics
[0131] In order to quantitatively analyze the experimental results of the model proposed in this invention, a 5-fold cross-validation method was used in the evaluation to ensure that the model was tested under different data changes. The data was divided into 5 folds, and 4 folds (80%) were used as training data each time, and the remaining fold (20%) was used as test data; for each fold, the data was expanded using the proposed time window partitioning method. The performance evaluation used some well-known performance indicators, including accuracy, specificity, sensitivity, and F1-Score. The following are the definitions of these indicators:
[0132]
[0133]
[0134]
[0135]
[0136] Where: TP (true positive) represents the number of positive samples correctly predicted as positive samples, TN (true negative) represents the number of negative samples correctly predicted as negative samples, FP (false positive) represents the number of negative samples mistakenly classified as positive samples, and FN (false negative) represents the number of positive samples mistakenly classified as negative samples.
[0137] Furthermore, in order to evaluate the performance of the classifier from different perspectives and help understand the advantages and disadvantages of the MMECNet in different aspects, this paper also uses the confusion matrix as a more intuitive evaluation indicator. The representation of the confusion matrix for TP, TN, FP and FN is shown in Table 2.
[0138] Table 2 Confusion matrix representation
[0139]
[0140] 3 Experimental design
[0141] In order to verify the effectiveness of the proposed MMECNet, three experiments were designed to verify the performance of MMECNet on the dataset, compare the performance of other models under the same dataset, and perform ablation experiments. The specific experimental design is as follows:
[0142] (1) Data Verification
[0143] The model performance of MMECNet was tested on normal vs. interictal vs. epileptic seizure (AB vs.CD vs.E), normal vs. epilepsy (AB vs.CD and AB vs.CDE), interictal vs. epileptic seizure (C vs.E, D vs.E and CD vs.E), and normal vs. epileptic seizure (A vs.E, B vs.E and AB vs.E). The accuracy, sensitivity, specificity, F1-Score and confusion matrix were selected as evaluation indicators to analyze the performance of the model on different data combinations.
[0144] (2) Comparative experiment
[0145] The performance of MMECNet was compared with other models in normal vs. interictal vs. epileptic seizure (AB vs.CD vs.E), normal vs. epilepsy (AB vs.CD and AB vs.CDE), interictal vs. epileptic seizure (C vs.E, D vs.E and CD vs.E) and normal vs. epileptic seizure (A vs.E, B vs.E and AB vs.E) groups. The evaluation indicators were accuracy, number of model parameters and model size.
[0146] (3) Ablation experiment
[0147] In order to verify the contribution of each module of the proposed method, five groups of sub-experiments were designed in the ablation experiment: time domain feature extraction backbone + classifier, time-frequency domain feature extraction backbone + classifier, dual-branch feature extraction backbone + global consistency module, dual-branch feature extraction backbone + local co-occurrence module and MMECNet. The performance of each model in normal vs. interictal vs. epilepsy (AB vs.CD vs.E) classification was verified separately.
[0148] 4 Comparison of experimental results
[0149] For the three-classification problem, the average performance of MMECNet in the five-fold cross validation on the data grouping AB vs. CD vs. E is shown in Table 3, and its accuracy reaches 99.88. From the confusion matrix, it can be seen that the main types of misclassification are AB vs. CD.
[0150] Table 3 Performance of three-classification problem (AB vs. CD vs. E)
[0151]
[0152] Similarly, for the normal vs. epilepsy patients in the binary classification problem (AB vs. CD and AB vs. CDE), as shown in the confusion matrix in Table 4, the main classification errors are also concentrated on misclassifying normal people as epilepsy patients.
[0153] Table 4 Average performance and confusion matrix for distinguishing normal and epileptic patients
[0154]
[0155] In distinguishing between normal and epileptic patients during seizures (Avs.E, B vs.E, and AB vs.E) and between epileptic patients during interictal seizures and epileptic patients during seizures (C vs.E, D vs.E, and CD vs.E), as shown in Tables 5 and 6, MMECNet achieved 100% accuracy.
[0156] Table 5 Average performance and confusion matrix for distinguishing interictal and epileptic seizures
[0157]
[0158] Table 6 Average performance and confusion matrix for distinguishing normal and epileptic seizures
[0159]
[0160] In order to compare the size of the model used by MMECNet and other SOTA solutions, as shown in Table 7, the number of parameters and memory size of each module in MMECNet are calculated respectively.
[0161] Table 7 MMECNet module parameter quantity and size
[0162] Modules Parameter quantity Memory size (MB) DiRepNet Backbone 170971 33.06 RepVGG4 Backbone 159760 10.72 Global Consistency 8964 0.74 Local Co-Occurrence 56744 4.71 MMECNet 396439 49.23
[0163] The performance comparison of MMECNet and other models is shown in the figure Fig. 9 As shown in the figure, the horizontal axis represents the model parameters, and the vertical axis represents the classification accuracy of the model in normal vs. interictal vs. epileptic seizure (AB vs. CD vs. E). Therefore, the closer the point in the figure is to the upper left corner, the higher the performance and efficiency of the model.
[0164] The quantitative comparison between MMECNet and other models is shown in Table 8. In all comparative experimental schemes, MMECNet has achieved better performance than others, and the number of model parameters and model size of MMECNet remain at a low level.
[0165] Table 8 Comparison between MMECNet and SOTA models
[0166] Methodology AB-CD-E AB-CDE CE DE CD-E Novel 1D-CNN 96.97 - 98.02 97.63 98.03 CLSTM 97 99 100 98 100 SeizureNet 97.07 - 99.5 98.5 99.33 DWT,CNN 99.4 - - - - EEG waveform,2D-CNN 97.86 - - - - MNL-Network 97.76 - 99.35 98.76 98.84 EWT,MF-RDCNN 99.82 - - - 100 DiRepNet 99.63 99.64 99.98 99.97 99.97 DiRepNet-S 99.61 99.46 99.97 99.97 99.94 MMECNet 99.88 99.71 100 100 100
[0167] Table 8
[0168] Methodology AE BE AB-E Total params Memory(MB) Novel 1D-CNN 99.52 99.11 99.38 168,763 33.91 CLSTM 100 100 100 13,298,051 148.48 SeizureNet 100 99 100 119,445 64.12 DWT,CNN - - - 10,333,395 352.61 EEG waveform,2D-CNN - - - 9,544,771 97.1x128 MNL-Network 99.93 99.72 99.75 120,903 13.48 EWT,MF-RDCNN - - 100 5,345,878 97.32 DiRepNet 100 100 100 661,128 48.8 DiRepNet-S 100 99.97 100 173,539 33.56 MMECNet 100 100 100 396439 49.23
[0169] In the ablation experiment, the performance of each model in the classification of normal vs. intermittent vs. epilepsy (AB vs. CD vs. E) was verified separately, using accuracy, specificity, sensitivity, F1 score and confusion matrix as performance indicators. The results of the ablation experiment are shown in Table 9. Compared with the single modality, the multimodal model has a certain improvement in classification accuracy, and the cross attention mechanism in the local co-occurrence module has a more significant improvement in accuracy than the simple feature addition of the global consistency module, and the combination of the two can further improve the classification accuracy of the model.
[0170] Table 9 Performance of each module in the ablation experiment
[0171] Submodel Acc(%) Sen(%) Spe(%) F1-Score(%) One-dimensional feature extraction module 99.61 99.67 99.70 99.67 2D feature extraction module 98.65 98.71 98.69 98.65 Feature extraction module + global consistency module 99.70 99.67 99.70 99.71 Feature extraction module + local co-occurrence module 99.83 99.85 99.87 99.85 MMECNet 99.88 99.86 99.91 99.88
[0172] Epileptic seizures are often associated with changes in brain waves at specific frequencies. Processing EEG through STFT and decomposing it into frequency content in specific time windows helps capture the time-frequency characteristics of the signal. In addition, calculating the first-order and second-order differences (delta and delta-delta) of the spectrogram helps capture the temporal speed and acceleration of the spectrum, which is crucial for accurately identifying the dynamic changes of EEG. The global consistency module can effectively integrate global features from different domains through global maximum pooling and dimensionality adjustment strategies, enhancing the model's overall grasp of different time and frequency features. The local co-occurrence module strengthens the model's ability to match relevant parts of the time domain and time-frequency domain features through the cross-attention mechanism, allowing the model to more carefully analyze and utilize the interaction between the two modalities.
[0173] In the present invention, a multimodal scheme is adopted to fuse the time domain features of epilepsy EEG with the time-frequency domain features of the time-frequency graph, and a multimodal epilepsy classification network (MMECNet) is proposed. First, the time-frequency graph and its role are introduced, and the acquisition scheme of the time-frequency graph in this chapter is explained. Then an overview of multimodality is introduced, and the feature extraction backbone of MMECNet, the cross-attention mechanism for fusing the time domain features with the time-frequency domain features of the time-frequency graph, and other module designs in MMECNet are described in detail. In order to verify the effectiveness of MMECNet and the contribution of each module, a series of verification experiments, comparative experiments and ablation experiments are designed. In the verification experiment, the performance and stability of MMECNet are verified in 9 different data combinations. In the comparative experiment, the accuracy, model size and model parameter amount of other models in three classifications are compared. In the ablation experiment, each part of the model is tested separately to compare the contribution of each part of the model. The experiment shows that MMECNet has good performance and robustness on the German Bonn epilepsy EEG dataset, and can achieve better results in model performance and model efficiency.
Claims
1. A training method for a multimodal lightweight epilepsy EEG signal classification model, the training method comprising: Adaptive decoupling of knowledge distillation, teacher model and student model. The adaptive decoupled knowledge distillation includes target class loss, non-target class loss, label loss and adaptive loss weight; The target class loss is used to calculate the distance between the output of the teacher model and the output of the student model on the target class; The non-target class loss is used to calculate the distance between the output of the teacher model and the output of the student model on the non-target class; The label loss is used to calculate the difference between the output of the student model and the data label; The adaptive loss weight is used to adaptively adjust the weights of the three losses during the model training process, so that the model can achieve better training effect; The teacher model includes one-dimensional RepVGG-A2 and two-dimensional RepVGG-B2SE; The student model includes a feature extraction module of a lightweight multimodal epilepsy detection model provided by the present invention.
2. The method for training a multimodal lightweight epilepsy EEG signal classification model according to claim 1, characterized in that: The target class loss and non-target class loss are determined by KL divergence; The label loss is determined by the cross entropy loss between the student model output and the data labels.
3. The training method of a multimodal lightweight epilepsy EEG signal classification model according to claim 1, characterized in that: In the initial stage of training, the weights of the three loss functions are initialized to 1. During the training process, the AdamW optimizer is used to optimize the three weights using the loss gradient. The expression of the adaptive loss weight is specifically:
4. A method for establishing a lightweight multimodal epilepsy detection model, the lightweight multimodal epilepsy detection model comprising: Feature extraction module, multimodal feature fusion module and classification module. The feature extraction module includes a one-dimensional feature extraction module and a two-dimensional feature extraction module; The one-dimensional feature extraction module includes an expansion reparameterization module and an SE expansion reparameterization module; The dilated reparameterization module includes dilated convolution and structural reparameterization blocks, which are used to extract features of different scales in one-dimensional EEG signals and control the receptive field of the model; The structure reparameterization block includes three convolution channels during the training phase, wherein the first channel includes convolution and batch normalization layers, the second channel includes convolution and batch normalization layers, and the third channel includes a batch normalization layer; The SE expansion reparameterization block includes an SE module and an expansion reparameterization block, which is used to increase the weight of the one-dimensional EEG signal feature; The two-dimensional feature extraction module includes a two-dimensional expansion reparameterization module and a two-dimensional SE expansion reparameterization module, which are used to extract features in the two-dimensional image and increase the weight of the two-dimensional image features; The multimodal feature fusion module includes a global consistency module and a local co-occurrence module; The global consistency module includes a global maximum pooling, a convolution layer and a projection layer, which is used to fuse the one-dimensional features and the two-dimensional features in the form of a jump connection, thereby playing a role in characterizing the complementarity of the two modalities in global representation; The projection layer includes a dimension transformation operation and a channel adjustment operation, which is used to map the one-dimensional feature to the same dimension and number of channels as the two-dimensional feature; The local co-occurrence module includes a cross-attention mechanism and a global average pooling to analyze the long-term interactions between samples in the two modalities and construct a feature vector from the two modalities; The classification module includes global average pooling and convolution for integrating the outputs of the global consistency module and the local co-occurrence module and obtaining a score for each category.
5. The method for establishing a lightweight multimodal epilepsy detection model according to claim 4, characterized in that: The one-dimensional EEG signal is transformed to obtain its time-frequency diagram, and a global consistency module and a local co-occurrence module are added after the feature extraction backbone. The global consistency module fuses the two modal features by mapping them to the same dimension in the form of jump connections, and the local co-occurrence module obtains the local correlation features of the two modalities through the cross-attention mechanism.
6. The time-frequency diagram according to claim 5, characterized in that The original epileptic EEG signal is transformed using a short-time Fourier transform (STFT) with a Hanning window to obtain its spectrogram. Further, the absolute values of these spectrograms (i.e., the amplitude of the STFT) are calculated. Next, the amplitude of the STFT is converted into a spectrogram with a decibel (dB) scale. In addition, the first-order difference (delta) and second-order difference (delta-delta) of the spectrogram are calculated. Finally, the dB-scaled spectrogram, delta, and delta-delta are combined to form a time-frequency graph containing three channels. The values of the time-frequency graph corresponding to each EEG signal are normalized to between 0 and 1, and the image length and width are scaled to 224×224.
7. The method for establishing a lightweight multimodal epilepsy detection model according to claim 4, characterized in that: The one-dimensional feature extraction module includes three dilation reparameterization blocks and one SE reparameterization module; The two-dimensional feature extraction module includes three two-dimensional structure reparameterization modules and one two-dimensional SE reparameterization module.
8. The method for establishing a lightweight multimodal epilepsy detection model according to claim 4, characterized in that: The global consistency module includes two global maximum pooling layers (GMP). For one-dimensional features, after GMP, they need to go through a projection layer, which includes a dimension expansion operation and a channel adjustment operation. For the fusion operation between the two modes, the feature values of the two modes are added by using a jump connection method. The local co-occurrence module includes a cross-attention mechanism and a global average pooling layer (GAP); The classification module includes feature concatenation operation, GAP and 1×1 convolution.
9. A classification method for a lightweight multimodal epilepsy detection model, the method comprising: Step 1, obtaining a two-dimensional time-frequency diagram of epileptic EEG signals; Step 2: input the one-dimensional EEG signal and the two-dimensional time-frequency graph into the lightweight multimodal epilepsy classification model as described above to obtain classification results of the one-dimensional EEG signal and the two-dimensional time-frequency graph.
Citation Information
Cited By
Prognosis prediction method and system for chronic Doc patient
CN120048519A
Electroencephalogram abnormal signal detection method based on step-by-step identification and multi-agent decision
CN120694664A
Electroencephalogram abnormal signal detection method based on step-by-step identification and multi-agent decision
CN120694664B
