An electroencephalogram signal data signal analysis method based on multi-modal deep learning

Abstract

The application belongs to the cross field of artificial intelligence and physiological signal and medical image processing fields, and provides a brain electrical signal data signal analysis method based on multi-modal deep learning. The purpose is to solve the problem that the existing scheme still needs to set the extracted signal features manually and cannot completely cover all effective features of west syndrome. The main scheme includes extracting original brain electrical signal data and original muscle electrical signal data to obtain binary data; data separation obtains multi-channel time-frequency domain data and divides training set data, validation set data and test set data; the neural network model of deep learning is set, and the best model parameter is obtained through training; then the data is input for classification, the abnormal brain electrical signal data and muscle electrical signal data in each time period are obtained, and the annotation is saved as an annotation file.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and physiological signal and medical image processing, and provides a method for analyzing electroencephalogram (EEG) signal data based on multimodal deep learning. Background Technology

[0002] West syndrome is a type of infantile epilepsy, also known as infantile spasms. Its main manifestations include infantile spasms, abnormal electroencephalogram (EEG), and developmental delay, most commonly occurring in infants aged 4 to 8 months. The physical manifestation of West syndrome is a sudden forward bending of the infant, with stiffening of the body, arms, and legs. These spasms are short-lived, typically lasting only a few seconds, but can occur two to one hundred times. Diagnosis of West syndrome requires an EEG. This type of epilepsy severely affects the infant's cognitive and learning development, thus requiring timely diagnosis and treatment.

[0003] Traditional manual visual EEG testing methods are slow, and their accuracy is limited by the experience of the personnel, resulting in large variations. This visual testing method not only easily causes visual fatigue for the staff, but is also inefficient and cannot make timely and effective judgments on the patient's condition.

[0004] In clinical practice, due to the presence of various types of EEG signals in West syndrome seizures during EEG monitoring, such as intracranial partial seizures, generalized complete seizures, K-complex waves, spindle waves, and body movement artifacts, the classification of this seizure is a typical multi-class problem.

[0005] In the field of automated detection, on the one hand, EEG signals are complex, non-stationary, random signals characterized by weak signals and large individual differences; on the other hand, in actual EEG monitoring, the number of samples exhibiting different EEG signals is uneven, with the number of samples of specific types of epileptic seizures being far lower than the number of normal EEG signals. This further complicates the detection task.

[0006] A high-frequency oscillation model of the scalp during seizures in infantile spasms, Publication No. (Publication)

[0007] CN112992341A discloses a high-frequency oscillation model of the scalp during seizures in infantile spasms, belonging to the field of electroencephalography (EEG) model technology. This model, trained by an EEG device, analyzes high-frequency oscillations in the scalp during EEG to determine whether they correspond to seizures of infantile spasms. It includes a data acquisition module, a data preprocessing module, a model feature extraction module, a model feature labeling module, a model training module, and a model testing module. This high-frequency oscillation model of the scalp during seizures in infantile spasms, trained using a large amount of historical case data, can label potential seizures of infantile spasms in EEG and output the potential probability of the disease based on the number of training iterations. This provides clinicians with rapid diagnostic evidence and suggestions, ensuring diagnostic accuracy while reducing the adverse effects of physicians' individual limitations in understanding the condition, thus improving the efficiency of infantile spasm diagnosis.

[0008] This technical method for predicting infantile spasms includes the following steps:

[0009] 1. Collect scalp electroencephalogram (EEG) data of infants diagnosed with infantile spasms during seizures. Each sample data includes EEG segments within 30 minutes before and after the seizure time.

[0010] 2. Integrate the sample data into a standardized model to obtain standardized and complete EEG data.

[0011] 3. Extract features related to high-frequency oscillations from EEG data, including the amplitude, duration, dynamic peak endpoint values, and power spectral entropy of EEG waves.

[0012] 4. Label the extracted features, establish a training dataset, and determine the correspondence between each sample data and infantile spasm attacks.

[0013] 5. Use a convolutional neural network to train the training dataset and generate a fully connected layer for 3D brain function image features.

[0014] 6. Use validation and test sets to validate and test the trained high-frequency oscillation model, adjust the model parameters, and enhance the model's ability to detect infantile spasms.

[0015] West syndrome is characterized by short-duration, frequent seizures. Existing methods lack sufficient data sampling. This invention samples the entire detection time and uses artificial intelligence technology to quickly and accurately identify the episodic phase of West syndrome, providing a basis for timely diagnosis and treatment.

[0016] This invention processes, transforms, compresses, and stores time-sampling data, effectively compressing data information and performing data augmentation, providing a larger dataset under conditions of limited data and time-consuming annotation.

[0017] Existing solutions still require manual setting of extracted signal features, which cannot fully cover all the effective features of West syndrome. However, this invention uses deep neural networks to effectively extract explicit and implicit information that can be recognized by humans from EEG data, thereby improving the model's learning and generalization abilities.

[0018] This invention uses a multimodal technique, sharing the EEG feature extraction module with other modalities, and adds electromyography (EMG) features, which is not present in existing methods.

[0019] By using validation and test sets, this invention can continuously iterate itself, optimize the model's parameters and performance, improve the model's accuracy and stability, and has self-evolution capabilities. Summary of the Invention

[0020] The purpose of this invention is to solve the problem that existing solutions still require manual setting of extracted signal features, which cannot fully cover all the effective features of West syndrome.

[0021] To achieve the above objectives, the present invention employs the following technical means:

[0022] A method for analyzing electroencephalogram (EEG) signal data based on multimodal deep learning includes the following steps:

[0023] Step 1: Preprocess and segment the raw EEG signal data and raw EMG signal data, and then convert and store the segmented data to obtain binary data.

[0024] Step 2: For the binary data obtained in Step 1, separate the data according to the set EEG signal and EMG signal channels, perform bandpass filtering and short-time Fourier transform on each signal channel, and convert and store the filtered and transformed time-frequency domain data to obtain multi-channel time-frequency domain data.

[0025] Step 3: Randomize the multi-channel time-frequency domain data obtained in Step 2 to obtain training set data, validation set data, and test set data;

[0026] Step 4: Set up the deep learning neural network model, including a deep convolutional neural network model for EEG signal processing, a deep convolutional neural network model for EMG signal processing, and a deep convolutional neural network model for processing both EEG and EMG signals simultaneously, and output the fusion of the two modalities, namely abnormal EEG signal data and EMG signal.

[0027] Step 5: Using the training set data, validation set data, and test set data obtained in Step 3, use the neural network model from Step 4 to train a classification model for whether West syndrome is flare-up, and obtain the optimal model parameters.

[0028] Step 6: For the EEG and EMG data of patients who need to predict West syndrome attacks, process the data using the technique in Step 2, then use the optimal model parameters saved in Step 5 to classify and obtain the annotations of abnormal EEG and EMG data for each time period, and save them as annotation files.

[0029] In the above scheme, step 1 specifically includes the following steps:

[0030] Step 1.1: Define a class DataTransformer to transform and extract raw EEG signal data. The DataTransformer class has two attributes, data_path and ch_names, which represent the location of the data file and the name of the required EEG or muscle signal channel, respectively.

[0031] Step 1.2: Define a method transform_data in the class DataTransformer to read the data file, retain the specified signal channels, extract event information, and split the data into units according to the event type. The method transform_data returns a list containing all unit data.

[0032] Step 1.3: Define a method save_data in the class DataTransformer to save the cell data as a binary file. The method save_data accepts one parameter, which represents the path to the file to save.

[0033] Step 1.4: Create a parallel computing module Parallelizer to accelerate the computation of instance methods of the DataTransformer class. The Parallelizer module has a property that represents the number of parallel processes.

[0034] Step 1.5: Define a method called parallelize in the Parallelizer module to process multiple data files in parallel and save the results as a batch of binary files. The parallelize method accepts two parameters, which represent the list of data files and the path to save the files.

[0035] In the above scheme, step 2 specifically includes the following steps:

[0036] Step 2.1: Define a class Filter to perform bandpass filtering on the EEG or muscle signals set by ch_names in Step 1. This class Filter has four attributes: low_freq, high_freq, sampling_rate, and order, which represent the low cutoff frequency, high cutoff frequency, sampling frequency, and filter order of the filter, respectively.

[0037] Step 2.2: Define a method get_coefficients in the Filter class to generate the filter coefficients and return an object representing the filter coefficients;

[0038] Step 2.3: Define a method filter_data in the Filter class to filter the data and return the filtered data. The filter_data method accepts one parameter, which represents the data to be filtered. The filter_data method calls the get_coefficients method to generate the filter coefficients and uses them to filter the data.

[0039] Step 2.4: Define a class Spectrogram to perform short-time Fourier transform on the EEG or muscle signal data set by ch_names in Step 1 and generate time-frequency plot data. The class Spectrogram has an attribute that represents the path of the binary file.

[0040] Step 2.5: Define a method `transform_data` in the `Spectrogram` class to perform a short-time Fourier transform on all signal channels in the binary file and return a list containing the time-frequency plot data of all channels. The `transform_data` method accepts four parameters, representing the low cutoff frequency, high cutoff frequency, sampling frequency, and filter order, respectively. The `transform_data` method creates an instance of the `Filter` class and uses it to perform bandpass filtering on each channel. Then, it uses the short-time Fourier transform function to perform a short-time Fourier transform on the filtered data and calculates the absolute value of the amplitude. Finally, it uses an image data generation function to process the absolute value of the amplitude to generate the time-frequency plot data.

[0041] Step 2.6: Create a parallel computing module Parallelizer to accelerate the computation of instance methods of the Spectrogram class. The Parallelizer module has a property that represents the number of parallel processes.

[0042] Step 2.7: Define a method `parallelize` in the `Parallelizer` module to process multiple binary files in parallel and save the results as batch time-frequency plot data. The `parallelize` method accepts five parameters, which represent the list of binary files, the path to save the files, the low cutoff frequency of the filter, the high cutoff frequency of the filter, and the filter order. The `parallelize` method creates multiple instances of the `Spectrogram` class and uses them to perform a short-time Fourier transform on each binary file, and saves the results as batch time-frequency plot data.

[0043] In the above scheme, step 3 specifically includes the following steps:

[0044] Step 3.1: Define a class FileCopier for copying files. The FileCopier class has two attributes: directory and ch_names, which represent the path of the source folder and the channel name of the EEG signal to be selected, respectively.

[0045] Step 3.2: Define a method copy_files in the class FileCopier to iterate through all files in the source folder and copy them to the target folder based on the filename. The method copy_files has one parameter, which represents the path of the target folder.

[0046] Step 3.2: In the copy_files method, use the os.listdir function to get the filenames of all files in the source folder and store them as a list. Then, iterate through each filename in the list, use the filename.endswith function and the '1' in filename statement to check if the filename meets the copying conditions, and use the shutil.copy function and os.path.join function to copy the files that meet the conditions to the target folder.

[0047] In the above scheme, step 4 specifically includes the following steps:

[0048] Step 4.1: Define a class NetConcat for a deep convolutional neural network model that simultaneously processes EEG and EMG signals. This class inherits from the nn.Module class and has two sub-networks, deepEEGnet and deepEMGnet, as attributes. deepEEGnet is the network used to process EEG signals, and the input to this network is the time-frequency map of the EEG signals processed by Spectrogram in Step 2. deepEMGnet is the network used to process EMG signals, and the input to this network is the time-frequency map of the muscle signals processed by Spectrogram in Step 2.

[0049] Both deepEEGnet and deepEMGnet use the GELU function and both use Bottleneck as the basic unit. The network formulas are as follows:

[0050] deepEEGnet()=ResNet(Bottleneck,[3,4,6,3])

[0051] deepEMGnet()=ResNet(Bottleneck,[3,4,6,3])

[0052] Wherein, ResNet represents the ResNet network formula defined in the subsequent part, Bottleneck represents the Bottleneck formula defined in the subsequent part, and [3, 4, 6, 3] represents the number of Bottlens in each residual layer;

[0053] Bottleneck defines a convolutional layer with a kernel size of 3x3, a convolutional layer with a kernel size of 1x1, and a bottleneck block consisting of three convolutional layers and a residual connection. The Bottleneck structure formula is as follows:

[0054] Bottleneck(x)=GELU(Conv1×1(GELU(Conv3×3(Conv1×1(x)))))+x)

[0055] Where Convk×k represents a 2D convolution with a kernel size of k, and GELU() represents the activation function;

[0056] The ResNet network formula is as follows:

[0057] ResNet(x)

[0058] =FC(AvgPool)

[0059] (Layer4(Layer3(Layer2(Layer1(MaxPool(GELU(GELU(BN(Conv7×7(x))))))))))

[0060] Where Convk×k represents a 2D convolution with a kernel size of k, BN() represents batch normalization, GELU() represents the activation function, MaxPool() represents max pooling, Layeri represents the i-th residual layer, AvgPool() represents adaptive average pooling, and FC() represents a fully connected layer.

[0061] The formula for merging a deepEEGnet-type network and a deepEEGnet-type network is as follows:

[0062] NetConcat=concat{deepEEGent,deepEEGent};

[0063] Step 4.2: In the __init__ method of the NetConcat class, accept the sampling frequency, the number of filters in the temporal convolutional layer, the momentum of the batch normalization layer, and the model version as parameters, and call the __init__ method of the parent class using the super function. Then, create two sub-networks using the deepEEGnet and deepEMGnet classes and pass in the same parameters.

[0064] Step 4.3: In the `forward` method of the `NetConcat` class, accept the input data as parameters and implement the forward propagation of the model using the following steps:

[0065] Step 4.3.1: Pass the input data through the EEG time-frequency data subnetwork to obtain the time-frequency characteristics of the EEG signal;

[0066] Step 4.3.2: Pass the input data through the muscle time-frequency data sub-network to obtain the time-frequency characteristics of the electromyographic signal;

[0067] Step 4.3.3: Concatenate the two features together in the first dimension to obtain the final output feature and return the fused feature.

[0068] In the above scheme, step 5 specifically includes the following steps:

[0069] Step 5.1: Define a class Trainer for training and validating the model. This class Trainer has four attributes: model, data_path, preprocess_method, and device, which represent the model object, the path to the data folder, the data preprocessing method, and the computing device, respectively.

[0070] Step 5.2: Define a method load_data in the Trainer class to load image data from a folder and return a dataset object containing images and labels. The load_data method calls a function that uses a variable to specify the data preprocessing method.

[0071] Step 5.3: Define a method split_data in the Trainer class to randomly split the dataset into two subsets, one for training and one for validation. The split_data method accepts two parameters, representing the split ratio. The split_data method calls a function and returns two dataset objects.

[0072] Step 5.4: Define a method get_dataloader in the Trainer class to convert two datasets into iterable data loader objects for training and validation, respectively. The method accepts one parameter, which represents the batch size. The get_dataloader method calls a function to set the order of the training data, set the number of parallel data loading processes, and returns two data loader objects.

[0073] Step 5.5: Define a method `train_and_validate` in this class to train and validate the model and save the optimal model parameters. The `train_and_validate` method accepts three parameters, representing the number of training epochs, the learning rate, and the path to the model parameter file. The `train_and_validate` method implements model training and validation using the following steps:

[0074] Step 5.5.1: Define a variable to record the best accuracy on the validation set, with an initial value of 0.0;

[0075] Step 5.5.2: Call the load_data, split_data, and get_dataloader methods to obtain the data loader objects for training and validation;

[0076] Step 5.5.3: Define an optimizer object to update model parameters;

[0077] Step 5.5.4: Define a loss function object to calculate the loss between the output and the label;

[0078] Step 5.5.5: Iterate through multiple training cycles using a loop. For each cycle, perform the following operations.

[0079] Step 5.5.5.1: Define a variable to accumulate the training loss, with an initial value of 0.0;

[0080] Step 5.5.5.2: Iterate through each batch in the training data loader using a loop, for each...

[0081] Batch, perform the following operations:

[0082] Step 5.5.5.2.1: Obtain images and labels from the batch;

[0083] Step 5.5.5.2.2: Transfer images and tags to the device;

[0084] Step 5.5.5.2.3: Clear the gradient cache;

[0085] Step 5.5.5.2.4: Transfer the model to the device as well;

[0086] Step 5.5.5.2.5: Calculate the output corresponding to the image;

[0087] Step 5.5.5.2.6: Calculate the loss between the output and the label;

[0088] Step 5.5.5.2.7: Calculate the gradient of the loss with respect to the model parameters;

[0089] Step 5.5.5.2.8: Update model parameters;

[0090] Step 5.5.5.2.9: Add the loss value to the training loss;

[0091] Step 5.5.5.2.10: Disable gradient calculation to save memory and speed up computation.

[0092] In the above scheme, step 6 specifically includes the following steps:

[0093] Step 6.1: Divide the EEG and EMG data of patients who need to be predicted for West syndrome attacks according to the set time window method, and perform data transformation according to the steps in Step 2 to obtain multi-channel time-frequency domain data.

[0094] Step 6.2: For the multi-channel time-frequency domain data in 6.1, use the best model parameters saved in step 5 to classify and give the prediction results, namely the annotation of abnormal EEG signal data and electromyography signal data.

[0095] Step 6.3: Annotate the abnormal EEG signal data and electromyography signal data obtained for each predicted time period and write them into the annotation file.

[0096] Because the present invention employs the above-mentioned technical means, it has the following beneficial effects:

[0097] (1) The technical effect is that it can quickly convert EEG data files into a format suitable for data analysis while preserving EEG signal channels and event information. The advantage of this invention is that it can save data processing time and space, and improve the accuracy and stability of data processing. The original data is a multi-channel, high-frequency floating-point number that occupies a lot of space. By converting it into 2D image data through steps 1 and 2, space can be saved significantly. For example, 30G of original data can be transformed into 400MB of time-frequency image data. At the same time, the image uses the network designed in step 4. The number of parameters of the 2D network for processing the image is larger than that of the 1D convolution for directly processing the original data. Therefore, the accuracy is high and the fluctuation is small.

[0098] (2) Data augmentation techniques were used to significantly increase the amount of usable data. The signal was segmented in step 1, and the segments of the feature regions could overlap in step 1.2. Furthermore, in steps 2 and 3, the number of EEG channels and electromyography signal channels set in step 1.1 were used instead of single-channel EEG, thereby increasing the amount of data and solving the problem of insufficient seizure data.

[0099] (3) A time-frequency model for EEG signals was constructed for West syndrome detection, and a large number of features were extracted from it using an end-to-end model instead of manually extracted features. The features of EEG signals and electromyography signals were extracted by the deep EEG net and deep EMG net designed in step 4, respectively. Both networks contain a large number of convolutional layers. By using convolutional layers, features from low-dimensional to high-dimensional data can be automatically extracted, thus eliminating the need for manual creation of useful features by experts. This solves the problem of the difficulty in promoting methods limited by professional knowledge, thereby reducing the amount of manual operation and lowering the threshold for users.

[0100] (4) A multi-scale, multi-branch network was used to integrate multimodal features, and both EEG and EMG data were used. The NetConcat network designed in step 4 was used to integrate and fuse EEG and EMG signals for judgment. Data from both EEG and EMG modalities can be used simultaneously. By using EMG signals as an auxiliary means, the misjudgment problem of using EEG signals alone was solved, thereby increasing the robustness of the model.

[0101] (5) This invention uses a multi-scale, multi-branch network to integrate multimodal features, and simultaneously uses EEG and EMG data. This distinction may play an important role in improving the robustness and generalization of West syndrome detection.

[0102] On the one hand, in actual clinical practice, this type of disease can present as either simple EEG seizures without movement or complete seizures involving movement throughout the body. Simple EEG seizures pose a lower risk to patients and should be excluded. These two types cannot be directly separated using EEG and related analytical methods in research, but they do differ in their electromyographic (EMG) signals. Introducing EMG signals is intended to classify these cases. Therefore, interference can be eliminated, improving the robustness of this network.

[0103] On the other hand, in addition to the characteristics of diseases contained in EEG signals, ESM signals also contain a large number of characteristics and differences between seizures and non-seizures. By using two different modalities of EEG and ESM signals, the number of features is increased, which can reduce false positives and false negatives, thereby improving robustness and generalization ability.

[0104] (6) A temporal frequency model for detecting West syndrome was developed, and a large number of features were extracted from it using an end-to-end model instead of manually extracted features. This method only requires converting the raw data into a two-dimensional image through steps 1 and 2, and then automatically extracting features using the convolutional layers of a neural network, specifically through the NetConcat network in step 4 and the two sub-networks deepEEGnet and deepEMGnet that make up NetConcat. This difference may play an important role in improving the accuracy and sensitivity of West syndrome detection.

[0105] (7) Data augmentation techniques significantly increase the amount of usable data. This difference may play an important role in improving the stability and reliability of West syndrome detection. The signal is segmented in step 1, and in step 1.2, the segments of the feature regions can overlap. Furthermore, steps 2 and 3 utilize the ch_names EEG and electromyography signal channels set in step 1.1, instead of single-channel EEG, thereby increasing the amount of data and addressing the problem of insufficient seizure data. The increased data volume can effectively enhance the performance of the neural network and improve accuracy.

[0106] (8) The technical advantage is that by adding a parallel computing module, EEG data files can be quickly converted into a format suitable for data analysis while preserving the EEG signal channels and event information. This difference may help save data processing time and space, and improve the accuracy and stability of data processing. Attached Figure Description

[0107] Figure 1 Step-by-step flowchart;

[0108] Figure 2 A flowchart for a neural network to determine the outcome of a seizure. Detailed Implementation

[0109] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.

[0110] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.

[0111] This invention provides a method for analyzing electroencephalogram (EEG) signal data based on multimodal deep learning, comprising the following steps:

[0112] Step 1: Preprocess and segment the raw EEG signal data and raw EMG signal data, and then convert and store the segmented data to obtain binary data.

[0113] Step 2: For the binary data obtained in Step 1, separate the data according to the set EEG signal and EMG signal channels, perform bandpass filtering and short-time Fourier transform on each signal channel, and convert and store the filtered and transformed time-frequency domain data to obtain multi-channel time-frequency domain data.

[0114] Step 3: Randomize the multi-channel time-frequency domain data obtained in Step 2 to obtain training set data, validation set data, and test set data;

[0115] Step 4: Set up the deep learning neural network models, including a deep convolutional neural network model for EEG signal processing, a deep convolutional neural network model for EMG signal processing, and a deep convolutional neural network model for simultaneously processing EEG and EMG signals. Output the fusion of the two modalities. The first modal input is the time-frequency map data obtained by short-time Fourier transform of EEG data, and the other modal input is the time-frequency map data obtained by short-time Fourier transform of EMG signal data.

[0116] On the one hand, in actual clinical practice, this type of disease can present as either simple EEG seizures without movement or complete seizures involving movement throughout the body. Simple EEG seizures pose a lower risk to patients and should be excluded. These two types cannot be directly separated in research using EEG and related analytical methods, but they do differ in their electromyographic (EMG) signals. Introducing EMG signals helps classify these cases. On the other hand, besides EEG signals containing disease characteristics, EMG signals also contain numerous characteristics and distinctions between seizures and non-seizure states. Introducing EMG signals increases the number of features, thereby reducing false positives and false negatives.

[0117] The first modal input is time-frequency data obtained by short-time Fourier transform of EEG data, and the other modal input is time-frequency data obtained by short-time Fourier transform of electromyography (EMG) signal data. EEG data and EMG signals come from different body locations and their variation patterns are different, so they are not equivalent and are two different modalities.

[0118] Step 5: Using the training set data, validation set data, and test set data obtained in Step 3, use the neural network model from Step 4 to train a classification model for whether West syndrome is flare-up, and obtain the optimal model parameters.

[0119] Step 6: For the EEG and EMG data of patients who need to predict West syndrome attacks, process the data using the technique in Step 2, then use the optimal model parameters saved in Step 5 to classify and obtain the annotations of abnormal EEG and EMG data for each time period, and save them as annotation files.

[0120] In the above technical solution, step 1 specifically includes the following steps:

[0121] Step 1.1: Define a class DataTransformer to transform and extract raw EEG signal data. The DataTransformer class has two attributes: data_path and ch_names, which represent the location of the data file and the name of the required signal channel, respectively.

[0122] Step 1.2: Define a method transform_data in the class DataTransformer to read the data file, retain the specified signal channels, extract event information, and split the data into units according to the event type. The method transform_data returns a list containing all unit data.

[0123] Step 1.3: Define a method save_data in the class DataTransformer to save the cell data as a binary file. The method save_data accepts one parameter, which represents the path to the file to save.

[0124] Step 1.4: Create a parallel computing module Parallelizer to accelerate the computation of instance methods of the DataTransformer class. The Parallelizer module has a property that represents the number of parallel processes.

[0125] Step 1.5: Define a method called parallelize in the Parallelizer module to process multiple data files in parallel and save the results as a batch of binary files. The parallelize method accepts two parameters, which represent the list of data files and the path to save the files.

[0126] In the above technical solution, step 2 specifically includes the following steps:

[0127] Step 2.1: Define a class Filter for bandpass filtering of EEG data. This class Filter has four attributes: low_freq, high_freq, sampling_rate, and order, which represent the low cutoff frequency, high cutoff frequency, sampling frequency, and filter order, respectively.

[0128] Step 2.2: Define a method get_coefficients in the Filter class to generate the filter coefficients and return an object representing the filter coefficients;

[0129] Step 2.3: Define a method filter_data in the Filter class to filter the data and return the filtered data. The filter_data method accepts one parameter, which represents the data to be filtered. The filter_data method calls the get_coefficients method to generate the filter coefficients and uses them to filter the data.

[0130] Step 2.4: Define a class Spectrogram to perform short-time Fourier transform on EEG signal data and generate time-frequency plot data. The class Spectrogram has an attribute that represents the path of the binary file.

[0131] Step 2.5: Define a method `transform_data` in the `Spectrogram` class (the `Spectrogram` class performs short-time Fourier transform, and the `Filter` class performs bandpass filtering; bandpass filtering must be performed before short-time Fourier transform. Here, the `Spectrogram` class uses an instance of the `Filter` class). This method performs short-time Fourier transform on all EEG channels in the binary file and returns a list containing time-frequency plot data for all channels. The `transform_data` method accepts four parameters, representing the low cutoff frequency, high cutoff frequency, sampling frequency, and filter order, respectively. The `transform_data` method creates an instance of the `Filter` class and uses it to perform bandpass filtering on each channel. Then, it uses the short-time Fourier transform function to perform short-time Fourier transform on the filtered data and calculates the absolute value of the amplitude. Finally, it uses an image data generation function to process the absolute value of the amplitude to generate time-frequency plot data.

[0132] Step 2.6: Create a parallel computing module Parallelizer to accelerate the computation of instance methods of the Spectrogram class. The Parallelizer module has a property that represents the number of parallel processes.

[0133] Step 2.7: Define a method `parallelize` in the `Parallelizer` module to process multiple binary files in parallel and save the results as batch time-frequency plot data. The `parallelize` method accepts five parameters, which represent the list of binary files, the path to save the files, the low cutoff frequency of the filter, the high cutoff frequency of the filter, and the filter order. The `parallelize` method creates multiple instances of the `Spectrogram` class and uses them to perform a short-time Fourier transform on each binary file, and saves the results as batch time-frequency plot data.

[0134] In the above technical solution, step 3 specifically includes the following steps:

[0135] Step 3.1: Define a class FileCopier for copying files. The FileCopier class has two attributes: directory and ch_names, which represent the path of the source folder and the channel name of the EEG signal to be selected, respectively.

[0136] Step 3.2: Define a method copy_files in the class FileCopier to iterate through all files in the source folder and copy them to the target folder based on the filename. The method copy_files has one parameter, which represents the path of the target folder.

[0137] Step 3.2: In the copy_files method, use the os.listdir function to get the filenames of all files in the source folder and store them as a list. Then, iterate through each filename in the list, use the filename.endswith function and the '1' in filename statement to check if the filename meets the copying conditions, and use the shutil.copy function and os.path.join function to copy the files that meet the conditions to the target folder.

[0138] In the above technical solution, step 4 specifically includes the following steps:

[0139] Step 4.1: Define a class `NetConcat` for a deep convolutional neural network model that simultaneously processes EEG and EMG signals. This class inherits from the `nn.Module` class (this neural network is based on the general PyTorch framework; `nn.Module` is a class already encapsulated within PyTorch, and all neural network architectures designed using the PyTorch framework need to use this class), and has two sub-networks, `deepEEGnet` and `deepEMGnet`, as attributes. (`deepEEGnet` is the network for processing EEG signals; its input is the time-frequency map of the EEG signals processed by Spectrogram in Step 2. `deepEMGnet` is the network for processing EMG signals; its input is the time-frequency map of the muscle signals processed by Spectrogram in Step 2.)

[0140] Both deepEEGnet and deepEMGnet use the GELU function to replace the original ReLU function in the resnet50 structure. Using the GELU function can avoid the neuron death problem caused by the ReLU function, that is, the problem that the activation function always outputs 0.

[0141] Define a deepEEGnet type network and a deepEMGnet type network, both using Bottleneck as the basic unit. The network formulas are as follows:

[0142] deepEEGnet()=ResNet(Bottleneck,[3,4,6,3])

[0143] deepEMGnet()=ResNet(Bottleneck,[3,4,6,3])

[0144] Wherein, ResNet represents the ResNet network formula defined in the subsequent part, Bottleneck represents the Bottleneck formula defined in the subsequent part, and [3, 4, 6, 3] represents the number of Bottlens in each residual layer.

[0145] Bottleneck defines a convolutional layer with a kernel size of 3x3, a convolutional layer with a kernel size of 1x1, and a bottleneck block consisting of the three convolutional layers and a residual connection. The Bottleneck structure formula is as follows:

[0146] Bottleneck(x) = GELU(Conv1×1(GELU(Conv3×3(Conv1×1(x)))) + x) where Convk×k represents a 2D convolution with kernel size k, and GELU() represents the activation function;

[0147] Convolutional Layer: A convolutional layer is a commonly used layer type in deep learning. It is mainly used to extract features from input data.

[0148] GELU Layer (Gaussian Error Linear Unit Layer): The GELU layer is an activation function. It provides a non-linear mapping with a smooth curve shape. The GELU activation function performs well in some deep learning tasks and can help the network learn more complex patterns and representations.

[0149] The ResNet network formula is as follows:

[0150] ResNet(x)

[0151] =FC(AvgPool)

[0152] (Layer4(Layer3(Layer2(Layer1(MaxPool(GELU(GELU(BN(Conv7×7(x))))))))))

[0153] Where Convk×k represents a 2D convolution with a kernel size of k, BN() represents batch normalization, GELU() represents the activation function, MaxPool() represents max pooling, Layeri represents the i-th residual layer, AvgPool() represents adaptive average pooling, and FC() represents a fully connected layer.

[0154] Convolutional Layer: A convolutional layer is a commonly used layer type in deep learning. It is primarily used to extract features from input data. By using convolutional kernels (also called filters) to perform convolution operations on the input data, convolutional layers can capture the local spatial features of the input data. Convolutional layers are widely used in computer vision tasks such as image classification, object detection, and image segmentation.

[0155] Batch Normalization (BN) Layer: BN layers are used to address the internal covariate shift problem during neural network training. By normalizing the input in each mini-batch of training samples, it ensures that the inputs of each layer in the network maintain a distribution with a mean of 0 and a variance of 1. BN layers can accelerate the training process of neural networks and improve the robustness and generalization ability of the model.

[0156] Fully Connected (FC) Layer: Also known as a fully connected layer or densely connected layer, in an FC layer, each neuron is connected to all neurons in the previous layer. FC layers are typically used to combine and map high-level features of the network for final classification tasks involving EEG or EMG signals.

[0157] GELU Layer (Gaussian Error Linear Unit Layer): The GELU layer is an activation function. It provides a non-linear mapping with a smooth curve shape. The GELU activation function performs well in some deep learning tasks and can help the network learn more complex patterns and representations.

[0158] MaxPool Layer: The MaxPool layer is a downsampling operation commonly used in convolutional neural networks. It divides the input feature map into non-overlapping rectangular regions and selects the maximum value from each region as the output. The MaxPool layer helps reduce the spatial dimensionality of the feature map while preserving the most salient features from the input feature map.

[0159] AvgPool (Average Pooling Layer): Similar to the MaxPoo1 layer, the AvgPool layer is also a downsampling operation. However, unlike the MaxPoo1 layer, the AvgPool layer calculates the average value of each region as its output, rather than the maximum value. The AvgPool layer can reduce the spatial dimensionality of the feature map.

[0160] The residual layer is a key component of a deep residual neural network (ResNet). Its role is to address the vanishing and exploding gradient problems during the training process of deep neural networks, thereby improving the training efficiency of deep networks. By stacking this layer, the neural network can extract more and higher-dimensional features, thus increasing its performance.

[0161] The multimodal network used in this paper consists of two DeepEEGNet-like networks, which are then fused to obtain a more powerful feature classifier. The network formula is shown below:

[0162] NetConcat=concat(deepEEGent,deepEEGent};

[0163] Step 4.2: In the __init__ method of the NetConcat class, accept the sampling frequency, the number of filters in the temporal convolutional layer, the momentum of the batch normalization layer, and the model version as parameters, and call the __init__ method of the parent class using the super function. Then, create two sub-networks using the deepEEGnet and deepEMGnet classes and pass in the same parameters.

[0164] Step 4.3: In the `forward` method of the `NetConcat` class, accept the input data as parameters and implement the forward propagation of the model using the following steps:

[0165] Step 4.3.1: Pass the input data through the EEG time-frequency data subnetwork to obtain the time-frequency characteristics of the EEG signal;

[0166] Step 4.3.2: Pass the input data through the muscle time-frequency data sub-network to obtain the time-frequency characteristics of the electromyographic signal;

[0167] Step 4.3.3: Concatenate the two features together in the first dimension to obtain the final output feature and return the fused feature.

[0168] In the above technical solution, step 5 specifically includes the following steps:

[0169] Step 5.1: Define a class Trainer for training and validating the model. This class Trainer has four attributes: model, data_path, preprocess method, and device, which represent the model object, the path to the data folder, the data preprocessing method, and the computing device, respectively.

[0170] Step 5.2: Define a method load_data in the Trainer class to load image data from a folder and return a dataset object containing images and labels. The load_data method calls a function that uses a variable to specify the data preprocessing method.

[0171] Step 5.3: Define a method split_data in the Trainer class to randomly split the dataset into two subsets, one for training and one for validation. The split_data method accepts two parameters, representing the split ratio. The split_data method calls a function and returns two dataset objects.

[0172] Step 5.4: Define a method get_dataloader in the Trainant class to convert two datasets into iterable data loader objects for training and validation, respectively. The method accepts one parameter, which represents the batch size. The get_dataloader method calls a function to set the order of the training data, set the number of parallel data loading processes, and returns two data loader objects.

[0173] Step 5.5: Define a method `train_and_validate` in this class to train and validate the model and save the optimal model parameters. The `train_and_validate` method accepts three parameters, representing the number of training epochs, the learning rate, and the path to the model parameter file. The `train_and_validate` method implements model training and validation using the following steps:

[0174] Step 5.5.1: Define a variable to record the best accuracy on the validation set, with an initial value of 0.0;

[0175] Step 5.5.2: Call the load_data, split_data, and get_dataloader methods to obtain the data loader objects for training and validation;

[0176] Step 5.5.3: Define an optimizer object to update model parameters;

[0177] Step 5.5.4: Define a loss function object to calculate the loss between the output and the label;

[0178] Step 5.5.5: Use a loop to iterate through multiple training cycles. For each cycle, perform the following operations: Step 5.5.5.1: Define a variable to accumulate the training loss, with an initial value of 0.0;

[0179] Step 5.5.5.2: Iterate through each batch in the training data loader using a loop, and for each batch, perform the following operations:

[0180] Step 5.5.5.2.1: Obtain images and labels from the batch;

[0181] Step 5.5.5.2.2: Transfer images and tags to the device;

[0182] Step 5.5.5.2.3: Clear the gradient cache;

[0183] Step 5.5.5.2.4: Transfer the model to the device as well;

[0184] Step 5.5.5.2.5: Calculate the output corresponding to the image;

[0185] Step 5.5.5.2.6: Calculate the loss between the output and the label;

[0186] Step 5.5.5.2.7: Calculate the gradient of the loss with respect to the model parameters;

[0187] Step 5.5.5.2.8: Update model parameters;

[0188] Step 5.5.5.2.9: Add the loss value to the training loss;

[0189] Step 5.5.5.2.10: Disable gradient calculation to save memory and speed up computation.

[0190] In the above technical solution, step 6 specifically includes the following steps:

[0191] Step 6.1: Divide the EEG and EMG data of patients who need to be predicted for West syndrome attacks according to the set time window method, and perform data transformation according to the steps in Step 2 to obtain multi-channel time-frequency domain data.

[0192] Step 6.2: For the multi-channel time-frequency domain data in 6.1, use the best model parameters saved in step 5 to classify and give the prediction results, namely the annotation of abnormal EEG signal data and electromyography signal data.

[0193] Step 6.3: Annotate the abnormal EEG signal data and electromyography signal data obtained for each predicted time period and write them into the annotation file.

[0194] Other prior art materials that may help in understanding the proposal of this application

[0195] A bandpass filter is a filter that allows signals within a specific frequency range or bandwidth to pass through while attenuating or rejecting signals outside that range. Continuous-time bandpass filtering: For a continuous-time signal x(t), its bandpass filtered output signal y(t) is:

[0196]

[0197] Here, h(t) is the impulse response of a bandpass filter, which can be composed of a cascaded or parallel low-pass filter and a high-pass filter. For example, if h(t) is composed of a cascaded low-pass filter and a high-pass filter, then:

[0198] h(t) = h L (t)*h H (t)

[0199] Where * represents convolution operation, h L (t) and h H (t) represents the impulse response of the low-pass filter and the high-pass filter, respectively.

[0200] The Short-Time Fourier Transform (STFT) is a Fourier correlation transform used to determine the frequency and phase content of a signal in a local time region. The formula for calculating the Short-Time Fourier Transform can take several forms:

[0201]

[0202] Here, w(t) is a window function, typically a finite-length function centered at zero, such as a Hanning window or a Gaussian window. S(τ, ω) is a complex function representing the phase and amplitude of the signal in time and frequency.

Claims

1. A multi-modal deep learning based electroencephalogram data signal analysis method, characterized in that, Includes the following steps: Step 1: Preprocess and segment the raw EEG signal data and raw EMG signal data, and then convert and store the segmented data to obtain binary data. Step 2: For the binary data obtained in Step 1, separate the data according to the set EEG signal and EMG signal channels, perform bandpass filtering and short-time Fourier transform on each signal channel, and convert and store the filtered and transformed time-frequency domain data to obtain multi-channel time-frequency domain data. Step 3: Randomize the multi-channel time-frequency domain data obtained in Step 2 to obtain training set data, validation set data, and test set data; Step 4: Configure the deep learning neural network models, including a deep convolutional neural network model for EEG signal processing, a deep convolutional neural network model for EMG signal processing, and a deep convolutional neural network model for simultaneously processing EEG and EMG signals, and output the fused results of the two modalities. Step 4 specifically includes the following steps: Step 4.1: Define a class NetConcat for a deep convolutional neural network model that simultaneously processes EEG and EMG signals. This class inherits from the nn.Module class and has two sub-networks, deepEEGnet and deepEMGnet, as attributes. deepEEGnet is the network for processing EEG signals, and its input is the time-frequency map of EEG signals processed by Spectrogram in Step 2. deepEMGnet is the network for processing EMG signals, and its input is the time-frequency map of muscle signals processed by Spectrogram in Step 2. Both deepEEGnet and deepEMGnet use the GELU function and both use Bottleneck as the basic unit. Step 4.2: In the __init__ method of the NetConcat class, accept the sampling frequency, the number of filters in the temporal convolutional layer, the momentum of the batch normalization layer, and the model version as parameters, and call the __init__ method of the parent class using the super function. Then, create two sub-networks using the deepEEGnet and deepEMGnet classes and pass in the same parameters. Step 4.3: In the `forward` method of the `NetConcat` class, accept the input data as parameters and implement the forward propagation of the model using the following steps: Step 4.3.1: Pass the input data through the EEG time-frequency data subnetwork to obtain the time-frequency characteristics of the EEG signal; Step 4.3.2: Pass the input data through the muscle time-frequency data sub-network to obtain the time-frequency characteristics of the electromyographic signal; Step 4.3.3: Concatenate the two features together in the first dimension to obtain the final output feature, and return the fused feature; Step 5: Using the training set data, validation set data, and test set data obtained in Step 3, use the neural network model from Step 4 to train a classification model for whether West syndrome is flare-up, and obtain the optimal model parameters. Step 6: For the EEG and EMG data of patients who need to predict West syndrome attacks, process the data using the technique in Step 2, then use the optimal model parameters saved in Step 5 to classify and obtain the annotations of abnormal EEG and EMG data for each time period, and save them as annotation files.

2. The electroencephalogram data signal analysis method based on multi-modal deep learning according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 1.1: Define a class DataTransformer to transform and extract raw EEG signal data. The DataTransformer class has two attributes, data_path and ch_names, which represent the location of the data file and the name of the required EEG or muscle signal channel, respectively. Step 1.2: Define a method transform_data in the class DataTransformer to read the data file, retain the specified signal channels, extract event information, and split the data into units according to the event type. The method transform_data returns a list containing all unit data. Step 1.3: Define a method save_data in the class DataTransformer to save the cell data as a binary file. The method save_data accepts one parameter, which represents the path to the file to save. Step 1.4: Create a parallel computing module Parallelizer to accelerate the computation of instance methods of the DataTransformer class. The Parallelizer module has a property that represents the number of parallel processes. Step 1.5: Define a method called parallelize in the Parallelizer module to process multiple data files in parallel and save the results as a batch of binary files. The parallelize method accepts two parameters, which represent the list of data files and the path to save the files.

3. The electroencephalogram data signal analysis method based on multi-modal deep learning according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Define a class Filter for bandpass filtering of EEG data. This class Filter has four attributes: low_freq, high_freq, sampling_rate, and order, which represent the low cutoff frequency, high cutoff frequency, sampling frequency, and filter order, respectively. Step 2.2: Define a method get_coefficients in the Filter class to generate the filter coefficients and return an object representing the filter coefficients; Step 2.3: Define a method filter_data in the Filter class to filter the data and return the filtered data. The filter_data method accepts one parameter, which represents the data to be filtered. The filter_data method calls the get_coefficients method to generate the filter coefficients and uses them to filter the data. Step 2.4: Define a class Spectrogram to perform short-time Fourier transform on EEG signal data and generate time-frequency plot data. The class Spectrogram has an attribute that represents the path of the binary file. Step 2.5: Define a method `transform_data` in the `Spectrogram` class to perform a short-time Fourier transform on all EEG channels in the binary file and return a list containing time-frequency plot data for all channels. The `transform_data` method accepts four parameters, representing the low cutoff frequency, high cutoff frequency, sampling frequency, and filter order, respectively. The `transform_data` method creates an instance of the `Filter` class and uses it to perform bandpass filtering on each channel. Then, it uses the short-time Fourier transform function to perform a short-time Fourier transform on the filtered data and calculates the absolute value of the amplitude. Finally, it uses an image data generation function to process the absolute value of the amplitude to generate time-frequency plot data. Step 2.6: Create a parallel computing module Parallelizer to accelerate the computation of instance methods of the Spectrogram class. The Parallelizer module has a property that represents the number of parallel processes. Step 2.7: Define a method `parallelize` in the `Parallelizer` module to process multiple binary files in parallel and save the results as batch time-frequency plot data. The `parallelize` method accepts five parameters, which represent the list of binary files, the path to save the files, the low cutoff frequency of the filter, the high cutoff frequency of the filter, and the filter order. The `parallelize` method creates multiple instances of the `Spectrogram` class and uses them to perform a short-time Fourier transform on each binary file, and saves the results as batch time-frequency plot data.

4. A method for analyzing electroencephalogram (EEG) signal data based on multimodal deep learning according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: Define a class FileCopier for copying files. The FileCopier class has two attributes: directory and ch_names, which represent the path of the source folder and the channel name of the EEG signal to be selected, respectively. Step 3.2: Define a method copy_files in the class FileCopier to iterate through all files in the source folder and copy them to the target folder based on the filename. The method copy_files has one parameter, which represents the path of the target folder. Step 3.2: In the copy_files method, use the os.listdir function to get the filenames of all files in the source folder and store them as a list. Then, iterate through each filename in the list, use the filename.endswith function and the '1' in filename statement to check if the filename meets the copying conditions, and use the shutil.copy function and os.path.join function to copy the files that meet the conditions to the target folder.

5. A method for analyzing electroencephalogram (EEG) signal data based on multimodal deep learning according to claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5.1: Define a class Trainer for training and validating the model. This class Trainer has four attributes: model, data_path, preprocess_method, and device, which represent the model object, the path to the data folder, the data preprocessing method, and the computing device, respectively. Step 5.2: Define a method load_data in the Trainer class to load image data from a folder and return a dataset object containing images and labels. The load_data method calls a function that uses a variable to specify the data preprocessing method. Step 5.3: Define a method split_data in the class Trainer to randomly split the dataset into two subsets, one for training and one for validation. The method split_data accepts two parameters, which represent the split ratio. The method split_data calls a function and returns two dataset objects. Step 5.4: Define a method get_dataloader in the Trainer class to convert two datasets into iterable data loader objects for training and validation, respectively. The method accepts one parameter, which represents the batch size. The get_dataloader method calls a function to set the order of the training data, set the number of parallel data loading processes, and returns two data loader objects. Step 5.5: Define a method `train_and_validate` in this class to train and validate the model and save the optimal model parameters. The `train_and_validate` method accepts three parameters, representing the number of training epochs, the learning rate, and the path to the model parameter file. The `train_and_validate` method implements model training and validation using the following steps: Step 5.5.1: Define a variable to record the best accuracy on the validation set, with an initial value of 0.0; Step 5.5.2: Call the load_data, split_data, and get_dataloader methods to obtain the data loader objects for training and validation; Step 5.5.3: Define an optimizer object to update model parameters; Step 5.5.4: Define a loss function object to calculate the loss between the output and the label; Step 5.5.5: Iterate through multiple training cycles using a loop. For each cycle, perform the following operations: Step 5.5.5.1: Define a variable to accumulate the training loss, with an initial value of 0.0; Step 5.5.5.2: Iterate through each batch in the training data loader using a loop, and for each batch, perform the following operations: Step 5.5.5.2.1: Obtain images and labels from the batch; Step 5.5.5.2.2: Transfer images and tags to the device; Step 5.5.5.2.3: Clear the gradient cache; Step 5.5.5.2.4: Transfer the model to the device as well; Step 5.5.5.2.5: Calculate the output corresponding to the image; Step 5.5.5.2.6: Calculate the loss between the output and the label; Step 5.5.5.2.7: Calculate the gradient of the loss with respect to the model parameters; Step 5.5.5.2.8: Update model parameters; Step 5.5.5.2.9: Add the loss value to the training loss; Step 5.5.5.2.10: Disable gradient calculation to save memory and speed up computation.

6. A method for analyzing electroencephalogram (EEG) signal data based on multimodal deep learning according to claim 1, characterized in that, Step 6 specifically includes the following steps: Step 6.1: Divide the EEG and EMG data of patients who need to be predicted for West syndrome attacks according to the set time window method, and perform data transformation according to the steps in Step 2 to obtain multi-channel time-frequency domain data. Step 6.2: For the multi-channel time-frequency domain data in 6.1, use the best model parameters saved in step 5 to classify and give the prediction results, namely the annotation of abnormal EEG signal data and electromyography signal data. Step 6.3: Annotate the abnormal EEG signal data and electromyography signal data obtained for each predicted time period and write them into the annotation file.

Images