A self-supervised learning method of Transformer encoder for EEG signal classification tasks

Abstract

The present invention discloses a Transformer encoder self-supervised learning method for an electroencephalogram (EEG) signal classification task. The method targets the scalp EEG signal classification task in a brain-computer interface, introduces a multi-layer wavelet packet reconstruction technique according to the human scalp EEG characteristics, reconstructs the EEG data signal into wavelet signals of different frequency bands, and designs a Transformer encoder-based self-supervised pre-training strategy, so that the encoder can generate a more refined and effective multi-scale refined feature representation for EEG signal data of different frequency bands without a label, improves the encoder's feature representation learning effect on scalp EEG signals and increases interpretability, and can effectively pre-train a feature extractor for the brain-computer interface classification task to better serve downstream classification tasks.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer interface, and more specifically, relates to a Transformer encoder self-supervised learning method for EEG signal classification tasks. Background Art

[0002] A brain-computer interface is a system that provides a direct interaction channel between the brain and external devices. An EEG-based brain-computer interface system refers to a system in which a person wears an EEG cap equipped with electrodes to collect brain wave (EEG, Electroencephalogram) signals through the electrodes, and then analyzes and decodes the EEG signals to determine the state or intention of the brain, perform disease detection, or further control external devices through EEG signals.

[0003] At present, one of the main challenges of brain-computer interface technology is that it is difficult to obtain enough high-quality annotated data in practical application scenarios. This makes it difficult to improve the accuracy of brain-computer interface classification tasks using large-parameter neural networks that have been widely studied in fields such as computer vision and natural language processing. Therefore, self-supervised learning is one of the effective ways to solve this problem by learning latent feature representations from unlabeled data to help downstream tasks classify. Self-supervised learning is mainly divided into pre-training stage and fine-tuning stage. The design of a good pre-training method is the key to extracting effective feature representations from data. Currently, contrastive learning is the most common pre-training method. However, contrastive learning methods are often based on the following two assumptions: (1) the enhanced views of the same data sample should have consistent semantic information in the latent representation space; (2) to construct effective data augmentation methods to learn effective feature representations. Unlike images, texts and other data with intuitive and dense information, the amount of information contained in EEG signals is sparse and not intuitive. Self-supervised pre-training methods based on contrastive learning need to design data augmentation methods for different tasks, which will introduce complex hyperparameter adjustment steps. In addition, bandpass filtering is a common method for EEG signal data enhancement, but since the effective EEG frequency band varies with the task, the signal after filtering out a part of the frequency band cannot guarantee the same semantic information as the original signal. In recent years, in addition to contrastive learning, the autoencoder model is also an emerging type of self-supervised pre-training algorithm, which has received extensive attention and research in fields such as computer vision and natural language processing. Compared with contrastive learning, the simplicity of the autoencoder model and its smaller reliance on data set-specific enhancement engineering make it more suitable for brain-computer interface tasks.

[0004] In EEG-based brain-computer interface tasks, unlike the image data used in computer vision tasks, which has a large amount of spatial redundancy, EEG signals are more nonlinear and have weaker structural self-similarity than image data. Therefore, self-supervised learning methods for image data often face the difficulty of converging the pre-training model loss in EEG tasks, and cannot obtain a good pre-training model, which makes it difficult to implement self-supervised learning algorithms in EEG tasks. Therefore, self-supervised learning methods for image data are difficult to apply to brain-computer interface tasks. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a Transformer encoder self-supervised learning method for EEG signal classification tasks. During the pre-training process, the Transformer encoder can learn a more effective multi-scale refined feature representation, thereby better serving downstream classification tasks.

[0006] To achieve the above object, according to a first aspect of the present invention, a Transformer encoder pre-training method for an EEG signal classification task is provided, comprising:

[0007] S1, perform wavelet packet decomposition and reconstruction on the EEG signal to obtain wavelet signals in multiple frequency bands;

[0008] S2, dividing each wavelet signal into signal segments of equal size, and masking at least one signal segment according to the same random mask sequence;

[0009] S3, inputting each wavelet signal that has completed the mask operation into the corresponding Transformer encoder and decoder for encoding and decoding, summing them according to the sampling points of the EEG signal to obtain the EEG reconstructed signal; using the Euclidean distance between the EEG reconstructed signal and the EEG signal as the pre-training loss, pre-training the Transformer encoder and decoder.

[0010] According to a second aspect of the present invention, a training method for an EEG signal classification model is provided, wherein the EEG signal classification model comprises a pre-trained Transformer encoder obtained by the pre-training method described in the first aspect and a classifier connected thereto in a one-to-one correspondence; the method comprises:

[0011] A1, using the pre-trained Transformer encoder as an initialized feature extractor, inputting each wavelet signal into the feature extractor corresponding to it to obtain a feature vector, and inputting it into the corresponding classifier to obtain a classification probability;

[0012] A2, based on the learnable weights, weighted summation is performed on the classification probabilities of each wavelet signal to obtain a model classification result; the cross entropy loss between the model classification result and the true label of the sample is used as the training loss to train the EEG signal classification model.

[0013] According to a third aspect of the present invention, there is provided an EEG signal classification method, characterized in that it includes: inputting the EEG signal to be classified into a trained EEG signal classification model obtained by using the training method described in the second aspect to obtain a classification result.

[0014] According to a fourth aspect of the present invention, there is provided a Transformer encoder pre-training device for an EEG signal classification task, comprising:

[0015] The wavelet packet decomposition and reconstruction module is used to perform wavelet packet decomposition and reconstruction on the EEG signal to obtain wavelet signals of multiple frequency bands;

[0016] A mask operation module, used for dividing each sub-wave signal into signal segments of equal size, and masking at least one signal segment according to the same random mask sequence;

[0017] The pre-training module is used to input each wavelet signal that has completed the mask operation into the corresponding Transformer encoder and decoder for encoding and decoding, and then sum them according to the sampling points of the EEG signal to obtain the EEG reconstructed signal; the Euclidean distance between the EEG reconstructed signal and the EEG signal is used as the pre-training loss to pre-train the Transformer encoder and decoder.

[0018] According to a fifth aspect of the present invention, there is provided a training device for an EEG signal classification model, wherein the EEG signal classification model comprises a pre-trained Transformer encoder obtained by using the pre-training device according to the fourth aspect and a classifier connected thereto in a one-to-one correspondence; the device comprises:

[0019] The classification probability acquisition module is used to use the pre-trained Transformer encoder as the initialized feature extractor, input each wavelet signal into the feature extractor corresponding to it to obtain a feature vector, and input it into the corresponding classifier to obtain the classification probability;

[0020] The training module is used to perform weighted summation on the classification probability of each wavelet signal based on the learnable weight to obtain the model classification result; and the cross entropy loss between the model classification result and the true label of the sample is used as the training loss to train the EEG signal classification model.

[0021] According to a sixth aspect of the present invention, there is provided an EEG signal classification device, comprising: inputting the EEG signal to be classified into a trained EEG signal classification model obtained by using the training device described in the fifth aspect to obtain a classification result.

[0022] According to the seventh aspect of the present invention, a computer storage medium is provided, in which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in the first aspect, or executes the method as described in the second aspect, or executes the method as described in the third aspect.

[0023] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0024] 1. The Transformer encoder pre-training method for EEG signal classification tasks provided by the present invention aims at the scalp EEG signal classification task in the brain-computer interface. According to the characteristics of human scalp EEG, the multi-layer wavelet packet reconstruction technology is introduced to reconstruct the EEG data signal into wavelet signals of different frequency bands, and a self-supervised pre-training strategy based on the Transformer encoder is designed, so that the encoder can generate more precise and effective multi-scale refined feature representation for EEG signal data of different frequency bands without labels, improve the feature representation learning effect of the encoder and increase the interpretability, and can train an effective feature extractor for the brain-computer interface classification task to better serve the downstream classification tasks.

[0025] 2. The training method of the EEG signal classification model provided by the present invention, for downstream classification tasks, supervisedly trains the pre-trained encoder and downstream classifier in an ensemble learning manner, so that they learn the relationship between EEG signal data and corresponding labels, and improves the feature extraction capability and classification performance for downstream tasks through a fine-tuning strategy that integrates multi-scale predicted label information. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a Transformer encoder pre-training method for an EEG signal classification task and a training method for an EEG signal classification model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] Since EEG signals have sparse information, low data structure similarity, and high nonlinearity, existing self-supervised learning methods for image data are often difficult to train, and thus cannot obtain effective feature representation, affecting the performance of downstream tasks.

[0029] Based on this, an embodiment of the present invention provides a Transformer encoder pre-training method for EEG signal classification tasks, such as Figure 1 As shown in the "pre-training phase" in , it includes:

[0030] S1, perform wavelet packet decomposition and reconstruction on the EEG signal to obtain wavelet signals in multiple frequency bands.

[0031] Specifically, the original EEG signal is decomposed and reconstructed by wavelet packets according to the frequency bands corresponding to the scalp EEG rhythm to obtain the wavelet signals of each frequency band.

[0032] Preferably, before the EEG signal is decomposed and reconstructed by wavelet packets, the method further comprises:

[0033] The EEG signal is preprocessed; the preprocessing includes bandpass filtering, detrending and channel re-referencing.

[0034] Specifically, the collected EEG data is first subjected to a 0-128 Hz bandpass filter and a detrending method to remove high-frequency noise and other artifacts unrelated to EEG in the collected original signal, so as to improve the signal-to-noise ratio of the EEG data and avoid the real EEG signal from being submerged. Furthermore, the processed EEG signal is subjected to channel re-reference, that is, the reconstructed EEG signal contains brain activity information at two electrode positions through channel combination, thereby reducing information loss.

[0035] Preferably, the multiple frequency bands include at least one of δ, θ, α, β, γ and other high-frequency component frequency bands; wherein the range of the other high-frequency component frequency bands is 64-128 Hz.

[0036] Specifically, according to the rhythm of human scalp EEG signals, we select δ (0-4 Hz), θ (4-8 Hz), α (8-16 Hz), β (16-32 Hz), γ (32-64 Hz), and other high-frequency component bands (64-128 Hz) to construct wavelet packets, and convert the above preprocessed and re-referenced EEG data X = {x1,…,x n Through wavelet packet decomposition and reconstruction, the reconstructed wavelet signal corresponding to each frequency band is obtained. Where n is the number of sampling points of the EEG signal.

[0037] S2, dividing each wavelet signal into signal segments of equal size, and masking at least one signal segment according to the same random mask sequence.

[0038] Preferably, after dividing each wavelet signal into signal segments of equal size, the method further comprises:

[0039] The signal segments of each wavelet signal are mapped in the same linear mapping manner to convert the original data into an embedding representation of a preset dimension (eg, 64 dimensions) to be suitable for the Transformer encoder.

[0040] Specifically, each wavelet signal is divided into The fragment sequence is composed of fragments and given a trainable position code to mark its sequence order. At the same time, a random length of The mask sequence Used to indicate whether the kth segment of each wavelet signal is masked.

[0041] S3, inputting each wavelet signal after the mask operation into the corresponding Transformer encoder and the decoder connected thereto respectively to obtain each decoded and reconstructed wavelet signal, and summing all the decoded and reconstructed wavelet signals according to the corresponding sampling points to obtain the EEG reconstructed signal; wherein the Transformer encoder and decoder are connected one by one; using the Euclidean distance between the EEG reconstructed signal and the EEG signal as the pre-training loss, the Transformer encoder and decoder are pre-trained, so that the pre-trained Transformer encoder can serve the downstream classification task and be used for feature extraction of the EEG signal.

[0042] Specifically, for each wavelet, a Transformer encoder is initialized And the corresponding Transformer decoder

[0043] The corresponding segments of each wavelet are masked according to the above mask sequence, and the generated sequence is input into the Transformer encoder corresponding to each wavelet to obtain the latent space feature representation of all segments (whether masked or not).

[0044] The generated latent space feature representation is input into the Transformer decoder corresponding to each wavelet In the above example, the corresponding decoding reconstructs the signals of each sub-wave

[0045] According to the characteristics of wavelet packet transform, all reconstructed wavelet signals are summed to obtain the reconstructed EEG signal

[0046] Construct pre-training tasks: Continuously iterate training to minimize pre-training loss: That is, the Euclidean distance between the reconstructed EEG signal S and the original EEG signal X is minimized, and the encoder and decoder parameters are updated.

[0047] When the specified number of training rounds (such as the 100th round) is performed, the current model parameters are saved as a pre-trained model.

[0048] Furthermore, the present invention provides a method for training an EEG signal classification model. Figure 1 As shown in the "fine-tuning stage" in the figure, the encoder part of each wavelet in the above-mentioned pre-trained model is used as the feature extractor of the downstream task, and connected to the classifier. The real label information of the downstream task and the integrated predicted label information of each wavelet are used for supervised training, and the model parameters of these encoders and classifiers are further updated to achieve fine-tuning to serve the downstream task.

[0049] The EEG signal classification model includes a pre-trained Transformer encoder obtained by the pre-training method described in any of the above embodiments and a classifier connected thereto in a one-to-one correspondence.

[0050] The method comprises:

[0051] A1, using the pre-trained Transformer encoder as an initialized feature extractor, inputting each wavelet signal into the feature extractor corresponding to it to obtain a feature vector, and inputting it into the corresponding classifier to obtain a classification probability.

[0052] Specifically, the above-mentioned pre-trained model is loaded, the decoder part in the model is deleted and only the encoder structure is retained, the encoder of each pre-trained sub-wave is used as the initialized feature extractor, and is connected to the classifier one by one to obtain the EEG signal classification model.

[0053] The wavelets of each frequency band of the preprocessed EEG signal are input into the pre-trained encoder to obtain the latent space feature representation, which is then input into the classifier corresponding to each encoder to obtain the probability of each wavelet being classified into each category by the classifier.

[0054] A2, based on the learnable weights, weighted summation is performed on the classification probabilities of each wavelet signal to obtain a model classification result; the cross entropy loss between the model classification result and the true label of the sample is used as the training loss to train the EEG signal classification model.

[0055] Preferably, the learnable weights are initialized to normalized identical weights and are continuously iteratively updated during the training process.

[0056] Specifically, the classifier prediction probabilities of the sub-waves in each frequency band are weighted and summed using learnable weights to obtain the final model classification results, where the learnable weights are initialized to the same normalized weights and iteratively updated in the subsequent training process; and the cross entropy loss between the model classification results and the true labels of the samples is calculated.

[0057] The model is continuously trained, and the weights, encoder and classifier parameters are updated until convergence, resulting in a fine-tuned model suitable for specific downstream classification tasks.

[0058] Furthermore, an embodiment of the present invention provides an EEG signal classification method, comprising: inputting an EEG signal to be classified into a trained EEG signal classification model obtained by using the training method described in any of the above embodiments to obtain a classification result.

[0059] Preferably, before the EEG signal to be classified is input into the trained EEG signal classification model, the method further includes: preprocessing the EEG signal to be classified.

[0060] The embodiment of the present invention provides a Transformer encoder pre-training device for an EEG signal classification task, comprising:

[0061] The wavelet packet decomposition and reconstruction module is used to perform wavelet packet decomposition and reconstruction on the EEG signal to obtain wavelet signals of multiple frequency bands;

[0062] A mask operation module, used for dividing each sub-wave signal into signal segments of the same size in the same way, and masking any signal segment;

[0063] The pre-training module is used to input each wavelet signal that has completed the mask operation into the corresponding Transformer encoder and decoder for encoding and decoding, and then sum them according to the sampling points of the EEG signal to obtain the EEG reconstructed signal; the Euclidean distance between the EEG reconstructed signal and the EEG signal is used as the pre-training loss to pre-train the Transformer encoder and decoder, so that the pre-trained Transformer encoder can serve the downstream classification task.

[0064] The embodiment of the present invention provides a training device for an EEG signal classification model, wherein the EEG signal classification model includes a pre-trained Transformer encoder obtained by using the pre-training device described in the above embodiment and a classifier connected thereto in a one-to-one correspondence;

[0065] The device comprises:

[0066] The classification probability acquisition module is used to use the pre-trained Transformer encoder as the initialized feature extractor, input each wavelet signal into the feature extractor corresponding to it to obtain a feature vector, and input it into the corresponding classifier to obtain the classification probability;

[0067] The training module is used to perform weighted summation on the classification probability of each wavelet signal based on the learnable weight to obtain the model classification result; and the cross entropy loss between the model classification result and the true label of the sample is used as the training loss to train the EEG signal classification model.

[0068] An embodiment of the present invention provides an EEG signal classification device, comprising: inputting an EEG signal to be classified into a trained EEG signal classification model obtained by using the training device described in the above embodiment to obtain a classification result.

[0069] An embodiment of the present invention provides a computer storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the Transformer encoder pre-training method for EEG signal classification tasks as described in any of the above embodiments, or executes the training method of the EEG signal classification model as described in any of the above embodiments, or executes the EEG signal classification method as described in the above embodiments.

[0070] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A Transformer encoder pre-training method for EEG signal classification tasks, characterized in that: include: S1, perform wavelet packet decomposition and reconstruction on the EEG signal to obtain wavelet signals in multiple frequency bands; The multiple frequency bands include at least two of δ, θ, α, β, γ or other high-frequency components, and the frequency band range of the other high-frequency components is 64-128 Hz; the wavelet signals of the wavelet signals of the multiple frequency bands are: n is the number of sampling points of the EEG signal; S2, dividing each wavelet signal into signal segments of equal size, and masking at least one signal segment according to the same random mask sequence; S3, inputting each wavelet signal that has completed the mask operation into the corresponding Transformer encoder and decoder for encoding and decoding, summing them according to the sampling points of the EEG signal to obtain the EEG reconstructed signal; using the Euclidean distance between the EEG reconstructed signal and the EEG signal as the pre-training loss, pre-training the Transformer encoder and decoder; The Transformer encoder of each wavelet signal encodes each wavelet signal that has completed the masking operation to obtain the latent space feature representation of all fragments, and the Transformer decoder of each wavelet signal decodes the latent space feature representation of all fragments to reconstruct each wavelet signal. The EEG reconstruction signal 2. The method according to claim 1, characterized in that Before the EEG signal is decomposed and reconstructed by wavelet packets, the method further comprises: Preprocessing the EEG signal; the preprocessing includes using bandpass filtering and detrending to remove artifacts, and performing channel re-referencing; After the sub-wave signals are divided into signal segments of equal size, the method further includes: The signal segments of each wavelet signal are mapped in the same linear mapping manner.

3. A training method for an EEG signal classification model, characterized in that: The EEG signal classification model includes a pre-trained Transformer encoder obtained by the pre-training method according to any one of claims 1 to 2 and a classifier connected thereto in a one-to-one correspondence; the method includes: A1, using the pre-trained Transformer encoder as an initialized feature extractor, inputting each wavelet signal into the feature extractor corresponding to it to obtain a feature vector, and inputting it into the corresponding classifier to obtain a classification probability; A2, based on the learnable weights, weighted summation is performed on the classification probabilities of each wavelet signal to obtain a model classification result; the cross entropy loss between the model classification result and the true label of the sample is used as the training loss to train the EEG signal classification model.

4. The method according to claim 3, characterized in that The learnable weights are initialized to normalized identical weights and are continuously iteratively updated during the training process.

5. A method for classifying electroencephalogram signals, characterized in that: include: The EEG signal to be classified is input into a trained EEG signal classification model obtained by the training method described in any one of claims 3-4 to obtain a classification result.

6. A Transformer encoder pre-training device for EEG signal classification tasks, characterized in that: include: Wavelet packet decomposition and reconstruction module is used to analyze the EEG signal X = {x1,…,x n } Perform wavelet packet decomposition and reconstruction to obtain wavelet signals of multiple frequency bands; The multiple frequency bands include at least two of δ, θ, α, β, γ or other high-frequency components, and the frequency band range of the other high-frequency components is 64-128 Hz; the wavelet signals of the wavelet signals of the multiple frequency bands are: n is the number of sampling points of the EEG signal; A mask operation module, used for dividing each sub-wave signal into signal segments of equal size, and masking at least one signal segment according to the same random mask sequence; A pre-training module is used to input each wavelet signal that has completed the mask operation into the corresponding Transformer encoder and decoder for encoding and decoding, and then sum them according to the sampling points of the EEG signal to obtain the EEG reconstructed signal; the Euclidean distance between the EEG reconstructed signal and the EEG signal is used as the pre-training loss to pre-train the Transformer encoder and decoder; The Transformer encoder of each wavelet signal encodes each wavelet signal that has completed the masking operation to obtain the latent space feature representation of all fragments, and the Transformer decoder of each wavelet signal decodes the latent space feature representation of all fragments to reconstruct each wavelet signal. The EEG reconstruction signal 7. A training device for an electroencephalogram signal classification model, characterized in that: The EEG signal classification model includes a pre-trained Transformer encoder obtained by using the pre-training device according to claim 6 and a classifier connected thereto in a one-to-one correspondence; the device includes: The classification probability acquisition module is used to use the pre-trained Transformer encoder as the initialized feature extractor, input each wavelet signal into the feature extractor corresponding to it to obtain a feature vector, and input it into the corresponding classifier to obtain the classification probability; The training module is used to perform weighted summation on the classification probability of each wavelet signal based on the learnable weight to obtain the model classification result; and the cross entropy loss between the model classification result and the true label of the sample is used as the training loss to train the EEG signal classification model.

8. An electroencephalogram signal classification device, characterized in that: include: The EEG signal to be classified is input into a trained EEG signal classification model obtained by using the training device described in claim 7 to obtain a classification result.

9. A computer storage medium storing instructions, wherein when the instructions are executed on a computer, the computer executes the method according to any one of claims 1 to 2, or the method according to any one of claims 3 to 4, or the method according to claim 5.

Images