Self-supervised representation learning method, system and storage medium for EEG signals

Abstract

The present invention proposes a self-supervised representation learning method, system and storage medium for EEG signals, which belongs to the technical field of signal processing and pattern recognition. The local latent representation and self-context representation of multi-channel EEG signal data of each time period are obtained by using an encoder, and a global representation is obtained to calculate the loss of the instantaneous time shift prediction task; the segment representation is obtained according to the self-context representation of each time period, and the correlation probability between different channels in different time periods is predicted, and the loss of the delayed time shift prediction task is calculated; the local latent representation of each time period is randomly replaced, and a new self-context representation is calculated according to the replaced new local latent representation, and the local latent representation corresponding to each original channel is predicted according to the new self-context representation. Whether it is replaced by other channels, the loss of the replacement discrimination learning prediction task is calculated; through three self-supervised tasks, self-supervised representation learning on EEG signal data is realized, and the learned representation is used for epileptic seizure prediction applications.

Description

Technical Field

[0001] The present invention relates to the technical field of signal processing and pattern recognition, and in particular to a self-supervisory representation learning method, system and storage medium for electroencephalogram (EEG) signals. Background Art

[0002] The patterns of EEG signals can greatly help us understand how the brain works normally and the mechanisms of related diseases. There are many studies on EEG signals, such as cognitive research, emotion recognition, and neurological diseases. There are two main methods for recording EEG signals, including non-invasive methods and invasive methods, but it is usually difficult to obtain labels for EEG signal data. As self-supervised learning has become the dominant learning paradigm in the absence of labels, more and more work has focused on using self-supervised learning to learn effective representations in massive unlabeled data. The method proposed in the present invention aims to perform self-supervised representation learning on scalp EEG (EEG) and stereo EEG (SEEG) signals, and use the learned representations for epileptic seizure prediction applications.

[0003] Since EEG signals are essentially time series data, general time series self-supervision algorithms can also be used for this task. Existing methods are based on specially designed proxy tasks, such as [Saeed, 2019], which transforms the data and trains the model to distinguish between the original data and the transformed data. [P.Sarkar, 2020] proposed a similar method by applying six transformations to the original data. In addition, [Aggarwal, 2019] learns representations with individual invariance by modeling local and global activity patterns. [Franceschi, 2019] uses time-based negative sampling and triplet loss to learn scalable multi-dimensional time series representations. The triplet loss objective ensures that similar time series have similar representations by minimizing the pairwise distance between positive samples (subsequences) while maximizing the distance between negative samples. Inspired by the outstanding results achieved by contrastive learning in the field of computer vision, some methods also use contrastive learning for time series data, such as CPC [Oord, 2018], which learns by predicting future representations in the latent space and has achieved great improvements in various speech recognition tasks. TS-TCC[Eldele, 2021] learns robust and discriminative representations by applying a temporal contrast module and a contextual contrast module. In addition, TST[Zerveas, 2021] learns representations by predicting mask values ​​of time series data input based on the Transformer[Vaswani, 2017] structure.

[0004] In addition to the above-mentioned time series self-supervised algorithms, there are also self-supervised representation learning algorithms designed specifically for EEG data. [Banville, 2020] used three self-supervised proxy tasks to learn the representation of EEG data, of which the relative position prediction and time series shuffling tasks were inspired by [Hyvarinen, 2016], and the other task was based on CPC [Oord, 2018]. [Mohsenvand, 2020] designed relevant data enhancement methods for EEG data and extended the self-supervised model SimCLR [Chen, 2020] in the field of computer vision to EEG data. [Cheng, 2020] designed a self-supervised method based on contrastive learning for EEG and ECG (electrocardiogram), learning individual-independent representations by leveraging subject-level information. The model proposed by [Tang, 2021] treats the EEG as a graph and proposes a graph neural network for automatic epilepsy detection and classification to reveal the connectivity between the electrodes of the EEG based on a self-supervised pre-training strategy.

[0005] Compared with EEG data with fixed specifications, SEEG data of different patients not only have different numbers of electrodes, but also different positions between electrodes. Designing self-supervised representation learning algorithms on SEEG data faces greater challenges than EEG data. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a self-supervised representation learning method, system and storage medium for EEG signal. Through three designed self-supervised tasks, self-supervised representation learning on EEG signal data is realized, and the learned representation is used for epileptic seizure prediction applications.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for learning self-supervised representation of EEG signals, comprising the following steps:

[0009] Obtain the original multi-channel EEG signal data of each time period, and use the encoder to obtain the local hidden representation and self-context representation of each time period; calculate the correlation of different channels in each time period according to the global context representation, and obtain a sparse fine-grained correlation matrix, and calculate the global representation of each channel in each time period according to the sparse fine-grained correlation matrix; predict the local hidden representation of the time period within k1 steps according to the global representation of all time periods, and calculate the loss of the instantaneous time shift prediction task;

[0010] According to the original multi-channel EEG signal data of each time period, the long-term correlation matrix between different channels in different time periods is calculated, and the long-term correlation pseudo-label is constructed; the segment representation is obtained according to the context representation of each time period, the correlation probability between different channels in different time periods is predicted, and the loss of the delay time shift prediction task is calculated;

[0011] Randomly replace the local latent representations of each time period, calculate new self-context representations according to the replaced new local latent representations, predict whether the local latent representations corresponding to each original channel are replaced by other channels according to the new self-context representations, and calculate the loss of the replacement discrimination learning prediction task;

[0012] The encoder is trained by combining the loss of the instantaneous time shift prediction task, the loss of the delayed time shift prediction task and the loss of the discriminative learning prediction task; the trained encoder is used to obtain the self-context representation of the original multi-channel EEG signal data to be processed, and the segment representation is obtained, and the segment representation is used as the learned EEG signal self-supervised representation for the epileptic seizure prediction task.

[0013] In a second aspect, the present invention provides an EEG signal self-supervised representation learning system for implementing the above-mentioned EEG signal self-supervised representation learning method.

[0014] In a third aspect, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-mentioned method for self-supervised representation learning of EEG signals.

[0015] Compared with the prior art, the present invention has the following beneficial effects: the present invention designs k-tuple features, base graph relationship capture mechanism and multi-task mechanism through graph neural network, which improves the calculation speed of approximate calculation of local base graph frequency on large-scale graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram showing a method for learning a self-supervised representation of an electroencephalogram signal according to an exemplary embodiment. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The accompanying drawings are only schematic diagrams of the present invention, and some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities, and these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0018] In this embodiment, the model used in the self-supervised representation learning method of EEG signals is denoted as Mbrain, which is a deep learning framework based on self-supervised learning, contrastive learning, multi-task learning, graph structure learning and graph neural network. The present invention can output a valid representation of each segment and each channel given a series of continuous multi-channel EEG signal data of fixed duration through the encoder in the pre-trained self-supervised module, and the representation can be applied to downstream epileptic seizure prediction tasks. Specifically, the present invention first performs representation learning on the original EEG signal data through a self-supervised learning module; then, the learned representation is used for training and testing of the epileptic seizure prediction module.

[0019] The overall framework of the present invention (MBrain) consists of three main modules: correlation learning module, time-shift prediction learning module and replacement discrimination learning module. The correlation learning module learns more accurate short-term data fine-grained correlation based on coarse-grained correlation; the time-shift prediction learning module closes the representations of signals with relatively more similar patterns on two time scales, and pushes away the representations of signals with relatively different patterns; the replacement discrimination learning module effectively learns commonalities while retaining signal differences to enhance the transferability of MBrain.

[0020] Figure 1 The framework diagram of the algorithm MBrain of the present invention is shown. On the far left is the original multi-channel EEG signal segment data, and the encoder is used to obtain local and global context representations. Then the general process of three self-supervised tasks is shown on the right. From top to bottom, they correspond to the replacement discriminant learning task, the instantaneous time shift prediction task, and the delayed time shift prediction task, where the instantaneous time shift prediction task uses the correlation matrix obtained by the proposed correlation learning algorithm. Through this multi-task mechanism, MBrain can learn effective representations from unlabeled EEG signal data.

[0021] The specific implementation methods of the three modules will be introduced below.

[0022] 1. Correlation learning module.

[0023] The overall synergy of brain regions usually reflects a person's specific functions and activities. In a normal state, the correlation between channels does not change much. However, there are significant differences between normal EEG signals and abnormal epileptic signals. Therefore, a dynamic learning strategy is needed to capture this variable correlation in the brain.

[0024] Through data observation, it was found that for EEG and SEEG data, the correlation matrices on two non-overlapping normal segments of the same patient were almost the same. In contrast, the correlation matrix in the epileptic state was very different from the correlation matrix in the normal state. These data observations confirm the above conclusions and support the use of the correlation matrix as a common attribute of EEG and SEEG data in the present invention. However, in the self-supervised learning stage, the label information of the brain signal cannot be obtained. Fortunately, thanks to the low frequency of epileptic seizures, the proportion of abnormal states is so small that their impact on the overall correlation calculation is negligible. In addition, experience shows that the more time periods involved in calculating the correlation matrix, the more stable the results and the smaller the variance. Therefore, the following stable coarse-grained correlation matrix can be obtained from all the data of a patient:

[0025]

[0026] in, represents the expected operation, that is, for all W long time periods s obtained from the original data t The fine-grained correlation matrix calculated in s is averaged; t，i represents the original EEG signal of the ith channel in time period t, s t，j represents the original EEG signal of the jth channel in time period t, A0(i, j) represents the element in the i-th row and j-th column in the coarse-grained correlation matrix A0, and i and j are both channel indices.

[0027] As shown in the previous conclusions, although a stable coarse-grained correlation matrix is ​​obtained in the desired manner, there are still slight changes in the fine-grained correlation matrix for normal data. In addition, there are significant differences in the correlations between normal and epileptic seizure states. Therefore, a fine-grained correlation matrix should allow a more accurate characterization of short-term correlations based on changes in A0. To achieve this goal, it is assumed that the true correlation matrix follows a Gaussian distribution on the elements, whose location parameters are the elements of A0 and the scale parameters will be learned from the data. Through the reparameterization method, the short-term correlation matrix for the t-th time period is sampled from the learned Gaussian distribution:

[0028]

[0029]

[0030] A 1，t (i, j) = A0(i, j) + σ t (i, j)×n t (i, j)

[0031] Among them, MLP represents multi-layer perceptron, represents the self-context representation of the τth step of the i-th channel in time period t, Determine the context representation of the jth channel at the τth step in time period t, n t (i, j) represents the distribution from the standard normal distribution The sampled value in , SoftPlus(.) is the SoftPlus activation function, σ t (i, j) represents the standard deviation of the Gaussian distribution followed by the correlation between the i-th and j-th channels in time period t, A 1，t (i, j) represents the correlation between the i-th and j-th channels in time period t.

[0032] Through the above correlation learning, the learnable variance allows the short-term correlation to change flexibly. Filter the elements whose weights are lower than the predefined positive threshold θ1 to obtain the sparse fine-grained correlation matrix A for the final time period t t :

[0033]

[0034] Among them, A t (i, j) represents the sparse fine-grained correlation matrix A t The element in the i-th row and j-th column in is used to represent the correlation between the i-th and j-th channels in the t-th time period after sparsification; in this way, the pseudo-correlation caused by low-frequency signals can be eliminated and sparsity can be enhanced, which is a common assumption in neuroscience-related models.

[0035] 2. Time-shift prediction learning module.

[0036] EEG signal data is non-stationary, which means that there are too many signal patterns that cannot be captured. In order to summarize the signal patterns in a more feasible representation space, the present invention adopts an indirect strategy, which is to bring closer the representations between signals with relatively similar patterns, and push away the representations between signals with relatively different patterns. Since it is desired to summarize the patterns in time and space, the patterns of different channels will be compared at different time steps, resulting in a time shift in the representation. In addition, considering that the original EEG signal is divided into multiple segments, the present invention implements this strategy on two time scales. For the representation of time periods within the same time step, instantaneous time shifts are used to summarize short-term patterns, while delayed time shifts are used to summarize long-term patterns between segments across different time steps.

[0037] (1) Instantaneous time shift.

[0038] The fine-grained correlation matrix obtained by the above-mentioned correlation learning module accurately reflects the correlation between segments within the same time step, and time segments with higher correlation should be more predictable at the representation level. The present invention sets the self-supervised learning task to predict local representation through global representation rather than raw time series data. Unlike the method of modeling the autoregressive correlation of only a single channel, the present invention makes full use of the correlation between channels. Specifically, the present invention obtains a global representation from the contextual representation of the channel itself and other channels, which is described in detail as follows.

[0039] First, use the nonlinear encoder g enc Map the observed time segment (including all channels) to a local d-dimensional latent representation in, is through g enc The length of the transformed sequence, s t It is the original EEG signal data with length C and C channels at time t, and the value of C is related to different patients. Then, the autoregressive model g ar The historical local information of each channel is summarized to obtain its own context representation as follows:

[0040]

[0041] Among them, z t，τ-1 represents the implicit representation of the τth step in the tth time period, represents the self-context representation of the τth step in time period t, Represents the self-context representation of the τth step of the i-th channel in time period t.

[0042] So far, only the contextual representation of all channels has been extracted independently. Taking the i-th channel as an example, according to the fine-grained correlation matrix A t , using the information of other channels to help predict the local representation of the i-th channel. Considering the matrix as a network structure, GNN is used to aggregate the information of other channels. In this embodiment, a layer of directed GCN is used to aggregate the information of other channels. The formula is as follows:

[0043]

[0044] Among them, Θ is the learnable transformation matrix, ReLU(.) is the ReLU activation function, represents the information of other channels before the aggregation of the i-th channel in time period t. Considering that this embodiment only aggregates the information of other channels, the self-loop in the directed GCN is removed. Finally, combined with Get the global representation c t，τ MBrain can predict the local representation within k1 steps based on the contrast loss InfoNCE

[0045]

[0046]

[0047] Among them, N t，i represents a random noise set of implicit representations containing a positive sample and several negative samples, Represents the set N t，i The implicit representation of a positive sample in , that is, the implicit representation of the ith channel in the τ+k1th step in the tth time period; c t，τ，i represents the global representation of the first τ steps of the i-th channel aggregation in time period t, z j Represents the set N t，i The hidden representation of the jth sample in is, represents the expected operation, that is, averaging the contrast loss of all channels in all time periods within k1 steps; represents the loss of the instantaneous time-shift prediction task; is the learnable bilinear matrix corresponding to the k1-th step prediction.

[0048] (2) Delay time shift.

[0049] When a neuron generates a potential difference above a certain threshold, an activity potential occurs. Considering that epileptic brain signals are disordered discharges from neurons, activity potentials are more frequent during epileptic seizures. In addition, activity potentials can be transmitted to other neurons along the connection structures in the brain. The physical distance of the connection structure will result in the temporal order of the activity potentials arriving at different brain regions. Therefore, there is a phenomenon of delayed time shift between brain regions that are far apart. The data observation results also confirmed that there is still a significant correlation between time periods across multiple time steps.

[0050] For the task of epileptic seizure prediction, there are large differences between normal and epileptic brain signals in terms of amplitude, frequency, and shape. This suggests that a reasonable pre-trained representation should have a certain ability to distinguish brain signals in the two states. If a simple classifier can easily predict whether two time periods are highly correlated, then there will be significant differences between the representations of time periods with lower correlation. Therefore, the present invention defines a long-term prediction task to learn more distinguishable segment representations. Similar to instantaneous time shifts, the cosine similarity matrix between time periods spanning several time steps is first calculated. Pseudo-labels are then constructed based on the cosine similarity matrix to guide the algorithm of the present invention to learn more similar representations between time periods with higher correlation.

[0051] For the ith channel in the tth time period, the long-term correlation matrix The calculation method is as follows:

[0052]

[0053] Where j = 1, ..., C represents the channel index, k2 = 1, ..., K2 represents the time step, and at most K2 time steps are traversed to calculate the delay time shift, s t，i Represents the data of the i-th channel in time period t, Represents the data of the jth channel in the time period t+k2, represents the long-term correlation value between the i-th channel in time period t and the j-th channel in time period t+k2. In order to encourage segments with higher correlation to be closer, a threshold θ2 is predefined in this embodiment to construct pseudo labels.

[0054]

[0055] in, A pseudo label indicating whether the i-th channel in time period t is related to the j-th channel in time period t+k2.

[0056] Using pseudo labels, based on segment representation h t To define the cross entropy loss for the delay time shift prediction task:

[0057]

[0058]

[0059]

[0060] in, is the predicted probability that two segments are highly correlated, h t，i The segment representation of the ith channel in time period t is represented by, The segment representation of the jth channel in the time period t+k2 is represented as follows: represents the expected operation, that is, the average of the cross entropy losses of all channels in all time periods with other channels within k2 steps, is the loss of the delayed time shift prediction task. In this embodiment, 50% of the labels are randomly selected from the set for efficient training.

[0061] 3. Replace the discriminative learning module.

[0062] In the specific method description of the previous two modules, the pre-training task of each channel was introduced. However, the target downstream task of the present invention, epileptic seizure prediction, focuses on each patient and each channel at the same time. In detail, there are differences in the characteristics of epileptic waves between different channels of a patient and between different patients. Through data observation, it was found that even in the same patient, the amplitude of the normal signal in one channel may be very similar to the amplitude of the epileptic signal in another channel. For different types of epilepsy patients, the epileptic seizure patterns are also very different. For this reason, the present invention designs an effective strategy to learn the commonalities of different channels while retaining their differences, so as to improve the transferability of the algorithm of the present invention and the accuracy of epilepsy prediction.

[0063] For commonality, the EEG signal data of each channel is first normalized to eliminate the impact of scale. All channels share the same encoder g enc and the autoregressive model g ar In order to make the representations of different channels more distinguishable, this paper proposes a replacement discrimination task to achieve this goal. t 15% of the local representation, the candidate set for replacement includes all sequences and all C channels in each sequence. Using the notation Represents the new local representation after replacement. Next, the pseudo label Y for this task can be generated t , as shown below:

[0064]

[0065] Among them, Y t (τ, i) represents the pseudo label of the i-th channel at the τ-th step, represents the new local representation after the i-th channel is replaced in the τ-th step of time period t, I(.) represents the channel index corresponding to the new local representation before it is replaced, τ and i traverse of sequences and C channels.

[0066] get After that, it is passed through the autoregressive model to obtain a new context representation Finally, a simple classifier is used to distinguish whether the context representation is replaced by other channels:

[0067]

[0068]

[0069] in, yes The predicted probability of being replaced. t，τ，i[.] represents the expected operation, that is, the average of the cross entropy loss in all channels in all time periods within τ steps, is the loss for the replacement discriminative learning prediction task.

[0070] When the discrimination accuracy is improved, the different channel representations output by the autoregressive model are easier to distinguish. This can preserve the differences between different channels and enhance the transferability of MBrain between patients. Combined with the multi-task loss function to jointly train the Mbrain model:

[0071]

[0072] After the training phase, the segment representation h obtained in the delayed time shift t It can be used for further epileptic seizure prediction tasks.

[0073] In a specific implementation of the present invention, the method for learning the self-supervised representation of EEG signals includes:

[0074] Step I, obtain the original multi-channel EEG signal data of each time period, use the encoder to obtain the local hidden representation and self-context representation of each time period; calculate the correlation of different channels in each time period according to the global context representation, obtain a sparse fine-grained correlation matrix, and calculate the global representation of each channel in each time period according to the sparse fine-grained correlation matrix; predict the local hidden representation of the time period within k1 steps according to the global representation of all time periods, and calculate the loss of the instantaneous time shift prediction task;

[0075] In this step, the specific implementation method is:

[0076] Step 1.1, use the nonlinear encoder g enc Map the EEG signal data of all channels in the observed time period to the local area and obtain the local implicit representation:

[0077]

[0078] Among them, s t is the EEG signal data of time period t, is through g enc The length of the transformed sequence, z t is the local implicit representation of time period t, Indicates the time period t Implicit representation of steps;

[0079] Using the autoregressive model g ar The historical local information of each channel is summarized to obtain its own context representation as follows:

[0080]

[0081] Among them, z t，τ-1 represents the implicit representation of the τth step in the tth time period, represents the self-context representation of the τth step in time period t, when

[0082] Step 1.2, calculate the sparse fine-grained correlation matrix:

[0083]

[0084] A 1，t (i, j) = A0(i, j) + σ t (i, j)×n t (i, j)

[0085] Among them, MLP represents multi-layer perceptron, represents the self-context representation of the τth step of the i-th channel in time period t, represents the context representation of the jth channel at the τth step in time period t, n t (i, j) represents the distribution from the standard normal distribution The sampled value in , SoftPlus(.) is the SoftPlus activation function, σ t (i, j) represents the standard deviation of the Gaussian distribution followed by the correlation between the i-th and j-th channels in time period t, A 1，t (i, j) represents the correlation between the i-th and j-th channels in time period t;

[0086] Filter the elements with correlation less than θ1 to obtain the sparse fine-grained correlation matrix A t :

[0087]

[0088] Among them, A t (i, j) represents the sparse fine-grained correlation matrix A t The element in the i-th row and j-th column in is used to represent the correlation between the i-th and j-th channels in the t-th time period after sparseness.

[0089] Step 1.3, calculate the global representation of each channel in each time period based on the sparse fine-grained correlation matrix:

[0090]

[0091]

[0092] in, represents the representation of other channel information before the aggregation of the i-th channel in time period t, Θ is a learnable transformation matrix, ReLU(.) is the ReLU activation function, c t，τrepresents the global representation of the first τ steps of the i-th channel aggregation in time period t,

[0093] Step 1.4, predict the local implicit representation of the time period within k1 steps based on the global representation of the current time period, and calculate the loss of the instantaneous time shift prediction task, including:

[0094]

[0095] Among them, N t，i represents a random noise set of implicit representations containing a positive sample and several negative samples, Represents the set N t，i The local hidden representation of a positive sample in , that is, the local hidden representation of the i-th channel in the τ+k1-th step of time period t, c t，τ，i represents the global representation of the first τ steps of the i-th channel aggregation in time period t, z j Represents the set N t，i The local hidden representation of the jth sample in is represents the expected operation, that is, averaging the contrast loss of all channels in all time periods within k1 steps; represents the loss of the instantaneous time-shift prediction task; is the learnable bilinear matrix corresponding to the k1-th step prediction; the superscript T indicates the transpose.

[0096] Step 2: Based on the original multi-channel EEG signal data of each time period, the long-term correlation matrix between different channels in different time periods is calculated to construct long-term correlation pseudo labels; the segment representation is obtained based on the context representation of each time period, the correlation probability between different channels in different time periods is predicted, and the loss of the delay time shift prediction task is calculated;

[0097] In this step, the specific implementation method is:

[0098] Step 2.1, calculate the long-term correlation matrix between different channels in different time periods:

[0099]

[0100] Where j = 1, ..., C represents the index of the channel, k2 represents the time step, s t，i Represents the data of the i-th channel in time period t, Represents the data of the jth channel in the time period t+k2, It represents the long-term correlation value between the i-th channel in time period t and the j-th channel in time period t+k2.

[0101] Step 2.2, construct long-term correlation pseudo labels:

[0102]

[0103] in, The pseudo label indicates whether the i-th channel in time period t is related to the j-th channel in time period t+k2, and θ2 is the correlation threshold.

[0104] Step 2.3, obtain the segment representation based on the context representation of each time period, and predict the correlation probability between different channels in different time periods. The formula is:

[0105]

[0106]

[0107] The loss of the delay time shift prediction task is calculated as:

[0108]

[0109] Among them, Pooling(.) represents the pooling operation, h t is the segment representation of time period t, is the time period t The self-context representation of the step, MLP(.) represents a multi-layer perceptron, is the predicted probability that two segments are highly correlated, h t，i represents the segment representation of the i-th channel in time period t, C represents the number of channels, The segment representation of the jth channel in the time period t+k2 is represented as follows: represents the expected operation, that is, the average of the cross entropy losses of all channels in all time periods with other channels within k2 steps, is the loss for the delay time-shift prediction task.

[0110] Step 3, randomly replace the local latent representation of each time period, calculate the new self-context representation according to the replaced new local latent representation, predict whether the local latent representation corresponding to each original channel is replaced by other channels according to the new self-context representation, and calculate the loss of the replacement discrimination learning prediction task;

[0111] In this step, the specific implementation method is:

[0112] Step 3.1: Randomly replace the local hidden representation z obtained in step 1 t 15% of the elements in the t time period are replaced to obtain the new local implicit representation according to Get the new self-context representation

[0113] Step 3.2, generate pseudo labels based on the replaced new local latent representation:

[0114]

[0115] in, is the new local implicit representation of the replaced time period t The elements in Y are the implicit representations of the ith channel in the tth time period and the τth step after replacement; t (τ, i) represents the pseudo label of the i-th channel in the τ-th step; I(.) represents the channel index corresponding to the new local latent representation before it is replaced;

[0116] Step 3.3, calculate the loss of the replacement discriminative learning prediction task:

[0117]

[0118]

[0119] in, is the new self-context representation The vector corresponding to the i-th channel in is the predicted probability that the ith channel is replaced by other channels, E t，τ，i [.] represents the expected operation, that is, the average of the cross entropy loss in all channels in all time periods within τ steps, is the loss for the replacement discriminative learning prediction task.

[0120] Step 4, training the encoder by combining the loss of the instantaneous time shift prediction task, the loss of the delayed time shift prediction task and the loss of the discriminative learning prediction task; using the trained encoder to obtain the self-context representation of the original multi-channel EEG signal data to be processed, and obtain the segment representation, and use the segment representation as the learned EEG signal self-supervised representation for the epileptic seizure prediction task.

[0121] In this embodiment, a self-supervised representation learning system for EEG signals is also provided, which is used to implement the above-mentioned embodiments, and will not be repeated hereafter. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible.

[0122] The system comprises:

[0123] An EEG signal acquisition module is used to acquire original multi-channel EEG signal data of each time period;

[0124] The encoder module is used to obtain the local latent representation and self-context representation of each time segment;

[0125] A correlation learning module is used to calculate the correlation of different channels in each time period according to the global context representation to obtain a sparse fine-grained correlation matrix;

[0126] The time-shift prediction learning module is used to calculate the global representation of each channel in each time period according to the sparse fine-grained correlation matrix; predict the local implicit representation of the time period within k1 steps according to the global representation of all time periods, and calculate the loss of the instantaneous time-shift prediction task; and, based on the original multi-channel EEG signal data of each time period, calculate the long-term correlation matrix between different channels in different time periods and construct long-term correlation pseudo labels; obtain the segment representation according to the self-context representation of each time period, predict the correlation probability between different channels in different time periods, and calculate the loss of the delayed time-shift prediction task;

[0127] A replacement discriminant learning module is used to randomly replace the local latent representations of each time period, calculate a new self-context representation based on the replaced new local latent representations, predict whether the local latent representations corresponding to each original channel are replaced by other channels based on the new self-context representations, and calculate the loss of the replacement discriminant learning prediction task;

[0128] A training module, which is used to train the encoder module by combining the loss values ​​calculated by the time-shift prediction learning module and the replacement discrimination learning module;

[0129] A self-supervised representation generation module uses a trained encoder to obtain the self-context representation of the original multi-channel EEG signal data to be processed, and obtains a segment representation based on the self-context representation of each time period, and uses the segment representation as the generated EEG signal self-supervised representation for the epileptic seizure prediction task.

[0130] The implementation process of the functions and effects of each module in the above-mentioned system is specifically detailed in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here. For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative labor. An embodiment of the present invention also provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the above-mentioned self-supervised representation learning method of EEG signals is implemented.

[0131] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0132] The technical effect of the present invention is verified by experiments below.

[0133] (1) Data description

[0134] SEEG dataset. The SEEG dataset used in this experiment was provided by a tertiary hospital. For patients with epilepsy, 4 to 10 invasive electrodes with 52 to 120 channels were used to record 1,000Hz or 2,000Hz signals. These numbers vary from patient to patient. It is worth noting that since SEEG data is collected through multiple channels at high frequency, this amount of data is very large. A total of 380 hours of SEEG signals with a size of 425GB were collected in this experiment. The corresponding training and test sets were generated for the four patients included in the dataset, and the experiments were repeated independently for each patient. The SEEG data were annotated at the channel level by experts. For the self-supervised learning stage, 1,000 10-second SEEG clips were randomly selected for training and validation. For the downstream epilepsy prediction task, 1,000 10-second SEEG clips were first utilized (80% for training and 20% for validation). For testing, another 510 10-second SEEG clips were sampled in this experiment with a positive-negative sampling ratio of 1:50.

[0135] EEG dataset. This experiment uses the TUSZ v1.5.2 dataset. It is the largest public EEG seizure database, containing 5,612 EEG records, 3,050 annotated seizures in clinical records, and 8 seizure types. The dataset contains 19 EEG channels under the standard 10-20 system. For experimental efficiency, this experiment sampled a small dataset from TUSZ and randomly extracted 3,000 12-second EEG segments for self-supervised learning. For the downstream epilepsy detection task, 3,000 12-second EEG segments were first obtained (80% for training and 20% for validation). For testing, this experiment sampled an additional 3,900 12-second EEG segments with a positive-negative sample ratio of 1:10. It is worth noting that the labels of the EEG data are coarse-grained, which means that for 1 12-second EEG segment, we only have a label of whether an epilepsy occurred in it.

[0136] (2) Description of the benchmark algorithm

[0137] This experiment compares the present invention with several existing benchmark methods.

[0138] SimCLR[Chen, 2020]: This is a simple yet effective framework for contrastive learning of visual representations, which we adapt to our application using time series specific data augmentation.

[0139] Triplet-Loss (T-Loss) [Franceschi, 2019]: This method uses time-based negative sampling and triplet loss to learn representations of time series segments.

[0140] TS-TCC[Eldele, 2021]: This is an unsupervised time series representation learning framework that learns robust and discriminative representations by applying a temporal contrast module and a contextual contrast module.

[0141] Time Series Transformer (TST) [Zerveas, 2021]: This is a framework for unsupervised representation learning of multivariate time series based on Transformer [Vaswani, 2017], which trains the model by predicting masked values ​​of time series data input.

[0142] GTS[Shang, 2021]: This is a time series prediction model that learns a graph structure across multiple time series and predicts them simultaneously using GNNs. Therefore, the model can learn useful representations from unlabeled time series data.

[0143] (3) Epileptic seizure prediction experiment

[0144] In the epileptic seizure prediction experiment, the model is first pre-trained on unlabeled data. After the pre-training phase is completed, the segment representations learned by the model are used for downstream epilepsy prediction tasks. Epileptic seizures usually last for more than 10 seconds, and the present invention suggests considering the correlation between channels. Therefore, the LSTM model is first used to independently encode the segment representation of each channel in chronological order (10 seconds in SEEG segments and 12 seconds in EEG segments). Then a layer of self-attention layer is applied to all channels in the same time step. Finally, a two-layer MLP classifier is used to predict whether epilepsy occurs within the time period. In addition, since the labels of EEG data are coarse-grained, each 12-second EEG segment representation is aggregated into one representation and then used for epileptic seizure prediction.

[0145] Table 1 Performance of epilepsy seizure prediction experiments on the SEEG dataset (including four different patients)

[0146]

[0147] Table 2 Performance of epilepsy prediction experiments on EEG dataset

[0148]

[0149] The SEEG dataset includes 4 patients, and the seizure prediction performance is shown in Table 1. Due to the imbalanced ratio of positive and negative samples in the SEEG dataset, it is not an appropriate choice to evaluate the performance of the model by only one of precision or recall. In the case of label imbalance, the F score is the main indicator for evaluating the performance of the model in the seizure prediction experiment. Especially in clinical applications, doctors pay more attention to discovering as many seizures as possible, so the F1 score and F2 score were selected in this experiment. Overall, MBrain outperformed all baseline algorithms in terms of F1 and F2 scores for all patients, proving that MBrain can fully learn effective representations from SEEG data.

[0150] Table 2 shows the experimental results of epileptic seizure prediction on the EEG dataset. According to the general evaluation scheme of the EEG dataset, the AUROC indicator is added in this experiment. Overall, MBrain outperforms all benchmark algorithms in F1 score, F2 score and AUROC on the EEG dataset. The model of the present invention is designed to learn channel-level representation, while the labels of the EEG dataset are coarse-grained, which requires a pooling operation on the representation output by the model of the present invention on the channel and time period to obtain a coarse-grained representation for epileptic seizure prediction. The pooling operation will lose a considerable amount of information in time and space, resulting in a decrease in performance on the EEG dataset.

[0151] (4) Transfer learning experiment

[0152] In the transfer learning experiment, the model is first pre-trained on the SEEG dataset of one patient (i.e., the source domain), and then the downstream model and classifier are trained on the SEEG dataset of another patient (i.e., the target domain), while the self-supervised model is fine-tuned. Finally, an epileptic seizure prediction experiment is performed on the target domain, and the results are compared with the results when both the self-supervised model and the downstream model are trained on the target domain. This time, transfer learning experiments were conducted on 12 cross-domain scenarios consisting of SEEG datasets composed of four patients.

[0153] Table 3 Performance of cross-patient transfer learning experiments (F2 score as evaluation indicator)

[0154]

[0155] Table 3 shows the performance of the transfer learning experiment in 12 cross-domain scenarios. More specifically, this experiment pre-trained the model on one patient and transferred it to the remaining three patients. B→A means that the model was pre-trained on patient B, and then the epileptic seizure prediction experiment was performed on patient A while the self-supervised model was fine-tuned. The results in the "Untransferred" row correspond to the performance of scenarios A→A, B→B, C→C, and D→D, as shown in Table 1. Compared with the results when both the self-supervised model and the downstream model were trained on the same patient, the F2-score of 11 out of the 12 cross-domain scenarios decreased by less than 15%. In addition, it can be observed that under the corresponding untransferred conditions shown in Table 1, half of the 12 cross-domain scenarios performed better than the best baseline algorithm. Overall, the experimental results show that MBrain has excellent transferability.

[0156] (5) Ablation experiment

[0157] This experiment studies the effectiveness of each module in the MBrain proposed in the present invention. Specifically, MBrain is compared with different model variants with different modules removed, as shown below. First, the correlation learning module is removed from the instantaneous time-shift prediction task, and only the contextual representation of one channel is used to predict the local representation of the same channel. This variant is denoted as MBrain-Graph. Secondly, the entire instantaneous time-shift prediction task is deleted, denoted as MBrain-Instant. Then, the delayed time-shift prediction task is deleted, denoted as MBrain-Delay. Finally, the replacement discrimination task is deleted, denoted as MBrain-Replace.

[0158] Table 4 Ablation experiment results

[0159]

[0160] Table 4 shows the results of the ablation study for patient A. It can be observed that the full MBrain achieves the best performance on all evaluation metrics, demonstrating the effectiveness of each module in the proposed model design. For MBrain-Instant, the significant drop in performance indicates that the proxy task of using contextual representations to predict local representations is very important and is the key to learning important representations in time series. For MBrain-Graph, the performance drop indicates that exploiting the correlation between channels can help learn more effective representations. In addition, the performance of MBrain-Delay and MBrain-Replace also drops significantly, indicating that modeling delayed time shifts and distinguishing the replaced data helps summarize and learn more discriminative representations.

[0161] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A self-supervised representation learning method for EEG signals, characterized in that: The following steps are involved: Obtain the original multi-channel EEG signal data of each time period, and use the encoder to obtain the local latent representation and self-context representation of each time period; According to the global context representation, the correlation of different channels in each time period is calculated to obtain a sparse fine-grained correlation matrix. According to the sparse fine-grained correlation matrix, the global representation of each channel in each time period is calculated; according to the global representation of all time periods, the local implicit representation of the time period within k1 steps is predicted, and the loss of the instantaneous time shift prediction task is calculated; According to the original multi-channel EEG signal data of each time period, the long-term correlation matrix between different channels in different time periods is calculated, and the long-term correlation pseudo-label is constructed; the segment representation is obtained according to the context representation of each time period, the correlation probability between different channels in different time periods is predicted, and the loss of the delay time shift prediction task is calculated; Randomly replace the local latent representations of each time period, calculate new self-context representations according to the replaced new local latent representations, predict whether the local latent representations corresponding to each original channel are replaced by other channels according to the new self-context representations, and calculate the loss of the replacement discrimination learning prediction task; The encoder is trained by combining the loss of the instantaneous time shift prediction task, the loss of the delayed time shift prediction task and the loss of the discriminative learning prediction task; the self-context representation of the original multi-channel EEG signal data to be processed is obtained by using the trained encoder, and a segment representation is obtained, and the segment representation is used as the learned EEG signal self-supervisory representation for the epileptic seizure prediction task; The local implicit representation of the time period within k1 steps is predicted based on the global representation of all time periods, and the loss of the instantaneous time shift prediction task is calculated. include: Among them, N t,i represents a random noise set of implicit representations containing a positive sample and several negative samples, Represents the set N t,i The local hidden representation of a positive sample in , that is, the local hidden representation of the i-th channel in the τ+k1-th step of time period t, c t,τ,i represents the global representation of the first τ steps of the i-th channel aggregation in time period t, z j Represents the set N t,i The local hidden representation of the jth sample in is represents the expected operation, that is, averaging the contrast loss of all channels in all time periods within k1 steps; represents the loss of the instantaneous time-shift prediction task; is the learnable bilinear matrix corresponding to the k1-th step prediction; the superscript represents transpose; The method of calculating the long-term correlation matrix between different channels in different time periods based on the original multi-channel EEG signal data in each time period and constructing the long-term correlation pseudo label includes: Where j = 1, ..., C represents the channel index, k2 represents the time step, s t,i Represents the data of the i-th channel in time period t, Represents the data of the jth channel in the time period t+k2, It represents the long-term correlation value between the i-th channel in time period t and the j-th channel in time period t+k2; Construct relevance pseudo labels: in, A pseudo label indicating whether the i-th channel in time period t is correlated with the j-th channel in time period t+k2, θ2 is the correlation threshold; The segment representation is obtained according to the context representation of each time period, the correlation probability between different channels in different time periods is predicted, and the loss of the delay time shift prediction task is calculated. The formula is: Among them, Pooling(.) represents the pooling operation, h t is the segment representation of time period t, is the time period t The self-context representation of the step, MLP(.) represents a multi-layer perceptron, is the predicted probability that two segments are highly correlated, h t,i represents the segment representation of the i-th channel in time period t, C represents the number of channels, The segment representation of the jth channel in the time period t+k2 is represented as follows: represents the expected operation, that is, the average of the cross entropy losses of all channels in all time periods with other channels within k2 steps, is the loss for the delay time-shift prediction task.

2. The method for self-supervised representation learning of EEG signals according to claim 1, characterized in that: The method of using the encoder to obtain the local implicit representation and the self-context representation of each time period includes: Using nonlinear encoder g enc Map the EEG signal data of all channels in the observed time period to the local area and obtain the local implicit representation: Among them, s t is the EEG signal data of time period t, is through g enc The length of the transformed sequence, z t is the local implicit representation of time period t, Indicates the time period t Implicit representation of steps; Using the autoregressive model g ar The historical local information of each channel is summarized to obtain its own context representation as follows: Among them, z t,τ-1 represents the implicit representation of the τth step in the tth time period, represents the self-context representation of the τth step in time period t, when 3. The method for self-supervised representation learning of EEG signals according to claim 1, characterized in that: The formula for calculating the correlation of different channels in each time period based on the global context representation to obtain the sparse fine-grained correlation matrix is: A 1,t (i,j)=A0(i,j)+σ t (i,j)×n t (i,j) Among them, MLP represents multi-layer perceptron, represents the self-context representation of the τth step of the i-th channel in time period t, represents the context representation of the jth channel at the τth step in time period t, n t (i, j) represents the value sampled from the standard normal distribution N(0,1), SoftPlus(.) is the SoftPlus activation function, σ t (i, j) represents the standard deviation of the Gaussian distribution followed by the correlation between the i-th and j-th channels in time period t, A 1,t (i,j) represents the correlation between the i-th and j-th channels in time period t; Filter the elements with correlation less than θ1 to obtain the sparse fine-grained correlation matrix A t : Among them, A t (i,j) represents the sparse fine-grained correlation matrix A t The element in the i-th row and j-th column in is used to represent the correlation between the i-th and j-th channels in the t-th time period after sparseness.

4. The method for self-supervised representation learning of EEG signals according to claim 3, characterized in that: The formula for calculating the global representation of each channel in each time period based on the sparse fine-grained correlation matrix is: in, represents the representation of other channel information before the aggregation of the i-th channel in time period t, Θ is a learnable transformation matrix, ReLU(.) is the ReLU activation function, c tτ represents the global representation of the first τ steps of the i-th channel aggregation in time period t, 5. The method for self-supervised representation learning of EEG signals according to claim 1, characterized in that: When calculating the loss of the replacement discriminative learning prediction task, it includes: Generate pseudo labels based on the replaced new local latent representation: in, is the new local implicit representation of the replaced time period t The elements in Y are the implicit representations of the ith channel in the tth time period and the τth step after replacement; t (τ,i) represents the pseudo label of the i-th channel in the τ-th step; I(.) represents the channel index corresponding to the new local latent representation before it is replaced; Compute the loss for the replacement discriminative learning prediction task: in, is the new self-context representation The vector corresponding to the i-th channel in is the predicted probability that the ith channel is replaced by other channels, E t,τ,i [.] represents the expected operation, that is, the average of the cross entropy loss in all channels in all time periods within τ steps, is the loss for the replacement discriminative learning prediction task.

6. A self-supervised representation learning system for EEG signals, used to implement the self-supervised representation learning method for EEG signals according to claim 1, characterized in that: The system comprises: An EEG signal acquisition module is used to acquire original multi-channel EEG signal data of each time period; The encoder module is used to obtain the local latent representation and self-context representation of each time segment; A correlation learning module is used to calculate the correlation of different channels in each time period according to the global context representation to obtain a sparse fine-grained correlation matrix; The time-shift prediction learning module is used to calculate the global representation of each channel in each time period according to the sparse fine-grained correlation matrix; predict the local implicit representation of the time period within k1 steps according to the global representation of all time periods, and calculate the loss of the instantaneous time-shift prediction task; and, based on the original multi-channel EEG signal data of each time period, calculate the long-term correlation matrix between different channels in different time periods and construct long-term correlation pseudo labels; obtain the segment representation according to the self-context representation of each time period, predict the correlation probability between different channels in different time periods, and calculate the loss of the delayed time-shift prediction task; The method of predicting the local implicit representation of the time period within k1 steps based on the global representation of all time periods and calculating the loss of the instantaneous time shift prediction task includes: Among them, N t,i represents a random noise set of implicit representations containing a positive sample and several negative samples, Represents the set N t,i The local hidden representation of a positive sample in , that is, the local hidden representation of the i-th channel in the τ+k1-th step of time period t, c t,τ,i represents the global representation of the first τ steps of the i-th channel aggregation in time period t, z j Represents the set N t,i The local hidden representation of the jth sample in is represents the expected operation, i.e., averaging the contrast loss of all channels in all time periods within k1 steps; L1 represents the loss of the instantaneous time shift prediction task; is the learnable bilinear matrix corresponding to the k1-th step prediction; the superscript represents transpose; The method of calculating the long-term correlation matrix between different channels in different time periods based on the original multi-channel EEG signal data in each time period and constructing the long-term correlation pseudo label includes: Where j = 1, ..., C represents the channel index, k2 represents the time step, s t,i Represents the data of the i-th channel in time period t, Represents the data of the jth channel in the time period t+k2, It represents the long-term correlation value between the i-th channel in time period t and the j-th channel in time period t+k2; Construct relevance pseudo labels: in, A pseudo label indicating whether the i-th channel in time period t is correlated with the j-th channel in time period t+k2, θ2 is the correlation threshold; The segment representation is obtained according to the context representation of each time period, the correlation probability between different channels in different time periods is predicted, and the loss of the delay time shift prediction task is calculated. The formula is: Among them, Pooling(.) represents the pooling operation, h t is the segment representation of time period t, is the time period t The self-context representation of the step, MLP(.) represents a multi-layer perceptron, is the predicted probability that two segments are highly correlated, h t,i represents the segment representation of the i-th channel in time period t, C represents the number of channels, The segment representation of the jth channel in the time period t+k2 is represented as follows: represents the expected operation, that is, the average of the cross entropy losses of all channels in all time periods with other channels within k2 steps, is the loss of the delayed time-shift prediction task; A replacement discriminant learning module is used to randomly replace the local latent representations of each time period, calculate new self-context representations according to the replaced new local latent representations, predict whether the local latent representations corresponding to each original channel are replaced by other channels according to the new self-context representations, and calculate the loss of the replacement discriminant learning prediction task; A training module, which is used to train the encoder module by combining the loss values ​​calculated by the time-shift prediction learning module and the replacement discrimination learning module; A self-supervised representation generation module uses a trained encoder to obtain the self-context representation of the original multi-channel EEG signal data to be processed, and obtains a segment representation based on the self-context representation of each time period, and uses the segment representation as the generated EEG signal self-supervised representation for the epileptic seizure prediction task.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the self-supervised representation learning method for EEG signals according to any one of claims 1 to 5.

Images