Cross-subject electroencephalogram signal classification method based on online test time domain adaptation

Abstract

The application discloses a cross-subject electroencephalogram signal classification method based on online test time domain adaptation, and steps of the method comprise the following steps: 1, pre-processing original EEG data, including removing noise, segmenting, extracting time-frequency features by using short-time Fourier transform, and obtaining source domain data and target domain data; 2, constructing a source model, a student model and a teacher model based on a CNN network, and training the source model by input data to obtain a pre-training source model; 3, initializing the student model and the teacher model by using the pre-training source model; 4, online optimizing the student model and the teacher model based on a mutual learning strategy on target data flow and realizing classification of electroencephalogram signals. The application can realize rapid classification of electroencephalogram signals under the condition of protecting the privacy of patients, so that the real-time demand of the classification system of electroencephalogram signals in an actual scene can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electroencephalogram signal classification, and particularly to a cross-subject electroencephalogram signal classification method based on online test time domain adaptation. BACKGROUND

[0002] Electroencephalogram (EEG) is a physiological technique for recording electrical activity of the brain. The observed patterns of neural activity from scalp and intracranial EEG are widely used to recognize and predict physiological and psychological states in the field of brain-computer interface such as emotion recognition, motor imagery, medical health, etc. Although deep neural networks (DNNs) have achieved unprecedented success in various brain-computer interface applications, such as seizure prediction, existing methods usually train the model in a patient-specific way, i.e., the training and testing data come from the same subject. Due to the time-varying characteristics and heterogeneity of electroencephalogram signals from different subjects, the performance of the model trained in the source domain will deteriorate severely when it is directly used in the target domain, resulting in poor robustness of the system.

[0003] Although some domain adaptation techniques have been used to solve the above problems, most of the existing domain adaptation techniques need to access the source domain data. Since the electroencephalogram signal contains sensitive physiological information of the user, this may raise privacy protection issues. Recently, researchers have proposed some source-free domain adaptation (SFDA) based methods. SFDA focuses on modifying a well-learned source model to the target source domain data without using any source domain data, which is beneficial to protect the privacy of the source domain data.

[0004] However, existing SFDA methods only focus on offline applications, which means that all electroencephalogram data of the target subject is available before training. In the training phase, offline SFDA optimizes the pre-trained source model using a certain amount of target data through multiple epochs, and then evaluates the performance of the task model using target test data. This setting has several limitations: first, waiting for pre-collection of sufficient unlabelled electroencephalogram samples of the target subject is inefficient and time-consuming. Second, since the offline algorithm needs to access the electroencephalogram data of the target subject multiple times and iterates through multiple epochs to train the classifier to obtain reliable performance, it further increases the cost of algorithm computation time. Third, when new electroencephalogram signal data needs to be classified, the task model needs to be retrained, which is costly. Finally, since electroencephalogram signals have time-varying characteristics and non-stationary characteristics, the classifier needs to be constantly adjusted to adapt to the new electroencephalogram data distribution, and the offline model cannot meet this demand. SUMMARY

[0005] The present application is a cross-subject electroencephalogram signal classification method based on online test time domain adaptation, which can realize dynamic prediction of the electroencephalogram signal of the target subject under the premise of protecting the privacy of the patient, so as to meet the real-time requirement of the electroencephalogram signal classification system in actual application.

[0006] In order to achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0007] The cross-subject electroencephalogram signal classification method based on online test time domain adaptation has the following characteristics:

[0008] Step 1.1, obtain a source domain data set of electroencephalogram signals with labeled category information, and select channel data of the original electroencephalogram signals in the source domain data set to obtain C channel source domain electroencephalogram signals; after slicing the C channel source domain electroencephalogram signals by a sliding window, the time-frequency features of the sliced source domain electroencephalogram signals are extracted by using short-time Fourier transform, and the input shape of the source domain electroencephalogram signals after short-time Fourier processing is reconstructed, so as to obtain N S C S class source domain electroencephalogram signal samples, denoted as source training set

[0009] Among them, represents the i-th source domain electroencephalogram signal sample in the j-th class of electroencephalogram signals, is the label corresponding to the electroencephalogram signal sample ; C S represents the number of categories of electroencephalogram signals, c represents the number of channels of the electroencephalogram signal sample, w represents the width of the sliding window, and h represents the height of the electroencephalogram signal sample;

[0010] Step 1.2, obtain a target domain data set of electroencephalogram signals without labeled category information, and select channel data of the original electroencephalogram signals in the target domain data set to obtain C channel target domain electroencephalogram signals; after slicing the C channel target domain electroencephalogram signals by a sliding window, the time-frequency features of the sliced target domain electroencephalogram signals are extracted by using short-time Fourier transform, and the input shape of the target domain electroencephalogram signals after short-time Fourier processing is reconstructed, so as to obtain a target domain data set composed of N T electroencephalogram segments Among them, Seg n represents the n-th electroencephalogram segment in the target domain data set D T , and Among them, represents the n-th electroencephalogram segment Seg T in the target domain data set D nA small batch of samples at time step t The b-th EEG signal sample in the data;

[0011] Step 2: Construct source model f and student model f based on CNN network. s Teacher model f t ;

[0012] The CNN network includes: a first convolutional module Conv3D, a second convolutional module Conv2D1, a third convolutional module Conv2D2, and a classification module;

[0013] The first convolutional module Conv3D sequentially includes: a first batch normalization layer, a first convolutional layer with stride (1, s1, s1) and kernel (c, k1, k1), a first ReLU nonlinear activation function layer, and a max pooling layer with kernel (1, d1, d1).

[0014] The second convolutional module Conv2D1 includes, in sequence: a second batch normalization layer, a second convolutional layer with stride s2 and kernel k2, a second ReLU nonlinear activation function layer, and a max pooling layer with kernel d2;

[0015] The third convolutional module Conv2D2 includes, in sequence: a third batch normalization layer, a third convolutional layer with stride s3 and kernel k3, a third ReLU nonlinear activation function layer, and a max pooling layer with kernel d3.

[0016] The classification module includes: two connection layers FC1 and FC2, and a first Sigmoid non-linear activation function;

[0017] Step 2.1: Initialize the weights of all convolutional kernels in the CNN network using the Kaiming initializer;

[0018] Step 2.2: Sample the EEG signal from the i-th source domain. The input is processed by the first convolutional module Conv3D in the source model f, which performs initial feature extraction and feature dimensionality reduction to obtain the first feature sequence. Where h′ represents the first feature sequence Height;

[0019] Step 2.3, the first feature sequence After being processed sequentially by the second convolutional module Conv2D1, the second feature sequence is obtained. Where c′ represents the second feature sequence The number of channels, w′ represents the second feature sequence. The width, h″ represents the second feature sequence. Height;

[0020] The second feature sequence After being processed by the third convolutional module Conv2D2, the third feature sequence is output. Where h″′ represents the third feature sequence The height, w″ represents the third feature sequence The width, c″ represents the third feature sequence The number of channels;

[0021] Step 2.4: For the third feature sequence The fourth feature sequence is obtained after the flattening operation.

[0022] Fourth characteristic sequence The input is processed by the classification module and the first fully connected layer FC1 and the first sigmoid nonlinear activation function to obtain the i-th segment of EEG signal sample. The fifth characteristic sequence Where 'a' represents the fifth feature sequence. The height of b represents the fifth feature sequence. The width;

[0023] The fifth feature sequence The sample is then fed into a second fully connected layer (FC2) for processing to obtain the source model f for the i-th segment of the EEG signal. The final Cs logit output values in, The source model f represents the prediction of the i-th segment of the EEG signal sample. The logit value belonging to class j, b′ represents The width of b, and b′ < b;

[0024] Step 2.5: Use the rebalancing loss function shown in equation (1) to establish the backpropagation loss function L. CE :

[0025]

[0026] In equation (1), L i,j This represents the i-th EEG signal sample in the j-th type of EEG signal. The loss is obtained from equation (2):

[0027]

[0028] In equation (2), Represents the i-th EEG signal sample The probability value of belonging to the j-th class is obtained from equation (3):

[0029]

[0030] Step 2.6: Based on the source training set D S The source model f is trained using the Adam optimizer, and the loss function L is calculated. CE Training stops when the number of training iterations reaches the set number or the loss error is less than the set threshold, thus obtaining the optimal source classification model f0.

[0031] Step 3: Initialize n = 1;

[0032] Step 4: Initialize t = 1;

[0033] Step 4.0: Initialize the student model f using the parameters of the optimal source classification model f0. s And teacher model f t Obtain the student model at time step t And teacher model f t t ;

[0034] Step 4.1: Take a small batch of samples Input the teacher model f at time step t respectively t t and student model In the middle, and respectively obtained the teacher model f t t The output contains Cs logit values. Student Model The output contains Cs logit values. in, Representing the teacher model f t t Predict a small batch of samples The bth EEG signal sample The logit value belonging to class j; Representing the student model f t s Predict a small batch of samples The bth EEG signal sample The logit value belonging to class j;

[0035] Step 4.1 Calculate the unsupervised loss L using equation (4). hybrid :

[0036] L hybrid =L ent +δL div +γL sce (4)

[0037] In equation (4), δ and γ both represent regularization coefficients, and L ent L represents the test entropy, which is obtained from equation (5). div Let L represent the KL divergence, which is obtained from equation (6). sce The symmetric cross-entropy is represented and obtained from equation (6);

[0038]

[0039] In the formula, Student model Predict a small batch of samples The bth EEG signal sample The probability value of belonging to the j-th class is obtained from equation (6):

[0040]

[0041]

[0042] In the formula, Representing the teacher model f t t Predict a small batch of samples The bth EEG signal sample The probability value of belonging to the j-th class is obtained from equation (8):

[0043]

[0044]

[0045] Step 4.2 Based on Using the Adam optimizer to optimize the student model at time step t Perform training and minimize the unsupervised loss function L. hybrid When the student model After one epoch, we obtain the student model at time step t+1. At the same time, the teacher model f is used in conjunction with equation (7). t t Parameters at time step t The teacher model f is updated to obtain the teacher model f. t t Parameters at time step t+1

[0046]

[0047] In equation (7), Student model The parameter at time step t+1; α is the sliding factor, and we have:

[0048]

[0049] In equation (8), and Refers to student model For a small batch of samples at time step t The average prediction probabilities obtained when making predictions are the most confident and the least confident, and we have:

[0050]

[0051] In equation (9), and Student model A small batch of samples at time step t The b-th sample

[0052] The most confident and second most confident prediction probabilities;

[0053] Step 4.3 uses a student model with a time step of t+1. For a small batch of samples Perform classification to obtain classification results;

[0054] After assigning t+1 to t in step 4.4, if t>T, it means Seg n If all samples have been classified, proceed to step 4.5; otherwise, return to step 4.0 and execute sequentially.

[0055] After assigning n+1 to n in step 4.5, if n>N T This indicates the target domain dataset. If all samples in the dataset have been classified, then proceed to step 4 sequentially.

[0056] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the cross-subject electroencephalogram (EEG) signal classification method, and the processor is configured to execute the program stored in the memory.

[0057] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the cross-subject electroencephalogram (EEG) signal classification method.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] 1. The application proposes an electroencephalogram signal state classification framework based on online test time domain adaptation technology (TSOTTA), which starts from an existing source model and continuously adapts it to the current test electroencephalogram data, the source domain data is unavailable, which helps to protect the privacy of patients; target data arrives in turn, and each target sample only needs to be accessed once, classification and adaptation are carried out at the same time, thereby meeting the real-time demand of the electroencephalogram classification system.

[0060] 2. The application designs a teacher-student online mutual learning strategy for the time-varying characteristics and heterogeneity of electroencephalogram data, updates the student model by optimizing the mixed loss, and the teacher model is the average value of the continuous student model. The teacher mode and the student mode are updated at the same time, and the student mode is supervised online; this strategy is beneficial to maintain the classification ability of the student model to the electroencephalogram signal of the source domain subject. The adjacent update strategy is beneficial to mine the correlation between electroencephalogram samples, and better electroencephalogram signal classification performance is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 It is a paradigm comparison diagram of the three domain adaptation algorithms involved in the application;

[0062] Figure 2 It is a diagram of the electroencephalogram signal classification method of the application;

[0063] Figure 3 It is a diagram of the model data division process in the application;

[0064] Figure 4 It is a diagram of the model self-training process in the application. DETAILED DESCRIPTION

[0065] In this embodiment, an electroencephalogram signal classification method based on online test time domain adaptation is mainly designed, which is an online electroencephalogram signal classification framework with privacy protection, which simultaneously carries out classification and adaptation of target domain electroencephalogram signals, and the specific paradigm is shown in part (c) of Figure 1 Unlike part (b) of Figure 1 Part (c) of Figure 1 The framework only uses a pre-trained source model and unlabeled target data to improve the performance of the pre-trained source model on the distribution of the target domain through online tuning, and the adaptation process does not use source domain data, and the target domain data arrives online. The overall framework of the method is shown in Figure 2 Specifically, the method is carried out in the following steps:

[0066] This method uses two public electroencephalogram data sets: CHB-MIT and Kaggle; as Figure 3As shown, to maximize the potential of available EEG data, this invention employs a leave-one-subject strategy to evaluate TSOTTA. Specifically, on the CHB-MIT dataset, this invention selects one patient as the target new patient and selects another 19 patients as source patients. In the Kaggle dataset, dog 5 was discarded because its EEG signal channel count differed from the other four dogs. Furthermore, this invention splits the source domain data into two subsets: 9 / 10 of the data is used as the training set x. s tr The remaining 1 / 10 was allocated to the validation set x s val Meanwhile, in order to simulate the scenario where test data arrives sequentially, this invention provides only one unlabeled EEG sample at each step, and this sample can only be accessed once.

[0067] Step 1.1: Obtain the EEG signal source domain dataset with labeled category information, and perform channel data selection on the raw EEG signals in the EEG signal source domain dataset to obtain C channels of source domain EEG signals; after slicing the C channels of source domain EEG signals through a sliding window, use Short Time Fourier Transform (SFT) to extract the time-frequency features of the sliced ​​source domain EEG signals, and reconstruct the input shape of the source domain EEG signals after SFT processing, thereby obtaining N S C with a time step size of t S Source domain EEG signal samples, denoted as the source training set.

[0068] in, This represents the i-th source domain EEG signal sample in the j-th type of EEG signal. EEG signal sample The corresponding tag; C S The number of categories of EEG signals is represented by c, the number of channels of the EEG signal sample is represented by w, the width of the sliding window is represented by h, and the height of the EEG signal sample is represented by h.

[0069] In the CHB-MIT EEG dataset, the number of channels was 22, the sliding window length was 30s, and the EEG signal sampling rate was 256Hz; in the Kaggle EEG dataset, the number of channels was 16, the sliding window length was 30s, and the EEG signal sampling rate was 400Hz.

[0070] Step 1.2, obtain an electroencephalogram target domain data set without labeled class information, and perform channel data selection on original electroencephalogram signals in the electroencephalogram target domain data set to obtain C-channel electroencephalogram signals in the target domain; after slicing the C-channel electroencephalogram signals in the target domain through a sliding window, the time-frequency features of the sliced electroencephalogram signals in the target domain are extracted by using short-time Fourier transform, and the input shape of the electroencephalogram signals in the target domain after short-time Fourier processing is reconstructed, so as to obtain N T electroencephalogram segments constituting a target domain data set wherein Seg n represents the n-th electroencephalogram segment in the target domain data set D T , and wherein, represents the n-th electroencephalogram segment Seg T in the target domain data set D n , and represents the b-th electroencephalogram sample in a small batch of samples at a time step t.

[0071] Step 2, constructing a source model f, a student model f s , and a teacher model f t based on a CNN network;

[0072] The CNN network comprises: a first convolution module Conv3D, a second convolution module Conv2D1, a third convolution module Conv2D2, and a classification module.

[0073] The first convolution module Conv3D comprises, in sequence: a first batch normalization layer, a first convolution layer with a step of (1, s1, s1) and a convolution kernel of (c, k1, k1), a first ReLU nonlinear activation function layer, and a maximum pooling layer with a convolution kernel of (1, d1, d1).

[0074] The second convolution module Conv2D1 comprises, in sequence: a second batch normalization layer, a second convolution layer with a step of s2 and a convolution kernel of k2, a second ReLU nonlinear activation function layer, and a maximum pooling layer with a convolution kernel of d2.

[0075] The third convolution module Conv2D2 comprises, in sequence: a third batch normalization layer, a third convolution layer with a step of s3 and a convolution kernel of k3, a third ReLU nonlinear activation function layer, and a maximum pooling layer with a convolution kernel of d3.

[0076] The classification module comprises: two connection layers FC1 and FC2, and a first Sigmoid nonlinear activation function.

[0077] Step 2.1, initializing all convolution kernel weights in the CNN network using a kaiming initializer;

[0078] Step 2.2: Sample the EEG signal from the i-th source domain. The input is processed by the first convolutional module Conv3D in the source model f, which performs initial feature extraction and dimensionality reduction to obtain the first feature sequence. Where h′ represents the first feature sequence Height;

[0079] Step 2.3, First Feature Sequence After being processed sequentially by the second convolutional module Conv2D1, the second feature sequence is obtained. Where c′ represents the second feature sequence The number of channels, w′ represents the second feature sequence. The width, h″ represents the second feature sequence. Height;

[0080] Second characteristic sequence After being processed by the third convolutional module Conv2D2, the third feature sequence is output. Where h″′ represents the third feature sequence The height, w″ represents the third feature sequence The width, c″ represents the third feature sequence The number of channels;

[0081] Step 2.4: For the third feature sequence The fourth feature sequence is obtained after the flattening operation.

[0082] Fourth characteristic sequence The input is processed by the classification module, and after passing through the first fully connected layer FC1 and the first sigmoid nonlinear activation function, the i-th segment of the EEG signal sample is obtained. The fifth characteristic sequence Where 'a' represents the fifth characteristic sequence The height of b represents the fifth feature sequence. The width;

[0083] The fifth feature sequence The sample is then fed into a second fully connected layer (FC2) for processing to obtain the source model f for the i-th segment of the EEG signal. The final Cs logit output values in, The source model f represents the prediction of the i-th segment of the EEG signal sample. The logit value belonging to class j, b′ represents The width of b, and b′ < b;

[0084] Step 2.5, the loss function L of back propagation is established by using the class rebalancing loss function shown in formula (1) CE :

[0085]

[0086] In formula (1), L i,j represents the loss of the i-th electroencephalogram signal sample in the j-th class of electroencephalogram signals, and is obtained by formula (2):

[0087]

[0088] In formula (2), P represents the probability value of the i-th electroencephalogram signal sample belonging to the j-th class, and is obtained by formula (3):

[0089]

[0090] In formula (3), L represents the logit value output by the source model f to the i-th electroencephalogram signal sample ;

[0091] Step 2.6, based on the source training set D S , the source model f is trained by using the Adam optimizer, and the loss function L CE is calculated. In this example, the batch size is set to 64, and the initial learning rate of the ADAM optimizer is set to 0.001. In addition, the early stopping technique is applied, and when the validation error does not decrease for 10 training periods, the model training is stopped, so as to obtain the optimal source classification model f0;

[0092] Step 3, the process of online optimization of the model is shown in Figure 4 , and n is initialized to 1;

[0093] Step 4, t is initialized to 1;

[0094] Step 4.0, the parameters of the optimal source classification model f0are used to initialize the student model f s and the teacher model f t respectively, so as to obtain the student model f and the teacher model f t t at the time step t;

[0095] Step 4.1, a small batch of samples is input into the teacher model f t t and the student model f at the time step t respectively, and the teacher model f t ​t Cs output logit values student model Cs output logit values wherein, denotes the teacher model f t t predicting a mini-batch of samples the b-th EEG signal sample in the mini-batch the logit value belonging to the j-th class; denotes the student model f t s predicting a mini-batch of samples the b-th EEG signal sample in the mini-batch the logit value belonging to the j-th class.

[0096] Step 4.1 calculating the loss unsupervised loss L hybrid using equation (4)

[0097] L hybrid = L ent + δL div + γL sce (4)

[0098] In equation (4), δ and γ both denote regularization coefficients, L ent denotes the test entropy and is obtained from equation (5), L div denotes the KL divergence and is obtained from equation (6), L sce denotes the symmetric cross-entropy and is obtained from equation (6);

[0099]

[0100] wherein, denotes the student model predicting a mini-batch of samples the b-th EEG signal sample in the mini-batch the probability value belonging to the j-th class and is obtained from equation (6):

[0101]

[0102]

[0103] wherein, denotes the teacher model f t t predicting a mini-batch of samples the b-th EEG signal sample in the mini-batch the probability value belonging to the j-th class and is obtained from equation (8):

[0104]

[0105]

[0106] where j represents the number of classes of the electroencephalogram sample;

[0107] Step 4.2 is based on The student model at time step t is trained using the Adam optimizer and the unsupervised loss loss function L is minimized hybrid When the student model goes through an epoch, the student model at time step t+1 is obtained Meanwhile, the teacher model f t t The parameters of the student model at time step t are updated and the teacher model f t t The parameters of the student model at time step t+1 are updated

[0108]

[0109] In formula (7), represents the student model at time step t+1; and a is a sliding factor, and has:

[0110]

[0111] In formula (8), and respectively represent the most confident and second confident average prediction probability of the student model when predicting a small batch of samples at time step t, and have:

[0112] In formula (9),

[0113] and represent the most confident and second confident prediction probability of the student model at time step t for the b-th sample in a small batch of samples ;

[0114]

[0115] Step 4.3 uses the student model at time step t+1 to classify a small batch of samples and obtains a classification result;

[0116] Step 4.4 assigns t+1 to t, and if t>T, it indicates that Seg​n all the samples in the target domain data set are classified, step 4.5 is executed, otherwise, step 4.0 is returned to be sequentially executed;

[0117] After step 4.5 assigns n+1 to n, if n>N T , it indicates that all the samples in the target domain data set are classified; otherwise, step 4 is returned to be sequentially executed.

[0118] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0119] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above method.

[0120] The online test time domain adaptation technology in the present application is respectively experimented on CHB-MIT and Kaggle two public data sets. Due to different experimental conditions and settings, the present application cannot be directly compared with existing electroencephalogram signal classification algorithms. In order to evaluate the effectiveness of TSOTTA, the present application compares it with typical online test time domain adaptation methods in the image field. In this example, four widely used evaluation indexes are used to measure the model performance. Sensitivity (SEN) refers to the proportion of all positive samples that are correctly classified as positive classes. The greater the SEN, the more accurate the positive sample classification ability of the classifier. Specificity (SPEC) represents the proportion of all negative samples that are correctly classified as negative classes. It represents the ability of the classifier to correctly classify negative samples. AUC is an important index for balancing the prediction performance of the model. The AUC value of a random classifier is 0.5, and the AUC value of a perfect classifier can reach 1. In the present application, paired t-test statistical analysis is performed on these methods based on AUC at a significance level of 0.05. It can measure whether the model is better than the comparison classifier. When the p-value is less than 0.05, it indicates that the model of the present application is significantly better than the comparison classifier at a significance level of 0.05. The specific statistical results on the CHB-MIT and Kaggle data sets are shown in Tables 3 and 4, respectively. Table 1 shows the performance comparison of TSOTTA and four comparison algorithms on the CHMB-MIT database; Table 2 shows the performance comparison of TSOTTA in the present application applied to the Kaggle data set and the four comparison methods. Experiments show that the TSOTTA proposed in the present application can effectively solve the online test time domain adaptation problem in the electroencephalogram signal classification task, and improve the generalization ability and classification ability of the model.

[0121] ​Table 1 Performance comparison of TSOTTA and four baseline methods on CHB-MIT database

[0122]

[0123]

[0124] Table 2 Performance comparison of TSOTTA and four baseline methods on Kaggle database

[0125]

[0126] Table 3 t-test statistical analysis of different domain adaptation methods on CHB-MIT dataset at a significance level of 0.05

[0127]

[0128] Table 4 t-test statistical analysis of different domain adaptation methods on Kaggle dataset at a significance level of 0.05

[0129]

[0130] In summary, the present application proposes an online test time domain adaptation framework for the problems of patient data privacy leakage and offline system unable to learn online in clinical practical application, which not only considers the problem of patient privacy protection, but also realizes online optimization of the task model through the teacher-student mutual learning strategy, so as to perform online classification setting of electroencephalogram signal and achieve faster and more accurate electroencephalogram signal classification effect.

Claims

1. A cross-subject electroencephalogram signal classification method based on online test time domain adaptation, characterized in that, The procedure is as follows: Step 1.1: Obtain the EEG signal source domain dataset with labeled category information, and perform channel data selection on the original EEG signals in the EEG signal source domain dataset to obtain source domain EEG signals with C channels; After slicing the source domain EEG signals of the C channels using a sliding window, the time-frequency features of the sliced ​​source domain EEG signals are extracted using Short-Time Fourier Transform (SFT), and the input shape of the source domain EEG signals after SFT processing is reconstructed to obtain N. S The step size for a period of time is C S Source domain EEG signal samples, denoted as the source training set. ,in, Indicates the j-th type of EEG signal. One source domain EEG signal sample, EEG signal sample The corresponding tag; C S The number of categories representing brainwave signals. The number of channels in the EEG signal sample is represented by w, the width of the sliding window is represented by h, and the height of the EEG signal sample is represented by h. Step 1.2: Obtain the target domain dataset of EEG signals without labeled category information, and perform channel data selection on the original EEG signals in the target domain dataset to obtain C channels of target domain EEG signals; after slicing the C channels of target domain EEG signals through a sliding window, use short-time Fourier transform to extract the time-frequency features of the sliced ​​target domain EEG signals, and reconstruct the input shape of the target domain EEG signals after short-time Fourier transform processing, thereby obtaining... The target domain dataset consists of several EEG signal segments. ,in, Represents the target domain dataset The nth segment of the EEG signal, and ,in, Represents the target domain dataset The nth EEG segment A small batch of samples at time step t The b-th EEG signal sample in the dataset; B represents the number of samples in a batch; Step 2, construct source model f, student model f based on CNN network s , teacher model f t ; The CNN network includes: a first convolutional module Conv3D, a second convolutional module Conv2D1, a third convolutional module Conv2D2, and a classification module; The first convolutional module Conv3D sequentially includes: a first batch normalization layer, a first convolutional layer with stride (1, s1, s1) and kernel (c, k1, k1), a first ReLU nonlinear activation function layer, and a max pooling layer with kernel (1, d1, d1). Second convolutional module The layers consist of: a second batch normalization layer, a second convolutional layer with a stride of s2 and a kernel of k2, a second ReLU nonlinear activation function layer, and a max pooling layer with a kernel of d2. The third convolutional module Conv2D2 includes, in sequence: a third batch normalization layer, a third convolutional layer with stride s3 and kernel k3, a third ReLU nonlinear activation function layer, and a max pooling layer with kernel d3. The classification module includes: two connection layers. and The first Sigmoid nonlinear activation function; Step 2.1: Initialize the weights of all convolutional kernels in the CNN network using the Kaiming initializer; Step 2.2, the first Source domain EEG signal samples The input is given to the source model f and passes through the first convolutional module in the source model f. Initial feature extraction and feature dimensionality reduction are performed to obtain the first feature sequence. ,in, Represents the first characteristic sequence Height; Step 2.3, the first feature sequence After being processed sequentially by the second convolutional module Conv2D1, the second feature sequence is obtained. ;in, Represents the second characteristic sequence The number of channels, Represents the second characteristic sequence width, Represents the second characteristic sequence Height; The second feature sequence After being processed by the third convolutional module Conv2D2, the third feature sequence is output. ,in, Represents the third characteristic sequence height, Represents the third characteristic sequence width, Represents the third characteristic sequence The number of channels; Step 2.4: For the third feature sequence The fourth feature sequence is obtained after the flattening operation. ; Fourth characteristic sequence The data is input into the classification module and passes through the first fully connected layer. After processing with the first Sigmoid nonlinear activation function, we obtain the... Segment EEG signal samples The fifth characteristic sequence Where a represents the fifth feature sequence The height of b represents the fifth feature sequence. The width; The fifth feature sequence Then input a second fully connected layer The process is performed to obtain the source model f for the first... Segment EEG signal samples The final Each logit output value ,in, The source model f represents the prediction of the i-th segment of the EEG signal sample. The logit value belonging to class j, express The width, and ; Step 2.5: Establish the backpropagation loss function using the rebalancing loss function shown in equation (1). : (1) In equation (1), This represents the i-th EEG signal sample in the j-th type of EEG signal. The loss is obtained from equation (2): (2) In equation (2), This indicates that the student model predicts the i-th EEG signal sample. The probability value of belonging to the j-th class is obtained from equation (3): (3) Step 2.6: Based on the source training set D S The source model f is trained using the Adam optimizer, and the loss function is calculated. Training stops when the number of training iterations reaches the set number or the loss error is less than the set threshold, thus obtaining the optimal source classification model f0. Step 3: Initialize n=1; Step 4: Initialize t=1; Step 4.0: Initialize the student model f using the parameters of the optimal source classification model f0. s And teacher model f t Obtain the student model at time step t Teacher Model ; Step 4.1: Take a small batch of samples Teacher models with time step t are input respectively. and student model In the middle, and respectively obtained teacher models. Output logit value Student model Output logit value ,in, Teacher model Predict a small batch of samples The bth EEG signal sample The logit value belonging to class j; Student model Predict a small batch of samples The bth EEG signal sample The logit value belonging to class j; Step 4.1 Calculate the unsupervised loss using equation (4). : (4) In equation (4), and Both represent regularization coefficients. Let the test entropy be represented, and it is obtained from equation (5). Let KL divergence be represented, and obtained from equation (6). The symmetric cross-entropy is represented and obtained from equation (6); (5) In the formula, Student model Predict a small batch of samples The bth EEG signal sample The probability value of belonging to the j-th class is obtained from equation (6): (6) (7) In the formula, Teacher model Predict a small batch of samples The bth EEG signal sample The probability value of belonging to the j-th class is obtained from equation (8): (8) (9) In equation (9), This indicates that the teacher model predicts the i-th EEG signal sample. The probability value of belonging to the j-th class. This indicates that the student model predicts the i-th EEG signal sample. The probability value of belonging to the j-th class; Step 4.2 Based on The Adam optimizer was used to optimize the student model at time step t. Perform training and minimize the unsupervised loss function. When the student model After one epoch, we obtain the student model at time step t+1. Meanwhile, the teacher model is analyzed using equation (7). Parameters at time step t Update the model to obtain the teacher model. Parameters at time step t+1 : (10) In equation (10), Student model The parameters at time step t+1; It is the sliding factor, and we have: (11) In equation (11), and Refers to student model For a small batch of samples at time step t The average prediction probabilities obtained when making predictions are the most confident and the least confident, and we have: ； (12) In equation (12), and Student model A small batch of samples at time step t The b-th sample The most confident and second most confident prediction probabilities; Step 4.3 uses a student model with a time step of t+1. For a small batch of samples Classify the data to obtain the classification results; After assigning t+1 to t in step 4.4, if t>T, then it means... If all samples have been classified, proceed to step 4.5; otherwise, return to step 4.0 and execute sequentially. After assigning n+1 to n in step 4.5, if n> This indicates the target domain dataset. If all samples in the dataset have been classified, then proceed to step 4 sequentially.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the cross-subject EEG signal classification method of claim 1, the processor being configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when run by a processor, performs the steps of the cross-subject EEG signal classification method of claim 1.

Images