An electroencephalogram signal privacy protection method based on knowledge distillation

Abstract

The application discloses a kind of electroencephalogram privacy protection methods based on knowledge distillation, comprising:1, the pre-processing of original data, including data selection, sample selection, data up-sampling and down-sampling;2, establish deep learning model based on convolutional neural network (CNN) and self-attention network (Transformer);3, source model pre-training, the data of subject is used to train source model, and the source model with good classification effect is obtained;4, forgetting training data selection, including the selection of forgetting subject data and the selection of retaining subject data;5, forgetting model training, relying on memory teacher model and forgetting teacher model, the forgetting model is supervised training.The application applies knowledge distillation to electroencephalogram classification model, can make the model effectively forget the electroencephalogram information of specific subject, and provide effective electroencephalogram data privacy protection for subject.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electroencephalogram signal classification, in particular to a privacy protection method for deleting specific subject electroencephalogram signal information from a deep learning model based on knowledge distillation. BACKGROUND

[0002] Electroencephalogram (EEG) is a physiological technique for recording the physiological electrical signals of the brain. The observed patterns of neural activity from scalp and intracranial EEG are widely used to recognize and predict physiological and psychological states in the field of brain-computer interface, such as emotion recognition, motor imagery, medical health, etc. Linear or nonlinear features extracted manually using traditional machine learning methods, such as auto-regressive coefficients and Lyapunov exponents, have achieved some success in strictly controlled experimental environments. However, these manually extracted features often require researchers to have rich professional knowledge and conduct a large number of experimental attempts. In addition, in real EEG recordings affected by various artifacts, manually extracted features often only cover part of the EEG information, resulting in poor robustness of the system.

[0003] Deep learning algorithms have been widely used in EEG signal classification and prediction due to their excellent generalization ability and strong ability to automatically learn efficient features. At present, most deep learning methods for EEG signal classification will first perform feature preprocessing, such as short-time Fourier transform and common spatial patterns. Although these preprocessing operations on the original EEG can obtain cleaner data, they may also lose some important information. In recent years, models using feature preprocessing and directly using raw EEG signals usually have more complex architectures and larger kernels, resulting in greater memory resource consumption and computing power.

[0004] At present, most deep learning algorithms for EEG signal classification are usually used as feature classifiers. Researchers extract time-domain features, frequency-domain features or time-frequency domain features from EEG signals based on existing professional knowledge, and then use deep learning algorithms for classification tasks. Although this method has achieved good classification performance, this approach requires deep mathematical knowledge for feature extraction, and it ignores the powerful data-driven capability of deep learning algorithms. Meanwhile, the extracted features are better data representations to some extent, but they also lose much spatial and temporal correlation information existing in the original multi-channel EEG data. There are also a few deep learning algorithms using end-to-end architecture, but they have not fully utilized the spatial and temporal correlation information existing in multi-channel EEG signals. Due to various conditions, the total amount of EEG signal data is severely insufficient, which greatly limits the development of EEG signal classification methods. And due to the serious data imbalance problem existing in different categories of EEG data, the development of classification methods is also greatly limited.

[0005] In recent years, people pay more and more attention to privacy protection issues. Electroencephalogram contains a lot of sensitive information that can reflect the physiological and psychological state of a person. When the electroencephalogram is used to train a model of a deep learning algorithm, the sensitive information will be recorded in the model in the form of model parameters. This information recorded in the form of parameters in the model can be extracted by'model reverse attack' method by the wrongdoers. This greatly threatens the privacy and security of the user. SUMMARY

[0006] To overcome the above shortcomings, the present application provides an electroencephalogram privacy protection end-to-end method based on knowledge distillation, which aims to solve the problem of electroencephalogram privacy protection in the deep learning scenario. By constructing a teacher-student framework, a student model that forgets the electroencephalogram information of a specific user is obtained under the supervision of the teacher model, thereby realizing the privacy protection of the user's electroencephalogram.

[0007] To achieve the above-mentioned application purposes, the following technical solutions are adopted:

[0008] The electroencephalogram privacy protection method based on knowledge distillation has the following characteristics:

[0009] Step 1, obtaining the electroencephalogram data of K subjects with labeled category information, and performing sliding slice processing on the electroencephalogram of each category to obtain N electroencephalogram samples with a time length of T and their corresponding category labels, denoted as a training sample set Wherein, x k,i ∈R 1×L×C represents the electroencephalogram data of the kth subject, x k,i represents the i-th electroencephalogram sample of the kth subject, C represents the number of channels of the electroencephalogram sample, L represents the length of the electroencephalogram sample, y k,i represents the corresponding category label, and n represents the number of samples of the electroencephalogram data of the subject.

[0010] Step 2, establishing a joint network based on CNN and Transformer, including a redundancy removal module, a joint module, and a weighted classification module.

[0011] Step 2.1, the redundancy removal module includes a first convolutional block and a second convolutional block in sequence.

[0012] Each convolutional block is composed of a convolutional layer, a normalization layer and a nonlinear activation layer in sequence.

[0013] The i-th electroencephalogram sample x k,iThe i-th feature map f of the k-th subject is input into the joint network and processed by the de-redundancy module, and the i-th feature map f of the k-th subject is output k,i ;

[0014] Step 2.2, the joint module processes the i-th feature map f k,i , and obtains the i-th local inter-channel feature f_c k,i and the i-th long-range intra-channel feature f_t k,i of the k-th subject;

[0015] Step 2.3, the weighting classification module processes f_c k,i and f_t k,i , and obtains the probability distribution of the i-th electroencephalogram signal sample x k,i of the k-th subject;

[0016] Step 3, model training, including training of a source model, a teacher model and a student model;

[0017] Step 3.1, training of the source model;

[0018] Based on the training sample set D, cross-entropy is used as the loss function of the joint network, and the joint network is trained using the Adam optimizer until the maximum number of iterations is reached or the loss is not decreasing, thereby obtaining the trained joint network as the source model;

[0019] Step 3.2, training of the teacher network, including training of a memory teacher model and a forgetting teacher model;

[0020] Step 3.2.1, the memory teacher model;

[0021] The memory teacher model is constructed based on the CNN and Transformer joint network, and the parameter weight of the source model is used as the parameter weight of the memory teacher model;

[0022] Step 3.2.2, the forgetting teacher model;

[0023] The forgetting teacher model is constructed based on the CNN and Transformer joint network, and the forgetting training sample set is selected from the electroencephalogram signal samples of a subject in the training sample set D as the electroencephalogram signal samples of a forgetting subject , and the electroencephalogram signal samples of the remaining subjects are used as the electroencephalogram signal samples of a retention subject; wherein, represents the i-th electroencephalogram signal sample of the forgetting subject, represents the corresponding class label;

[0024] The EEG signal sample D of the forgetting subject f The cross entropy is used as the loss function of the forgetting teacher model, and the Adam optimizer is used to train the forgetting teacher model until the maximum iteration number is reached or the loss is not reduced, so that the trained forgetting teacher model is obtained;

[0025] Step 3.3, training of the student model, comprising: data selection and training of the student model;

[0026] Step 3.3.1, data selection;

[0027] Part of the EEG signal samples are selected from the EEG signal samples of the retention subject, and part of the EEG signal samples are also selected from the EEG signal samples D of the forgetting subject f , and jointly constitute a joint training sample set;

[0028] Step 3.3.2, training of the student model;

[0029] Step a, constructing the student model based on the CNN and the Transformer joint network, and setting the weight parameters of the student model as the weight parameters of the source model;

[0030] Step b, constructing the joint loss function of the student model, comprising: retention loss and forgetting loss;The retention loss is the KL divergence between the probability distribution of the sample output by the memory teacher model and the student model, and the forgetting loss is the inverse of the KL divergence between the probability distribution of the sample output by the forgetting teacher model and the student model;The joint loss function is the weighted sum of the retention loss and the forgetting loss;

[0031] Step c, based on the joint training sample set, using the Adam optimizer to train the student model, and calculating the joint loss function until the maximum iteration number is reached or the loss is not reduced, so that the student model which removes the data information of the forgetting subject and retains the data information of the retention subject is obtained, and is used for privacy protection of the EEG data of the forgetting subject.

[0032] The EEG signal privacy protection method based on knowledge distillation provided by the application also has the characteristics that the joint module comprises: a parallel CNN branch and a Transformer branch;

[0033] Step 2.2.1, the CNN branch is based on a convolutional neural network and comprises a plurality of Conv modules, each Conv module comprises six similar convolutional blocks, and taking the first Conv module as an example, the convolutional blocks contained therein are in turn: a third convolutional block, a fourth convolutional block, a fifth convolutional block, a sixth convolutional block, a seventh convolutional block and an eighth convolutional block;

[0034] The convolution kernel size in the third, fifth and seventh convolution blocks is the same, and the convolution kernel size in the fourth, sixth and eighth convolution blocks is the same.

[0035] The i-th feature map f k,i is input into the CNN branch, and after being processed by the six convolution blocks in turn, the i-th local inter-channel feature f_c k,i of the k-th subject is output.

[0036] Step 2.2.2, the Transformer branch is based on a self-attention network and includes, in turn, a mapping module and a self-attention module.

[0037] The mapping module is composed of, in turn, a transpose layer, a merging layer and a fully connected sub-block.

[0038] The i-th feature map f k,i is input into the Transformer branch and processed by the mapping module to obtain the i-th input token P k,i of the k-th subject.

[0039] The self-attention module is composed of a plurality of Trans sub-modules with the same structure.

[0040] The i-th input token P k,i is input into the self-attention module for processing, and the i-th long-range intra-channel feature f_t k,i of the k-th subject is output.

[0041] The weighted classification module includes, in turn, a feature fusion block and a classification block.

[0042] Step 2.3.1, the feature fusion module includes parallel CNN logical branches and Transformer logical branches and a weighted fusion layer, wherein the CNN logical branches include, in turn, a convolution layer, a merging layer and two fully connected layers, and the Transformer logical branches include, in turn, a segmentation layer and a fully connected layer.

[0043] f_c k,i is input into the CNN logical branch for processing, and the i-th CNN branch logical output l_c k,i of the k-th subject is obtained.

[0044] f_t k,i is input into the Transformer logical branch for processing, and the i-th Transformer branch logical output l_t k,i of the k-th subject is obtained.

[0045] l_c k,iand l_t k,i Input into the weighted fusion layer for processing, and get the i-th fusion logic output l_r of the k-th subject k,i ;

[0046] Step 2.3.2, the classification block comprises: a softmax layer;

[0047] Input l_r k,i Into the classification block, and processed by the softmax layer, get the i-th electroencephalogram sample x k,i The probability distribution belongs to Indicates the probability that the sample x k,i Belongs to the m-th electroencephalogram sample, and M represents the number of classes of electroencephalogram samples.

[0048] The electronic device comprises a memory and a processor, and the feature is that the memory is used to store a program supporting the processor to execute the electroencephalogram privacy protection method, and the processor is configured to execute the program stored in the memory.

[0049] The computer readable storage medium stores a computer program, and the feature is that when the computer program is run by a processor, the steps of the electroencephalogram privacy protection method are executed.

[0050] Compared with the prior art, the beneficial effects of the present application are:

[0051] 1. The present application uses a deep neural network to perform an electroencephalogram classification task, without any denoising processing and backend processing, to complete end-to-end classification of electroencephalogram signals, and has higher timeliness and robustness than previous electroencephalogram classification methods.

[0052] 2. The present application combines the advantages of CNN and Transformer, fuses the inter-channel correlation of multi-channel electroencephalogram signals extracted by the CNN branch and the intra-channel information of multi-channel electroencephalogram extracted by the Transformer, and improves the accuracy of the classification result.

[0053] 3. The present application uses knowledge distillation to perform a user privacy protection task, deletes the electroencephalogram information of the forgotten subjects in the model in a supervised training manner by constructing a teacher-student framework, while retaining the electroencephalogram information of the retained subjects, thereby realizing privacy protection of the retained patient electroencephalogram signals.

[0054] 4、The application utilizes the characteristics of deep neural network feature learning, randomly selects a small part of data of the subjects to delete the electroencephalogram information of the forgotten subjects, compared with the retraining method, only a small amount of original data is used, not only the time and resource consumption of model training are reduced, but also better privacy protection is provided for the subjects. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 It is a training flowchart;

[0056] Figure 2 It is a schematic diagram of the CNN and Transformer combined network in the application. DETAILED DESCRIPTION

[0057] In this embodiment, a knowledge distillation-based electroencephalogram signal privacy protection deep learning method is used. In the first step, a combined network based on CNN and Transformer is used to train a source model with good classification effect on all subject data. The parameters of the source model are used to initialize the memory teacher model and the student model. In the second step, the same network is used to train a forgotten teacher model on the forgotten subject data. In the third step, the memory teacher model and the forgotten teacher model are used to jointly guide the training of the student model, as shown in Figure 1 The method is performed according to the following steps:

[0058] Step 1, obtain the electroencephalogram signal data of K subjects with labeled category information, and perform sliding slice processing on the electroencephalogram signal of each category to obtain N electroencephalogram signal samples with a time length of T and their corresponding category labels, denoted as a training sample set Wherein, x k,i ∈R 1×L×C represents the electroencephalogram signal data of the kth subject, x k,i represents the i-th electroencephalogram signal sample of the kth subject, C represents the number of channels of the electroencephalogram signal sample, L represents the length of the electroencephalogram signal sample, y k,i represents the corresponding category label, and n represents the number of samples of the electroencephalogram signal data of the subject. In this example, the number of channels of the electroencephalogram signal is 22, the sample length is 256*T (the signal sampling rate is 256Hz), and T is 4s. The data set used is the public electroencephalogram data set: CHB-MIT electroencephalogram data set.

[0059] Step 2, establish a combined network based on CNN and Transformer, including: a redundancy removal module, a joint module, and a weighted classification module. The overall structure of the combined network is shown in part (a) of Figure 2 , wherein Stem represents the redundancy removal module, the part between Stem and the classifier is the joint module, and the classifier represents the weighted classification module.

[0060] The step 2.1, the redundancy removing module comprises: a first convolution block and a second convolution block.

[0061] Each convolution block is sequentially composed of a convolution layer, a normalization layer and a nonlinear activation layer; in the present example, the convolution kernel size of the convolution layer in the first convolution block is (32, 1), the step size is (10, 1), and the padding size is (16, 1); the convolution kernel size of the convolution layer in the second convolution block is (3, 3), the step size is (2, 1), and the padding size is (1, 0); the normalization layer uses batch normalization, and the nonlinear activation layer uses a ReLU function. The preprocessed sample x k,i ∈R 1×L×C , 1 represents the number of channels, L represents the length of the sample, and C represents the number of channels of the sample. In the redundancy removing module, the first convolution block is processed first to obtain and then the second convolution block is processed to obtain The redundancy removing module can be represented as:

[0062] f(x) = ReLU(BN(Conv(ReLU(BN(Conv(x))))) (1)

[0063] In formula (1), Conv represents a convolution layer, BN represents a batch normalization layer, and ReLU represents a nonlinear activation layer.

[0064] Step 2.2, the joint module, comprising: a parallel CNN branch and a Transformer branch;

[0065] Step 2.2.1, the CNN branch is based on a convolutional neural network and is composed of multiple Conv modules, each of which is composed of six convolution blocks similar in structure. Taking the first Conv module as an example, the convolution blocks it contains are sequentially: a third convolution block, a fourth convolution block, a fifth convolution block, a sixth convolution block, a seventh convolution block and an eighth convolution block, as shown in the Conv module in subfigure (b) in Figure 2 In the present example, the convolution blocks in the CNN branch each sequentially comprise a convolution layer, a regularization layer and a nonlinear activation layer. The convolution kernel size of the convolution layer in the third, fifth and seventh convolution blocks is (3, 3), and the padding size is (1, 1); the convolution kernel size of the convolution layer in the fourth, sixth and eighth convolution blocks is (1, 1). The third and sixth convolution blocks expand the number of channels of the sample. The regularization layer adopts batch normalization, and the nonlinear activation layer adopts a ReLU function.

[0066] The specific implementation process is as follows: the feature map is input into the CNN branch, first input into the third convolution block, expand its channel number, and obtain the feature map The feature map is then input into the fourth and fifth convolutional blocks, keeping the number of channels unchanged; it is then input into the sixth convolutional block, expanding its number of channels to 64, thus obtaining the feature map. The feature maps are then fed into the seventh and eighth convolutional blocks for processing, keeping the number of channels constant, to obtain the output features of the CNN branches.

[0067]

[0068] Step 2.2.2: The Transformer branch is based on a self-attention network and includes, in sequence, a mapping module and a self-attention module. The mapping module is... Figure 2 Projection of neutron graph (a), self-attention module as follows Figure 2 The Trans module of subgraph (b) in the diagram is shown;

[0069] The mapping module consists of a transpose layer, a merge layer, and a fully connected sub-block. In this example, the transpose layer swaps the dimensions of the feature maps, and the merge layer has two functions: first, to merge the dimensions of the feature maps; and second, to merge the randomly initialized classes. Embedded into the feature map, the fully connected sub-block contains a fully connected layer, a regularization layer, and a non-linear activation layer. The normalization layer in the fully connected sub-block uses layer normalization, and the non-linear activation layer uses the GELU function. The specific implementation process of the mapping module is as follows: Feature Map The input is first fed into the transpose layer, and then processed to obtain the output. The two-dimensional output is obtained after merging layer processing.

[0070] The self-attention module consists of multiple structurally similar Trans sub-modules, with the specific structure as follows: Figure 2 As shown in the Trans module of neutron graph (b), in this example, the Trans submodule sequentially includes a first fully connected sub-block, a variable-dimensional layer, a multi-head attention block, and a second fully connected sub-block. The two fully connected sub-blocks have the same structure but different internal parameter settings. Both contain a fully connected layer, a normalization layer, and a nonlinear activation layer. Here, the normalization layer uses layer normalization, and the nonlinear activation layer uses the GELU function. The multi-head attention block sequentially includes a segmentation layer, an attention layer, and a restoration layer. Its function is to divide the input into three feature elements—query, key, and value—based on the pre-set number of attention heads and process them.

[0071] The specific implementation process of the Trans submodule is as follows: The two-dimensional feature is input into the Trans submodule, and first processed by the first fully connected sub-block to obtain the dimension-expanded feature f. l k,i ∈R (C+1)×EE represents the size of the pre-defined feature dimension after expansion; it is then input into the variable-dimensional layer, where the features are transformed into words P according to the pre-defined number of attention heads. k,i ∈R 3 ×B×H×(C+1)×(E / (3*H)) B represents the number of samples in a batch, and H represents the number of attention heads. The tokens are input into the multi-head attention block, and the tokens are segmented into three feature elements—query, key, and value (hereinafter referred to as q, k, and v)—according to the first dimension. These three feature elements can be represented as follows:

[0072]

[0073] The attention features are calculated using these three feature elements. The calculation process is as follows:

[0074]

[0075] Where, k T This represents the transpose of feature element k. Let represent the column number of feature element k, and softmax(·) denote the softmax function. The calculated attention features are input into the second fully connected sub-block, and processed to obtain the feature output f_t of the Transformer branch. k,j ∈R (C+1)×64 .

[0076] Step 2.3: The weighted classification module includes, in sequence, a feature fusion block and a classification block, with the specific structure as follows: Figure 2 The subgraph (c) in the diagram is shown;

[0077] Step 2.3.1: The feature fusion module includes parallel CNN logical branches and Transformer logical branches, as well as a weighted fusion layer. The CNN logical branch sequentially includes a convolutional layer, a merging layer, and two fully connected layers. The Transformer logical branch sequentially includes a segmentation layer and a fully connected layer. In this example, the convolutional kernel size of the convolutional layer in the CNN logical branch is (6,2), and the stride is (6,4). This convolutional layer reduces the number of channels in the model features. The merging layer merges different dimensions of the features, and the two fully connected layers reduce the dimensionality of the features, transforming them into logical outputs. The segmentation layer in the Transformer logical branch separates the class tokens embedded in the features, and the fully connected layer transforms the separated class tokens into logical outputs. The weighted fusion layer fuses the logical outputs of the CNN and Transformer logical branches into a joint logical output. The specific implementation is as follows: Feature map f_c k ,iThe input to the CNN's logical branch first goes through convolutional layers to reduce the number of channels and feature dimensions; then it goes through merging layers to combine different dimensions; finally, it goes through fully connected layers to obtain the logical output l_c. k,i Feature map f_t k,j The input is fed into the Transformer logic branch, where it first passes through a segmentation layer to extract the embedded class token (c_t∈R). 1×64 The logic output l_t is obtained by processing c_t through a fully connected layer. k,i Finally, the two logical outputs are weighted and summed to obtain the joint logical output.

[0078] Step 2.3.2: The classification block includes a softmax layer;

[0079] l_r k,i The input is fed into the classification block and processed by the softmax layer to obtain the i-th EEG signal sample x of the k-th subject. k,i The probability distribution it belongs to {p i,1 ,p i,2 ,...,p i,m ,...,p i,M}, p i,m Indicates sample x k,i The probability of an EEG sample belonging to the m-th class is given by M, where M represents the number of classes of the EEG sample. In this example, the class corresponding to the highest probability value is selected as the i-th EEG signal sample x. k,i The prediction category.

[0080] Step 3: Model training, including training the source model, teacher model, and student model. The specific training process is as follows: Figure 2 As shown;

[0081] Step 3.1: Training the source model;

[0082] Based on the training sample set D, cross-entropy is used as the loss function for the joint network, and the Adam optimizer is used to train the joint network until the maximum number of iterations is reached or the loss no longer decreases, thus obtaining the trained joint network as the source model. The data selection and process for training the source model are as follows: Figure 1 The left half of the image is shown. In this example, the number of samples in a batch during source model training is set to 64, the initial learning rate of the Adam optimizer is set to 0.0001, and the maximum number of iterations is set to 10.

[0083] Step 3.2, Training of the teacher network, including training of the memory teacher model and the forgotten teacher model;

[0084] Step 3.2.1: Memorize the teacher model;

[0085] The memory teacher model is constructed based on a joint network, and the parameter weights of the source model are used as the parameter weights of the memory teacher model, such as... Figure 1 The intermediate domain adaptation network and the teacher C network are shown.

[0086] Step 3.2.2, Forgotten Teacher Model;

[0087] The forgotten teacher model is constructed based on the aforementioned CNN and Transformer joint network. The forgotten training sample set is the EEG signal sample of a subject selected from the training sample set D as the EEG signal sample of the forgotten subject. The EEG signal samples of the remaining subjects were used as retained subjects' EEG signal samples; among them, This represents the i-th EEG signal sample from the forgotten subject. express Corresponding category labels; based on EEG signal samples D from forgotten subjects f Cross-entropy was used as the loss function for the forgotten teacher model, and the Adam optimizer was used to train the forgotten teacher model until the maximum number of iterations was reached or the loss no longer decreased, thus obtaining a trained forgotten teacher model. The data selection and training process are as follows: Figure 1 The upper right part is shown in the image. In this example, when training the forgotten teacher model, the number of samples in a batch is set to 64, the initial learning rate of the Adam optimizer is 0.00001, and the maximum number of iterations is set to 10.

[0088] Step 3.3, Training the student model, includes: data selection and training the student model;

[0089] Step 3.3.1: Data selection;

[0090] A subset of EEG signal samples was selected from the retained EEG signal samples of the subjects, and samples from the forgotten EEG signal samples D were also selected. f A subset of EEG signal samples were also selected and used to form a joint training sample set. In this example, the proportion of retained subject data was 0.1, with one-tenth of the data from each retained subject being selected as training data. The proportion of forgotten subject data was 1, with all data from forgotten subjects being selected as training data.

[0091] Step 3.3.2: Training the student model;

[0092] Step a: Construct a student model based on the CNN and Transformer joint network, and set the initial weight parameters of the student model to be the weight parameters of the source model;

[0093] Step b: Construct the joint loss function for the student model, including: retention loss and forgetting loss; the retention loss is the KL divergence between the probability distributions of the samples output by the teacher model and the student model, and the forgetting loss is the reciprocal of the KL divergence between the probability distributions of the samples output by the teacher model and the student model. The joint loss function is a weighted sum of the retention loss and the forgetting loss. Figure 1 As shown in the lower right part, in this example, the formula for calculating the retention loss is:

[0094]

[0095] In equation (4), KL represents divergence; T g (·) and S(·) represent the memory teacher model and student model, respectively, and δ(·) represents the softmax function. and s i These represent the memory teacher model and student model respectively for the i-th EEG signal sample x. k ,i The probability distribution to which the output samples belong.

[0096] The formula for calculating forgetting loss is:

[0097]

[0098] In equation (5), T e (·) represents the forgotten teacher model. This represents the forgetting teacher model for the i-th EEG signal sample x. k,i The probability distribution to which the output samples belong.

[0099] The formula for calculating the joint loss function is:

[0100]

[0101] In equation (6), l u Indicates forgetting the label, when l u =0 indicates that the EEG signal sample of the forgotten subject has been forgotten. u =1 indicates that the subject's EEG signal sample is retained; α and β represent the proportion coefficients of memory loss and forgetting loss in the combined loss, respectively.

[0102] Step c: Based on the joint training sample set, train the student model using the Adam optimizer and calculate the joint loss function. The process continues until the maximum number of iterations is reached or the loss no longer decreases, resulting in a student model that removes forgotten subject data while retaining the retained data. This model is used to protect the privacy of EEG data from forgotten subjects. In this example, during student model training, the batch size is set to 64 samples, the initial learning rate of the Adam optimizer is 0.00001, and the maximum number of iterations is set to 1.

[0103] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0104] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0105] Evaluation of Method Effectiveness: Although deep learning networks are widely used in academia and industry, their specific behavioral mechanisms remain largely a black box. Therefore, in this example, it is currently difficult to clearly understand how the model forgets the subject's data. However, statistical indicators, used to evaluate the model's classification results, can reflect whether the model has forgotten data. When the model performs poorly on the data of a specific subject, it can be said that the model has forgotten the data of that specific subject. Tables 1 and 2 show the statistical indicators of the model's classification results on the forgotten and retained subject data after forgetting training, respectively. As shown in Table 1, on the forgotten subject data, all statistical indicators show a significant decrease before and after using the method proposed in this invention. As shown in Table 2, on the retained subject data, after using the method proposed in this invention, all statistical indicators remain almost unchanged before and after forgetting. Experiments show that the joint network based on CNN and Transformer proposed in this invention can effectively extract local inter-channel information and long-range intra-channel information from EEG signals, enabling the model to achieve high classification performance. The proposed knowledge distillation-based deep learning method for protecting the privacy of EEG signals can effectively forget the data of specific subjects while keeping the data of other subjects essentially unchanged.

[0106] Table 1. Classification performance of EEG signals from forgotten subjects on the CHB-MIT database.

[0107]

[0108] Table 2 shows the average performance of EEG signal classification for subjects retained on the CHB-MIT database.

[0109]

[0110] In summary, the joint network based on CNN and Transformer proposed in this invention can effectively utilize the temporal information and spatial correlations in EEG signals to improve the accuracy of EEG signal sample classification. Furthermore, the proposed knowledge distillation-based deep learning method for EEG signal privacy protection can effectively delete EEG data from specific subjects while maintaining the data of other subjects largely unchanged. Compared to retraining methods, this method uses only a small amount of EEG data, significantly reducing time and computational costs.

Claims

1. A knowledge distillation based electroencephalogram signal privacy protection method, characterized in that, is carried out as follows: Step 1, obtain the electroencephalogram data of K subjects with labeled category information, and perform sliding slice processing on the electroencephalogram of each category to obtain N electroencephalogram samples with a time length of T and the corresponding category labels, denoted as a training sample set wherein, represents the electroencephalogram data of the kth subject, x k,i ∈R 1×L×C represents the i th electroencephalogram sample of the k th subject, C represents the number of channels of the electroencephalogram sample, L represents the length of the electroencephalogram sample, y k,i represents x k,i corresponding category label, n represents the number of samples of the electroencephalogram data of the subject; Step 2, a joint network based on CNN and Transformer is established, including: a redundancy removal module, a joint module, and a weighted classification module; Step 2.1, the redundancy removal module sequentially includes: a first convolutional block and a second convolutional block; Each convolutional block is sequentially composed of a convolutional layer, a normalization layer, and a nonlinear activation layer; The i-th electroencephalogram signal sample x k,i is input into the joint network and processed by the de-redundancy module, and an i-th feature map f k,i of the k-th subject is output Step 2.2, the joint module processes the i-th feature map f k,i After processing, the i-th local inter-channel feature f_c k,i and the i-th long-range intra-channel feature f_t k,i of the k-th subject are obtained. Step 2.

3. The weighted classification module processes f_c k,i and f_t k,i to obtain the probability distribution to which the i-th electroencephalogram sample x k,i of the k-th subject belongs. Step 3, model training, including training of a source model, a teacher model, and a student model; Step 3.1, training of the source model; Based on the training sample set D, cross-entropy is used as the loss function of the joint network, and the joint network is trained using the Adam optimizer until the maximum number of iterations is reached or the loss is no longer decreasing, thereby obtaining the trained joint network as the source model; Step 3.2, training of the teacher network, including training of a memory teacher model and a forgetting teacher model; Step 3.2.1, the memory teacher model; The memory teacher model is constructed based on the CNN and Transformer joint network, and the parameter weights of the source model are used as the parameter weights of the memory teacher model; Step 3.2.2, the forgetting teacher model; The forgetting teacher model is constructed based on the CNN and the Transformer joint network, and a forgetting training sample set is obtained by selecting the electroencephalogram sample of a subject from the training sample set D as the electroencephalogram sample of a forgetting subject The electroencephalogram samples of the remaining subjects are taken as the electroencephalogram samples of reserved subjects; wherein, Xi represents the i-th electroencephalogram sample of the forgetting subject, Xi represents The corresponding category label; The EEG signal sample D based on the forgetting subject f Cross-entropy is used as the loss function of the forgetting teacher model, and the forgetting teacher model is trained by using an Adam optimizer until a maximum iteration number is reached or the loss is not decreased, so as to obtain the trained forgetting teacher model. Step 3.3, training of the student model, including data selection and training of the student model; Step 3.3.1, data selection; A part of the electroencephalogram signal samples is selected from the electroencephalogram signal samples of the retention subject, and a part of the electroencephalogram signal samples is also selected from the electroencephalogram signal samples D of the amnesia subject, and the two parts jointly constitute a joint training sample set; f A part of the electroencephalogram signal samples is selected from the electroencephalogram signal samples of the retention subject, and a part of the electroencephalogram signal samples is also selected from the electroencephalogram signal samples D of Step 3.3.2, training of the student model; Step a, constructing the student model based on the CNN and Transformer joint network, and setting the weight parameters of the student model to be the weight parameters of the source model; Step b, constructing the joint loss function of the student model, including a retention loss and a forgetting loss; the retention loss is the KL divergence between the probability distribution of the samples output by the memory teacher model and the student model, and the forgetting loss is the inverse of the KL divergence between the probability distribution of the samples output by the forgetting teacher model and the student model; the joint loss function is the weighted sum of the retention loss and the forgetting loss; Step c, based on the joint training sample set, the student model is trained using the Adam optimizer, and the joint loss function l is calculated, until the maximum number of iterations is reached or the loss is no longer decreasing, thereby obtaining a student model that removes the information of the forgetting subject's data while retaining the information of the retention subject's data, which is used for privacy protection of the EEG data of the forgetting subject.

2. The knowledge distillation-based electroencephalogram privacy protection method according to claim 1, characterized in that, The joint module includes a parallel CNN branch and a Transformer branch; Step 2.2.1, the CNN branch is based on a convolutional neural network and is composed of multiple Conv modules, each of which is composed of six convolutional blocks with similar structures; taking the first Conv module as an example, it includes the following convolutional blocks in sequence: a third convolutional block, a fourth convolutional block, a fifth convolutional block, a sixth convolutional block, a seventh convolutional block, and an eighth convolutional block; Among them, the convolution kernel sizes of the third, fifth, and seventh convolutional blocks are the same, and the convolution kernel sizes of the fourth, sixth, and eighth convolutional blocks are the same; The i-th feature map f k,i is input into the CNN branch and sequentially processed through six convolutional blocks, and the i-th local inter-channel feature f_c k,i of the k-th subject is output Step 2.2.2, the Transformer branch is based on a self-attention network and sequentially includes a mapping module and a self-attention module; The mapping module is sequentially composed of a transposition layer, a merging layer and a fully connected sub-block; The i-th feature map f k,i After being input into the Transformer branch and processed by the mapping module, the i-th input token P of the k-th subject is obtained k,i ; The self-attention module is composed of multiple Trans sub-modules with the same structure; the ith input token P k,i After being input into the self-attention module for processing, the ith long-range intra-channel feature f_t of the kth subject is output k,i .

3. The knowledge distillation-based electroencephalogram privacy protection method according to claim 2, characterized in that, The weighted classification module sequentially includes a feature fusion block and a classification block; Step 2.3.1, the feature fusion module contains parallel CNN logic branches and Transformer logic branches and a weighted fusion layer, wherein the CNN logic branches sequentially contain a convolution layer, a merging layer and two fully connected layers, and the Transformer logic branches sequentially contain a segmentation layer and a fully connected layer; f_c k,i input into the CNN logic branch for processing, and obtain the logic output l_c k,i of the i-th CNN branch of the k-th subject f_t k,i input into the Transformer logic branch for processing and obtain the i-th Transformer branch logic output l_t k,i of the k-th subject Input into the weighted fusion layer for processing, and get the i-th fusion logic output l_r k,i of the k-th subject k,i Input into the weighted fusion layer for processing, and get the i-th fusion logic output l_r k,i of the k-th subject Step 2.3.2, the classification block contains a softmax layer; l_r k,i Input into the classification block and processed through a softmax layer to obtain the i-th electroencephalogram signal sample x k,i The probability distribution belongs to The sample x k,i The probability that the sample belongs to the m-th electroencephalogram sample, and M represents the number of electroencephalogram sample categories.

4. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the electroencephalogram signal privacy protection method in any one of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the electroencephalogram signal privacy protection method in any one of claims 1-3.

Images