Electroencephalogram signal decoding method, device, equipment and medium

Abstract

The invention belongs to the technical field of signal decoding, and relates to an electroencephalogram signal decoding method and device, equipment and a medium. The method comprises the following steps: acquiring a data set of electroencephalogram signals, and obtaining a training set and a test set; constructing a teacher model, and inputting the long-length electroencephalogram signals in the training set into the teacher model for training to obtain a pre-trained teacher model; constructing a student model, respectively inputting the long-length electroencephalogram signal and the short-length electroencephalogram signal in the training set into the pre-training teacher model and the student model, and then calculating the characteristic distillation loss based on frequency mask generation distillation by taking the middle layer characteristics of the pre-training teacher model as transmitted knowledge to obtain the total loss of the student model; performing supervised training on the student model by adopting the pre-trained teacher model to obtain an optimized student model; and decoding the short-length electroencephalogram signals in the test set by adopting an optimized student model. According to the invention, the decoding precision of the short-length electroencephalogram signal can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of signal decoding, and in particular to a method, device, equipment and medium for decoding electroencephalogram (EEG) signals. Background Art

[0002] Brain Computer Interface (BCI) technology can quickly and directly convert brain signals into commands to control external devices without going through peripheral nerves and muscle tissue, providing a channel for human-computer integration and further improving the efficiency of human-computer collaboration.

[0003] Among them, electroencephalogram (EEG) has become the main mode for implementing BCI systems due to its non-invasiveness, high temporal resolution and convenience. The key to designing a high-performance brain-computer interface system based on EEG signals is to develop a fast and accurate EEG signal decoding method to analyze and identify the operator's EEG signals in the shortest possible time and obtain the operator's control intention.

[0004] However, short-length EEG signals face problems such as low signal-to-noise ratio and fewer features, resulting in low decoding accuracy, which limits the practical application of brain-computer interface systems. Therefore, improving the recognition accuracy of short-length EEG signals and thus improving the information transmission rate of the system is still the research focus of BCI systems based on EEG signals. A large number of previous studies have consistently shown that, under the same conditions, longer EEG signals tend to lead to higher decoding accuracy. Therefore, the generative method has become a common method to extend the signal length to improve the decoding performance of short-length EEG signals. However, the generative method is difficult to train, and when the number of categories increases, the generative method will find it difficult to simulate the real data distribution.

[0005] In recent years, knowledge distillation has achieved remarkable results in improving model performance in many fields such as computer vision, natural language processing, and human activity recognition. It can transfer knowledge from a pre-trained teacher model to a lightweight student model, thereby improving the student model performance without increasing the test time cost.

[0006] However, most existing knowledge distillation methods force the student model to completely imitate the intermediate features or output probability distribution of the teacher model, which may have a negative impact on training and reduce the student's generalization ability. In addition, most knowledge distillation methods focus on reducing the computational and storage costs of the model, and no method is used to improve the performance of short-length EEG signals. Summary of the invention

[0007] Based on this, it is necessary to provide a method, device, equipment and medium for decoding electroencephalogram signals in response to the above-mentioned technical problems, which can improve the decoding accuracy of short-length electroencephalogram signals.

[0008] A method for decoding an electroencephalogram signal, comprising: Acquire a data set of electroencephalogram signals, and obtain a training set and a test set, wherein both the training set and the test set contain long-length electroencephalogram signals and short-length electroencephalogram signals; Construct a teacher model, input the long-length EEG signals in the training set into the teacher model for training, and when the teacher model converges, obtain a pre-trained teacher model based on the long-length EEG signals in the test set; Construct a student model, input the long-length EEG signals and short-length EEG signals in the training set into the pre-trained teacher model and the student model respectively, then use the intermediate layer features of the pre-trained teacher model as the transferred knowledge, calculate the feature distillation loss based on frequency mask generation distillation, and obtain the total loss of the student model, so as to supervise the student model with the pre-trained teacher model and obtain the optimized student model; The optimized student model is used to decode the short-length EEG signals in the test set.

[0009] In one embodiment, after respectively inputting the long-length EEG signal and the short-length EEG signal in the training set into the pre-trained teacher model and the student model, the method further includes: The probability distribution of the output of the pre-trained teacher model is used as the transferred knowledge, the output distillation loss is calculated, and the total loss of the student model is obtained based on the feature distillation loss and the output distillation loss.

[0010] In one embodiment, calculating the output distillation loss includes: The non-target category distillation loss is calculated based on the KL divergence, and the inter-class relationship distillation loss is calculated based on the Pearson correlation coefficient; According to the non-target category distillation loss and the inter-class relationship distillation loss, the output distillation loss is obtained.

[0011] In one embodiment, calculating the feature distillation loss for frequency mask-based distillation generation includes: Generate a random mask matrix based on the characteristics of the sample in the pre-trained teacher model and the student model:

[0012] In the formula, For the i The first in the random mask matrix of samples c The value of the channel, For the samples, For the cchannels, indicating the channel position in the feature map and satisfying , is the number of channels of EEG signal samples, is a random number between 0 and 1, is the hyperparameter representing the mask rate; According to the random mask matrix, generate the masked student features:

[0013] In the formula, is the masked student feature, is the characteristic of the sample in the student model, For the i A random mask matrix of samples; The generator reconstructs the complete feature representation of the student model based on the masked student features, so that the reconstructed student model features are close to the teacher model features:

[0014] in,

[0015] In the formula, It is a generator consisting of two layers of one-dimensional convolution with a kernel size of 3 and a layer of RELU activation function. is the feature of the sample in the pre-trained teacher model, and There are two different convolutional layers; According to the random mask matrix and the generation process of the generator, the feature distillation loss based on frequency mask generation distillation is obtained:

[0016] In the formula, is the characteristic distillation loss, is the total number of EEG signal samples, is the number of channels of EEG signal samples, For the i The first sample in the pre-trained teacher model c The characteristics of the channels, For the i The sample is in the student model c The characteristics of a channel.

[0017] In one embodiment, calculating the non-target category distillation loss based on KL divergence includes: The softmax function with a temperature coefficient is used to calculate the probability value of each non-target category of the output predicted by the pre-trained teacher model and the student model:

[0018] In the formula, For the i In the sample j The probability value of each category, When the temperature coefficient is Time to i The sample j Logit-like value The calculated probability value is The output of the last fully connected layer of the model In the sample The logit value of each category, is the temperature coefficient, For different categories, For the i The logit values ​​of different categories in samples; Normalize the probability values ​​for each non-target class:

[0019] In the formula, is the normalized probability value of each non-target category, is the target category; According to the normalized probability, the non-target category distillation loss is calculated based on the KL divergence:

[0020] In the formula, is the total number of EEG signal samples, K is the number of categories of EEG signal samples, is the distillation loss for non-target categories, is the non-target category probability vector output by the pre-trained teacher model, is the non-target category probability vector output by the student model, For the i The KL divergence loss of non-target categories of samples, For the pre-trained teacher model i The non-target category probability vector output by sample prediction, For the student model i The non-target category probability vector output by sample prediction, For the pre-trained teacher model i The prediction output of samples k The probability value of non-target categories, For the student model i The prediction output of samples kThe probability value of non-target categories. In one embodiment, the inter-class relationship distillation loss is calculated based on the Pearson correlation coefficient, including: Based on the pre-trained teacher model and student model, calculate the Pearson correlation coefficient:

[0021] In the formula, is the Pearson correlation coefficient, For the pre-trained teacher model i The probability vector of sample output, For the student model i The probability vector of sample output, is the covariance function, is the standard deviation function, For the pre-trained teacher model i The prediction output of samples j The predicted probability value of each category, For the student model i The prediction output of samples j The predicted probability value of each category, is the mean predicted by the pre-trained teacher model, is the mean predicted by the student model; Based on the Pearson correlation coefficient, calculate the Pearson distance between the pre-trained teacher model and the student model:

[0022] In the formula, is the Pearson distance, For the student model i The probability vector of sample output; According to the Pearson distance, the inter-class relationship distillation loss is calculated:

[0023] In the formula, is the inter-class relationship distillation loss, is the probability vector output by the pre-trained teacher model, is the probability vector output by the student model.

[0024] In one embodiment, according to the feature distillation loss and the output distillation loss, the total loss of the student model is obtained, including:

[0025] in,

[0026] In the formula, is the total loss of the student model, is the cross entropy loss, is the output distillation loss, is the characteristic distillation loss, and For different hyperparameters, is the cross entropy function, For labels, When the temperature coefficient The probability vector output by the student model when is 1, It is the logit vector output by the last fully connected layer of the student model.

[0027] A decoding device for electroencephalogram signals, comprising: An acquisition module is used to acquire a data set of electroencephalogram signals and obtain a training set and a test set, wherein both the training set and the test set contain long-length electroencephalogram signals and short-length electroencephalogram signals; The teacher module is used to construct a teacher model. The long-length EEG signals in the training set are input into the teacher model for training. When the teacher model converges, a pre-trained teacher model is obtained according to the long-length EEG signals in the test set. The student module is used to construct a student model. The long-length EEG signals and short-length EEG signals in the training set are input into the pre-trained teacher model and the student model respectively. Then, the intermediate layer features of the pre-trained teacher model are used as the transferred knowledge, and the feature distillation loss based on frequency mask generation distillation is calculated to obtain the total loss of the student model. The student model is supervised and trained using the pre-trained teacher model to obtain an optimized student model. The decoding module is used to decode the short-length EEG signals in the test set using the optimized student model.

[0028] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0029] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0030] The decoding method, device, equipment and medium of the electroencephalogram signal are as follows: first, in view of the problem of low decoding accuracy of short-length EEG signals, a new framework is introduced to transfer the knowledge of the pre-trained long-length EEG teacher model to the short-length EEG student model in a loose matching manner, thereby improving the decoding performance and decoding accuracy of the student model for short-length EEG signals, and then improving the performance of the BCI system; secondly, the feature distillation loss based on frequency mask generation distillation is calculated, and the feature knowledge is transferred from the teacher to the student in a loose matching manner to improve the classification performance; then, in order to better realize the knowledge transfer, a knowledge distillation method based on loose matching is proposed, that is, the feature knowledge based on mask generation, the non-target class distribution knowledge and the inter-class relationship knowledge are transferred from the teacher to the student in a loose matching manner to further improve the classification performance. The present invention can better utilize the knowledge in the teacher model, so that the student model can improve the accuracy of the system as much as possible under the premise of meeting the system response speed requirements, thereby improving the information transmission rate and practicality of the system, which is of great significance and practical value for improving the information transmission rate of the brain-computer interface system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is an application scenario diagram of a method for decoding an electroencephalogram signal in an embodiment; Figure 2 is a schematic flow chart of a method for decoding an electroencephalogram signal in one embodiment; Figure 3 A schematic diagram of a conventional characteristic distillation method in one embodiment; Figure 4 is a schematic diagram of a feature distillation method based on generative alignment in one embodiment; Figure 5 is a schematic diagram of the architecture of a method for decoding an electroencephalogram signal in one embodiment; Figure 6 The comparison experimental results of the average information transmission rate of the method of the present application and other methods on a data set in one embodiment; Figure 7 The comparison experimental results of the average information transmission rate of the method of the present application and other methods on another data set in one embodiment; Figure 8 is a structural block diagram of a decoding device for electroencephalogram signals in one embodiment; Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0033] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.

[0034] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0035] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0036] The present application provides a method for decoding an electroencephalogram signal, which can be applied to Figure 1 In the application scenario diagram shown, the terminal 102 communicates with the server 104 through the network, and the terminal 102 may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 may be various portal websites, servers corresponding to the backend of the work system, etc.

[0037] The present application provides a method for decoding an electroencephalogram signal, such as Figure 2 The flowchart shown in FIG. 1 shows a method for applying the method to Figure 1 The following is an example of a terminal in the example, including: Step 202, obtaining a data set of EEG signals, and obtaining a training set and a test set, wherein both the training set and the test set contain long-length EEG signals and short-length EEG signals.

[0038] In this step, an example is given below: First, obtain two public EEG datasets containing 35 and 10 subjects respectively (there can also be only one dataset. When there are two datasets, the number of subjects in the two datasets is different).

[0039] Then, the original EEG data was preprocessed by filtering, segmentation and other operations to obtain two groups of long-length EEG signal and short-length EEG signal sample pairs with 40 and 12 class labels respectively. ,in and Respectively i Contains long EEG samples with sampling points and A short EEG sample of sampling points, For the i The labels corresponding to the samples are N is the total number of EEG signal samples. For different groups of data, N The values ​​are different.

[0040] Finally, the leave-one-out cross-validation method is used to divide the sample pairs in each set of public data sets into training sets and test sets.

[0041] It should be noted that long length and short length are relative, not specific.

[0042] Step 204, construct a teacher model, input the long-length EEG signals in the training set into the teacher model for training, until the teacher model converges, and obtain a pre-trained teacher model based on the long-length EEG signals in the test set.

[0043] Specifically: Build the SSVEPformer model based on the Transformer architecture as the teacher model. The model includes a fast Fourier transform module, a channel combination module, a Transformer encoder, and a multi-layer perceptron module. Among them, the fast Fourier transform module can transform EEG time domain signals of different lengths into frequency domain signals of fixed length; the channel combination module calculates the weighted combination of all channels through convolution operations, thereby reducing noise and amplifying features that are critical to EEG signal classification; the Transformer encoder consists of six identical and continuous sub-encoders, which complete the feature encoding and attention learning process through a convolutional neural network module and a multi-layer perceptron module; the final multi-layer perceptron module classifies the extracted features; The long-length EEG signal in the training set is used as the input of the teacher model, and the cross entropy function is used to supervise the teacher model and update the model parameters until the model converges; When the teacher model converges, the teacher model with the best performance on the test set is used as the pre-trained teacher model for subsequent guided student network model training.

[0044] In this step, each module of the teacher model belongs to the existing technology and will not be described in detail here.

[0045] Step 206, construct a student model, input the long-length EEG signal and the short-length EEG signal in the training set into the pre-trained teacher model and the student model respectively, then use the intermediate layer features of the pre-trained teacher model as the transferred knowledge, calculate the feature distillation loss based on the frequency mask generation distillation, and obtain the total loss of the student model, so as to supervise the student model with the pre-trained teacher model to obtain the optimized student model.

[0046] in, Calculate the feature distillation loss based on frequency mask generative distillation, including: Generate a random mask matrix based on the characteristics of the sample in the pre-trained teacher model and the student model:

[0047] In the formula, For the i The first in the random mask matrix of samples c The value of the channel, For the samples, For the c channels, indicating the channel position in the feature map and satisfying , is the number of channels of EEG signal samples, is a random number between 0 and 1, is the hyperparameter representing the mask rate; According to the random mask matrix, generate the masked student features:

[0048] In the formula, is the masked student feature, For the i The characteristics of samples in the student model, , represents the vector dimension, For the i A random mask matrix of samples; The generator reconstructs the complete feature representation of the student model based on the masked student features, so that the reconstructed student model features are close to the teacher model features. The generation process is expressed as:

[0049] in,

[0050] In the formula, It is a generator consisting of two layers of one-dimensional convolution with a kernel size of 3 and a layer of RELU activation function. For the i The features of samples in the pre-trained teacher model, , and There are two different convolutional layers; According to the random mask matrix and the generation process of the generator, the feature distillation loss based on frequency mask generation distillation is obtained:

[0051] In the formula, is the characteristic distillation loss, is the total number of EEG signal samples, is the number of channels of EEG signal samples, For the i The first sample in the pre-trained teacher model c The characteristics of the channels, For the i The sample is in the student model c The characteristics of a channel.

[0052] According to the feature distillation loss, the total loss of the student model is obtained; then the stochastic gradient descent (SGD) optimizer is used to minimize the total loss of the student model, that is, the overall loss function of the student model mentioned above. The optimizer can update and adjust the parameters of the model during the training process to minimize the loss function, thereby optimizing the parameters of the initial model, and then using the pre-trained teacher model to supervise the training of the student model to obtain an optimized student model.

[0053] In this step, a SSVEPformer model based on the Transformer architecture is built as the student model, and the model structure is the same as that of the teacher model.

[0054] The generation process forces the student model to learn how to infer complete feature representations from incomplete information, thereby learning richer feature representations, improving its robustness and generalization ability, and reducing overfitting.

[0055] Step 208, using the optimized student model to decode the short-length EEG signals in the test set.

[0056] In this step, the decoding performance of the student model can be evaluated by calculating the classification accuracy and information transfer rate (ITR).

[0057] In another embodiment, after respectively inputting the long-length EEG signals and the short-length EEG signals in the training set into the pre-trained teacher model and the student model, it also includes: using the probability distribution of the output of the pre-trained teacher model as the transferred knowledge, calculating the output distillation loss, and obtaining the total loss of the student model based on the feature distillation loss and the output distillation loss.

[0058] The calculation of the output distillation loss includes: calculating the non-target category distillation loss based on the KL divergence, and calculating the inter-class relationship distillation loss based on the Pearson correlation coefficient; and obtaining the output distillation loss according to the non-target category distillation loss and the inter-class relationship distillation loss.

[0059] Specifically: The softmax function with a temperature coefficient is used to calculate the probability value of each non-target category of the output predicted by the pre-trained teacher model and the student model:

[0060] In the formula, For the i In the sample j The probability value of each category, When the temperature coefficient is Time to i The sample j The logit value of each category The calculated probability value is The output of the last fully connected layer of the model In the sample The logit value of each category, is the temperature coefficient, for different categories (the sum here is the sum of all categories, j categories), For the i The logit values ​​of different categories in samples; For multi-classification tasks, the probability distribution of non-target classes is a collection of many classes excluding the target class. Since the predicted probability values ​​of the target class by the student model and the teacher model are different, the probability distribution of non-target classes needs to be normalized. The probability value of each non-target class is normalized:

[0061] In the formula, is the normalized probability value of each non-target category, is the target category; According to the normalized probability, the non-target category distillation loss is calculated based on the KL divergence:

[0062] In the formula, is the total number of EEG signal samples, K is the number of categories of EEG signal samples, is the distillation loss for non-target categories, is the non-target category probability vector output by the pre-trained teacher model, is the non-target category probability vector output by the student model, For the i The KL divergence loss of non-target categories of samples, For the pre-trained teacher model i The non-target category probability vector output by sample prediction, For the student model i The non-target category probability vector output by sample prediction, For the pre-trained teacher model i The prediction output of samples k The probability value of non-target categories, For the student model i The prediction output of samples k The probability value of non-target categories.

[0063] The Pearson Correlation Coefficient (PCC) is an indicator used to measure the degree of correlation between two variables. It reflects the linear relationship and the direction of the correlation between the two variables. Therefore, the Pearson correlation coefficient is used to measure the correlation between the probability distribution of the teacher model output and the probability distribution of the student model output. According to the pre-trained teacher model and student model, the Pearson correlation coefficient is calculated:

[0064] In the formula, is the Pearson correlation coefficient, For the pre-trained teacher model i The probability vector of sample output, For the student model i The probability vector of sample output, is the covariance function, is the standard deviation function, For the pre-trained teacher model i The prediction output of samples j The predicted probability value of each category, For the student model i The prediction output of samples j The predicted probability value of each category, is the mean predicted by the pre-trained teacher model, is the mean predicted by the student model; The Pearson correlation coefficient varies in the range of [-1, 1]. When the correlation coefficient value is 0, it means that there is no linear relationship between the two variables. Therefore, the Pearson distance (PD) is used as the correlation loss between the probability distribution of the teacher model output and the probability distribution of the student model output. According to the Pearson correlation coefficient, the Pearson distance between the pre-trained teacher model and the student model is calculated:

[0065] In the formula, is the Pearson distance, For the student model i The probability vector of sample output; According to the Pearson distance, the inter-class relationship distillation loss is calculated:

[0066] In the formula, is the inter-class relationship distillation loss, is the probability vector output by the pre-trained teacher model, is the probability vector output by the student model.

[0067] According to the non-target category distillation loss and the inter-class relationship distillation loss, the output distillation loss (also the total logit loss of the student model) is obtained:

[0068] In the formula, is the output distillation loss.

[0069] According to the feature distillation loss and output distillation loss, the total loss of the student model is obtained, including:

[0070] in,

[0071] In the formula, is the total loss of the student model, is the cross entropy loss, is the output distillation loss, is the characteristic distillation loss, and are different hyperparameters used to balance the effects of cross entropy loss, output distillation loss, and feature distillation loss. is the cross entropy function, For labels, When the temperature coefficient The probability vector output by the student model when is 1, It is the logit vector output by the last fully connected layer of the student model.

[0072] In this embodiment, if Figure 3 and Figure 4 As shown, based on the logit loss, the relationship between the teacher network and the student network in the probability distribution of each instance can be retained through the maximum linear correlation on the basis of the exact matching of the prediction probability of the non-target category, so it can be better generalized to the entire distribution of the output. Compared with the traditional feature distillation method, this application is a more robust and effective method.

[0073] like Figure 5 The architecture diagram shown in FIG. Indicates that the model parameters are fixed. In this architecture, the feature knowledge generated based on the mask belongs to the intermediate feature level, and the non-target class distribution knowledge and inter-class relationship knowledge belong to the probability level. This application performs knowledge distillation from different levels, which can further improve the classification and decoding performance.

[0074] The above-mentioned electroencephalogram signal decoding method, firstly, for the problem of low decoding accuracy of short-length EEG signals, introduces a new framework, transfers the knowledge of the pre-trained long-length EEG teacher model to the short-length EEG student model in a loose matching manner, thereby improving the decoding performance and decoding accuracy of the student model for short-length EEG signals, and then improving the performance of the BCI system; secondly, calculates the feature distillation loss based on frequency mask generation distillation, and transfers the feature knowledge from the teacher to the student in a loose matching manner to improve the classification performance; then, in order to better realize knowledge transfer, a knowledge distillation method based on loose matching is proposed, that is, the feature knowledge based on mask generation, the non-target class distribution knowledge and the inter-class relationship knowledge are transferred from the teacher to the student in a loose matching manner to further improve the classification performance. The present invention can better utilize the knowledge in the teacher model, so that the student model can improve the accuracy of the system as much as possible under the premise of meeting the system response speed requirements, thereby improving the information transmission rate and practicality of the system, which is of great significance and practical value for improving the information transmission rate of the brain-computer interface system.

[0075] In a specific embodiment, in order to verify the performance improvement effect of the knowledge distillation method based on relaxation matching of an embodiment of the present invention on EEG signals of different lengths, data of 2.0s length is used as long-length EEG data, and data of 0.5s, 0.6s, 0.7s, 0.8s, 0.9s, and 1.0s are used as short-length EEG data respectively.

[0076] The batch size during training is set to 64, the epoch is 100, and the learning rate is 0.001. Set to 0.5, the temperature coefficient Set to 4. In order to balance the impact of each loss on the student model training, α , β and are set to 0.8 and .

[0077] In this embodiment, a leave-one-out cross-validation method is used to conduct comparative experiments and ablation experiments to verify the practicality of the method based on relaxed matching knowledge distillation (RMKD) in the embodiment of the present invention in decoding short-length EEG signals.

[0078] Specifically, the data of one subject in the data set is selected as the test set, and the data of the remaining subjects are selected as the training set. During the training process, all training data will be shuffled to avoid the training parameters randomly generated by the model affecting the experimental results.

[0079] The short-length EEG signals to be classified in the test set are input into the student network, and the average classification accuracy (Accuracy) and information transfer rate (ITR) of all subjects are calculated as the performance evaluation indicators of the EEG signal recognition method. ITR refers to the amount of information transmitted by a single target selection. It is not only related to the classification accuracy, but also takes into account the number of categories and the speed of single target recognition. It is a comprehensive performance indicator for measuring the performance of the brain-computer interface system. The calculation formula of ITR is as follows:

[0080] in, C is the number of classification targets, P is the classification accuracy, t Select a time for a single target.

[0081] In order to evaluate the performance of the method described in this embodiment, the classification results of the described method and deep learning methods EEGNet, SSVEPNet, CCNN and other knowledge distillation algorithms were statistically analyzed when the EEG signal lengths were 0.5s, 0.6s, 0.7s, 0.8s, 0.9s, and 1.0s, respectively. Figure 6 The experimental results of the comparison of the average information transmission rate of the method of the present application and other methods on a data set are shown in FIG. Figure 7 The experimental results of the comparison of the average information transmission rate of the method of the present application and other methods on another data set are shown in Table 1, the average classification accuracy results of different methods in the comparison experiment are shown in Table 1, and the average classification accuracy results of different methods in the ablation experiment are shown in Table 2.

[0082] Table 1 - Average classification accuracy of different methods in comparative experiments

[0083] From the comparative experimental results, it can be seen that different knowledge distillation algorithms all perform well, the recognition accuracy is improved to a certain extent, and the average classification accuracy and information transmission rate of the method proposed in this embodiment are better than those of other comparative methods. Therefore, the method based on relaxed matching knowledge distillation proposed in this embodiment can better transfer the knowledge in the teacher model to the student model, thereby improving the decoding performance of the student model for short-length EEG signals.

[0084] Table 2 - Average classification accuracy of different methods in ablation experiments

[0085] From the ablation experiment results, it can be seen that the method based on relaxed matching knowledge distillation proposed in this embodiment is superior to all the comparison methods. This result proves the advantage of the method described in this embodiment in distilling useful knowledge from features and outputs, further improving the decoding accuracy of short-length EEG signals, thereby improving the information transmission rate of the system, which is of great significance and practical value for promoting the practical application of brain-computer interfaces.

[0086] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0087] The present application also provides a decoding device for electroencephalogram signals, such as Figure 8 As shown, in one embodiment, it includes: an acquisition module 802, a teacher module 804, a student module 806 and a decoding module 808, wherein: An acquisition module 802 is used to acquire a data set of electroencephalogram signals and obtain a training set and a test set, wherein both the training set and the test set contain long-length electroencephalogram signals and short-length electroencephalogram signals; The teacher module 804 is used to construct a teacher model, input the long-length EEG signals in the training set into the teacher model for training, and obtain a pre-trained teacher model based on the long-length EEG signals in the test set when the teacher model converges; The student module 806 is used to construct a student model, input the long-length EEG signal and the short-length EEG signal in the training set into the pre-trained teacher model and the student model respectively, and then use the intermediate layer features of the pre-trained teacher model as the transferred knowledge, calculate the feature distillation loss based on the frequency mask generation distillation, and obtain the total loss of the student model, so as to supervise the student model with the pre-trained teacher model to obtain the optimized student model; The decoding module 808 is used to decode the short-length EEG signals in the test set using the optimized student model.

[0088] For the specific definition of a decoding device for an electroencephalogram signal, please refer to the definition of a decoding method for an electroencephalogram signal in the above text, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0089] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for decoding an electroencephalogram signal is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0090] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0092] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0093] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0094] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

[0095] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for decoding an electroencephalogram signal, characterized in that: include: Acquire a data set of electroencephalogram signals, and obtain a training set and a test set, wherein both the training set and the test set contain long-length electroencephalogram signals and short-length electroencephalogram signals; Construct a teacher model, input the long-length EEG signals in the training set into the teacher model for training, and when the teacher model converges, obtain a pre-trained teacher model based on the long-length EEG signals in the test set; Construct a student model, input the long-length EEG signals and short-length EEG signals in the training set into the pre-trained teacher model and the student model respectively, then use the intermediate layer features of the pre-trained teacher model as the transferred knowledge, calculate the feature distillation loss based on frequency mask generation distillation, and obtain the total loss of the student model, so as to supervise the student model with the pre-trained teacher model and obtain the optimized student model; The optimized student model is used to decode the short-length EEG signals in the test set.

2. The method for decoding an electroencephalogram signal according to claim 1, characterized in that: After inputting the long-length EEG signals and short-length EEG signals in the training set into the pre-trained teacher model and the student model respectively, it also includes: The probability distribution of the output of the pre-trained teacher model is used as the transferred knowledge, the output distillation loss is calculated, and the total loss of the student model is obtained based on the feature distillation loss and the output distillation loss.

3. The method for decoding an electroencephalogram signal according to claim 2, characterized in that: Calculate the output distillation loss, including: The non-target category distillation loss is calculated based on the KL divergence, and the inter-class relationship distillation loss is calculated based on the Pearson correlation coefficient; According to the non-target category distillation loss and the inter-class relationship distillation loss, the output distillation loss is obtained.

4. A method for decoding an electroencephalogram signal according to any one of claims 1 to 3, characterized in that: Calculate the feature distillation loss based on frequency mask generative distillation, including: Generate a random mask matrix based on the characteristics of the sample in the pre-trained teacher model and the student model: In the formula, For the i The first in the random mask matrix of samples c The value of the channel, For the samples, For the c channels, indicating the channel position in the feature map and satisfying , is the number of channels of EEG signal samples, is a random number between 0 and 1, is the hyperparameter representing the mask rate; According to the random mask matrix, generate the masked student features: In the formula, is the masked student feature, is the characteristic of the sample in the student model, For the i A random mask matrix of samples; The generator reconstructs the complete feature representation of the student model based on the masked student features, so that the reconstructed student model features are close to the teacher model features: in, In the formula, It is a generator consisting of two layers of one-dimensional convolution with a kernel size of 3 and a layer of RELU activation function. is the feature of the sample in the pre-trained teacher model, and There are two different convolutional layers; According to the random mask matrix and the generation process of the generator, the feature distillation loss based on frequency mask generation distillation is obtained: In the formula, is the characteristic distillation loss, is the total number of EEG signal samples, is the number of channels of EEG signal samples, For the i The first sample in the pre-trained teacher model c The characteristics of the channels, For the i The sample is in the student model c The characteristics of a channel.

5. The method for decoding an electroencephalogram signal according to claim 3, characterized in that: The non-target category distillation loss is calculated based on KL divergence, including: The softmax function with a temperature coefficient is used to calculate the probability value of each non-target category of the output predicted by the pre-trained teacher model and the student model: In the formula, For the i In the sample j The probability value of each category, When the temperature coefficient is Time to i The sample j The logit value of each category The calculated probability value is The output of the last fully connected layer of the model In the sample The logit value of each category, is the temperature coefficient, For different categories, For the i The logit values ​​of different categories in samples; Normalize the probability values ​​for each non-target class: In the formula, is the normalized probability value of each non-target category, is the target category; According to the normalized probability, the non-target category distillation loss is calculated based on the KL divergence: In the formula, is the total number of EEG signal samples, K is the number of categories of EEG signal samples, is the distillation loss for non-target categories, is the non-target category probability vector output by the pre-trained teacher model, is the non-target category probability vector output by the student model, For the i The KL divergence loss of the non-target category of samples, For the pre-trained teacher model i The non-target category probability vector output by sample prediction, For students, the model i The non-target category probability vector output by sample prediction, For the pre-trained teacher model i The prediction output of samples k The probability value of non-target categories, For students, the model i The prediction output of samples k The probability value of non-target categories.

6. The method for decoding an electroencephalogram signal according to claim 3, characterized in that: The inter-class relationship distillation loss is calculated based on the Pearson correlation coefficient, including: Based on the pre-trained teacher model and student model, calculate the Pearson correlation coefficient: In the formula, is the Pearson correlation coefficient, For the pre-trained teacher model i The probability vector of sample output, For students, the model i The probability vector of sample output, is the covariance function, is the standard deviation function, For the pre-trained teacher model i The prediction output of samples j The predicted probability value of each category, For students, the model i The prediction output of samples j The predicted probability value of each category, is the mean predicted by the pre-trained teacher model, is the mean predicted by the student model; Based on the Pearson correlation coefficient, calculate the Pearson distance between the pre-trained teacher model and the student model: In the formula, is the Pearson distance, For students, the model i The probability vector of sample output; According to the Pearson distance, the inter-class relationship distillation loss is calculated: In the formula, is the inter-class relationship distillation loss, is the probability vector output by the pre-trained teacher model, is the probability vector output by the student model.

7. A method for decoding an electroencephalogram signal according to claim 2 or 3, characterized in that: According to the feature distillation loss and output distillation loss, the total loss of the student model is obtained, including: in, In the formula, is the total loss of the student model, is the cross entropy loss, is the output distillation loss, is the characteristic distillation loss, and For different hyperparameters, is the cross entropy function, For labels, When the temperature coefficient The probability vector output by the student model when is 1, It is the logit vector output by the last fully connected layer of the student model.

8. A decoding device for electroencephalogram signals, characterized in that: include: An acquisition module is used to acquire a data set of electroencephalogram signals and obtain a training set and a test set, wherein both the training set and the test set contain long-length electroencephalogram signals and short-length electroencephalogram signals; The teacher module is used to construct a teacher model. The long-length EEG signals in the training set are input into the teacher model for training. When the teacher model converges, a pre-trained teacher model is obtained according to the long-length EEG signals in the test set. The student module is used to construct a student model. The long-length EEG signals and short-length EEG signals in the training set are input into the pre-trained teacher model and the student model respectively. Then, the intermediate layer features of the pre-trained teacher model are used as the transferred knowledge, and the feature distillation loss based on frequency mask generation distillation is calculated to obtain the total loss of the student model. The student model is supervised and trained using the pre-trained teacher model to obtain an optimized student model. The decoding module is used to decode the short-length EEG signals in the test set using the optimized student model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images