Cross-subject and cross-scene transfer learning method based on spatial-temporal characteristics of electroencephalogram signals

Abstract

The invention provides a cross-subject and cross-scene transfer learning method and device based on spatial-temporal characteristics of electroencephalogram signals, and relates to the technical field of brain-computer interfaces. The method comprises the following steps: preprocessing an original electroencephalogram signal to obtain a spatial-temporal characteristic signal; performing convolution processing through a time-space feature convolution network according to the spatial-temporal feature signal to obtain migration feature data in a training stage; training a Transform model by using the migration feature data in the training stage to obtain an electroencephalogram migration learning model; acquiring a learning stage electroencephalogram signal; performing preprocessing and convolution processing on the learning-stage electroencephalogram signals to obtain learning-stage migration feature data; according to the training stage migration feature data and the learning stage migration feature data, updating a time-space feature convolutional network and an electroencephalogram migration learning model; and carrying out transfer learning based on the updated convolutional network and the updated transfer learning model according to the electroencephalogram signal in the learning stage. The invention provides an efficient and stable cross-subject and cross-scene transfer learning method based on spatial-temporal characteristics of electroencephalogram signals.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a cross-subject and cross-scenario transfer learning method and device based on the spatiotemporal characteristics of electroencephalogram signals. Background Art

[0002] The brain-computer interface (BCI) is a communication pathway between the brain and the external environment that does not rely on peripheral nerves and muscle tissue. BCI identifies the subject's intention by decoding electroencephalography (EEG). BCI can convert neural activity into a series of output commands to control computers, robots, etc. BCI can perform cognitive assessments by detecting the subject's EEG. In different users and diverse scenarios, it is necessary to improve the adaptability and robustness of the BCI system. Transfer learning is a key technology for the widespread application of BCI.

[0003] Transfer learning is a machine learning method that uses knowledge learned in one field to improve learning in another related field. Transfer learning can improve the generalization ability of the model when data is limited or the cost of marking is high. In BCI research, due to differences in EEG between different subjects, the EEG of the same subject in different scenarios is also different, and the data collection of each subject is difficult and the scenarios are complex, which limits the promotion and application of BCI. Therefore, how to transfer the model to other subjects and scenarios through a small amount of subject data to achieve cross-subject and cross-scenario model application will significantly reduce the demand for model training and achieve more efficient EEG decoding control, which is of great significance for the promotion and application of BCI.

[0004] In existing BCI research, there are accurate decoding algorithm models for EEG processing analysis of single subjects and scenes to efficiently identify and control user intentions and cognitive assessments, such as EEG Convolutional Neural Network (EEGNet), Deep Neural Network (DeepNet), Parameter Attribute (ParaAtt) and other algorithm models. However, these methods usually rely on a large amount of individual training data, and each subject needs to train the model separately, which has great limitations in practical applications.

[0005] The application of transfer learning in BCI has also made some progress. By migrating the training model of one subject to other subjects or different scenarios, transfer learning can reduce the demand for new data and improve the adaptability and generalization ability of the model. For example, the use of EEG differential entropy features, source analysis, conditional framework and other technologies can achieve cross-subject and cross-scenario migration from the level of EEG data features and network architecture. Due to the large EEG gap between different subjects, the signal characteristics of some subjects are difficult to transfer, resulting in unstable model performance and difficulty in meeting the requirements of practical applications.

[0006] As a deep learning model based on the self-attention mechanism, the Transformer model has also begun to show great potential in the field of BCI in recent years. Its powerful sequence modeling ability enables it to effectively capture the spatiotemporal features in EEG and improve decoding accuracy. Some studies have tried to use Transformer to decode EEG, and achieved better results than traditional methods. However, there are still some challenges in the application of Transformer in transfer learning of BCI. Transformer models usually contain a large number of parameters, the training process is complex and the computational cost is high, and a small amount of migration data is difficult to have an impact on large parameter models. Therefore, how to make full use of the self-attention mechanism to better capture and utilize the characteristics of EEG requires further research and optimization.

[0007] In the existing technology, there is a lack of an efficient and stable cross-subject and cross-scenario transfer learning method based on the spatiotemporal characteristics of EEG signals. Summary of the invention

[0008] In order to solve the technical problems that the existing EEG decoding algorithm in the prior art performs well in decoding the EEG signal of a single subject, but relies on a large amount of individual training data; the signal characteristics of some subjects are difficult to transfer due to the large differences between different subjects, resulting in instability of the current transfer learning method; the Transformer model is difficult to fully utilize the self-attention mechanism to capture the spatiotemporal characteristics of EEG, the embodiment of the present invention provides a cross-subject and cross-scenario transfer learning method and device based on the spatiotemporal characteristics of EEG signals. The technical solution is as follows:

[0009] On the one hand, a cross-subject and cross-scenario transfer learning method based on the spatiotemporal characteristics of EEG signals is provided, the method is implemented by a transfer learning device, and the method includes:

[0010] Acquire original EEG signals; pre-process the original EEG signals to obtain spatiotemporal characteristic signals;

[0011] According to the spatiotemporal feature signal, convolution processing is performed through a time-space feature convolution network to obtain migration feature data in the training phase;

[0012] The Transformer model is trained using the migration feature data in the training phase to obtain an EEG migration learning model;

[0013] Acquire an EEG signal in the learning phase; preprocess the EEG signal in the learning phase to obtain a preprocessed signal in the learning phase; perform convolution processing on the preprocessed signal in the learning phase through the time-space feature convolution network to obtain migration feature data in the learning phase;

[0014] According to the feature data migrated in the training phase and the feature data migrated in the learning phase, the time-space feature convolution network is updated to obtain an updated feature convolution network;

[0015] According to the migration feature data of the learning stage, the EEG migration learning model is updated to obtain an updated migration learning model;

[0016] According to the EEG signals in the learning phase, transfer learning is performed based on the updated convolutional network and the updated transfer learning model.

[0017] On the other hand, a cross-subject and cross-scenario transfer learning device based on the spatiotemporal features of EEG signals is provided, and the device is applied to a cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals, and the device comprises:

[0018] A data preprocessing module is used to obtain raw EEG signals; preprocess the raw EEG signals to obtain spatiotemporal characteristic signals;

[0019] A feature convolution module, used to perform convolution processing through a time-space feature convolution network according to the spatiotemporal feature signal to obtain migration feature data in the training phase;

[0020] A model training module, used to train the Transformer model using the migration feature data in the training phase to obtain an EEG migration learning model;

[0021] A migration data acquisition module is used to acquire EEG signals in the learning phase; preprocess the EEG signals in the learning phase to obtain preprocessed signals in the learning phase; and convolute the preprocessed signals in the learning phase through the time-space feature convolution network to obtain migration feature data in the learning phase.

[0022] A convolutional model updating module, used to update the time-space feature convolutional network according to the feature data migrated in the training phase and the feature data migrated in the learning phase, to obtain an updated feature convolutional network;

[0023] A transfer learning model updating module, used to update the EEG transfer learning model according to the migration feature data of the learning stage to obtain an updated transfer learning model;

[0024] A transfer learning module is used to perform transfer learning based on the EEG signals in the learning stage, the updated convolutional network and the updated transfer learning model.

[0025] On the other hand, a transfer learning device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned cross-subject and cross-scenario transfer learning methods based on the spatiotemporal characteristics of EEG signals is implemented.

[0026] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned cross-subject and cross-scenario transfer learning methods based on the spatiotemporal characteristics of EEG signals.

[0027] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0028] The present invention proposes a cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals. By standardizing and extracting multi-dimensional features of EEG, the model can still obtain good performance with less training data, reducing the cost and time of data collection and model training. By optimizing the transfer learning algorithm, the data loss of the transfer data and the training data is calculated during the transfer process, and the feature convolutional network is adjusted through back propagation, so that the temporal and spatial feature convolutional network can extract transfer data close to the characteristics of the training data, effectively enhancing the adaptability and generalization ability of the model in different subjects and different scenarios, and significantly improving the effect and stability of transfer learning. ; By optimizing the structure of the Transformer model, introducing the multi-head attention mechanism and position embedding matrix, the self-attention mechanism is fully utilized, the Transformer model's ability to capture and utilize EEG spatiotemporal features is improved, the model structure is simplified, and the training complexity and computational cost are reduced; in the process of transfer learning, by freezing two encoders (axe_x and axe_y), the stability of feature extraction is ensured, so that after the migration data features are added, the model can more effectively focus on updating the encoder of the ave_z dimension, which not only reduces unnecessary computational burden, but also further improves the efficiency and effect of transfer learning. The present invention is an efficient and stable cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 It is a flow chart of a cross-subject and cross-scenario transfer learning method based on the spatiotemporal characteristics of EEG signals provided by an embodiment of the present invention;

[0031] Figure 2 It is a block diagram of a cross-subject and cross-scenario transfer learning device based on spatiotemporal features of EEG signals provided by an embodiment of the present invention;

[0032] Figure 3 It is a structural schematic diagram of a transfer learning device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0034] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0035] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0036] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0037] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0038] The embodiment of the present invention provides a cross-subject and cross-scenario transfer learning method based on the spatiotemporal characteristics of EEG signals. The method can be implemented by a transfer learning device, which can be a terminal or a server. Figure 1 The flowchart of the cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals is shown. The processing flow of the method may include the following steps:

[0039] S1. Obtaining original EEG signals; preprocessing the original EEG signals to obtain spatiotemporal characteristic signals.

[0040] Optionally, the original EEG signal is preprocessed to obtain a spatiotemporal characteristic signal, including:

[0041] Performing waveform processing on the original EEG signal to obtain a preprocessed EEG signal; the waveform processing includes filtering, baseline correction and artifact removal;

[0042] Performing spatiotemporal redistribution on the preprocessed EEG signal to obtain a spatiotemporal distributed EEG signal;

[0043] Data cleaning is performed on spatiotemporally distributed EEG signals to obtain spatiotemporally characteristic signals; data cleaning includes standardization and null value filling.

[0044] In a feasible implementation, the training data EEG signal is filtered, baseline corrected, and artifacts removed in the preprocessing part; and the time and space are redistributed to convert the two-dimensional EEG signal into three-dimensional data containing electrode distribution information and time information. The dimensions of the three-dimensional data are represented as a x 、a y ,t d ; Then the data is standardized and filled with empty values ​​to maintain the integrity and rationality of the data. The preprocessing part converts the EEG into a dimension size of (a x , a y , t d ) input.

[0045] S2. According to the spatiotemporal feature signal, convolution processing is performed through the time-space feature convolution network to obtain the migration feature data in the training phase.

[0046] Optionally, according to the spatiotemporal feature signal, convolution processing is performed through a time-space feature convolution network to obtain migration feature data in the training phase, including:

[0047] According to the spatiotemporal characteristic signals, the time channel convolution is performed using the time convolution kernel to obtain the EEG time characteristics;

[0048] According to the spatiotemporal characteristic signals, spatial channel convolution is performed using the spatial convolution kernel to obtain EEG spatial features;

[0049] Batch normalization is performed on the EEG temporal features and EEG spatial features to obtain the migration feature data in the training phase.

[0050] In a feasible implementation manner, the present invention uses n size (a x , a y , 3) The temporal convolution kernel convolves the temporal channel to extract the temporal features of EEG.

[0051] Where n is the number of convolution kernels. In a convolution layer, each convolution kernel generates a feature map, and n corresponds to the number of output feature maps or the depth of the layer; x , a y , the two values ​​represent the size of the convolution kernel in the spatial dimension, that is, the size of the convolution kernel in height and width. 3 represents the size of the convolution kernel in the channel dimension of the input data.

[0052] Using a size of (3, 3, t d-2 )’s spatial convolution kernel is used to convolve the spatial channels and extract the spatial features under the electrode distribution information.

[0053] The shape of the input data is (H, W, t d ), H and W represent the spatial dimensions (height and width) of the data, t d Represents the channel dimension (may correspond to different time points or different features). The size is (3, 3, t d-2 ) will slide with a 3x3 window in the spatial dimension and cover all the data from the second channel to the last channel in the channel dimension (i.e., t d-2 channels).

[0054] The purpose of this convolution operation is to extract features in the spatial dimension while maintaining information in the temporal dimension. For example, in video processing, motion features in a series of image frames can be extracted; in multi-channel biological signal processing, the spatiotemporal features of the interaction between different electrodes can be extracted.

[0055] Perform batch normalization and pooling on the data to unify the dimensions of the three-dimensional EEG signal to ave. Where ave_x=ave_y=ave_z=ave, x, y, z represent three dimensions. The sample size of a batch of data is batch_size. After preprocessing and spatiotemporal feature extraction, the data dimension of a batch is (batch_size, n, ave_x, ave_y, ave_z).

[0056] S3. Use the feature data transferred in the training phase to train the Transformer model to obtain the EEG transfer learning model.

[0057] Optionally, the Transformer model is trained using the feature data migrated during the training phase to obtain an EEG transfer learning model, including:

[0058] Embedding space mapping is performed on the migrated feature data in the training phase to obtain pre-processed training data;

[0059] Constructing a position embedding matrix according to the preprocessed training data; adding the position embedding matrix to the preprocessed training data to obtain the position embedding training data;

[0060] According to the position embedding training data, the encoder to be trained is trained to obtain an encoder;

[0061] Based on the multi-head attention mechanism, the training data is embedded according to the position, and the output is obtained by encoding.

[0062] Construct a loss function based on the position embedding training data and output data;

[0063] According to the loss function, the Transformer model is optimized to obtain the EEG transfer learning model.

[0064] In a feasible implementation, a projection layer is used to map the batch training data, mapping the data of dimension (batch_size, n, ave_x, ave_y, ave_z) to the embedding space emb_size, that is, the batch data dimension is (batch_size, emb_size, ave_x, ave_y, ave_z).

[0065] Perform position embedding on batch data. Create a position embedding matrix with a shape of (1, emb_size, ave_x, ave_y, ave_z) and add the position embedding matrix to the data, while the dimensional information of the data remains unchanged. Train an encoder for decoding for each of the three dimensions of the EEG. Use the attention mechanism to calculate Q (query matrix), K (key matrix), and V (value matrix) from the input data. Among them, Q=XWQ, K=XWK, and V=XWV. The dimensions of WQ, WK, and WV are all (emb_size, emb_size).

[0066] The present invention introduces a multi-head attention mechanism, where the number of heads is head_num, the dimension of each head is head_dim, and emb_size=head_num×head_dim. For the ave_x dimension, the Q, K, and V matrices are projected and rearranged to the (batch_size, head_num, ave_y×ave_z, head_dim) dimension, the attention weight is calculated by the dot product operation of Q and K and divided by the scaling factor, the softmax function is applied to the data to obtain an attention matrix with a shape size of (batch_size, head_num, ave2, ave2), the attention matrix is ​​multiplied by V and the output is rearranged to the original shape, that is, the encoder output of the ave_x dimension is (batch_size, emb_size, ave2). Similarly, the encoder outputs of ave_y and ave_z dimensions are both (batch_size, emb_size, ave2). The three encoder outputs are merged into (batch_size, emb_size, 3ave2) and mapped to the required category num_class through the Reduce layer and the fully connected layer.

[0067] Wherein, the encoder includes an axe_x encoder, an axe_y encoder and an ave_z encoder;

[0068] The encoder corresponds to the three dimensions of the input data of the Transformer model.

[0069] In a feasible implementation manner, in the present invention, the dimension of each EEG signal sample is (ave_x, ave_y, ave_z).

[0070] S4. Obtaining EEG signals in the learning phase; preprocessing the EEG signals in the learning phase to obtain preprocessed signals in the learning phase; performing convolution processing on the preprocessed signals in the learning phase through a time-space feature convolution network to obtain migration feature data in the learning phase.

[0071] In a feasible implementation, the migration data is preprocessed and the spatiotemporal features are extracted, and each migration data is converted into the form of (ave_x, ave_y, ave_z).

[0072] S5. Update the time-space feature convolution network according to the feature data migrated in the training phase and the feature data migrated in the learning phase to obtain an updated feature convolution network.

[0073] Optionally, updating the time-space feature convolution network according to the feature data migrated in the training phase and the feature data migrated in the learning phase to obtain an updated feature convolution network includes:

[0074] The divergence loss is calculated based on the feature data migrated in the training phase and the feature data migrated in the learning phase to obtain the KL divergence loss;

[0075] According to the KL divergence loss, the time-space feature convolutional network is back-propagated to obtain an updated feature convolutional network.

[0076] In a feasible implementation, the KL divergence (KL) loss of the same category migration data and training data is calculated and back propagation is used to adjust the temporal feature convolution network and the spatial feature convolution network, encouraging the temporal feature convolution network and the spatial feature convolution network to extract migration data features close to training data features.

[0077] S6. Update the EEG transfer learning model according to the migration feature data in the learning stage to obtain an updated transfer learning model.

[0078] Optionally, updating the EEG transfer learning model according to the migration feature data in the learning phase to obtain an updated transfer learning model includes:

[0079] Construct a migration position embedding matrix based on the migration feature data of the learning phase; add the migration position embedding matrix to the migration feature data of the learning phase to obtain the embedded migration feature data of the learning phase;

[0080] Freeze the axe_x encoder and the axe_y encoder, train the ave_z encoder based on the feature data migrated in the post-embedding learning phase, and obtain the updated ave_z encoder;

[0081] Based on the multi-head attention mechanism, according to the feature data of the learning stage migration, the output is encoded through the axe_x encoder, axe_y encoder and the updated ave_z encoder to obtain the transfer learning output data;

[0082] Construct a cross entropy loss function based on the feature data transferred during the learning phase and the output data of transfer learning;

[0083] According to the cross entropy loss function, the EEG transfer learning model is optimized and trained to obtain an updated transfer learning model.

[0084] In a feasible implementation, the two encoders axe_x and axe_y are frozen, the migration data features are added to the position embedding, enter the three encoders, the Reduce layer and the fully connected layer, use the cross entropy loss function for back propagation, and update the ave_z encoder to realize the update of the EEG transfer learning model.

[0085] S7. Perform transfer learning based on updating the convolutional network and the transfer learning model according to the EEG signals in the learning stage.

[0086] In a feasible implementation, the present invention proposes a method for converting two-dimensional EEG signals into three-dimensional data containing electrode distribution information and time information to ensure the integrity and rationality of the data. The temporal and spatial features in the EEG signal are extracted through multi-layer convolution kernels, and the feature fusion is performed in combination with the Transformer model. This method can not only effectively extract the complex temporal and spatial features in the EEG signal, but also improve the efficiency and accuracy of feature extraction.

[0087] By calculating the KL divergence loss of the migration data and the training data, and using back propagation to adjust the temporal feature convolutional network and the spatial feature convolutional network. In the process of cross-subject and cross-scenario migration, the present invention proposes a strategy of freezing the two encoders axe_x and axe_y and focusing on updating the ave_z encoder.

[0088] The present invention proposes a cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals. By standardizing and extracting multi-dimensional features of EEG, the model can still obtain good performance with less training data, reducing the cost and time of data collection and model training. By optimizing the transfer learning algorithm, the data loss of the transfer data and the training data is calculated during the transfer process, and the feature convolutional network is adjusted through back propagation, so that the temporal and spatial feature convolutional network can extract transfer data close to the characteristics of the training data, effectively enhancing the adaptability and generalization ability of the model in different subjects and different scenarios, and significantly improving the effect and stability of transfer learning. ; By optimizing the structure of the Transformer model, introducing the multi-head attention mechanism and position embedding matrix, the self-attention mechanism is fully utilized, the Transformer model's ability to capture and utilize EEG spatiotemporal features is improved, the model structure is simplified, and the training complexity and computational cost are reduced; in the process of transfer learning, by freezing two encoders (axe_x and axe_y), the stability of feature extraction is ensured, so that after the migration data features are added, the model can more effectively focus on updating the encoder of the ave_z dimension, which not only reduces unnecessary computational burden, but also further improves the efficiency and effect of transfer learning. The present invention is an efficient and stable cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals.

[0089] Figure 2 This is a block diagram of a cross-subject and cross-scenario transfer learning device based on the spatiotemporal features of EEG signals according to an exemplary embodiment. The device is used for a cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals. Figure 2The device includes a data preprocessing module 210, a feature convolution module 220, a model training module 230, a migration data acquisition module 240, a convolution model update module 250, a migration learning model update module 260 and a migration learning module 270. Among them:

[0090] The data preprocessing module 210 is used to obtain the original EEG signal; preprocess the original EEG signal to obtain the spatiotemporal characteristic signal;

[0091] The feature convolution module 220 is used to perform convolution processing through a time-space feature convolution network according to the spatiotemporal feature signal to obtain migration feature data in the training phase;

[0092] A model training module 230 is used to train the Transformer model using the feature data transferred during the training phase to obtain an EEG transfer learning model;

[0093] The migration data acquisition module 240 is used to obtain the EEG signal in the learning stage; pre-process the EEG signal in the learning stage to obtain the pre-processed signal in the learning stage; and convolute the pre-processed signal in the learning stage through a time-space feature convolution network to obtain the migration feature data in the learning stage.

[0094] A convolutional model updating module 250, used to update the time-space feature convolutional network according to the feature data migrated in the training phase and the feature data migrated in the learning phase, to obtain an updated feature convolutional network;

[0095] The transfer learning model updating module 260 is used to update the EEG transfer learning model according to the migration feature data of the learning stage to obtain an updated transfer learning model;

[0096] The transfer learning module 270 is used to perform transfer learning based on updating the convolutional network and the transfer learning model according to the EEG signals in the learning stage.

[0097] Optionally, the data preprocessing module 210 is further configured to:

[0098] Performing waveform processing on the original EEG signal to obtain a preprocessed EEG signal; the waveform processing includes filtering, baseline correction and artifact removal;

[0099] Performing spatiotemporal redistribution on the preprocessed EEG signal to obtain a spatiotemporal distributed EEG signal;

[0100] Data cleaning is performed on spatiotemporally distributed EEG signals to obtain spatiotemporally characteristic signals; data cleaning includes standardization and null value filling.

[0101] Optionally, the feature convolution module 220 is further used to:

[0102] According to the spatiotemporal characteristic signals, the time channel convolution is performed using the time convolution kernel to obtain the EEG time characteristics;

[0103] According to the spatiotemporal characteristic signals, spatial channel convolution is performed using the spatial convolution kernel to obtain EEG spatial features;

[0104] Batch normalization is performed on the EEG temporal features and EEG spatial features to obtain the migration feature data in the training phase.

[0105] Optionally, the model training module 230 is further used to:

[0106] Embedding space mapping is performed on the migrated feature data in the training phase to obtain pre-processed training data;

[0107] Constructing a position embedding matrix according to the preprocessed training data; adding the position embedding matrix to the preprocessed training data to obtain the position embedding training data;

[0108] According to the position embedding training data, the encoder to be trained is trained to obtain an encoder;

[0109] Based on the multi-head attention mechanism, the training data is embedded according to the position, and the output is obtained by encoding.

[0110] Construct a loss function based on the position embedding training data and output data;

[0111] According to the loss function, the Transformer model is optimized to obtain the EEG transfer learning model.

[0112] Wherein, the encoder includes an axe_x encoder, an axe_y encoder and an ave_z encoder;

[0113] The encoder corresponds to the three dimensions of the input data of the Transformer model.

[0114] Optionally, the convolution model updating module 250 is further configured to:

[0115] The divergence loss is calculated based on the feature data migrated in the training phase and the feature data migrated in the learning phase to obtain the KL divergence loss;

[0116] According to the KL divergence loss, the time-space feature convolutional network is back-propagated to obtain an updated feature convolutional network.

[0117] Optionally, the transfer learning model updating module 260 is further used to:

[0118] Construct a migration position embedding matrix based on the migration feature data of the learning phase; add the migration position embedding matrix to the migration feature data of the learning phase to obtain the embedded migration feature data of the learning phase;

[0119] Freeze the axe_x encoder and the axe_y encoder, train the ave_z encoder based on the feature data migrated in the post-embedding learning phase, and obtain the updated ave_z encoder;

[0120] Based on the multi-head attention mechanism, according to the feature data of the learning stage migration, the output is encoded through the axe_x encoder, axe_y encoder and the updated ave_z encoder to obtain the transfer learning output data;

[0121] Construct a cross entropy loss function based on the feature data transferred during the learning phase and the output data of transfer learning;

[0122] According to the cross entropy loss function, the EEG transfer learning model is optimized and trained to obtain an updated transfer learning model.

[0123] The present invention proposes a cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals. By standardizing and extracting multi-dimensional features of EEG, the model can still obtain good performance with less training data, reducing the cost and time of data collection and model training. By optimizing the transfer learning algorithm, the data loss of the transfer data and the training data is calculated during the transfer process, and the feature convolutional network is adjusted through back propagation, so that the temporal and spatial feature convolutional network can extract transfer data close to the characteristics of the training data, effectively enhancing the adaptability and generalization ability of the model in different subjects and different scenarios, and significantly improving the effect and stability of transfer learning. ; By optimizing the structure of the Transformer model, introducing the multi-head attention mechanism and position embedding matrix, the self-attention mechanism is fully utilized, the Transformer model's ability to capture and utilize EEG spatiotemporal features is improved, the model structure is simplified, and the training complexity and computational cost are reduced; in the process of transfer learning, by freezing two encoders (axe_x and axe_y), the stability of feature extraction is ensured, so that after the migration data features are added, the model can more effectively focus on updating the encoder of the ave_z dimension, which not only reduces unnecessary computational burden, but also further improves the efficiency and effect of transfer learning. The present invention is an efficient and stable cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals.

[0124] Figure 3 is a schematic diagram of a transfer learning device provided by an embodiment of the present invention, such as Figure 3 As shown, the transfer learning device may include the above Figure 2 The transfer learning device 310 for cross-subjects and cross-scenarios based on the spatiotemporal features of EEG signals is shown. Optionally, the transfer learning device 310 may include a first processor 2001 .

[0125] Optionally, the transfer learning device 310 may further include a memory 2002 and a transceiver 2003 .

[0126] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0127] Combine the following Figure 3 The components of the transfer learning device 310 are introduced in detail:

[0128] The first processor 2001 is the control center of the transfer learning device 310, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0129] Optionally, the first processor 2001 may perform various functions of the transfer learning device 310 by running or executing a software program stored in the memory 2002 , and calling data stored in the memory 2002 .

[0130] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0131] In a specific implementation, as an embodiment, the transfer learning device 310 may also include multiple processors, such as Figure 3 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0132] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0133] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0134] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0135] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0136] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and be transmitted through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0137] It should be noted that Figure 3 The structure of the transfer learning device 310 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0138] In addition, the technical effects of the transfer learning device 310 can refer to the technical effects of the cross-subject and cross-scenario transfer learning method based on the spatiotemporal characteristics of EEG signals described in the above method embodiment, which will not be repeated here.

[0139] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0140] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0141] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0142] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0143] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0144] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0145] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0150] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0151] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A cross-subject and cross-scenario transfer learning method based on the spatiotemporal characteristics of EEG signals, characterized in that: The method comprises: Acquire original EEG signals; pre-process the original EEG signals to obtain spatiotemporal characteristic signals; According to the spatiotemporal feature signal, convolution processing is performed through a time-space feature convolution network to obtain migration feature data in the training phase; The Transformer model is trained using the migration feature data in the training phase to obtain an EEG migration learning model; Acquire an EEG signal in the learning phase; preprocess the EEG signal in the learning phase to obtain a preprocessed signal in the learning phase; perform convolution processing on the preprocessed signal in the learning phase through the time-space feature convolution network to obtain migration feature data in the learning phase; According to the feature data migrated in the training phase and the feature data migrated in the learning phase, the time-space feature convolution network is updated to obtain an updated feature convolution network; According to the migration feature data of the learning stage, the EEG migration learning model is updated to obtain an updated migration learning model; According to the EEG signals in the learning phase, transfer learning is performed based on the updated convolutional network and the updated transfer learning model.

2. The cross-subject and cross-scenario transfer learning method based on the spatiotemporal characteristics of EEG signals according to claim 1 is characterized in that: The preprocessing of the original EEG signal to obtain a spatiotemporal characteristic signal includes: Performing waveform processing on the original EEG signal to obtain a preprocessed EEG signal; the waveform processing includes filtering, baseline correction and artifact removal; Performing spatiotemporal redistribution on the preprocessed EEG signal to obtain a spatiotemporal distributed EEG signal; The spatiotemporal distribution EEG signal is cleaned to obtain spatiotemporal characteristic signals; the data cleaning includes standardization and null value filling.

3. The cross-subject and cross-scenario transfer learning method based on spatiotemporal features of EEG signals according to claim 1 is characterized in that: The step of performing convolution processing on the spatiotemporal feature signal through a time-space feature convolution network to obtain migration feature data in the training phase includes: According to the spatiotemporal characteristic signal, a time channel convolution is performed using a time convolution kernel to obtain an EEG time feature; According to the spatiotemporal characteristic signal, a spatial channel convolution is performed using a spatial convolution kernel to obtain EEG spatial characteristics; Batch normalization is performed on the EEG temporal features and the EEG spatial features to obtain migration feature data in the training phase.

4. The cross-subject and cross-scenario transfer learning method based on spatiotemporal features of EEG signals according to claim 1 is characterized in that: The method of training the Transformer model using the migration feature data in the training phase to obtain the EEG migration learning model includes: Performing embedding space mapping on the migration feature data in the training phase to obtain pre-processed training data; Constructing a position embedding matrix according to the preprocessed training data; adding the position embedding matrix to the preprocessed training data to obtain position embedding training data; According to the position embedding training data, the encoder to be trained is trained to obtain an encoder; Based on the multi-head attention mechanism, embedding the training data according to the position, encoding and outputting through the encoding, and obtaining output data; Constructing a loss function based on the position embedding training data and the output data; According to the loss function, the Transformer model is optimized to obtain an EEG transfer learning model.

5. The cross-subject and cross-scenario transfer learning method based on spatiotemporal features of EEG signals according to claim 4 is characterized in that: The encoder includes an axe_x encoder, an axe_y encoder and an ave_z encoder; The encoder corresponds to the three dimensions of the input data of the Transformer model.

6. The cross-subject and cross-scenario transfer learning method based on spatiotemporal features of EEG signals according to claim 1 is characterized in that: The updating of the time-space feature convolution network according to the migration feature data in the training phase and the migration feature data in the learning phase to obtain an updated feature convolution network includes: Performing divergence loss calculation based on the migration feature data in the training phase and the migration feature data in the learning phase to obtain a KL divergence loss; According to the KL divergence loss, the time-space feature convolutional network is back-propagated to obtain an updated feature convolutional network.

7. The cross-subject and cross-scenario transfer learning method based on spatiotemporal features of EEG signals according to claim 5 is characterized in that: The updating of the EEG transfer learning model according to the migration feature data of the learning stage to obtain an updated transfer learning model includes: Constructing a migration position embedding matrix according to the migration feature data in the learning phase; adding the migration position embedding matrix to the migration feature data in the learning phase to obtain the embedded migration feature data in the learning phase; Freeze the axe_x encoder and the axe_y encoder, train the ave_z encoder according to the feature data migrated in the post-embedding learning phase, and obtain an updated ave_z encoder; Based on the multi-head attention mechanism, according to the feature data of the learning stage migration, encoding and outputting are performed through the axe_x encoder, the axe_y encoder and the updated ave_z encoder to obtain the transfer learning output data; Constructing a cross entropy loss function based on the migration feature data in the learning phase and the migration learning output data; According to the cross entropy loss function, the EEG transfer learning model is optimized and trained to obtain an updated transfer learning model.

8. A cross-subject and cross-scenario transfer learning device based on the spatiotemporal features of EEG signals, the cross-subject and cross-scenario transfer learning device based on the spatiotemporal features of EEG signals is used to implement the cross-subject and cross-scenario transfer learning method based on the spatiotemporal features of EEG signals as claimed in any one of claims 1 to 7, characterized in that: The device comprises: A data preprocessing module is used to obtain raw EEG signals; preprocess the raw EEG signals to obtain spatiotemporal characteristic signals; A feature convolution module, used to perform convolution processing through a time-space feature convolution network according to the spatiotemporal feature signal to obtain migration feature data in the training phase; A model training module, used to train the Transformer model using the migration feature data in the training phase to obtain an EEG migration learning model; A migration data acquisition module is used to acquire EEG signals in the learning phase; preprocess the EEG signals in the learning phase to obtain preprocessed signals in the learning phase; and convolute the preprocessed signals in the learning phase through the time-space feature convolution network to obtain migration feature data in the learning phase. A convolutional model updating module, used to update the time-space feature convolutional network according to the feature data migrated in the training phase and the feature data migrated in the learning phase, to obtain an updated feature convolutional network; A transfer learning model updating module, used to update the EEG transfer learning model according to the migration feature data of the learning stage to obtain an updated transfer learning model; The transfer learning module is used to perform transfer learning based on the EEG signal in the learning stage, the updated convolutional network and the updated transfer learning model.

9. A transfer learning device, characterized in that: The transfer learning device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images