A cross-subject and cross-scene transfer learning method based on electroencephalogram signal space-time features

Abstract

This invention provides a method and apparatus for cross-subject, cross-scene transfer learning based on the spatiotemporal features of electroencephalogram (EEG) signals, relating to the field of brain-computer interface technology. The method includes: preprocessing raw EEG signals to obtain spatiotemporal feature signals; performing convolution processing on the spatiotemporal feature signals using a temporal-spatial feature convolutional network to obtain training-stage transfer feature data; training a Transformer model using the training-stage transfer feature data to obtain an EEG transfer learning model; acquiring EEG signals during the learning stage; preprocessing and performing convolution processing on the learning-stage EEG signals to obtain learning-stage transfer feature data; updating the temporal-spatial feature convolutional network and the EEG transfer learning model based on the training-stage and learning-stage transfer feature data; and performing transfer learning based on the updated convolutional network and the updated transfer learning model using the learning-stage EEG signals. This invention provides an efficient and stable cross-subject, cross-scene transfer learning method based on the spatiotemporal features of EEG signals.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and in particular to a method and device for cross-subject and cross-scene transfer learning based on the spatiotemporal characteristics of electroencephalogram (EEG) signals. Background Technology

[0002] Brain-computer interfaces (BCIs) are communication pathways between the brain and the external environment that do not rely on peripheral nerves and muscle tissue. BCIs identify the subject's intentions by decoding electroencephalography (EEG) signals. BCIs can translate neural activity into a series of output commands to control computers, robots, and other devices. Cognitive assessments can be performed by analyzing the subject's EEG. Improving the adaptability and robustness of BCI systems in different users and diverse scenarios is crucial; transfer learning is a key technology for the widespread application of BCIs.

[0003] Transfer learning is a machine learning method that leverages knowledge learned in one domain to improve learning performance in another related domain. It can enhance a model's generalization ability even with limited data or high labeling costs. In BCI research, the differences in EEG patterns among different participants, as well as variations in the EEG patterns of the same participant in different scenarios, coupled with the difficulty of data collection for each participant and the complexity of the scenarios, limit the widespread application of BCI. Therefore, understanding how to transfer models to other participants and scenarios using limited participant data to achieve cross-participant and cross-scenario model application can significantly reduce model training requirements and achieve more efficient EEG decoding control, which is of great significance for the widespread application of BCI.

[0004] In existing BCI research, precise decoding algorithms exist for EEG processing analysis of individual subjects and scenarios, enabling efficient identification and control of user intent and cognitive assessment. These include algorithms such as EEG Convolutional Neural Network (EEGNet), Deep Neural Network (DeepNet), and ParaAtt (Parameter Attribute). However, these methods typically rely on large amounts of individual training data, and each subject requires individual model training, which presents significant limitations in practical applications.

[0005] Transfer learning has also made some progress in BCI (Browser Computational Learning). By transferring a model trained on one subject to other subjects or different scenarios, transfer learning can reduce the need for new data and improve the model's adaptability and generalization ability. For example, techniques such as EEG differential entropy features, source analysis, and conditional frames are used to achieve cross-subject and cross-scenario transfer at the levels of EEG data features and network architecture, respectively. However, due to the significant differences in EEG patterns among different subjects, the signal features of some subjects are difficult to transfer, resulting in unstable model performance and difficulty in meeting the requirements of practical applications.

[0006] As a deep learning model based on self-attention mechanisms, the Transformer model has shown great potential in the field of Biological Sequence Interpretation (BCI) in recent years. Its powerful sequence modeling capabilities enable it to effectively capture spatiotemporal features in EEG, improving decoding accuracy. Existing research has attempted to decode EEG using Transformers, achieving better results than traditional methods. However, applying Transformers to transfer learning in BCI still faces some challenges. Transformer models typically contain a large number of parameters, making training complex and computationally expensive. Furthermore, a small amount of transfer data is unlikely to significantly impact models with large parameters. Therefore, further research and optimization are needed to fully utilize the self-attention mechanism to better capture and leverage EEG features.

[0007] In the existing technology, there is a lack of an efficient and stable transfer learning method across subjects and scenarios based on the spatiotemporal characteristics of EEG signals. Summary of the Invention

[0008] To address the existing technical problems of high performance in decoding individual EEG signals using existing EEG decoding algorithms, which rely heavily on large amounts of individual training data; the instability of current transfer learning methods due to significant differences in EEG signals among different subjects, and the difficulty of fully utilizing the self-attention mechanism of Transformer models to capture the spatiotemporal features of EEG, this invention provides a cross-subject, cross-scene transfer learning method and apparatus based on the spatiotemporal features of EEG signals. The technical solution is as follows:

[0009] On the one hand, a cross-subject, cross-scene transfer learning method based on the spatiotemporal features of EEG signals is provided. This method is implemented by a transfer learning device and includes:

[0010] Acquire raw EEG signals; preprocess the raw EEG signals to obtain spatiotemporal feature signals;

[0011] Based on the spatiotemporal feature signals, convolution processing is performed through a time-space feature convolutional network to obtain transfer feature data during the training phase.

[0012] The Transformer model is trained using the transfer feature data from the training phase to obtain an EEG transfer learning model.

[0013] Acquire EEG signals during the learning phase; preprocess the EEG signals during the learning phase to obtain preprocessed signals during the learning phase; perform convolution processing on the preprocessed signals during the learning phase through the temporal-spatial feature convolutional network to obtain transfer feature data during the learning phase.

[0014] The temporal-spatial feature convolutional network is updated based on the transfer feature data from the training phase and the transfer feature data from the learning phase to obtain an updated feature convolutional network.

[0015] Based on the transfer feature data of the learning stage, the EEG transfer learning model is updated to obtain an updated transfer learning model;

[0016] Based on the EEG signals during the learning phase, transfer learning is performed using the updated convolutional network and the updated transfer learning model.

[0017] On the other hand, a cross-subject, cross-scene transfer learning device based on spatiotemporal features of EEG signals is provided. This device is applied to a cross-subject, cross-scene transfer learning method based on spatiotemporal features of EEG signals. The device includes:

[0018] The data preprocessing module is used to acquire raw EEG signals and preprocess the raw EEG signals to obtain spatiotemporal feature signals.

[0019] The feature convolution module is used to perform convolution processing on the spatiotemporal feature signals through a time-space feature convolution network to obtain transfer feature data during the training phase.

[0020] The model training module is used to train the Transformer model using the transfer feature data from the training phase to obtain an EEG transfer learning model.

[0021] The transfer data acquisition module is used to acquire EEG signals during the learning phase; preprocess the EEG signals during the learning phase to obtain preprocessed signals during the learning phase; and perform convolution processing on the preprocessed signals during the learning phase through the temporal-spatial feature convolutional network to obtain transfer feature data during the learning phase.

[0022] The convolutional model update module is used to update the temporal-space feature convolutional network based on the transfer feature data of the training phase and the transfer feature data of the learning phase, so as to obtain the updated feature convolutional network.

[0023] The transfer learning model update module is used to update the EEG transfer learning model based on the transfer feature data of the learning stage to obtain an updated transfer learning model.

[0024] The transfer learning module is used to perform transfer learning based on the EEG signals during the learning phase, using the updated convolutional network and the updated transfer learning model.

[0025] On the other hand, a transfer learning device is provided, the transfer learning device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the methods described above for cross-subject, cross-scene transfer learning based on spatiotemporal features of EEG signals is implemented.

[0026] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of cross-subject, cross-scene transfer learning based on spatiotemporal features of EEG signals.

[0027] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0028] This invention proposes a cross-subject, cross-scenario transfer learning method based on the spatiotemporal features of EEG signals. By standardizing and extracting multidimensional features from EEG, the model can still achieve good performance with limited training data, reducing the cost and time of data collection and model training. By optimizing the transfer learning algorithm, calculating the data loss between the transfer data and training data during the transfer process, and adjusting the feature convolutional network through backpropagation, the temporal and spatial feature convolutional networks can extract transfer data with features close to those of the training data. This effectively enhances the model's adaptability and generalization ability across different subjects and scenarios, significantly improving the effectiveness and stability of transfer learning. This invention optimizes the Transformer model structure by introducing a multi-head attention mechanism and a position embedding matrix, fully utilizing the self-attention mechanism to enhance the Transformer model's ability to capture and utilize EEG spatiotemporal features. This simplifies the model structure and reduces training complexity and computational cost. During transfer learning, freezing two encoders (axe_x and axe_y) ensures the stability of feature extraction. This allows the model to more effectively focus on updating the ave_z dimension encoder after the addition of transfer data features, reducing unnecessary computational burden and further improving the efficiency and effectiveness of transfer learning. This invention is an efficient and stable cross-subject, cross-scene transfer learning method based on spatiotemporal features of EEG signals. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of a cross-subject, cross-scene transfer learning method based on the spatiotemporal features of EEG signals provided in an embodiment of the present invention;

[0031] Figure 2 This is a block diagram of a cross-subject, cross-scene transfer learning device based on the spatiotemporal features of electroencephalogram (EEG) signals provided in an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram of the structure of a transfer learning device provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0034] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0035] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0036] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0037] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0038] This invention provides a cross-subject, cross-scene transfer learning method based on the spatiotemporal characteristics of electroencephalogram (EEG) signals. This method can be implemented using a transfer learning device, which can be a terminal or a server. Figure 1 The flowchart shown is for a cross-subject, cross-scene transfer learning method based on the spatiotemporal features of EEG signals. The processing flow of this method may include the following steps:

[0039] S1. Acquire raw EEG signals; preprocess the raw EEG signals to obtain spatiotemporal characteristic signals.

[0040] Optionally, the raw EEG signal is preprocessed to obtain spatiotemporal feature signals, including:

[0041] The raw EEG signal is subjected to waveform processing to obtain a preprocessed EEG signal; waveform processing includes filtering, baseline correction and artifact removal.

[0042] Spatiotemporal redistribution of preprocessed EEG signals was performed to obtain spatiotemporally distributed EEG signals;

[0043] Data cleaning is performed on spatiotemporally distributed EEG signals to obtain spatiotemporally characteristic signals; data cleaning includes standardization and null value filling.

[0044] In one feasible implementation, the training data EEG signal is filtered, baseline corrected, and artifact removed in the preprocessing section; and time and space are redistributed to convert the two-dimensional EEG signal into three-dimensional data containing electrode distribution information and time information, where the dimensions of the three-dimensional data are represented as a. x a y t d The data is then standardized and padded with null values ​​to maintain its integrity and rationality. The preprocessing section converts the EEG data to a dimension of (a... x , a y , t d Input of ).

[0045] S2. Based on the spatiotemporal feature signals, perform convolution processing through a time-space feature convolutional network to obtain transfer feature data during the training phase.

[0046] Optionally, based on the spatiotemporal feature signals, convolution processing is performed using a time-space feature convolutional network to obtain transfer feature data from the training phase, including:

[0047] Based on the spatiotemporal feature signals, temporal convolution kernels are used to perform temporal channel convolution to obtain EEG temporal features;

[0048] Based on the spatiotemporal feature signals, spatial channel convolution is performed using spatial convolution kernels to obtain the spatial features of EEG.

[0049] Batch normalization was performed on the temporal and spatial features of EEG to obtain transfer feature data during the training phase.

[0050] In one feasible implementation, the present invention uses n elements of size (a x , a y 3) The temporal convolution kernel is used to convolve the temporal channels to extract the temporal features of the EEG.

[0051] Where n is the number of convolutional kernels. In a convolutional layer, each convolutional kernel generates a feature map, and n corresponds to the number of output feature maps or the depth of the layer; a x , a y The two values ​​represent the size of the convolution kernel in the spatial dimensions, that is, the dimensions of the convolution kernel in the height and width. 3 represents the size of the convolution kernel in the channel dimension of the input data.

[0052] Using a value of (3, 3, t) d-2 The spatial convolution kernel is used to convolve the spatial channels to extract spatial features under the electrode distribution information.

[0053] The shape of the input data is (H, W, t). d H and W represent the spatial dimensions (height and width) of the data, t d This represents the channel dimension (which may correspond to different time points or different features). Its size is (3, 3, t). d-2 The convolutional kernel will slide in a 3x3 window in the spatial dimension and cover all data from the second channel to the last channel in the channel dimension (i.e., t). d-2 (One channel).

[0054] The purpose of this convolution operation is to extract spatial features while preserving temporal information. For example, in video processing, motion features can be extracted from a series of image frames; in multi-channel biosignal processing, spatiotemporal features of interactions between different electrodes can be extracted.

[0055] The data undergoes batch normalization and pooling to unify the dimensions of the EEG 3D signals to `ave`. Here, `ave_x` = `ave_y` = `ave_z` = `ave`, where `x`, `y`, and `z` represent the three dimensions. If the sample size of a batch is `batch_size`, then after preprocessing and spatiotemporal feature extraction, the dimensions of a batch are (batch_size, n, ave_x, ave_y, ave_z).

[0056] S3. Use the transfer feature data from the training phase to train the Transformer model and obtain the EEG transfer learning model.

[0057] Optionally, the Transformer model is trained using transfer feature data from the training phase to obtain an EEG transfer learning model, including:

[0058] Embedding space mapping is performed on the transfer feature data during the training phase to obtain preprocessed training data;

[0059] Construct a location embedding matrix based on the preprocessed training data; add the location embedding matrix to the preprocessed training data to obtain the location embedding training data;

[0060] The encoder is trained by embedding training data at the location and then trained to obtain the encoder.

[0061] Based on the multi-head attention mechanism, training data is embedded according to position, and the output data is obtained by encoding.

[0062] A loss function is constructed based on the location embeddings of the training data and the output data;

[0063] Based on the loss function, the Transformer model is optimized to obtain the EEG transfer learning model.

[0064] In one feasible implementation, a projection layer is used to map the batch training data, mapping data with dimensions (batch_size, n, ave_x, ave_y, ave_z) to the embedding space emb_size, that is, the batch data dimension is (batch_size, emb_size, ave_x, ave_y, ave_z).

[0065] Position embedding is performed on the batch data. A position embedding matrix with a shape and size of (1, emb_size, ave_x, ave_y, ave_z) is created and added to the data, while the dimensionality of the data remains unchanged. An encoder is trained for each of the three dimensions of each EEG for decoding. An attention mechanism is used to compute Q (query matrix), K (key matrix), and V (value matrix) from the input data. Where Q = XWQ, K = XWK, and V = XWV. The dimensions of WQ, WK, and WV are all (emb_size, emb_size).

[0066] This invention introduces a multi-head attention mechanism, where the number of heads is head_num, the dimension of each head is head_dim, and emb_size = head_num × head_dim. For the ave_x dimension, the Q, K, and V matrices are projected and rearranged to the (batch_size, head_num, ave_y × ave_z, head_dim) dimension. Attention weights are calculated by the dot product of Q and K and divided by a scaling factor. The softmax function is applied to process the data to obtain an attention matrix with a shape of (batch_size, head_num, ave2, ave2). The attention matrix is ​​multiplied by V, and the output is rearranged to the original shape, i.e., the encoder output of the ave_x dimension is (batch_size, emb_size, ave2). Similarly, the encoder outputs of the ave_y and ave_z dimensions are both (batch_size, emb_size, ave2). The outputs of the three encoders are merged into (batch_size, emb_size, 3ave2), and the outputs are mapped to the required class num_class through the Reduce layer and the fully connected layer.

[0067] The encoders include the axe_x encoder, the axe_y encoder, and the ave_z encoder;

[0068] The encoder corresponds to the three dimensions of the input data of the Transformer model.

[0069] In one feasible implementation, the dimension of each EEG signal sample in this invention is (ave_x, ave_y, ave_z).

[0070] S4. Acquire EEG signals during the learning phase; preprocess the EEG signals during the learning phase to obtain preprocessed signals during the learning phase; perform convolution processing on the preprocessed signals during the learning phase through a temporal-spatial feature convolutional network to obtain transfer feature data during the learning phase.

[0071] In one feasible implementation, the migration data is preprocessed and spatiotemporal features are extracted, and each migration data is transformed into the form of (ave_x, ave_y, ave_z).

[0072] S5. Based on the transfer feature data from the training phase and the transfer feature data from the learning phase, update the temporal-spatial feature convolutional network to obtain the updated feature convolutional network.

[0073] Optionally, the temporal-spatial feature convolutional network is updated based on the transfer feature data from the training phase and the learning phase to obtain an updated feature convolutional network, including:

[0074] The KL divergence loss is calculated based on the transfer feature data from the training phase and the transfer feature data from the learning phase.

[0075] Based on the KL divergence loss, backpropagation is performed on the temporal-spatial feature convolutional network to obtain the updated feature convolutional network.

[0076] In one feasible implementation, the Kullback-Leibler divergence (KL) loss between the same category of transferred data and training data is calculated, and backpropagation is used to adjust the temporal feature convolutional network and the spatial feature convolutional network, encouraging the temporal feature convolutional network and the spatial feature convolutional network to extract features from the transferred data that are close to the features of the training data.

[0077] S6. Based on the transfer feature data of the learning stage, update the EEG transfer learning model to obtain the updated transfer learning model.

[0078] Optionally, the EEG transfer learning model is updated based on the transfer feature data of the learning stage to obtain an updated transfer learning model, including:

[0079] A transfer position embedding matrix is ​​constructed based on the transfer feature data of the learning stage; the transfer position embedding matrix is ​​added to the transfer feature data of the learning stage to obtain the embedded transfer feature data of the learning stage.

[0080] Freeze the axe_x encoder and axe_y encoder, and train the ave_z encoder based on the transfer feature data from the post-embedding learning stage to obtain the updated ave_z encoder;

[0081] Based on the multi-head attention mechanism, the transfer learning output data is obtained by encoding the transfer feature data in the learning stage through the axe_x encoder, axe_y encoder and the update ave_z encoder.

[0082] Construct a cross-entropy loss function based on the transfer feature data and transfer learning output data during the learning phase;

[0083] Based on the cross-entropy loss function, the EEG transfer learning model is optimized and trained to obtain an updated transfer learning model.

[0084] In one feasible implementation, the two encoders axe_x and axe_y are frozen, the transfer data features are added to the position embedding, and then fed into three encoders, a Reduce layer and a fully connected layer. Backpropagation is performed using the cross-entropy loss function to update the ave_z encoder to update the EEG transfer learning model.

[0085] S7. Based on the EEG signals during the learning phase, perform transfer learning by updating the convolutional network and the transfer learning model.

[0086] In one feasible implementation, this invention proposes a method for converting two-dimensional EEG signals into three-dimensional data containing electrode distribution information and temporal information, ensuring the integrity and rationality of the data. Temporal and spatial features in the EEG signal are extracted using multi-layer convolutional kernels, and feature fusion is performed using a Transformer model. This method not only effectively extracts complex spatiotemporal features from EEG signals but also improves the efficiency and accuracy of feature extraction.

[0087] By calculating the KL divergence loss between the transfer data and the training data, and using backpropagation to adjust the temporal and spatial feature convolutional networks, this invention proposes a strategy to freeze the axe_x and axe_y encoders and focus on updating the ave_z encoder during cross-subject and cross-scene transfer processes.

[0088] This invention proposes a cross-subject, cross-scenario transfer learning method based on the spatiotemporal features of EEG signals. By standardizing and extracting multidimensional features from EEG, the model can still achieve good performance with limited training data, reducing the cost and time of data collection and model training. By optimizing the transfer learning algorithm, calculating the data loss between the transfer data and training data during the transfer process, and adjusting the feature convolutional network through backpropagation, the temporal and spatial feature convolutional networks can extract transfer data with features close to those of the training data. This effectively enhances the model's adaptability and generalization ability across different subjects and scenarios, significantly improving the effectiveness and stability of transfer learning. This invention optimizes the Transformer model structure by introducing a multi-head attention mechanism and a position embedding matrix, fully utilizing the self-attention mechanism to enhance the Transformer model's ability to capture and utilize EEG spatiotemporal features. This simplifies the model structure and reduces training complexity and computational cost. During transfer learning, freezing two encoders (axe_x and axe_y) ensures the stability of feature extraction. This allows the model to more effectively focus on updating the ave_z dimension encoder after the addition of transfer data features, reducing unnecessary computational burden and further improving the efficiency and effectiveness of transfer learning. This invention is an efficient and stable cross-subject, cross-scene transfer learning method based on spatiotemporal features of EEG signals.

[0089] Figure 2 This is a block diagram illustrating a cross-subject, cross-scene transfer learning device based on spatiotemporal features of electroencephalogram (EEG) signals, according to an exemplary embodiment. The device is used for a cross-subject, cross-scene transfer learning method based on spatiotemporal features of EEG signals. (Refer to...) Figure 2The device includes a data preprocessing module 210, a feature convolution module 220, a model training module 230, a transfer data acquisition module 240, a convolutional model update module 250, a transfer learning model update module 260, and a transfer learning module 270. Among them:

[0090] Data preprocessing module 210 is used to acquire raw EEG signals and preprocess the raw EEG signals to obtain spatiotemporal feature signals.

[0091] The feature convolution module 220 is used to perform convolution processing on the spatiotemporal feature signals through a time-space feature convolution network to obtain transfer feature data during the training phase.

[0092] The model training module 230 is used to train the Transformer model using the transfer feature data from the training phase to obtain the EEG transfer learning model.

[0093] The transfer data acquisition module 240 is used to acquire EEG signals during the learning phase; preprocess the EEG signals during the learning phase to obtain preprocessed signals during the learning phase; and perform convolution processing on the preprocessed signals during the learning phase through a temporal-spatial feature convolutional network to obtain transfer feature data during the learning phase.

[0094] The convolutional model update module 250 is used to update the temporal-space feature convolutional network based on the transfer feature data during the training phase and the transfer feature data during the learning phase, so as to obtain the updated feature convolutional network.

[0095] The transfer learning model update module 260 is used to update the EEG transfer learning model based on the transfer feature data of the learning stage to obtain an updated transfer learning model.

[0096] The transfer learning module 270 is used to perform transfer learning based on the EEG signals during the learning phase, by updating the convolutional network and the transfer learning model.

[0097] Optionally, the data preprocessing module 210 is further used for:

[0098] The raw EEG signal is subjected to waveform processing to obtain a preprocessed EEG signal; waveform processing includes filtering, baseline correction and artifact removal.

[0099] Spatiotemporal redistribution of preprocessed EEG signals was performed to obtain spatiotemporally distributed EEG signals;

[0100] Data cleaning is performed on spatiotemporally distributed EEG signals to obtain spatiotemporally characteristic signals; data cleaning includes standardization and null value filling.

[0101] Optionally, the feature convolution module 220 is further used for:

[0102] Based on the spatiotemporal feature signals, temporal convolution kernels are used to perform temporal channel convolution to obtain EEG temporal features;

[0103] Based on the spatiotemporal feature signals, spatial channel convolution is performed using spatial convolution kernels to obtain the spatial features of EEG.

[0104] Batch normalization was performed on the temporal and spatial features of EEG to obtain transfer feature data during the training phase.

[0105] Optionally, the model training module 230 is further used for:

[0106] Embedding space mapping is performed on the transfer feature data during the training phase to obtain preprocessed training data;

[0107] Construct a location embedding matrix based on the preprocessed training data; add the location embedding matrix to the preprocessed training data to obtain the location embedding training data;

[0108] The encoder is trained by embedding training data at the location and then trained to obtain the encoder.

[0109] Based on the multi-head attention mechanism, training data is embedded according to position, and the output data is obtained by encoding.

[0110] A loss function is constructed based on the location embeddings of the training data and the output data;

[0111] Based on the loss function, the Transformer model is optimized to obtain the EEG transfer learning model.

[0112] The encoders include the axe_x encoder, the axe_y encoder, and the ave_z encoder;

[0113] The encoder corresponds to the three dimensions of the input data of the Transformer model.

[0114] Optionally, the convolutional model update module 250 is further used for:

[0115] The KL divergence loss is calculated based on the transfer feature data from the training phase and the transfer feature data from the learning phase.

[0116] Based on the KL divergence loss, backpropagation is performed on the temporal-spatial feature convolutional network to obtain the updated feature convolutional network.

[0117] Optionally, the transfer learning model update module 260 is further used for:

[0118] A transfer position embedding matrix is ​​constructed based on the transfer feature data of the learning stage; the transfer position embedding matrix is ​​added to the transfer feature data of the learning stage to obtain the embedded transfer feature data of the learning stage.

[0119] Freeze the axe_x encoder and axe_y encoder, and train the ave_z encoder based on the transfer feature data from the post-embedding learning stage to obtain the updated ave_z encoder;

[0120] Based on the multi-head attention mechanism, the transfer learning output data is obtained by encoding the transfer feature data in the learning stage through the axe_x encoder, axe_y encoder and the update ave_z encoder.

[0121] Construct a cross-entropy loss function based on the transfer feature data and transfer learning output data during the learning phase;

[0122] Based on the cross-entropy loss function, the EEG transfer learning model is optimized and trained to obtain an updated transfer learning model.

[0123] This invention proposes a cross-subject, cross-scenario transfer learning method based on the spatiotemporal features of EEG signals. By standardizing and extracting multidimensional features from EEG, the model can still achieve good performance with limited training data, reducing the cost and time of data collection and model training. By optimizing the transfer learning algorithm, calculating the data loss between the transfer data and training data during the transfer process, and adjusting the feature convolutional network through backpropagation, the temporal and spatial feature convolutional networks can extract transfer data with features close to those of the training data. This effectively enhances the model's adaptability and generalization ability across different subjects and scenarios, significantly improving the effectiveness and stability of transfer learning. This invention optimizes the Transformer model structure by introducing a multi-head attention mechanism and a position embedding matrix, fully utilizing the self-attention mechanism to enhance the Transformer model's ability to capture and utilize EEG spatiotemporal features. This simplifies the model structure and reduces training complexity and computational cost. During transfer learning, freezing two encoders (axe_x and axe_y) ensures the stability of feature extraction. This allows the model to more effectively focus on updating the ave_z dimension encoder after the addition of transfer data features, reducing unnecessary computational burden and further improving the efficiency and effectiveness of transfer learning. This invention is an efficient and stable cross-subject, cross-scene transfer learning method based on spatiotemporal features of EEG signals.

[0124] Figure 3 This is a schematic diagram of the structure of a transfer learning device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the transfer learning device may include the above-mentioned Figure 2 The illustrated device is a cross-subject, cross-scene transfer learning device based on the spatiotemporal characteristics of EEG signals. Optionally, the transfer learning device 310 may include a first processor 2001.

[0125] Optionally, the transfer learning device 310 may also include a memory 2002 and a transceiver 2003.

[0126] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0127] The following is combined with Figure 3 A detailed description of each component of the transfer learning device 310 is provided below:

[0128] The first processor 2001 is the control center of the transfer learning device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0129] Optionally, the first processor 2001 can perform various functions of the transfer learning device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0130] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0131] In a specific implementation, as one example, the transfer learning device 310 may also include multiple processors, for example... Figure 3 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0132] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0133] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the migration learning device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0134] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0135] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0136] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the transfer learning device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0137] It should be noted that, Figure 3 The structure of the transfer learning device 310 shown in the diagram does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0138] Furthermore, the technical effect of the transfer learning device 310 can be referred to the technical effect of the cross-subject, cross-scene transfer learning method based on the spatiotemporal characteristics of EEG signals described in the above method embodiments, which will not be repeated here.

[0139] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0140] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0141] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0142] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0143] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0144] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0145] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0146] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-subject, cross-scene transfer learning method based on spatiotemporal features of electroencephalogram (EEG) signals, characterized in that, The method includes: Acquire raw EEG signals; preprocess the raw EEG signals to obtain spatiotemporal feature signals; Based on the spatiotemporal feature signals, convolution processing is performed through a time-space feature convolutional network to obtain transfer feature data during the training phase. The Transformer model is trained using the transfer feature data from the training phase to obtain an EEG transfer learning model. The step of training the Transformer model using the transfer feature data from the training phase to obtain the EEG transfer learning model includes: The training phase transfer feature data is embedded spatially mapped to obtain preprocessed training data; A location embedding matrix is ​​constructed based on the preprocessed training data; the location embedding matrix is ​​added to the preprocessed training data to obtain location embedding training data; The encoder to be trained is trained by embedding training data at the aforementioned locations to obtain the encoder. Based on the multi-head attention mechanism, training data is embedded according to the position, and the output data is obtained by encoding through the encoding. A loss function is constructed based on the location embedding training data and the output data; Based on the loss function, the Transformer model is optimized to obtain the EEG transfer learning model. The encoders include an axe_x encoder, an axe_y encoder, and an ave_z encoder; The encoder corresponds to the three dimensions of the input data of the Transformer model; Acquire EEG signals during the learning phase; preprocess the EEG signals during the learning phase to obtain preprocessed signals during the learning phase; perform convolution processing on the preprocessed signals during the learning phase through the temporal-spatial feature convolutional network to obtain transfer feature data during the learning phase. The temporal-spatial feature convolutional network is updated based on the transfer feature data from the training phase and the transfer feature data from the learning phase to obtain an updated feature convolutional network. The step of updating the temporal-spatial feature convolutional network based on the transfer feature data from the training phase and the transfer feature data from the learning phase to obtain an updated feature convolutional network includes: The KL divergence loss is calculated based on the transfer feature data from the training phase and the transfer feature data from the learning phase. Based on the KL divergence loss, the temporal-space feature convolutional network is backpropagated to obtain the updated feature convolutional network; Based on the transfer feature data of the learning stage, the EEG transfer learning model is updated to obtain an updated transfer learning model; The step of updating the EEG transfer learning model based on the transfer feature data of the learning stage to obtain an updated transfer learning model includes: A migration position embedding matrix is ​​constructed based on the learning stage migration feature data; the migration position embedding matrix is ​​added to the learning stage migration feature data to obtain the embedded learning stage migration feature data; Freeze the axe_x encoder and the axe_y encoder, and train the ave_z encoder based on the transfer feature data of the post-embedding learning stage to obtain the updated ave_z encoder; Based on the multi-head attention mechanism, according to the transfer feature data of the learning stage, the transfer learning output data is obtained by encoding and outputting through the axe_x encoder, the axe_y encoder and the update ave_z encoder. Construct a cross-entropy loss function based on the transfer feature data of the learning stage and the transfer learning output data; The EEG transfer learning model is optimized and trained according to the cross-entropy loss function to obtain an updated transfer learning model. Based on the EEG signals during the learning phase, transfer learning is performed using the updated feature convolutional network and the updated transfer learning model.

2. The cross-subject, cross-scene transfer learning method based on spatiotemporal features of EEG signals according to claim 1, characterized in that, The preprocessing of the raw EEG signal to obtain spatiotemporal feature signals includes: The original EEG signal is subjected to waveform processing to obtain a preprocessed EEG signal; the waveform processing includes filtering, baseline correction, and artifact removal. The preprocessed EEG signals are spatiotemporally redistributed to obtain spatiotemporally distributed EEG signals; The spatiotemporal distributed EEG signals are cleaned to obtain spatiotemporal feature signals; the data cleaning includes standardization and null padding.

3. The cross-subject, cross-scene transfer learning method based on spatiotemporal features of EEG signals according to claim 1, characterized in that, The step of obtaining transfer feature data during the training phase by performing convolution processing through a time-space feature convolutional network based on the spatiotemporal feature signal includes: Based on the spatiotemporal feature signals, temporal channel convolution is performed using a temporal convolution kernel to obtain EEG temporal features; Based on the spatiotemporal feature signals, spatial channel convolution is performed using spatial convolution kernels to obtain EEG spatial features. The EEG temporal features and EEG spatial features are batch normalized to obtain transfer feature data for the training phase.

4. A cross-subject, cross-scene transfer learning device based on spatiotemporal features of EEG signals, wherein the cross-subject, cross-scene transfer learning device based on spatiotemporal features of EEG signals is used to implement the cross-subject, cross-scene transfer learning method based on spatiotemporal features of EEG signals as described in any one of claims 1-3, characterized in that, The device includes: The data preprocessing module is used to acquire raw EEG signals and preprocess the raw EEG signals to obtain spatiotemporal feature signals. The feature convolution module is used to perform convolution processing on the spatiotemporal feature signals through a time-space feature convolution network to obtain transfer feature data during the training phase. The model training module is used to train the Transformer model using the transfer feature data from the training phase to obtain an EEG transfer learning model. The transfer data acquisition module is used to acquire EEG signals during the learning phase; preprocess the EEG signals during the learning phase to obtain preprocessed signals during the learning phase; and perform convolution processing on the preprocessed signals during the learning phase through the temporal-spatial feature convolutional network to obtain transfer feature data during the learning phase. The convolutional model update module is used to update the temporal-space feature convolutional network based on the transfer feature data of the training phase and the transfer feature data of the learning phase, so as to obtain the updated feature convolutional network. The transfer learning model update module is used to update the EEG transfer learning model based on the transfer feature data of the learning stage to obtain an updated transfer learning model. The transfer learning module is used to perform transfer learning based on the EEG signals of the learning stage, the updated feature convolutional network, and the updated transfer learning model.

5. A transfer learning device, characterized in that, The transfer learning device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 3.

Images