P300 spelling method based on convolutional mixer network

Abstract

This invention discloses a P300 spelling method based on a convolutional mixer network, comprising: acquiring EEG samples and performing preprocessing operations; using a temporal convolutional mixer to effectively extract the temporal features of the EEG samples, improving the network's learning ability for temporal information; using a spatial mixer to compress and mix spatial features to extract effective spatial features; using a pooling layer to perform average pooling operations on the spatial features; using an output layer to generate the probability that the EEG sample contains a P300 waveform; determining the row and column of the EEG sample containing the P300 waveform, determining the position of the target character, and outputting the spelled character. This invention can effectively mine the deep spatiotemporal feature relationships of EEG signals through a convolutional mixer network, enhance the network's ability to extract valuable features, significantly improve the P300 character spelling performance, and has high practicality.

Description

Technical Field

[0001] This invention relates to the technical field of P300 character spelling, and in particular to a P300 spelling method based on a convolutional mixer network. Background Technology

[0002] Brain-computer interface (BCI) technology provides a form of communication that does not rely on physical movement by directly interpreting brain activity and translating it into control commands for external devices. Character spelling based on the P300 waveform in event-related potentials (ERPs) is a typical application, allowing users to output specific characters using brainwave signals. However, accurate detection of P300 waveforms presents challenges due to low signal-to-noise ratios, noise sensitivity, and spatiotemporal complexity.

[0003] Current research on P300 detection mainly falls into two categories: traditional machine learning methods and deep learning methods. Traditional machine learning methods primarily rely on manual feature extraction and classical classifiers, with relatively transparent feature selection and classification models, offering strong interpretability. In P300 signal decoding applications, methods such as Support Vector Machines, Linear Discriminant Analysis (LDA), and Stepwise LDA are widely used. However, these traditional methods heavily rely on manual features and are susceptible to inter-trial variation, resulting in poor robustness and generalization ability. With the rise of deep learning, more and more methods based on deep neural networks are being applied to P300 signal decoding. Convolutional neural networks are widely used to automatically learn spatiotemporal features in EEG signals, effectively improving P300 detection accuracy and performing exceptionally well in character recognition tasks. The EEGNet model, by incorporating a deep separable convolutional structure, optimizes spatiotemporal pattern extraction of EEG signals, further enhancing its generalization ability in complex EEG data. Capsule networks can better handle the hierarchical relationships of EEG signals, significantly improving the decoding effect of P300 signals. These deep learning methods can automatically extract more complex and representative features, thus significantly outperforming traditional methods. However, deep learning models often require a large amount of training data to avoid overfitting, and their robustness remains a challenge in low signal-to-noise ratio environments. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a P300 spelling method based on a convolutional mixer network. This method can more effectively extract the spatiotemporal features of EEG samples and utilize label information to enhance the similarity of samples within the same category and expand the differences between different categories, thereby optimizing the feature space and reducing the network's dependence on a large amount of labeled data, providing more stable and accurate spelling performance.

[0005] To achieve the above objectives, the technical solution provided by this invention is as follows: a P300 spelling method based on a convolutional mixer network, wherein the convolutional mixer network includes a temporal convolutional mixer, a spatial convolutional mixer, a pooling layer, and an output layer; the temporal convolutional mixer captures important features of EEG samples in the temporal dimension through convolution operations; the spatial convolutional mixer is used to extract correlation features between different channels in the spatial dimension, enhancing the expression of spatial information; the pooling layer reduces the dimensionality of features and highlights important features through average pooling operations; the output layer maps the extracted features to a low-dimensional space and outputs the probability that the EEG sample contains a P300 waveform, thereby determining the character to be spelled.

[0006] The specific implementation of the P300 spelling method includes:

[0007] EEG samples were acquired and preprocessed to obtain standardized and normalized EEG samples.

[0008] The pre-processed EEG samples were processed using a pre-trained convolutional mixer network as follows:

[0009] The preprocessed EEG samples are mixed using a temporal convolution mixer to effectively extract the temporal features of the EEG samples, resulting in a temporal feature map. The temporal convolution mixer includes an embedding layer, depthwise convolution, and pointwise convolution. Both depthwise and pointwise convolutions are followed by an ELU activation function and batch normalization. A residual connection is established between the embedding layer and the pointwise convolution. The embedding layer divides the EEG samples into smaller temporal blocks and expands them into multiple feature maps. The depthwise convolution performs individual convolution on each feature map to obtain local temporal features. The pointwise convolution fuses the local temporal features to optimize global information. The ELU activation function and batch normalization enhance the network's expressive power and optimize feature distribution. The residual connection prevents information loss.

[0010] The temporal feature map output by the temporal convolutional mixer is fed into the spatial convolutional mixer to extract effective spatial features and obtain a spatial feature map. The spatial convolutional mixer includes convolution, ELU activation function and batch normalization. The convolution compresses and mixes temporal features at the spatial level, mapping multiple channels into one channel. The ELU activation function and batch normalization are used to improve the network's expressive power.

[0011] A pooling layer is used to perform average pooling on the spatial feature map output by the spatial convolution mixer, which reduces the spatial dimension, reduces computational complexity, and extracts more significant features to obtain pooled features.

[0012] The pooling features output by the pooling layer are flattened and fully connected using the output layer, and then output to a single neuron to obtain the probability that the EEG sample contains a P300 waveform.

[0013] Using the aforementioned convolutional mixer network for prediction, the probability that the EEG samples corresponding to multiple flashes in a character spelling task contain the P300 waveform is obtained. The row and column of the EEG sample containing the P300 waveform are determined, thereby determining the position of the target character and realizing the output of the spelled character.

[0014] Furthermore, EEG samples were obtained from the P300 spelling device, which contains a 6×6 character matrix with 36 characters. When a user wants to spell a certain character, that character becomes the target character. During the fixation on the target character, each row and each column of the character matrix flashes once, with the first six flashes being column flashes and the last six being row flashes, for a total of 12 flashes. These 12 flashes recorded 12 EEG samples. When the row or column containing the target character flashed, the user's brain produced a distinct P300 waveform; when the row or column not containing the target character flashed, the user's brain did not produce a distinct P300 waveform. A P300 waveform is generated. By detecting the presence of the P300 waveform in the EEG sample, the row and column of the stimulus are identified, thereby inferring the target character that the user is paying attention to. The acquired EEG sample is 2 seconds long and contains data from 32 channels. To obtain the complete P300 waveform, the EEG data time window from 0 to 610 ms after the visual stimulus is extracted and downsampled to 78 time points. The EEG data from the 32 channels is bandpass filtered from 0.5 to 20 Hz, and the data is standardized and normalized. Finally, the obtained EEG sample X = {x1, x2, ..., x...} i ,…,x N The dimension of} is 78×32, where x i Let X be the i-th EEG sample, and N represent the number of EEG samples input into the convolutional mixer network. EEG samples X include two categories: those containing P300 waveforms and those without. The label Y = {y1, y2, ..., y...} for each EEG sample. i …,y N} indicates whether the EEG sample contains a P300 waveform; if it does, the label is 1, and if it does not, the label is 0, where y i The label is the one corresponding to the i-th EEG sample.

[0015] Furthermore, the temporal convolutional mixer extracts temporal features from the preprocessed EEG samples, outputting a temporal feature map with both local and global features, as follows:

[0016] First, convolution is used as the embedding layer to divide the preprocessed EEG samples into small time blocks and expand the input EEG samples into 40 feature maps.

[0017] F embedding =Conv(X)

[0018] In the formula, F embedding Represents the expanded feature map; Conv represents the convolution operation.

[0019] Then, depthwise convolution is used to convolve each feature map individually with a specific kernel, generating a depth feature map with 40 channels to extract local spatial features for each channel; ELU activation function and batch normalization are applied after depthwise convolution to optimize the global feature distribution and improve the network's performance and expressive power; linear residual connections are added between the embedding layer and pointwise convolution to preserve the overall trend of temporal changes and prevent information loss.

[0020] F depth =F embedding +BN(ELU(ConvDepthwise(F embedding )))

[0021] In the formula, F depth represents deep feature map, BN represents batch normalization, ELU represents ELU activation function, and ConvDepthwise represents depthwise convolution.

[0022] Since depthwise convolution cannot effectively capture the feature correlations between channels, pointwise convolution is used to fuse the features generated by depthwise convolution from various dimensions, ensuring global information fusion. ELU and batch normalization are also applied after pointwise convolution to obtain the temporal feature map F. temporal :

[0023] F temporal =BN(ELU(ConvPointwise(F depth )))

[0024] In the formula, ConvPointwise represents pointwise convolution.

[0025] Furthermore, the spatial convolutional mixer extracts and fuses spatial features from the temporal feature map, outputting a spatial feature map, as follows:

[0026] The spatial convolutional mixer modifies the temporal feature map F along the spatial dimension. temporal Convolution is performed to integrate valuable features extracted from different channels. It compresses all channel information into a single channel, thereby reducing data dimensionality while achieving global spatial feature extraction and fusion. The spatial convolution mixer also uses the ELU activation function and batch normalization to further improve the training efficiency and performance of the network, ultimately obtaining the spatial feature map F. spatial :

[0027] F spatial =BN(ELU(Conv(F temporal ))).

[0028] Furthermore, the pooling layer performs average pooling on the obtained spatial feature map to further compress and aggregate spatiotemporal information, reduce the number of parameters, and minimize the risk of overfitting, resulting in the pooled feature F. pooling :

[0029] F pooling =AvgPooling(F spatial )

[0030] In the formula, AvgPooling represents the average pooling operation.

[0031] Furthermore, the output layer first pools the feature F pooling This high-dimensional data is flattened into a one-dimensional representation, and then the extracted features are mapped through a fully connected layer before being output to a single neuron. The output layer introduces a Sigmoid activation function to restrict the output value to the range [0, 1], representing the probability P that the EEG sample contains a P300 waveform. target :

[0032] P target =Sigmoid(Linear(Flatten(F pooling )))

[0033] In the formula, Sigmoid represents the Sigmoid activation function, Linear represents a fully connected layer, and Flatten represents the flattening operation.

[0034] Furthermore, the loss function for training the convolutional mixer network is a hybrid loss function, consisting of supervised contrastive loss L. S and cross-entropy loss L C The structure comprises two loss functions: supervised contrastive loss establishes a better decision boundary in the feature space, enhancing the network's understanding of the data and improving its ability to identify rare classes; cross-entropy loss allows for explicit classification of EEG samples through probability output, directly optimizing classification accuracy; this hybrid loss function enables the network to extract different feature representations for samples within each class, thereby significantly improving the network's feature learning ability and classification accuracy, while also improving the network's robustness and generalization ability, and reducing dependence on a large amount of labeled data; among them, supervised contrastive loss L... S The pooling features output by the pooling layer are used for calculation:

[0035]

[0036] In the formula, S + Representative and EEG sample xi A set of samples with the same label, where p represents S. + The samples in |S + |Representative and EEG Samples x i The number of samples with the same label in the same batch, S represents the number of samples X excluding EEG sample X. i The set of other samples, where s represents a sample in S, u i Representing EEG sample x i The corresponding pooling feature, u p Representative and EEG sample x i Pooling features corresponding to samples with the same label, u s Representative of EEG sample x i Pooling features corresponding to other EEG samples, where τ represents temperature, are used to control the smoothness of training and the impact on difficult samples.

[0037] Cross-entropy loss L C The probability of the output waveform containing P300 is used for calculation:

[0038]

[0039] In the formula, p i Representing EEG sample x i The probability of the output layer containing the P300 waveform;

[0040] The total loss L of the network is the supervised comparison loss L. S and cross-entropy loss L C Mixed losses as a result:

[0041] L=λL S +(1-λ)L C

[0042] In the formula, λ represents the regularization parameter, which is used to balance the contributions between supervised contrast loss and cross-entropy loss.

[0043] Furthermore, the specific operations for using the convolutional mixer network to predict and output spelled characters are as follows:

[0044] In each round of the spelling task, each row and column of the character matrix flashes once, with the first six flashes being column flashes and the last six being row flashes, for a total of 12 flashes. These 12 flashes record 12 EEG samples. The P300 waveform only appears when the row and column containing the target character flashes. Since the target row and column are unique, a probability maximization strategy is used to determine the row and column containing the target character.

[0045]

[0046] In the formula, c represents the column where the target character is located, k represents the row where the target character is located, and P j The probability that the EEG sample corresponding to the j-th flash contains P300 is output by the convolutional mixer network; the target character spelled by the user can be determined by the row and column where the target character is located, thereby realizing the output of the P300 spelled character.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] 1. This invention designs a convolutional mixer network that segments the input EEG sample into smaller time blocks, integrates local and global information using methods such as depthwise convolution and separable convolution, and then extracts spatiotemporal hybrid features to obtain spatiotemporal dependencies for P300 waveform detection, thereby realizing P300 character spelling.

[0049] 2. This invention designs a hybrid loss function that integrates supervised contrastive learning into network training, alleviating the limitations of using cross-entropy loss alone, which results in unclear classification edges and susceptibility to noise. This joint supervision method enables the convolutional mixer network to learn deep features that simultaneously possess inter-class separability and intra-class compactness, thereby improving the network's prediction accuracy and generalization ability.

[0050] 3. This invention combines the idea of ​​convolutional mixer network with supervised contrastive learning, which can achieve good P300 character spelling performance and can still be effectively trained with the support of a small amount of training data, thereby saving the cost and time of data collection and annotation, and has broad application prospects.

[0051] In summary, this invention can effectively mine the deep feature relationships of EEG signals in the temporal and spatial dimensions, and make full use of label information to enhance the network's ability to extract valuable features from EEG signals. It significantly improves the network's P300 detection capability and character spelling performance, facilitates rapid deployment, and has high practical performance. Attached Figure Description

[0052] Figure 1 This is a framework diagram of the method of the present invention.

[0053] Figure 2 This is a schematic diagram of the structure of a convolutional mixer network.

[0054] Figure 3 This is a schematic diagram of the P300 character speller; the "S" in the white box at the top of the diagram represents the target character, and the characters within the black area form a character matrix that can be spelled.

[0055] Figure 4 This is a schematic diagram of the temporal convolution mixer; ELU in the diagram is the ELU activation function.

[0056] Figure 5 This is a schematic diagram of the spatial convolution mixer; ELU in the diagram is the ELU activation function. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0058] This embodiment discloses a P300 spelling method based on a convolutional mixer network. The convolutional mixer network includes a temporal convolutional mixer, a spatial convolutional mixer, a pooling layer, and an output layer. The temporal convolutional mixer captures important features of EEG samples in the temporal dimension through convolution operations. The spatial convolutional mixer extracts correlation features between different channels in the spatial dimension, enhancing the expression of spatial information. The pooling layer reduces the dimensionality of features and highlights important features through average pooling operations. The output layer maps the extracted features to a low-dimensional space and outputs the probability of P300 waveforms in the EEG sample, thereby determining the character to be spelled. In summary, this invention designs a temporal and spatial convolutional mixer to extract the spatiotemporal features of EEG signals, and introduces supervised contrastive learning to optimize the feature space, improve classification performance, and achieve high-precision P300 character spelling. The framework of this invention is as follows: Figure 1 As shown, the structure of the convolutional mixer network is as follows: Figure 2 As shown, the specific spelling method for P300 includes the following steps:

[0059] EEG samples were acquired and preprocessed to obtain standardized and normalized EEG samples.

[0060] The pre-processed EEG samples were processed using a pre-trained convolutional mixer network as follows:

[0061] The preprocessed EEG samples are blended using a temporal convolutional mixer to effectively extract the temporal features of the EEG samples, resulting in a temporal feature map. The temporal convolutional mixer includes an embedding layer, depthwise convolution, and pointwise convolution. Both depthwise and pointwise convolutions are followed by an ELU activation function and batch normalization. A residual connection is established between the embedding layer and the pointwise convolution. The embedding layer divides the EEG samples into smaller temporal blocks and expands them into 40 feature maps. The depthwise convolution performs individual convolution on each feature map to obtain local temporal features. The pointwise convolution fuses the local temporal features to optimize global information. The ELU activation function and batch normalization enhance the network's expressive power and optimize feature distribution. The residual connection prevents information loss.

[0062] The temporal feature map output by the temporal convolutional mixer is fed into the spatial convolutional mixer to extract effective spatial features and obtain a spatial feature map. The spatial convolutional mixer includes convolution, ELU activation function and batch normalization. The convolution compresses and mixes temporal features at the spatial level, mapping multiple channels into one channel. The ELU activation function and batch normalization are used to improve the network's expressive power.

[0063] A pooling layer is used to perform average pooling on the spatial feature map output by the spatial convolution mixer, which reduces the spatial dimension, reduces computational complexity, and extracts more significant features to obtain pooled features.

[0064] The pooling features output by the pooling layer are flattened and fully connected using the output layer, and then output to a single neuron to obtain the probability that the input EEG sample contains a P300 waveform.

[0065] Using the aforementioned convolutional mixer network for prediction, the probability that the EEG samples corresponding to 12 flashes in a character spelling task contain the P300 waveform is obtained. The row and column of the EEG sample containing the P300 waveform are determined, thereby determining the position of the target character and realizing the output of the spelled character.

[0066] Specifically, EEG samples are acquired and preprocessed, as follows:

[0067] EEG samples were obtained from a traditional P300 speller, which contains a 6×6 character matrix with a total of 36 characters, such as... Figure 3 As shown. When a user attempts to spell a character, that character becomes the target character. During the focus on this target character, each row and column of the character matrix flashes once: the first six flashes are column flashes, and the last six are row flashes, for a total of 12 flashes. These 12 flashes record 12 EEG samples. When the row or column containing the target character flashes, the user's brain produces a distinct P300 waveform; when a row or column not containing the target character flashes, the user's brain does not produce a P300 waveform. By detecting the presence of the P300 waveform in the EEG samples, the row and column of the stimulus are identified, thereby inferring the target character the user is focusing on. The acquired EEG samples are 2 seconds long and contain data from 32 channels. To obtain the complete P300 waveform, a time window of EEG data from 0 to 610 ms from the start of the visual stimulus is extracted and downsampled to 78 time points. The 32 channels of EEG data are bandpass filtered from 0.5 to 20 Hz, and the data is standardized and normalized. Finally, the obtained EEG sample X = {x1, x2, ..., x...} i ,…,x N The dimension of} is 78×32, where x iLet X be the i-th EEG sample, and N represent the number of EEG samples input into the convolutional mixer network. EEG samples X include two categories: those containing P300 waveforms and those without. The label Y = {y1, y2, ..., y...} for each EEG sample. i …,y N} indicates whether the EEG sample contains a P300 waveform; if it does, the label is 1, and if it does not, the label is 0, where y i The label is the one corresponding to the i-th EEG sample.

[0068] Specifically, the temporal convolution mixer includes an embedding layer, depthwise convolutions, and pointwise convolutions; an ELU activation function and batch normalization following the depthwise and pointwise convolutions; and residual connections between the embedding layer and the pointwise convolutions, such as... Figure 4 As shown, the preprocessed EEG samples are used to extract temporal features using the temporal convolution mixer, outputting a temporal feature map with both local and global features:

[0069] First, convolution is used as the embedding layer to divide the preprocessed EEG samples into smaller time blocks, and the input EEG samples are expanded into 40 feature maps.

[0070] F embedding =Conv(X)

[0071] In the formula, F embedding Represents the expanded feature map; Conv represents the convolution operation.

[0072] Then, depthwise convolution is used to convolve each feature map individually with a specific kernel, generating a depth feature map with 40 channels to extract local spatial features for each channel; ELU activation function and batch normalization are applied after depthwise convolution to optimize the global feature distribution and improve the network's performance and expressive power; linear residual connections are added between the embedding layer and pointwise convolution to preserve the overall trend of temporal changes and prevent information loss.

[0073] F depth =F embedding +BN(ELU(ConvDepthwise(F embedding )))

[0074] In the formula, F depth represents deep feature map, BN represents batch normalization, ELU represents ELU activation function, and ConvDepthwise represents depthwise convolution.

[0075] Since depthwise convolution cannot effectively capture the feature correlations between channels, pointwise convolution is used to fuse the features generated by depthwise convolution from various dimensions, ensuring global information fusion. ELU and BatchNorm are also applied after pointwise convolution to obtain the temporal feature map F. temporal :

[0076] F temporal =BN(ELU(ConvPointwise(F depth )))

[0077] In the formula, ConvPointwise represents pointwise convolution;

[0078] Specifically, the spatial convolution mixer includes convolution, ELU activation function, and batch normalization, such as... Figure 5 As shown, it extracts and fuses spatial features from the temporal feature map, outputting a spatial feature map:

[0079] Spatial convolutional mixer modulates the temporal feature map F along the spatial dimension. temporal Convolution is performed to integrate valuable features extracted from different channels. It compresses all channel information into a single channel, thereby reducing data dimensionality while achieving global spatial feature extraction and fusion. This module also uses the ELU activation function and batch normalization to further improve the network's training efficiency and performance, ultimately obtaining the spatial feature map F. spatial :

[0080] F spatial =BN(ELU(Conv(F temporal )))

[0081] Specifically, the pooling layer performs average pooling on the obtained spatial feature map to further compress and aggregate spatiotemporal information, reduce the number of parameters, and minimize the risk of overfitting, thereby obtaining the pooled feature F. pooling :

[0082] F pooling =AvgPooling(F spatial )

[0083] In the formula, AvgPooling represents the average pooling operation.

[0084] Specifically, the output layer performs a fully connected operation on the pooling layer output to obtain the probability that the input EEG sample contains a P300 waveform, and then performs iterative training of the network using a hybrid loss function consisting of supervised contrastive loss and cross-entropy loss.

[0085] The output layer first pools the feature F poolingThis high-dimensional data is flattened into a one-dimensional representation, and then the extracted features are mapped through a fully connected layer before being output to a single neuron. A sigmoid activation function is introduced to restrict the output value to the range [0, 1], representing the probability P that the target EEG sample contains a P300 waveform. target :

[0086] P target =Sigmoid(Linear(Flatten(F pooling )))

[0087] In the formula, Sigmoid represents the Sigmoid activation function, Linear represents a fully connected layer, and Flatten represents the flattening operation;

[0088] The loss function for network training is a hybrid loss function, consisting of supervised contrastive loss L. S and cross-entropy loss L C The structure is as follows: Supervised contrastive loss establishes a better decision boundary in the feature space, enhancing the network's understanding of the data and improving its ability to identify rare classes. Cross-entropy loss allows for explicit classification of EEG samples through probability output, directly optimizing classification accuracy. This hybrid loss function enables the network to extract different feature representations for samples within each class, significantly improving the model's feature learning ability and classification accuracy. It also enhances the network's robustness and generalization ability, reducing dependence on large amounts of labeled data. Among these, the supervised contrastive loss L... S The pooling features output by the pooling layer are used for calculation:

[0089]

[0090] In the formula, S + Representative and EEG sample x i A set of samples with the same label, where p represents S. + The samples in |S + |Representative and EEG Samples x i The number of samples with the same label in the same batch, S represents the number of samples X excluding EEG sample X. i The set of other samples, where s represents a sample in S, u i Representing EEG sample x i The corresponding pooling feature, u p Representative and EEG sample x i Pooling features corresponding to samples with the same label, u s Representative of EEG sample x i Pooling features corresponding to other EEG samples, where τ represents temperature and is set to 0.07, are used to control the smoothness of training and the impact on difficult samples.

[0091] Cross-entropy loss L C The probability of the output waveform containing P300 is used for calculation:

[0092]

[0093] In the formula, p i Representing EEG sample x i The probability of the output layer containing the P300 waveform;

[0094] The total loss L of the network is the supervised comparison loss L. S and cross-entropy loss L C Mixed losses as a result:

[0095] L=λL S +(1-λ)L C

[0096] In the formula, λ represents the regularization parameter, which is used to balance the contributions between supervised contrast loss and cross-entropy loss.

[0097] Specifically, using the aforementioned convolutional mixer network for prediction, the probability that the EEG samples corresponding to 12 flashes in a character spelling task contain the P300 waveform is obtained. This allows us to determine the row and column of the EEG sample containing the P300 waveform, thereby identifying the target character's position and enabling the output of the spelled character.

[0098] In each round of the spelling task, each row and column of the character matrix flashes once, with the first six flashes being column flashes and the last six being row flashes, for a total of 12 flashes. These 12 flashes record 12 EEG samples. The P300 waveform only appears when the row and column containing the target character flashes. Since the target row and column are unique, a probability maximization strategy is used to determine the row and column containing the target character:

[0099]

[0100] In the formula, c represents the column where the target character is located, k represents the row where the target character is located, and P j The probability that the EEG sample corresponding to the j-th flash contains P300 is output by the convolutional mixer network; the target character spelled by the user can be determined by the row and column where the target character is located, thus realizing the output of the P300 spelled character.

[0101] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A P300 spelling method based on a convolutional mixer network, characterized in that, The convolutional mixer network includes a temporal convolutional mixer, a spatial convolutional mixer, a pooling layer, and an output layer. The temporal convolutional mixer captures important features of EEG samples at the temporal level through convolutional operations. The spatial convolutional mixer extracts correlation features between different channels in the spatial dimension, enhancing the expression of spatial information. The pooling layer reduces the dimensionality of features and highlights important features through average pooling operations. The output layer maps the extracted features to a low-dimensional space and outputs the probability of the EEG sample containing a P300 waveform, thereby determining the spelled character. The specific implementation of the P300 spelling method includes: EEG samples were acquired and preprocessed to obtain standardized and normalized EEG samples. The pre-processed EEG samples were processed using a pre-trained convolutional mixer network as follows: The preprocessed EEG samples are mixed using a temporal convolution mixer to effectively extract the temporal features of the EEG samples, resulting in a temporal feature map. The temporal convolution mixer includes an embedding layer, depthwise convolution, and pointwise convolution. Both depthwise and pointwise convolutions are followed by an ELU activation function and batch normalization. A residual connection is established between the embedding layer and the pointwise convolution. The embedding layer divides the EEG samples into smaller temporal blocks and expands them into multiple feature maps. The depthwise convolution performs individual convolution on each feature map to obtain local temporal features. The pointwise convolution fuses the local temporal features to optimize global information. The ELU activation function and batch normalization enhance the network's expressive power and optimize feature distribution. The residual connection prevents information loss. The temporal feature map output by the temporal convolutional mixer is fed into the spatial convolutional mixer to extract effective spatial features and obtain a spatial feature map. The spatial convolutional mixer includes convolution, ELU activation function and batch normalization. The convolution compresses and mixes temporal features at the spatial level, mapping multiple channels into one channel. The ELU activation function and batch normalization are used to improve the network's expressive power. A pooling layer is used to perform average pooling on the spatial feature map output by the spatial convolution mixer, which reduces the spatial dimension, reduces computational complexity, and extracts more significant features to obtain pooled features. The pooling features output by the pooling layer are flattened and fully connected using the output layer, and then output to a single neuron to obtain the probability that the EEG sample contains a P300 waveform. Using the aforementioned convolutional mixer network for prediction, the probability that the EEG samples corresponding to multiple flashes in a character spelling task contain the P300 waveform is obtained. The row and column of the EEG sample containing the P300 waveform are determined, thereby determining the position of the target character and realizing the output of the spelled character.

2. The P300 spelling method based on a convolutional mixer network according to claim 1, characterized in that, EEG samples were obtained from a P300 spelling device, which contains a 6×6 character matrix with 36 characters. When a user wants to spell a character, that character becomes the target character. During the fixation on the target character, each row and each column of the character matrix flashes once, with the first six flashes being column flashes and the last six being row flashes, for a total of 12 flashes. These 12 flashes recorded 12 EEG samples. When the row or column containing the target character flashes, the user's brain produces a noticeable P300 waveform; when rows or columns not containing the target character flash, the user's brain does not produce a waveform. The P300 waveform was generated; the presence of the P300 waveform in the EEG sample was detected to identify the row and column of the stimulus, thereby inferring the target character that the user was paying attention to; the acquired EEG sample was 2 seconds long and contained data from 32 channels. To obtain the complete P300 waveform, the EEG data time window from 0 to 610 ms from the start of the visual stimulus was extracted and downsampled to 78 time points; the EEG data from the 32 channels were bandpass filtered from 0.5 to 20 Hz, and the data were standardized and normalized; finally, the obtained EEG sample X = {x1, x2, ..., x...} i ,…,x N The dimension of} is 78×32, where x i Let X be the i-th EEG sample, and N represent the number of EEG samples input into the convolutional mixer network. EEG samples X include two categories: those containing P300 waveforms and those without. The label Y = {y1, y2, ..., y...} for each EEG sample. i …,y N } indicates whether the EEG sample contains a P300 waveform; if it does, the label is 1, and if it does not, the label is 0, where y i The label is the one corresponding to the i-th EEG sample.

3. The P300 spelling method based on a convolutional mixer network according to claim 2, characterized in that, The temporal convolutional mixer extracts temporal features from the preprocessed EEG samples, outputting a temporal feature map with both local and global features, as follows: First, convolution is used as the embedding layer to divide the preprocessed EEG samples into small time blocks and expand the input EEG samples into 40 feature maps. F embedding =Conv(X) In the formula, F embedding Represents the expanded feature map; Conv represents the convolution operation. Then, depthwise convolution is used to convolve each feature map individually with a specific convolution kernel, generating a depth feature map with 40 channels to extract local spatial features for each channel; ELU activation function and batch normalization are applied after deep convolution to optimize the global feature distribution and improve the network's performance and expressive power; linear residual connections are added between the embedding layer and pointwise convolution to preserve the overall trend of temporal changes and prevent information loss. F depth =F embedding +BN(ELU(ConvDepthwise(F embedding ))) In the formula, F depth represents deep feature map, BN represents batch normalization, ELU represents ELU activation function, and ConvDepthwise represents depthwise convolution. Since depthwise convolution cannot effectively capture the feature correlations between channels, pointwise convolution is used to fuse the features generated by depthwise convolution from various dimensions, ensuring global information fusion. ELU and batch normalization are also applied after pointwise convolution to obtain the temporal feature map F. temporal : F temporal =BN(ELU(ConvPointwise(F depth ))) In the formula, ConvPointwise represents pointwise convolution.

4. The P300 spelling method based on a convolutional mixer network according to claim 3, characterized in that, The spatial convolutional mixer extracts and fuses spatial features from the temporal feature map, outputting a spatial feature map as follows: The spatial convolutional mixer modifies the temporal feature map F along the spatial dimension. temporal Convolution is performed to integrate valuable features extracted from different channels. It compresses all channel information into a single channel, thereby reducing data dimensionality while achieving global spatial feature extraction and fusion. The spatial convolution mixer also uses the ELU activation function and batch normalization to further improve the training efficiency and performance of the network, ultimately obtaining the spatial feature map F. spatial : F spatial =BN(TWO(Conv(F temporal )))。 5. The P300 spelling method based on a convolutional mixer network according to claim 4, characterized in that, The pooling layer performs average pooling on the obtained spatial feature map to further compress and aggregate spatiotemporal information, reduce the number of parameters, and minimize the risk of overfitting, resulting in the pooled feature F. pooling : F pooling =AvgPooling(F spatial ) In the formula, AvgPooling represents the average pooling operation.

6. The P300 spelling method based on a convolutional mixer network according to claim 5, characterized in that, The output layer first pools the feature F. pooling This high-dimensional data is flattened into a one-dimensional representation, and then the extracted features are mapped through a fully connected layer before being output to a single neuron. The output layer introduces a Sigmoid activation function to restrict the output value to the range [0, 1], representing the probability P that the EEG sample contains a P300 waveform. target : P target =Sigmoid(Linear(Flatten(F pooling ))) In the formula, Sigmoid represents the Sigmoid activation function, Linear represents a fully connected layer, and Flatten represents the flattening operation.

7. The P300 spelling method based on a convolutional mixer network according to claim 6, characterized in that, The loss function for training the convolutional mixer network is a hybrid loss function, consisting of supervised contrastive loss L. S and cross-entropy loss L C The structure comprises two loss functions: supervised contrastive loss establishes a better decision boundary in the feature space, enhancing the network's understanding of the data and improving its ability to identify rare classes; cross-entropy loss allows for explicit classification of EEG samples through probability output, directly optimizing classification accuracy; this hybrid loss function enables the network to extract different feature representations for samples within each class, thereby significantly improving the network's feature learning ability and classification accuracy, while also improving the network's robustness and generalization ability, and reducing dependence on a large amount of labeled data; among them, supervised contrastive loss L... S The pooling features output by the pooling layer are used for calculation: In the formula, S + Representative and EEG sample x i A set of samples with the same label, where p represents S. + The samples in |S + |Representative and EEG Samples x i The number of samples with the same label in the same batch, S represents the number of samples X excluding EEG sample X. i The set of other samples, where s represents a sample in S, u i Representing EEG sample x i The corresponding pooling feature, u p Representative and EEG sample x i Pooling features corresponding to samples with the same label, u s Representative of EEG sample x i Pooling features corresponding to other EEG samples, where τ represents temperature, are used to control the smoothness of training and the impact on difficult samples. Cross-entropy loss L C The probability of the output waveform containing P300 is used for calculation: In the formula, p i Representing EEG sample x i The probability of the output layer containing the P300 waveform; The total loss L of the network is the supervised comparison loss L. S and cross-entropy loss L C Mixed losses as a result: L=λL S +(1-λ)L C In the formula, λ represents the regularization parameter, which is used to balance the contributions between supervised contrast loss and cross-entropy loss.

8. The P300 spelling method based on a convolutional mixer network according to claim 7, characterized in that, The specific steps for using the convolutional mixer network to predict and output spelled characters are as follows: In each round of the spelling task, each row and column of the character matrix flashes once, with the first six flashes being column flashes and the last six being row flashes, for a total of 12 flashes. These 12 flashes record 12 EEG samples. The P300 waveform only appears when the row and column containing the target character flashes. Since the target row and column are unique, a probability maximization strategy is used to determine the row and column containing the target character. In the formula, c represents the column where the target character is located, k represents the row where the target character is located, and P j The probability that the EEG sample corresponding to the j-th flash contains P300 is output by the convolutional mixer network; the target character spelled by the user can be determined by the row and column where the target character is located, thereby realizing the output of the P300 spelled character.

Images