A visual brain-computer interface signal decoding method and system

Abstract

The present invention relates to the fields of brain science and computer technology, and in particular to a visual brain-computer interface signal decoding method and system thereof. The method comprises constructing a visual brain-computer filter bank, acquiring EEG data, and transforming it into multiple sub-band signals; constructing a convolutional neural network to extract the signal features corresponding to each sub-band signal, and splicing the signal features along the channel to form an aggregated feature map; constructing a temporal kernel selection network to calculate the weight values ​​of the spliced ​​feature map to obtain a weighted feature map; flattening the weighted feature map to obtain a one-dimensional feature vector; and constructing a classification module to map and classify the one-dimensional feature vector and output the stimulation frequency of the EEG data. The present invention optimizes feature extraction by selecting a temporal kernel, accurately capturing the characteristic pattern of steady-state visual evoked potential signals. By expanding the receptive field of the convolution kernel, task-related feature patterns are emphasized, significantly improving the classification performance and generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the fields of brain science and computer technology, and in particular to a visual brain-computer interface signal decoding method and system thereof. Background Art

[0002] Brain-Computer Interfaces (BCIs) aim to establish a direct, efficient connection between the brain and external devices capable of processing information. This allows for novel information exchange and device control, completely breaking the constraints of traditional brain information output models. Their application prospects in healthcare are particularly compelling. For example, they offer alternative motor control methods for patients with movement disorders, assisting in the early diagnosis and rehabilitation monitoring of neurological diseases; in smart home scenarios, they enable users to conveniently control home appliances through their brains; and in the entertainment industry, they create new immersive and interactive experiences.

[0003] Steady-State Visual Evoked Potential (SSVEP), a key paradigm in BCI technology, is derived from the brain's regular electrophysiological response to visual stimuli of specific frequencies. When the human or animal visual system receives visual stimuli that flicker at a fixed frequency or exhibit a specific pattern of changes, neuronal activity in the occipital cortex is synchronously modulated, resulting in the electroencephalogram (EEG) signal exhibiting periodic rhythmic changes closely related to the stimulus frequency. Specifically, this is manifested as significant power spectrum peaks at the fundamental frequency or multiples of the stimulus frequency. This characteristic provides a critical signal foundation and implementation path for the construction of BCIs based on SSVEP.

[0004] However, traditional SSVEP-based BCI signal decoding methods have obvious limitations:

[0005] (1) Although the untrained detection method is simple to operate and does not require special training for the user, due to the non-stationarity of EEG activity and variability between different users, its decoding accuracy in practical applications is low when the data length is short, making it difficult to meet the needs of practical applications.

[0006] (2) Although the detection method based on user training can optimize the classification model parameters through the training data of specific users to achieve higher decoding performance, the classification accuracy and robustness are often insufficient when dealing with complex multi-target recognition tasks, making it difficult for BCI technology based on SSVEP to be popularized and promoted on a large scale. Summary of the Invention

[0007] The present invention aims to at least improve one of the technical problems existing in the prior art. To this end, the present invention proposes a visual brain-computer interface signal decoding method and system.

[0008] The technical solutions of the present invention are as follows:

[0009] A visual brain-computer interface signal decoding method, comprising:

[0010] S101, constructing a filter bank for a visual brain-computer interface, wherein the filter bank includes a plurality of bandpass filters;

[0011] S102, acquiring EEG data, and transforming the EEG data into a plurality of sub-band signals according to the filter bank;

[0012] S103, constructing a convolutional neural network, wherein the convolutional neural network includes multiple convolutional layers;

[0013] S104, extracting signal features corresponding to the subband signals at each convolutional layer according to the convolutional neural network, and concatenating the signal features along the channels to form an aggregated feature map;

[0014] S105, constructing a temporal kernel selection network, and calculating weight values ​​for the spliced ​​feature maps according to the temporal kernel selection network, wherein the temporal kernel selection network includes dilated convolution and grouped convolution;

[0015] S106, weighting the spliced ​​feature map according to the calculated weight value to obtain a weighted feature map;

[0016] S107, flattening the weighted feature map to obtain a one-dimensional feature vector;

[0017] S108, constructing a classification module, wherein the classification module includes a fully connected layer and a normalized exponential (Softmax) function;

[0018] S109 , mapping and classifying the one-dimensional feature vector according to the fully connected layer and the Softmax function in the classification module to output the stimulation frequency of the EEG data.

[0019] In a possible technical solution, further, S105 specifically includes:

[0020] In the concatenated feature map, the features at different positions of the feature map are calculated respectively through the dilated convolution and grouped convolution on the two branches of the temporal kernel selection network;

[0021] According to the dilated convolution and grouped convolution operations of the two branches: and , based on dilated convolution and grouped convolution to construct two convolutions with different expanded receptive fields to capture more comprehensive feature information. The relationship is as follows:

[0022] ,

[0023] in, is the receptive field, d is the dilation rate, and k is the convolution kernel;

[0024] The size of the dilated convolution receptive field relative to the normal convolution receptive field of the same convolution kernel size can be expressed as follows:

[0025] ;

[0026] Feature extraction is performed based on two convolutions with different receptive fields to obtain rich contextual information in the input feature map. ,as follows

[0027] ;

[0028] In the formula is the input feature map, For branches dilated convolution and grouped convolution operations;

[0029] A selection mechanism is used to temporally select features from convolution kernels of different scales to obtain the temporal selection mask ;

[0030] Perform dot product calculation based on the temporal selection mask, and obtain a temporal selection weight map with N channels based on the dot product calculation and a convolution operation with a kernel size of (1,1). , ( )

[0031] .

[0032] Where, is the dot product calculation, is the convolution calculation, is the i-th channel, and N is the number of channels.

[0033] In a possible technical solution, further, in S106, specifically, outputting the input feature map and time selection weight graph The dot product in the element direction, that is , to obtain the weighted feature map.

[0034] In a possible technical solution, further, S107 includes flattening the weighted feature map into a one-dimensional feature vector.

[0035] In a possible technical solution, further, S109 includes transforming the expanded one-dimensional vector into a probability map through a fully connected layer.

[0036] In a possible technical solution, further, the convolutional layer includes:

[0037] The first convolutional layer is used to combine the signals of different electrodes;

[0038] The second convolutional layer is used to reduce the complexity of the network by downsampling the time dimension;

[0039] The third convolutional layer is used to extract temporal features.

[0040] In a possible technical solution, further, the filter bank includes three band-pass filters, each with a frequency range of 6 Hz to 90 Hz, 14 Hz to 90 Hz, and 22 Hz to 90 Hz, for converting single EEG data into three sub-band signals.

[0041] In a possible technical solution, further, the convolutional neural network is shared by each sub-band signal, and after the signal features of each sub-band signal are extracted, the signal features are spliced.

[0042] The visual brain-computer interface signal decoding method of the present invention integrates a filter bank, a convolutional neural network, and a temporal kernel selection network. The filter bank captures frequency domain information from different frequency bands, extracts temporal features from the frequency and spatial domains based on the convolutional neural network, and optimizes feature extraction by selecting a temporal kernel using a temporal kernel selection network. The present invention accurately captures the characteristic patterns of steady-state visual evoked potential signals and provides richer feature representations compared to other deep learning-based models. Specifically, the temporal kernel selection network includes dilated convolution and grouped convolution, which expands the receptive field of the decoding method, reduces the computational complexity of the model, and extracts temporal features important for steady-state visual evoked potential frequency recognition by selecting a temporal kernel. By expanding the receptive field of the convolution kernel, task-related feature patterns are emphasized, significantly improving the classification performance and generalization ability of the model.

[0043] A visual brain-computer interface signal decoding system, wherein the above method is used for decoding, comprising:

[0044] A first building block is used to build a filter bank for a visual brain-computer interface, wherein the filter bank includes a plurality of bandpass filters;

[0045] an acquisition module, configured to acquire EEG data and transform the EEG data into a plurality of sub-band signals according to the filter bank;

[0046] A second building block is used to build a convolutional neural network, wherein the convolutional neural network includes multiple convolutional layers;

[0047] An extraction module is used to extract signal features corresponding to the sub-band signals under each convolutional layer according to a convolutional neural network, and to concatenate the signal features along the channel to form an aggregated feature map;

[0048] A third construction module is used to construct a temporal kernel selection network, and calculate weight values ​​for the spliced ​​feature maps according to the temporal kernel selection network, wherein the temporal kernel selection network includes dilated convolution and grouped convolution;

[0049] A calculation module, configured to weight the concatenated feature maps according to the calculated weight values ​​to obtain weighted feature maps;

[0050] A fourth building module is used to build a classification module, wherein the classification module includes a fully connected layer and a Softmax function;

[0051] a flattening module, configured to flatten the weighted feature map according to the fully connected layer to obtain a feature map of a one-dimensional feature vector;

[0052] The output module is used to classify the one-dimensional feature vector according to the Softmax function in the classification module to output the stimulation frequency of the EEG data.

[0053] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned visual brain-computer interface signal decoding method when executing the computer program.

[0054] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is a flow chart of a method for decoding visual brain-computer interface signals according to an embodiment of the present invention;

[0057] Figure 2 2 is a schematic diagram of a decoding process of a visual brain-computer interface signal decoding method according to an embodiment of the present invention;

[0058] Figure 3 2 is a schematic diagram of a convolutional neural network structure of a visual brain-computer interface signal decoding method according to an embodiment of the present invention;

[0059] Figure 4 2. It is a schematic diagram of a temporal kernel selection network structure of a visual brain-computer interface signal decoding method according to an embodiment of the present invention;

[0060] Figure 5 is a schematic diagram of the classification accuracy and information transmission rate achieved by four methods in the embodiment on the Benchmark dataset;

[0061] Figure 6 Schematic diagram of the classification accuracy and information transmission rate achieved by the four methods in the embodiment on the BETA dataset. DETAILED DESCRIPTION

[0062] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] It should be noted that when an element is referred to as being “fixed to” another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being “connected to” another element, it may be directly connected to the other element or there may be an intermediate element at the same time.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0065] In the specification, claims, and accompanying drawings of this application, the terms "first," "second," "third," and the like are used to distinguish different objects and are not used to describe a particular order. Furthermore, the terms "including," "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a list of steps or elements may be included, or alternatively, steps or elements not listed may be included, or other steps or elements may be included that are inherent to the process, method, product, or apparatus.

[0066] Only portions relevant to the present application are shown in the accompanying drawings, not all of them. Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the various operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently, or simultaneously. In addition, the order of the various operations can be rearranged. The process can be terminated when its operations are completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0067] As used in this specification, the terms "component," "module," "system," "unit," and the like are used to refer to computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or distributed between two or more computers. In addition, these units can be executed from various computer-readable media having various data structures stored thereon. Units can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from a second unit interacting with another unit in a local system, a distributed system, and / or a network, such as the Internet, which interacts with other systems via signals).

[0068] Example 1

[0069] like Figures 1 to 6 As shown, this embodiment provides a visual brain-computer interface signal decoding method, which includes:

[0070] S101, constructing a filter bank for a visual brain machine, the filter bank comprising a plurality of bandpass filters, wherein three of the bandpass filters are designed using 6th-order Butterworth infinite impulse response (IIR) filters, with a maximum passband attenuation of 3 dB and a minimum stopband attenuation of 30 dB;

[0071] S102, acquiring EEG data, and transforming the EEG data into a plurality of sub-band signals according to the filter bank;

[0072] It should be noted that, in this embodiment, the filter bank includes three band-pass filters, each with a frequency range of 6 Hz to 90 Hz, 14 Hz to 90 Hz, and 22 Hz to 90 Hz, for converting a single EEG data into three sub-band signals.

[0073] S103, constructing a convolutional neural network, wherein the convolutional neural network includes multiple convolutional layers; the convolutional neural network is shared by each sub-band signal, and after extracting the signal features of each sub-band signal, the signal features are spliced.

[0074] S104, extracting signal features corresponding to the subband signals at each convolutional layer according to the convolutional neural network, and concatenating the signal features along the channels to form an aggregated feature map;

[0075] It should be noted that, in this embodiment, the convolutional layer includes:

[0076] The first convolutional layer consists of 60 kernels of size , the step length is , filled with "valid" filters, is the number of electrodes of the EEG signal, used to combine the signals of different electrodes. The random deactivation rate of this convolutional layer is 0.05;

[0077] The second convolutional layer consists of 60 kernels of size , the step length is , filled with "valid" filters, the complexity of the network is reduced by downsampling in the time dimension. The activation function of this layer is "Elu" and the random inactivation rate is 0.05;

[0078] The third convolutional layer consists of 60 kernels with a size of , the step length is , filled with "same" filters, used to extract temporal features. The random inactivation rate of this layer is 0.95;

[0079] In this embodiment, the three sub-band signals are spliced ​​along the feature channel dimension through the output of the convolutional network to form a feature map with 180 channels.

[0080] S105, constructing a temporal kernel selection network, and calculating weight values ​​for the spliced ​​feature maps based on the temporal kernel selection network; specifically comprising:

[0081] In the concatenated feature map, the features at different positions of the feature map are calculated respectively through the dilated convolution and grouped convolution on the two branches of the temporal kernel selection network;

[0082] According to the dilated convolution and grouped convolution operations of the two branches: and , based on dilated convolution and grouped convolution to construct two convolutions with different expanded receptive fields to capture more comprehensive feature information. The relationship is as follows:

[0083] ,

[0084] in, is the receptive field, d is the dilation rate, and k is the convolution kernel;

[0085] The size of the dilated convolution receptive field relative to the normal convolution receptive field of the same convolution kernel size can be expressed as follows:

[0086] ,;

[0087] Feature extraction is performed based on two convolutions with different receptive fields to obtain contextual information in the input feature map. ,as follows

[0088] ;

[0089] In the formula is the input feature map, For branches dilated convolution and grouped convolution operations;

[0090] A selection mechanism is used to temporally select features from convolution kernels of different scales to obtain the temporal selection mask ;

[0091] Perform dot product calculation based on the temporal selection mask, and obtain a temporal selection weight map with N channels based on the dot product calculation and a convolution operation with a kernel size of (1,1). ( )

[0092] ;

[0093] Where, is the dot product calculation, is the convolution calculation, is the i-th channel, and N is the number of channels.

[0094] S106, weighting the spliced ​​feature map according to the calculated weight value to obtain a weighted feature map; specifically, outputting the input feature map and time selection weight graph The dot product in the element direction, that is , to obtain the weighted feature map.

[0095] S107, flattening the weighted feature map to obtain a one-dimensional feature vector;

[0096] S108, constructing a classification module, wherein the classification module includes a fully connected layer and a Softmax function;

[0097] It should be noted that S108 specifically includes: using cross entropy loss and center loss The linear combination of , build a classification module containing a fully connected layer and a Softmax function, where:

[0098] ,

[0099] In the formula is the regularization parameter.

[0100] Cross Entropy Loss The definition is as follows

[0101]

[0102] In the formula and are the true labels and the predicted labels, The number of classes.

[0103] Center loss The definition is as follows

[0104]

[0105] In the formula is the first The feature vector of the sample, is the feature center of the class to which the sample belongs, which is updated when each batch of samples is trained. The number of classes.

[0106] S109 , mapping and classifying the one-dimensional feature vector according to the fully connected layer and the Softmax function in the classification module to output the stimulation frequency of the EEG data.

[0107] It should be noted that, in this embodiment, the features from the convolution kernels of different scales are time-selected to obtain the time-selective mask , specifically including:

[0108] First, concatenate the two temporal features along the channel dimension

[0109] ,

[0110] Where, and All are time characteristics.

[0111] Then, along the channel dimension Perform average pooling and max pooling , extract channel dimensions respectively Key features in and as follows

[0112]

[0113]

[0114] Secondly, concatenate these two feature maps into a two-channel feature map , through a kernel size of Convolution and Sigmoid function, transforming it into a time-selective mask .

[0115] The visual brain-computer interface signal decoding method of the present invention integrates a filter bank, a convolutional neural network, and a temporal kernel selection network. The present invention uses a filter bank to capture frequency domain information from different frequency bands, extracts temporal features from the frequency and spatial domains based on a convolutional neural network, and optimizes feature extraction by adjusting different receptive fields using a temporal kernel selection network. The visual brain-computer interface signal decoding method of the present invention accurately captures the characteristic patterns of steady-state visual evoked potential signals, providing a richer feature representation compared to other deep learning-based models. Specifically, the temporal kernel selection network incorporates dilated and grouped convolutions, which expand the receptive field of the decoding method and reduce the computational complexity of the model. By expanding the receptive field of the convolution kernel, task-related characteristic patterns are emphasized, significantly improving the model's classification performance and generalization ability. The present invention uses a linear combination of cross-entropy loss and center loss as the objective function for model optimization, which not only enhances the compactness of intra-class feature signals but also expands the distance between inter-class feature signals, further improving the model's classification performance and generalization ability. Test results on two SSVEP-BCI datasets validate the superior performance of the model.

[0116] Example 2

[0117] This embodiment also provides a method for verifying the high accuracy of the visual brain-computer interface signal decoding method described in Example 1, which specifically includes the following steps:

[0118] S1, select the EEG data of n subjects as the database, each EEG data corresponds to a target frequency, select one subject as the test subject, and the rest as other subjects;

[0119] S2, constructing a visual brain-computer interface signal decoding model, importing the other subject data into the visual brain-computer interface one by one for training to obtain a visual brain-computer pre-training model, wherein the visual brain-computer interface signal decoding model includes the visual brain-computer interface signal decoding architecture in Example 1 for implementing the decoding method;

[0120] S3, dividing the test subject data into a training set and a test set, and training the visual brain-computer pre-training model based on the training set to obtain a training model corresponding to the test subject;

[0121] S4, importing the test set into the training model to output the stimulation frequency of the test set;

[0122] S5, comparing the target frequency corresponding to the test subject data with the stimulation frequency, and calculating the EEG test accuracy rate based on the EEG accuracy number N. , output EEG test accuracy;

[0123] In S6, other subjects are selected one by one as test subjects, and S2 to S6 are repeated to calculate the average classification accuracy.

[0124] This embodiment provides the following specific implementation cases:

[0125] Regarding the data set, this embodiment prepares two sets of data sets, which are as follows:

[0126] ① Benchmark Dataset: This dataset collects 64-electrode EEG data from 35 healthy subjects at a sampling frequency of 1000 Hz. They participated in a BCI spelling experiment with 40 stimulus targets / frequencies. These 40 stimulus targets were encoded using a frequency-phase joint modulation method, with a frequency range of 8.0 Hz to 15.8 Hz, an increment of 0.2 Hz, and a phase difference of 0.5π between two adjacent stimulus signals. Each subject performed six blocks of experiments, each consisting of 40 trials corresponding to 40 stimulus targets. Each trial lasted 6 seconds, consisting of a 0.5-second target cue, a 5-second visual stimulus, and a 0.5-second rest period after the end of the experiment. This dataset was collected in an electromagnetically shielded laboratory.

[0127] ②BETA dataset (Benchmark dataset for event-based transaction analysis): The experiments in this dataset used the same experimental paradigm and target encoding method as the Benchmark dataset. The two datasets also had the same number of electrode channels and sampling rate. 70 healthy subjects participated in a BCI spelling experiment, which consisted of 4 blocks, each containing 40 experiments corresponding to 40 stimulation targets. Each experiment lasted 2 seconds for the first 15 subjects, and 3 seconds for the remaining subjects. A short break was provided between the two block experiments to avoid visual fatigue. This dataset was collected outside a laboratory without electromagnetic protection and has a relatively lower signal-to-noise ratio compared to the Benchmark dataset.

[0128] like Figure 2 As shown, the visual brain-computer interface signal decoding method in Example 1 is used to decode the above data set, and the steps are as follows:

[0129] 1. Data preprocessing: For a given dataset, the subject's continuous EEG data is subjected to channel extraction, downsampling rate, and single-experiment data interception.

[0130] 1.1. Electrode Channel Selection: From the numerous channels of raw EEG recordings, nine electrode channels located in the occipital region (Pz, PO5, PO3, POz, PO4, PO6, O1, Oz, and O2) were selected based on the physiological characteristics of EEG signal distribution on the scalp and the response patterns of SSVEP signals. Previous research and practice have demonstrated that these channels have high sensitivity and response relevance to SSVEP signals, effectively capturing changes in EEG activity associated with visual stimulation and providing a reliable data foundation for subsequent signal processing and analysis.

[0131] 1.2 Data downsampling: If the sampling frequency of the original EEG data is too high, it will not only increase the burden of data storage and processing, but may also introduce unnecessary noise and redundant information. Therefore, if the sampling frequency of the original EEG data is too high, it should be downsampled to around 250Hz to reduce the complexity of data processing;

[0132] 1.3. Time Segment Capture: Data was captured within a specific time window starting from the onset of the task prompt, i.e., the time period [0.64s, (0.64 + d)s]. 0.5s accounts for the time required for the subject's gaze to shift to the stimulus target, 0.14s is the latency of the visual system, and d is the data length used for frequency discrimination.

[0133] 2. Construct a sub-band filter bank: Design a filter bank consisting of three band-pass filters with frequency ranges of 6Hz to 90Hz, 14Hz to 90Hz, and 22Hz to 90Hz, respectively, to transform the EEG signal of a single experiment into three sub-band signals.

[0134] 2.1. Three bandpass filters are designed using 6th-order Butterworth infinite impulse response (IIR) filters with a maximum passband attenuation of 3 dB and a minimum stopband attenuation of 30 dB.

[0135] 3. Feature extraction: Construct a convolutional neural network consisting of three convolutional layers, which is shared by the three sub-band signals, extract signal features from each sub-band, and connect the three feature signals along the feature channel dimension. The convolutional neural network structure is as follows Figure 3 At a sampling rate of 250 Hz, taking EEG data with a length of 0.4s and 9 electrodes as an example, the parameters of the convolutional neural network are shown in Table 1.

[0136] Table 1 Convolutional neural network parameters for EEG data with a sampling rate of 250 Hz, a length of 0.4 s and 9 electrodes

[0137]

[0138] 3.1、The first convolutional layer consists of 60 kernels with a size of , the step length is , filled with "valid" filters, (In this embodiment ) is the number of electrodes of the EEG signal, which is used to combine the signals of different electrodes. The random inactivation rate of this convolutional layer is 0.05;

[0139] 3.2, the second convolutional layer consists of 60 kernels with a size of , the step length is , filled with "valid" filters, the complexity of the network is reduced by downsampling in the time dimension. The activation function of this layer is "Elu" and the random inactivation rate is 0.05;

[0140] 3.3, the third convolutional layer consists of 60 kernels with a size of , the step length is , filled with "same" filters, used to extract temporal features. The random inactivation rate of this layer is 0.95;

[0141] 3.4. Connect the three sub-band signals along the feature channel dimension through the output of the convolution module to form a feature map with 180 channels.

[0142] 4. Feature selection: Construct a time kernel selection network to automatically calculate the weights of the input feature signals, focus on important time positions and select key time features by weighting them. Taking two dilated convolution and grouped convolution branches as an example, the time kernel selection network structure is as follows: Figure 4 At a sampling rate of 250 Hz, taking EEG data with a length of 0.4 s and 9 electrodes as an example, the parameters of the temporal kernel selection network are shown in Table 2.

[0143] Table 2. Temporal kernel selection network parameters for EEG data with a length of 0.4 s and 9 electrodes at a sampling rate of 250 Hz.

[0144]

[0145] 4.1. In the concatenated feature map, the features at different positions of the feature map are calculated by dilated convolution and grouped convolution on the two branches of the temporal kernel selection network.

[0146] According to the dilated convolution and grouped convolution operations of the two branches: and , based on dilated convolution and grouped convolution to construct two convolutions with different expanded receptive fields to capture more comprehensive feature information. The relationship is as follows

[0147] ,

[0148] in, is the receptive field, d is the dilation rate, and k is the convolution kernel;

[0149] The size of the dilated convolution receptive field relative to the normal convolution receptive field of the same convolution kernel size can be expressed as follows

[0150]

[0151] 4.2. In order to obtain rich contextual information in the input features, two convolutions with different receptive fields are used for feature extraction as follows

[0152]

[0153] Where, is the input feature map, For branches The two dilated convolution parameter sets are selected as and , the size of the dilated convolution receptive field relative to the ordinary convolution receptive field of the same convolution kernel size is respectively and The group size is set to Each feature map is further divided into The convolutional layer , in order to combine the characteristic information of each group;

[0154] 4.3. Use a selection mechanism to temporally select features from convolution kernels of different scales. First, concatenate two temporal features along the channel dimension.

[0155]

[0156] Then, along the channel dimension Perform average pooling and max pooling , respectively extract Key features in and as follows

[0157]

[0158]

[0159] Second, concatenate these two feature maps into a two-channel feature map , through a kernel size of Convolution and Sigmoid function, transforming it into a time-selective mask .

[0160] 4.4, through a dot product operation and a kernel size of The convolution operation ( ), converting the features generated by convolution kernels of different scales into a Time-selective weight map of channels ( )

[0161]

[0162] Finally, the output of the temporal selection network is the dot product of the input features and the temporal selection weight map in the element direction .

[0163] 5. Build a classification module: Using the linear combination of cross entropy loss and center loss as the target loss function, design a classification module that includes a fully connected layer and a softmax function, and train and optimize the model parameters;

[0164] 5.1. The feature map output by the temporal kernel selection module is flattened into a one-dimensional feature vector. This vector is then transformed into class probabilities using a fully connected layer and a softmax function. The class label of the feature map is determined by selecting the class with the highest probability.

[0165] 5.2. Using Cross Entropy Loss and center loss The linear combination of

[0166]

[0167] In the formula is the regularization parameter (in this embodiment, ). Cross entropy loss The definition is as follows

[0168]

[0169] In the formula and are the true labels and the predicted labels, Number of classes. Center loss The definition is as follows

[0170]

[0171] In the formula is the first The feature vector of the sample, is the feature center of the class to which the sample belongs, which is updated when each batch of samples is trained.

[0172] 6. Model training and frequency recognition: First, use the training data of other subjects to train the visual brain-computer interface signal decoding model (hereinafter referred to as FBCNN-TKS) to obtain a visual brain-computer pre-training model. Then use the training data of the test subject to train the visual brain-computer pre-training model, and use the trained training model corresponding to the test subject to classify the test data, and determine the frequency of the test signal based on the probability of the Softmax output.

[0173] It should be noted that in this embodiment, the Adam optimizer was selected for FBCNN-TKS training, with an initial learning rate of 0.0001. The SGD optimizer was used to optimize the center loss to encourage similar features to cluster toward the center of the class. The random dropout rates of the first, second, and third convolutional layers were set to 0.05, 0.05, and 0.95, respectively.

[0174] It should be noted that in this embodiment, the training process of FBCNN-TKS is divided into two stages. In the first stage, the subjects in a dataset are divided into test subjects and training subjects using the leave-one-out method. That is, any subject is used as a test subject once, and the other subjects are used as training subjects. Their data is used as training data to train a pre-trained model for each test subject.

[0175] It should be noted that in the second stage, the leave-one-out method is used to divide the data of each test subject into training data and test data, that is, each experimental block is used as test data once, and the remaining experimental blocks are used as training data to fine-tune the training model corresponding to the test subject and obtain the visual brain-computer pre-training model specific to the test subject. ;

[0176] It should be noted that, in this embodiment, the visual brain-computer pre-training model specific to the test subject is used. , use the neural network classification function to classify the test data of the test subjects and obtain the first Test experiments The labels (i.e. stimulation frequencies) are as follows

[0177]

[0178] (7) Performance evaluation: The performance of the BCI system based on SSVEP is usually evaluated using two indicators: classification accuracy (P) and information transfer rate (ITR). Classification accuracy P is defined as the ratio of the number of correctly classified trials to the total number of trials; while the information transfer rate ITR is defined as follows:

[0179]

[0180] In the formula is the number of classes, is the average time of a single trial, including visual attention and gaze shift time between two trials (for both datasets, the gaze shift time is 0.5s).

[0181] It is worth noting that the classification accuracy and ITR are the average accuracy and ITR of all subjects and all experimental blocks in a dataset. To objectively evaluate the performance of FBCNN-TKS, we compared it with three baseline methods: a traditional machine learning (ML) method eTRCA, a classic deep learning network (DL) Conv-CA, and a DL network deep neural network (DNN).

[0182] 1) Impact of data length: Figure 5 The relationship between the classification accuracy (accuracy) and information transmission rate (ITR) achieved by the four methods on the Benchmark dataset and the five data lengths (0.2, 0.4, 0.6, 0.8, and 1.0) is shown. Figure 5 (a) shows the relationship between classification accuracy (i.e., accuracy) and the length of the five data. Figure 5 (b) shows the relationship between the information transfer rate (ITR) and the length of the five data points. The "I"-shaped bars corresponding to each data point in the figure represent the error bars of each data point, indicating the standard error of each data point. The number of asterisks * corresponds to the p-value of the significant difference obtained by the paired sample t-test between FBCNN-TKS and DNN (i.e., the p-value obtained by the paired sample t-test between FBCNN-TKS and DNN): . As can be seen from the figure, FBCNN-TKS achieved the highest classification accuracy and ITR at all data lengths. In addition, at all data lengths, the accuracy and ITR of FBCNN-TKS were significantly better than the best baseline method DNN (p<0.05). As can be seen from the figure, the classification accuracy of the four methods increased with the increase of data length, but the ITR first increased and then decreased with the increase of data length, and the highest ITR was achieved at 0.4 seconds. At a data length of 0.4 seconds, the accuracy and ITR achieved by FBCNN-TKS were 83.10% and 251.54 bits / minute, respectively, which were significantly higher than the accuracy of 79.89% and ITR of 235.66 bits / minute achieved by DNN.

[0183] Figure 6 The relationship between the classification accuracy and ITR of the four methods on the BETA dataset and the five data lengths is shown. Figure 6 (a) shows the relationship between classification accuracy (i.e., accuracy) and the length of the five data. Figure 6 (b) shows the information transfer rate (ITR) as a function of the five data lengths. As can be seen from the figure, the accuracy and ITR achieved by the four methods follow the same pattern as those achieved on the benchmark dataset. FBCNN-TKS also achieved the highest classification accuracy and ITR for all data lengths. The difference is that on the BETA dataset, the difference in classification accuracy and ITR between FBCNN-TKS and DNN is even more significant for all data lengths except 0.6s. At a data length of 0.4 seconds, FBCNN-TKS achieved an accuracy of 72.98% and an ITR of 203.47 bits / minute, respectively, significantly exceeding the DNN's accuracy of 67.51% and ITR of 179.67 bits / minute.

[0184] 2) Impact of Training Data Size: The size of the training data has a significant impact on the decoding accuracy of the DL network. Therefore, we conducted classification experiments on FBCNN-TKS with different numbers of training blocks. We used the leave-one-out method to select each block from all blocks as a test block. From the remaining blocks, we sequentially selected different numbers of blocks as training blocks. The classification results for a data length of 0.4 seconds are shown in Table 3. As can be observed from the table, the accuracy of FBCNN-TKS continues to improve with the increase in training blocks, but the rate of improvement gradually slows down. However, FBCNN-TKS can also achieve acceptable classification accuracy (70%) when using one training block for the Benchmark dataset and two training blocks for the BETA dataset.

[0185] Table 3. Classification accuracy and standard error (mean ± standard error, %) achieved by FBCNN-TKS using different numbers of training blocks.

[0186]

[0187] 3) Model Complexity: To evaluate the computational complexity of FBCNN-TKS, we compared it with DNN in terms of model parameters, training time, and testing time. The training time for the first and second phases represents the time required for pre-training and fine-tuning on a single subject, while the testing time represents the time required to test a single subject. During evaluation, the data length and number of electrode channels for both datasets were set to 0.4s and 9, respectively. The batch sizes for Benchmark and BETA were set to 100 and 120, respectively, and the training epochs were set to 1000. The results are shown in Table 4. As can be seen, despite extracting more channel feature maps and additionally using a kernel selection mechanism to select temporal information at key locations, the total number of parameters in FBCNN-TKS does not increase significantly compared to DNN. This is due to the use of shared CNN modules in FBCNN-TKS and the use of dilated and grouped convolutions in the TKS module, which significantly reduce model parameters. There is little difference in fine-tuning training and testing time between the two models. Compared to DNN, FBCNN-TKS has a longer pre-training time. In practical applications, the pre-training stage can usually be completed offline, and the global model trained with data from other subjects can be used to replace the current model, avoiding excessively long pre-training time and thus improving the deployment efficiency of the model.

[0188] Table 4. Training (first and second phase) and testing time (seconds) of FBCNN-TKS and DNN

[0189]

[0190] These results fully demonstrate that the method provided by the present invention for decoding visual BCI signals by fusing sub-band convolutional neural networks with a temporal kernel selection mechanism is extremely effective and helps promote the practical application of visual BCI systems.

[0191] Example 3

[0192] This embodiment provides a visual brain-computer interface signal decoding system, which includes:

[0193] A first building block is used to build a filter bank for a visual brain-computer interface, wherein the filter bank includes a plurality of bandpass filters;

[0194] an acquisition module, configured to acquire EEG data and transform the EEG data into a plurality of sub-band signals according to the filter bank;

[0195] A second building block is used to build a convolutional neural network, wherein the convolutional neural network includes multiple convolutional layers;

[0196] An extraction module is used to extract signal features corresponding to the sub-band signals under each convolutional layer according to a convolutional neural network, and to concatenate the signal features along the channel to form an aggregated feature map;

[0197] A third construction module is used to construct a temporal kernel selection network, and calculate weight values ​​for the spliced ​​feature maps according to the temporal kernel selection network, wherein the temporal kernel selection network includes dilated convolution and grouped convolution;

[0198] A calculation module, configured to weight the concatenated feature maps according to the calculated weight values ​​to obtain weighted feature maps;

[0199] A fourth building module is used to build a classification module, wherein the classification module includes a fully connected layer and a Softmax function;

[0200] a flattening module, configured to flatten the weighted feature map according to the fully connected layer to obtain a feature map of a one-dimensional feature vector;

[0201] The output module is used to classify the one-dimensional feature vector according to the Softmax function in the classification module to output the stimulation frequency of the EEG data.

[0202] A visual brain-computer interface signal decoding system in an embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), etc., which is not specifically limited in the embodiment of the present application.

[0203] In the embodiment of the present application, a visual brain-computer interface signal decoding system can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0204] The visual brain-computer interface signal decoding system provided in the embodiment of the present application can realize Figure 1The various processes of the method embodiment of a visual brain-computer interface signal decoding method are implemented, and will not be repeated here to avoid repetition.

[0205] A visual brain-computer interface signal decoding system according to an embodiment of the present invention integrates a filter bank, a convolutional neural network, and a temporal kernel selection network. The present invention uses a filter bank to capture frequency domain information from different frequency bands, extracts temporal features from the frequency and spatial domains based on a convolutional neural network, and optimizes feature extraction by selecting a temporal kernel using a temporal kernel selection network. The visual brain-computer interface signal decoding method of the present invention accurately captures the characteristic patterns of steady-state visual evoked potential signals, providing a richer feature representation than other deep learning-based models. Specifically, the temporal kernel selection network incorporates dilated and grouped convolutions, expanding the receptive field of the decoding method and reducing the model's computational complexity. Furthermore, the temporal kernel selection network extracts temporal features important for steady-state visual evoked potential frequency recognition. By expanding the receptive field of the convolution kernel, task-relevant feature patterns are emphasized, significantly improving the model's classification performance and generalization ability. The present invention uses a linear combination of cross-entropy loss and center loss as the objective function for model optimization, which not only enhances the compactness of intra-class feature signals but also increases the distance between inter-class feature signals, further improving the model's classification performance and generalization ability. The test results on two SSVEP-BCI datasets verify the superior performance of the model.

[0206] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, each process of the above-mentioned visual brain-computer interface signal decoding method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0207] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned visual brain-computer interface signal decoding method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0208] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0209] Example 4

[0210] This embodiment provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the visual brain-computer interface signal decoding method as described above.

[0211] Example 5

[0212] This embodiment provides a computer storage medium, wherein the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the visual brain-computer interface signal decoding method as described above.

[0213] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as a limitation to the invention.

[0214] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0215] Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present embodiment application. The appearance of this phrase in various positions in the specification does not necessarily mean that they are all the same embodiments, nor are they independent or alternative embodiments that are mutually exclusive with other embodiments. It can be understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0216] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A visual brain-computer interface signal decoding method, characterized in that: include: S101, constructing a filter bank for a visual brain-computer interface, wherein the filter bank includes a plurality of bandpass filters; S102, acquiring EEG data, and transforming the EEG data into a plurality of sub-band signals according to the filter bank; S103, constructing a convolutional neural network, wherein the convolutional neural network includes multiple convolutional layers; S104, extracting signal features corresponding to the subband signals at each convolutional layer according to the convolutional neural network, and concatenating the signal features along the channels to form an aggregated feature map; S105: construct a temporal kernel selection network, and calculate weight values ​​for the spliced ​​feature maps according to the temporal kernel selection network, wherein the temporal kernel selection network includes dilated convolution and grouped convolution, specifically including: In the concatenated feature map, the features at different positions of the feature map are calculated respectively through the dilated convolution and grouped convolution on the two branches of the temporal kernel selection network; According to the dilated convolution and grouped convolution operations of the two branches: and , based on dilated convolution and grouped convolution to construct two convolutions with different expanded receptive fields, the expanded receptive fields The relationship is as follows: ， in, is the receptive field, d is the dilation rate, and k is the convolution kernel; The size of the dilated convolution receptive field relative to the normal convolution receptive field of the same convolution kernel size is expressed as follows: ； Feature extraction is performed based on two convolutions with different receptive fields to obtain information in the input feature map. , as follows ， In the formula is the input feature map, For branches dilated convolution and grouped convolution operations; Temporally select features from convolution kernels of different scales to obtain temporally selective masks ; Perform dot product calculation based on the temporal selection mask, and obtain a temporal selection weight map with N channels based on the dot product calculation and a convolution operation with a kernel size of (1,1). , ； Where, is the dot product calculation, is the convolution calculation, is the i-th channel, N is the number of channels; S106, weighting the spliced ​​feature map according to the calculated weight value to obtain a weighted feature map; S107, flattening the weighted feature map to obtain a one-dimensional feature vector; S108, constructing a classification module, wherein the classification module includes a fully connected layer and a normalized exponential function; S109 , mapping and classifying the one-dimensional feature vector according to the fully connected layer and the normalized exponential function in the classification module to output the stimulation frequency of the EEG data.

2. The visual brain-computer interface signal decoding method according to claim 1, characterized in that: Specifically, in S106, the input feature map is output. and time selection weight graph The dot product in the element direction, that is , to obtain the weighted feature map.

3. The visual brain-computer interface signal decoding method according to claim 1, characterized in that: The convolutional layer includes: The first convolutional layer consists of 60 kernels of size , the step length is , filled with "valid" filters, where is the number of electrodes of the EEG signal, used to combine the signals of different electrodes; The second convolutional layer consists of 60 kernels of size , the step length is , filled with "valid" filters, which are used to reduce the complexity of the network by sampling in the time dimension; The third convolutional layer consists of 60 kernels with a size of , the step length is , filled with "same" filters, used to extract temporal features.

4. The visual brain-computer interface signal decoding method according to claim 1, characterized in that: The filter bank includes three band-pass filters, each with a frequency range of 6 Hz to 90 Hz, 14 Hz to 90 Hz, and 22 Hz to 90 Hz, and is used to transform single EEG data into three sub-band signals.

5. The visual brain-computer interface signal decoding method according to claim 1, characterized in that: The bandpass filter is designed using a sixth-order Butterworth infinite impulse response filter, with a maximum passband attenuation of 3 dB and a minimum stopband attenuation of 30 dB.

6. A visual brain-computer interface signal decoding system, characterized in that: Decoding is performed using the method according to any one of claims 1 to 5, comprising: A first building block is used to build a filter bank for a visual brain-computer interface, wherein the filter bank includes a plurality of bandpass filters; an acquisition module, configured to acquire EEG data and transform the EEG data into a plurality of sub-band signals according to the filter bank; A second building block is used to build a convolutional neural network, wherein the convolutional neural network includes multiple convolutional layers; An extraction module is used to extract signal features corresponding to the sub-band signals under each convolutional layer according to a convolutional neural network, and to concatenate the signal features along the channel to form an aggregated feature map; The third construction module is used to construct a temporal kernel selection network, and calculate the weight value of the spliced ​​feature map according to the temporal kernel selection network, wherein the temporal kernel selection network includes dilated convolution and grouped convolution, specifically including: In the concatenated feature map, the features at different positions of the feature map are calculated respectively through the dilated convolution and grouped convolution on the two branches of the temporal kernel selection network; According to the dilated convolution and grouped convolution operations of the two branches: and , based on dilated convolution and grouped convolution to construct two convolutions with different expanded receptive fields, the expanded receptive fields The relationship is as follows: ， in, is the receptive field, d is the dilation rate, and k is the convolution kernel; The size of the dilated convolution receptive field relative to the normal convolution receptive field of the same convolution kernel size is expressed as follows: ； Feature extraction is performed based on two convolutions with different receptive fields to obtain information in the input feature map. , as follows ， In the formula is the input feature map, For branches dilated convolution and grouped convolution operations; Temporally select features from convolution kernels of different scales to obtain temporally selective masks ; Perform dot product calculation based on the temporal selection mask, and obtain a temporal selection weight map with N channels based on the dot product calculation and a convolution operation with a kernel size of (1,1). , ； Where, is the dot product calculation, is the convolution calculation, is the i-th channel, N is the number of channels; A calculation module, configured to weight the concatenated feature maps according to the calculated weight values ​​to obtain weighted feature maps; a fourth building module, configured to build a classification module, wherein the classification module includes a fully connected layer and a normalized exponential function; a flattening module, configured to flatten the weighted feature map according to the fully connected layer to obtain a feature map of a one-dimensional feature vector; An output module is used to classify the one-dimensional feature vector according to the normalized exponential function in the classification module to output the stimulation frequency of the EEG data.

7. A computer device, characterized in that: It includes a memory and a processor, the memory stores a computer program, and the processor implements the visual brain-computer interface signal decoding method as described in any one of claims 1 to 5 when executing the computer program.

Images