Eeg signal decoding method based on spatio-temporal convolution and residual connection

Abstract

The application discloses a brain electrical signal decoding method based on space-time convolution and residual connection, and comprises the following steps: step 1, constructing a visual evoked brain electrical signal data set, collecting the space-time characteristics of the brain electrical signal, and collecting the brain electrical signal generated by visual stimulation of a subject; step 2, constructing a brain electrical signal decoding classification network CR-model based on space-time convolution and residual connection, collecting the space-time characteristics of the brain electrical signal in step 1, and performing deep fusion extraction on the space-time characteristics, so that the high time resolution of the brain electrical signal and the spatial connection embodied by the multi-channel signals are fully utilized, and the classification accuracy of the visual stimulation brain electrical signal is improved. The application is used for solving the problems that the existing method cannot fully utilize the spatial information between the multi-channel signals of the brain electrical signal and the connection between the time information and the spatial information is not close.

Description

Technical Field

[0001] This invention belongs to the field of EEG signal decoding technology, specifically relating to an EEG signal decoding method based on spatiotemporal convolution and residual connections. Background Technology

[0002] Cognitive neuroscience research is an important part of bio-inspired intelligence research, and decoding brain representations and deciphering visual cognitive processes are long-term goals pursued in cognitive neuroscience research. Using appropriate biomedical techniques to observe and record brain structure and function, and utilizing non-invasive brain-computer interface technology to decode and identify brain ideologies, reveal cognitive processes, and propose strategies and methods to improve work and life efficiency are currently hot research directions in neurocomputational science and bioinformatics.

[0003] Early research on brain-computer interfaces explored the feasibility of decoding brain activity from electroencephalogram (EEG) signals. Currently, much research focuses on exploring new methods for recognizing thought patterns. However, achieving self-recognition without categorization requires transforming thought patterns into a captureable form. While reconstructing thought patterns is extremely difficult, basic artificial intelligence models can be built using neuroimaging techniques and deep learning to reconstruct simple images. Currently, two commonly used techniques in theoretical research and experimental practice are functional magnetic resonance imaging (fMRI) and electroencephalography (EEG). FMRI offers high spatial resolution and can provide detailed information about different brain regions, but it requires expensive equipment and significant computational resources. This has led many researchers to focus on the relatively cheaper and simpler EEG methods. EEG, as a non-invasive technique that allows direct observation of brain electrical signals without external stimulation, has been widely used in clinical diagnosis and treatment.

[0004] With the development of computer science, deep learning-based EEG signal classification technology has been widely applied. Compared to traditional brain-computer interface systems that rely on manual feature selection, it significantly reduces the dependence on human resources. Furthermore, deep learning models possess stronger adaptability and generalization capabilities, making EEG signal processing more efficient and accurate. In particular, convolutional neural networks (CNNs), with their local perception and weight sharing characteristics, exhibit strong spatial feature extraction capabilities when processing EEG signals. Through multi-layer convolution and pooling operations, CNNs can capture relevant features in EEG signals at multiple scales. Residual networks, by introducing residual connections, solve the gradient vanishing and gradient exploding problems encountered by traditional neural networks when training deep models, and can more effectively capture deep and high-level features in EEG signals, thereby improving the accuracy of classification and prediction.

[0005] Patent application CN115828059A, entitled "A Method for Decoding Motor Imagery EEG Based on Spatial-Frequency Image Feature Extraction and Classification," discloses a method for decoding EEG information based on Fast Fourier Transform (FFT) and Convolutional Neural Network (CNN). This method first uses a Butterworth filter to filter the raw EEG signal to a target frequency band of 8-30 Hz. Then, it uses FFT to transform the signal from the time domain to the frequency domain. Next, it concatenates the multi-channel EEG signals to generate a spatial-frequency energy map. This spatial-frequency image is then used as input to a CNN to obtain the final signal classification result. Alternatively, this method uses FFT to transform the signal from the time domain to the frequency domain, concatenates the selected channels, and simultaneously represents the spatial and frequency domain features of the signal as an image. The generated spatial-frequency image is then input into a CNN for feature extraction and classification. This method utilizes the spatial and frequency domain information of EEG signals for EEG decoding and demonstrates good accuracy. However, the method has the drawback of ignoring the temporal information of EEG signals and failing to consider the relationship between temporal and spatial information, resulting in insufficient utilization of the temporal information of EEG signals and a weak connection between temporal and spatial information.

[0006] Existing methods fail to fully utilize the spatial information between multiple channels of EEG signals and do not closely link the spatiotemporal information of EEG signals, resulting in decreased classification accuracy and poor generalizability of existing feature extraction and classification models. Summary of the Invention

[0007] To overcome the problems existing in the prior art, the present invention aims to provide a brain signal decoding method (CR-Model) based on spatiotemporal convolution and residual connections. It utilizes temporal convolutional networks and spatial convolutional networks to capture the temporal and spatial features of brain signals, respectively, and uses residual connection networks to extract deep features and improve the performance and stability of the model. This addresses the problems of insufficient utilization of spatial information between multi-channel signals of brain signals and weak connection between temporal and spatial information in existing methods.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for decoding EEG signals based on spatiotemporal convolution and residual connections includes the following steps;

[0010] Step 1: Construct a visually evoked EEG signal dataset, collect the spatiotemporal characteristics of EEG signals, and use them to stimulate subjects' vision to generate EEG signals.

[0011] Step 2: Construct the EEG signal decoding and classification network CR-model based on spatiotemporal convolution and residual connections to extract the spatiotemporal features of the EEG signals collected in Step 1, and perform deep fusion extraction of the spatiotemporal features to make full use of the high temporal resolution of EEG signals and the spatial connections reflected by multiple channels to improve the classification accuracy of visual stimulus EEG signals.

[0012] Step 1 specifically involves:

[0013] Step 1.1, Stimulus image selection;

[0014] The EEG signal data collected in the experiment were event-related electrical signals induced by object images as stimulus sources. Images with prominent subjects and minimal background interference were selected as visual stimuli to highlight different salient targets and were used for the visual observation of the subjects in step 1.2 to induce the subjects to produce different visual stimuli. The selected visual stimuli were derived from different image categories in the ImageNet image dataset.

[0015] Step 1.2, Experimental Paradigm Design;

[0016] During the recording of EEG signals, each type of image is displayed one by one in a trial;

[0017] Step 1.3, EEG signal acquisition;

[0018] Subjects observed the stimulus images selected in step 1.1 according to the experimental paradigm of step 1.2, and the resulting visual stimulus EEG signals were collected and processed using the NeuroScan EEG signal acquisition system.

[0019] Step 1.4: Construct an EEG signal dataset;

[0020] The processed EEG dataset was divided into training, validation, and test sets, and the EEG signals of the same image corresponding to all subjects were placed in the same set.

[0021] In step 1.2, after a trial display ends, the screen displays a black background as a rest period to reduce the interference of the previous trial on the current trial.

[0022] In step 1.3, the EEG signal acquisition system includes a Quik-Cap 128-channel electrode cap, a SynAmps 2128-channel EEG signal amplifier, and the EEG signal acquisition and analysis software Curry 7.

[0023] The Quik-Cap 128-channel electrode cap is used to detect the subject's EEG trigger signal, the SynAmps 2128-channel EEG signal amplifier amplifies the EEG signal, and the EEG signal acquisition and analysis software Curry 7 is used to collect the subject's EEG signal and perform subsequent processing.

[0024] The distribution of the 128-channel EEG signal acquisition system adopts the international 10-20 system, where 10 and 20 refer to the actual distance between adjacent electrodes being 10% or 20% of the total distance between the front and back or left and right sides of the skull. All EEG signals are referenced to the FCz electrode. The experimental paradigm design software used in the experiment is the psychological design software E-Prime.

[0025] In step 1.4, the sampling frequency of the EEG device is 1000Hz.

[0026] Step 2 specifically involves:

[0027] Step 2.1, construct the temporal convolution module;

[0028] The size of the preprocessed EEG signal in step 1 is 128×440 when inputting into the temporal convolution block (where 128 is the number of EEG signal channels and 440 is the number of sampling points in the time dimension); in the temporal convolution block, one-dimensional convolution is applied independently to each channel to process the input signal along the time dimension; temporal feature extraction is performed.

[0029] Step 2.2, construct the spatial convolution module;

[0030] After extracting the temporal features, the signal is input into the spatial convolution module to perform channel-level spatial analysis. By applying one-dimensional convolution in the channel dimension, the correlation between different channels is found within the corresponding time interval, which can decode and extract the spatial feature information of the EEG signal.

[0031] Step 2.3, construct the residual module;

[0032] After decoding and extracting the corresponding spatiotemporal features in the time and space dimensions, the CR-Model connects the obtained spatiotemporal features to the residual network. The final residual block, composed of a set of residual connection layers, performs two-dimensional convolution on the decoded spatiotemporal features to extract deeper relationships and features from the signal. Before adding the input to the residual, each residual layer performs two convolutions (with batch normalization and ReLU activation). Then, the output is provided to the final convolutional layer. Finally, it is processed using a fully connected layer with the same size as the joint embedding dimension.

[0033] In step 2.1, the temporal convolution block includes a set of one-dimensional convolutions with different dilation coefficients, and then their outputs are mapped and concatenated to obtain an output tensor. Each element of this tensor does not have a temporal correspondence with a single sample in the original signal, but by adjusting the kernel size and dilation, it can decode and extract information from a specific time span in the EEG signal.

[0034] In step 2.2, the spatial convolution module further includes multiple one-dimensional convolutional layers whose outputs are concatenated to make the "rows" of the channel continuously appended to the signal matrix; then, each spatial one-dimensional convolution is operated with a different kernel size, and batch normalization and ReLU activation are performed after all convolutional layers in the temporal and spatial blocks.

[0035] The beneficial effects of this invention are:

[0036] This invention utilizes convolutional neural networks and spatiotemporal feature analysis methods. It captures the temporal and spatial features of EEG signals through temporal and spatial convolutional networks, respectively, and employs residual connection networks to extract depth features. In accurately decoding visual EEG signals, it achieves high classification accuracy and demonstrates good model generalization ability. By fully integrating the spatiotemporal information of EEG signals and utilizing residual connection networks to extract depth features and improve model performance and stability, this invention effectively extracts depth visual features from EEG signals and exhibits good model generalization ability. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the process of the present invention.

[0038] Figure 2 This is a network model diagram of the present invention.

[0039] Figure 3 This is a schematic diagram of the EEG signal acquisition paradigm.

[0040] Figure 4 This is a schematic diagram of a 128-channel electrode cap.

[0041] Figure 5 This diagram illustrates the accuracy of each model in classification tasks of 2-40 classes. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings.

[0043] This invention employs a feature extraction network based on a spatiotemporal convolutional network and a residual connection network. The feature extraction network model based on the spatiotemporal convolutional network and the residual connection network includes: a temporal convolutional module, a spatial convolutional module, a residual module, and a softmax classification module. The following section, in conjunction with the appendix... Figure 1 and attached Figure 2 The present invention will now be described in further detail.

[0044] See attached document Figure 1 The present invention includes the following steps:

[0045] Step 1: Construct a dataset of visually evoked EEG signals.

[0046] Step 1.1, Stimulus image selection.

[0047] The EEG data collected in the experiment were event-related electrical signals induced by object images as stimuli. Images with prominent subjects and minimal background interference were selected as visual stimuli to highlight different salient targets and elicit different visual stimuli from the participants. The final visual stimuli were derived from 40 image categories in the ImageNet image dataset, whose numbers and Chinese names are shown in Table 1.1.

[0048] Table 1.1 Stimulus Image Numbers and Chinese Names

[0049] 0 German Shepherd 10 canoe 20 golf 30 parachute 1 Egyptian cat 11 cell phone 21 piano 31 Billiards Case 2 Grey butterfly 12 mug 22 soldering iron 32 radio 3 horse 13 convertible 23 Pumpkin lantern 33 SLR camera 4 Capuchin monkey 14 computer 24 messenger bag 34 gun 5 elephant 15 Electronic watch 25 missile 35 shoe 6 panda 16 electric guitar 26 Gloves 36 banana 7 Anemonefish 17 train 27 Mountain bike 37 pizza 8 airplane 18 coffee machine 28 tent 38 daisy 9 broom 19 Folding chair 29 pajamas 39 mushroom

[0050] Each category contained 50 images, for a total of 2000 images. All 2000 images were stripped of black borders and converted to a 500×500 resolution. The experiment involved six participants: one female and five males. All participants were in good health, without any known medical conditions or risk factors, and were well-educated with normal cognitive and judgment abilities. The signal acquisition process adhered to scientific research standards. In this experiment, the six participants viewed 40 categories of images, with 50 images in each category, for a total of approximately 12,000 images viewed.

[0051] Step 1.2, Experimental Paradigm Design.

[0052] EEG signal acquisition paradigms such as Figure 3 As shown. During the recording of EEG signals, each category of images was displayed one by one in a trial, with each image presented for 0.5 seconds, for a total of 25 seconds per trial. After each trial, the screen displayed a black background for 10 seconds as a rest period to reduce interference from the previous trial. The experiment included 40 categories, therefore, the EEG recording time for each subject was up to 1400 seconds.

[0053] Step 1.3, EEG signal acquisition.

[0054] This experiment used a 128-channel EEG signal acquisition system called NeuroScan to collect and process EEG signals generated by visual stimuli. The system includes... Figure 4The Quik-Cap 128-channel electrode cap, SynAmps2 128-channel EEG signal amplifier, and Curry 7 EEG signal acquisition and analysis software shown were distributed according to the international 10-20 system, where 10 and 20 refer to the actual distance between adjacent electrodes being 10% or 20% of the total anterior-posterior or lateral distance of the skull. All EEG signals were referenced to the FCz electrode, and the experimental paradigm design software used was the psychological design software E-Prime.

[0055] Before conducting the EEG signal acquisition experiment, to ensure the quality of the experimental data, subjects need to be in good physiological condition. Therefore, subjects are required to get sufficient rest the day before the experiment to maintain high energy levels and ensure they can concentrate during the experiment, minimizing the negative effects of blinking and drowsiness. Simultaneously, subjects need to clean their scalps to ensure that the electrode impedance remains below 5kΩ after the electrolyte injection. Researchers also ensure that subjects wear the EEG cap correctly, aligning its midline with the center of the subject's brain to collect EEG signals from the correct location. Care must also be taken during electrolyte injection to avoid injecting too much electrolyte at once, which could cause the electrodes to stick together and affect the experimental results.

[0056] In experiments involving the acquisition of electroencephalogram (EEG) signals, participants were required to choose a comfortable position and remain as still as possible to avoid interference from motion signals. Participants needed to maintain an appropriate distance from the screen, approximately four times the height of the stimulus image. The experiment took place in a well-lit, soundproof, and isolated room to minimize external interference. Researchers monitored the participants in real-time from an adjacent room, promptly addressing any interference or problems encountered.

[0057] Step 1.4: Construct an EEG signal dataset.

[0058] The EEG sampling frequency is 1000Hz. Theoretically, each image can correspond to 500 sampling points. However, due to factors related to the acquisition equipment and processing software, the data lengths obtained are not uniform. Therefore, to unify the data format and eliminate interference from bad samples, it is necessary to exclude sampling sequences with fewer than 480 and more than 500 sampling points, and to add sampling points with an amplitude of 0 to sequences with a length of 480-500 to unify them into a 500 sampling sequence length. To eliminate the influence of the previous image sequence on the current image sequence, the first 20 and last 40 sampling points of the EEG signal corresponding to each image are discarded, and 440 sampling points recorded in 20-460ms are used as valid EEG data for the experiment. To effectively avoid overfitting when training the algorithm using the processed EEG dataset, and to verify the generalization ability of the algorithm and conduct final performance evaluation, the processed EEG dataset is divided into training, validation, and test sets in a 4:1:1 ratio, and the EEG signals of the same image corresponding to all subjects are placed in the same set.

[0059] Step 2: Construct an EEG signal decoding and classification network based on spatiotemporal convolution and residual connections.

[0060] Step 2.1, construct the temporal convolution module.

[0061] The two main purposes of the temporal convolution block are to extract important features and reduce the size of the input signal. After preprocessing, the EEG signal is input into the temporal convolution block with a size of 128×440 (where 128 is the number of EEG signal channels and 440 is the number of sampling points in the time dimension). In the temporal convolution block, one-dimensional convolution is applied independently to each channel to process the input signal along the time dimension. To obtain relevant features at multiple time scales, the temporal block includes a set of one-dimensional convolutions with different dilation coefficients. Their outputs are then mapped and concatenated to obtain an output tensor. Each element of this tensor does not have a temporal correspondence with a single sample in the original signal, but by adjusting the kernel size and dilation, it can extract information over a specific time span. The structure of the temporal convolution module is shown in Table 2.1.

[0062] Table 2.1 Temporal Convolutional Block Structure

[0063]

[0064]

[0065] Step 2.2, construct the spatial convolution module.

[0066] The spatial convolutional block aims to extract effective spatial feature information by applying one-dimensional convolutions along the channel dimension to find the correlation between different channels within a corresponding time interval. Similar to the temporal convolutional module, the spatial convolutional module also includes multiple one-dimensional convolutional layers whose outputs are concatenated. In this case, the channel dimensions are ordered so that the "rows" of the channels are continuously appended to the signal matrix. Then, each spatial one-dimensional convolution operates with a different kernel size. Batch normalization and ReLU activation are performed after all convolutional layers in the temporal and spatial blocks. The spatial convolutional module structure is shown in Table 2.2.

[0067] Table 2.2 Spatial Convolution Block Structure

[0068]

[0069] Step 2.3: Construct the residual module.

[0070] After extracting the corresponding spatiotemporal features in the time and space dimensions, the model performs two-dimensional convolution on the extracted spatiotemporal features using a final residual block composed of a set of residual connected layers to extract deeper relationships and features from the signal. Each residual layer performs two convolutions (with batch normalization and ReLU activation) before adding the input to the residuals. The output is then fed to the final convolutional layer. Finally, a fully connected layer with the same size as the joint embedding dimension is used for processing. The residual connected module structure is shown in Table 2.3.

[0071] Table 2.3 Residual Connector Block Structure

[0072]

[0073]

[0074] By appropriately adding a Softmax layer after the fully connected layer of the encoder, the proposed encoder was subjected to EEG signal classification tests to understand its ability to decode visual information from neural data.

[0075] Step 3: Comparative analysis of experimental results.

[0076] Step 3.1: Comparison of classification results between this method and other methods.

[0077] The experiment used the evoked stimulus EEG signal dataset proposed in step 1. The hardware configuration used in the experiment was as follows: the graphics processors were NVIDIA GeForce RTX 2080Ti and NVIDIA TITAN RTX.

[0078] In the brain-visual data classification task, several publicly available EEG signal classification models are first evaluated using the entire dataset and the complete EEG signal time series (i.e., 20-460ms). The comparison method is described below:

[0079] (1)LSTM: Only the classic long short-term memory network is used as the EEG signal encoder to learn EEG signal features, and Softmax is used as the classifier to predict labels.

[0080] (2) EEGNet: It is a fully convolutional network that is widely used in various tasks related to EEG signals;

[0081] (3)CNN-VGG16: Visual feature representation of EEG signals based on VGG16 to guide EEG signal classification experiments;

[0082] (4) SyncNet: An improvement on convolutional neural networks, which learns the interaction between EEG signal data at different time steps through convolution operations;

[0083] (5) CRAM: It combines convolutional networks and recurrent networks. First, the EEG signal is divided into slices along the time domain. The slices are encoded using a convolutional neural network. Then, a recurrent network based on attention mechanism is used to further explore the temporal dynamics between different slices.

[0084] Meanwhile, the data was processed using bandpass filters with different cutoff frequencies, including high-frequency gamma-band filters (55-95Hz), beta-band to mid-frequency gamma-band filters (14-70Hz), and full-frequency filters (5-95Hz), and the classification performance was tested within these frequency ranges. The experimental results are shown in Table 3.1.

[0085] Table 3.1 Comparison of classification results between the method proposed in this invention and other methods

[0086]

[0087]

[0088] Analyzing the results in Table 3.1, the EEG signal classification model proposed in this invention, which combines spatiotemporal convolution and residual connection networks for spatiotemporal feature analysis, shows a certain degree of improvement in classification accuracy compared to the listed EEG signal classification methods. This fully demonstrates that the proposed method can effectively decode vision-related human brain activity. Further analysis of the impact of different bandpass filters on EEG signal processing in Table 3.1 reveals that the EEG signals processed by the high-frequency gamma band filter exhibit higher classification accuracy. This is because the gamma band contains richer cognitive information and filters out low-frequency noise interference.

[0089] Step 3.2, Classification accuracy of CR-Net for EEG signal sampling segments of different durations.

[0090] Using EEG signal segments with different sampling durations as input, the classification results are shown in Table 3.2.

[0091] Table 3.2 Classification accuracy of EEG signal sampling segments of different durations

[0092]

[0093] As shown in Table 3.2, inputting the entire EEG signal segment yields the best classification performance. Ignoring any part will affect the classification result. This intuitive rule is also true for other models.

[0094] Step 3.3: Compare the classification accuracy of each subject and each model.

[0095] In the case of the best classification performance, the impact of different subjects and different models on classification accuracy was further evaluated under time intervals (20-460ms) and high-frequency gamma bandpass filtering (55-95Hz), as shown in Table 3.3.

[0096] Table 3.3 Classification accuracy of each subject and each model

[0097]

[0098] Table 3.3 shows that the different participants significantly influenced the experimental results, and the performance of the classic Long Short-Term Memory Network was relatively stable across all participants. All models showed relatively low accuracy in decoding the EEG signals of participant 1, possibly due to participant 1's unfocused mental state. However, the CR-Net proposed in this invention achieved a significant success rate of 95.56% on the EEG signals extracted from participant 4. Other models also performed well on participant 4's data, indicating that the EEG data from participant 4 was of higher quality.

[0099] Step 3.4: Compare the classification accuracy of each model under different numbers of categories.

[0100] To address the issue of insufficient generalization performance in existing models, the model proposed in this invention was compared with other existing models on a dataset collected in the paper "Ahmed H, Wilbur RB, Bharadwaj HM, et al. Object classification from randomized EEG trials[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021:3845-3854." Similar to the dataset used in step 1, both datasets used 40 images of the same category from ImageNet. However, the dataset in the aforementioned paper employed a single-image, randomly presented paradigm without adhering to a category, and to expand the dataset, approximately 1000 images were selected for each category, totaling 40,000 images for EEG signal data collection. In the experiment, 400 images were presented per trial, each image for 2 seconds, with a 1-second interval between images, for a total trial duration of 20 minutes and 20 seconds. The classification accuracy of each model for classes 2-40 is as follows: Figure 5 As shown.

[0101] Table 3.4 lists the classification accuracy of each model for the dataset proposed in the literature "Ahmed H, Wilbur RB, Bharadwaj HM, et al. Object classification from randomized EEG trials[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2021:3845-3854." with 2, 3, 4, 5, 10, 20, 30 and 40 classes respectively.

[0102] Table 3.4 Classification accuracy of various models under different numbers of categories

[0103]

[0104] Table 3.4 shows that when the number of categories is small, the proposed model outperforms existing methods, demonstrating good generalizability. However, as the number of categories increases, the classification ability of all models drops sharply. This indicates that accurate classification of randomly sampled EEG signals from single stimuli is a problem that current technology struggles to solve and will remain a key issue that the field of EEG signal processing needs to address in the future.

Claims

1. A method for decoding electroencephalogram (EEG) signals based on spatiotemporal convolution and residual connections, characterized in that, Includes the following steps; Step 1: Construct a visually evoked EEG signal dataset, collect the spatiotemporal characteristics of EEG signals, and use them to stimulate subjects' vision to generate EEG signals. Step 2: Construct the EEG signal decoding and classification network CR-model based on spatiotemporal convolution and residual connections to extract the spatiotemporal features of the EEG signals collected in Step 1, and perform deep fusion extraction of the spatiotemporal features. By utilizing the high temporal resolution of EEG signals and the spatial connections reflected by multiple channels, the classification accuracy of visual stimulus EEG signals can be improved. Step 2 specifically involves: Step 2.1, construct the temporal convolution module; The size of the preprocessed EEG signal from step 1 when inputting the temporal convolution block is [size missing]. In the temporal convolutional block, one-dimensional convolution is applied independently to each channel to process the input signal along the time dimension; temporal feature extraction is then performed. Step 2.2, construct the spatial convolution module; After extracting the temporal features, the signal is input into the spatial convolution module to perform channel-level spatial analysis. By applying one-dimensional convolution in the channel dimension, the correlation between different channels is found within the corresponding time interval, which can decode and extract the spatial feature information of the EEG signal. Step 2.3, construct the residual module; After decoding and extracting the corresponding spatiotemporal features in the time and space dimensions, the CR-Model connects the obtained spatiotemporal features to a residual network. The final residual block, composed of a set of residual connection layers, performs two-dimensional convolution on the decoded spatiotemporal features to extract deeper relationships and features from the signal. Each residual layer performs two convolutions before adding the input to the residual. Then, the output is provided to the final convolutional layer. Finally, it is processed using a fully connected layer with the same size as the joint embedding dimension. In step 2.1, the temporal convolution block includes a set of one-dimensional convolutions with different dilation coefficients, and then their outputs are mapped and concatenated to obtain an output tensor. Each element of this tensor does not correspond to a single sample in the original signal in time, but by adjusting the kernel size and dilation, it can decode and extract information of a specific time span in the EEG signal. In step 2.2, the spatial convolution module further includes multiple one-dimensional convolutional layers whose outputs are concatenated to make the "rows" of the channel continuously appended to the signal matrix. Then, each spatial one-dimensional convolution operates with a different kernel size, and batch normalization is performed after all convolutional layers in the temporal and spatial blocks. activation.

2. The EEG signal decoding method based on spatiotemporal convolution and residual connections according to claim 1, characterized in that, Step 1 specifically involves: Step 1.1, Stimulus image selection; The EEG signal data collected in the experiment consisted of event-related electrical signals induced by object images as stimulus sources. Images with prominent subjects and minimal background interference were selected as visual stimuli, highlighting different salient targets, and used for visual observation by the subjects in step 1.2 to elicit different visual stimuli from the subjects. The selected visual stimuli were derived from... Different image categories in the image dataset; Step 1.2, Experimental Paradigm Design; During the recording of EEG signals, each type of image is displayed one by one in a trial; Step 1.3, EEG signal acquisition; Subjects observed the stimulus images selected in step 1.1 according to the experimental paradigm of step 1.2, and the resulting visual stimulus EEG signals were processed using... The brainwave signal acquisition system is used to collect and process the signals; Step 1.4: Construct an EEG signal dataset; The processed EEG dataset was divided into training, validation, and test sets, and the EEG signals of the same image corresponding to all subjects were placed in the same set.

3. The EEG signal decoding method based on spatiotemporal convolution and residual connections according to claim 2, characterized in that, In step 1.2, after a trial display ends, the screen displays a black background as a rest period to reduce the interference of the previous trial on the current trial.

4. The EEG signal decoding method based on spatiotemporal convolution and residual connections according to claim 2, characterized in that, In step 1.3, the EEG signal acquisition system includes a Quik-Cap 128-channel electrode cap, a SynAmps2 128-channel EEG signal amplifier, and the EEG signal acquisition and analysis software Curry 7.

5. The EEG signal decoding method based on spatiotemporal convolution and residual connections according to claim 4, characterized in that, The Quik-Cap 128-channel electrode cap is used to detect the subject's EEG trigger signal, the SynAmps2 128-channel EEG signal amplifier amplifies the EEG signal, and the Curry 7 EEG signal acquisition and analysis software is used to collect the subject's EEG signal and perform subsequent processing.

6. The EEG signal decoding method based on spatiotemporal convolution and residual connections according to claim 4, characterized in that, The 128-channel EEG signal acquisition system adopts the international 10-20 system, where 10 and 20 refer to the actual distance between adjacent electrodes being 10% or 20% of the total anterior-posterior or lateral distance of the skull. All EEG signals are... The electrodes were used as a reference, and the experimental paradigm design software used in the experiment was the psychological design software E-Prime.

7. The EEG signal decoding method based on spatiotemporal convolution and residual connections according to claim 2, characterized in that, In step 1.4, the sampling frequency of the EEG device is 1000. .

Images