Electroencephalogram signal visual decoding method and system based on neural co-occurrence pattern learning

Abstract

The application discloses a brain electrical signal visual decoding method and system based on neural co-occurrence pattern learning, and comprises a neural co-occurrence Transformer model, specifically comprises a space-time convolution feature extraction module and a neural co-occurrence encoder, and the neural co-occurrence encoder comprises feature identity embedding and a Transformer encoding block.The feature identity embedding is that a set of learnable vector parameters are initialized as unique identity identifiers of each feature block, and the parameters are directly superimposed on the feature blocks output by the space-time convolution feature extraction module, so that the neural activity patterns stably co-occurring under specific visual stimulation are automatically discovered and focused; then the neural activity patterns are input into the Transformer encoding block to obtain brain electrical representation vectors; and the brain electrical signal is decoded by adopting a multi-modal contrast learning mode and discarding traditional time position coding and instead capturing the co-occurrence patterns of neural signals, so that the visual decoding precision of the brain electrical signal is significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interface (BCI), artificial intelligence and neuroscience data analysis technology, and specifically to a method and system for visual decoding of electroencephalogram (EEG) signals based on neural co-occurrence pattern learning, which uses deep learning technology to decode visual object information from electroencephalogram (EEG) signals or magnetoencephalogram (MEG) signals. Background Technology

[0002] Decoding visual information directly from brain activity is a challenge in the field of brain-computer interfaces. Currently, deep learning-based decoding methods have made some progress, with the most representative related technologies falling into two main categories: The first category is discriminative methods based on contrastive learning. These methods typically align EEG signal features with visual features through a contrastive learning framework. While they have made some progress in object classification or retrieval tasks, they often ignore the stable co-occurrence structure of EEG signals in the spectral and spatial dimensions, making them quite sensitive to noise.

[0003] The second category is reconstruction methods based on generative models. These methods typically introduce a mapper to map EEG signals to a latent space to guide image generation or reconstruction. However, such methods often rely on strict time-series alignment to train the mapper; when there are jitters in the subject's neural response latency, the model's decoding and reconstruction performance can be significantly affected.

[0004] Given the technical shortcomings of existing EEG decoding models, which generally rely on strict time series alignment and thus are extremely sensitive to "latency jitter" in EEG signals and have poor robustness, there is still an urgent need for a visual decoding scheme for EEG signals that can maintain stable decoding performance even when there are fluctuations in the subject's neural reaction time.

[0005] In summary, current technologies for processing EEG signals have not yet fully addressed the key issue of "how to decouple time dependence and utilize stable neural co-occurrence patterns." Therefore, there is an urgent need for a visual decoding method for EEG signals that can decouple absolute time and location dependence and focus on capturing intrinsic co-occurrence patterns of neural features. Summary of the Invention

[0006] The purpose of this invention is to address the problems existing in the prior art by proposing a visual decoding method and system for electroencephalogram (EEG) signals based on neural co-occurrence pattern learning. This invention breaks away from the traditional method's dependence on strict time sequences of EEG signals by constructing a unique Neural Co-occurrence Transformer (NCT) model.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a visual decoding method for EEG signals based on neural co-occurrence pattern learning. The decoding method includes the following: data acquisition and preprocessing: collecting raw EEG signals of subjects when viewing visual stimuli, performing downsampling, bandpass filtering and segmentation processing, and converting them into a standardized two-dimensional data matrix as input to the neural co-occurrence Transformer model; Construct a neural co-occurrence Transformer model: including a spatiotemporal convolutional feature extraction module and a neural co-occurrence encoder NCT-C. The spatiotemporal convolutional feature extraction module includes a temporal convolutional extraction part, a spatial feature aggregation part, and a pointwise feature projection. The temporal convolution extraction section is used to slide independently along the time axis on each EEG channel, and uses the convolution kernel that slides along the time axis to extract neural oscillation features of specific frequency bands, and outputs the feature map into the spatial feature aggregation section; The point-by-point feature projection is used to project the input feature map into the target high-dimensional space in an up-dimensional manner. The spatial feature aggregation part utilizes cross-channel convolution kernels to aggregate spatial topological features of different EEG channels; The neural co-occurrence encoder includes feature identity embedding, a Transformer encoding block containing residual connections and layer normalization. The feature identity embedding is: initializing a set of learnable vector parameters and directly superimposing these parameters onto the feature blocks output by the spatiotemporal convolutional feature extraction module. These vector parameters do not contain any information about the temporal order, but serve as unique identifiers for each feature block, forcing the subsequent self-attention mechanism to abandon its focus on "temporal order" and instead use multi-head self-attention to calculate the similarity of content between feature blocks, thereby automatically discovering and focusing on neural activity patterns that co-occur stably under specific visual stimuli. After feature identity embedding processing, a feature block sequence with injected identity embedding and permutation invariance is obtained. The feature block sequence is input into a Transformer coding block containing residual connections and layer normalization. Multi-head self-attention is used to calculate the co-occurrence association weight between feature blocks, thereby obtaining the EEG representation vector output by the neural co-occurrence encoder. A neural co-occurrence Transformer model was trained using multimodal contrastive learning with input data, and the trained neural co-occurrence Transformer model was used to decode EEG signals.

[0008] Furthermore, the specific processing method for the feature identity embedding is as follows: initializing a set of learnable parameter matrices corresponding to the number of feature blocks. ; through formula The parameter matrix is ​​superimposed onto the feature block sequence P; where... It is used only to identify the unique identity of the feature block and does not contain the temporal position information of the feature block sequence in the original signal, so that the neural co-occurrence encoder has permutation invariance.

[0009] Furthermore, the self-attention mechanism operates in an invariant manner, and its computation process includes:

[0010]

[0011] In this module, MHSA stands for Multi-Head Self-Attention Module, FFN stands for Feedforward Neural Network, LayerNorm stands for Layer Normalization, and Output stands for the generated EEG representation vector.

[0012] Furthermore, the multimodal contrastive learning training employs loss functions including EEG-image contrastive loss and EEG-text contrastive loss, with the total loss function defined as:

[0013] Where α represents the weighting coefficient used to balance the contributions of the visual modality and the text modality; The contrast loss term representing the alignment of EEG features with image features; The contrast loss representing the alignment of EEG features with text features is the auxiliary regularization loss term, which is calculated in the same way as the image alignment loss term.

[0014] Furthermore, the Transformer encoding block, which includes residual connections and layer normalization, is a single-layer network structure configured to complete the co-occurrence combination calculation of all features within one forward computation cycle, in order to limit the model's overfitting to the temporal order.

[0015] Furthermore, the segment length is 50-300ms, and data from a specific time window is extracted as the model input sample E based on the timing of stimulus presentation.

[0016] Furthermore, the neural co-occurrence Transformer model also includes a cross-modal semantic projection layer, which adopts a multilayer perceptron structure and sequentially includes: a linear layer, layer normalization and nonlinear activation functions, and the final output linear layer; after the mapping of the cross-modal semantic projection layer, the model outputs the final EEG feature vector, which is consistent with the image features and text features extracted by the target pre-trained multimodal model in terms of feature dimension.

[0017] Secondly, the present invention provides a visual decoding system for electroencephalogram (EEG) signals based on neural co-occurrence pattern learning. The decoding method includes: a data acquisition module for acquiring and preprocessing EEG signals; a feature extraction module for converting EEG signals into a high-dimensional feature block sequence; a co-occurrence encoding module configured to process the feature block sequence by superimposing learnable feature identity embeddings instead of using temporal position encoding, and extracting co-occurrence features using a self-attention mechanism; and a semantic decoding module for outputting visual decoding results based on the co-occurrence features.

[0018] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for visual decoding of EEG signals based on neural co-occurrence pattern learning.

[0019] Fourthly, the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the above-described method for visual decoding of EEG signals based on neural co-occurrence pattern learning.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: First, this invention significantly improves the visual decoding accuracy of EEG signals by abandoning traditional temporal location encoding and instead capturing the co-occurrence patterns of neural signals. In a zero-shot image retrieval task across 200 classes on the publicly available standard dataset THINGS-EEG, the Neural Co-occurrence Transformer (NCT) of this invention achieved a Top-1 retrieval accuracy of 32.25%. Compared to the widely used EEGNetv4 model, the accuracy of this invention is improved by approximately 7.5%; performance is also improved compared to representative methods based on generative reconstruction. These results demonstrate that for complex visual decoding tasks, modeling based on co-occurrence combinations can effectively alleviate the decoding bottleneck problem caused by the strict temporal sequence dependence of existing deep learning models.

[0021] Secondly, this invention exhibits strong anti-interference and robustness, effectively overcoming the problem of latency jitter in the subject's neural responses. Some existing temporal modeling methods rely on absolute time positions; once a time shift occurs in the signal, the model performance deteriorates. This invention innovatively introduces a feature identity embedding strategy, transforming the EEG feature sequence into a disordered feature set, giving the model permutation invariance. This means that regardless of the specific time of occurrence of key neural oscillation features on the timeline, as long as they coexist within the same time window, the model can correctly identify them. Experiments show that even when the order of input features is artificially shuffled, the decoding performance of this invention remains stable, demonstrating good robustness.

[0022] Furthermore, this invention reveals the intrinsic mechanism by which the brain processes visual information, demonstrating a high degree of neuroscientific interpretability. Visual analysis of the attention flow within the model reveals that, without introducing any prior anatomical knowledge, this invention automatically generates a dual-flow structure highly consistent with the human brain's visual processing mechanism: information flow clearly originates from the occipital visual cortex to the parietal lobe (dorsal pathway) responsible for spatial perception and the temporal lobe (ventral pathway) responsible for object recognition. This finding confirms that the co-occurrence features extracted in this invention are not random noise in the data, but rather genuine and effective neurocognitive representations, providing a new perspective for the mechanistic research of brain-computer interfaces.

[0023] Finally, this invention solves the problems of low signal-to-noise ratio and semantic ambiguity in EEG signals through an optimized multimodal semantic alignment strategy. This invention employs a CLIP-based multimodal contrastive learning framework and prioritizes low-level visual details in image features through a weighted loss function, while utilizing text features to provide high-level semantic constraints. This ensures that the model can still achieve accurate zero-shot retrieval and decoding based on feature co-occurrence patterns when facing unseen image categories. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the training framework based on contrastive learning provided in an embodiment of the present invention. It illustrates the process of semantic alignment between the NCT model and a frozen multimodal pre-trained model (such as CLIP) image / text encoder.

[0025] Figure 2 This is a schematic diagram of the structure of the Neural Co-occurrence Transformer (NCT) model provided in an embodiment of the present invention.

[0026] Figure 3 This is the performance of the embodiments of the present invention on the THINGS-EEG dataset, where (a) is the similarity matrix, (b) is the t-SNE visualization clustering of EEG features, and (c) is an example of the qualitative results of zero-shot image retrieval.

[0027] Figure 4 This is a diagram of the internal mechanism analysis of the model in this embodiment of the invention. In the diagram, (a) shows the brain regions and their corresponding functional divisions, as well as the transmission process of attention flow; (b) shows the visualization process of attention flow inside the model, which verifies the connection pattern of the simulated dorsal / ventral pathway of the human brain.

[0028] Figure 5 The image shows a comparison of decoding results with and without feature identity embedding, where NCT-C indicates the use of feature identity embedding and NTC-S indicates the use of traditional time series modeling. Detailed Implementation

[0029] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of protection of this application.

[0030] This invention is a visual decoding system for EEG signals with permutation invariance: After extracting features from the EEG signal through a spatiotemporal convolutional feature extraction module and converting it into a high-dimensional feature block sequence, instead of introducing positional encoding to represent temporal order, an independent and learnable feature identity embedding vector is assigned to each feature block. The feature identity embedding vector is used only to identify the unique identity of the feature block. By superimposing this vector with the corresponding high-dimensional EEG feature block and inputting it into a Transformer encoding block with multi-head self-attention, the model ignores the absolute order of features on the time axis when calculating feature association weights, and instead captures feature combinations that stably co-occur within the same time window, i.e., neural co-occurrence patterns. This fundamentally decouples the model from the dependence on absolute temporal position, improving the accuracy and anti-interference ability of the EEG signal visual decoding system when there are latency jitters.

[0031] The term "co-occurrence" refers to the combined occurrence pattern of specific frequency band oscillations (spectral dimension) generated by different brain regions (spatial dimension) within the same time window when a subject is subjected to visual stimulation. This invention proposes that as long as key neural features (such as alpha waves in the occipital lobe and theta waves in the parietal lobe) co-occur in a single stimulus, the visual content can be determined, without needing to strictly define the millisecond-level order in which these features appear. Therefore, this invention eliminates traditional temporal location encoding and significantly improves the model's robustness to EEG latency jitter by learning the inherent co-occurrence relationships of feature sets, achieving accurate decoding of visual objects.

[0032] Example 1 The EEG signal visual decoding method based on neural co-occurrence pattern learning in this embodiment includes the following steps: Step 1 (Model Training Stage): Obtain the training EEG signals of the subjects and the corresponding visual stimulus images and text labels to construct a training dataset; construct a neural co-occurrence Transformer model that includes a spatiotemporal convolution feature extraction module and a neural co-occurrence encoder; use the training dataset to perform multimodal contrastive learning training on the neural co-occurrence Transformer model to obtain a trained neural co-occurrence Transformer model, which serves as the EEG decoding network; Step 2 (Feature Extraction): Obtain the raw online EEG signals of the subject when receiving visual stimuli, and preprocess them to construct a data tensor; input the data tensor into the trained neural co-occurrence Transformer model, and extract the discrete feature block sequence decoupled from global temporal dependence in its spatiotemporal convolution feature extraction module; Step 3 (Co-occurrence Encoding): The feature block sequence described in Step 2 is input into the neural co-occurrence encoder of the trained neural co-occurrence Transformer model. The neural co-occurrence encoder NCT-C removes the temporal position encoding and adds a learnable feature identity embedding to each feature block, thereby transforming the feature block sequence into an unordered feature set. The self-attention mechanism in the neural co-occurrence encoder is used to perform global association modeling on the unordered feature set, capture the co-occurrence patterns between feature blocks, and generate EEG representation vectors. Step 4 (Semantic Decoding): Map the EEG representation vector to the target semantic space and decode the visual stimulus content by calculating similarity.

[0033] Example 2 This embodiment uses a visual decoding method for EEG signals based on neural co-occurrence pattern learning. The specific steps are as follows: Step 1: Acquisition and Standardized Preprocessing of EEG Data Before practical application or model training, the raw EEG signals need to be denoised and formatted to form a standard two-dimensional data matrix that can be directly processed by the subsequent neural network.

[0034] The specific processing steps are as follows: Data Acquisition: Raw EEG signals of subjects were recorded using a multi-channel EEG acquisition device while receiving visual stimuli (e.g., image display). In a preferred embodiment, 63-channel electrodes were used to record the subjects' EEG activity in the rapid sequence visual presentation (RSVP) paradigm. This embodiment collected EEG data from 10 subjects. Figure 5 The image shows the original electroencephalograms of different subjects, where Sub-01 represents the first subject, and so on.

[0035] Downsampling and denoising: To reduce computational complexity and remove environmental and physiological noise, the sampling rate of the original EEG signal is first downsampled to a target frequency (preferably 250Hz). Then, a bandpass filter is applied to the downsampled signal to remove extremely low-frequency baseline drift artifacts and high-frequency electromyographic interference. The passband range of the bandpass filter is preferably from 0.1Hz to 100Hz.

[0036] Epoching: Based on the event markers of the visual stimulus presentation, continuous EEG data within a specific time window is extracted as an independent sample. In a preferred embodiment, the data segment from 0ms to 1000ms after the stimulus presentation is extracted.

[0037] Channel-level standardization: To eliminate dimensional differences between different electrode channels and accelerate the convergence of subsequent models, the segmented data is subjected to Z-Score standardization by channel, ensuring that the data distribution of each channel meets the requirements of a mean of 0 and a standard deviation of 1. After the above processing, the EEG data sample corresponding to a single visual stimulus is formatted into a standardized two-dimensional data matrix E, where... C represents the number of EEG signal channels (e.g., 63 channels), and T represents the total number of time sampling points within the extracted time window (e.g., T = 250 sampling points under a sampling rate of 250Hz and a window of 0ms–1000ms). The matrix E serves as the direct input to the subsequent spatiotemporal convolutional feature extraction module.

[0038] The preprocessed EEG signals and corresponding visual stimulus images, along with text labels (i.e., the file names of each image), are used to construct a training dataset. Step 2: Constructing the Neural Co-occurrence Transformer (NCT) Model The neural co-occurrence Transformer (NCT) model (see...) Figure 2 The system includes a spatiotemporal convolutional feature extraction module (the backbone of the convolutional neural network), a neural co-occurrence encoder (NCT-C), and a cross-modal semantic projection layer. The spatiotemporal convolutional feature extraction module comprises a temporal convolution extraction part, a spatial feature aggregation part, and a pointwise feature projection part. Specifically, Temporal Convolution Extraction: The input data—a normalized two-dimensional data matrix E—first enters the temporal convolution extraction section. In this section, a set of learnable two-dimensional filters is used to independently convolve each channel along the time axis, with the kernel size set to [value missing]. ,in This represents the temporal receptive field. The mathematical expression for the temporal convolution extraction part is:

[0039] Where E represents the preprocessed, standardized two-dimensional data matrix (the dimension is the number of channels multiplied by the number of sampling time points). Represents the learnable temporal convolutional kernel weights (size set to...). (Used to extract oscillation features in a specific frequency band), * indicates a two-dimensional convolution operation, BatchNorm and ELU represent batch normalization and exponential linear unit nonlinear activation functions, respectively. This is the output of the temporal convolution extraction part.

[0040] Spatial Convolution: Following temporal convolution extraction, the spatial feature aggregation part is performed. This part uses depthwise convolution layers to aggregate signals along the channel dimension, learning the spatial topological relationships between signals from different brain regions (e.g., occipital and parietal lobes). The mathematical expression for spatial feature aggregation is as follows:

[0041] Where AvgPool represents the average pooling downsampling operation performed along the time dimension. The spatial convolution kernel weights in the depth direction (with a size of (250, 1) are used to learn and aggregate the spatial topological relationships between different brain regions in the channel dimension). This refers to the extracted spatiotemporal feature map.

[0042] 3. Pointwise Feature Projection: The output of the spatial feature aggregation part is reshaped through 1*1 convolution and dimension alignment operations to obtain the feature block sequence P, which is the output of the spatiotemporal convolutional feature extraction module. Pointwise feature projection can project the input feature map into the target high-dimensional space (64 dimensions).

[0043] It is important to note that, unlike traditional EEG processing front-ends that preserve the complete temporal sequence, the spatiotemporal convolutional feature extraction module in this embodiment is not an isolated feature extractor, but a customized front-end that provides a spatial topological basis for downstream co-occurrence patterns. Its output feature block sequence P is a local oscillation fragment decoupled from global temporal features. The physical significance of this forced discretization is that it severs the long-range temporal dependence of traditional EEG features, providing a data foundation for the subsequent co-occurrence encoder to abandon positional encoding and focus on the intrinsic combination of features.

[0044] 4. A Neural Co-occurrence Encoder (NCT-C) was constructed, comprising feature identity embedding, Transformer encoding blocks with residual connections and layer normalization. This structure fundamentally changes the traditional reliance on temporal modeling of EEG signals. Unlike Recurrent Neural Networks (RNNs) or standard Transformers that force temporal sequence order through positional encoding (denoted as NCT-S), NCT-C treats the extracted feature block sequence P as an unordered set of features, introducing a unique "Feature Identity Embedding" strategy. Specifically, the system initializes a set of learnable vector parameters and directly superimposes these parameters onto the input feature blocks. It is particularly important to emphasize that these vector parameters do not contain any information about temporal order (such as "100th millisecond" or "200th millisecond"), but rather serve as unique identifiers for each feature block. This design forces subsequent self-attention mechanisms to abandon their focus on "temporal order" and instead utilize multi-head self-attention to compute the similarity of content between feature blocks, thereby automatically discovering and focusing on neural activity patterns that stably co-occur under specific visual stimuli. By using Transformer encoding blocks containing residual connections and layer normalization, highly abstract EEG semantic representations with permutation invariance can be extracted from an unordered feature set, exhibiting strong robustness to latency jitter commonly found in EEG signals. This invention does not use positional encodings representing temporal order (e.g., 0,1,2,…,T0,1,2,…,T), but instead initializes a set of learnable vector parameters. .

[0045] (1) Feature Identity Embedding: The system initializes a set of learnable vector parameters. The vector parameters are then directly superimposed onto the feature block sequence P:

[0046] in, The embedding matrix consists of learnable vector parameters that do not contain time-series information and are only used to identify the unique identity of features. This represents a sequence of feature blocks with injected identity embedding and permutation invariance, which is the input to a Transformer encoded block containing residual connections and layer normalization.

[0047] The Feature Identity Embedding not only omits time-series information, but more importantly, through end-to-end training, the system forces each row of the embedding matrix to spontaneously specialize during training, specifically for learning particular types of neurophysiological patterns (such as alpha oscillations or P300 components). This transforms the neural network from traditional location-based encoding that records the timing of feature occurrences to essentially 'establishing classification slots for functional features,' replacing chronological order with slot identifiers.

[0048] (2) Self-attention mechanism: embedding the injected identity into a feature block sequence that has permutation invariance. The input is fed into a single-layer Transformer encoding block. Each layer contains a multi-head self-attention mechanism (MHSA) and a feedforward neural network (FFN). The computation process is as follows:

[0049]

[0050] in: This represents a multi-head self-attention mechanism used to calculate the co-occurrence association weights between feature blocks; This represents a feedforward neural network used for nonlinear transformation of features; Presentation layer normalization operation; This represents the intermediate feature representation after processing by the self-attention mechanism. This represents the EEG semantic representation of the output of a Transformer coding block. The number of Transformer coding blocks, including residual connections and layer normalization, can be 1-3. When there are multiple coding blocks, the output of the previous single-layer Transformer coding block is used as the input of the next single-layer Transformer coding block.

[0051] Furthermore, to avoid the model implicitly relearning temporal dependencies in extremely deep network layers, in a preferred embodiment of the present invention, the Transformer encoder is preferably a single-layer multi-head self-attention architecture. This single-layer structure, in conjunction with feature identity embedding, forces the model to complete the co-occurrence combination calculation of all visual response features within a single forward computation cycle. This constraint not only significantly reduces the number of model parameters and computational complexity but also maximizes the limitation of the model's tendency to overfit temporal sequences, thereby significantly improving the system's robustness to EEG latency jitter.

[0052] In the self-attention mechanism modeling steps described above, since the input feature blocks have been assigned feature identity embeddings rather than temporal location encodings, the attention weight matrix is ​​calculated during multi-head self-attention (MHSA) computation. The physical meaning of [the concept] has undergone a fundamental change. Specifically, the weight calculation for multi-head self-attention satisfies the following relationship:

[0053] Where z is the input feature. and For the projection matrix of the query and key, For the feature dimension. The attention weights here. Instead of measuring the sequential transition probability between time step i and time step j, it directly measures the pairwise co-occurrence strengths (i.e., co-occurrence association weights) of the i-th neurophysiological feature and the j-th neurophysiological feature under the same visual stimulus time window. Based on this, it automatically discovers and focuses on EEG feature combinations that stably co-occur in different samples, thereby achieving true co-occurrence pattern learning.

[0054] 5. Cross-modal semantic projection: Following the neural co-occurrence encoder, a cross-modal semantic projection layer is also included. Since the EEG representation vectors extracted by the neural co-occurrence encoder often do not match the feature dimensions of the target pre-trained multimodal model (such as CLIP), the cross-modal semantic projection layer is configured to nonlinearly map the EEG representation vectors to the target shared multimodal semantic space.

[0055] Specifically, such as Figure 2 As shown, the cross-modal semantic projection layer adopts a multilayer perceptron (MLP) structure, which sequentially includes: a linear layer (B, 50, 64), layer normalization and nonlinear activation function (ELU activation function), and the final output linear layer (B, 64). B represents the amount of data input to the model during each batch of training, 50 represents the sequence length, and 64 represents the feature dimension of the overall model output. After mapping by the cross-modal semantic projection layer, the model outputs the final EEG feature vector, which can be directly cosine similar to the features extracted by the CLIP model (which can extract both text and image features) in terms of both dimension and distribution. This provides a structural basis for subsequent multimodal contrastive learning loss calculation and zero-shot cross-modal retrieval.

[0056] The third step is the multimodal contrastive learning and training phase. like Figure 1As shown, in the model training and semantic alignment stages, this invention employs a cross-modal contrastive learning framework for end-to-end optimization. The target pre-trained multimodal model uses the CLIP model as the "teacher model," and its parameters are frozen to maintain its strong semantic representation capabilities. For each input EEG sample, the system extracts its corresponding visual stimulus image and text description, and obtains image feature vectors and text feature vectors through the CLIP model's image encoder and text encoder, respectively. The training objective function is constructed based on InfoNCE loss, aiming to maximize the cosine similarity between the EEG representation vector and the corresponding image / text feature vector, while minimizing its similarity with other non-matching samples within the batch. Regarding the weight allocation of the loss function, this embodiment found that image features contain low-level visual details (such as color and texture) crucial for decoding, while text features provide high-level semantic constraints. Therefore, a higher image loss weight is set, forming an optimization strategy that prioritizes visual information and supplements it with semantic information, thereby guiding the NCT model to learn EEG representations that contain both visual details and semantic consistency. By using a pre-trained CLIP model to extract image feature embeddings of visual stimulus images and text feature embeddings of corresponding text descriptions, the neural co-occurrence encoder NCT-C outputs EEG representation vectors, which are then used to obtain EEG feature vectors through cross-modal semantic projection.

[0057] Loss function construction: The InfoNCE loss function (ClipLoss) is used to maximize the cosine similarity between the image feature embedding of the stimulus image and the text feature embedding of the corresponding text description.

[0058] The contrastive loss term for aligning EEG representation vectors with image features is:

[0059] in, Indicates the batch size during training. Indicates the first Cosine similarity (positive sample pair) between an EEG sample (NCT model processing result) and its corresponding correct image sample features (CLIP processing result); Indicates the first Cosine similarity (negative sample pairs) between features of an EEG sample and other image samples in batch j. This represents a learnable temperature parameter used to adjust the smoothness of the similarity distribution.

[0060] The total loss function is then:

[0061] in: This represents the weighting coefficient used to balance the contributions of the visual and textual modalities; The contrast loss term represents the alignment of EEG samples with image features; The auxiliary regularization loss term represents the alignment of EEG samples with text features.

[0062] The neural co-occurrence Transformer (NCT) model was obtained through training.

[0063] Step 4: Online Visual Decoding (Inference) Stage Based on the above system architecture, this invention demonstrates significant performance advantages in practical zero-shot image retrieval tasks. In the inference phase, a retrieval library containing features of all candidate images is first constructed. After real-time acquisition of the subject's EEG signals, a trained Neural Co-occurrence Transformer (NCT) model is used to generate an EEG query vector (i.e., an EEG feature vector). The similarity between this query vector and all image features in the retrieval library is calculated, and the image with the highest similarity is finally output as the decoding result.

[0064] Experimental data show that the method of this invention achieves high accuracy in the 200-class zero-shot retrieval task of the THINGS-EEG dataset, as shown in the results. Figure 3 As shown, the decoding performance of subjects viewing different types of images, along with the visualization of EEG features, demonstrates that the model can effectively learn the semantic information of images (distinguishing between animals, objects, and other categories), and exhibits advantages over existing representative methods. Further mechanistic analysis reveals that, despite the absence of any prior neuroscience knowledge, the NCT model spontaneously exhibits an attentional flow structure similar to that of the human brain after training. Its internal information flow clearly flows from the occipital visual cortex to the parietal lobe responsible for spatial processing and the temporal lobe responsible for object recognition, which highly coincides with the dorsal and ventral visual pathways of the brain. Furthermore, even when the order of input features is artificially shuffled, the decoding performance of NCT remains stable, further confirming that this invention successfully decouples time dependence and achieves efficient, robust, and biologically interpretable visual decoding by capturing stable neural co-occurrence patterns.

[0065] In this embodiment, the experiment is based on the PyTorch deep learning framework and runs on a compute node configured with an NVIDIA GPU. Training uses the Adam optimizer, with an initial learning rate set to 0.001, a batch size of 128, and a temperature parameter in the InfoNCE loss function. The value was set to 0.07. This invention was validated on the publicly available dataset THINGS-EEG. The table below shows the prediction comparison results of different models.

[0066]

[0067] Decoding Accuracy: In the 200-way zero-shot image retrieval task, the method of this invention achieves a high Top-1 accuracy, outperforming several existing representative methods. Mechanism Analysis: Through gradient-weighted class activation heatmap Grad-CAM and attention flow analysis (e.g., Figure 4 The study found that the Neural Co-occurrence Transformer (NCT) model can automatically form information pathways from the occipital lobe to the parietal and temporal lobes, consistent with the dorsal and ventral visual pathways in the human brain. Anti-interference verification: By shuffling the order of input features, the model's performance was almost unaffected, demonstrating that it utilizes stable co-occurrence patterns rather than fragile temporal dependencies. Without feature identity embedding (NCT-S), less information is considered, and temporal dependencies are high; the comparison results are as follows... Figure 5 As shown.

[0068] The above description is merely a preferred embodiment of the present invention. Based on the above description, those skilled in the art can make various changes and modifications without departing from the technical concept of this invention. The technical scope of this invention is not limited to the contents of the specification but must be determined according to the scope of the claims.

[0069] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A method for visual decoding of electroencephalogram (EEG) signals based on neural co-occurrence pattern learning, characterized in that, The decoding method includes the following: Data acquisition and preprocessing: Raw EEG signals were collected from subjects while they viewed visual stimuli. The signals were then downsampled, bandpass filtered, and segmented to convert them into a standardized two-dimensional data matrix, which was used as input to the neural co-occurrence Transformer model. Construct a neural co-occurrence Transformer model: including a spatiotemporal convolutional feature extraction module and a neural co-occurrence encoder NCT-C. The spatiotemporal convolutional feature extraction module includes a temporal convolutional extraction part, a spatial feature aggregation part, and a pointwise feature projection. The temporal convolution extraction section is used to slide independently along the time axis on each EEG channel, and uses the convolution kernel that slides along the time axis to extract neural oscillation features of specific frequency bands, and outputs the feature map into the spatial feature aggregation section; The point-by-point feature projection is used to project the input feature map into the target high-dimensional space in an up-dimensional manner. The spatial feature aggregation part utilizes cross-channel convolution kernels to aggregate spatial topological features of different EEG channels; The neural co-occurrence encoder includes feature identity embedding, a Transformer encoding block containing residual connections and layer normalization. The feature identity embedding is: initializing a set of learnable vector parameters and directly superimposing these parameters onto the feature blocks output by the spatiotemporal convolutional feature extraction module. These vector parameters do not contain any information about the temporal order, but serve as unique identifiers for each feature block, forcing the subsequent self-attention mechanism to abandon its focus on "temporal order" and instead use multi-head self-attention to calculate the similarity of content between feature blocks, thereby automatically discovering and focusing on neural activity patterns that co-occur stably under specific visual stimuli. After feature identity embedding processing, a feature block sequence with injected identity embedding and permutation invariance is obtained. The feature block sequence is input into a Transformer coding block containing residual connections and layer normalization. Multi-head self-attention is used to calculate the co-occurrence association weight between feature blocks, thereby obtaining the EEG representation vector output by the neural co-occurrence encoder. A neural co-occurrence Transformer model was trained using multimodal contrastive learning with input data, and the trained neural co-occurrence Transformer model was used to decode EEG signals.

2. The method according to claim 1, characterized in that, A training dataset was constructed using preprocessed EEG signals and corresponding visual stimulus images and text labels; The objective function of the training process is constructed based on InfoNCE loss, which aims to maximize the cosine similarity between the EEG representation vector and the corresponding image / text feature vector, while minimizing its similarity with other unmatched samples in the batch.

3. The method according to claim 2, characterized in that, The image features / text feature vectors are obtained through processing using a pre-trained multimodal model CLIP.

4. The method according to claim 2, characterized in that, The total loss function is: ， in: This represents the weighting coefficient used to balance the contributions of the visual and textual modalities; The contrast loss term represents the alignment of EEG samples with image features; This represents the contrast loss for aligning EEG samples with text features.

5. The method according to claim 1, characterized in that, The point-by-point feature projection process is as follows: the output of the spatial feature aggregation part is reshaped by 1*1 convolution and dimension alignment to obtain the feature block sequence P.

6. The method according to claim 1, characterized in that, The mathematical expression for the temporal convolution extraction part is: ， Where E represents the preprocessed, standardized two-dimensional data matrix. The asterisk (*) represents the learnable temporal convolution kernel weights, * represents the two-dimensional convolution operation, and BatchNorm and ELU represent the batch normalization operation and the exponential linear unit nonlinear activation function, respectively. This is the output of the temporal convolution extraction part; The mathematical expression for the spatial feature aggregation part is as follows: ， Where AvgPool represents the average pooling downsampling operation performed along the time dimension; The spatial convolution kernel weights in the depth direction are used to learn and aggregate the spatial topological relationships between different brain regions in the channel dimension. This is the extracted spatiotemporal feature map.

7. The method according to any one of claims 1-6, characterized in that, The neural co-occurrence Transformer model also includes a cross-modal semantic projection layer, which adopts a multilayer perceptron structure and sequentially includes: a linear layer, layer normalization and nonlinear activation functions, and the final output linear layer; after the mapping of the cross-modal semantic projection layer, the model outputs the final EEG feature vector, which is consistent with the image features and text features extracted by the target pre-trained multimodal model in terms of feature dimension.

8. A visual decoding system for electroencephalogram (EEG) signals based on neural co-occurrence pattern learning, comprising the method described in any one of claims 1-7, characterized in that, include: The data acquisition module is used to acquire and preprocess EEG signals; The feature extraction module is used to convert EEG signals into high-dimensional feature block sequences; The co-occurrence encoding module is configured not to use temporal position encoding, but to process the feature block sequence by superimposing learnable feature identity embeddings, and to extract co-occurrence features using a self-attention mechanism; The semantic decoding module is used to output visual decoding results based on co-occurrence features.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the EEG signal visual decoding method according to any one of claims 1-7.

10. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the EEG signal visual decoding method based on neural co-occurrence pattern learning as described in any one of claims 1-7.

Images