A multi-stage image reconstruction method based on electroencephalogram visual semantics extraction

Abstract

The application discloses a multi-stage image reconstruction method based on electroencephalogram visual semantic extraction, comprising the following steps: acquiring an electroencephalogram visual data set, and pre-processing electroencephalogram signals and image data; constructing a NeuroFormer electroencephalogram encoder, combining image attention, channel attention and space-time convolution mechanism to extract electroencephalogram space-time features, and using a pre-trained CLIP-ViT-H / 14 image encoder to realize cross-modal shared feature space alignment training; mapping the electroencephalogram signal to be recognized into an electroencephalogram embedding, and performing zero-shot classification by comparing with a test category image embedding prototype; performing multi-stage diffusion reconstruction based on the electroencephalogram embedding, including semantic representation generation, latent space structure sketch generation and cascaded diffusion refinement reconstruction, and obtaining a final image. The application can effectively improve the electroencephalogram visual semantic expression ability, cross-modal generalization ability and image reconstruction quality, and realize electroencephalogram image reconstruction with consistent structure and accurate semantics.

Description

Technical Field

[0001] This invention belongs to the field of EEG visual decoding and EEG signal processing technology, specifically relating to a multi-stage image reconstruction method based on EEG visual semantic extraction. Background Technology

[0002] Visual decoding is an important research direction in cognitive neuroscience and brain-inspired artificial intelligence, aiming to infer and reconstruct the visual content perceived by an individual from neural activity. In recent years, researchers have attempted to use different neuroimaging techniques to reverse-engineer the way the human brain encodes natural images in order to classify and reconstruct visual content. Among them, functional magnetic resonance imaging (fMRI) has been widely used in visual decoding tasks due to its high spatial resolution and has made significant progress in image classification and image reconstruction. However, fMRI relies on changes in blood oxygen levels to indirectly reflect neural activity, has low temporal resolution, and is expensive and difficult to process in real time, which limits its application in fast visual dynamic decoding and large-scale scenarios.

[0003] In contrast, electroencephalography (EEG), with its advantages of millisecond-level temporal resolution, portability, and low cost, has become an important approach to understanding human visual processing. EEG can directly record potential changes caused by neuronal electrical activity, capturing the brain's response to rapid visual stimuli in real time, making it more suitable for visual decoding and non-invasive brain-computer interfaces (BCIs). Nevertheless, EEG-based visual decoding still faces significant challenges, including insufficient spatial resolution, strong noise interference, large inter-subject variability, and a limited availability of high-quality paired data for training. These factors significantly limit the generalization ability and stability of EEG-based visual decoding models.

[0004] To improve EEG visual decoding performance, existing technology CN119478100A introduces multimodal learning methods to jointly model EEG signals with image or text semantic information. Previous work has achieved zero-shot classification or preliminary EEG-to-image reconstruction through contrastive learning, trimodal semantic alignment, or image generation strategies based on diffusion models. Simultaneously, research has also constructed EEG encoders to model spatiotemporal dynamics and improve decoding performance. For example, existing technology discloses an EEG image generation method based on autoregressive pre-training and feature alignment, which generates EEG images by autoregressively encoding EEG signals and combining a CLIP model with a stable diffusion model. However, this type of method still has the following shortcomings: First, existing methods mainly rely on autoregressive coding structures to model EEG sequences, which do not adequately consider the spatial topological relationships between EEG electrodes and the functional associations across brain regions, making it difficult to fully explore the spatial dependency features between EEG channels, resulting in limited EEG feature expression capabilities. Secondly, existing methods usually focus on single-stage semantic mapping or direct image generation, lacking hierarchical modeling of image semantic and structural information. This makes it difficult to simultaneously take into account high-level semantic consistency and low-level structural details, which can easily lead to problems such as structural distortion, texture blurring, or semantic shift in the generated image. Third, existing technologies mostly use a limited number of EEG-image paired samples for training. In zero-sample scenarios where the training category and the test category are inconsistent, the model's cross-class generalization ability is weak, making it difficult to achieve stable category transfer and semantic matching. Fourth, most existing diffusion generation methods directly use EEG features to constrain the diffusion model, lacking an independent modeling process for prior image structure. This results in insufficient ability to preserve object contours, spatial layout, and color distribution during the generation process, affecting the final image reconstruction quality. Fifth, existing methods in the process of aligning EEG and visual semantics usually rely on only a single feature mapping or a simple feature alignment method, which lacks effective suppression of EEG noise, cross-subject differences and feature distribution shifts under limited sample conditions, resulting in insufficient model training stability and robustness.

[0005] Therefore, under the current technological conditions, there is an urgent need for an EEG visual decoding method that can more effectively capture the spatial dependencies between EEG channels, improve multimodal semantic alignment capabilities, and achieve high-quality image reconstruction under limited data conditions, in order to solve the shortcomings of existing technologies in feature representation, model generalization, and image generation quality. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multi-stage image reconstruction method based on EEG visual semantic extraction.

[0007] The objective of this invention can be achieved through the following technical solutions: This invention provides a multi-stage image reconstruction method based on electroencephalogram (EEG) visual semantic extraction, comprising the following steps: S1. Obtain the public dataset, preprocess the EEG signals and corresponding image data in the public dataset, and perform data augmentation processing respectively to obtain the augmented EEG data and the augmented image data. S2, input the enhanced EEG data into the EEG encoder to extract the EEG embedding, input the enhanced image data into the image encoder to extract the image embedding, and perform shared feature space alignment training on the EEG embedding and the image embedding based on cross-modal contrastive learning to obtain the trained EEG encoder; S3, input the EEG signal to be identified into the trained EEG encoder for feature mapping to obtain the corresponding EEG embedding, and perform zero-shot classification based on the similarity between the EEG embedding of the EEG signal to be identified and the image features of the test category to obtain the corresponding classification result; S4, performing multi-stage image reconstruction based on the EEG embedding of the EEG signal to be identified and the classification result, including: generating an image semantic representation using a diffusion model conditioned on the EEG embedding of the EEG signal to be identified; mapping the EEG embedding of the EEG signal to be identified to the latent space of the diffusion model autoencoder to generate a structural sketch; fusing the image semantic representation and the structural sketch and inputting them into a cascaded diffusion model, and combining the classification result to refine the structure and complete the semantics of the image, generating the final reconstructed image.

[0008] Furthermore, the publicly available dataset is an EEG-visual dataset constructed based on the fast sequence visual presentation paradigm. The EEG-visual dataset includes EEG signals and corresponding image data of multiple subjects during the viewing of natural images. The image data consists of visual stimulus images acquired synchronously with the EEG signals. The EEG-visual dataset includes a training set and a test set, wherein the image categories in the training set and the test set do not overlap. The training set is used for cross-modal alignment training of the EEG encoder, and the test set is used for zero-shot classification and image reconstruction.

[0009] Furthermore, the preprocessing of the EEG signals and corresponding image data in the publicly available dataset, followed by data augmentation processing to obtain augmented EEG data and augmented image data, specifically includes: The EEG signals are subjected to test segmentation, baseline correction, signal downsampling, and repeated averaging to obtain stable EEG response data. The test segmentation is to extract a preset time window after the presentation of visual stimulus as a single EEG test. The baseline correction is to use the average signal within a preset time period before the stimulus presentation as the baseline signal. The repeated averaging is to average the results of multiple EEG tests corresponding to the same visual stimulus. The image data is resized and input into a pre-trained image encoder to extract image features, thereby obtaining a visual semantic representation corresponding to the EEG signal; Adaptive Mixup data augmentation was performed on EEG signals and image features respectively. EEG signals were enhanced by mixing in the time and frequency domains respectively, while image features were enhanced by linear mixing in the image embedding space. The time-domain and frequency-domain enhancement of the EEG signals includes: selecting first and second EEG image sample pairs of different categories from the training data, and performing time-domain mixing and frequency-domain mixing respectively based on the mixing weights. in, and These represent different categories of electroencephalogram (EEG) signals. This indicates that a Fast Fourier Transform is performed on the EEG signal. Indicates mixed weights; This represents the EEG signal after time-domain mixing; This represents the spectral characteristics of the electroencephalogram (EEG) after frequency domain mixing; The time-domain mixing results and the frequency-domain mixing results are weighted and fused to obtain the enhanced EEG data: in, This indicates enhanced EEG data. This represents the inverse fast Fourier transform function. and These represent the weighting coefficients corresponding to the time-domain mixing result and the frequency-domain mixing result, respectively; The linear blending enhancement of the image features includes: in, and These represent different categories of visual stimulus images. This represents the feature extraction function of the image encoder. Indicates the enhanced image features; The mixed weight It is obtained by sampling from the Beta distribution and adaptively adjusted according to the category distribution of the training samples and the energy of the EEG signals.

[0010] Furthermore, the EEG encoder is a NeuroFormer encoder, which includes a graph attention module, a channel attention module, a spatiotemporal convolution module, and a multilayer perceptron output layer; The graph attention module is used to construct a channel graph structure for multiple electrode channels corresponding to EEG signals, and learn the spatial dependency relationship between different electrode channels through the graph attention mechanism to obtain the spatial correlation features of EEG signals. The channel attention module is used to adaptively allocate feature weights for different electrode channels based on a self-attention mechanism, thereby enhancing the response characteristics of key electrode channels and suppressing noise channel interference. The spatiotemporal convolution module is used to perform joint convolution operations on the EEG signals in both time and space dimensions to extract the spatiotemporal coupling features of the EEG signals. The multilayer perceptron output layer is used to map the extracted EEG features to a shared feature space to obtain an EEG embedding corresponding to the image embedding. The graph attention module, channel attention module, spatiotemporal convolution module, and multilayer perceptron output layer are cascaded through residual connections.

[0011] Furthermore, the image encoder is a pre-trained CLIP-ViT-H / 14 image encoding model, used to extract visual semantic features from the input image data to obtain the corresponding image embedding; The image encoder keeps its parameters frozen during cross-modal alignment training and provides a unified visual semantic alignment target for the EEG encoder.

[0012] Furthermore, S2 specifically includes: The enhanced EEG data is input into the EEG encoder to obtain the EEG embedding, and the enhanced image data is input into the image encoder to obtain the image embedding. The EEG embedding and the image embedding are then normalized. An intra-batch similarity matrix was constructed based on the normalized EEG embedding and image embedding, and the matching relationship between samples of different modalities was calculated by dot product method. Based on the similarity matrix, a bidirectional cross-modal contrastive learning objective function is constructed, which includes contrast loss from EEG to image and contrast loss from image to EEG: in, This indicates the contrast loss from the EEG to the image direction. This indicates the number of samples in a training batch. Indicates the first A brainwave implant, This indicates the corresponding image embedding. Indicates the first batch Image embedding, This represents the similarity calculation function. Indicates the temperature coefficient; in, This indicates the contrast loss from the image to the EEG direction. Indicates the first A brainwave implant; A total loss function is constructed based on the contrast loss from EEG to image direction and the contrast loss from image to EEG direction: in, This represents the total loss function for cross-modal alignment. The parameters of the EEG encoder are updated by backpropagation using the total loss function, and the EEG encoder is trained by gradient descent.

[0013] Furthermore, S3 specifically includes: The EEG signal to be identified is input into the trained EEG encoder for feature mapping to obtain the corresponding EEG embedding. Based on the similarity matching between the EEG embedding and the image embedding prototype corresponding to the test category in the shared feature space, the image embedding prototype is a category-level semantic representation obtained by encoding the image features of the test category by an image encoder; The cosine similarity between the EEG embedding and the prototype image embeddings of each test category is calculated. The cosine similarity is calculated by normalizing the similarity through vector inner product and combining vector norm. The image corresponding to the test category with the highest similarity is embedded into the category corresponding to the prototype as the final classification result, thereby achieving zero-shot classification based on EEG embedding. In the zero-shot classification process, EEG training samples of the test category are not used; instead, category generalization matching is performed solely based on the shared feature space learned during the training phase.

[0014] Furthermore, the generation of image semantic representation using a diffusion model conditioned on the EEG embedding of the EEG signal to be identified specifically includes: The EEG is embedded into the conditional encoding module of the pre-trained diffusion model to obtain the conditional semantic guidance vector, which is represented as: in, This represents the EEG embedding output by the EEG encoder; Represents a conditional mapping network; Represents the conditional semantics guiding vector; The diffusion model uses the visual semantic distribution in the image embedding space as the learning target. Based on the conditional semantic guidance vector, the diffusion denoising process is conditionally constrained, so that the diffusion model gradually recovers the semantic embedding of the target image during the iterative denoising process. In the diffusion model In the step-by-step denoising process, the noise residual is jointly estimated based on the current noise state and the conditional semantic guidance vector, and the image semantic embedding is generated through stepwise optimization. The semantic embedding output by the diffusion model is used as an image semantic representation to represent the high-level semantic information corresponding to the EEG embedding. The image semantic representation is located in the same semantic space as the image encoder output, ensuring cross-stage alignment with subsequent structure generation modules.

[0015] Furthermore, the step of mapping the EEG embedding of the EEG signal to be identified to the latent space generation structure sketch of the diffusion model autoencoder specifically includes: Image latent space representation is obtained based on a pre-trained Stable Diffusion autoencoder, and the image latent space representation is used to characterize the structural information of the image. A regression mapping network is constructed to embed EEG into the latent space of an image, and the EEG embedding is used as input features to output the corresponding latent space structure vector. The regression mapping network is trained by minimizing the reconstruction error between the EEG embedding and the image latent space representation, thereby establishing a one-to-one mapping relationship between the EEG embedding and the structural latent variables. The structural latent vectors are input into the Stable Diffusion decoder for preliminary decoding to obtain a structural sketch. The structural sketch is used to represent the low-frequency structural information of the target image, including the object outline, spatial layout, and color block distribution, and is used to provide structural prior constraints for subsequent diffusion reconstruction.

[0016] Furthermore, the process of fusing the image semantic representation with the structural sketch and inputting it into a cascaded diffusion model, and then combining the classification results to refine the image structure and complete its semantics to generate the final reconstructed image, specifically includes: The image semantic representation and the structural sketch are respectively feature-encoded to obtain semantic condition vector and structural condition vector; A joint conditional vector is constructed based on feature splicing or weighted fusion, and the joint conditional vector is input into the first-stage UNet network of the cascaded diffusion model to generate the initial reconstructed image. The initial reconstructed image and the category semantic code corresponding to the classification result are jointly input into the second stage UNet network of the cascaded diffusion model, wherein the category semantic code is obtained by encoding the category label text or the category prototype image by a pre-trained image encoder; The second-stage UNet network iteratively denoises and optimizes details of the initial reconstructed image, thereby correcting image structural errors and completing semantic information. The final output is a reconstructed image after multi-stage optimization.

[0017] Compared with the prior art, the present invention has the following advantages: (1) Existing EEG visual decoding methods typically rely on temporal modeling or simple convolutional structures to extract features from EEG signals. They lack sufficient modeling of the spatial topological relationships between different EEG electrodes and the neural activity associations across brain regions. This results in the inability to fully utilize the spatial dependency information in EEG signals, leading to problems such as insufficient feature expression ability, weak cross-category semantic discrimination ability, and poor stability of reconstruction results. This invention constructs a NeuroFormer encoder, which includes a graph attention module, a channel attention module, and a spatiotemporal convolution module, to construct the EEG electrode channels as a graph structure. It combines graph attention mechanism, self-attention mechanism, and spatiotemporal joint convolution to perform spatial and temporal joint modeling of EEG signals. This achieves the collaborative extraction of functional associations and spatiotemporal dynamic features between EEG channels, thereby effectively enhancing the expressive ability of EEG features, improving the separability between EEG patterns corresponding to different visual semantics, and further improving zero-shot classification accuracy and semantic consistency and structural stability in the image reconstruction process.

[0018] (2) Existing EEG image generation methods typically use EEG features to apply single-stage conditional constraints to the diffusion model, lacking a hierarchical modeling mechanism for image semantic and structural information. This leads to problems such as semantic shift, object structure distortion, spatial layout disorder, and missing texture details in the generated images, making it difficult to simultaneously ensure high-level semantic consistency and low-level structural accuracy. This invention constructs a multi-stage image reconstruction process, using a conditional diffusion model to generate image semantic representations and latent space mapping to generate structural sketches, and further using a cascaded diffusion model for structural refinement and semantic completion. This achieves phased decoupling modeling of image semantic and structural information, enabling high-level visual semantics and low-level structural contours to be optimized and constrained respectively. This effectively improves the structural integrity, spatial layout accuracy, and semantic fidelity of the reconstructed images, further improving the image generation quality in complex visual scenes.

[0019] (3) Most existing EEG visual decoding methods rely on a limited number of EEG-image paired samples for supervised training. When the training category and the test category are inconsistent, the model is prone to feature distribution shift and decreased category generalization ability, resulting in a significant reduction in classification and image reconstruction performance in zero-shot scenarios. This invention constructs a shared feature space between EEG embedding and image embedding and adopts a bidirectional cross-modal contrastive learning strategy based on InfoNCE loss to perform unified semantic alignment between EEG modality and visual modality. This enables the model to learn high-level semantic feature representations shared across categories, thereby achieving effective matching between EEG signals and unknown category visual semantics without relying on EEG training data of the test category. This improves the model's category generalization ability and cross-modal transfer ability in zero-shot scenarios.

[0020] (4) Due to low spatial resolution, high noise, and significant differences among subjects, EEG signals in the prior art are prone to overfitting, unstable feature distribution, and insufficient robustness across samples during model training, especially when the training samples are limited. This invention performs experimental segmentation, baseline correction, downsampling, and repeated averaging on EEG signals, and further combines an adaptive Mixup enhancement strategy that integrates the time and frequency domains. This introduces time-domain dynamic perturbation and frequency-domain feature fusion at the EEG signal level, while simultaneously performing embedded spatial linear mixing enhancement on image features. This effectively expands the distribution of training samples, reduces the impact of EEG noise and individual differences on model training, improves the model's adaptability to complex EEG distributions and training stability, and further enhances robustness and generalization performance under limited data conditions.

[0021] (5) Existing EEG image generation methods typically rely on a single visual generation path and lack independent constraints on prior information about image structure. This results in generated images that, while semantically similar to the target category in some areas, are prone to instability in terms of overall structural layout, object contours, and color distribution. This invention maps EEG embeddings to the latent space of a Stable Diffusion autoencoder and uses latent space structure vectors to generate structural sketches. This allows the model to explicitly learn low-frequency structural information and spatial layout priors of the image before image generation, thereby providing stable structural constraints for image reconstruction during subsequent diffusion generation. This effectively reduces the structural drift problem in generated images and improves spatial consistency and visual realism in complex target scenes.

[0022] (6) Existing cross-modal semantic alignment methods typically employ unidirectional feature mapping or simple feature matching, which can easily lead to insufficient alignment constraints between EEG features and visual features, thereby affecting the accuracy of EEG semantic decoding and subsequent image reconstruction. This invention achieves bidirectional semantic consistency constraints by simultaneously constructing bidirectional contrastive loss functions for both "EEG to image" and "image to EEG," and by constructing a cross-modal similarity matrix based on intra-batch samples. This enables EEG embeddings and image embeddings to perform more stable clustering and separation in a shared semantic space, thereby improving the accuracy of cross-modal feature matching, reducing semantic confusion between different categories, and further enhancing the accuracy of zero-shot classification and image reconstruction.

[0023] (7) Existing EEG image generation methods typically only utilize EEG features for conditional guidance during the diffusion generation stage, resulting in insufficient utilization of category-level semantic information. This can easily lead to problems such as missing semantic details or unclear category features in the generated images. This invention introduces category semantic encoding based on classification results into the second-stage UNet network of the cascaded diffusion model and performs iterative refinement in conjunction with the initial reconstructed image. This enables the model to further utilize high-level category-level semantic information to complete and correct image texture, local structure, and visual details, thereby effectively enhancing the category consistency, detail realism, and visual recognizability of the generated images. Attached Figure Description

[0024] Figure 1 This is a flowchart of the multi-stage image reconstruction method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the framework for visual decoding and image reconstruction based on electroencephalography (EEG) according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the graph attention-enhanced EEG encoder NeuroFormer according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the MindCascade image reconstruction result, a multi-stage image reconstruction process according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] Example 1: This embodiment provides a multi-stage image reconstruction method based on electroencephalogram (EEG) visual semantic extraction, such as... Figure 1 , Figure 2 As shown, it includes the following steps: S1. Obtain the public dataset, preprocess the EEG signals and corresponding image data in the public dataset, and perform data augmentation processing respectively to obtain the augmented EEG data and the augmented image data. In one specific implementation, the publicly available EEG-visual dataset ThingsEEG is obtained. The ThingsEEG dataset is constructed based on the Rapid Serial Visual Presentation (RSVP) paradigm and contains EEG signals from multiple subjects viewing natural scene images, along with corresponding visual stimulus images. The image categories corresponding to the training and test sets in the dataset are completely mutually exclusive. During the training phase, only the training categories are used to complete cross-modal feature alignment, while during the testing phase, the untrained categories are used to complete zero-shot classification and image reconstruction, thereby verifying the model's generalization ability under unknown category conditions. This data partitioning method can avoid overfitting the model to fixed categories and improve the practical application capability of EEG-visual semantic decoding.

[0027] The raw EEG signals were first processed by segmentation, with the presentation of the visual stimulus as the zero point, and a time window of 0–1000 ms after the stimulus was selected as a single EEG test. Since the neural response evoked by visual stimulus is mainly concentrated in the short period after stimulus, selecting this time range can preserve the visual cognition-related EEG activity more completely. Subsequently, baseline correction was performed, using the average potential in the 200 ms period before stimulus presentation as the baseline signal, and offset correction was performed on the entire test signal to eliminate the influence of DC drift and low-frequency noise between different tests, thereby improving the consistency of EEG responses between different tests.

[0028] Furthermore, the EEG signal sampling rate was downsampled from 1000 Hz to 250 Hz. Since the main effective frequency band of visually evoked EEG signals is concentrated in the low-frequency region, downsampling still preserves the main neural activity information, while effectively reducing data redundancy and model training complexity, and improving the efficiency of subsequent network training. Subsequently, the average of 4 or 80 repeated trials corresponding to the same image stimulus was performed to reduce random noise and transient artifact interference, improve the signal-to-noise ratio, and obtain a more stable representation of the neural response. The EEG signal after repeated averaging more closely approximates the subject's true evoked response to visual stimuli, which is beneficial for subsequent semantic feature extraction.

[0029] When preprocessing visual image data, the image size is first uniformly adjusted to the input resolution corresponding to the CLIP-ViT-H / 14 model to ensure that the visual input matches the pre-trained model structure. Then, a fixed-weight CLIP-ViT-H / 14 image encoder is used to extract high-dimensional semantic features from the images. The CLIP model has already been pre-trained on large-scale image and text data and has strong visual semantic expression capabilities. Therefore, freezing its parameters can maintain the stability of the visual semantic space and avoid degradation of the visual feature space due to the limited scale of EEG data.

[0030] To improve the robustness of model training under limited data conditions, adaptive Mixup data augmentation was performed on both the EEG and image modalities. The EEG modality employed a data-level augmentation approach, simultaneously performing hybrid augmentation in the time and frequency domains; the image modality used a feature-level augmentation approach, performing linear interpolation mixing in the CLIP image embedding space. Compared to traditional single time-domain augmentation, the simultaneous introduction of frequency-domain augmentation further preserves the spectral distribution characteristics of EEG signals, enabling the augmented EEG samples to possess both temporal dynamic information and frequency-domain rhythmic features, thereby improving the model's ability to learn complex EEG patterns.

[0031] Specifically, two pairs of EEG-image samples from different categories are randomly selected from the training data. and And sample the mixed weights from the Beta distribution: in, Indicates the sample mixture weights; This represents the Beta distribution parameter, used to control the sample mixing intensity. When When the size is small, the augmented sample is closer to the original sample; when As the number of samples increases, the mixing degree between different categories improves. Using a Beta distribution to generate mixing coefficients ensures a continuous transition of augmented samples between different categories, thereby improving the model's generalization ability to class boundaries.

[0032] Subsequently, EEG signal mixing was performed in both the time and frequency domains: After performing a Fourier transform on the EEG signal, frequency domain mixing is then performed. in, and These represent different categories of electroencephalogram (EEG) signals. This indicates that a Fast Fourier Transform is performed on the EEG signal. Indicates mixed weights; This represents the EEG signal after time-domain mixing; This represents the EEG spectral characteristics after frequency domain mixing. Temporal domain mixing can enhance the continuity of EEG waveform changes and improve the model's adaptability to different temporal dynamic patterns. Frequency domain enhancement can maintain the statistical properties between different EEG rhythm components and improve the model's robustness to frequency pattern changes. Since visual cognitive activity is closely related to frequency band changes such as alpha and beta waves, performing enhancement in the frequency domain is beneficial for improving neurosemantic expression capabilities.

[0033] After completing the time-domain and frequency-domain enhancements, the frequency-domain features are restored to the time-domain using the inverse fast Fourier transform, and then weighted and fused with the time-domain enhancement results. in, This indicates enhanced EEG data. This represents the inverse fast Fourier transform function. and These represent the weighting coefficients corresponding to the time-domain mixing result and the frequency-domain mixing result, respectively; in this embodiment... ; By jointly fusing the time-domain and frequency-domain enhancement results, the waveform and spectral structures of EEG signals can be preserved simultaneously, avoiding information loss caused by a single enhancement method. The dual-pathway fusion strategy achieves a balance between temporal continuity and frequency stability in the enhanced samples, which is beneficial for improving the model training stability under cross-subject conditions.

[0034] Image modalities are augmented at the feature level in the CLIP semantic embedding space: in, and These represent different categories of visual stimulus images. This represents the feature extraction function of the image encoder. Indicates the enhanced image features; Since the enhancement operation is performed in the semantic embedding space, it can maintain the continuity of visual semantics and avoid the structural distortion problem caused by direct pixel-level mixing.

[0035] Linear blending is performed synchronously with the corresponding tags: in, and These represent the category labels corresponding to the two samples; This represents the enhanced soft label. Using soft label training can reduce the model's over-reliance on a single class and improve the smoothness of classification boundaries.

[0036] Furthermore, to enhance the adaptive capabilities of data augmentation, adaptive MixRatio and adaptive Adjustment strategy. The adaptive MixRatio dynamically adjusts the mixing ratio based on the class distribution in the current batch. When the class distribution in the batch is unbalanced, it increases the mixing probability of minority class samples to alleviate the class bias problem; adaptive... The enhancement intensity is dynamically adjusted based on the mean square energy of the EEG signal. When the EEG signal energy is high, the blending degree is appropriately reduced to avoid excessive perturbation of high-confidence neural responses; when the EEG signal energy is low, the blending degree is increased to enhance the feature representation ability of weak signal samples. Through the above adaptive enhancement mechanism, the model's adaptability to complex EEG patterns and limited sample conditions can be improved, further enhancing cross-modal semantic alignment and subsequent image reconstruction performance.

[0037] S2, input the enhanced EEG data into the EEG encoder to extract the EEG embedding, input the enhanced image data into the image encoder to extract the image embedding, and perform shared feature space alignment training on the EEG embedding and image embedding based on cross-modal contrastive learning to obtain the trained EEG encoder; In one specific implementation, the EEG encoder is a NeuroFormer encoder, such as... Figure 3 As shown, the NeuroFormer neural encoder includes: a graph attention (GA) module, a channel attention (CA) module, a temporal-spatioconvolution (TSConv) module, and a multilayer perceptron (MLP) output layer; each module uses a residual structure. The NeuroFormer encoder employs a combination of graph structure modeling and spatiotemporal joint feature extraction to enhance its ability to express the spatial topological relationships and temporal dynamic features of EEG signals. Since EEG signals originate from the collaborative activity of neuronal populations in different brain regions, processing individual electrode channels alone is insufficient to effectively describe the functional connectivity between brain regions. Therefore, a graph attention mechanism is introduced during the encoding stage to model the spatial dependencies between electrodes.

[0038] Specifically, each EEG electrode is treated as a node in a graph structure, and a fully connected channel graph is constructed using GATConv. The graph attention module adaptively establishes information propagation relationships between different brain regions by learning the attention weights between different nodes.

[0039] After extracting the spatial features of the graph structure, the importance of different EEG channels is adaptively weighted using a channel attention module. Since not all EEG channels contribute equally to visual semantic expression under different visual stimulation conditions, the channel attention mechanism can highlight the responses of key brain regions and reduce interference from noisy channels.

[0040] After completing the spatial dependency modeling, the spatiotemporal convolution module is further used to extract the temporal dynamic patterns of EEG signals. EEG signals are essentially temporal neural activity data, with different time periods corresponding to different cognitive processing stages. Therefore, modeling only spatial relationships is insufficient to fully represent the visual cognitive process. The spatiotemporal convolution module performs convolution operations simultaneously in both the temporal and spatial dimensions, extracting the dynamic changes in EEG signals through local receptive fields.

[0041] Subsequently, the features processed by graph attention, channel attention, and spatiotemporal convolution are input into the output layer of the multilayer perceptron and mapped into a 1024-dimensional shared semantic space to generate EEG embedding features consistent with the CLIP image embedding dimension. A unified embedding dimension ensures that EEG features and visual semantic features are similar to each other within the same semantic space, improving cross-modal semantic alignment capabilities.

[0042] The image modality employs a pre-trained CLIP-ViT-H / 14 model as the image encoder to extract high-dimensional visual semantic embeddings from the input images. The CLIP model has already been pre-trained on large-scale image and text data and possesses strong open semantic representation capabilities; therefore, its parameters are frozen during training, and only the EEG encoder is trained. Freezing the visual encoder parameters maintains a stable visual semantic space distribution, avoiding drift in the visual semantic space due to the limited scale of EEG data, thus providing a stable and reliable semantic alignment target for EEG features.

[0043] Subsequently, an intra-batch similarity matrix was constructed based on the normalized EEG embeddings and image embeddings, and the semantic matching relationship between samples of different modalities was calculated using a dot product approach. The contrast loss corresponding to the EEG-to-image direction is expressed as: in, This indicates the contrast loss from the EEG to the image direction. This indicates the number of samples in a training batch. Indicates the first A brainwave implant, This indicates the corresponding image embedding. Indicates the first batch Image embedding, This represents the similarity calculation function. Indicates the temperature coefficient; The contrast loss from the corresponding image to the EEG direction is represented as: in, This indicates the contrast loss from the image to the EEG direction. Indicates the first A brainwave implant; Bidirectional contrastive learning can simultaneously constrain the semantic consistency in both directions, namely "EEG → image" and "image → EEG", so that the distance between corresponding sample pairs in the shared semantic space continues to approach, while the distance between samples of different categories gradually increases, thereby enhancing cross-modal semantic discrimination ability.

[0044] Finally, construct the total loss function: in, This represents the total loss function for cross-modal alignment. The parameters of the EEG encoder are updated by backpropagation using the total loss function, and the EEG encoder is trained by gradient descent.

[0045] The image encoder uses a pre-trained CLIP-ViT-H / 14 model to extract high-dimensional semantic image embeddings, while keeping the weights frozen to stabilize the visual feature distribution.

[0046] Cross-modal contrastive learning includes the following steps: The enhanced EEG data is input into the EEG encoder to obtain the EEG embedding, and the enhanced image data is input into the image encoder to obtain the image embedding. The EEG embedding and the image embedding are then normalized. An intra-batch similarity matrix was constructed based on the normalized EEG embedding and image embedding, and the matching relationship between samples of different modalities was calculated by dot product method. A bidirectional cross-modal contrastive learning objective function is constructed based on the similarity matrix, which includes contrast loss from EEG to image and contrast loss from image to EEG: in, This indicates the contrast loss from the EEG to the image direction. This indicates the number of samples in a training batch. Indicates the first A brainwave implant, This indicates the corresponding image embedding. Indicates the first batch Image embedding, This represents the similarity calculation function. Indicates the temperature coefficient; in, This indicates the contrast loss from the image to the EEG direction. Indicates the first A brainwave implant; A total loss function is constructed based on the contrast loss from EEG to image and the contrast loss from image to EEG: in, This represents the total loss function for cross-modal alignment. The parameters of the EEG encoder are updated by backpropagation using the total loss function, and the EEG encoder is trained by gradient descent.

[0047] S3, input the EEG signal to be identified into the trained EEG encoder for feature mapping to obtain the corresponding EEG embedding, and perform zero-shot classification based on the similarity between the EEG embedding of the EEG signal to be identified and the image features of the test category to obtain the corresponding classification result; In one specific implementation, the trained EEG encoder shares a unified semantic feature space with the image encoder. Any input EEG signal can be mapped to a corresponding EEG embedding, and zero-shot classification is achieved through semantic similarity with the prototype features of the test category image. Since the categories in the training set and the test set are independent of each other, the classification process does not require the use of EEG training samples corresponding to the test category, thus enabling cross-modal semantic inference of unknown categories.

[0048] In practice, the EEG signal to be identified is input into the trained NeuroFormer encoder, and the spatiotemporal neural response features are extracted layer by layer through the graph attention module, channel attention module and spatiotemporal convolution module. The EEG embedding is then mapped to the shared semantic space through the output layer of the multilayer perceptron. Image prototype features for the test category are obtained by encoding images of the corresponding category using a pre-trained CLIP-ViT-H / 14 image encoder. To reduce the impact of fluctuations in individual samples, mean aggregation is performed on multiple image embeddings within the same category to obtain category-level semantic prototypes. in, Indicates the first Image prototype embedding; Indicates the first The number of images corresponding to the class; Indicates the first The first in the class Image; This represents the feature extraction function of the image encoder.

[0049] Then, the cosine similarity between the EEG embeddings and the image prototypes of each category was calculated: in, Indicates the first The first EEG sample and the first Similarity between class image prototypes; This represents the vector dot product operation. Since normalization has been performed, cosine similarity can more stably reflect cross-modal semantic consistency, reducing the impact of differences in feature scales across different modalities on classification results.

[0050] The test categories include visual categories such as animals, plants, vehicles, furniture, electronic devices, food, and natural scenes. Different categories correspond to different image semantic prototypes. Each category did not participate in the EEG encoder parameter optimization during the training phase, in order to achieve EEG semantic recognition under zero-shot conditions.

[0051] Ultimately, the category with the highest similarity is used as the prediction result corresponding to the EEG signal; In this implementation, the shared feature space is trained through cross-modal contrastive learning, ensuring that EEG embeddings and visual semantic embeddings maintain a consistent semantic topological relationship in a high-dimensional space. Compared to traditional supervised classification methods that rely on fixed-category training data, this method leverages the category generalization ability in the visual semantic space to recognize EEG data from categories not included in the training, thereby improving the model's generalization performance and cross-category transfer ability in zero-shot scenarios. Simultaneously, by using category prototype aggregation, the semantic shift caused by individual image differences is reduced, improving the stability and robustness of zero-shot classification results.

[0052] S4, performing multi-stage image reconstruction based on the EEG embedding and classification results of the EEG signal to be identified, including: generating image semantic representation using a diffusion model conditioned on the EEG embedding of the EEG signal to be identified; mapping the EEG embedding of the EEG signal to be identified to the latent space of the diffusion model autoencoder to generate a structural sketch; fusing the image semantic representation and the structural sketch and inputting them into the cascaded diffusion model, and combining the classification results to refine the structure and complete the semantics of the image, generating the final reconstructed image.

[0053] In one specific implementation, the multi-stage image reconstruction employs the MindCascade cascade generation framework. Through a step-by-step reconstruction approach of "semantic generation—structural restoration—detail refinement," it jointly models high-level visual semantic information and low-level structural information from EEG signals, thereby improving the semantic consistency and structural realism of the reconstructed image. Compared to single-stage direct generation, this multi-stage reconstruction strategy can reduce the impact of EEG signal noise on image generation stability and improve image fidelity in complex scenes.

[0054] S4.1, using a diffusion model conditioned on EEG embeddings, learns the semantic embedding distribution of images to obtain preliminary image representations.

[0055] In practice, the EEG data output from the NeuroFormer encoder is embedded into the input conditional diffusion model, and a conditional mapping network is used to generate semantic conditional vectors. in, This represents the EEG embedding output by the EEG encoder; Represents a conditional mapping network; Represents the conditional semantics guiding vector; Subsequently, using the image semantic embedding extracted by the CLIP image encoder as the learning objective of the diffusion model, a conditional probability distribution between EEG semantics and visual semantics was established: in, Represents image semantic embedding; This represents the probability distribution for generating image semantic embeddings given EEG embedding conditions.

[0056] The diffusion model progressively adds Gaussian noise to the image semantic embedding during the forward diffusion stage, and progressively recovers the target semantic features based on EEG conditional semantic vectors during the reverse denoising stage. In the first... In the denoising process, the diffusion model predicts the noise residual through the UNet network. The diffusion model is trained by minimizing the mean square error between the predicted noise and the actual noise. The conditional diffusion approach allows EEG features to progressively constrain the image semantic generation process. Compared to direct regression, this effectively alleviates the semantic drift problem caused by high noise levels in EEG signals and improves the consistency between the generated semantic representation and the actual visual content. The final output image semantic embedding serves as a preliminary image representation for subsequent structure generation and image refinement stages.

[0057] S4.2, EEG embedding is mapped to the latent space of the Stable Diffusion autoencoder to generate a structural sketch.

[0058] In practice, a pre-trained Stable Diffusion autoencoder is used to encode the latent space of real images to obtain latent variables of the images. Then, a regression mapping network between EEG embedding and the latent space of the images is constructed. The corresponding latent space structure vector is predicted through EEG embedding and input into the Stable Diffusion decoder for image decoding to generate a structural sketch.

[0059] The structural sketch mainly contains low-frequency structural information of the target image, including the target outline, spatial layout, subject position, and color block distribution. Since EEG signals are highly sensitive to changes in overall visual structure, latent spatial structure mapping can prioritize the recovery of the overall image structure, providing a stable structural prior for subsequent image refinement stages, thereby reducing structural distortion problems that occur during diffusion generation.

[0060] S4.3 uses the fusion of image semantic embedding and structural sketch as the conditional input to the first UNet of the cascaded diffusion model, and the classification result as the conditional input to the second UNet, guiding the generation of the final high-quality reconstructed image.

[0061] In practice, the semantic representation of the image generated by S4.1 and the structural sketch generated by S4.2 are feature-encoded, and joint conditional features are constructed by feature splicing or weighted fusion. Joint conditional features are used as conditional inputs to the first-level UNet network to guide the diffusion model in generating the initial reconstructed image. Since semantic noise, texture loss, and local structural blurring may still exist in the initial reconstruction stage, semantic refinement is further performed using the classification results obtained in stage S3.

[0062] In practice, a pre-trained CLIP encoder is used to semantically encode the classification labels, and the initial reconstructed image and category semantic features are jointly input into the second-level UNet network to further optimize the image structure and semantic details. The classification labels provide explicit category semantic constraints, which helps to enhance the texture consistency, color consistency, and local detail representation of the target object.

[0063] The final output is a reconstructed image optimized through multiple stages. The entire reconstruction process integrates EEG semantic features, latent space structural features, and category semantic information to achieve joint constraints from high-level semantics to low-level structure. This ensures the overall structural accuracy of the image while improving the semantic consistency, clarity, and visual realism of the generated image.

[0064] Example 2: This embodiment aims to verify the effectiveness of the proposed EEG visual semantic extraction and multi-stage image reconstruction method on a large public dataset, including zero-shot classification performance and EEG-driven image reconstruction performance.

[0065] In this example, the publicly available ThingsEEG dataset was used for performance validation. This dataset was acquired using the RapidSerial Visual Presentation (RSVP) paradigm and includes EEG recordings of 10 subjects viewing natural images. The training set contains 1654 semantic concepts, with 10 images per category, displayed 4 times; the test set contains 200 semantic concepts, with 1 image per category, displayed 80 times. Each image is presented for 100ms, with a 100ms blank screen between images. A 64-channel EASYCAP device was used, with 63 channels reserved for analysis, and a sampling rate of 1000Hz. The EEG signals from this dataset were preprocessed as follows: first, the period from 0 to 1000ms after stimulation was truncated; then, baseline correction was performed using the 200ms period before stimulation; then, the 1000Hz was downsampled to 250Hz; finally, the average of repeated trials of the same image was calculated to improve the signal-to-noise ratio (SNR). The images from this dataset were input into a pre-trained CLIP-ViT-H-14 image encoder to extract 1024-dimensional semantic embeddings.

[0066] This embodiment is based on the PyTorch framework and was trained and tested on a single A6000 GPU. The experimental details are as follows: 740 training samples were randomly selected as the validation set during training; the optimizer used was Adam with an initial learning rate of 0.0002, β1 and β2 of 0.5 and 0.999 respectively, and a batch size of 64; the model weights were saved when the validation loss remained optimal for 40 consecutive rounds during training; the image reconstruction part was implemented using the StableDiffusionv1-5-pruned-emaonly model.

[0067] On the 200-class zero-shot classification task of the ThingsEEG dataset, the EEG encoder NeuroFormer of this invention achieved a Top-1 accuracy of 34.90% and a Top-5 accuracy of 67.60%, which is a significant performance improvement compared to the random baseline (Top-1: 0.5%, Top-5: 2.5%).

[0068] Table 1. Average accuracy of NeuroFormer encoder compared to existing methods for 200-channel zero-shot classification with 10 subjects. NeuroFormer demonstrated superior accuracy across multiple subjects, indicating that it possesses stronger cross-modal alignment and cross-subject generalization capabilities.

[0069] Table 2 shows the results of the ablation experiment. Experimental results show that using adaptive Mixup improved Top-1 classification accuracy by 2.25% and Top-5 accuracy by 2.85%, indicating that the adaptive Mixup strategy effectively enhances the model's generalization ability by increasing the diversity of training data. Further analysis of the roles of each module in NeuroFormer reveals that the GA and CA modules improve the model's perception of spatial topology and channel correlation, while the TSConv module is responsible for capturing key local spatiotemporal features. These three modules together constitute a complete spatiotemporal modeling system, playing an indispensable role in improving EEG decoding performance.

[0070] like Figure 4 This paper presents the partial image reconstruction results of MindCascade on the ThingsEEG dataset for subject 8.

[0071] The results demonstrate that MindCascade can successfully reconstruct coarse-grained category information (such as lambs, sailboats, and omelets) while preserving fine-grained semantic details (such as the lamb's posture, the sailboat's background, and the omelet's shape). Compared to the ground truth (GT) image, the generated image exhibits high consistency in color, shape, and semantics, validating the effectiveness of MindCascade in EEG-guided image generation.

[0072] Table 3 presents the quantitative assessment of the image reconstruction quality for subject 8. Experimental results show that although MindCascade performs slightly worse than other methods on low-level visual fidelity metrics (such as PixCorr), it outperforms these methods on high-level visual features such as CLIP and Inception scores, demonstrating MindCascade's advantage in high semantic consistency.

[0073] The experiments in this example fully verify that: the NeuroFormer encoder proposed in this invention can efficiently extract spatiotemporal features and achieve high accuracy in zero-shot classification tasks; the MindCascade multi-stage image reconstruction method proposed in this invention can effectively recover visual information in EEG and generate images with high semantic consistency; and the adaptive Mixup and multimodal contrastive learning framework improve cross-modal semantic alignment and generalization capabilities.

[0074] This example demonstrates that the method of the present invention can effectively extract visual semantics from EEG signals and reconstruct corresponding visual images under unsupervised and unknown category conditions, and has important practical application and scientific research value.

[0075] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-stage image reconstruction method based on electroencephalogram (EEG) visual semantic extraction, characterized in that, Includes the following steps: S1. Obtain the public dataset, preprocess the EEG signals and corresponding image data in the public dataset, and perform data augmentation processing respectively to obtain the augmented EEG data and the augmented image data. S2, input the enhanced EEG data into the EEG encoder to extract the EEG embedding, input the enhanced image data into the image encoder to extract the image embedding, and perform shared feature space alignment training on the EEG embedding and the image embedding based on cross-modal contrastive learning to obtain the trained EEG encoder; S3, input the EEG signal to be identified into the trained EEG encoder for feature mapping to obtain the corresponding EEG embedding, and perform zero-shot classification based on the similarity between the EEG embedding of the EEG signal to be identified and the image features of the test category to obtain the corresponding classification result; S4, performing multi-stage image reconstruction based on the EEG embedding of the EEG signal to be identified and the classification result, including: generating an image semantic representation using a diffusion model conditioned on the EEG embedding of the EEG signal to be identified; mapping the EEG embedding of the EEG signal to be identified to the latent space of the diffusion model autoencoder to generate a structural sketch; fusing the image semantic representation and the structural sketch and inputting them into a cascaded diffusion model, and combining the classification result to refine the structure and complete the semantics of the image, generating the final reconstructed image.

2. The multi-stage image reconstruction method based on EEG visual semantic extraction according to claim 1, characterized in that, The publicly available dataset is an EEG-visual dataset constructed based on the fast sequence visual presentation paradigm. The EEG-visual dataset includes EEG signals and corresponding image data of multiple subjects during the viewing of natural images. The image data consists of visual stimulus images acquired synchronously with the EEG signals. The EEG-visual dataset includes a training set and a test set, and the image categories in the training set and the test set do not overlap. The training set is used for cross-modal alignment training of the EEG encoder, and the test set is used for zero-shot classification and image reconstruction.

3. The multi-stage image reconstruction method based on EEG visual semantic extraction according to claim 1, characterized in that, The process of preprocessing the EEG signals and corresponding image data in the publicly available dataset, and then performing data augmentation processing to obtain augmented EEG data and augmented image data, specifically includes: The EEG signals are subjected to test segmentation, baseline correction, signal downsampling, and repeated averaging to obtain stable EEG response data. The test segmentation is to extract a preset time window after the presentation of visual stimulus as a single EEG test. The baseline correction is to use the average signal within a preset time period before the stimulus presentation as the baseline signal. The repeated averaging is to average the results of multiple EEG tests corresponding to the same visual stimulus. The image data is resized and input into a pre-trained image encoder to extract image features, thereby obtaining a visual semantic representation corresponding to the EEG signal; Adaptive Mixup data augmentation was performed on EEG signals and image features respectively. EEG signals were enhanced by mixing in the time and frequency domains respectively, while image features were enhanced by linear mixing in the image embedding space. The time-domain and frequency-domain enhancement of the EEG signals includes: selecting first and second EEG image sample pairs of different categories from the training data, and performing time-domain mixing and frequency-domain mixing respectively based on the mixing weights. in, and These represent different categories of electroencephalogram (EEG) signals. This indicates that a Fast Fourier Transform is performed on the EEG signal. Indicates mixed weights; This represents the EEG signal after time-domain mixing; This represents the spectral characteristics of the electroencephalogram (EEG) after frequency domain mixing; The time-domain mixing results and the frequency-domain mixing results are weighted and fused to obtain the enhanced EEG data: in, This indicates enhanced EEG data. This represents the inverse fast Fourier transform function. and These represent the weighting coefficients corresponding to the time-domain mixing result and the frequency-domain mixing result, respectively; The linear blending enhancement of the image features includes: in, and These represent different categories of visual stimulus images. This represents the feature extraction function of the image encoder. Indicates the enhanced image features; The mixed weight It is obtained by sampling from the Beta distribution and adaptively adjusted according to the category distribution of the training samples and the energy of the EEG signals.

4. The multi-stage image reconstruction method based on EEG visual semantic extraction according to claim 1, characterized in that, The EEG encoder is a NeuroFormer encoder, which includes a graph attention module, a channel attention module, a spatiotemporal convolution module, and a multilayer perceptron output layer. The graph attention module is used to construct a channel graph structure for multiple electrode channels corresponding to EEG signals, and learn the spatial dependency relationship between different electrode channels through the graph attention mechanism to obtain the spatial correlation features of EEG signals. The channel attention module is used to adaptively allocate feature weights for different electrode channels based on a self-attention mechanism, thereby enhancing the response characteristics of key electrode channels and suppressing noise channel interference. The spatiotemporal convolution module is used to perform joint convolution operations on the EEG signals in both time and space dimensions to extract the spatiotemporal coupling features of the EEG signals. The multilayer perceptron output layer is used to map the extracted EEG features to a shared feature space to obtain an EEG embedding corresponding to the image embedding. The graph attention module, channel attention module, spatiotemporal convolution module, and multilayer perceptron output layer are cascaded through residual connections.

5. The multi-stage image reconstruction method based on EEG visual semantic extraction according to claim 1, characterized in that, The image encoder is a pre-trained CLIP-ViT-H / 14 image encoding model, which is used to extract visual semantic features from input image data to obtain the corresponding image embedding; The image encoder keeps its parameters frozen during cross-modal alignment training and provides a unified visual semantic alignment target for the EEG encoder.

6. The multi-stage image reconstruction method based on EEG visual semantic extraction according to claim 1, characterized in that, S2 specifically includes: The enhanced EEG data is input into the EEG encoder to obtain the EEG embedding, and the enhanced image data is input into the image encoder to obtain the image embedding. The EEG embedding and the image embedding are then normalized. An intra-batch similarity matrix was constructed based on the normalized EEG embedding and image embedding, and the matching relationship between samples of different modalities was calculated by dot product method. Based on the similarity matrix, a bidirectional cross-modal contrastive learning objective function is constructed, which includes contrast loss from EEG to image and contrast loss from image to EEG: in, This indicates the contrast loss from the EEG to the image direction. This indicates the number of samples in a training batch. Indicates the first A brainwave implant, This indicates the corresponding image embedding. Indicates the first batch Image embedding, This represents the similarity calculation function. Indicates the temperature coefficient; in, This indicates the contrast loss from the image to the EEG direction. Indicates the first A brainwave implant; A total loss function is constructed based on the contrast loss from EEG to image direction and the contrast loss from image to EEG direction: in, This represents the total loss function for cross-modal alignment. The parameters of the EEG encoder are updated by backpropagation using the total loss function, and the EEG encoder is trained by gradient descent.

7. The multi-stage image reconstruction method based on EEG visual semantic extraction according to claim 1, characterized in that, S3 specifically includes: The EEG signal to be identified is input into the trained EEG encoder for feature mapping to obtain the corresponding EEG embedding. Based on the similarity matching between the EEG embedding and the image embedding prototype corresponding to the test category in the shared feature space, the image embedding prototype is a category-level semantic representation obtained by encoding the image features of the test category by an image encoder; The cosine similarity between the EEG embedding and the prototype image embeddings of each test category is calculated. The cosine similarity is calculated by normalizing the similarity through vector inner product and combining vector norm. The image corresponding to the test category with the highest similarity is embedded into the category corresponding to the prototype as the final classification result, thereby achieving zero-shot classification based on EEG embedding. In the zero-shot classification process, EEG training samples of the test category are not used; instead, category generalization matching is performed solely based on the shared feature space learned during the training phase.

8. The multi-stage image reconstruction method based on EEG visual semantic extraction according to claim 1, characterized in that, The generation of image semantic representation using a diffusion model conditioned on the EEG embedding of the EEG signal to be identified specifically includes: The EEG is embedded into the conditional encoding module of the pre-trained diffusion model to obtain the conditional semantic guidance vector, which is represented as: in, This represents the EEG embedding output by the EEG encoder; Represents a conditional mapping network; Represents the conditional semantics guiding vector; The diffusion model uses the visual semantic distribution in the image embedding space as the learning target. Based on the conditional semantic guidance vector, the diffusion denoising process is conditionally constrained, so that the diffusion model gradually recovers the semantic embedding of the target image during the iterative denoising process. In the diffusion model In the step-by-step denoising process, the noise residual is jointly estimated based on the current noise state and the conditional semantic guidance vector, and the image semantic embedding is generated through stepwise optimization. The semantic embedding output by the diffusion model is used as an image semantic representation to represent the high-level semantic information corresponding to the EEG embedding. The image semantic representation is located in the same semantic space as the image encoder output, ensuring cross-stage alignment with subsequent structure generation modules.

9. The multi-stage image reconstruction method based on EEG visual semantic extraction according to claim 1, characterized in that, The step of mapping the EEG embedding of the EEG signal to be identified to the latent space generation structure sketch of the diffusion model autoencoder specifically includes: Image latent space representation is obtained based on a pre-trained Stable Diffusion autoencoder, and the image latent space representation is used to characterize the structural information of the image. A regression mapping network is constructed to embed EEG into the latent space of an image, and the EEG embedding is used as input features to output the corresponding latent space structure vector. The regression mapping network is trained by minimizing the reconstruction error between the EEG embedding and the image latent space representation, thereby establishing a one-to-one mapping relationship between the EEG embedding and the structural latent variables. The structural latent vectors are input into the Stable Diffusion decoder for preliminary decoding to obtain a structural sketch. The structural sketch is used to represent the low-frequency structural information of the target image, including the object outline, spatial layout, and color block distribution, and is used to provide structural prior constraints for subsequent diffusion reconstruction.

10. The multi-stage image reconstruction method based on electroencephalogram visual semantic extraction according to claim 1, characterized in that, The process of fusing the image semantic representation with the structural sketch and inputting it into a cascaded diffusion model, combined with the classification results, to refine the image structure and complete its semantics, generating the final reconstructed image, specifically includes: The image semantic representation and the structural sketch are respectively feature-encoded to obtain semantic condition vector and structural condition vector; A joint conditional vector is constructed based on feature splicing or weighted fusion, and the joint conditional vector is input into the first-stage UNet network of the cascaded diffusion model to generate the initial reconstructed image. The initial reconstructed image and the category semantic code corresponding to the classification result are jointly input into the second stage UNet network of the cascaded diffusion model, wherein the category semantic code is obtained by encoding the category label text or the category prototype image by a pre-trained image encoder; The second-stage UNet network iteratively denoises and optimizes details of the initial reconstructed image, thereby correcting image structural errors and completing semantic information. The final output is a reconstructed image after multi-stage optimization.

Images