Method for reconstructing human visual information according to electroencephalogram signals

Abstract

The invention discloses a method for reconstructing human visual information according to electroencephalogram signals, is applied to the field of artificial intelligence, and aims to solve the problems that an existing visual reconstruction task has a certain progress on the coarse-grained scale, but details of visual stimulation can not be accurately restored by the methods, and the visual stimulation effect is poor. And a large amount of important semantic information of the reconstructed image is lost. According to the method, a new embedding alignment mode is adopted to reduce the parameter quantity of the model and enable the electroencephalogram signal characteristics to better represent fine-grained information of visual stimulation, and meanwhile, a better frequency domain encoder is used to improve the classification accuracy of the model.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence, and in particular relates to a human visual information reconstruction technology. Background Art

[0002] The human brain is an extremely complex and mysterious biological structure. It consists of approximately 86 billion neurons, which are interconnected by trillions of synapses to form a highly complex network. This complex network structure gives humans a series of advanced cognitive functions, including learning, memory, emotion, language, decision-making, and consciousness. The brain's processing power is extremely powerful, capable of processing and integrating a large amount of information from the senses in a very short time and producing sophisticated responses. Brain decoding focuses on interpreting neural activity patterns to understand cognitive and sensory processes, and attempts to reverse restore these processes. In-depth research on brain function can not only improve the present invention's understanding of human beings themselves, but also promote progress in multiple scientific fields, especially medicine and artificial intelligence.

[0003] The study of the human visual system is an important research branch of brain decoding. Although the rapid development of deep learning in recent years has greatly improved the performance of computer vision systems, the human brain still has unparalleled efficiency and robustness in processing complex visual information. The encoding and decoding of visual information are two key links in the human brain's processing of visual data. The encoding process involves converting external visual stimuli, such as light and color, into neural signals in the brain, while decoding is to reconstruct the original visual image from these neural signals. Exploring how the human brain performs these complex visual tasks and trying to simulate these processes is crucial to improving the intelligence level of machine vision systems. By building a model that mimics the visual processing capabilities of the human brain, the present invention can enhance the perceptual intelligence of the machine, making it more efficient and accurate in image recognition, scene understanding, and other vision-related artificial intelligence applications. In the field of visual decoding, researchers use neuroimaging technology to measure the brain's neural response to visual stimuli to reconstruct visual experience. A common visual decoding method is to map brain activity to the latent space of a generative model, which then reconstructs the visual stimuli.

[0004] Visual reconstruction tasks have made some progress at a coarse-grained scale, but these methods often fail to accurately restore the details of visual stimuli, resulting in the loss of a large amount of important semantic information in the reconstructed image. This loss of information limits the understanding of how the brain processes complex visual scenes, and also affects the quality and practicality of the reconstructed image. In addition, these methods usually require a lot of computing resources, which may be an insurmountable obstacle for some research teams.

[0005] Existing visual stimulus reconstruction methods include the following two categories:

[0006] A. Reconstructing visual stimuli based on brain signals

[0007] A common method for decoding the visual process of the brain is to reconstruct the visual stimuli received by the human brain. Most previous studies were limited to classifying visual stimuli and did not actually reconstruct visual stimuli. In the field of image generation, using feature vectors to generate images has become a cutting-edge trend in research. Among them, technologies such as variational autoencoders (VAE) and generative adversarial networks (GAN) have demonstrated the ability to transform from abstract features to specific images. These methods can generate new images that are visually indistinguishable from real images by learning the underlying representation of data. The development of these technologies has provided a basis for visual stimulus reconstruction. Early visual reconstruction tasks mostly used these two models as generative models. However, the generation quality of VAE is poor and it is prone to overfitting when the data set is small. Although GAN has high generation quality, the training process is unstable and it is extremely difficult to find the global optimal solution. The performance of the generative model restricts the development of the task of decoding visual stimuli. It is still challenging to achieve high-fidelity, lightweight models and comprehensive subject visual brain decoding.

[0008] B. Reconstructing visual stimuli using diffusion models

[0009] In recent years, the progress of image generation technology has greatly improved the quality of generated images. As an emerging image generation technology, diffusion models have attracted widespread attention due to their unique advantages. Diffusion models are based on Markov chains and Bayesian principles. They start from a simple noise distribution, gradually introduce structural information, and finally generate complex images that match the target distribution. In the context of conditional generation, diffusion models are able to accept additional conditional information, such as text descriptions, category labels, or other data features, to guide the generation process, thereby producing images consistent with these conditions. The advantage of this method lies in the smoothness and controllability of its generation process, which makes the generated images not only visually realistic, but also closely related to the given conditions in terms of content.

[0010] Applying the conditional generation principle of the diffusion model to brain signals, especially EEG and fMRI features, opens up new possibilities for neuroscience research and brain-computer interface technology. As key technologies for capturing brain activity, EEG and fMRI provide rich neurobiological data that reflect the dynamic changes of the brain under different cognitive tasks and emotional states. By inputting these signals as conditions into the diffusion model, researchers can explore the generation of visual images that match specific neural activity patterns, which not only helps to better understand how the brain processes and encodes information, but also provides new ideas for developing personalized neurorehabilitation and cognitive training applications. (Chen et al., 2022) proposed MinD-vis, which demonstrated the feasibility of using the diffusion model for visual signal reconstruction. Specifically, MinD-vis first effectively learns brain features by imitating the brain's visual encoding through sparse coding mask brain construction, converts fMRI data into sparse coding features with locality constraints, and then uses it as a conditional input to fine-tune the pre-trained latent diffusion model, and finally completes the reconstruction of visual stimuli. (Liu et al., 2023) proposed rainCLIP, a task-independent brain decoding model based on fMRI that can reconstruct visual stimuli with high semantic fidelity, bridging the gap between brain activity, images, and text. (Ozcelik et al., 2023) proposed a two-stage scene reconstruction framework called Brain Diffuser for reconstructing natural images based on fMRI signals. (Sun et al., 2023) proposed a contrast and diffusion framework to decode real images from fMRI recordings, thereby gaining insight into the relationship between diffusion models and the human brain visual system. (Han et al., 2024) proposed MindFormer, a model specifically designed to generate fMRI conditioned feature vectors that can be used to stabilize the conditional generation of diffusion models. However, the cost of fMRI is relatively high, the equipment is expensive to purchase and maintain, and fMRI equipment requires special environmental conditions, such as a room that is protected from magnetic field interference, which limits its availability in certain research and clinical settings.

[0011] EEG is a method of measuring brain activity that records electrical signals generated by brain neurons by placing a series of electrodes on the scalp. These electrical signals reflect electrical activity in the brain and can be used to study brain function, diagnose neurological diseases, and develop brain-computer interfaces. Compared with fMRI, EEG is less expensive and the acquisition equipment is lighter. Therefore, using EEG to reconstruct visual stimuli in the brain shows great potential. DreamDiffusion (Bai et al., 2023) uses CLIP to provide additional supervision, which enables alignment of EEG, text, and image embedding spaces even with limited EEG-image pair data.

[0012] Although all of the above methods have achieved certain results, they are all based on fine-tuning large models to achieve visual stimulus reconstruction. The shortcomings of this method are obvious. First, it eliminates the model's original ability to generate images using text, and this fine-tuning usually requires a lot of computing resources. Secondly, the fine-tuned model is usually not reusable because the EEG prompt function cannot be directly transferred to other custom models derived from the same text-to-image base model. In addition, the new model is usually incompatible with existing structural control tools, which poses a major challenge to downstream applications. Finally, the pre-trained diffusion model has been maturely trained on a large number of datasets, and fine-tuning it with scarce EEG-image data can easily lead to overfitting. Summary of the invention

[0013] In order to solve the above problems, the present invention proposes a method for reconstructing human visual information based on EEG signals, which can achieve better classification accuracy and image reconstruction quality while reducing the number of model parameters.

[0014] The technical solution adopted by the present invention is: a method for reconstructing human visual information based on EEG signals, comprising:

[0015] S0, obtaining an image of the scene observed by the human eye at the same moment, and collecting the EEG sequence corresponding to the human body at that moment;

[0016] S1, using the time-frequency domain encoder to extract the time-domain features and frequency-domain features of the EEG sequence;

[0017] S2, input the splicing results of the time domain features and frequency domain features of the EEG sequence into the semantic classifier to obtain the EEG classification label;

[0018] S3, using the pre-trained IP-Adapter model to generate image feature embedding based on the image of the scene observed by the human eye obtained in step S0;

[0019] S4, based on the image feature embedding obtained in step S3, aligning the splicing results of the time domain features and frequency domain features of the EEG sequence extracted in step S1;

[0020] S5. The time domain features and frequency domain features of the aligned EEG sequence are concatenated together with the EEG classification labels and used as conditions of the cascade diffusion model for multi-level semantic visual reconstruction.

[0021] Beneficial effects of the present invention: The method of the present invention achieves better classification accuracy and image reconstruction quality while having fewer model parameters. Specifically, the present invention adopts a new embedding alignment method to reduce the number of model parameters and enable the EEG signal features to better represent the fine-grained information of visual stimuli, while using a better frequency domain encoder to improve the classification accuracy of the model. The method of the present invention has the following advantages:

[0022] 1. IP-adapter embedding is used to replace CLIP embedding, which reduces the feature dimension and thus reduces the computational cost;

[0023] 2. The improved frequency domain encoder improves the performance of EEG signal classification, thereby enhancing the quality of coarse-grained visual reconstruction;

[0024] 3. In order to verify the method of the present invention, the present invention conducted qualitative and quantitative experiments using the ImageNetEEG dataset, and the results showed the superiority of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of the present invention;

[0026] Figure 2 A qualitative comparison of the reconstruction results of the method of the present invention and the prior art is provided;

[0027] Figure 3 The figure shows the quality comparison of the reconstruction results between the method of the present invention and the prior art. DETAILED DESCRIPTION

[0028] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0029] The present invention proposes a new research method, which directly embeds and aligns brain features with the generation conditions of the diffusion model. The reconstruction of visual stimuli is completed by directly embedding and aligning EEG time-frequency domain features with the coarse-grained and fine-grained semantic features of visual stimuli, which greatly simplifies the scalability and efficiency of the visual reconstruction task.

[0030] like Figure 1As shown, the present invention consists of three parts: an EEG time-frequency domain encoder, an alignment network, and a cascade diffusion model. The time-domain encoder extracts time-domain features by reconstructing the masked signal, and the frequency encoder uses Conv-LSTM to obtain frequency-domain features through fast Fourier transform (FFT). The time-frequency domain features are concatenated and then input into the alignment network. The alignment network is fine-tuned as a whole under the supervision of the IP-Adapter image embedding, which makes it possible to align the EEG features with the fine-grained semantics of the IP-Adapter space. Finally, the aligned EEG fine-grained embedding is used together with the coarse-grained embedding of the classification result as a condition of the cascade diffusion model for multi-level semantic visual reconstruction.

[0031] The following is a detailed description of the implementation process of the three parts:

[0032] 1. EEG time-frequency domain encoder

[0033] Temporal Encoder: Mask pre-training is a self-supervised learning method that trains the model by predicting the masked parts of the input so that it learns rich contextual representations.

[0034] The work of this invention adopts the same time domain encoder structure as DreamDiffusion. Specifically, given an input EEG sequence x, whose shape is (B, C, L), where B is the batch size, C is the number of signal channels, and L is the signal sequence length, it is divided into N signal blocks in the time domain, and each signal block is mapped to the D-dimensional space to obtain Z∈R N×D The mapping is done using one-dimensional convolution, which is done as follows:

[0035] x′=proj(x)

[0036] The kernel and stride size of the one-dimensional convolution are both equal to the signal block size K. The result is a new tensor x′ with shape (B, D, P) where

[0037] Then transpose x′ and put the feature in the last dimension to get:

[0038] Z = transpose(x′)

[0039] Then, a portion of the signal blocks are randomly masked. For a given input sequence z, the algorithm steps for randomly masking the sequence according to the mask ratio mask_ratio are as follows:

[0040]

[0041] Finally, the MAE (Masked Autoencoder) architecture is used to predict the masked signal blocks based on the contextual clues of the signal blocks. Subsequently, these embeddings are input into multiple VIT (Vision Transformer) blocks to effectively capture the complex dependencies of EEG signals and enhance the ability of feature representation. It uses the self-attention mechanism to obtain the interaction information between different parts of the input data. Its core calculation can be expressed as:

[0042]

[0043] Where Q = ZW q , K=ZW k 、V=ZW V are the query, key, and value matrices from the EEG signal block features. Z is the feature of the EEG signal block, W q , W k , W v is a learnable weight matrix. d is a scalar that normalizes the dimensions in the softmax function.

[0044] By reconstructing the masked signal, the pre-trained time domain encoder can gain in-depth insights into the EEG characteristics of different populations and various brain activities, thereby completing the EEG feature extraction.

[0045] The loss function of the reconstructed signal is as follows:

[0046]

[0047] Where: B represents the batch size of the input samples, N represents the number of signal blocks, and p ij represents the reconstructed value of the jth signal block of the ith sample, i.e., the output result of the above encoder; t ij represents the actual value of the jth signal block of the ith sample, m ij represents the mask value of the jth signal block of the ith sample, m ij =0 means that the signal block is not shielded, m ij =1 indicates that the signal block is masked. The sample here is the EEG signal sequence mentioned above.

[0048] Frequency domain encoder: In the prior art, LSTM is used as the frequency domain feature extraction network. However, a major disadvantage of LSTM is that it does not consider the spatial dependency of frequency domain signals. Therefore, the present invention uses Conv-LSTM as the frequency domain feature extraction network. Conv-LSTM overcomes the limitations of traditional RNN and CNN by combining CNN and LSTM. Conv-LSTM introduces spatial convolution operations and temporal recurrent operations, while retaining the memory unit and gating mechanism in LSTM, which can capture temporal and spatial features and has better generalization ability. Given an input tensor X t and the previous state (H t-1 ,C t-1 ), the core calculation of the ConvLSTM unit can be described as:

[0049] combined=concat(X t ,H t-1 )

[0050] combined_conv=Convld(combined)

[0051] (I t ,F t ,O t ,G t )=split(combined_conv,4)

[0052]

[0053]

[0054] Among them, σ represents the sigmoid activation function, which is used to map real numbers between 0 and 1, that is, I t ,F t ,O t ,G t represents a real number, Indicates that I t ,F t ,O t ,G t They are mapped to values ​​between 0 and 1 respectively; tanh represents the hyperbolic tangent activation function, Convld represents the one-dimensional convolution operation, and split represents dividing the convolution output into four parts. During the training process, the present invention uses the visual stimulus label of EEG as supervision and uses CE (CrossEntropy, cross entropy loss) as the loss function for training with reference to the process of BrainVis. Regarding the description of the visual stimulus label of EEG, for example, the EEG signal induced by seeing the dog image, the label of this section is dog.

[0055] Semantic classifier: After the time domain encoder and the frequency domain encoder are trained, the present invention concatenates the time domain features and the frequency domain features, and then uses the labels of visual stimuli as supervision to train the semantic classifier. The classifier is a multilayer perceptron (MP) stacked by two linear layers. The trained semantic classifier can obtain coarse-grained semantic information of the category label. The loss function used in the training process is CE cross entropy loss.

[0056] 2. Align the network

[0057] Different from previous works that use CLIP embedding as alignment guidance, this work uses the pre-trained IP-Adapter model to generate image feature embedding as alignment guidance. The core concept of IP-Adapter revolves around its innovative decoupled cross-attention mechanism. Specifically, IP-Adapter does not use a single cross-attention layer to process text and image features at the same time, but introduces a cross-attention layer dedicated to image features. Given an image feature f i , the new cross attention Z ip The output is calculated as follows:

[0058]

[0059] Where Q = ZW q , K′=f i W′ k , V′=f i W′ v is the query, key, and value matrix from the image features; f i Represents the image features. W′ k and W′ v is the corresponding weight matrix. d is a scalar used to normalize the dimensions in the softmax function.

[0060] Those skilled in the art should note that the pre-trained IP-Adapter model is a known technology, and the specific training process is not described in detail in the present invention. Please refer to: Ye, Hu, et al. "Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models." arXiv preprintarXiv:2308.06721(2023).

[0061] The pre-trained IP-Adapter model in the present invention takes the actual image seen by the human body of the collected EEG signal as input to generate image feature embedding.

[0062] This separation enables the model to focus on learning more detailed specific image features, thereby enhancing its ability to capture unique features of visual data, solving the problem that DreamDiffusion cannot reconstruct fine-grained semantics by fine-tuning large pre-trained models using CLIP image feature embeddings. This training ensures that the neural representation is closely aligned with the visual feature embeddings of the IP-Adapter, which helps to reconstruct accurate and reliable images from EEG signals. In addition, the IP-Adapter image feature embedding used in this work has a dimension of only 4*768, which is more compact than the CLIP text feature embedding of 77*768 used by BrainVis, greatly reducing the number of parameters of the alignment network and alleviating the requirements of the training process on the equipment.

[0063] The present invention follows the training process of BrainVis and uses a network composed of fully connected layers with residual connections to achieve embedding alignment, such as Figure 1 The network shown in FIG. 1 specifically includes two weight layers, namely, fully connected layers, and the two fully connected layers are connected via residual connections. The difference is that the present invention does not need to align the class label embedding and the additionally generated text description embedding at the same time, but only needs to align the image feature embedding. The EEG feature embedding f that maximizes the alignment output eeg and the image feature embedding generated by the pre-trained IP-Adapter model img The cosine similarity of , the loss function is defined as follows:

[0064]

[0065] Figure 1 The linear rectification function in is the activation function of the residual connection.

[0066] 3. Cascade Diffusion Model

[0067] It may be difficult to generate the desired image using precise text prompts because it may not always capture what you want to express, so adding images next to text prompts helps the model better understand what it should generate and can produce more accurate results. In contrast, although the work of the present invention uses IP-Adapter image embedding as a guide for alignment, since the alignment process cannot be completely accurate, the reconstruction process will inevitably cause loss of semantic information and noise increase. In order to solve this problem, the present invention uses a cascade diffusion model to perform image reconstruction tasks. Specifically:

[0068] The present invention firstly performs reverse diffusion in latent space based on the aligned EEG time-frequency domain embedding.

[0069] The embedding is updated in each iteration and injected into the current image state to affect the direction and results of denoising. Finally, after multiple iterations of denoising, the latent representation generated by the model is decoded into an image that is both clear and consistent with the semantic information contained in the EEG time-frequency domain embedding.

[0070] However, due to semantic noise and information loss in the embedding, the reconstructed image quality is not high. Therefore, the present invention uses clear EEG classification labels to perform secondary reconstruction on the reconstruction results of EEG time-frequency domain embedding to improve the latent image that is not completely denoised during the back diffusion process. This can reduce semantic noise and obtain higher quality results.

[0071] The technical effects of the present invention are described below in combination with specific data:

[0072] Dataset: The dataset used in the present invention contains EEG records from six subjects (five men and one woman) when viewing object images. The experiment used 40 different and easily distinguishable categories from ImageNet. Easy to distinguish means that each category is significantly different from other categories (such as German Shepherd-parachute), rather than different subcategories of the same object (German Shepherd-Siberian Husky). Each category contains 50 images, totaling 2000 images as visual stimuli. The experiment displays different images of one category each time, each for 0.5 seconds. After each category is displayed, there is a 10-second black screen time to reset the visual pathway. The entire experimental process lasted 1400 seconds, which is about 23 minutes and 20 seconds. After the experiment, 536 records were discarded because the time was too short or the changes were too large, and finally 11466 128-channel EEG sequences were obtained. In order to prevent interference caused by the previously displayed image, the present invention intercepts 40-460 milliseconds of data from each sequence for the experiment, and divides the data into a training set, a test set, and a validation set in a ratio of 8:1:1.

[0073] Quantitative value: The present invention refers to the parameter settings of previous work (Bai et al., 2023 and Fu et al., 2024). For EEG signals, the present invention uses the signal after filtering at 5-95HZ, and sets the data length to 512 by interpolation. The mask rate of the time domain encoder is 75%, and 500 rounds of training are performed. The frequency domain encoder uses a 3-layer conv-LSTM for 1000 rounds of training, and then the time-frequency domain features are aligned for 500 rounds. The linear classifier is trained for 50 rounds, and finally the 1.5 version of the stable diffusion model is used for image generation. Those skilled in the art should note that the 1.5 version here is only the version number of the stable diffusion model in this embodiment. In practical applications, the stable diffusion model obtained by the final training can be used for image generation.

[0074] Evaluation Metrics

[0075] This paper uses a variety of indicators for evaluation to measure the generation quality of the model from various angles:

[0076] EEG classification accuracy (CA): used to evaluate the precision of EEG classification, which shows the degree of correlation between EEG features and coarse-grained sub-semantics of visual stimuli.

[0077] 50-way Top-1 classification accuracy (GA): used to evaluate the semantic accuracy of the generated image. The present invention feeds the generated image and the actual image into a classifier pre-trained on the ImageNet1K dataset, and then checks whether the top-1 predicted categories of the generated image and the actual image match in 50 predefined categories. If the classification labels of the generated image and the actual image are consistent, it can be considered that the generated image is consistent with the semantics of the actual image.

[0078] Inception Score (IS): The diversity of generated images is evaluated by calculating the entropy of the generated image category distribution, and the authenticity of the generated images is evaluated by comparing the average entropy of the generated images with the real images.

[0079] Fréchet Inception Distance (FID): used to evaluate the perceptual difference between generated images and real images. One advantage of FID is that it does not require pairwise comparison of generated images, which makes it more efficient than some pixel-based metrics when evaluating a large number of images. In addition, FID is also considered to be an evaluation metric that is more in line with human visual perception because it focuses on high-level semantic features of images rather than low-level pixel features.

[0080] Results and comparison with other methods

[0081] Most of the indicators used in previous image reconstruction work to evaluate the generation quality only adopted GA or initial score, which cannot comprehensively evaluate the quality of the reconstructed image. The work of BrainVis solves this problem by evaluating the reconstruction effect from two aspects: image reconstruction and EEG classification. The work of this invention further uses the various evaluation indicators used by BrainVis and uses a new indicator to evaluate the generation quality. The baseline selections for image reconstruction include DreamDiffusion, Brain2Image, NeuroVision, DCLS-GAN, ESG-ADA and MB2C; the baselines for EEG classification include EEGCN, Brain2Image, S-EEGNet and KDSTFT.

[0082] Quantitative results

[0083] The present invention uses the trained model to perform four image reconstructions for each EEG sequence of each subject to test the reconstruction quality. For the image reconstruction quality, the results in Table 1 show that the BrainAdapter of the present invention achieves the best in terms of GA and FID, which are 2.4% and 4.1% higher than the existing best methods, respectively. This shows that the image reconstruction of the present invention has the best overall quality and can better capture the semantic information contained in the visual data. The results in Table 2 show that the BrainAdapter of the present invention is 10.15%, 4.45%, 0.61% and 10.32% higher than the existing best methods in Top1, Top3, Top5 CA and F1 scores, respectively. This shows that the EEG encoding method proposed in the present invention has the best EEG representation capability. In addition, as shown in Table 3, the method of the present invention uses only 188M parameters, which is far lower than the 297M parameters of reamDiffusion and the 518M parameters of BrainVis. This shows that the method of the present invention helps to reduce energy consumption during model training and reasoning, which is particularly important for application scenarios where models need to be deployed in resource-constrained environments.

[0084] Table 1 Comparison of the effects of the method of the present invention and the prior art on the three indicators of GA, IS and FID

[0085]

[0086] Table 2 Comparison of the effects of the method of the present invention and the prior art on the four indicators of Top1, Top3, Top5 CA and F1 score

[0087]

[0088] Table 3 Comparison of model parameters

[0089]

[0090] Qualitative results

[0091] like Figure 2As shown, the image reconstruction results of the present invention significantly demonstrate the ability to capture GT in detail, whether in posture, color or other details. The fine-grained reconstruction method of the present invention successfully captures the deep semantics of GT, thanks to the advanced feature extraction capability of the model of the present invention. This capability makes the reconstructed image visually very similar to the original image, retaining key visual information and structural integrity. In addition, the reconstruction results of the present invention also excel in color accuracy and texture details, which are key factors in image realism. In the coarse-grained reconstruction stage, the present invention utilizes the powerful generation capability of the diffusion model to supplement and improve the results of fine-grained reconstruction. This two-stage approach not only improves the efficiency of reconstruction, but also enhances the robustness of the results, ensuring high-quality output even in the face of challenging image content.

[0092] exist Figure 3 In the present invention, the reconstructed images of three categories, namely horses, dogs and pumpkin lanterns, were selected for comparison with the diffusion-based DreamDiffusion. It is worth noting that the original model of DreamDiffusion used 120k data from MOAAB to train MSM, while in the comparison of the present invention, both the model of the present invention and DreamDifusion used only 20k data from ImageNetEEG to train MSM. According to Table 1, the performance of DreamDifusion dropped significantly when the amount of MSM training data was insufficient, while the model of the present invention performed well with the same amount of data. Figure 3 It can be seen that the image quality of the present invention is higher, which highlights the effectiveness of the method of the present invention.

[0093] In order to prove that each part of the method of the present invention plays a positive role in the image reconstruction task, the present invention conducts an ablation experiment. In Table 4, the present invention conducts an ablation study through the EEG classification task to analyze the impact of the main components in the EEG encoder on the EEG representation. The results show that both time domain features and frequency domain features have a positive impact on the performance, among which frequency domain feature extraction has a higher impact on EEG classification.

[0094] Table 4 Ablation analysis of EEG encoder components

[0095]

[0096] The present invention realizes a method for converting EEG signals to visual images by combining time-frequency domain feature extraction, deep learning alignment network and cascade diffusion model. This method can not only extract and align deep features of EEG data, but also generate visual content with rich semantic information. Better classification effect and image reconstruction quality are achieved with a smaller number of parameters. Through this method, the present invention can have a deeper understanding of how the brain processes visual information and provide new perspectives and tools for research in related fields.

[0097] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for reconstructing human visual information based on electroencephalogram signals, characterized in that: include: S0, obtaining an image of the scene observed by the human eye at the same moment, and collecting the EEG sequence corresponding to the human body at that moment; S1, using the time-frequency domain encoder to extract the time-domain features and frequency-domain features of the EEG sequence; S2, input the splicing results of the time domain features and frequency domain features of the EEG sequence into the semantic classifier to obtain the EEG classification label; S3, using the pre-trained IP-Adapter model to generate image feature embedding based on the image of the scene observed by the human eye obtained in step S0; S4, based on the image feature embedding obtained in step S3, aligning the splicing results of the time domain features and frequency domain features of the EEG sequence extracted in step S1; S5. The time domain features and frequency domain features of the aligned EEG sequence are concatenated together with the EEG classification labels and used as conditions of the cascade diffusion model for multi-level semantic visual reconstruction.

2. The method for reconstructing human visual information based on EEG signals according to claim 1, characterized in that: The time-frequency domain encoder uses the same time-domain encoder structure as DreamDiffusion to extract the time-domain features of the EEG sequence; Conv-LSTM is used as the frequency-domain encoder to extract the frequency-domain features of the EEG sequence.

3. The method for reconstructing human visual information based on EEG signals according to claim 2, characterized in that: It also includes training the time domain encoder and the frequency domain encoder, splicing the time domain features and frequency domain features output by the trained time domain encoder and the frequency domain encoder, and then inputting the spliced ​​results into the semantic classifier.

4. The method for reconstructing human visual information based on EEG signals according to claim 3, characterized in that: The semantic classifier is specifically a multi-layer perceptron composed of several stacked linear layers.

5. The method for reconstructing human visual information based on EEG signals according to claim 4, characterized in that: Step S4 uses a network composed of fully connected layers with residual connections to align the time domain features and frequency domain features of the EEG sequence.

6. The method for reconstructing human visual information based on EEG signals according to claim 5, characterized in that: The network structure used for alignment in step S3 includes two fully connected layers, and the two fully connected layers are connected through residual connections.

7. The method for reconstructing human visual information based on EEG signals according to claim 6, characterized in that: The method also includes training the network structure used for alignment. The loss function used in the training process is: maximizing the cosine similarity between the alignment results of the time domain features and frequency domain features of the EEG sequence and the image feature embedding in step S3.

Images