Noninvasive electroencephalogram visual image restoration method based on deep learning model DiffEEG

Abstract

The invention discloses a noninvasive electroencephalogram visual image restoration method based on a model DiffEEG, and the method comprises the following steps: S1, carrying out the feature construction of a difference form on an acquired original electroencephalogram signal, and carrying out the self-adaptive adjustment of different signal dimensions through combining with a feature weighting and structure adjustment mechanism, S2, carrying out the vectorization processing of electroencephalogram features, and carrying out the reconstruction of an electroencephalogram visual image. The method comprises the steps of S1, carrying out feature mapping operation projection to obtain EEG representation, S3, carrying out modeling on the image representation by using VAE to obtain I1, mapping electroencephalogram features into condition vectors, obtaining image feature representation I2, and driving a GIT model to output a corresponding text T1, and S4, inputting I1, I2 and T1 as guide signals into a framework, and finally outputting a result. The invention provides a DiffEEG model for non-invasive electroencephalogram visual image restoration, features are extracted by adopting a difference mechanism and channel attention, a visual reconstruction framework is input, and a reconstructed image is output. According to the method, the effectiveness of the DiffEEG in EEG visual reconstruction is verified by an experimental result, and the EEG feature extraction accuracy is improved.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interfaces, and specifically designs a non-invasive EEG visual image restoration method based on the DiffEEG model. Background Technology

[0002] Brain-computer interfaces (BCIs) enable direct communication between the brain and external devices without relying on the peripheral nervous system, making them an important research direction in human-computer interaction and medical rehabilitation. Due to their low cost and non-invasiveness, electroencephalography (EEG) has been widely applied in various BCI paradigms, and reconstructing visual stimuli from brain signals is one of the most promising research areas. Initially, EEG signal analysis relied primarily on visual examination of the amplitude and frequency changes over time. However, with the development of digital technology, this methodology has changed significantly, gradually shifting towards a more comprehensive analysis of the temporal and spatial characteristics of EEG signals (Craik, et al., Journal of Neural Engineering, 16.3: 031001 (2019)). Therefore, EEG is considered a powerful tool for capturing brain function, especially on millisecond-scale timescales. Despite these advantages, EEG has traditionally suffered from poor spatial resolution, making it difficult to accurately pinpoint specific brain regions on the scalp where neural activity is measured (Li, et al., Sensors, 22.15: 5865 (2022)). In recent years, increasing research has focused on applying EEG to more complex tasks, such as image recognition and reconstruction (Mai, et al., arXiv preprint arXiv, 2308.07428 (2023)). Current research has significantly improved the accuracy of image recognition tasks, demonstrating the potential of EEG as a bridge between neural activity and visual representation (Spampinato, et al., Proceedings of the IEEE conference on computer vision and pattern recognition, 6809–6817 (2017); Kavasidis, et al., Proceedings of the 25th ACM international conference on Multimedia, 1809–1817 (2017)).

[0003] To achieve cross-modal conversion from EEG signals to images, it is necessary to reduce the significant gap between low signal-to-noise ratio temporal neural signals and high-dimensional structured visual representations. Existing methods typically rely on deep neural networks for feature extraction and image generation. Generative models such as generative adversarial networks and diffusion models have significantly improved the accuracy and effectiveness of EEG signal-based methods, enabling the generation of photorealistic images from neural signals (Kumar, et al. Personal and Ubiquitous Computing, 22.1:185-199 (2018)). Current research has demonstrated the feasibility of decoding natural images from EEG signals and developed an innovative framework for aligning EEG responses with paired image stimuli (Bai, et al., arXiv preprint arXiv:2306.16934 (2023)). Cross-modal alignment is a key step in brain activity-driven image reconstruction, aiming to reduce the modal differences between EEG signals and visual images and achieve effective sharing of textual information. The VAE model proposed by Diederik et al. achieves data generation and reconstruction by learning the latent distribution of data (Kingma, DP, et al. arXiv preprint arXiv:1312.6114 (2013).). Based on this method, Brain2Image is the first framework to use VAE to guide the generation of images from EEG features (Kavasidis, Isaak, et al. Proceedings of the 25th ACM international conference on Multimedia, 1809–1817 (2017).). In addition, Palazzo et al. designed a conditional GAN ​​framework based on generative adversarial networks to generate corresponding images using EEG signals as conditional inputs (Palazzo, et al., Proceedings of the IEEE international conference on computer vision, 3410–3418 (2017).). Currently, the introduction of multimodal pre-trained models such as CLIP has made contrastive learning the mainstream method for cross-modal alignment (Li, et al., arXivpreprint arXiv:2403.07721 (2024).).

[0004] The aforementioned non-invasive EEG-based visual reconstruction methods often struggle to capture the fine-grained dynamic changes in the time series of EEG signals for text comprehension. Existing methods typically treat EEG as a static feature, ignoring the temporal information inherent in the signal's evolution over time, making it difficult for the model to establish accurate cross-modal correspondences. To address these issues, this patent proposes a novel EEG visual reconstruction algorithm based on the DiffEEG model. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a non-invasive EEG visual image reconstruction method based on the DiffEEG model. Under in-subject conditions, the method performs differential operations on EEG data, extracts its temporal features using a Transformer structure, mines spatial information in EEG signals by fusing channels, and trains EEG feature data using both the VAE framework and a diffusion model to generate low-resolution and high-resolution image data. Finally, it combines text information generated by a GIT model to output a reconstructed image.

[0006] To achieve the aforementioned objectives, the technical solution adopted by this invention is as follows: a non-invasive EEG visual image restoration method based on the DiffEEG model, comprising the following steps: S1. The acquired raw EEG signals are constructed using differential features, and feature weighting and structural adjustment mechanisms are combined to adaptively adjust different signal dimensions. S2. The EEG features are vectorized and projected through feature mapping operations to obtain EEG representations. S3. The image representation is modeled using VAE to obtain I1, and the EEG features are mapped to conditional vectors to obtain image feature representations I2, which drive the GIT model to output the corresponding text T1. S4. I1, I2, and T1 are used as guiding signals input into the framework, and the final output result is obtained.

[0007] Furthermore, the DiffEEG of S1 includes an iTS substructure, a differential structure for difference information extraction, a substructure for feature recalibration, and simultaneously sets an embedding structure and an output signal mapping structure.

[0008] Furthermore: EEG signals are modeled using feature-based structures. Encode the corresponding information to obtain Differential computation is introduced for modeling to generate intermediate representations containing differential information. Using a compression-based feature modulation structure for Through the transformation by the rearrangement unit and the linear mapping module, the final output data format is as follows: .

[0009] Furthermore: the differential operation in S1 will convert the EEG signal after feature extraction by the iTS module... The result of signal sequence differencing can be expressed as: The final conditionally guided representation of the model is obtained by calculating the difference between each EEG signal and the previous one: For the above output values Weighted aggregation is performed to obtain the signal enhancement representation, and the output is represented as follows. , and Combinatorial update, the output is represented as Build a differential encoder and set up The parameter controls the number of layers; each layer performs differential coding operations, based on... Perform a loop operation and output the result. .

[0010] Furthermore: S2 characterizes the differentiated EEG. Encode and characterize the dependencies between different electrodes for output. To standardize the encoding format, the data shape is rearranged to... , obtain data .

[0011] Further: S3 sets the DiffEEG model described in S1 to training mode, obtaining data I1, which is represented as image data. For each , represented as Add positional encoding to each feature vector The output is ,Will The encoder is passed in, and the final output data is represented as follows: Train the DiffEEG model and calculate the output of the DiffEEG model. With visual representation The distance between the two: , will be trained and The input to the decoder is sampled from the latent distribution. Output I1, expressed as follows: .

[0012] Further: the S4 according to As a generation condition, let c represent At the same time, image feature data As a monitoring signal The expression is as follows: Using a trained model, the real image data can be predicted from random noise, as shown in the following expression: Utilize the generated The data is input into the Git model to generate the corresponding language description T1. Image data is then used to obtain image description information using an autoregressive approach, as shown in the following expression: The output is T1 until the training is complete.

[0013] Furthermore, the S4 framework inputs data I1, as well as the generated data representations I2 and T1, into the framework, and transforms the data into the final callable prompt information through the IPAdapter, and outputs the results using SDXL-turbo technology.

[0014] The beneficial effects of this invention are as follows: The proposed model, DiffEEG, is used for non-invasive EEG visual image reconstruction. It performs differential operations on experimental EEG signal data and uses a Transformer structure to extract temporal features from the EEG signals, capturing dynamic temporal variations. A channel attention mechanism is also incorporated to learn spatial information within the signal channels, resulting in richer feature information. Finally, a projection layer maps the extracted EEG features to feature data aligned with high-dimensional image data. A multimodal alignment framework is then constructed to reconstruct the visual stimulus image. Experimental results from this invention validate the effectiveness of the DiffEEG network in non-invasive EEG visual image reconstruction, further enhancing the performance of brain-computer interface visual reconstruction applications. Attached Figure Description

[0015] Figure 1 This is a flowchart of a non-invasive EEG visual image restoration method based on the DiffEEG model according to the present invention.

[0016] Figure 2 This is a schematic diagram of the structure of the non-invasive EEG visual image restoration of the present invention.

[0017] Figure 3 This is a schematic diagram of the structure of the model DiffEEG of the present invention.

[0018] Figure 4 This is a schematic diagram of the differential Transformer module of the present invention. Detailed Implementation

[0019] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0020] Example 1:

[0021] like Figure 1 As shown, in one embodiment of the present invention, a non-invasive EEG visual image restoration method based on the DiffEEG model includes the following steps: S1. The acquired raw EEG signals are constructed using differential features, and feature weighting and structural adjustment mechanisms are combined to adaptively adjust different signal dimensions. S2. The EEG features are vectorized and projected through feature mapping operations to obtain EEG representations. S3. The image representation is modeled using VAE to obtain I1, and the EEG features are mapped to conditional vectors to obtain image feature representations I2, which drive the GIT model to output the corresponding text T1. S4. I1, I2, and T1 are used as guiding signals input into the framework, and the final output result is obtained.

[0022] Furthermore, the DiffEEG of S1 includes an iTS substructure, a differential structure for difference information extraction, a substructure for feature recalibration, and simultaneously sets an embedding structure and an output signal mapping structure.

[0023] Furthermore: EEG signals are modeled using feature-based structures. Encode the corresponding information to obtain Differential computation is introduced for modeling to generate intermediate representations containing differential information. Using a compression-based feature modulation structure for Through the transformation by the rearrangement unit and the linear mapping module, the final output data format is as follows: .

[0024] Furthermore: the differential operation in S1 will convert the EEG signal after feature extraction by the iTS module... The result of signal sequence differencing can be expressed as: The final conditionally guided representation of the model is obtained by calculating the difference between each EEG signal and the previous one: For the above output values Weighted aggregation is performed to obtain the signal enhancement representation, and the output is represented as follows. , and Combinatorial update, the output is represented as Build a differential encoder and set up The parameter controls the number of layers; each layer performs differential coding operations, based on... Perform a loop operation and output the result. .

[0025] Furthermore: S2 characterizes the differentiated EEG. Encode and characterize the dependencies between different electrodes for output. To standardize the encoding format, the data shape is rearranged to... , obtain data .

[0026] Further: S3 sets the DiffEEG model described in S1 to training mode, obtaining data I1, which is represented as image data. For each , represented as Add positional encoding to each feature vector The output is ,Will The encoder is passed in, and the final output data is represented as follows: Train the DiffEEG model and calculate the output of the DiffEEG model. With visual representation The distance between the two: , will be trained and The input to the decoder is sampled from the latent distribution. Output I1, expressed as follows: .

[0027] Further: the S4 according to As a generation condition, let c represent At the same time, image feature data As a monitoring signal The expression is as follows: Using a trained model, the real image data can be predicted from random noise, as shown in the following expression: Utilize the generated The data is input into the Git model to generate the corresponding language description T1. Image data is then used to obtain image description information using an autoregressive approach, as shown in the following expression: The output is T1 until the training is complete.

[0028] Furthermore, the S4 framework inputs data I1, as well as the generated data representations I2 and T1, into the framework, and transforms the data into the final callable prompt information through the IPAdapter, and outputs the results using SDXL-turbo technology.

[0029] Example 2: This embodiment describes a specific experiment provided in Embodiment 1.

[0030] The experimental dataset was the publicly available THINGS-EEG dataset, which contains EEG signals collected from 10 subjects during a visual task. Each subject completed the same experiment four times, resulting in 10 independent datasets. The training set contained 16,540 different training image conditions, each repeated four times, while the test set contained 200 test image conditions, totaling approximately 82,160 image experiments. To ensure the integrity and richness of the experimental data, the original 63-channel EEG data were retained, and bandpass filters ranging from 0.1Hz to 100Hz were applied, maintaining a sampling rate of 1000Hz. All experiments were run on a system equipped with two NVIDIA RTX 4090 GPUs. All models were implemented using the Python 3.10.0 and PyTorch 2.1.0 deep learning open-source frameworks, and CUDA 12.1 was used for model training and testing. In the experiment, MSE and CLIPLLoss were used as model loss functions, and the Adam optimizer was used for parameter learning. The data batch size was set to 64, and the epochs were set to 50 rounds. The training iterations continued until the training set loss tended to stabilize.

[0031] To fully demonstrate the effectiveness of this invention, the experiment required the model to identify the corresponding target image from 200 image categories based solely on EEG signals, without relying on any target category label supervision. The experiment used Top-1 and Top-5 accuracy metrics to measure the textual consistency between the model's output EEG features and image feature representations. The Top-1 accuracy metric reflects whether the model can accurately locate a unique corresponding image, while the Top-5 accuracy metric focuses more on whether the model can rank the target image within the top five, thus evaluating the model's accuracy.

[0032] Accuracy: Accuracy refers to the number of samples correctly predicted by the model ( ) accounts for the total sample size ( The proportion of ) can be expressed mathematically as: .

[0033] The experiment used cosine similarity as an indicator to evaluate the consistency between EEG features and image features, with a test sample size of [number missing]. The feature of the i-th EEG sample is represented as The feature set of all candidate images is The formula is as follows: The index sequence after sorting in descending order of similarity is as follows: , , ,in This represents the index of the image with the highest similarity to the i-th sample. This represents the index of the real image corresponding to the text of EEG sample i. This is an indicator function; it takes the value 1 if the condition is true, and 0 otherwise.

[0034] The experimental results obtained after training based on the above indicators are shown in Table 1. To verify the performance and effectiveness of the network model constructed in this invention, it was compared with current mainstream methods such as LSTM, Conformer, EEGNet, BraVL, NICE, and ATM. The results in Table 1 show the performance of the DiffEEG model constructed in this invention compared with other methods in terms of accuracy. The DiffEEG model achieved an average Top-1 accuracy of 34.8% and a Top-5 accuracy of 65.8%, which are higher than the performance indicators of other existing models in both Top-1 and Top-5 accuracy scores, indicating that the DiffEEG model constructed in this invention has better performance for non-invasive EEG visual image reconstruction.

[0035] In summary, this experiment was conducted on different models using the same public dataset, verifying the effectiveness of the DiffEEG model in non-invasive EEG visual image reconstruction.

[0036] .

Claims

1. A non-invasive EEG visual image reconstruction method based on the DiffEEG model, characterized in that, Includes the following steps: S1. The acquired raw EEG signals are constructed using differential features, and feature weighting and structural adjustment mechanisms are combined to adaptively adjust different signal dimensions. S2. The EEG features are vectorized and projected through feature mapping operations to obtain EEG representations. S3. The image representation is modeled using VAE to obtain I1, and the EEG features are mapped to conditional vectors to obtain image feature representations I2, which drive the GIT model to output the corresponding text T1. S4. I1, I2, and T1 are used as guiding signals input into the framework, and the final output result is obtained.

2. The non-invasive EEG visual image reconstruction method based on the DiffEEG model according to claim 1, characterized in that, The DiffEEG of S1 includes an iTS substructure, a differential structure for extracting difference information, a substructure for feature recalibration, and an embedding structure and an output signal mapping structure.

3. The non-invasive EEG visual image restoration method based on the model DiffEEG according to claim 2, characterized in that, EEG signals through feature modeling structures Encode the corresponding information to obtain Differential computation is introduced for modeling to generate intermediate representations containing differential information. Using a compression-based feature modulation structure for Through the transformation by the rearrangement unit and the linear mapping module, the final output data format is as follows: .

4. The non-invasive EEG visual image reconstruction method model DiffEEG according to claim 1, characterized in that, The differential operation in S1 will convert the EEG signal after feature extraction by the iTS module into a differential signal. The result of signal sequence differencing can be expressed as: The final conditionally guided representation of the model is obtained by calculating the difference between each EEG signal and the previous one: For the above output values Weighted aggregation is performed to obtain the signal enhancement representation, and the output is represented as follows. , and Combinatorial update, the output is represented as Build a differential encoder and set up The parameter controls the number of layers; each layer performs differential coding operations, based on... Perform a loop operation and output the result. .

5. The non-invasive EEG visual image reconstruction method model DiffEEG according to claim 1, characterized in that, S2 represents the differential EEG. Encode and characterize the dependencies between different electrodes for output. To standardize the encoding format, the data shape is rearranged to... , obtain data .

6. The non-invasive EEG visual image reconstruction method based on model DiffEEG according to claim 1, characterized in that, Step S3 sets the DiffEEG model described in S1 of claim 1 to training mode, obtaining data I1, where the image data is represented as follows: For each , represented as Add positional encoding to each feature vector The output is ,Will The encoder is passed in, and the final output data is represented as follows: Train the DiffEEG model and calculate the output of the DiffEEG model. With visual representation The distance between the two: , will be trained and The input to the decoder is sampled from the latent distribution. Output I1, expressed as follows: .

7. The non-invasive EEG visual image reconstruction method based on the model DiffEEG according to claim 1, characterized in that, The S4 obtained according to claim 5 As a generation condition, let c represent Simultaneously, using the image feature data obtained according to claim 6 As a monitoring signal The expression is as follows: Using a trained model, the real image data can be predicted from random noise, as shown in the following expression: Utilize the generated The data is input into the Git model to generate the corresponding language description T1. Image data is then used to obtain image description information using an autoregressive approach, as shown in the following expression: The output is T1 until the training is complete.

8. The non-invasive EEG visual image reconstruction method based on model DiffEEG according to claim 1, characterized in that, The S4 framework inputs the data I1 from claim 6 and the data representations I2 and T1 generated in claim 7 into the framework, converts the data into the final callable prompt information through the IPAdapter, and outputs the results using SDXL-turbo technology.

Images