EEG-fNIRS visual information decoding method
By combining EEG and fNIRS signals and utilizing a Transformer encoder and a multimodal attention fusion module, the problem of insufficient single-modal signals in visual information reconstruction in existing technologies is solved, achieving high-fidelity visual image reconstruction. This method is particularly suitable for patients with neuromuscular pathway damage and provides a high-bandwidth brain-computer communication method.
Patent Information
- Application Number
- CN202512020012.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies have not effectively combined the high temporal resolution of EEG with the portability and spatial resolution advantages of fNIRS for visual information reconstruction tasks, resulting in limited information content of single-modal signals, inaccurate semantics of reconstructed images, and loss of detail.
The EEG-fNIRS visual information decoding method is adopted. By simultaneously acquiring EEG and fNIRS time-domain signals, a Transformer encoder is used to generate feature embedding vectors. High-fidelity visual images are reconstructed through a multimodal attention fusion module and a dual decoding strategy. Combining the complementary advantages of EEG and fNIRS, a diffusion model is used to enrich and optimize image details.
It significantly improves the feature robustness and image fidelity of visual information reconstruction, and is particularly suitable for patients with neuromuscular pathway damage. It provides a high-bandwidth, non-invasive brain-computer communication method, enhancing the freedom and richness of information transmission.
Smart Images
Figure CN122066796A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of brain-computer interface information processing, and in particular to an EEG-fNIRS visual information decoding method. Background Technology
[0002] Decoding imagined images directly from the brain has always been a significant challenge in the field of human-computer interaction. In recent years, research on reconstructing visual content using electroencephalography (EEG) or functional magnetic resonance imaging (fMRI) signals has made continuous progress. At the same time, the rise of diffusion models has provided a strong algorithmic foundation for high-fidelity image generation, and functional near-infrared spectroscopy (fNIRS) technology has also provided a relatively convenient and non-invasive brain-computer interaction window. However, research on combining the high temporal resolution of EEG with the portability and spatial resolution advantages of fNIRS for visual information reconstruction tasks is still lacking and is a technical problem that urgently needs to be solved in the current field. Summary of the Invention
[0003] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide an EEG-fNIRS visual information decoding method.
[0004] The objective of this invention is achieved through the following technical solution:
[0005] A method for decoding EEG-fNIRS visual information includes the following steps:
[0006] Simultaneously acquire the EEG time-domain signal and fNIRS time-domain signal generated when the subject views the visual stimulus image, and obtain the EEG time-spectrum and fNIRS time-spectrum after processing;
[0007] Input the EEG time spectrum and the fNIRS time spectrum into the Transformer encoder to generate EEG feature embedding vector and fNIRS feature embedding vector respectively.
[0008] The EEG feature embedding vector and the fNIRS feature embedding vector are input together into a multimodal attention fusion module to generate a hybrid embedding vector;
[0009] A two-stage decoding strategy is used to reconstruct the hybrid embedding vector into the final image.
[0010] Furthermore, the EEG time-lapse spectrogram and the fNIRS time-lapse spectrogram are input into the Transformer encoder to generate EEG feature embedding vectors and fNIRS feature embedding vectors, respectively, as follows:
[0011] The EEG time-spectral map and the fNIRS time-spectral map are spatially divided into sequences composed of patches, which are then further formed into patch embedding sequences.
[0012] The Patch embedding sequence is input into the Transformer encoder, and each modality outputs a high-dimensional feature vector, resulting in EEG feature embedding vectors and fNIRS feature embedding vectors.
[0013] Furthermore, the Transformer encoder is multi-layered, with each layer including a multi-head self-attention mechanism and a feedforward neural network.
[0014] Furthermore, the multimodal attention fusion module includes a bidirectional cross-attention unit and a gated fusion unit. Furthermore, a two-stage decoding strategy is employed to reconstruct the final image from the hybrid embedding vectors, specifically as follows:
[0015] The first-stage decoding strategy is as follows: the hybrid embedding vector is input in parallel into two independent decoding paths. The two independent decoding paths include an image-guided information decoding path and a text-guided information decoding path, which respectively obtain a low-resolution preliminary reconstructed image containing basic spatial information and a "semantic description" text information describing the core content of the image.
[0016] The second-stage decoding strategy specifically involves inputting the initially reconstructed image and the "semantic description" text information into the Stabel Diffusion XL diffusion model to generate the reconstructed image.
[0017] Furthermore, in the text guidance information decoding path, the hybrid embedding vector is adapted to dimensions through a linear projection layer and then input into a large language model, which decodes it into a "semantic description" text information describing the core content of the image.
[0018] Furthermore, the gated fusion unit dynamically learns fusion weights through a neural network and adaptively weights and sums the two enhanced EEG feature vectors and fNIRS feature vectors to ultimately generate a unified hybrid embedding vector.
[0019] Furthermore, the Stabel Diffusion XL diffusion model employs IP-Adapter technology.
[0020] Furthermore, the image patches that make up the Patch sequence are linearly projected, flattened, and mapped into a fixed-dimensional vector to form the Patch embedding sequence.
[0021] Furthermore, the sequence of patches includes multiple fixed-size, non-overlapping patches.
[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0023] To address the issues of limited information content in single-modal signals, inaccurate semantics, and lack of detail in reconstructed images in existing non-invasive visual decoding tasks, this invention proposes an EEG-fNIRS visual information decoding method. This method converts EEG and fNIRS signals into time-spectral graphs and utilizes a gated multimodal attention module for feature interaction and deep fusion, fully combining the complementary advantages of EEG and fNIRS signals and significantly improving feature robustness. This aims to achieve accurate reconstruction from blurred brain signals to high-fidelity visual images.
[0024] This invention further designs a parallel decoding and joint guidance strategy: using VAE to map fused features into a preliminary image containing spatial layout as a structural cue; at the same time, using a large language model to decode homologous features into semantic text describing the core content as a content cue; finally, using a diffusion model, the final image is generated under the dual constraints of the above structure and semantics. Attached Figure Description
[0025] Figure 1 This is a schematic diagram of the process of the present invention;
[0026] Figure 2 This is a detailed internal structural diagram of the multimodal attention fusion module proposed in this invention. Detailed Implementation
[0027] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.
[0028] like Figure 1 As shown, a multimodal guided EEG-fNIRS visual information decoding method includes the following steps:
[0029] S1 simultaneously acquires EEG time-domain (EEG) signals and fNIRS time-domain (fNIRS) signals generated when subjects view visual stimulus images, and performs preprocessing and time-frequency conversion on each signal to obtain a two-dimensional time-frequency spectrum X that can simultaneously represent time and frequency information. eeg and X fnirs This serves as the input for subsequent feature extraction.
[0030] The time-frequency conversion uses time-frequency analysis methods such as short-time Fourier transform to convert a one-dimensional time-domain signal into a two-dimensional time-frequency spectrum that can simultaneously represent time and frequency information.
[0031] S2 feeds the EEG time-spectral image and the fNIRS time-spectral image into two independent Transformer encoders. First, the time-spectral image is segmented into several image patches. Then, the Transformer encoder performs deep feature extraction on the image patch sequence, generating EEG feature embedding vectors and fNIRS feature embedding vectors, specifically including:
[0032] S21: For the EEG time-spectral image and fNIRS time-spectral image generated in step S1, they are spatially segmented into a sequence of multiple fixed-size, non-overlapping patches. Then, each image patch is linearly projected, flattened, and mapped into a fixed-dimensional vector to form a patch embedding sequence.
[0033] S22: The Patch embedding sequence is fed into a multi-layer Transformer encoder. Through the encoder's internal multi-head self-attention mechanism and feedforward neural network, the time-frequency features in the sequence are deeply modeled to capture their inherent complex dependencies. After processing by the multi-layer encoder, each modality ultimately outputs a high-dimensional feature vector, referred to as the EEG feature embedding vector and the fNIRS feature embedding vector, respectively.
[0034] Furthermore, to achieve efficient feature extraction, this invention employs a convolution-enhanced Transformer encoder to embed and encode the temporal spectrograms of the two modes respectively.
[0035] First, the time-spectrum diagram X∈R of any mode is... H×W×C The image is segmented into N non-overlapping image patches, which are then mapped to embedding vectors using a linear projection matrix E, while also incorporating a positional code E. pos This forms the initial sequence:
[0036]
[0037] Where [·; ·] represents sequence concatenation. Let be the learnable projection matrix, and D be the embedding dimension.
[0038] Subsequently, Z0 is input into a multi-layer convolutional enhanced Transformer encoder. For the l-th layer, its output Z... l The calculation is as follows:
[0039] Z l =Conv-FFN(MSA(Z) l-1 )+Z l-1 )+MSA(Z l-1 )
[0040] Here, MSA(·) represents the multi-head self-attention module, used to model global time-frequency dependencies; Conv-FFN(·) is the convolutional augmentation feedforward network, specifically defined as:
[0041] Conv-FFN(Z)=PWConv2(GELU(BN(DWConv(PWConv1(Z)))))
[0042] Here, PWConv represents 1×1 pointwise convolution, DWConv represents k×k depthwise convolution, BN is batch normalization, and GELU is the activation function. This structure effectively integrates global attention mechanisms and local convolutional perception capabilities, significantly improving the robustness of representing the time-frequency features of brain signals. After L layers of encoding, it outputs EEG feature embedding vectors H. eeg With fNIRS feature embedding vector H eeg .
[0043] S3 inputs the EEG feature embedding vector and the fNIRS feature embedding vector into a multimodal attention fusion module. This module first enhances the two features through bidirectional cross-attention, and then dynamically weights and integrates the enhanced features through a gated fusion unit, finally generating a unified hybrid embedding vector.
[0044] like Figure 2 As shown, to further achieve multimodal complementary fusion, this invention designs a multimodal attention fusion module that includes bidirectional cross-attention and gating mechanisms.
[0045] include:
[0046] S31: The multimodal attention fusion module first includes a bidirectional cross-attention unit. Through two parallel cross-attention calculations, an EEG feature vector enhanced with fNIRS information and an fNIRS feature vector enhanced with EEG information are generated respectively.
[0047] S32: The two enhanced feature vectors are input together into the gated fusion unit. This unit adopts a gating mechanism, dynamically learns the fusion weights through a neural network, and adaptively weights and sums the two enhanced features to finally generate a unified hybrid embedding vector.
[0048] First, cross-attention is used to enhance the features of both modalities. Using EEG as the query and fNIRS as the key and value, the enhanced EEG features are as follows:
[0049]
[0050] Similarly, the enhanced Hfnirs′ can be obtained. Then, dynamic weighting is performed using a gated fusion unit:
[0051] α=σ(W gate ·[H′ eeg ;H′ fnirs ]+b gate ),
[0052] H mix =α⊙H′ eeg +(1-α)⊙H′ fnirs
[0053] Where σ is the Sigmoid function, ⊙ represents element-wise multiplication, and W gate With b gate These are learnable parameters.
[0054] S4 employs a two-stage decoding strategy to reconstruct the hybrid embedding vector into the final image.
[0055] S41: The hybrid embedding vector generated in step S3 is input in parallel to two independent decoding paths to generate multimodal guidance information: In the image guidance information decoding path, the hybrid embedding vector is input to a variational autoencoder, which reconstructs a low-resolution "preliminary reconstruction" image containing basic spatial information through its encoding and decoding process, serving as the image condition for subsequent refinement; at the same time, in the text guidance information decoding path, the hybrid embedding vector is dimensionally adapted through a linear projection layer and then input to a large language model, which decodes it into a "semantic description" text describing the core content of the image, serving as the text condition.
[0056] The language model described is a Transformer-based text generation network. After adaptation and fine-tuning of the head network with hybrid embedding features, it is able to generate semantic descriptive text from brain signal embeddings. The core content of the image refers to the high-level semantic information of the image, mainly key information such as the image category, which corresponds to the text description output by the large model; while the basic spatial information may refer to the approximate geometric shape, color layout, and other information of the image, which corresponds to the prior information of the blurred image generated by the VAE.
[0057] S42: The "fuzzy reconstructed" image and "semantic description" text generated in step S41 are simultaneously input into the Stabel Diffusion XL diffusion model. This model uses IP-Adapter technology to encode the "fuzzy reconstructed" image as an image cue, which is then combined with the "semantic description" text cue. Through a multi-step iterative denoising process, the image is enriched and optimized for detail, ultimately generating a high-fidelity final reconstructed image.
[0058] Furthermore, to achieve high-fidelity image reconstruction, this invention employs a dual-guided decoding strategy.
[0059] Hybrid embedding H mix Two parallel input paths:
[0060] (1) Structural path: A low-resolution preliminary image is generated through a variational autoencoder (VAE) decoder. As a hint of spatial structure;
[0061] (2) Semantic path: through the linear projection layer W p After aligning the dimensions, the input to the large language model generates a semantic text description T. sem =LLM(Prefix=W p H mix (), used as a content prompt.
[0062] Ultimately, With T sem Simultaneously input the Stable Diffusion XL diffusion model. This model uses IP-Adapter technology to encode the image into image cues. img The text is then CLIP encoded into a text prompt. txt Denoising generation is performed under dual constraints, and its training objective is to minimize:
[0063] L diff =E[|∈-∈ θ (z t ,t,c txt ,c img )| 2 ]
[0064] This results in the output of a high-fidelity reconstructed image with accurate semantics and rich details.
[0065] The EEG-fNIRS visual information decoding method described in this invention is particularly suitable for special populations who have lost the ability to express themselves verbally or physically due to damage to neuromuscular pathways (such as patients with aphasia, locked-in syndrome, or high-level paraplegia). By capturing their brain representation of the target visual content, the system can reconstruct the image of their "thoughts" in real time and present it on the interactive interface, thereby establishing a novel brain-computer communication channel that does not require peripheral motor involvement. Compared to traditional assistive devices that rely on eye tracking or weak electromyographic signals, this method directly decodes advanced visual intentions, significantly improving the freedom and richness of information transmission, and providing a potential high-bandwidth, non-invasive means of communication for patients with severe communication impairments.
[0066] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A method for decoding EEG-fNIRS visual information, characterized in that, Includes the following steps: Simultaneously acquire the EEG time-domain signal and fNIRS time-domain signal generated when the subject views the visual stimulus image, and further obtain the EEG time-spectrum and fNIRS time-spectrum; Input the EEG time spectrum and the fNIRS time spectrum into the Transformer encoder to generate EEG feature embedding vector and fNIRS feature embedding vector respectively. The EEG feature embedding vector and the fNIRS feature embedding vector are input together into a multimodal attention fusion module to generate a hybrid embedding vector; A two-stage decoding strategy is used to reconstruct the hybrid embedding vector into the final image.
2. The EEG-fNIRS visual information decoding method according to claim 1, characterized in that, The EEG time-of-flight spectrogram and the fNIRS time-of-flight spectrogram are input into the Transformer encoder to generate EEG feature embedding vectors and fNIRS feature embedding vectors, respectively. The EEG time-spectral map and the fNIRS time-spectral map are spatially divided into sequences composed of patches, which are then further formed into patch embedding sequences. The Patch embedding sequence is input into the Transformer encoder, and each modality outputs a high-dimensional feature vector, resulting in EEG feature embedding vectors and fNIRS feature embedding vectors.
3. The EEG-fNIRS visual information decoding method according to claim 2, characterized in that, The Transformer encoder is multi-layered, with each layer including a multi-head self-attention mechanism and a feedforward neural network.
4. The EEG-fNIRS visual information decoding method according to claim 1, characterized in that, The multimodal attention fusion module includes a bidirectional cross-attention unit and a gating fusion unit.
5. The EEG-fNIRS visual information decoding method according to claim 1, characterized in that, A two-stage decoding strategy is employed to reconstruct the final image from the hybrid embedding vectors, specifically as follows: The first-stage decoding strategy is as follows: the hybrid embedding vector is input in parallel into two independent decoding paths. The two independent decoding paths include an image-guided information decoding path and a text-guided information decoding path, which respectively obtain a low-resolution preliminary reconstructed image containing basic spatial information and a "semantic description" text information describing the core content of the image. The second-stage decoding strategy specifically involves inputting the preliminary reconstructed image and "semantic description" text information into the Stabel Diffusion XL diffusion model to generate the reconstructed image.
6. The EEG-fNIRS visual information decoding method according to claim 5, characterized in that, In the text-guided information decoding path, the hybrid embedding vector is adapted to dimensions through a linear projection layer and then input into a language model to be decoded into a "semantic description" text information describing the core content of the image.
7. The EEG-fNIRS visual information decoding method according to claim 4, characterized in that, The gated fusion unit dynamically learns fusion weights through a neural network and adaptively weights and sums the two enhanced EEG feature vectors and fNIRS feature vectors to ultimately generate a unified hybrid embedding vector.
8. The EEG-fNIRS visual information decoding method according to claim 5, characterized in that, The StabelDiffusion XL diffusion model uses IP-Adapter technology.
9. The EEG-fNIRS visual information decoding method according to claim 2, characterized in that, Linear projection is performed on the image patches that make up the Patch sequence, flattening them and mapping them into a fixed-dimensional vector to form the Patch embedding sequence.
10. The EEG-fNIRS visual information decoding method according to claim 9, characterized in that, The sequence of patches consists of multiple fixed-size, non-overlapping patches.