Brain decoding system based on functional magnetic resonance image and potential diffusion model

Abstract

The invention discloses a brain decoding system based on a functional magnetic resonance image and a potential diffusion model. The brain decoding system comprises a functional magnetic resonance image feature extraction module, an image generation module and an image output module, the functional magnetic resonance image feature extraction module combines spherical convolution and a mask auto-encoder, neural activity features related to visual stimulation are extracted from functional magnetic resonance imaging fMRI data, and the image generation module generates an image with high semantic information by using the neural activity features as conditions. And the output module processes and outputs the generated image. The method solves the major challenges of reconstructing visual stimulation from fMRI data, including the problems of low signal-to-noise ratio and high dimension, effectively reduces the dimension of fMRI data, accurately captures the distance between brain regions, and learns local and global brain activation modes by introducing a feature learning algorithm based on surface fMRI data; besides, the design of an anti-noise potential diffusion model (NR-LDM) further improves the quality of a reconstructed image by reducing the influence of fMRI noise, an image with more accurate semantic information can be generated, and improvement of the accuracy of brain decoding based on fMRI data is facilitated.

Description

Technical Field

[0001] The present invention relates to the fields of neuroscience and computer vision technology, and in particular to a brain decoding system based on functional magnetic resonance imaging and a potential diffusion model. Background Art

[0002] With the rapid development of neuroscience and artificial intelligence technology, researchers have gradually begun to explore how to decode and reconstruct brain activity, especially through functional magnetic resonance imaging (fMRI) data to predict and reproduce external visual images perceived by humans. fMRI technology can capture changes in blood oxygen levels when the brain processes visual information, reflecting the activity of specific brain areas. However, fMRI signals have high dimensions, low signal-to-noise ratio, and complex time series characteristics, making it an extremely challenging task to extract neural activity features directly related to visual stimuli and convert them into images.

[0003] Traditional brain activity decoding methods usually rely on linear models or simple neural networks. Although these methods can decode fMRI signals to a certain extent, the generated images are usually of low quality and lack high-level semantic information. In order to solve these problems, in recent years, researchers have introduced more complex deep learning models, such as generative adversarial networks (GANs) and variational autoencoders (VAEs), to enhance decoding capabilities and improve the quality of generated images. However, these models still have problems such as insufficient generalization ability, unstable training, and difficulty in ensuring the quality of generated images when processing complex brain activity signals.

[0004] In this context, diffusion models, as an emerging generative model, have demonstrated great potential in high-quality image generation. Diffusion models can capture complex data distribution and generate high-fidelity images by gradually adding perturbations to the noise in the latent space and reversely generating clear images. However, how to effectively apply diffusion models to the field of brain activity decoding, especially in the process of generating high semantic information images from fMRI data, remains an unresolved problem. Summary of the invention

[0005] Purpose of the invention: The present invention provides a brain decoding system based on functional magnetic resonance imaging and latent diffusion model, which can efficiently and accurately reconstruct external visual images from functional magnetic resonance imaging (fMRI) data.

[0006] Technical solution: The brain decoding system based on functional magnetic resonance imaging and latent diffusion model described in the present invention includes: a functional magnetic resonance imaging feature extraction module, an image generation module and an image output module; the functional magnetic resonance imaging feature extraction module combines spherical convolution and masked autoencoder to extract neural activity features related to visual stimulation from functional magnetic resonance imaging fMRI data, the image generation module uses these neural activity features as conditions to generate images with high semantic information, and the output module processes and outputs the generated images.

[0007] Furthermore, the functional magnetic resonance imaging feature extraction module includes a spherical convolution part, a masked autoencoder part and a spherical deconvolution part; the spherical convolution part includes a spherical convolution layer and a spherical pooling layer, and the masked autoencoder part includes a Vision Transformer (ViT) encoder and a ViT decoder. The spherical deconvolution part includes a series of spherical deconvolution layers. The fMRI data first passes through the spherical convolution part. The spherical convolution layer effectively captures the local features of the data by smoothly moving the DirectNeighbor (DiNe) convolution kernel on the sphere while maintaining the integrity of the spatial information. The spherical pooling layer uses average pooling to aggregate small areas in the feature map into a single value, reducing the amount of calculation and extracting the main features. The feature map after spherical convolution will pass through the spherical pooling layer. Through the spherical convolution part, the dimension of the feature map gradually increases, indicating that higher-level features are gradually extracted. The obtained feature map is sent to the masked autoencoder part, which is implemented based on ViT. Through the ViT encoder, the fMRI representation is obtained, and then the representation is sent to the ViT decoder to reconstruct the feature map. This process forces the model to learn how to reconstruct the lost information, thereby capturing the hidden structure of the data. The reconstructed feature maps are processed by the spherical deconvolution part, and these feature maps are gradually restored to the dimension and spatial resolution of the original data.

[0008] Furthermore, the intermediate features extracted by the ViT encoder in the mask autoencoder are used for subsequent generation tasks, and the mean square error is used as the reconstruction loss (L recon ) to measure the similarity between the input data x and the reconstructed data x':

[0009]

[0010] Where S is the number of samples and s represents the sth sample.

[0011] Furthermore, the image generation module is based on the latent diffusion model (LDM), which operates on the image latent space, including the forward diffusion process and the reverse denoising process.

[0012] Furthermore, the forward diffusion process generates a series of latent variables y0, y1, ..., y by iteratively adding Gaussian noise to the image. T , while the reverse process aims to learn a denoising function ∈ θ To remove noise, y T After receiving the fMRI representation extracted by the feature extraction module, the image generation module first obtains the main fMRI representation through pseudo principal component analysis (pseudo PCA), and then uses the main fMRI representation as a control condition to control the LDM to generate images from noise;

[0013] The pseudo PCA part uses linear projection (W k ) to approximate PCA to extract the main neurons in the network, identify and retain the key features (principal components) that capture most of the data, while discarding less important components that are usually associated with noise, thereby effectively removing noise. For the fMRI representation obtained in the feature extraction module, fMRI , the main fMRI representation z was obtained through pseudo PCA fMRI , the specific formula is as follows:

[0014] Z fMRI =e fMRI ×W k .

[0015] Furthermore, the alignment loss (L Align ), to ensure that pseudo-PCA learns the main semantic concepts while minimizing the impact of noise; L Align We quantify the difference between the embedding of generated images and the primary fMRI embedding, thereby improving the effectiveness of fMRI features for visual stimulus reconstruction; for the representation of generated images gen , through the projection layer ρ θ becomes and z fMRI The same dimension, calculate the mean square error as the loss, the specific formula is as follows:

[0016]

[0017] Furthermore, in order to provide more powerful guidance for the reconstruction task, cross-attention conditioning and time-step conditioning are integrated into LDM to calculate the true noise ∈ and the predicted noise ∈ θ (y t ,t,z fMRI ,σ θ (z fMRI )) is taken as the loss, and the specific formula is as follows:

[0018]

[0019] Where y0 represents the latent variable of a given image, z is the conditional control variable, and t represents the time step. is the corresponding mathematical expectation, z fMRI represents the main fMRI representation, σ θ (z fMRI ) represents the dimension projector σ θ The time step embedding of ∈ represents the real noise, ∈ θ is a set of denoising functions, which are implemented as U-Net.

[0020] Furthermore, the perceptual loss (L PSM ), L PSM Computationally generated image representation (e gen ) and the real image representation (e orig ), guiding the model to give priority to high-level semantic information. The specific formula is as follows:

[0021]

[0022] Furthermore, the overall objective function is as follows:

[0023] L total =L DC-LDM +λ1*L Align +λ2*L PSM

[0024] Among them, λ1 and λ2 are used to balance the relative importance of alignment loss and perceptual loss.

[0025] Furthermore, the image output module performs a series of processing on the generated images to ensure optimal visualization of the data for subsequent analysis and interpretation.

[0026] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the present invention solves the major challenges of reconstructing visual stimuli from fMRI data, including its low signal-to-noise ratio and high dimensionality. By introducing a feature learning algorithm based on surface fMRI data, the dimensionality of fMRI data is effectively reduced, the distance between brain regions is accurately captured, and local and global brain activation patterns are learned; in addition, the design of the noise-resistant latent diffusion model (NR-LDM) further improves the quality of reconstructed images by alleviating the impact of fMRI noise, and images with more accurate semantic information can be generated, which helps to improve the accuracy of brain decoding based on fMRI data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the structure of the brain decoding system of the present invention.

[0028] Figure 2Schematic diagram of the functional magnetic resonance imaging feature extraction module of the present invention.

[0029] Figure 3 Schematic diagram of the image generation module of the present invention.

[0030] Figure 4 Comparison plots of generated images for all compared methods. DETAILED DESCRIPTION

[0031] like Figure 1 As shown, a brain decoding system based on functional magnetic resonance imaging and latent diffusion model includes: a functional magnetic resonance imaging feature extraction module, an image generation module and an image output module; the functional magnetic resonance imaging feature extraction module combines spherical convolution and mask autoencoder to extract neural activity features related to visual stimulation from functional magnetic resonance imaging (fMRI) data, the image generation module uses these neural activity features as conditions to generate images with high semantic information, and the output module processes and outputs the generated images.

[0032] like Figure 2 As shown in the figure, the functional magnetic resonance imaging feature extraction module includes a spherical convolution part, a masked autoencoder part and a spherical deconvolution part; the spherical convolution part includes a spherical convolution layer and a spherical pooling layer, the masked autoencoder part includes a ViT encoder and a ViT decoder, and the spherical deconvolution part includes a series of spherical deconvolution layers; the fMRI data first passes through the spherical convolution part, the spherical convolution layer effectively captures the local features of the data by smoothly moving the DiNe convolution kernel on the spherical surface while maintaining the integrity of the spatial information, and the spherical pooling layer uses an average pooling method to aggregate small areas in the feature map into a single value, which reduces the amount of calculation and To extract the main features, the feature map after spherical convolution will pass through the spherical pooling layer. After the spherical convolution part, the dimension of the feature map gradually increases, indicating that higher-level features are gradually extracted; the obtained feature map is sent to the masked autoencoder part, which is implemented based on ViT. Through the ViT encoder, the fMRI representation is obtained, and then the representation is sent to the ViT decoder to reconstruct the feature map. This process forces the model to learn how to reconstruct the lost information, thereby capturing the hidden structure of the data; the reconstructed feature maps are processed by the spherical deconvolution part, and these feature maps are gradually restored to the dimension and spatial resolution of the original data.

[0033] The intermediate features extracted by the ViT encoder in the mask autoencoder are used for subsequent generation tasks, and the mean square error is used as the reconstruction loss (L recon ) to measure the similarity between the input data x and the reconstructed data x':

[0034]

[0035] Where S is the number of samples and s represents the sth sample.

[0036] like Figure 3 As shown in Figure 1, the image generation module is based on the latent diffusion model (LDM), which operates on the image latent space, including the forward diffusion process and the reverse denoising process.

[0037] The forward diffusion process generates a series of latent variables y0, y1, ..., y by iteratively adding Gaussian noise to the image. T , while the reverse process aims to learn a denoising function ∈ θ To remove noise, y T After receiving the fMRI representation extracted by the feature extraction module, the image generation module first obtains the main fMRI representation through pseudo PCA, and then uses the main fMRI representation as a control condition to control LDM to generate images from noise;

[0038] The pseudo PCA part uses linear projection (W k ) to approximate PCA to extract the main neurons in the network, identify and retain the key features (principal components) that capture most of the data, while discarding less important components that are usually associated with noise, thereby effectively removing noise. For the fMRI representation obtained in the feature extraction module, fMRI , the main fMRI representation z was obtained through pseudo PCA fMRI , the specific formula is as follows:

[0039] z fMRI =e fMRI ×W k .

[0040] The alignment loss (L Align ), to ensure that pseudo-PCA learns the main semantic concepts while minimizing the impact of noise; L Align We quantify the difference between the embedding of generated images and the primary fMRI embedding, thereby improving the effectiveness of fMRI features for visual stimulus reconstruction; for the representation of generated images gen , which is transformed into z through the projection layer fMRI The same dimension, calculate the mean square error as the loss, the specific formula is as follows:

[0041]

[0042] In order to provide more powerful guidance for the reconstruction task, cross-attention conditioning and time-step conditioning are also integrated into LDM to calculate the true noise ∈ and the predicted noise ∈ θ (y t ,t,z fMRI ,σ θ (zfMRI )) is taken as the loss, and the specific formula is as follows:

[0043]

[0044] Where y0 represents the latent variable of a given image, z is the conditional control variable, and t represents the time step. is the corresponding mathematical expectation, z fMRI represents the main fMRI representation, σ θ (z fMRI ) represents the dimension projector σ θ The time step embedding of ∈ represents the real noise, ∈ θ is a set of denoising functions, which are implemented as U-Net.

[0045] Added perceptual loss (L PSM ), L PSM Computationally generated image representation (e gen ) and the real image representation (e orig ), guiding the model to give priority to high-level semantic information. The specific formula is as follows:

[0046]

[0047] The overall objective function is as follows:

[0048] L total =L DC-LDM +λ1*L Align +λ2*L PSM

[0049] Among them, λ1 and λ2 are used to balance the relative importance of alignment loss and perceptual loss.

[0050] The image output module performs a series of processing on the generated images to ensure the best visualization of the data and facilitate subsequent analysis and interpretation.

[0051] Numerous experiments were conducted on three public datasets for comprehensive evaluation, including NSD, BOLD5000 and GOD. The NSD dataset collected fMRI images of 8 subjects while viewing more than 70,000 high-resolution natural scene images. The BOLD5000 dataset includes fMRI imaging data from 4 subjects, recording their brain activity while viewing images from COCO, ImageNet, and Scene Recognition databases, with a total of 5,254 images. The GOD dataset contains 1,250 different images from 200 different categories. N-way-Top-K is used as the indicator, which is a classification indicator that can effectively measure the semantic fidelity of the generated image. The larger the value, the better. As shown in Tables 1 and Figure 4 As shown, it can be seen from the experimental results that compared with the existing methods, the present invention has better performance on all data sets and can generate images with higher semantic information and more credibility.

[0052] Table 1 Comparative experiments between the system of the present invention and existing brain decoding methods on three data sets

[0053]

[0054]

[0055] At the same time, ablation experiments were conducted to verify the effectiveness of the system and verify each part of the image generation module. The experimental results are shown in Table 2. In Table 2, LDM is used as the baseline model, and perceptual loss, pseudo PCA, and alignment loss are added to reduce the influence of noise in the learned fMRI feature representation. After adding perceptual loss, the semantic fidelity of the generated image is improved. After further adding pseudo PCA to extract the main fMRI representation, the quality is further improved. Finally, alignment loss is incorporated to effectively learn the main semantic concepts. It can be seen from the experimental results that each part can effectively improve the performance of the model.

[0056] Table 2 Validity verification of each part of the system image generation module

[0057]

Claims

1. A brain decoding system based on functional magnetic resonance imaging and latent diffusion model, characterized in that: include: Functional magnetic resonance imaging feature extraction module, image generation module and image output module; the functional magnetic resonance imaging feature extraction module combines spherical convolution and mask autoencoder to extract neural activity features related to visual stimulation from functional magnetic resonance imaging (fMRI) data, the image generation module uses these neural activity features as conditions to generate images with high semantic information, and the output module processes and outputs the generated images.

2. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 1, characterized in that: The functional magnetic resonance imaging feature extraction module includes a spherical convolution part, a masked autoencoder part and a spherical deconvolution part; the spherical convolution part includes a spherical convolution layer and a spherical pooling layer, the masked autoencoder part includes a ViT encoder and a ViT decoder, and the spherical deconvolution part includes a series of spherical deconvolution layers; the fMRI data first passes through the spherical convolution part, the spherical convolution layer smoothly moves the DiNe convolution kernel on the sphere, and the spherical pooling layer uses average pooling to aggregate small areas in the feature map into a single value. The feature map after spherical convolution will pass through the spherical pooling layer. Through the spherical convolution part, the dimension of the feature map gradually increases, indicating that higher-level features are gradually extracted; the obtained feature map is sent to the masked autoencoder part, which is implemented based on ViT. Through the ViT encoder, the fMRI representation is obtained, and then the representation is sent to the ViT decoder to reconstruct the feature map; The reconstructed feature maps are processed by the spherical deconvolution part, and these feature maps are gradually restored to the dimension and spatial resolution of the original data.

3. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 2, characterized in that: The intermediate features extracted by the ViT encoder in the mask autoencoder are used for subsequent generation tasks, and the mean square error is used as the reconstruction loss L recon To measure the similarity between the input data x and the reconstructed data x': Where S is the number of samples and s represents the sth sample.

4. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 1, characterized in that: The image generation module is based on the latent diffusion model (LDM), which operates on the image latent space and includes a forward diffusion process and a reverse denoising process.

5. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 4, characterized in that: The forward diffusion process generates a series of latent variables y0, y1, ..., y by iteratively adding Gaussian noise to the image. T , while the reverse process aims to learn a denoising function ∈ θ To remove noise, y T After receiving the fMRI representation extracted by the feature extraction module, the image generation module first obtains the main fMRI representation through pseudo principal component analysis pseudo PCA, and then uses the main fMRI representation as a control condition to control LDM to generate images from noise; The pseudo PCA part uses linear projection W k To approximate PCA to extract the main neurons in the network, identify and retain the key features that capture most of the data, while discarding less important components that are usually related to noise. For the fMRI representation obtained in the feature extraction module, fMRI , the main fMRI representation z was obtained through pseudo PCA fMRI , the specific formula is as follows: with fMRI =e fMRI ×W k 。 6. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 5, characterized in that The alignment loss L Align , L Align We quantify the difference between the embeddings of generated images and the primary fMRI embeddings; for the representation of the generated images, we gen , through the projection layer ρ θ becomes and z fMRI The same dimension, calculate the mean square error as the loss, the specific formula is as follows:

7. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 6, characterized in that: In order to provide more powerful guidance for the reconstruction task, cross-attention conditioning and time-step conditioning are also integrated into LDM to calculate the true noise ∈ and the predicted noise ∈ θ (y t ,t,z fMRI ,σ θ (z fMRI )) is taken as the loss, and the specific formula is as follows: Where y0 represents the latent variable of a given image, z is the conditional control variable, and t represents the time step. is the corresponding mathematical expectation, z fMRI represents the main fMRI representation, σ θ (z fMRI ) represents the dimension projector σ θ The time step embedding of ∈ represents the real noise, ∈ θ is a set of denoising functions, which are implemented as U-Net.

8. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 7, characterized in that: Added perceptual loss L PSM , L PSM Computationally generated image representations gen and real image representation orig The similarity between them guides the model to give priority to high-level semantic information. The specific formula is as follows:

9. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 8, characterized in that: The overall objective function is as follows: L total =L DC-LDM +λ1*L Align +λ2*L PSM Among them, λ1 and λ2 are used to balance the relative importance of alignment loss and perceptual loss.

10. The brain decoding system based on functional magnetic resonance imaging and latent diffusion model as claimed in claim 1, characterized in that: The image output module performs a series of processing on the generated images to ensure the best visualization of the data and facilitate subsequent analysis and interpretation.

Images