Cross-subject brain decoding system based on functional magnetic resonance image and feature decoupling

Abstract

The invention discloses a cross-subject brain decoding system based on a functional magnetic resonance image and feature decoupling. The cross-subject brain decoding system comprises a functional magnetic resonance image feature extraction module, an image generation module and an image output module, the functional magnetic resonance image feature extraction module extracts neural activity features related to visual stimulation from functional magnetic resonance imaging fMRI data, the image generation module comprises a feature decoupling module and a diffusion transformer module, and the feature decoupling module extracts common features among subjects from the neural activity features; the diffusion transformer module generates a high-accuracy image by using the common features as conditions, and the output module processes and outputs the generated image. According to the method, the features of the brain activity signals are extracted through the mask auto-encoder, the common features are extracted through the decoupling module and combined with the diffusion transformer, and external visual images can be efficiently and accurately reconstructed from functional magnetic resonance imaging (fMRI) data of different subjects.

Description

Technical Field

[0001] The present invention relates to the technical fields of neuroscience and computer vision, and in particular, to a cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling. Background Art

[0002] Reconstructing visual stimuli from functional magnetic resonance imaging (fMRI) signals is a key challenge in cognitive neuroscience and artificial intelligence research. This research not only promotes our understanding of the neural representation of visual perception but also provides important insights into the mechanism by which the brain processes complex sensory information.

[0003] Early fMRI decoding methods mainly used linear models such as ridge regression. Although these models are interpretable, they are difficult to capture the complex non-linear relationship between neural responses and visual stimuli. The rise of deep learning has changed this field, and generative adversarial networks (GANs) and diffusion models have achieved remarkable success in generating high-quality images from brain activity. For example, Cortex2Image first maps fMRI into a feature vector and a noise vector, and then generates images through an instance-conditioned generative adversarial network (IC-GAN). Meanwhile, MinD-Vis uses the fMRI representation learned by MAE to guide the latent diffusion model (LDM) to generate credible images through cross-attention. However, existing methods mainly focus on intra-individual visual decoding and are difficult to cope with the variability of inter-individual neural responses, limiting the accuracy of cross-subject decoding.

[0004] The specificity of inter-individual fMRI signals stems from genetic, environmental, and signal length differences, which pose a major challenge to the development of a unified brain decoding model. Existing cross-individual decoding methods, such as MindBridge, usually address the challenge of heterogeneous brain signals through two key technical components: First, use linear layers or adaptive pooling operations to achieve dimensional alignment of the input signals; Second, adopt individual-specific modules to extract common semantic features across individuals for image generation, thereby effectively decoding the brain signals of different individuals. However, the dependence of these methods on individualized parameters limits their scalability and hinders their functional generalization ability on new individuals, ultimately limiting their ability to identify general neural representations. Summary of the Invention

[0005] Object of the Invention: The present invention provides a cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling. By extracting the features of brain activity signals through a masked autoencoder, extracting common features via a decoupling module, and combining with a diffusion transformer, it can efficiently and accurately reconstruct external visual images from the functional magnetic resonance imaging (fMRI) data of different subjects.

[0006] Technical solution: A cross-subject brain decoding system based on functional magnetic resonance imaging (fMRI) and feature decoupling according to the present invention includes: an fMRI feature extraction module, an image generation module, and an image output module; the fMRI feature extraction module extracts neural activity features related to visual stimuli from fMRI data, the image generation module includes a feature decoupling module and a diffusion transformer module, the feature decoupling module extracts common features among subjects from the neural activity features, the diffusion transformer module uses these common features as conditions to generate images with high accuracy, and the output module processes and outputs the generated images.

[0007] Further, the fMRI feature extraction module includes an fMRI encoder and an fMRI decoder, both of which are implemented based on Vision Transformer (ViT). The fMRI data is masked to randomly mask a part of the fMRI data, and through the fMRI encoder, an fMRI representation is obtained, and then this representation is sent to the fMRI decoder to reconstruct the fMRI data.

[0008] Further, a two-stage training strategy is adopted to obtain the fMRI representation. In the first stage, the model is pre-trained on the large-scale publicly available Human Connectome Project S1200 (HCP S1200) dataset. In the second stage, based on the pre-trained model, the Natural Scenes Dataset (NSD) dataset is used for further fine-tuning to adapt to the specific task of visual stimulus reconstruction. By pre-training on the large-scale HCP S1200 dataset (778,600 fMRI images) and fine-tuning on the NSD dataset, this method can enhance the model's ability to extract brain neural patterns and improve the generalization performance across datasets and subjects.

[0009] Further, the system extracts meaningful features from complex spherical data. The intermediate features extracted by the fMRI encoder in the masked autoencoder are used for subsequent generation tasks, and the mean squared error is used as the reconstruction loss L recon to measure the similarity between the input data x and the reconstructed data x':

[0010]

[0011] where N is the number of samples.

[0012] Furthermore, the image generation module includes a feature decoupling module and a diffusion transformer module; after receiving the fMRI representation extracted by the functional magnetic resonance imaging feature extraction module, the image generation module obtains the common features through the feature decoupling module, and then uses the common features as control conditions to control the diffusion transformer module DiT to generate pictures from noise. This method effectively reduces the inter-subject variability in the brain decoding model.

[0013] Furthermore, the feature decoupling module uses channel attention and spatial attention as decouplers to capture the weights of channel information and spatial information, and these weights are used to capture the private features I of the subjects i , while the inverse weights are used to capture the common features S among the subjects i , and the specific formulas are as follows:

[0014] S i = z i ⊙ (1 - ψ(z i ))

[0015] I i = z i ⊙ ψ(z i )

[0016] Among them, the operation ⊙ represents element-wise multiplication, and ψ() represents the attention function.

[0017] Furthermore, an auxiliary subject classification task is introduced to ensure the ability of the decoupling module to distinguish common features and private features. For the private features I obtained by the decoupling module i , a multi-layer perceptron f cls is used for classification, and cross-entropy loss is used as the loss function. The specific formula is as follows:

[0018]

[0019] Among them, y i is the true subject label.

[0020] Furthermore, the diffusion transformer module DiT is a diffusion model based on the transformer architecture, including a forward diffusion process and a reverse denoising process. The forward diffusion process generates a series of latent variables y0, y1......, y by iteratively adding Gaussian noise to the image T , while the reverse process denoises by learning a denoising function ∈ θ to denoise y TRestore to y0. DiT completely replaces the classic U-Net architecture in the diffusion model with a transformer architecture, which has strong scaling capabilities and can efficiently capture dependencies in the data and generate high-quality results. DiT generates images by introducing noise and training a neural network to reverse the process of noise addition, combined with a transformer model.

[0021] Furthermore, the adaptive layer normalization method is used to fine-tune DiT, and the common feature S extracted by the feature decoupling module i is used as a regulatory control signal to generate adaptive scaling and offset parameters. These adaptive parameters are used to perform layer normalization on the input weights to modulate the layer normalization parameters in each transformer block to predict the noise ∈ added during the diffusion process. The specific formula is as follows:

[0022]

[0023] where y0, y t are latent variables, t is the time step, and ∈ θ is a set of noise reduction functions, which are implemented as transformers;

[0024] The overall objective function is as follows:

[0025] L total = L DiT + λ1 * L Classify

[0026] where λ1 is used to balance the relative importance of the classification loss.

[0027] Furthermore, the image output module performs a series of post-processing on the decoded image to ensure the best visualization effect of the data for subsequent analysis and interpretation.

[0028] Beneficial effects: Compared with the prior art, the present invention has the following remarkable advantages: The present invention solves the major challenge of reconstructing visual stimuli from fMRI data, including the problem of differences in data among different subjects. By introducing a feature learning algorithm based on surface fMRI data, it effectively addresses the problem of different data sizes, reduces the dimension of fMRI data, accurately captures the distances between brain regions, and learns the global brain activation pattern; In addition, the design of the feature decoupling module further improves the generalization across subjects by extracting common features among subjects, effectively improving the quality of the reconstructed image; It can generate images with more accurate details and richer semantic information, which helps to improve the accuracy of cross-subject brain decoding based on fMRI data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the system structure of the present invention.

[0030] Figure 2 Schematic diagram of the functional magnetic resonance imaging feature extraction module of the present invention.

[0031] Figure 3 Schematic diagram of the image generation module of the present invention.

[0032] Figure 4 Comparison chart of the generated images for all comparison methods. Detailed implementation manners

[0033] As Figure 1 shown, a cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling includes: a functional magnetic resonance imaging feature extraction module, an image generation module, and an image output module; the functional magnetic resonance imaging feature extraction module extracts neural activity features related to visual stimuli from functional magnetic resonance imaging (fMRI) data, the image generation module includes a feature decoupling module and a diffusion transformer module, the feature decoupling module extracts common features among subjects from the neural activity features, the diffusion transformer module uses these common features as conditions to generate highly accurate images, and the output module processes and outputs the generated images.

[0034] The functional magnetic resonance imaging feature extraction module is mainly implemented based on a masked autoencoder and adopts a two-stage training strategy. Among them, the masked autoencoder part includes an fMRI encoder and an fMRI decoder, and both the fMRI encoder and the fMRI decoder are implemented based on Vision Transformer (ViT).

[0035] In the initial stage of the feature extraction module, the fMRI data first passes through the fMRI encoder. Before inputting into the fMRI encoder, we add a mask to the fMRI data, randomly masking a part of the fMRI data, obtaining an fMRI representation through the fMRI encoder, and then sending this representation into the fMRI decoder to reconstruct the fMRI data. This process forces the model to learn how to reconstruct the lost information, thereby capturing the hidden structure of the data. In addition, in order to learn a more general functional magnetic resonance imaging representation, we adopt a two-stage training strategy. First, we pre-train the model on the large-scale publicly available Human Connectome Project S1200 (HCP S1200) dataset, which includes 778,600 fMRI images from 1200 subjects performing seven cognitive tasks. This pre-training enhances the robustness of the model and the ability to learn general fMRI representations. Based on this pre-trained model, we further fine-tune it using the Natural Scenes Dataset (NSD) dataset (training split) to make it adapt to the specific task of visual stimulus reconstruction.

[0036] Through the feature extraction module, the system can extract meaningful features from complex spherical data and learn the deep structure of the data through the intermediate representation of the masked autoencoder. We use the intermediate features extracted by the fMRI encoder in the masked autoencoder for subsequent generation tasks. We use the mean squared error as the reconstruction loss (L recon ) to measure the similarity between the input data x and the reconstructed data x':

[0037]

[0038] where N is the number of samples.

[0039] The image generation module uses the Diffusion Transformer (DiT) module as the backbone and integrates a feature decoupling module to extract common features among subjects for final decoding.

[0040] DiT operates on the image latent space and consists of two main parts: the forward diffusion process and the reverse denoising process. The forward diffusion process generates a series of latent variables y0, y1......, y by iteratively adding Gaussian noise to the image T , while the reverse process aims to denoise by learning a denoising function ∈ θ to restore y T to y0. After receiving the fMRI representation extracted by the feature extraction module, the image generation module first obtains the common features through the feature decoupling module, and then uses the common features as the control condition to control DiT to generate images from noise.

[0041] For the fMRI representation z i obtained in the feature extraction module, we obtain the common feature (S i ) through the feature decoupling module. The feature decoupling module uses channel attention and spatial attention as decouplers to capture the weights of channel information and spatial information, which are used to capture the private features of the subjects (I i ), while the inverse weights are used to capture the common features among the subjects (S i ), and the specific formula is as follows:

[0042] S i = z i ⊙ (1 - ψ(z i ))

[0043] I i = z i ⊙ ψ(z i )

[0044] where the operation ⊙ represents element-wise multiplication, and ψ() represents the attention function.

[0045] We also introduced an auxiliary subject classification task to ensure the ability of the decoupling module to distinguish common features and private features. Specifically, for the private features (I i ) obtained by the decoupling module, we used a multi-layer perceptron (f cls ) for classification and used cross-entropy loss as the loss function. The specific formula is as follows:

[0046]

[0047] where y i is the true subject label.

[0048] To provide more powerful guidance for the reconstruction task, we used the adaptive layer normalization method to fine-tune DiT. Specifically, the common features (S i ) were used as the adjustment control signal to modulate the layer normalization parameters in each transformer block to predict the noise ∈ added during the diffusion process. The specific formula is as follows:

[0049]

[0050] where y0, y t are the latent variables, t is the time step, and ∈ θ is a set of noise reduction functions, which are implemented as transformers.

[0051] The overall objective function is as follows:

[0052] L total = L DiT + λ1 * L Classify

[0053] where λ1 is used to balance the relative importance of the classification loss.

[0054] After training the above model, we obtained a model for cross-subject brain decoding based on functional magnetic resonance imaging. When we receive new subject data and input the functional magnetic resonance imaging data, we can decode the corresponding images.

[0055] The core function of the image output module is to perform a series of post-processing on the decoded images to ensure the best visualization effect of the data for subsequent analysis and interpretation. This module integrates a variety of image processing techniques, including denoising and smoothing, color correction, etc., aiming to improve the image quality and meet the user's needs.

[0056] We conducted numerous experiments on the large public dataset NSD for a comprehensive evaluation. The NSD dataset collected fMRI images of 8 subjects while they were viewing over 70,000 high-resolution natural scene images. We adopted N-way-Top-K as the semantic metric, which is a classification metric that can effectively measure the semantic fidelity of the generated images, and the larger the value, the better. In addition, we also used SSIM and PCC as the fine-grained metrics to evaluate the detail accuracy of the generated images, and the larger the value, the better. As shown in Table 1, Figure 2 it can be seen from the experimental results that our method performs better than the existing methods in all experimental settings and can generate images with better detail information and higher semantic fidelity.

[0057] Table 1. Comparative experiments of our system and existing brain decoding methods on three datasets

[0058]

[0059]

[0060] At the same time, we conducted ablation experiments on the effectiveness of our system and verified the important parts of the system. The experimental results are shown in Table 2. In Table 2, we gradually removed the two-stage training strategies in the feature decoupling module and the feature extraction module from the system, thus obtaining two baselines: w / o decoupling module, w / o decoupling module + pre-training. After adding pre-training, both the detail information and semantic fidelity of the generated images were significantly improved. After further adding the decoupling module to extract the common features among subjects, the quality of the generated images under each experimental setting was further improved. From the experimental results, we can see that each part of our method can effectively improve the model performance.

[0061] Table 2. Verification of the effectiveness of the important parts of the system

[0062]

Claims

1. A cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling, characterized in that, Including: A functional magnetic resonance imaging (fMRI) feature extraction module, an image generation module, and an image output module; the fMRI feature extraction module extracts neuroactivity features related to visual stimuli from fMRI data. The image generation module includes a feature decoupling module and a diffusion transformer module. The feature decoupling module extracts common features among subjects from the neuroactivity features, and the diffusion transformer module uses these common features as conditions to generate highly accurate images. The output module processes and outputs the generated images.

2. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling according to claim 1, wherein The fMRI feature extraction module includes an fMRI encoder and an fMRI decoder, both of which are implemented based on Vision Transformer (ViT). The fMRI data is masked to randomly obscure a part of the fMRI data. Through the fMRI encoder, an fMRI representation is obtained, and then this representation is fed into the fMRI decoder to reconstruct the fMRI data.

3. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling according to claim 2, wherein The two-stage training strategy is adopted to obtain the fMRI representation. In the first stage, the model is pre-trained on the large-scale publicly available Human Connectome Project S1200 (HCP S1200) dataset. In the second stage, based on the pre-trained model, the Natural Scenes Dataset (NSD) dataset is used for further fine-tuning to adapt to the specific task of visual stimulus reconstruction.

4. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling according to claim 2, wherein The system extracts meaningful features from complex spherical data. The intermediate features extracted by the fMRI encoder in the masked autoencoder are used for subsequent generation tasks, and the mean squared error is used as the reconstruction loss L recon to measure the similarity between the input data x and the reconstructed data x': Where N is the number of samples.

5. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling according to claim 1, characterized in that, The image generation module includes a feature decoupling module and a diffusion transformer module; after receiving the fMRI representation extracted by the fMRI feature extraction module, the image generation module obtains common features via the feature decoupling module, and then uses these common features as control conditions to control the diffusion transformer module DiT to generate images from noise.

6. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling according to claim 5, wherein The feature decoupling module uses channel attention and spatial attention as decouplers to capture the weights of channel information and spatial information, and these weights are used to capture the private features I of the subject i , while the inverse weights are used to capture the common features S among subjects i , and the specific formula is as follows: S i = z i ⊙(1 - ψ(z i )) I i = z i ⊙ ψ(z i ) Where the operation ⊙ represents element-wise multiplication, and ψ() represents the attention function.

7. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling as claimed in claim 5, wherein An auxiliary subject classification task is introduced to ensure the ability of the decoupling module to distinguish between common features and private features. For the private feature I obtained by the decoupling module i , a multi-layer perceptron f cls is used for classification, and the cross-entropy loss is used as the loss function. The specific formula is as follows: where y i is the true subject label.

8. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling according to claim 5, characterized in that, The Diffusion Transformer module DiT is a diffusion model based on the transformer architecture, including a forward diffusion process and a reverse denoising process. The forward diffusion process generates a series of latent variables y by iteratively adding Gaussian noise to the image + , y 1......, y . , and the reverse process denoises by learning a denoising function ∈ θ to restore y . to y + ..

9. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling according to claim 5, characterized in that Use the adaptive layer normalization method to fine-tune DiT, and utilize the common feature S extracted by the feature decoupling module i As the adjustment control signal, generate adaptive scaling and offset parameters, and use these adaptive parameters to perform layer normalization on the input weights, modulating the layer normalization parameters in each transformer block to predict the noise ∈ added during the diffusion process. The specific formula is as follows: where y + , y5 is a latent variable, t is a time step, and ∈9 is a set of noise reduction functions implemented as a transformer; The overall objective function is as follows: L 5o5al = L 1i2 + λ1 * L Classify Where λ1 is used to balance the relative importance of the classification loss.

10. The cross-subject brain decoding system based on functional magnetic resonance imaging and feature decoupling according to claim 1, wherein The image output module performs a series of post-processing on the decoded images to ensure the best visualization effect of the data for subsequent analysis and interpretation.

Images