Brain image decoding method based on CLIP feature fusion and neural layer mapping

Abstract

This invention discloses a brain image decoding method based on CLIP feature fusion and neural layer mapping. The method includes acquiring and preprocessing functional magnetic resonance imaging (fMRI) data of a subject observing an image; dividing the preprocessed fMRI data into low-level and high-level visual regions; constructing a bi-branch mapping network and performing hierarchical mapping; fusing multi-layer CLIP visual features; completing training; and generating images using a diffusion model. This method, without introducing additional VAE detail channels, utilizes the fine-grained visual detail information contained in the intermediate layers of the CLIP visual encoder and combines low-level / high-level functional partitioning of the visual cortex to perform hierarchical modeling and mapping of fMRI signals. This improves the semantic consistency and visual detail fidelity of the reconstructed image while reducing the size of trainable parameters and implementation complexity.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of brain-computer interface, artificial intelligence and medical image processing. Specifically, it relates to a brain image decoding method based on CLIP feature fusion and neural layer mapping. In particular, it relates to a brain image decoding method that utilizes multi-layer visual feature fusion of the contrastive language-image pre-trained model CLIP, and combines the functional hierarchy of the visual cortex to perform partition mapping of fMRI signals, thereby driving a diffusion generation model to achieve visual stimulus image reconstruction. Background Technology

[0002] Brain image decoding technology aims to reconstruct visual content perceived or viewed by a subject based on brain activity signals, and has significant application value in fields such as brain-computer interfaces, human-computer interaction, neuroscience research, and clinical auxiliary assessment. Functional magnetic resonance imaging (fMRI), as a non-invasive brain imaging technique, can acquire blood oxygen level-dependent (BOLD) signals related to visual stimuli across the entire brain and form high-dimensional neural response patterns at the voxel level. Effectively mapping these high-dimensional fMRI signals into representations usable for image generation is key to achieving high-quality image reconstruction.

[0003] In recent years, the development of deep learning, especially large-scale pre-trained visual and generative models, has driven the rapid iteration of fMRI image reconstruction methods. In existing technologies, a mainstream approach typically employs a Contrastive Language-Image Pre-training (CLIP) model as a cross-modal representation space. This maps fMRI signals to the final layer embeddings of the CLIP visual encoder and / or CLIP text encoder, and then uses these features as conditional inputs or guiding signals to drive a diffusion model to generate images. This type of method can, to a certain extent, ensure the semantic consistency of the reconstruction results, meaning that the generated images have a high degree of matching with real stimuli at the category, scene, or conceptual level. However, CLIP final layer features are usually dominated by abstract semantics and have limited ability to express fine-grained visual details such as texture, edges, and local shapes. This results in deficiencies in structural detail and local fidelity in the generated images, manifesting as unclear object outlines, missing textures, or distorted local structures.

[0004] To compensate for the aforementioned lack of detail, another approach in existing technologies often introduces additional detail recovery channels. For example, fMRI signals are further mapped to the low-level latent space of a variational autoencoder (VAE) or image generation model to provide stronger structural and texture constraints. While this type of method can improve detail reconstruction capabilities, it typically requires training an additional mapping network from fMRI to VAE latent variables and coupling it with the diffusion generation process. This leads to a significant increase in the number of trainable parameters, a complex training process, and increased computational and storage overhead, making it unsuitable for deployment and application in resource-constrained scenarios. Furthermore, the introduction of multi-channel pipelines increases the difficulty of system engineering implementation and the cost of parameter tuning, and improper coordination constraints between different channels may also lead to semantic drift or instability.

[0005] On the one hand, from a neuroscience perspective, the human visual cortex exhibits a clear hierarchical organizational pattern: early visual regions, represented by V1, V2, and V3, primarily process basic visual features such as edges, orientation, color, and texture; while higher visual regions, represented by LOC, FFA, PPA, and VWFA, are more involved in the integration and expression of high-level semantic information such as complex objects, faces, scenes, and text. Correspondingly, the CLIP visual encoder also exhibits hierarchical representation characteristics: the middle layers often contain richer local structural and object detail information, while the final layers are more inclined towards semantic abstraction. Current technologies generally map fMRI signals within the visual cortex indiscriminately to the feature space of the final CLIP layer, failing to fully utilize the functional hierarchical differences of the visual cortex and the complementarity of multiple layers of features within the CLIP. This easily leads to hierarchical mismatch between brain signals and target representations, limiting cross-modal alignment accuracy and decoding stability. Meanwhile, although CLIP intermediate layer features have strong detail representation capabilities, their direct use in reconstruction may introduce noise representations unrelated to the target semantics, leading to problems such as background clutter, irrelevant objects, or local artifacts in the generated image. Therefore, how to reasonably utilize CLIP multi-layer visual features and establish a corresponding mapping with the visual cortex hierarchical structure without introducing additional VAE detail channels, so as to balance semantic consistency and detail fidelity, and reduce model complexity and parameter scale, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a brain image decoding method based on CLIP feature fusion and neural layer mapping, in order to solve the problems in the prior art that rely on high-parameter VAE pipelines due to ignoring CLIP intermediate layer information, and the problem that the reconstruction image details are lost due to simply relying on the semantic features of the final layer.

[0007] This invention employs the following technical solution to achieve its objective: a brain image decoding method based on CLIP feature fusion and neural layer mapping, comprising the following steps: (1) Acquire functional magnetic resonance imaging (fMRI) data of subjects observing images and perform preprocessing; (2) The preprocessed functional magnetic resonance imaging (fMRI) data were divided into low-level and high-level visual regions: the detail fMRI signal corresponding to the low-level visual region was... The semantic fMRI signal corresponding to the higher visual areas is ; (3) Construct a dual-branch mapping network and perform hierarchical mapping: Construct a dual-branch mapping network containing an image branch and a text branch to realize hierarchical mapping from fMRI signals to CLIP features, wherein: the text branch receives semantic fMRI signals. Output the predicted CLIP text features The predicted CLIP text features are used to provide semantic constraints on the generated image; the image branch receives fMRI signals. With semantic fMRI signals Output the predicted CLIP visual intermediate layer detail features respectively. Compared with the predicted CLIP visual final layer semantic features ; (4) Multi-layer CLIP visual feature fusion: fusion of predicted CLIP visual intermediate layer detail features Compared with the predicted CLIP visual final layer semantic features The average value is used for fusion to obtain the fused image features. The fused image features It contains fine-grained visual information and high-level semantic information; (5) Complete training and generate images using a diffusion model: Construct a supervisory signal pair to train the bi-branch mapping network in step (3), and input the fMRI data to be decoded into the trained bi-branch mapping network to obtain the image. and The data is then input into the Versatile Diffusion model to generate the corresponding reconstructed image.

[0008] Optionally, in step (1), acquiring the functional magnetic resonance imaging (fMRI) data of the subject when observing the image includes acquiring the stimulus image data corresponding to the visual stimulus image of the subject and the text description data corresponding to the stimulus image.

[0009] Optionally, the low-level visual region divided in step (2) is at least one of V1, V2, and V3, and the high-level visual region is at least one of LOC, FFA, PPA, and VWFA.

[0010] Optionally, in step (4), the CLIP ViT-L / 14 visual encoder is used to extract features from the visual stimulus image of the subject, and the CLIP text encoder is a text encoder that is compatible with CLIP ViT-L / 14.

[0011] Optionally, in step (4), when extracting features from the visual stimulus image of the subject, 10 to 20 layers of features from the stimulus image are extracted and averaged to obtain detailed features, which are used to characterize local texture, shape, and structural details, denoted as... The last layer of features from the stimulus image is extracted to represent high-level semantic and category information, denoted as . The intermediate layer detail features of the stimulus image are fused with the final layer semantic features by calculating the average value to form a fused CLIP visual feature, denoted as . .

[0012] Optionally, in step (3), a dual-branch mapping network is constructed, including a text branch and an image branch, to map the obtained hierarchical fMRI signals to the CLIP feature space. The text branch is used to map the semantic fMRI signals... Mapped to the final semantic feature space of the CLIP text encoder, the output is the predicted CLIP text features. Image branches are used to provide semantic conditions for diffusion generation; image branches are used to extract detailed fMRI signals. With semantic fMRI signals These are mapped to the intermediate layer detail feature space and the final layer semantic feature space of the CLIP visual encoder, respectively, to output the predicted features. and .

[0013] Optionally, in step (5), the dual-branch mapping network is trained using the real CLIP visual intermediate layer detail features and real CLIP visual final layer semantic features extracted by the CLIP visual encoder from the stimulus image, and the real CLIP text features extracted by the CLIP text encoder from the text branch description as supervision signals.

[0014] Optionally, in step (5), during training, the CLIP visual encoder and CLIP text encoder are frozen, and only the parameters of the dual-branch mapping network are updated.

[0015] Optionally, in step (5), the Versatile Diffusion receives the predicted CLIP text features simultaneously when generating the image. With the fused image features As a conditional control signal.

[0016] Compared with the prior art, the present invention has the following beneficial effects: without introducing additional VAE detail channels, the method utilizes the fine-grained visual detail information contained in the intermediate layer of the CLIP visual encoder, and combines the low-level / high-level functional partitions of the visual cortex to perform hierarchical modeling and mapping of fMRI signals, thereby improving the semantic consistency and visual detail fidelity of the reconstructed image, and reducing the scale of trainable parameters and implementation complexity. Attached Figure Description

[0017] Figure 1 This is a flowchart of the dual-branch mapping network decoding method proposed in this invention; Figure 2 This is a schematic diagram of the multi-layer mapping between fMRI features and CLIP features proposed in this invention. Detailed Implementation

[0018] The following detailed description, in conjunction with the technical solution of this invention, provides a specific implementation of the brain image decoding method based on multi-layer CLIP feature fusion and neural hierarchical mapping described in this invention. For ease of description, key symbols are conventionally used below: detailed fMRI signals are denoted as... Semantic fMRI signal is denoted as Predicted CLIP text features are denoted as Predicting CLIP visual intermediate layer detail features is denoted as... The predicted semantic features of the CLIP visual final layer are denoted as The fused predicted image features are denoted as .

[0019] A brain image decoding method based on CLIP feature fusion and neural layer mapping includes the following steps: (1) Acquire functional magnetic resonance imaging (fMRI) data of subjects observing images and perform preprocessing; (2) The preprocessed functional magnetic resonance imaging (fMRI) data were divided into low-level and high-level visual regions: the detail fMRI signal corresponding to the low-level visual region was... The semantic fMRI signal corresponding to the higher visual areas is ; (3) Construct a dual-branch mapping network and perform hierarchical mapping: Construct a dual-branch mapping network containing an image branch and a text branch to realize hierarchical mapping from fMRI signals to CLIP features, wherein: the text branch receives semantic fMRI signals. Output the predicted CLIP text features The predicted CLIP text features are used to provide semantic constraints on the generated image; the image branch receives fMRI signals. With semantic fMRI signals Output the predicted CLIP visual intermediate layer detail features respectively. Compared with the predicted CLIP visual final layer semantic features ; (4) Multi-layer CLIP visual feature fusion: fusion of predicted CLIP visual intermediate layer detail features Compared with the predicted CLIP visual final layer semantic features The average value is used for fusion to obtain the fused image features. The fused image features It contains fine-grained visual information and high-level semantic information; (5) Complete training and generate images using a diffusion model: Construct a supervisory signal pair to train the bi-branch mapping network in step (3), and input the fMRI data to be decoded into the trained bi-branch mapping network to obtain the image. and The data is then input into the Versatile Diffusion model to generate the corresponding reconstructed image.

[0020] This embodiment will describe the details of each step in the order of the above method.

[0021] In step (1), fMRI data is acquired when the subject views natural image stimuli. Simultaneously, stimulus image data corresponding one-to-one with the fMRI data, and textual description data corresponding to the stimulus images (e.g., natural language descriptions of the stimulus images) are acquired. The fMRI data may be single-trial beta plots, event-related response features, or other signal forms that characterize visual stimulus-induced brain activity. The fMRI data is preprocessed to reduce noise and improve cross-trial consistency. The preprocessing includes: denoising, abnormal voxel removal, standardization, normalization, inter-session alignment and / or drift correction, etc.

[0022] Specifically, the fMRI data can be a single-trial beta estimate for each stimulus image, or a response vector obtained by modeling the stimulus presentation period. Each training sample includes at least one stimulus image. ; fMRI response vector corresponding to the stimulus image; set of text descriptions corresponding to the stimulus image Preprocessing of fMRI data includes: (a) Voxel standardization / normalization: The response of each voxel is z-score standardized within the training set, or the sample vector is normalized to reduce the difference in the dimensions of different voxels. (b) Denoising: Voxels with abnormal fluctuations are removed or truncated, and voxels with low signal-to-noise ratio are screened out; (c) Sample alignment: Establish an index mapping between stimulus images, text descriptions and fMRI samples according to the stimulus presentation order to ensure a one-to-one correspondence among the three; (d) Vectorization: The voxel signals within the ROI of each sample are flattened into vectors in a fixed order and used as inputs to the subsequent mapping network.

[0023] In step (2), based on the functional hierarchical organization of the visual cortex, the fMRI voxels of visually related brain regions are divided into two levels of ROI signals, including the fMRI signals of the entire visual region, defined as detail signals, denoted as... fMRI signals in higher visual areas are defined as semantic fMRI signals, denoted as... Among them, detail signals are used to characterize neural responses related to basic visual features such as edges, colors, and textures, and optionally include V1, V2, V3, and higher-level visual areas; semantic fMRI is used to characterize neural responses related to higher-level semantics such as objects, faces, scenes, and text, including higher-level visual areas such as LOC, FFA, PPA, and VWFA.

[0024] The specific steps are as follows: (a) Source of ROI definition: The set of voxels contained in each ROI can be determined by pre-defined brain region atlases such as the Glasser atlas or the Difumo atlas.

[0025] (b) Detail fMRI: This includes fMRI signals from low-level visual regions such as V1, V2, and V3, as well as fMRI signals from high-level visual regions. The voxel signals from these ROIs are stitched together to obtain the detail fMRI signal. .

[0026] (c) Semantic fMRI: This includes fMRI signals from higher visual regions such as LOC, FFA, PPA, and VWFA. The voxel signals of this type of ROI are stitched together to obtain the semantic fMRI signal. .

[0027] In step (3), a dual-branch mapping network is constructed, including a text branch and an image branch, to map the hierarchical fMRI signals obtained in step (2) to the CLIP feature space. The text branch is used to map the semantic fMRI signals... Mapped to the final semantic feature space of the CLIP text encoder, the output is the predicted CLIP text features. This is used to provide semantic conditions for diffusion generation. Image branching is used to extract detailed fMRI signals. With semantic fMRI signals These are mapped to the intermediate layer detail feature space and the final layer semantic feature space of the CLIP visual encoder, respectively, to output the predicted features. and The dual-branch mapping networks are all implemented using a multilayer perceptron (MLP). Each branch includes an encoding network and an output head for feature mapping, so that the dimension of the output feature is consistent with the dimension of the corresponding CLIP target feature, which facilitates loss calculation during training and serves as conditional input for the generative model during the inference phase.

[0028] Specifically: (a) Image branch: used to output intermediate layer detail prediction features and final layer semantic prediction features, and further fuse them to form The image branch includes: detail mapping subnetwork: input Output Semantic mapping subnetwork: Input Output ; Fusion module: for and Calculate the average, merge, and pass through the book to obtain .

[0029] (b) Text branch: used to translate semantic fMRI signals Mapped to CLIP text feature space, output The text branch includes: a semantic fMRI feature mapping network: the input is... The output is the same as Same-dimensional vector representation; Output head: generates predicted text features .

[0030] In step (4), CLIP ViT-L / 14 is used as the feature extraction backbone network. The CLIP visual encoder is used to extract visual features from the stimulus image, and the CLIP text encoder is used to extract text features from the text description. To ensure the stability of the training supervision signal, the CLIP parameters are preferably frozen during the training of the mapping network. For each image stimulus, the corresponding middle 10-20 layers of features for the CLIP ViT-L / 14 model are extracted, and averaged to obtain the middle layer detail features. The final semantic features are obtained by extracting the features from the last layer. The intermediate layer detail features are fused with the final layer semantic features to obtain the fused visual features. The fused visual features serve as the training target features for the image branch and are also used as one of the visual conditional inputs to the diffusion generation model during inference. For the text branch, the text descriptions corresponding to the image stimuli are input into the CLIP text encoder, and the final layer features are extracted to obtain the final text embedding. For a given image stimulus with multiple text descriptions, the semantic features of the text are obtained by averaging the corresponding last-layer text embeddings. ; In step (5), a supervision signal is constructed during training, and hierarchical correspondences are learned in the dual-branch mapping network. In the text branch, semantic fMRI signals are... Input text branch And compared with the true CLIP text features obtained by the CLIP text encoder from the text description. Alignment training is performed. For cases where the same stimulus image corresponds to multiple text descriptions, the average of the features from the multiple texts can be optionally taken as the true measure. In image branch alignment, detailed fMRI signals are captured. The detail mapping subnetwork of the input image branch is obtained and with real intermediate layer details Alignment; semantic fMRI signals The semantic mapping subnetwork of the input image branch obtains and with the true final layer semantic features Alignment; and based on prediction and Computational prediction fusion features Features that blend with reality Alignment.

[0031] Specifically: the CLIP visual encoder and CLIP text encoder are frozen, and only the parameters of the two-branch mapping network are updated to reduce training costs and improve training stability. The supervised objectives for hierarchical mapping training include: text targets: real text features. Image detail target: Real intermediate layer detail features Image semantic target: Real last-layer semantic features Image fusion target: realistically fused visual features The network forward pass yields the following prediction results: The difference between the predictions and the actual values ​​is calculated to update the mapping network parameters. During training, the mean squared error (MSE) loss function is used, which includes: ; ; ; .

[0032] In step (5), the fMRI data of the sample to be decoded undergoes the same preprocessing as in step (1), and is then divided into layers according to step (2). and The features are then input into the trained dual-branch mapping network to obtain the predicted text features. Visual features fused with prediction .Will and Simultaneously, it is input as a condition to Versatile Diffusion to control the diffusion generation process to generate reconstructed images, and finally outputs the reconstruction results corresponding to the fMRI signal.

[0033] Specific Implementation: To verify the feasibility and effectiveness of the method of the present invention, this embodiment uses the publicly available Natural Scenes Dataset (NSD) for experiments. This dataset contains high-resolution fMRI data generated by subjects viewing natural image stimuli under a 7T MRI scanner, and provides textual descriptions corresponding to the stimulus images. This embodiment employs a subject-specific setting, i.e., a separate two-branch mapping network from fMRI to CLIP feature space is trained for each subject.

[0034] In this embodiment, the fMRI input voxels are selected from the visually relevant ROIs provided by NSD and are divided based on the functional hierarchy of the visual cortex: low-level visual regions (including V1, V2, and V3) and high-level visual regions together correspond to detail fMRI signals. Semantic fMRI signals corresponding to high-level visual regions (including LOC, FFA, PPA, and VWFA) The CLIP model uses the ViT-L / 14 version. The average value of features from layers 10 to 20 of the CLIP visual encoder is extracted as the true intermediate layer detail features, and features from layer 24 are extracted as the true final layer semantic features. These two are fused for image branch supervision, while the final layer features extracted by the CLIP text encoder are used as the text branch supervision target. During the training phase, the parameters of CLIP and the Versatile Diffusion diffusion model are frozen, and only the parameters of the dual-branch mapping network described in this invention are trained.

[0035] This embodiment uses low-level visual consistency metrics such as pixel correlation (PixCorr) and structural similarity (SSIM), as well as high-level semantic consistency metrics based on deep network features (including Inception feature similarity, CLIP similarity, EfficientNet-B distance, and SwAV distance) to evaluate the reconstructed images. Experimental results are shown in Table 1. This invention achieves image reconstruction with both semantic consistency and detail fidelity without requiring additional VAE detail channels. Furthermore, because it only trains the mapping network from fMRI to the CLIP feature space, the number of trainable parameters is smaller than that of schemes introducing additional VAE mapping branches, verifying the comprehensive advantages of this invention in terms of reconstruction quality and parameter efficiency.

[0036] Table 1 Experimental results of embodiments of the present invention

Claims

1. A brain image decoding method based on CLIP feature fusion and neural layer mapping, characterized in that, Includes the following steps: (1) Acquire functional magnetic resonance imaging (fMRI) data of subjects observing images and perform preprocessing; (2) The preprocessed functional magnetic resonance imaging (fMRI) data were divided into low-level and high-level visual regions: the detail fMRI signal corresponding to the low-level visual region was... The semantic fMRI signal corresponding to the higher visual areas is ; (3) Constructing a dual-branch mapping network and performing hierarchical mapping: Constructing a dual-branch mapping network containing image and text branches to achieve hierarchical mapping from fMRI signals to CLIP features, wherein: Text branch receives semantic fMRI signals Output the predicted CLIP text features The predicted CLIP text features are used to provide semantic constraints on the generated image; Image branch receives fMRI signals With semantic fMRI signals Output the predicted CLIP visual intermediate layer detail features respectively. Compared with the predicted CLIP visual final layer semantic features ; (4) Multi-layer CLIP visual feature fusion: fusion of predicted CLIP visual intermediate layer detail features Compared with the predicted CLIP visual final layer semantic features The average value is used for fusion to obtain the fused image features. The fused image features It contains fine-grained visual information and high-level semantic information; (5) Complete training and generate images using a diffusion model: Construct a supervisory signal pair to train the bi-branch mapping network in step (3), and input the fMRI data to be decoded into the trained bi-branch mapping network to obtain the image. and The data is then input into the Versatile Diffusion model to generate the corresponding reconstructed image.

2. The brain image decoding method based on CLIP feature fusion and neural layer mapping according to claim 1, characterized in that, In step (1), acquiring functional magnetic resonance imaging (fMRI) data of the subject when observing the image includes acquiring stimulus image data corresponding to the subject's visual stimulus image and text description data corresponding to the stimulus image.

3. The brain image decoding method based on CLIP feature fusion and neural layer mapping according to claim 1, characterized in that, The low-level visual regions divided in step (2) are at least one of V1, V2, and V3, and the high-level visual regions are at least one of LOC, FFA, PPA, and VWFA.

4. The brain image decoding method based on CLIP feature fusion and neural layer mapping according to claim 2, characterized in that, In step (4), the CLIP ViT-L / 14 visual encoder is used to extract features from the visual stimulus image of the subject. The CLIP text encoder is a text encoder that is compatible with CLIP ViT-L / 14.

5. The brain image decoding method based on CLIP feature fusion and neural layer mapping according to claim 4, characterized in that, In step (4), when extracting features from the visual stimulus image of the subject, 10 to 20 layers of features from the stimulus image are extracted and averaged to obtain detailed features, which are used to characterize local texture, shape, and structural details, denoted as... ; The last layer of features from the stimulus image is extracted and used to represent high-level semantic and category information, denoted as . The intermediate layer detail features of the stimulus image are fused with the final layer semantic features by calculating the average value to form a fused CLIP visual feature, denoted as . .

6. The brain image decoding method based on CLIP feature fusion and neural layer mapping according to claim 1, characterized in that, In step (3), a dual-branch mapping network is constructed, including a text branch and an image branch, to map the obtained hierarchical fMRI signals to the CLIP feature space. The text branch is used to map the semantic fMRI signals. Mapped to the final semantic feature space of the CLIP text encoder, the output is the predicted CLIP text features. Image branches are used to provide semantic conditions for diffusion generation; image branches are used to extract detailed fMRI signals. With semantic fMRI signals These are mapped to the intermediate layer detail feature space and the final layer semantic feature space of the CLIP visual encoder, respectively, to output the predicted features. and .

7. The brain image decoding method based on CLIP feature fusion and neural layer mapping according to claim 5, characterized in that, In step (5), the dual-branch mapping network is trained using the real CLIP visual intermediate layer detail features and real CLIP visual final layer semantic features extracted by the CLIP visual encoder from the stimulus image, as well as the real CLIP text features extracted by the CLIP text encoder from the text branch description.

8. The brain image decoding method based on CLIP feature fusion and neural layer mapping according to claim 7, characterized in that, In step (5), during training, the CLIP visual encoder and CLIP text encoder are frozen, and only the parameters of the dual-branch mapping network are updated.

9. The brain image decoding method based on CLIP feature fusion and neural layer mapping according to claim 1, characterized in that, In step (5), the Versatile Diffusion receives the predicted CLIP text features simultaneously when generating the image. With the fused image features As a conditional control signal.

Images