Iterative visual perception image reconstruction method based on diffusion model

Abstract

The invention particularly relates to an iterative visual perception image reconstruction method based on a diffusion model, and the method is designed based on an image generation model Stable diffusion, and is divided into two stages: in the first stage, a decoder is trained to decode an fMRI signal into three spaces, namely a text feature space, an image VAE feature space and an image CLIP feature space; and in the second stage, the decoded features are input into a Stable diffusion model to carry out image generation and alignment. Experiments carried out on an NSD data set show that a reconstruction result can be kept consistent with a stimulation image in semantics and structure, and the effectiveness of a model structure is explained.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an iterative visual perception image reconstruction method based on a diffusion model. Background Technology

[0002] Existing research indicates that the brain's visual center (located in the occipital lobe of the cerebral cortex) processes information acquired from retinal receptors and then reconstructs the appearance of the external environment in the brain. In this process, the eye merely acts as a window for visual formation; the brain's visual center is truly responsible for visual formation, perception, and regulation. The task of reconstructing stimulus images from brain signals can be explained as a decoding process of brain signals.

[0003] Numerous studies have attempted to decode text, graphics, and images from brain signals using deep learning methods such as convolutional neural networks and long short-term memory networks. However, existing methods for reconstructing visually perceived images using diffusion models rely heavily on semantic annotations paired with images, which often lack image detail. Furthermore, current methods for preprocessing functional magnetic resonance imaging (fMRI) signals lack consideration for effectively decoding signals, which also hinders accurate reconstruction results.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To address the issues of missing semantic annotation details, redundant fMRI signal preprocessing, and insufficient accuracy in reconstruction results in existing technologies, this invention provides an iterative visual perception image reconstruction method based on a diffusion model. This method is designed based on the stable diffusion image generation model and improves the quality of the reconstructed image by optimizing the feature extraction method and the iterative algorithm of the reconstruction process. Experiments on the NSD dataset show that, compared with the real stimulus image, the reconstruction result retains more image details while maintaining accuracy.

[0006] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0007] According to a first aspect of the present invention, an iterative visual perception image reconstruction method based on a diffusion model is provided, the method comprising: Step 1: fMRI signal preprocessing based on the Masked Autoencoder (MAE) concept Step 1.1: Construct a self-supervised training task based on the MAE concept, and pre-train the mask module with the goal of recovering fMRI signals; Step 1.2: Use the pre-trained mask module to filter the fMRI signal and remove redundant signals; Step 2: Generation of fine-grained image descriptions Step 2.1: Design a cue template on the multimodal large model VisualGLM-6B, the template focusing on the object of interest, details and background information of the image; Step 2.2: Select images from the training set and generate 5 detailed descriptions for each image using VisualGLM-6B; Step 3: FastL2LiR-based fMRI signal decoding Step 3.1: Encode the image description generated in Step 2 using CLIP's text encoder to obtain image semantic features, train a FastL2LiR linear regression decoder, and map the fMRI signal to the image semantic space. ; Step 3.2: Encode the natural image using VQ-VAE and train the FastL2LiR decoder to map the fMRI signal to the latent space features of the image. ; Step 3.3: Encode the natural image using the CLIP image encoder and train the FastL2LiR decoder to map the fMRI signal to the multi-layer visual features of CLIP. ; Step 4: Image Reconstruction and Alignment Based on Stable Diffusion Step 4.1: Apply the image semantic features obtained in Step 3 Image latent space features Inputting the Stablediffusion model initially generates a reconstructed image containing semantic and structural information. ; Step 4.2: Utilize the CLIP features of each image layer obtained in Step 3 via CLIP visual branch For reconstructed images Encode the features, extract the lower linear layer features, and calculate the encoded features and... The L2 distance is used to backpropagate and update features. The image reconstruction results are iteratively optimized.

[0008] In some exemplary embodiments, step 1.1 specifically includes the following steps: 1D convolution was performed on the fMRI signals of the left and right brains respectively, and they were divided into multiple patches. After segmentation, the fMRI output feature size of the left brain was 380×1024, and the fMRI output feature size of the right brain was 410×1024. Channel compression of fMRI signals was performed using the Conv1d model to reduce the feature dimension to 1 dimension to obtain the scalar importance score of each patch. The softmax activation function was used to generate the weight of each block. After sorting by weight, the top 75% of the blocks were retained and the position number ids restore was recorded. The preserved fMRI sequence is concatenated with the learnable [cls] token, input into a 24-layer Transformer Encoder to extract deep features, and then processed by LayerNorm to obtain the encoded features; The encoded features are projected from 1024 dimensions to 512 dimensions through a linear layer, and a [mask]token is concatenated at the unselected sequence positions to obtain the encoder output features. As input to the decoder; After adding a 512-dimensional sin-cos decoder positional encoding, it is input into an 8-layer Transformer Decoder for reconstruction. The feature size is then mapped back to the original size through a linear layer, and self-supervised training is performed using the MSE loss on the masked block as the reconstruction loss.

[0009] In some exemplary embodiments, the image semantic feature extraction method in step 3.1 specifically includes: Each image corresponds to 5 image descriptions. The text encoder is called on each description to obtain CLIP text features. The 5 features are then averaged and pooled to obtain the 15×768-dimensional text features corresponding to the image. The preprocessed left and right brain fMRI signals are spliced ​​together, and the text features are flattened and z-score standardized, while the mean and variance are preserved. The FastL2LiR model is used as a decoder to learn the fit between the fMRI signal and the text features. On the test set, the fMRI signal is decoded using the obtained decoder and then reshaped back to 15×768 to obtain the semantic information with CLIP text feature space.

[0010] In some exemplary embodiments, the image CLIP feature extraction in step 3.3 specifically involves: ViT / B-32 was chosen as the backbone for pre-training CLIP. First, features from the six linear layers of the CLIP visual branch were extracted from the CLIP model. Then, the prediction accuracy of fMRI for each feature dimension was calculated using 5-fold cross-validation on a partial training set. The most important information for the decoding task was retained, and highly relevant feature indices were selected, retaining the top 25% of feature indices. Finally, the selected CLIP features and normalized fMRI data were used to train the decoding model FastL2LiR.

[0011] In some exemplary embodiments, the image reconstruction based on the Stable diffusion model in step 4.1 specifically involves: The two inputs of the Stable diffusion model and Set as a learnable variable for subsequent iterations. At each iteration, DDIMSampler performs time-step denoising to reconstruct a pixel-level image with a size of 1×3×512×512. Next, the CLIP encoder is used to extract features from the reconstructed image at six linear layers, and the MSE loss is calculated by comparing them with the target features pre-decoded by fMRI. After calculating the loss, the features are updated in reverse. The reconstruction iteration is performed, with the number of iterations set to 150. One image is generated every 5 iterations, and all generated reconstructed images are saved.

[0012] According to a second aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the iterative visual perception image reconstruction method based on a diffusion model as described in the first aspect.

[0013] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the iterative visual perception image reconstruction method based on the diffusion model described in the first aspect above.

[0014] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the diffusion-based iterative visual perception image reconstruction method described in the first aspect above by executing the executable instructions.

[0015] The present invention provides a visual perception image reconstruction method based on fMRI. This method is designed based on the Stable diffusion image generation model and consists of two stages. The first stage trains a decoder to decode the fMRI signal into three spaces: a text feature space, an image VAE feature space, and an image CLIP feature space. The second stage inputs the decoded features into the Stable diffusion model for image generation and alignment. Experiments on the NSD dataset show that the reconstruction result is semantically and structurally consistent with the stimulus image, demonstrating the effectiveness of the model structure. Compared with existing technologies, it has the following advantages: 1. Based on the MAE concept, fMRI signals are processed to remove redundant signals and obtain information that is more conducive to decoding semantics.

[0016] 2. Use VisualGLM-6B to generate fine-grained image descriptions for natural images, train fMRI signals to perform depth alignment with image semantics, and thus obtain more accurate and detailed reconstructed images.

[0017] 3. The reconstruction process uses a constraint strategy designed based on the structural features of the image for iterative optimization, further optimizing details such as the position and size of objects in the reconstruction results.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 This is a schematic diagram of the visual perception task corresponding to the method described in the example of the present invention.

[0021] Figure 2 This is a model diagram of the method described in the example of the present invention.

[0022] Figure 3 This is a diagram of the fMRI feature self-supervised screening module of the method described in the example of the present invention.

[0023] Figure 4 This is a schematic diagram of the Stable diffusion principle in the method described in the example of the present invention.

[0024] Figure 5This is a partial reconstruction result diagram of the method described in the example of the present invention.

[0025] Figure 6 This is a comparison chart of the reconstruction results of the method described in this invention and other methods. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0027] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] Research using human brain fMRI data to precisely reconstruct visual stimuli from brain activity has profound implications for advancing the development of brain-computer interfaces and understanding the working mechanisms of the human brain. Figure 1 This diagram illustrates the visual perception image reconstruction task corresponding to this invention. Intuitively, it involves reconstructing the visual perception image based on the human brain's fMRI, where the visual perception image is the image observed by the human and is the target of this reconstruction task. For clarity, in this invention, "visual perception image" can be described as "natural image" or "stimulus image."

[0029] Existing methods rely too heavily on semantic annotations from paired natural images during training, but these annotations lack image detail. Furthermore, existing methods do not adequately consider the effective decoding signal during fMRI signal preprocessing, which also hinders accurate reconstruction results.

[0030] To address the shortcomings and deficiencies of existing technologies, this example implementation provides an iterative visual perception image reconstruction method based on a diffusion model. This method reconstructs visual perception images from brain signals, more specifically, it reconstructs images observed using functional magnetic resonance imaging (fMRI) of the human brain. This method is designed based on the image generation model Stablediffusion, which consists of three sub-modules: an fMRI feature preprocessing module based on MAE, a decoder training module based on linear regression, and a visual perception image reconstruction module based on Stable diffusion. A schematic diagram of the overall model is shown below. Figure 2 As shown, the details are as follows: 1. MAE-based fMRI feature preprocessing module When decoding fMRI signals, since the fMRI signals contain effective information related to the semantics of the visual stimulus image, this invention proposes the following data preprocessing scheme to extract the fMRI signals with effective decoding semantics, obtain subsequent decoding input signals with higher accuracy, and thus ensure the accuracy of the reconstructed image semantics. The feature preprocessing module is as follows: Figure 3 As shown, during data acquisition, the fMRI signal corresponding to each stimulus image comes from multiple brain regions. In reality, because fMRI signals originate from different brain regions, and different brain regions respond differently to visual information, these signals contain redundant information for the task of decoding fMRI signals into image semantics. Therefore, this invention designs a self-supervised masking method to perform preliminary filtering of large-scale fMRI signals. A filtering ratio of 25% allows for signal selection while retaining effective information.

[0031] 1.1 Feature Filtering Mask Module Based on One-Dimensional Convolution With a fixed patch size of 50, the resting-state fMRI time series of the left and right brains were divided into blocks, and the entire signal was segmented into several blocks of length. A patch sequence of 50 Then, each patch is mapped to a 1024-dimensional embedding space using a 1×1024 Conv1d convolutional kernel. Ultimately, the left brain receives a 380×1024 tensor, and the right brain receives a 410×1024 feature map. .

[0032] To implement a learnable masking strategy, the 1024-dimensional embedded features of each patch are first compressed using a 1×1 Conv1d algorithm to reduce the feature dimension to 1 dimension, resulting in a scalar importance score for each patch. Then, softmax is used to normalize these scores into probability weights, and all patches are sorted from highest to lowest probability. According to a pre-set mask ratio of 0.75, only the first 75% of the sorted patches are retained, while the remaining patches are masked. Simultaneously, the position indices of the retained patches in the original sequence are saved as IDs restore, facilitating the restoration of masked positions and loss calculation in the subsequent decoding stage. The calculation formula for this process is:

[0033]

[0034] 1.2 Transformer-based encoding and decoding The preserved fMRI patch sequence is concatenated with learnable [cls] tokens and fed into a 24-layer Transformer Encoder to extract depth features. LayerNorm then yields a 1024-dimensional encoded representation. Next, a linear layer projects the encoded features to 512 dimensions, and then [mask] tokens, equal in number to the number of missing blocks, are padded according to the order of ids restore, forming the decoder input. Subsequently, 512-dimensional sin-cos decoder positional encodings are superimposed, and the signal is reconstructed through an 8-layer Transformer Decoder. Finally, the fMRI values ​​are mapped back to their original size via MPL, and self-supervised training is performed only on the masked blocks, calculating MSE loss. Taking the left brain signal as an example, the network structure parameters for each layer are set as shown in Table 1 below: Table 1 Transformer-based encoding and decoding

[0035] 2. Decoder Training Based on FastL2LiR The decoder was obtained by selecting the Fast L2-regularized Linear Regression method. For example... Figure 2 As shown, the obtained decoder is used to decode the fMRI signal, mapping the fMRI signal to three spaces: image semantic space, image latent space, and image CLIP space.

[0036] In the first decoder, the target of fMRI signal fitting is the image description encoded using the CLIP text encoder. The semantic features of the image decoded on the test set are represented as C. The decoder training process is as follows: First, images in the training set are selected, and VisualGLM-6B is used to generate detailed semantic descriptions for each image. Each image corresponds to 5 image descriptions, describing the subject, background, and details of the image. The pre-trained Stable Diffusion uses the CLIP text encoder with a feature space dimension of 77×768, where 77 refers to the maximum number of tokens and 768 refers to the encoding dimension of each token. Considering the length of the text description corresponding to the image, only the first 15×768 dimensions of the flattened features are fitted in practice. In the subsequent reconstruction process, C is used as text input to ensure that the semantics of the subsequent reconstruction results are consistent with the natural image.

[0037] Since stable diffusion is a latent space diffusion model, the target of fMRI signal fitting is not obtained from the high-dimensional image space, but rather by compressing the image into the latent space. Stable diffusion uses VQ-VAE to encode the image, extracting the latent space features (1×4×64×64 dimensions) of the training set images, and then mapping fMRI to this latent space, thereby achieving image latent space compression. In the second decoder, the target of fMRI signal fitting is the natural image features encoded using VQ-VAE, and the latent space features decoded on the test set are represented as Z.

[0038] The target of fMRI signal fitting in the third decoder is the natural image features encoded by the CLIP image encoder. To align the structural information of the reconstructed image with the original image without losing semantic information, this invention selects the CLIP low-level visual features to control the reconstructed image. First, features from Linear-2, Linear-4, Linear-6, Linear-8, Linear-10, and Linear-12 layers in the CLIP visual branch are extracted. For each layer, 859 sets of data are divided from the training set as a validation set. The 5-fold cross-validation method is used to calculate the prediction accuracy (Pearson correlation coefficient) of fMRI for each layer's features. Then, only the features with the highest prediction accuracy are selected. Next, all training set data is used to refit these 25% of features. When reconstructing images in the test set, these 25% of features serve as guidance for aligning the original and reconstructed images. The CLIP spatial features of the decoded image on the test set are represented as follows: .

[0039] The three features mentioned above are used in the second stage of image reconstruction and alignment, among which, and These two features were used to rebuild the module. Features are used to align modules.

[0040] 3. Image Reconstruction and Alignment Based on Stable Diffusion The decoder obtains features Z and C after decoding the fMRI. These two features are used as input to the Stable diffusion model for image reconstruction.

[0041] The diffusion model includes a forward diffusion process and a backward denoising process. Definition The image is the original image, and T is the total diffusion time step. For the noisy image at the final time step, let q(·|·) represent the posterior distribution of the latent variables during the forward diffusion process, and let p(·|·) represent the prior distribution of the real data during the reverse denoising process. During the forward diffusion process, Gaussian noise is gradually added to the original image to obtain the posterior distribution of the noisy image, and the following three conditions are satisfied during the noisy process: (1) the latent vector at each step Dimensions and inputs (1) The dimensions are kept consistent; (2) The latent vector at each time step t is encoded as a Gaussian distribution that depends only on the latent vector at the previous time step. (3) The parameters of the Gaussian distribution of the latent vector at each time step t change with the time step, and satisfy the constraint that the Gaussian distribution at the final time step is the standard Gaussian distribution, i.e. The formula for this process is as follows:

[0042] In the reverse denoising process, the original image can be obtained by estimating the prior distribution of the image at each time step and continuously sampling from that distribution. Optimizing the noise matching term in the diffusion model ultimately involves training a neural network to process the noisy image at any time step. As input, to predict the original raw image At this point, the optimization objective is transformed into minimizing the prediction error. The optimization of summing the noise matching terms over all time steps can be approximated as minimizing the expected prediction error at each time step, and this optimization objective can be approximated by random sampling:

[0043] Figure 4 The training process of the diffusion model is summarized. During the forward noise addition process, for the image input... First, by setting the hyperparameters By continuously adding noise to the original image, the latent variable distribution at each time step t is obtained. Then, in the reverse denoising process, the time code t at each time step and the latent space sampled from that time step are used. As input to U-Net, the original image U-Net is trained as the prediction target.

[0044] Because the images reconstructed by the Stable diffusion model lack constraints, they are prone to structural inconsistencies with natural images. Therefore, this invention designs an alignment strategy to optimize the reconstruction results. Specifically, the image obtained by the decoder in stage one... Features can be used for alignment, first reconstructing the image. Encoded using the CLIP model's image encoder. , and then with Calculate the loss:

[0045] After calculating the loss, the feature Z is updated in reverse, and reconstruction iterations are performed to optimize the structural information of the reconstructed image, such as the pose and size of objects in the image. In this invention, the number of iterations is set to 150, with one image generated every 5 iterations. The generated reconstructed images are saved in a designated folder.

[0046] The steps in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.

[0047] Example 1 An iterative visual perception image reconstruction method based on a diffusion model combines the Stable diffusion model from the field of image generation with the decoding of human brain fMRI signals to reconstruct visual perception images. By introducing fine-grained semantic information from the image, the method compensates for the lack of detail in image annotation, making the semantic information decoded from fMRI more accurate and focusing on more fine-grained attribute information in the image, thus better matching the content of the stimulus image. The decoded semantic information is input into the Stable diffusion model and multiple alignment iterations are performed using structural features to reconstruct more natural and accurate image content. The method includes the following steps: Step 1: fMRI signal preprocessing based on MAE concept; Step 1.1, Pre-training of the mask method. Each stimulus image corresponds to a set of fMRI signals. Based on the idea of ​​MAE (Masked Autoencoders), the mask module is pre-trained. The self-supervised task is to recover the fMRI signal. The model output is the predicted fMRI value of the masked block. The selection ratio is set to 25%. The loss function is MSE (Mean Square Error). The loss is calculated only on the masked block. The final prediction result of the mask module is retained to preprocess the fMRI signal. Step 1.2, fMRI signal preprocessing. A pre-trained mask method is used to perform preliminary screening and filtering of large-scale fMRI signals.

[0048] Step 2, generation of fine-grained image descriptions; Step 2.1, designing prompts. On the multimodal large model VisualGLM-6B, design prompt templates that focus on the main objects, details, and background of the image to generate image descriptions; Step 2.2, Image Description Generation. Natural images (i.e., visually perceived images) corresponding to the fMRI signals in the training set are selected. It is important to emphasize that these natural images are only visible as sources of ground truth labels during the decoder training phase in Step 3; they are not visible during the decoder inference phase. Using the multimodal large model VisualGLM-6B as the description generator, each natural image in the training set is input into VisualGLM-6B. Based on the prompt template designed in Step 2.1, five detailed text descriptions containing objects, details, and background are generated for each image. These descriptions will be used later in Step 3.1 to construct supervised labels for semantic features.

[0049] Step 3, fMRI signal decoding based on FastL2LiR; Step 3.1, Image Semantic Feature Extraction. During the training phase, the image descriptions corresponding to the natural images in the training set from Step 2.2 are used as text input, and the CLIP text encoder is invoked to encode them. The features of the five descriptions corresponding to each image are averaged and pooled to obtain the ground truth image semantic features for that image. Subsequently, the preprocessed fMRI signal from Step 1.2 is used as input, and its corresponding ground truth image semantic features are used as the regression target to train the Fast L2-regularized Linear Regression (FastL2LiR) linear regression decoder. During the inference phase, this decoder maps the fMRI signal to the image semantic space. ; Step 3.2, Image VQ-VAE Latent Space Feature Extraction. Stable diffusion uses VQ-VAE to encode images, thereby achieving image latent space compression. Therefore, during the training phase, the natural images (i.e., visually perceived images) corresponding to the fMRI signals in the training set are first encoded using VQ-VAE. The preprocessed fMRI signals are used as input, and the encoded latent space features are used as the regression target to train the FastL2LiR decoder. During the inference phase, this decoder maps the fMRI signals to the image latent space features. ; Step 3.3, Image CLIP Feature Extraction. During the training phase, the CLIP image encoder encodes the natural images (i.e., visually perceived images) corresponding to the fMRI signals in the training set, extracting features from multiple linear layers in the CLIP visual branch as ground truth values ​​for the image CLIP features. Using the preprocessed fMRI signal as input and the filtered CLIP visual feature ground truth values ​​as the regression target, the FastL2LiR decoder is trained. During the inference phase, this decoder maps the fMRI signal to the multi-layer visual features of the CLIP. This is used as the reconstruction input for continuously optimizing Stable diffusion through backpropagation gradient in step 4. Step 4: Image reconstruction and alignment based on stable diffusion; Step 4.1, Image Reconstruction. Utilize the semantic features of the image obtained from decoding in Step 3. Image latent space features When input into a Stable diffusion model for image reconstruction, a preliminary reconstructed image containing semantic and structural information can be generated. ; Step 4.2, iterative alignment optimization. The CLIP features of each image layer are obtained in Step 3. It can be used to align the reconstructed visual perception image Let the CLIP visual branch be... It encodes structural information such as position and size at lower levels. First, the reconstructed image... use Encode, extract lower linear layer features (Linear-k, k=2,4,6……12), then use Calculate the L2 distance between the two:

[0050] in i This represents the feature layer number of Linear-k, and the value decoded in the first step is updated based on the calculated L2 distance through backpropagation. Iterative optimization of image reconstruction is performed to achieve alignment of the reconstruction results.

[0051] Furthermore, the pre-trained mask method described in step 1.1 specifically includes: fMRI signals originate from different brain regions, and these regions respond differently to visual information. A self-supervised masking method is used to filter and extract key decoding information from a large-scale fMRI signal. 1D convolution was performed on the fMRI signals of the left and right brains respectively, and the fMRI was divided into multiple patches. The output feature size of the left brain fMRI after segmentation was 380×1024, and the output feature size of the right brain fMRI was 410×1024. The Conv1d model was used to compress the channels of fMRI, reducing the feature dimension to 1 dimension, and obtaining the scalar importance score of each patch. Then, the softmax activation function was used to generate weights for each block, and the blocks were sorted according to the weights. The top 75% of the blocks were retained and the position numbers (ids restore) of the retained blocks were recorded. The preserved fMRI sequence is concatenated with the learnable [cls] token and fed into a 24-layer TransformerEncoder to extract deep features. Finally, the encoded features are obtained through LayerNorm. The encoded features are first projected from 1024 dimensions to 512 dimensions through a linear layer. To restore the original fMRI signal size, [mask] tokens are concatenated at the unselected sequence locations to obtain the encoder output features. As input to the decoder; After adding 512-dimensional sin-cos decoder position encoding, the data is fed into an 8-layer Transformer Decoder for reconstruction; finally, it is mapped back to the original feature size through linear layers, outputting a feature map consistent with the original fMRI size. The MSE loss on the masked block is used as the reconstruction loss for self-supervised training.

[0052] Furthermore, the image semantic feature extraction method described in step 3.1 specifically includes: The text encoder weights of the Stable diffusion model are loaded to encode image descriptions. The Stable diffusion model uses a pre-trained CLIP text encoder. The pre-training process aligns image features and text features in the same vector space, thereby bridging the semantic gap between modalities. Each image corresponds to 5 image descriptions. The text encoder is called on each description to obtain CLIP text features. Average pooling is then performed on the 5 features to obtain the 15×768-dimensional text features corresponding to the image.

[0053] Preprocessed left and right brain fMRI signals were spliced ​​together. Text features were flattened and z-score standardized, with the mean and variance preserved. A FastL2LiR model was used as the decoder to learn the fit between the fMRI signals and text features. On the test set, the fMRI signals were decoded using the obtained decoder and then reshaped back to 15×768 to obtain the semantic information corresponding to the CLIP text feature space.

[0054] Furthermore, the image CLIP feature extraction described in step 3.3 specifically involves: ViT / B-32 was chosen as the backbone for pre-training CLIP. First, features (38400 dimensions) from the six linear layers of the CLIP visual branch were extracted. Since features with low decoding accuracy on these layers can lead to erroneous image reconstructions, a 5-fold cross-validation method was used on a subset of the training set to calculate the fMRI prediction accuracy (Pearson correlation coefficient) for each feature dimension before fitting the features to each layer. This process retained the most important information for the decoding task, selecting highly correlated feature indices and retaining the top 25%. Finally, the selected CLIP features and normalized fMRI data were used to train the FastL2LiR decoding model.

[0055] Furthermore, the image reconstruction based on the Stable diffusion model described in step 4.1 specifically includes: The two inputs z and c of the Stable diffusion model are set as learnable variables for subsequent iterations. z represents the latent space features of the image decoded from the fMRI signal, and c represents the semantic features of the text description corresponding to the image decoded from the fMRI signal. At each iteration, DDIMSampler performs time-step denoising to reconstruct a pixel-level image with a size of 1×3×512×512.

[0056] Next, the CLIP encoder is used to extract features at the six linear layers of the reconstructed image, and the MSE loss is calculated with the target features pre-decoded by fMRI. After calculating the loss, the features z are updated in reverse, and reconstruction iteration is performed. The number of iterations is set to 150, and one image is generated every 5 iterations. All generated reconstructed images are saved.

[0057] The invention is further illustrated below through simulation experiments: Experimental Methods for Visual Perception Image Reconstruction Based on fMRI The Natural Scenes Dataset (NSD), currently the largest neuroimaging dataset, was used, containing densely sampled functional magnetic resonance imaging (fMRI) data from eight participants. During 30-40 MRI scans, each participant viewed 9,000-10,000 natural scenes of different colors (22,000-30,000 repetitions) simultaneously with fMRI acquisition. Whole-brain gradient echo EPI was used for 7T scans at 1.8 mm isovoxel resolution and a 1.6-second TR. The images viewed by the participants were from the MSCOCO dataset, and text descriptions were extracted from the COCOID of the stimulus images. Detailed information about the NSD data used in the experiment is shown in Table 2.

[0058] Table 2. Detailed information on NSD data.

[0059] This method uses two evaluation metrics to assess the reconstruction results: image similarity index PSNR (Peak Signal-to-Noise Ratio) and Pearson correlation coefficient. We selected the model proposed by Takagi in 2023 for comparison. The metric results are shown in Table 3, and the visualization of the reconstruction results is as follows. Figure 6 As shown in the figure, PSNR measures the degree of image distortion and the similarity between two images; a higher PSNR indicates a smaller difference between the two images. The Pearson correlation coefficient measures the linear relationship between two random variables X and Y. A value of 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and a value close to 0 indicates no correlation. The reconstruction results of this method are semantically and structurally closer to the original image, demonstrating the effectiveness and advancement of the proposed method.

[0060] Figure 5 More reconstruction results are shown, where the red boxes represent the original natural images, i.e., the stimulus images, and the images below the red boxes are the reconstructed images obtained from the fMRI signals using the method proposed in this invention. It can be seen that the reconstructed images maintain semantic and structural consistency with the natural images, thus verifying the effectiveness of the reconstruction method proposed in this invention.

[0061] Table 3 Results on image consistency

[0062] In summary, the two-stage image reconstruction model of this invention achieved more accurate experimental results on the NSD dataset. During data preprocessing, a preprocessing scheme based on the characteristics of fMRI signals was designed to remove redundant signals, resulting in fMRI signals more conducive to decoding semantic information for subsequent image reconstruction. In the two-stage image reconstruction algorithm, reconstruction iterations based on alignment strategies and backpropagation enable the semantic and structural information of the reconstructed image to be aligned with the natural image. Furthermore, a large model is used to construct fine-grained annotation information for the natural image, supplementing more semantic details and thus optimizing the quality of the reconstructed image.

[0063] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0064] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0065] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.

Claims

1. An iterative visual perception image reconstruction method based on a diffusion model, characterized in that, The method includes: Step 1: fMRI signal preprocessing based on the Masked Autoencoder (MAE) concept Step 1.1: Construct a self-supervised training task based on the MAE concept, and pre-train the mask module with the goal of recovering fMRI signals; Step 1.2: Use the pre-trained mask module to filter the fMRI signal and remove redundant signals; Step 2: Generation of fine-grained image descriptions Step 2.1: Design a cue template on the multimodal large model VisualGLM-6B, the template focusing on the object of interest, details and background information of the image; Step 2.2: Select images from the training set and generate 5 detailed descriptions for each image using VisualGLM-6B; Step 3: FastL2LiR-based fMRI signal decoding Step 3.1: Encode the image description generated in Step 2 using CLIP's text encoder to obtain image semantic features, train a FastL2LiR linear regression decoder, and map the fMRI signal to the image semantic space. ; Step 3.2: Encode the natural image using VQ-VAE and train the FastL2LiR decoder to map the fMRI signal to the latent space features of the image. ; Step 3.3: Encode the natural image using the CLIP image encoder and train the FastL2LiR decoder to map the fMRI signal to the multi-layer visual features of CLIP. ; Step 4: Image Reconstruction and Alignment Based on Stable Diffusion Step 4.1: Apply the image semantic features obtained in Step 3 Image latent space features Inputting into the Stable diffusion model initially generates a reconstructed image containing semantic and structural information. ; Step 4.2: Utilize the CLIP features of each image layer obtained in Step 3 via CLIP visual branch For reconstructed images Encode the features, extract the lower linear layer features, and calculate the encoded features and... The L2 distance is used to backpropagate and update features. The image reconstruction results are iteratively optimized.

2. The iterative visual perception image reconstruction method based on a diffusion model according to claim 1, characterized in that, Step 1.1 specifically includes the following steps: 1D convolution was performed on the fMRI signals of the left and right brains respectively, and they were divided into multiple patches. After segmentation, the fMRI output feature size of the left brain was 380×1024, and the fMRI output feature size of the right brain was 410×1024. Channel compression of fMRI signals was performed using the Conv1d model to reduce the feature dimension to 1 dimension to obtain the scalar importance score of each patch. The softmax activation function was used to generate the weight of each block. After sorting by weight, the top 75% of the blocks were retained and the position number ids restore was recorded. The preserved fMRI sequence is concatenated with the learnable [cls] token, input into a 24-layer Transformer Encoder to extract deep features, and then processed by LayerNorm to obtain the encoded features; The encoded features are projected from 1024 dimensions to 512 dimensions through a linear layer, and a [mask] token is concatenated at the unselected sequence positions to obtain the encoder output features. As input to the decoder; After adding a 512-dimensional sin-cos decoder positional encoding, it is input into an 8-layer Transformer Decoder for reconstruction. The feature size is then mapped back to the original size through a linear layer, and self-supervised training is performed using the MSE loss on the masked block as the reconstruction loss.

3. The iterative visual perception image reconstruction method based on a diffusion model according to claim 1, characterized in that, The image semantic feature extraction method in step 3.1 specifically includes: Each image corresponds to 5 image descriptions. The text encoder is called on each description to obtain CLIP text features. The 5 features are then averaged and pooled to obtain the 15×768-dimensional text features corresponding to the image. The preprocessed left and right brain fMRI signals are spliced ​​together, and the text features are flattened and z-score standardized, while the mean and variance are preserved. The FastL2LiR model is used as a decoder to learn the fit between the fMRI signal and the text features. On the test set, the fMRI signal is decoded using the obtained decoder and then reshaped back to 15×768 to obtain the semantic information with CLIP text feature space.

4. The iterative visual perception image reconstruction method based on a diffusion model according to claim 1, characterized in that, Step 3.3, image CLIP feature extraction, specifically involves: ViT / B-32 was chosen as the backbone for pre-training CLIP. First, features from the six linear layers of the CLIP visual branch were extracted from the CLIP model. Then, the prediction accuracy of fMRI for each feature dimension was calculated using 5-fold cross-validation on a partial training set. The most important information for the decoding task was retained, and highly relevant feature indices were selected, retaining the top 25% of feature indices. Finally, the selected CLIP features and normalized fMRI data were used to train the decoding model FastL2LiR.

5. The iterative visual perception image reconstruction method based on a diffusion model according to claim 1, characterized in that, Step 4.1, image reconstruction based on the Stable diffusion model, specifically involves: The two inputs of the Stable diffusion model and Set as a learnable variable for subsequent iterations. At each iteration, DDIMSampler performs time-step denoising to reconstruct a pixel-level image with a size of 1×3×512×512. Next, the CLIP encoder is used to extract features from the reconstructed image at six linear layers, and the MSE loss is calculated by comparing them with the target features pre-decoded by fMRI. After calculating the loss, the features are updated in reverse. The reconstruction iteration is performed, with the number of iterations set to 150. One image is generated every 5 iterations, and all generated reconstructed images are saved.

6. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the iterative visual perception image reconstruction method based on the diffusion model as described in any one of claims 1 to 5.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the iterative visual perception image reconstruction method based on the diffusion model as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the iterative visual perception image reconstruction method based on the diffusion model according to any one of claims 1 to 5 by executing the executable instructions.

Images