Fmri video neural decoding method based on visual perception and semantic consistency

Abstract

The application discloses an fMRI video neural decoding method based on visual perception and semantic consistency, and steps are as follows: acquiring fMRI signals and corresponding video data; constructing an fMRI input vector; extracting a color graph, a spatial depth graph, frame semantic embedding and text description semantic embedding from the video data; taking the fMRI input vector as a feature, taking the color graph and the spatial depth graph as labels, training a two-stage training double-tower mask autoencoder to train a visual perception channel to obtain a prediction of the color graph and the spatial depth graph; taking the fMRI input vector as a feature, taking the frame semantic embedding and the text description semantic embedding as labels, training a semantic channel to obtain a prediction of fMRI semantic embedding; and based on the prediction of the color graph, the spatial depth graph and the fMRI semantic embedding, performing multi-condition guidance on a video diffusion model to obtain decoding output. The application can realize fMRI video neural decoding with high visual perception consistency and semantic consistency.

Description

Technical Field

[0001] This invention relates to the field of visual neural decoding technology, and more specifically to an fMRI video neural decoding method based on visual perception and semantic consistency. Background Technology

[0002] Video neural decoding technology has become a research hotspot in neuroscience, brain-computer interfaces, and artificial intelligence in recent years. Its function is to reconstruct the video watched by the subject from functional magnetic resonance imaging (fMRI) data. Existing fMRI video neural decoding methods have the following limitations:

[0003] (1) The decoding is performed using simple models, but there is a lack of biological modeling of the human visual system;

[0004] (2) Learning and reconstructing the video at the semantic level alone cannot capture the complex changes in brain activity in the human visual system, and results in insufficient visual perception consistency of the reconstruction results.

[0005] (3) The lack of multimodal modeling at the visual perception level resulted in the failure to fully explore the visual perception information in fMRI data;

[0006] (4) The video reconstruction effect is poor and lacks generalization ability. Summary of the Invention

[0007] To overcome the shortcomings and deficiencies of existing technologies, this invention provides an fMRI video neural decoding method based on visual perception and semantic consistency. At the visual perception level, this invention utilizes a dual-stage trained dual-tower mask autoencoder to perform bimodal visual perception learning of the color perception pathway and the spatial depth perception pathway, achieving high-quality multimodal visual perception reconstruction of color perception and spatial depth perception. At the semantic level, the fMRI input vector is aligned with frame semantic embedding and text description semantic embedding in a trimodal manner, achieving high-quality reconstruction of high-level semantic information of the video. Combining the visual perception and semantic levels, the technology of controllable image generation is extended to video generation, and a video generation diffusion model is used to achieve fMRI video neural decoding with high visual perception consistency and semantic consistency.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] This invention provides an fMRI video neural decoding method based on visual perception and semantic consistency, comprising the following steps:

[0010] Acquire fMRI signals and their corresponding video data;

[0011] Data preprocessing is performed on the fMRI signals to construct the fMRI input vector;

[0012] Extract color maps, spatial depth maps, frame semantic embeddings, and text description semantic embeddings from video data;

[0013] Using fMRI input vectors as features and color maps and spatial depth maps as labels, a visual perception pathway is trained based on a dual-tower mask autoencoder trained in two stages to obtain predictions of color maps and spatial depth maps.

[0014] Using fMRI input vectors as features and frame semantic embeddings and text description semantic embeddings as labels, a semantic pathway is trained to obtain the prediction of fMRI semantic embeddings.

[0015] The prediction of color map, spatial depth map, and fMRI semantic embedding are used as guiding conditions for the video diffusion model. The video diffusion model is guided by multiple conditions to obtain the decoding output.

[0016] As a preferred technical solution, fMRI signals are preprocessed to construct fMRI input vectors, specifically including:

[0017] The original fMRI signal was denoised and registered to a standard space;

[0018] Data from voxel points located in the visual brain region are extracted to obtain the fMRI input vector.

[0019] As a preferred technical solution, color maps, spatial depth maps, frame semantic embeddings, and text description semantic embeddings are extracted from video data, specifically including:

[0020] For each frame of the video data, bilinear interpolation is used to downsample and adjust the resolution, and bilinear interpolation is used to upsample and obtain the color map.

[0021] A spatial depth map is extracted for each frame based on a depth estimation model;

[0022] Extract CLIP image embeddings from each frame of the video data, and then perform average calculations to obtain the frame semantic embeddings;

[0023] Text descriptions are extracted from each frame of the video data, the text descriptions are connected using conjunctions, and then rewritten based on a large language model to obtain the semantic embedding of the text descriptions.

[0024] As a preferred technical solution, a dual-tower mask autoencoder based on two-stage training is used to train the visual perception pathway, specifically including:

[0025] The first stage of training involves training a single-modal mask autoencoder within its respective modality using color maps, spatial depth maps, and fMRI input vectors.

[0026] The second phase of training involves cross-modal training using color maps, spatial depth maps, and fMRI input vectors.

[0027] As a preferred technical solution, when training a single-modal mask autoencoder, the loss function of the color image and the fMRI input vector is a weighted sum of the L1 loss and the MSE loss.

[0028] When training a single-modal masked autoencoder using a spatiotemporal depth map, the loss function is a weighted sum of the L1 loss and the MSE loss, with an additional total variational regularization term.

[0029] When performing cross-modal training on color maps and fMRI input vectors, the loss function is a weighted sum of InfoNCE loss, L1 loss, and MSE loss;

[0030] When performing cross-modal training on spatial depth maps and fMRI input vectors, the loss function is a weighted sum of InfoNCE loss, L1 loss, and MSE loss, with an additional total variational regularization term.

[0031] As a preferred technical solution, the semantic pathway is trained by aligning the fMRI input vector with the frame semantic embedding and the text description semantic embedding in a three-modal manner, mapping the fMRI input vector to the CLIP semantic latent space, and the loss function is a weighted sum of semantic embedding similarity loss, L1 loss and MSE loss.

[0032] As a preferred technical solution, the prediction of color map, spatial depth map, and fMRI semantic embedding are used as guiding conditions for the video diffusion model, providing multi-condition guidance for the video diffusion model, specifically including:

[0033] Color map prediction, spatial depth map prediction, and fMRI semantic embedding are used as guiding conditions for the video diffusion model through controllable generation techniques.

[0034] The image modal conditions corresponding to the color map and spatial depth map are used to obtain controllable generation adapter features using controllable generation techniques, and then input into the feature map of the diffusion model;

[0035] The embedding modality conditions corresponding to the fMRI semantic embedding are input into the video diffusion model using a cross-attention mechanism;

[0036] Adjust the video diffusion model and the controllable generation adapter, and obtain the decoded output based on the adjusted controllable generation adapter and the video diffusion model.

[0037] The present invention also provides an fMRI video neural decoding system based on visual perception and semantic consistency, for implementing the above-mentioned fMRI video neural decoding method based on visual perception and semantic consistency, including: a data acquisition module, an fMRI input vector construction module, a data extraction module, a visual perception pathway training module, a semantic pathway training module, and a multi-condition guidance module;

[0038] The data acquisition module is used to acquire fMRI signals and their corresponding video data;

[0039] The fMRI input vector construction module is used to preprocess fMRI signals and construct fMRI input vectors;

[0040] The data extraction module is used to extract color maps, spatial depth maps, frame semantic embeddings, and text description semantic embeddings from video data.

[0041] The visual perception pathway training module is used to train the visual perception pathway based on a dual-tower mask autoencoder with fMRI input vector as features and color map and spatial depth map as labels, and to obtain the prediction of color map and spatial depth map.

[0042] The semantic pathway training module is used to train the semantic pathway by using the fMRI input vector as a feature and the frame semantic embedding and text description semantic embedding as labels, so as to obtain the prediction of the fMRI semantic embedding.

[0043] The multi-condition guidance module is used to use the prediction of color map, spatial depth map, and fMRI semantic embedding as guidance conditions for the video diffusion model, and to guide the video diffusion model in multiple ways to obtain the decoding output.

[0044] The present invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described fMRI video neural decoding method based on visual perception and semantic consistency.

[0045] The present invention also provides a computer device, including a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the fMRI video neural decoding method based on visual perception and semantic consistency as described above.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] (1) This invention introduces a dual-modal visual perception learning framework in the fMRI video neural decoding task, and establishes color perception pathway and spatial depth perception pathway respectively, thereby achieving high-fidelity reconstruction of low-level visual perception features. Compared with existing decoding methods that rely only on a single image modality or global semantic information, this invention can better capture visual processing features at different levels in the visual cortex of the brain, and significantly improve the visual perception consistency and detail fidelity of video reconstruction.

[0048] (2) Based on a dual-stage training dual-tower mask autoencoder, this invention performs dual-modal visual perception learning of color perception pathway and spatial depth perception pathway. Through a dual-stage training strategy of single-modal pre-training and cross-modal alignment, it achieves efficient cross-modal mapping from fMRI signals to visual images (color map and depth map). It can maintain the high-dimensional brain activity features of fMRI representation space and fully transfer the spatial information of video modality, which significantly enhances cross-domain learning ability and generalization performance.

[0049] (3) This invention has the ability to reconstruct multi-level semantic alignment and high semantic consistency. It adopts a contrastive learning strategy at the semantic level and introduces the InfoNCE loss function. By aligning the fMRI input vector, frame semantic embedding and text description semantic embedding, it achieves deep integration of fMRI and multimodal semantic space. This three-modal alignment mechanism effectively improves the model's expressive ability at the semantic level, making the generated video more consistent with the original video in terms of semantic logic and scene content.

[0050] (4) This invention uses multi-condition guided video diffusion generation to achieve high-quality video reconstruction. In the video generation stage, the prediction of color map, spatial depth map, and fMRI semantic embedding are used as multi-condition inputs. By introducing a framework that combines a controllable generation adapter with a diffusion model, fine control of the video generation process is achieved. The multi-condition classifier-free guidance strategy effectively balances visual perception and semantic consistency. The generated video shows high-quality reconstruction results in color, spatial hierarchy, and semantic content.

[0051] (5) This invention forms a complete hierarchical neural decoding process from fMRI signals to color perception, spatial depth perception, semantic understanding and then to video generation. It constructs a hierarchical neural decoding system from low-level perception to high-level semantics, realizes the collaborative modeling of visual perception and semantic understanding, and makes full use of the brain's multi-level information processing characteristics. Compared with traditional single-layer mapping or end-to-end black box methods, this invention has significant advantages in biological interpretability and reconstruction accuracy. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the fMRI video neural decoding method based on visual perception and semantic consistency of the present invention.

[0053] Figure 2 This is a schematic diagram of the implementation architecture of the fMRI video neural decoding method based on visual perception and semantic consistency of the present invention;

[0054] Figure 3 This is a schematic diagram of the implementation architecture of the visual perception pathway of the present invention;

[0055] Figure 4 This is a schematic diagram of the semantic path implementation architecture of the present invention;

[0056] Figure 5 This is a schematic diagram showing the visualization comparison results of the fMRI video neural decoding method based on visual perception and semantic consistency of the present invention with other methods. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] Example 1

[0059] like Figure 1 , Figure 2 As shown, this embodiment provides an fMRI video neural decoding method based on visual perception and semantic consistency, including the following steps:

[0060] S1: Acquire functional magnetic resonance imaging (fMRI) signals and video data viewed by the subjects, and establish a pairing relationship between the fMRI signals and the video data viewed by the subjects;

[0061] S2: Perform data preprocessing on functional magnetic resonance imaging (fMRI) signals, denoise the raw fMRI signals and register them to a standard space, and extract data of voxel points located in the visual brain region to construct the fMRI input vector;

[0062] In this embodiment, the functional magnetic resonance imaging (fMRI) signal is preprocessed using the FreeSurfer tool, fMRIPrep tool, ciftify tool, and multimodal surface mapping (MSM) method to obtain the registered cortical standard fs_LR-32k spherical space.

[0063] Specifically, for structural data, the FreeSurfer tool was used to perform craniotomy, segmentation, and surface reconstruction on T1-weighted structural images; for functional data, the fMRIPrep tool was used for motion correction, slice time correction, field distortion correction, and registration. This process yields images with the following shape: fMRI data, of which, , , Indicates three-dimensional spatial resolution. This indicates the time frame number, i.e., the change data of 3D brain MRI imaging over time. The Ciftify toolbox is used to map the functional data from voxel space to CIFTI space, including: mapping the functional data to the cortical surface generated by FreeSurfer (surface mapping); downsampling the data from a standard fs_LR spherical template with 164k vertices to 32k vertices; applying Gaussian spatial smoothing with a full width at most 4 mm to improve the signal-to-noise ratio; and applying a temporal high-pass filter to remove low-frequency drift. After this step, a shape of [shape missing] can be obtained. fMRI data, of which, The number of points in CIFTI space, which is the result of cortical surface segmentation in three-dimensional space. The point set extracted from the sample; spherical registration was performed using the multimodal surface mapping (MSM) method to align the cortical surfaces of different subjects, including: mapping each subject's cortical surface to a standard spherical template and using the MSM method for nonlinear alignment to minimize morphological differences between surfaces;

[0064] In this embodiment, voxel data located in the visual brain region are extracted. Specifically, the V1, V2, V3, and V4 brain regions defined by HCP-MMP1.0 are selected as the visual brain regions. After this step, the shape can be obtained as follows: fMRI data, of which, This represents the number of visual brain region subsets extracted from the CIFIT space;

[0065] S3: Preprocess the video data, specifically including:

[0066] Spatially, the resolution of the video was adjusted to 256×256 using bilinear interpolation, and temporally, the frame rate of the video was adjusted to 3FPS. Each 2 seconds was used as a video sample. The paired functional magnetic resonance imaging (fMRI) signal lagged 4 seconds behind the video signal. The average of all paired fMRI frames within a video sample window could be taken as the corresponding fMRI signal.

[0067] In this embodiment, the video data is processed into 6 frames per 2 seconds. Its color map and spatial depth map are extracted for training the visual perception pathway, and its frame semantic embedding and text description semantic embedding are extracted for semantic pathway calculation. Specifically, the CLIP model image encoder extracts CLIP image embedding for each frame image, and then performs average calculation to obtain the frame semantic embedding. For each frame image, the graph-to-text model is used to obtain the text description, and the English word "then" is used to concatenate the text description into a sentence. The resulting sentence is input into the CLIP text encoder to obtain the text description semantic embedding.

[0068] In this embodiment, when extracting the color map, each frame is subjected to lossy spatial downsampling bilinear interpolation, the resolution is adjusted to 8×8, and then bilinear interpolation is used to upsample to 256×256.

[0069] In this embodiment, when extracting the spatial depth map, a depth estimation model is used to extract the spatial depth map for each frame, and the output of the model is normalized to a value range of [0, 1] using the sigmoid function. The depth estimation model can be the DepthFM model, the DepthAnything model, or the DepthAnything V2 model.

[0070] In this embodiment, when extracting the text description of the video, a graph-based text model is used to extract English text descriptions for each frame, and the English conjunction "then" is used to connect each sentence. Then, a large language model is used to rewrite and polish the sentences to finally obtain a complete video text description. The graph-based text model can be the BLIP model, BLIP2 model, or BLIP3 model, and the large oracle model can be the GPT-4o model, DeepSeek model, or Qwen model.

[0071] S4: Using the fMRI input vector as features and the color map and spatial depth map as labels, a dual-tower mask autoencoder based on two-stage training is used to train the visual perception pathway and obtain the prediction of the color map and spatial depth map.

[0072] like Figure 3 As shown, a dual-tower masked autoencoder architecture with two-stage training is used when training the visual perception pathway, specifically:

[0073] The first stage involves training a single-modal mask autoencoder within each of the video visual perception modal and the fMRI modal. The video visual perception modal consists of a color map for color perception and a spatial depth map for spatial depth perception, while the fMRI modal consists of the fMRI input vector. The training yields a color map encoder, a spatial depth map encoder, an fMRI encoder, a color map decoder, a spatial depth map decoder, and an fMRI decoder.

[0074] Let the color diagram be The spatial depth map is The fMRI input vector is The encoder and decoder of the color map mask autoencoder are respectively... and The encoder and decoder of the spatial depth map mask autoencoder are respectively and The encoder and decoder of the fMRI mask autoencoder are respectively and Then the training of the single-modal autoencoder in the first stage can be expressed as:

[0075] ;

[0076] ;

[0077] ;

[0078] in, This indicates that the input data is randomly masked;

[0079] During the first phase of training, you can choose to load a pre-trained model. For the fMRI modality, the pre-trained model is loaded from Mind-Vis' fMRI mask autoencoder. For the video modality (i.e., color map and spatial depth map), the pre-trained model is loaded from the VideoMAE-B model.

[0080] For the fMRI modality, color map modality, and spatial depth map modality, in the single-modality masked autoencoder training phase, the encoder input consists of a learnable [CLS] word embedding and the embedding of the input data (fMRI input vector embedding, 16×16 block embedding of the color map, and 16×16 block embedding of the spatial depth map). The masking rate of the model is set to 100%, meaning that the decoder input consists of the [CLS] word embedding output by the encoder and a trainable [MASK] word embedding. Through the first phase of training, the encoder and decoder for each modality are obtained, and the decoder can decode based on only one [CLS] embedding.

[0081] The second stage combines the fMRI encoder with the color map decoder and the spatial depth map decoder to obtain the cross-modal mapping capability of color map and spatial depth map from fMRI input vector, thereby realizing fMRI video decoding with high spatial perception consistency.

[0082] The second phase of cross-modal training is as follows:

[0083] ;

[0084] ;

[0085] During the second phase of cross-modal encoder-decoder training, the fMRI encoder, color map encoder, spatial depth map encoder, color map decoder, and spatial depth map decoder are loaded, while the color map and spatial depth map encoders are frozen.

[0086] The training in this embodiment has two optimization objectives: aligning the [CLS] word embeddings of the fMRI encoder and the video encoder (i.e., the color map and spatial depth map encoders) and reconstructing the color map and spatial depth map modalities. These two optimization objectives are accomplished by the contrastive learning loss function and the color map and spatial depth map reconstruction loss function, respectively. After this two-stage dual-tower masked autoencoder is trained, given the fMRI input vector, the fMRI encoder can output the [CLS] word embedding containing visual perception information, and reconstruct the color perception map and spatial depth perception map through the video decoder based on the learned [MASK] word embedding.

[0087] In the first phase of training, for the fMRI input vector and color map, the loss function is a weighted sum of the L1 loss and the MSE loss, expressed as:

[0088] ;

[0089] For output as a spatial depth map To smooth the output spatial depth map, the model introduces an additional total variation regularization term, defined as:

[0090] ;

[0091] in, and These are the height and width of the image, respectively. and Let represent the pixel coordinates of the image, then the loss function for the spatial depth map is:

[0092] ;

[0093] The reconstruction loss is also used in the second phase to ensure the second optimization objective;

[0094] To align the [CLS] word embeddings of the fMRI and color and spatial depth maps (i.e., the first optimization objective), an additional InfoNCE loss function is introduced, expressed as:

[0095] ;

[0096] in, and These represent two [CLS] word embeddings that need to be aligned, for fMRI color perception. The [CLS] word embedding indicates fMRI. The [CLS] word embedding representing the color map is relevant for fMRI spatial depth perception. The [CLS] word embedding indicates fMRI. [CLS] word embeddings representing spatial depth maps Indicates batch size, This is a hyperparameter, representing the temperature coefficient;

[0097] For fMRI color perception, the loss function is:

[0098] ;

[0099] For fMRI spatial depth sensing, the loss function is:

[0100] ;

[0101] After the second stage of training is completed, the model can output predicted color maps and spatial depth maps based on the fMRI input vector, thereby completing the spatial perception pathway.

[0102] S5: Using the fMRI input vector as features, frame semantic embedding and text description semantic embedding as labels, and InfoNCE loss as the loss function for contrastive learning, the semantic pathway is trained to obtain the prediction of fMRI semantic embedding.

[0103] In this embodiment, when training the semantic pathway, contrastive learning is used and the InfoNCE loss function is applied, which is defined as follows:

[0104] ;

[0105] in, This is a hyperparameter, called the temperature coefficient;

[0106] In this embodiment, the semantic path training performs trimodal alignment between the fMRI input vector and the frame semantic embedding and text description semantic embedding of the video, thereby mapping the fMRI input vector to the CLIP semantic latent space and achieving fMRI video decoding with high semantic consistency.

[0107] like Figure 4 As shown, the CLIP model maps image and text descriptions to a semantic latent space, while the learning objective of the fMRI encoder is to map fMRI to the same latent space and align them. In the latent space, it brings fMRI closer to positive image and text samples and pushes fMRI further away from negative image and text samples.

[0108] In this embodiment, bidirectional InfoNCE loss is used when training the fMRI semantic pathway, as follows:

[0109] ;

[0110] in, Represents the InfoNCE loss function, which will Substituting the [CLS] word embedding from fMRI, Substituting the video's frame semantic embedding and text description semantic embedding into the equations, the semantic embedding similarity loss from the contrastive learning method is:

[0111] ;

[0112] in, [CLS] word embedding for fMRI encoder, The average [CLS] word embeddings are the outputs of the CLIP image encoder for each of the 6 frames of a 2-second video sample. The text descriptions of the six frames of a 2-second video sample are concatenated into a sentence and then input into the CLIP text encoder to obtain the [CLS] word embedding;

[0113] Furthermore, to achieve closer semantic embedding, in addition to using InfoNCE as the loss function, L1 and MSE losses are also introduced. Therefore, the final fMRI semantic pathway loss function is:

[0114] ;

[0115] After the above training process, the model can output a predicted semantic embedding based on the fMRI input vector. This semantic embedding is aligned with the CLIP latent space and contains rich natural image and text semantic information, thereby completing the fMRI semantic perception pathway.

[0116] S6: The predictions of color map, depth map, and fMRI semantic embedding are used as guiding conditions for the video diffusion model through controllable generation technology. The diffusion model and controllable generation adapter module are fine-tuned to obtain the video prediction.

[0117] In this embodiment, when using controllable generation technology to guide the video diffusion model under multiple conditions, the conditions of the image modalities (color map and spatial depth map) are introduced into the feature map of the diffusion model using controllable generation adapter features, and the conditions of the embedding modalities (fMRI semantic embedding) are introduced into the diffusion model using a cross-attention mechanism.

[0118] Let the color map output by the fMRI visual perception pathway predict as Depth map prediction is The fMRI semantic embedding prediction output by the fMRI semantic perception pathway is Then the diffusion model at time step The predicted noise can be expressed as:

[0119] ;

[0120] in, Indicated by The U-Net skeleton for the diffusion model with parameters. To add noisy samples to the latent space, during the inference phase, multi-conditional classifier-free guidance (CFG) is used:

[0121] ;

[0122] Among them, negative conditions , , This is the average result over the training dataset;

[0123] In this embodiment, the controllable generation adapter adopts the ControlNet model or the T2I-Adapter model;

[0124] This invention constructs an fMRI visual perception pathway, which, given an fMRI input vector, can predict the color map and spatial depth map of a corresponding video sample. It also constructs an fMRI semantic perception pathway, which, given an fMRI input vector, can predict the CLIP spatial semantic embedding of a corresponding video sample. By combining the obtained color map, spatial depth map, and CLIP semantic embeddings, the final natural video is reconstructed. This goal is achieved through joint fine-tuning of the fMRI visual perception pathway, the fMRI semantic perception pathway, the video diffusion model, and the controllable generation model. During the fine-tuning of the video diffusion model, the controllable generation adapters used are the color map adapter and depth map adapter of the T2I-Adapter, and the diffusion model used is based on Stable Diffusion. The Tune-A-Video image-to-video fine-tuning model in version 1.4 only trains the color map adapter, depth map adapter, and the self-attention head, mutual attention head, and temporal attention head of the diffusion model during video fine-tuning. The parameters of all other modules are frozen. After joint fine-tuning, this invention can predict color map videos and spatial map videos through the fMRI visual perception pathway, predict CLIP spatial semantic embeddings through the fMRI semantic perception pathway, and obtain natural image outputs based on the fine-tuned, controllable generator adapter and video diffusion model. Unlike existing technical solutions, this invention, by introducing both visual and semantic perception pathways, ultimately achieves natural image decoding outputs that possess both visual perception consistency and semantic consistency.

[0125] This embodiment specifically selects the CC2017 dataset, which contains 4320 training and test sets for three subjects. 3 and 1200 There are 3 samples, where each training set sample is repeated 3 times and used as an independent sample, and each test set sample is repeated 5 times. The average of the samples is used as a test sample.

[0126] When training the fMRI visual perception pathway, for the first stage, a learning rate of 100% was used. The optimizer is Adam, and its parameters are: , Regularization weight decays to The batch size is 192; when training the fMRI single-modal mask autoencoder, the fMRI input vector is randomly sparsified to 0 with a 20% probability; when training the color map and spatial depth map mask autoencoders, the image is randomly flipped horizontally or vertically with a 50% probability and randomly rotated. to Brightness, contrast, saturation (colormap only), and hue (colormap only) were randomly adjusted to a factor of 0.8 to 1.2; the hyperparameters of the loss function were... For the second stage, the random sparsity probability of fMRI was adjusted to 15%, and Mixup enhancement was performed with a 20% probability; the batch size was 160; and the hyperparameters of the loss function were... , , , During fMRI semantic pathway training, the random sparsity probability of fMRI was 20%, and the batch size was 128. During fine-tuning of the controllable video diffusion model, the batch size was 16, and the model iterated a total of 20,000 times.

[0127] In this embodiment, a total of 8 metrics are selected to measure the video neural decoding performance of the model, including 4 metrics for measuring visual perception consistency: FID (the smaller the better), Hue, PSNR, and SSIM, and 4 metrics for measuring semantic consistency: Video2-way, Video 50-way, Frame 2-way, and Frame 50-way (the rest are better the larger the better).

[0128] The experimental results of this embodiment compared with other methods are shown in Table 1 below:

[0129] Table 1. Comparison of experimental results between the video neural decoding method of this embodiment and other methods.

[0130]

[0131] Therefore, it can be seen that the method in this embodiment significantly outperforms other methods in FID, Hue, and PSNR, demonstrating high visual perceptual consistency. Furthermore, it significantly outperforms other methods in Video-2, Video-50, Frame-2, and Frame-50, demonstrating high semantic consistency. Figure 5 As shown, the visualization comparison results of the method of the present invention and other methods are obtained. The video stimulus in the figure is the original sample. The closer the output results of each method are to the original sample, the better the effect. Compared with other methods, the method of the present invention can not only achieve a high degree of consistency between the reconstructed video and the original video at the semantic level, but also achieve a high degree of similarity at the visual perception level. The position of the main object in the reconstructed video matches that of the original video.

[0132] Example 2

[0133] This embodiment provides an fMRI video neural decoding system based on visual perception and semantic consistency, used to implement the fMRI video neural decoding method based on visual perception and semantic consistency in Embodiment 1, including: a data acquisition module, an fMRI input vector construction module, a data extraction module, a visual perception pathway training module, a semantic pathway training module, and a multi-condition guidance module;

[0134] In this embodiment, the data acquisition module is used to acquire fMRI signals and their corresponding video data;

[0135] In this embodiment, the fMRI input vector construction module is used to preprocess the fMRI signal and construct the fMRI input vector;

[0136] In this embodiment, the data extraction module is used to extract color maps, spatial depth maps, frame semantic embeddings, and text description semantic embeddings from video data;

[0137] In this embodiment, the visual perception pathway training module is used to train the visual perception pathway based on a dual-tower mask autoencoder with fMRI input vector as features and color map and spatial depth map as labels, to obtain predictions of color map and spatial depth map.

[0138] In this embodiment, the semantic pathway training module is used to train the semantic pathway by using the fMRI input vector as a feature and the frame semantic embedding and text description semantic embedding as labels, so as to obtain the prediction of the fMRI semantic embedding.

[0139] In this embodiment, the multi-condition guidance module is used to use the prediction of the color map, the prediction of the spatial depth map, and the prediction of the fMRI semantic embedding as guidance conditions for the video diffusion model, and to guide the video diffusion model in multiple conditions to obtain the decoding output.

[0140] Example 3

[0141] This embodiment provides a storage medium, which may be a ROM, RAM, disk, optical disk, or other storage medium. The storage medium stores one or more programs. When the programs are executed by a processor, they implement the fMRI video neural decoding method based on visual perception and semantic consistency as described in Embodiment 1.

[0142] Example 4

[0143] This embodiment provides a computing device, which may be a desktop computer, laptop computer, smartphone, PDA handheld terminal, tablet computer or other terminal device with display function. The computing device includes a processor and a memory. The memory stores one or more programs. When the processor executes the program stored in the memory, it implements the fMRI video neural decoding method based on visual perception and semantic consistency of Embodiment 1.

[0144] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A neural decoding method for fMRI video based on visual perception and semantic consistency, characterized in that, Includes the following steps: Acquire fMRI signals and their corresponding video data; Data preprocessing is performed on the fMRI signals to construct the fMRI input vector; Extract color maps, spatial depth maps, frame semantic embeddings, and text description semantic embeddings from video data; Text descriptions are extracted from each frame of the video data, the text descriptions are connected using conjunctions, and rewritten based on a large language model to obtain the semantic embedding of the text descriptions. Using fMRI input vectors as features and color and spatial depth maps as labels, a visual perception pathway is trained based on a two-stage trained dual-tower mask autoencoder to obtain predictions for color and spatial depth maps, specifically including: The first stage of training involves training a single-modal mask autoencoder within its respective modality using color maps, spatial depth maps, and fMRI input vectors. Let the color diagram be The spatial depth map is The fMRI input vector is The encoder and decoder of the color map mask autoencoder are respectively... and The encoder and decoder of the spatial depth map mask autoencoder are respectively and The encoder and decoder of the fMRI mask autoencoder are respectively and During training, for the fMRI modality, the fMRI mask autoencoder is loaded from Mind-Vis; for the color map and spatial depth map, it is loaded from the VideoMAE-B model. Therefore, the first stage of single-modal autoencoder training is represented as follows: ； ； ； in, This indicates that the input data is randomly masked; The second phase of training involves cross-modal training using color maps, spatial depth maps, and fMRI input vectors. The second phase of cross-modal training is as follows: ； ； Using fMRI input vectors as features and frame semantic embeddings and text description semantic embeddings as labels, a semantic pathway is trained to obtain the prediction of fMRI semantic embeddings. Training the semantic pathway includes: aligning the fMRI input vector with the frame semantic embedding and the text description semantic embedding in a three-modal manner, and mapping the fMRI input vector to the CLIP semantic latent space; The prediction of color map, spatial depth map, and fMRI semantic embedding are used as guiding conditions for the video diffusion model. The video diffusion model is guided by multiple conditions to obtain the decoded output.

2. The fMRI video neural decoding method based on visual perception and semantic consistency according to claim 1, characterized in that, Data preprocessing of fMRI signals to construct fMRI input vectors includes: The original fMRI signal was denoised and registered to a standard space; Data from voxel points located in the visual brain region are extracted to obtain the fMRI input vector.

3. The fMRI video neural decoding method based on visual perception and semantic consistency according to claim 1, characterized in that, Extracting color maps, spatial depth maps, and frame semantic embeddings from video data, specifically including: For each frame of the video data, bilinear interpolation downsampling is used to adjust the resolution, and bilinear interpolation upsampling is used to obtain the color map; A spatial depth map is extracted for each frame based on a depth estimation model; The CLIP image embedding is extracted from each frame of the video data, and then the average is calculated to obtain the frame semantic embedding.

4. The fMRI video neural decoding method based on visual perception and semantic consistency according to claim 1, characterized in that, When training a single-modal masked autoencoder, the loss function for the color image and fMRI input vector is a weighted sum of the L1 loss and the MSE loss. When training a single-modal masked autoencoder using a spatial depth map, the loss function is a weighted sum of the L1 loss and the MSE loss, with an additional total variational regularization term. When performing cross-modal training on color maps and fMRI input vectors, the loss function is a weighted sum of InfoNCE loss, L1 loss, and MSE loss; When performing cross-modal training on spatial depth maps and fMRI input vectors, the loss function is a weighted sum of InfoNCE loss, L1 loss, and MSE loss, with an additional total variational regularization term.

5. The fMRI video neural decoding method based on visual perception and semantic consistency according to claim 1, characterized in that, In the step of training the semantic pathway, the loss function is a weighted sum of semantic embedding similarity loss, L1 loss, and MSE loss.

6. The fMRI video neural decoding method based on visual perception and semantic consistency according to claim 1, characterized in that, The prediction of color map, spatial depth map, and fMRI semantic embedding are used as guiding conditions for the video diffusion model, providing multi-condition guidance for the model. Specifically, this includes: Color map prediction, spatial depth map prediction, and fMRI semantic embedding are used as guiding conditions for the video diffusion model through controllable generation techniques. The image modal conditions corresponding to the color map and spatial depth map are used to obtain controllable generation adapter features using controllable generation techniques, and then input into the feature map of the diffusion model; The embedding modality conditions corresponding to the fMRI semantic embedding are input into the video diffusion model using a cross-attention mechanism; Adjust the video diffusion model and the controllable generation adapter, and obtain the decoded output based on the adjusted controllable generation adapter and video diffusion model.

7. An fMRI video neural decoding system based on visual perception and semantic consistency, characterized in that, The method for implementing the fMRI video neural decoding method based on visual perception and semantic consistency as described in any one of claims 1-6 includes: a data acquisition module, an fMRI input vector construction module, a data extraction module, a visual perception pathway training module, a semantic pathway training module, and a multi-condition guidance module; The data acquisition module is used to acquire fMRI signals and their corresponding video data; The fMRI input vector construction module is used to preprocess fMRI signals and construct fMRI input vectors; The data extraction module is used to extract color maps, spatial depth maps, frame semantic embeddings, and text description semantic embeddings from video data. The visual perception pathway training module is used to train the visual perception pathway based on a dual-tower mask autoencoder with fMRI input vector as features and color map and spatial depth map as labels, and to obtain the prediction of color map and spatial depth map. The semantic pathway training module is used to train the semantic pathway by using the fMRI input vector as a feature and the frame semantic embedding and text description semantic embedding as labels, so as to obtain the prediction of the fMRI semantic embedding. The multi-condition guidance module is used to use the prediction of color map, spatial depth map, and fMRI semantic embedding as guidance conditions for the video diffusion model, and to guide the video diffusion model in multiple ways to obtain the decoding output.

8. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the fMRI video neural decoding method based on visual perception and semantic consistency as described in any one of claims 1-6.

9. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the fMRI video neural decoding method based on visual perception and semantic consistency as described in any one of claims 1-6.

Images