A method for reconstructing images from brain activity based on multimodal diffusion models

Abstract

The present invention discloses a method for reconstructing images from brain activity based on a multimodal diffusion model, comprising the following steps: A1, preprocessing collected electroencephalogram (EEG) data; A2, inputting the preprocessed EEG data into an EEG feature extraction network to obtain EEG data features; A3, inputting the EEG data features into an image coding regression model, an image embedding regression model, and a text embedding regression model, respectively, to obtain image coding results, CLIP image embedding vectors, and CLIP text embedding vectors; A4, using the image coding results as image input to the multimodal diffusion model, the CLIP image embedding vectors as image latent variables, and the CLIP text embedding vectors as cue word latent variables, performing denoising diffusion decoding, and obtaining a reconstructed image. The present invention can generate highly recognizable and clear images from EEG signals.

Description

Technical Field

[0001] The present invention relates to the fields of brain-computer interface and artificial intelligence, and in particular to a method for reconstructing images from brain activities based on a multimodal diffusion model. Background Art

[0002] Vision is one of the primary ways humans acquire information about the outside world. The vast majority of information humans acquire comes from vision, which is closely related to perception, emotion, and memory. Furthermore, for visual stimuli to be perceived and generate meaningful information, corresponding brain activity is essential. This brain activity generates electrical signals, which are collected by appropriate brain-computer interface devices and, with the help of deep learning models, can accomplish tasks such as classification and image reconstruction.

[0003] Early image generation algorithms such as GAN had problems such as training difficulty, large data requirements, and mode collapse. The quality and diversity of reconstructed images were difficult to guarantee. At the same time, the cost of collecting large amounts of EEG data was high, making GAN training even more difficult. Summary of the Invention

[0004] The present invention mainly solves the technical problems existing in the prior art, such as the difficulty of image reconstruction and the high data volume requirement, and provides a method for reconstructing images from brain activity based on a multimodal diffusion model.

[0005] The present invention solves the above technical problems mainly through the following technical solutions: a method for reconstructing images from brain activity based on a multimodal diffusion model, comprising the following steps:

[0006] A1. Preprocess the collected EEG data;

[0007] A2. Input the preprocessed EEG data into the EEG feature extraction network to obtain EEG data features;

[0008] A3. Input the EEG data features into the image encoding regression model, image embedding regression model, and text embedding regression model respectively to obtain the image encoding result, CLIP image embedding vector (CLIP image embedding), and CLIP text embedding vector (CLIP text embedding);

[0009] A4. Use the image encoding result as the image input of the multimodal diffusion model, the CLIP image embedding vector as the image latent variable input of the multimodal diffusion model, and the CLIP text embedding vector as the cue word latent variable input of the multimodal diffusion model. Perform denoising diffusion decoding to obtain a reconstructed image.

[0010] The multimodal diffusion model includes an image encoder, an image decoder, a plurality of residual blocks, a plurality of fully connected residual blocks, a plurality of image cross-attention layers and a plurality of text cross-attention layers; the input of the image encoder is the image encoding result, which is output to the first residual block, the output of the first residual block enters the first image cross-attention layer and the first text cross-attention layer, the output of the first image cross-attention layer and the first text cross-attention layer enters the second residual block, and the output of the first text cross-attention layer also enters the first fully connected residual block, the output of the first fully connected residual block and the second residual block enters the second text cross-attention layer, and the output of the second residual block also enters the second image cross Attention layer, the output of the second image cross-attention layer and the second text cross-attention layer enters the third residual block, and the output of the second text cross-attention layer also enters the second fully connected residual block, the output of the second fully connected residual block and the third residual block enter the third text cross-attention layer, and the output of the third residual block also enters the third image cross-attention layer, the output of the third image cross-attention layer and the third text cross-attention layer enter the fourth residual block, the output of the fourth residual block enters the image decoder, and the image decoder outputs the final reconstructed image; all image cross-attention layers input the CLIP image embedding vector, and all text cross-attention layers input the CLIP text embedding vector.

[0011] This model combines two input modalities: text and images. It achieves cross-modal information sharing through a shared global layer and exchangeable data and context layers. The global layer is shared across all modalities and tasks and is responsible for processing common features. The data layer is responsible for processing data from a specific modality (such as images or text), and the context layer is responsible for processing contextual information of the input modality (such as text embeddings or image embeddings). Through this design, the multimodal diffusion model can share information between different tasks while maintaining the ability to handle specific tasks.

[0012] The multimodal diffusion model uses a cross-attention layer as the context layer, and its structure is similar to the traditional LatentDiffusion:

[0013] Attention(Q,K,V)=softmax(QK T / d 0.5 )×V;

[0014] Q = W Q (i) ×φ i (z t ), K = W K (i) ×τ θ (y), V = W V (i) ×τθ (y);

[0015] Among them, φ i (z t ) represents the intermediate result of U-Net at time point t, τ θ is the embedding vector obtained by projecting the EEG signal after encoding, W Q (i) 、W K (i) and W V (i) It is a learnable parameter matrix; secondly, a fully connected residual block is used to convert the original 768-dimensional latent variables output by the text cross-attention layer into 320*4 features as the text data layer, and the latent variables output by the image cross-attention layer are used as the image data layer through residual connection, which helps to reduce the difference between the text and image embedding spaces, thereby accelerating convergence and improving performance.

[0016] A multimodal diffusion model is used for image reconstruction. The input of the diffusion model includes both the image encoding results of the deep variational autoencoder as the base map representing low-level information such as color and contour as the image modality, and the text embedding vector corresponding to the text input as the text modality. It combines the advantages of the two diffusion workflows of text generation and image rewriting to achieve better reconstruction effects.

[0017] Preferably, the objective function during training of the multimodal diffusion model is:

[0018] ;

[0019] Among them, x0 is the original image, q is the image after adding noise in the intermediate process, p is the noise added in each forward diffusion; E represents the expectation, which is used to calculate the average value of the random variable, θ represents the learnable model parameter, and T represents the Tth moment of adding noise, that is, x 1:T represents the joint probability distribution of noise added from the first time to the time T.

[0020] Preferably, the channels selected for EEG data are O1, Oz, O2, PO7, PO3, POz, PO4, PO8, P7, P5, P3, P1, Pz, P2, P4, P6, and P8 near the occipital and parietal cortices. Preprocessing involves first removing DC noise with a 1 Hz high-pass filter and then normalizing the data using a whitening transform to obtain preprocessed EEG data. This processed EEG data can reduce the weight of channels with high noise content and increase the signal-to-noise ratio of the EEG signal.

[0021] Preferably, the EEG feature extraction network includes a three-layer long short-term memory network and a linear fully connected layer, and the training process of the EEG feature extraction network is:

[0022] The preprocessed EEG data is fed into the EEG feature extraction network to obtain EEG data features, and then the Triplet-Loss loss of the EEG data features is calculated. The formula is:

[0023] ;

[0024] Among them, x a Represents the current data, x p Represents data with the same label as the current data (positive sample), x n represents data with different labels from the current data (negative samples), f θ is the learnable neural network parameter, that is, the parameter of the feature extraction network, and β is the margin term, which is used to control the distance between positive and negative samples; training is completed by back-propagating the gradient.

[0025] Preferably, after training the EEG feature extraction network, a test set is input and the K-Means clustering accuracy is calculated using the true labels of the images and the features extracted from the corresponding EEG. If the K-Means clustering accuracy is greater than 0.9, the training is completed.

[0026] Preferably, the image coding regression model, the image embedding regression model and the text embedding regression model are all ridge regression models, and the weight coefficients of the image coding regression model, the image embedding regression model and the text embedding regression model are determined by the following method:

[0027] T101, use the EEG feature extraction network to extract EEG features and standardize them to make their mean 0 and standard deviation 1;

[0028] T102. Use a deep variational autoencoder to extract features from the images in the training data of the training set to obtain image coding features; use the ridge regression method to perform regression training on the EEG features and image coding features, and save the weights to obtain the weight coefficients of the image coding regression model;

[0029] T103. Use the CLIP model to convert the text in the training data of the training set into CLIP text feature data; use the ridge regression method to perform regression training on the EEG features and CLIP text features, and save the weights to obtain the weight coefficients of the text embedding regression model; the CLIP (Contrastive Language-Image Pre-Training) model uses a pre-trained existing model;

[0030] T104. Use the CLIP model to convert the images in the training data of the training set into CLIP image feature data; use the ridge regression method to perform regression training on the EEG features and CLIP image features, and save the weights to obtain the weight coefficients of the image embedding regression model.

[0031] Extracting the embedding vector corresponding to the image from the CLIP image is as follows: first convert the image to 512*512 resolution, then convert it to a tensor floating point array, then map the pixel values ​​from the [0,1] interval to [-1,1], and then call the clip_encode_vision method of the Transformer library;

[0032] With these three sets of weights, we can use the ridge regression model to convert EEG features into the required image encoding results, CLIP image embedding vectors, and CLIP text embedding vectors. We can then measure the regression performance by calculating the Euclidean distance and Pearson correlation coefficient between the predicted results and the test set.

[0033] The training set includes EEG data, corresponding real pictures (used to obtain real CLIP image features and deep variational autoencoder encoding feature maps), and corresponding text (used to obtain real CLIP text features).

[0034] Preferably, the deep variational autoencoder comprises two parts: an encoder and a decoder. The encoder encodes the input image into feature maps of different resolutions after multiple downsampling, and feeds the feature maps into the subsequent decoder in the form of a dictionary. The key of the dictionary is the resolution, and the value is the feature map corresponding to the resolution.

[0035] The structure of the encoder block is as follows: the first layer starts with a two-dimensional convolution layer with a 3*3 convolution kernel, 3 input channels, 512 output channels, and a padding and stride of 1 to perform convolution operations on the input image. Subsequently, it passes through several encoder blocks. Each encoder block consists of four layers of two-dimensional convolution layer networks. The convolution kernels of the four layers are 1*1, 3*3 or 1*1, 3*3 or 1*1, 1*1 respectively. The convolution kernel size of the two middle layers depends on the pixels of the input feature map. If the pixel is greater than 2, it is 3*3, otherwise it is 1*1. The number of input channels of the four-layer network is 1*3. The numbers of the output channels are 128, 128, 128, and 512 respectively, and the activation function is GELU. For the encoder, residual connections are also used to alleviate the gradient vanishing problem. When passing parameters to the encoder block, a residual=True is passed, indicating that the residual connection of the encoder block is enabled. The encoder block then adds the input of the encoder block to the result of the 1*1 convolution. If the encoder block needs to be downsampled, it is finally subjected to two-dimensional average pooling according to the downsampling rate to obtain the downsampled feature map.

[0036] Based on the aforementioned encoder block structure, the encoder structure is as follows: the resolution of the first to eleventh encoder blocks is 64, and no downsampling is performed; the resolution of the twelfth encoder block is 64, and the downsampling rate is 2; the resolution of the thirteenth to thirty-second encoder blocks is 32, and no downsampling is performed; the resolution of the thirty-third encoder block is 32, and the downsampling rate is 2; the resolution of the thirty-fourth to forty-second encoder blocks is 16, and no downsampling is performed; the resolution of the forty-third encoder block is 16, and the downsampling rate is 2; the resolution of the forty-fourth to fifty-first encoder blocks is 8, and no downsampling is performed; the resolution of the fifty-second encoder block is 8, and the downsampling rate is 2; the resolution of the fifty-third to fifty-ninth encoder blocks is 4, and no downsampling is performed; the resolution of the sixtieth encoder block is 4, and the downsampling rate is 4; the resolution of the sixtieth to sixty-sixth encoder blocks is 1, and no downsampling is performed; and finally the encoder outputs a dictionary, the key of which is the image resolution, and the value is the feature map corresponding to the resolution.

[0037] Preferably, the decoder is responsible for performing a series of feature fusion and upsampling operations on the feature maps of different resolutions output by the encoder to gradually restore the resolution of the image, and output an image feature map, which contains two low-level information: the color and contour of the original image; the decoder is composed of several decoder blocks, and the decoder block contains three sub-modules. The first sub-module Block1 is used to extract the mean and variance of the latent variables from the feature map of the encoder, inputs the concatenation of the current resolution feature map and the encoder feature map, and outputs the mean and variance of the latent variables; the second sub-module Block2 is used to calculate the mean and variance of the prior distribution, inputs the current resolution feature map, and outputs the prior mean, prior variance and a residual feature of the latent variables; the third sub-module Block3 is a residual module, which is used to further process the feature map.

[0038] The specific decoder structure is as follows: the resolution of the first and second decoder blocks is 1, and no resolution fusion is performed; the third decoder block upsamples the feature map of resolution 1 and fuses it with the feature map of resolution 4; the resolution of the fourth to sixth decoder blocks is 4, and no resolution fusion is performed; the seventh decoder block upsamples the feature map of resolution 4 and fuses it with the feature map of resolution 8; the resolution of the eighth to fourteenth decoder blocks is 8, and no resolution fusion is performed; the fifteenth upsamples the feature map of resolution 8 and fuses it with the feature map of resolution 16; the resolution of the sixteenth to thirtieth decoder blocks is 16, and no resolution fusion is performed; the thirty-first decoder block upsamples the feature map of resolution 16 and fuses it with the feature map of resolution 32; the resolution of the thirty-second to sixty-second decoder blocks is 32, and no resolution fusion is performed; the sixty-third decoder block upsamples the feature map of resolution 32 and fuses it with the feature map of resolution 64; the resolution of the sixty-fourth to seventy-fifth decoder blocks is 64, and no resolution fusion is performed; finally, the image coding features of 64 resolution are obtained through 1*1 two-dimensional convolution.

[0039] By encoding the image using a deep variational autoencoder, we can obtain the underlying features of the image's contour and color.

[0040] As a preference, when training a deep variational autoencoder, in order to make the feature map obtained by image encoding as similar as possible to the original image, a structural similarity metric is used as the loss function:

[0041] ;

[0042] Among them, μ x and σ x Represent the mean and variance of the real picture, μ M(x) and σ M(x) where C1 and C2 represent the mean and variance of the feature map after encoding by the deep variational autoencoder, respectively. C1 and C2 are constants for stable calculation. They are empirical values, and C1 = 6.5025 and C2 = 58.5225 are generally used.

[0043] The substantial effects brought about by the present invention are: selecting EEG channels related to visual function brain areas; establishing a long short-term memory network to extract features, adopting the contrastive learning method for training, selecting Triplet-Loss as the loss function, so that the EEG data corresponding to different samples can be distinguished, and measuring the feature extraction effect by K-Means clustering; encoding the target image through a deep variational autoencoder to extract underlying features such as color and contour, and using the features as the image modality input of the multimodal diffusion model; with the help of the multimodal diffusion model and CLIP, the image stimulus and the corresponding text information are converted into vectors, avoiding the problem of traditional generative models such as GAN requiring a large amount of training data and the problem of difficulty in training, and at the same time providing more semantic information than the original diffusion model that only uses CLIP to embed vector input; in the alignment of EEG features with the intermediate parameters of the multimodal diffusion model, the use of linear regression method can also achieve good results, further reducing the calculation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of a method for reconstructing images from brain activity based on a multimodal diffusion model according to the present invention;

[0045] Figure 2 It is a schematic diagram of the structure of a multimodal diffusion model of the present invention;

[0046] Figure 3 1 is a schematic diagram of an encoder block of a deep variational autoencoder of the present invention;

[0047] Figure 4 1 is a schematic diagram of a decoder block of a deep variational autoencoder of the present invention. DETAILED DESCRIPTION

[0048] The technical solution of the present invention will be further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0049] Example: A method for reconstructing images from brain activity based on a multimodal diffusion model, such as Figure 1 As shown, the following steps are included:

[0050] A1. Preprocess the collected EEG data. Preprocessing includes channel selection, intercepting EEG signals in a specific time period, and normalization. The input EEG data size is n_channel*time, and the output preprocessed EEG data size is n_selected_channel*time. If it is the training phase, the images in the training data also need to be preprocessed, that is, the image resolution size is unified to 512*512.

[0051] A2. Input the preprocessed EEG data into the EEG feature extraction network to obtain EEG data features. The size of the input EEG data is time*n_channel after the transposition of the preprocessed EEG signal, and the corresponding output feature is a one-dimensional vector of length 128.

[0052] A3. Input the EEG data features into the image coding regression model, image embedding regression model, and text embedding regression model respectively to obtain the image coding result, CLIP image embedding vector, and CLIP text embedding vector;

[0053] A4. Use the image encoding result as the image input of the multimodal diffusion model, the CLIP image embedding vector as the image latent variable input of the multimodal diffusion model, and the CLIP text embedding vector as the cue word latent variable input of the multimodal diffusion model. Perform denoising diffusion decoding to obtain a reconstructed image.

[0054] like Figure 2 As shown, the multimodal diffusion model includes an image encoder, an image decoder, several residual blocks, several fully connected residual blocks, several image cross-attention layers and several text cross-attention layers. The input of the image encoder is the image encoding result, which is output to the first residual block. The output of the first residual block enters the first image cross-attention layer and the first text cross-attention layer. The output of the first image cross-attention layer and the first text cross-attention layer enters the second residual block. At the same time, the output of the first text cross-attention layer also enters the first fully connected residual block. The output of the first fully connected residual block and the second residual block enters the second text cross-attention layer. At the same time, the output of the second residual block also enters the second image cross-attention layer. The output of the second image cross-attention layer and the second text cross-attention layer enters the third residual block. At the same time, the output of the second text cross-attention layer also enters the second fully connected residual block. The output of the second fully connected residual block and the third residual block enter the third text cross-attention layer. At the same time, the output of the third residual block also enters the third image cross-attention layer. The output of the third image cross-attention layer and the third text cross-attention layer enters the fourth residual block. The output of the fourth residual block enters the image decoder, and the image decoder outputs the final reconstructed image. All image cross-attention layers input the CLIP image embedding vector, and all text cross-attention layers input the CLIP text embedding vector.

[0055] In the figure, the light blue module is the data layer, the dark blue module is the context layer, and the weights of the white module are fixed and do not participate in gradient updates during training. The image encoder is the AutoKL image encoder, and the image decoder is the AutoKL image decoder. This model combines the combination of two input modalities, text and image, and realizes cross-modal information sharing through a shared global layer and exchangeable data layer and context layer. Among them, the global layer is shared among all modalities and tasks and is responsible for processing common features. The data layer is responsible for processing data of a specific modality (such as images or text). The context layer is responsible for processing the contextual information of the input modality (such as text embedding or image embedding). Through this design, the multimodal diffusion model can share information between different tasks while maintaining the ability to process specific tasks;

[0056] The multimodal diffusion model uses a cross-attention layer as the context layer, and its structure is similar to the traditional LatentDiffusion:

[0057] Attention(Q,K,V)=softmax(QK T / d 0.5 )×V;

[0058] Q = W Q (i) ×φ i (z t ), K = W K (i) ×τ θ (y), V = W V (i) ×τ θ (y);

[0059] Among them, φ i (z t ) represents the intermediate result of U-Net at time point t, τ θ is the embedding vector obtained by projecting the EEG signal after encoding, W Q (i) 、W K (i) and W V (i) It is a learnable parameter matrix; secondly, a fully connected residual block is used to convert the original 768-dimensional latent variables output by the text cross-attention layer into 320*4 features as the text data layer, and the latent variables output by the image cross-attention layer are used as the image data layer through residual connection, which helps to reduce the difference between the text and image embedding spaces, thereby accelerating convergence and improving performance.

[0060] The objective function when training the multimodal diffusion model is:

[0061] ;

[0062] Among them, x0 is the original image, q is the image after adding noise in the intermediate process, p is the noise added in each forward diffusion; E represents the expectation, which is used to calculate the average value of the random variable, θ represents the learnable model parameter, and T represents the Tth moment of adding noise, that is, x 1:T represents the joint probability distribution of adding noise from the first time to the time T, x 0:T Represents the joint probability distribution from the original data to the noise added at time T.

[0063] The channels selected for the EEG data are O1, Oz, O2, PO7, PO3, POz, PO4, PO8, P7, P5, P3, P1, Pz, P2, P4, P6, and P8 near the occipital and parietal cortices. The preprocessing step is to first remove DC noise with a 1 Hz high-pass filter and then normalize it using a whitening transformation to obtain the preprocessed EEG data.

[0064] The EEG feature extraction network consists of a three-layer long short-term memory network and a linear fully connected layer. The training process of the EEG feature extraction network is as follows:

[0065] The preprocessed EEG data is fed into the EEG feature extraction network to obtain EEG data features, and then the Triplet-Loss loss L of the EEG data features is calculated. triplet , the formula is:

[0066] ;

[0067] Among them, x a Represents the current data, x p Represents data with the same label as the current data (positive sample), x n represents data with different labels from the current data (negative samples), f θ is the learnable neural network parameter, that is, the parameter of the feature extraction network, and β is the margin term, which is used to control the distance between positive and negative samples; training is completed by back-propagating the gradient.

[0068] After training the EEG feature extraction network, the test set is input and the K-Means clustering accuracy is calculated using the true labels of the images and the corresponding EEG extracted features. If the K-Means clustering accuracy is greater than 0.9, the training is completed.

[0069] The image coding regression model, image embedding regression model and text embedding regression model are all ridge regression models. The weight coefficients of the image coding regression model, image embedding regression model and text embedding regression model are determined by the following method:

[0070] T101, use the EEG feature extraction network to extract EEG features and standardize them to make their mean 0 and standard deviation 1;

[0071] T102. Use a deep variational autoencoder to extract features from the images in the training data of the training set to obtain image coding features; use the ridge regression method to perform regression training on the EEG features and image coding features, and save the weights to obtain the weight coefficients of the image coding regression model;

[0072] T103. Use the CLIP model to convert the text in the training data of the training set into CLIP text feature data; use the ridge regression method to perform regression training on the EEG features and CLIP text features, and save the weights to obtain the weight coefficients of the text embedding regression model; the CLIP model uses a pre-trained existing model;

[0073] T104. Use the CLIP model to convert the images in the training data of the training set into CLIP image feature data; use the ridge regression method to perform regression training on the EEG features and CLIP image features, and save the weights to obtain the weight coefficients of the image embedding regression model.

[0074] Extracting the embedding vector corresponding to the image from the CLIP image is as follows: first convert the image to 512*512 resolution, then convert it to a tensor floating point array, then map the pixel values ​​from the [0,1] interval to [-1,1], and then call the clip_encode_vision method of the Transformer library;

[0075] With these three sets of weights, we can use the ridge regression model to convert EEG features into the required image encoding results, CLIP image embedding vectors, and CLIP text embedding vectors. We can then measure the regression performance by calculating the Euclidean distance and Pearson correlation coefficient between the predicted results and the test set.

[0076] The deep variational autoencoder consists of two parts: an encoder and a decoder. The encoder encodes the input image into feature maps of different resolutions after multiple downsampling, and feeds them into the subsequent decoder in the form of a dictionary. The key of the dictionary is the resolution, and the value is the feature map corresponding to the resolution.

[0077] like Figure 3As shown in the figure, the structure of the encoder block is as follows: the first layer starts with a two-dimensional convolution layer with a 3*3 convolution kernel, 3 input channels, 512 output channels, and a padding and stride of 1 to perform convolution operations on the input image. Subsequently, it passes through several encoder blocks. Each encoder block consists of four layers of two-dimensional convolution layer networks. The convolution kernels of the four layers are 1*1, 3*3 or 1*1, 3*3 or 1*1, 1*1 respectively. The convolution kernel size of the two middle layers depends on the pixels of the input feature map. If the pixel is greater than 2, it is 3*3, otherwise it is 1*1. The input channel of the four-layer network is 1*3. The number of channels is 512, 128, 128, and 128 respectively, and the number of output channels is 128, 128, 128, and 512 respectively. The activation function is GELU. For the encoder, residual connection is also used to alleviate the gradient vanishing problem. That is, when passing parameters to the encoder block, a residual=True is passed, which means that the residual connection of the encoder block is enabled. The encoder block then adds the input of the encoder block and the result of the 1*1 convolution. If the encoder block needs to be downsampled, it is finally subjected to two-dimensional average pooling according to the downsampling rate to obtain the downsampled feature map.

[0078] Based on the aforementioned encoder block structure, the encoder structure is as follows: the resolution of the first to eleventh encoder blocks is 64, and no downsampling is performed; the resolution of the twelfth encoder block is 64, and the downsampling rate is 2; the resolution of the thirteenth to thirty-second encoder blocks is 32, and no downsampling is performed; the resolution of the thirty-third encoder block is 32, and the downsampling rate is 2; the resolution of the thirty-fourth to forty-second encoder blocks is 16, and no downsampling is performed; the resolution of the forty-third encoder block is 16, and the downsampling rate is 2; the resolution of the forty-fourth to fifty-first encoder blocks is 8, and no downsampling is performed; the resolution of the fifty-second encoder block is 8, and the downsampling rate is 2; the resolution of the fifty-third to fifty-ninth encoder blocks is 4, and no downsampling is performed; the resolution of the sixtieth encoder block is 4, and the downsampling rate is 4; the resolution of the sixtieth to sixty-sixth encoder blocks is 1, and no downsampling is performed; and finally the encoder outputs a dictionary, the key of which is the image resolution, and the value is the feature map corresponding to the resolution.

[0079] The decoder is responsible for performing a series of feature fusion and upsampling operations on the feature maps of different resolutions output by the encoder to gradually restore the resolution of the image, and output the image feature map, which contains the low-level information such as color and contour of the original image; the decoder consists of several decoder blocks, such as Figure 4As shown in the figure, the decoder block mainly consists of three sub-modules. The first sub-module Block1 is used to extract the mean and variance of the latent variables from the feature map of the encoder. It inputs the concatenation of the current resolution feature map and the encoder feature map, and outputs the mean and variance of the latent variables. The second sub-module Block2 is used to calculate the mean and variance of the prior distribution. It inputs the current resolution feature map and outputs the prior mean, prior variance and a residual feature of the latent variable. The third sub-module Block3 is the residual module, which is used to further process the feature map.

[0080] The specific decoder structure is as follows: the resolution of the first and second decoder blocks is 1, and no resolution fusion is performed; the third decoder block upsamples the feature map of resolution 1 and fuses it with the feature map of resolution 4; the resolution of the fourth to sixth decoder blocks is 4, and no resolution fusion is performed; the seventh decoder block upsamples the feature map of resolution 4 and fuses it with the feature map of resolution 8; the resolution of the eighth to fourteenth decoder blocks is 8, and no resolution fusion is performed; the fifteenth upsamples the feature map of resolution 8 and fuses it with the feature map of resolution 16; the resolution of the sixteenth to thirtieth decoder blocks is 16, and no resolution fusion is performed; the thirty-first decoder block upsamples the feature map of resolution 16 and fuses it with the feature map of resolution 32; the resolution of the thirty-second to sixty-second decoder blocks is 32, and no resolution fusion is performed; the sixty-third decoder block upsamples the feature map of resolution 32 and fuses it with the feature map of resolution 64; the resolution of the sixty-fourth to seventy-fifth decoder blocks is 64, and no resolution fusion is performed; finally, the image coding features of 64 resolution are obtained through 1*1 two-dimensional convolution.

[0081] When training a deep variational autoencoder, in order to make the feature map obtained by image encoding as similar as possible to the original image, a structural similarity metric is used as the loss function:

[0082] ;

[0083] Among them, μ x and σ x Represent the mean and variance of the real picture, μ M(x) and σ M(x) where C1 and C2 represent the mean and variance of the feature map after encoding by the deep variational autoencoder, respectively. C1 and C2 are constants for stable calculation. They are empirical values, and C1 = 6.5025 and C2 = 58.5225 are generally used.

[0084] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

[0085] Although this article frequently uses terms such as multimodal diffusion model and regression model, the use of other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitations is contrary to the spirit of the present invention.

Claims

1. A method for reconstructing images from brain activity based on a multimodal diffusion model, characterized in that The following steps are involved: A1. Preprocess the collected EEG data; A2. Input the preprocessed EEG data into the EEG feature extraction network to obtain EEG data features; A3. Input the EEG data features into the image coding regression model, image embedding regression model, and text embedding regression model respectively to obtain the image coding result, CLIP image embedding vector, and CLIP text embedding vector; A4. Use the image encoding result as the image input of the multimodal diffusion model, the CLIP image embedding vector as the image latent variable input of the multimodal diffusion model, and the CLIP text embedding vector as the cue word latent variable input of the multimodal diffusion model. Perform denoising diffusion decoding to obtain a reconstructed image. The multimodal diffusion model includes an image encoder, an image decoder, a plurality of residual blocks, a plurality of fully connected residual blocks, a plurality of image cross-attention layers and a plurality of text cross-attention layers; the input of the image encoder is the image encoding result, which is output to the first residual block, the output of the first residual block enters the first image cross-attention layer and the first text cross-attention layer, the output of the first image cross-attention layer and the first text cross-attention layer enters the second residual block, and the output of the first text cross-attention layer also enters the first fully connected residual block, the output of the first fully connected residual block and the second residual block enters the second text cross-attention layer, and the output of the second residual block also enters the second image cross Attention layer, the output of the second image cross attention layer and the second text cross attention layer enters the third residual block, and the output of the second text cross attention layer also enters the second fully connected residual block, the output of the second fully connected residual block and the third residual block enter the third text cross attention layer, and the output of the third residual block also enters the third image cross attention layer, the output of the third image cross attention layer and the third text cross attention layer enter the fourth residual block, the output of the fourth residual block enters the image decoder, and the image decoder outputs the final reconstructed image; all image cross attention layers input the CLIP image embedding vector, and all text cross attention layers input the CLIP text embedding vector; The objective function during training of the multimodal diffusion model is: ； Among them, x0 is the original image, q is the image after adding noise in the intermediate process, p is the noise added in each forward diffusion; the hollow E represents the expectation, which is used to calculate the average value of the random variable, θ represents the learnable model parameter, and T represents the Tth moment of adding noise, that is, x 1:T represents the joint probability distribution of adding noise from the first time to the time T, x 0:T Represents the joint probability distribution from the original data to the noise added at time T.

2. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 1, characterized in that: The channels selected for the EEG data are O1, Oz, O2, PO7, PO3, POz, PO4, PO8, P7, P5, P3, P1, Pz, P2, P4, P6, and P8 near the occipital and parietal cortices. The preprocessing step is to first remove DC noise with a 1 Hz high-pass filter and then normalize it using a whitening transformation to obtain the preprocessed EEG data.

3. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 1 or 2, characterized in that: The EEG feature extraction network includes a three-layer long short-term memory network and a linear fully connected layer. The training process of the EEG feature extraction network is as follows: The preprocessed EEG data is fed into the EEG feature extraction network to obtain EEG data features, and then the Triplet-Loss loss of the EEG data features is calculated. The formula is: ； Among them, x a Represents the current data, x p Indicates data with the same label as the current data, x n Indicates data with different labels from the current data, f θ are the learnable neural network parameters, i.e., the parameters of the feature extraction network, and β is the margin term.

4. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 3, characterized in that: After training the EEG feature extraction network, the test set is input and the K-Means clustering accuracy is calculated using the true labels of the images and the corresponding EEG extracted features. If the K-Means clustering accuracy is greater than 0.9, the training is completed.

5. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 1, characterized in that: The image coding regression model, image embedding regression model and text embedding regression model are all ridge regression models. The weight coefficients of the image coding regression model, image embedding regression model and text embedding regression model are determined by the following method: T101, use the EEG feature extraction network to extract EEG features and standardize them to make their mean 0 and standard deviation 1; T102. Use a deep variational autoencoder to extract features from the images in the training data of the training set to obtain image coding features; use the ridge regression method to perform regression training on the EEG features and image coding features, and save the weights to obtain the weight coefficients of the image coding regression model; T103. Use the CLIP model to convert the text in the training data of the training set into CLIP text feature data; use the ridge regression method to perform regression training on the EEG features and CLIP text features, and save the weights to obtain the weight coefficients of the text embedding regression model; T104. Use the CLIP model to convert the images in the training data of the training set into CLIP image feature data; use the ridge regression method to perform regression training on the EEG features and CLIP image features, and save the weights to obtain the weight coefficients of the image embedding regression model.

6. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 5, characterized in that: The deep variational autoencoder consists of two parts: an encoder and a decoder. The encoder encodes the input image into feature maps of different resolutions after multiple downsampling, and feeds them into the subsequent decoder in the form of a dictionary. The key of the dictionary is the resolution, and the value is the feature map corresponding to the resolution. The structure of the encoder block is as follows: the first layer starts with a two-dimensional convolutional layer with a 3*3 convolution kernel, 3 input channels, 512 output channels, and a padding and stride of 1 to perform convolution operations on the input image. Subsequently, it passes through several encoder blocks. Each encoder block consists of four layers of two-dimensional convolutional layer networks. The convolution kernels of the four layers are 1*1, 3*3 or 1*1, 3*3 or 1*1, 1*1. The convolution kernel size of the two middle layers depends on the pixels of the input feature map. If the pixel is greater than 2, it is 3*3, otherwise it is 1*1. The input channels of the four layers are 512, 128, 128, and 128 respectively, and the output channels are 128, 128, 128, and 512 respectively. The activation function is GELU. If the encoder block needs to be downsampled, it will be subjected to two-dimensional average pooling according to the downsampling rate to obtain the downsampled feature map. Based on the aforementioned encoder block structure, the encoder structure is as follows: the resolution of the first to eleventh encoder blocks is 64, and no downsampling is performed; the resolution of the twelfth encoder block is 64, and the downsampling rate is 2; the resolution of the thirteenth to thirty-second encoder blocks is 32, and no downsampling is performed; the resolution of the thirty-third encoder block is 32, and the downsampling rate is 2; the resolution of the thirty-fourth to forty-second encoder blocks is 16, and no downsampling is performed; the resolution of the forty-third encoder block is 16, and the downsampling rate is 2; the resolution of the forty-fourth to fifty-first encoder blocks is 8, and no downsampling is performed; the resolution of the fifty-second encoder block is 8, and the downsampling rate is 2; the resolution of the fifty-third to fifty-ninth encoder blocks is 4, and no downsampling is performed; the resolution of the sixtieth encoder block is 4, and the downsampling rate is 4; the resolution of the sixtieth to sixty-sixth encoder blocks is 1, and no downsampling is performed; and finally the encoder outputs a dictionary, the key of which is the image resolution, and the value is the feature map corresponding to the resolution.

7. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 6, characterized in that: The decoder is responsible for performing a series of feature fusion and upsampling operations on the feature maps at different resolutions output by the encoder to gradually restore the image resolution and output an image feature map containing two low-level information: color and contour of the original image. The decoder consists of several decoder blocks, each of which contains three submodules. The first submodule, Block1, is used to extract the mean and variance of the latent variables from the encoder's feature map. It inputs the concatenation of the current resolution feature map and the encoder feature map, and outputs the mean and variance of the latent variables. The second submodule, Block2, is used to calculate the mean and variance of the prior distribution. It inputs the current resolution feature map and outputs the prior mean, prior variance, and a residual feature of the latent variables. The third submodule Block3 is a residual module, which is used to further process the feature map; The decoder structure is as follows: the first and second decoder blocks have a resolution of 1 and no resolution fusion is performed; the third decoder block upsamples the feature map of resolution 1 and fuses it with the feature map of resolution 4; the fourth to sixth decoder blocks have a resolution of 4 and no resolution fusion is performed; the seventh decoder block upsamples the feature map of resolution 4 and fuses it with the feature map of resolution 8; The resolution of the eighth to fourteenth decoder blocks is 8, and no resolution fusion is performed; The fifteenth one upsamples the feature map of resolution 8 and fuses it with the feature map of resolution 16; The 16th to 30th decoder blocks have a resolution of 16 and no resolution fusion is performed; the 31st decoder block upsamples the feature map of resolution 16 and fuses it with the feature map of resolution 32; The resolution of the 32nd to 62nd decoder blocks is 32, and no resolution fusion is performed; the 63rd decoder block upsamples the feature map of resolution 32 and fuses it with the feature map of resolution 64; the resolution of the 64th to 75th decoder blocks is 64, and no resolution fusion is performed; finally, the image coding features of 64 resolution are obtained through 1*1 two-dimensional convolution.

8. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 7, characterized in that: When training a deep variational autoencoder, a structural similarity metric is used as the loss function: ； Among them, μ x and σ x Represent the mean and variance of the real picture, μ M(x) and σ M(x) They represent the mean and variance of the feature map after encoding by the deep variational autoencoder, respectively. C1 and C2 are constants for stable calculation.

Images