Method for reconstructing image from brain activity based on multi-modal diffusion model

Abstract

The invention discloses a method for reconstructing an image from brain activity based on a multi-modal diffusion model. The method comprises the following steps: A1, preprocessing collected electroencephalogram data; a2, inputting the preprocessed electroencephalogram data into an electroencephalogram feature extraction network to obtain electroencephalogram data features; a3, respectively inputting the electroencephalogram data features into an image coding regression model, an image embedding regression model and a text embedding regression model to obtain an image coding result, a CLIP image embedding vector and a CLIP text embedding vector; and A4, taking the image coding result as image input of the multi-modal diffusion model, taking the CLIP image embedding vector as an image hidden variable, taking the CLIP text embedding vector as a cue word hidden variable, and carrying out de-noising diffusion decoding to obtain a reconstructed image. According to the method, the pictures which are relatively high in recognizability and relatively clear can be generated from the electroencephalogram signals.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface and artificial intelligence, and in particular to a method for reconstructing images from brain activities based on a multimodal diffusion model. Background Art

[0002] Vision is one of the main ways for humans to obtain information from the outside world. The vast majority of information obtained by humans comes from vision, which is closely related to human perception, emotions, and memory. At the same time, visual stimulation cannot be perceived and produce meaningful information without corresponding brain activities. These brain activities can generate electrical signals, which are collected through corresponding brain-computer interface devices, and with the help of some deep learning models, tasks such as classification and image reconstruction can be completed.

[0003] Early image generation algorithms such as GAN had problems such as difficulty in training, large amounts of data required, and mode collapse. The quality and diversity of reconstructed images were difficult to guarantee. At the same time, the cost of collecting large amounts of EEG data was high, making GAN training even more difficult. Summary of the invention

[0004] The present invention mainly solves the technical problems existing in the prior art such as the difficulty of image reconstruction and the high data volume requirement, and provides a method for reconstructing images from brain activities based on a multimodal diffusion model.

[0005] The present invention aims to solve the above technical problems mainly through the following technical solutions: a method for reconstructing images from brain activities based on a multimodal diffusion model, comprising the following steps: A1. Preprocess the collected EEG data; A2, input the preprocessed EEG data into the EEG feature extraction network to obtain EEG data features; A3. Input the EEG data features into the image coding regression model, image embedding regression model and text embedding regression model respectively to obtain the image coding result, CLIP image embedding vector (CLIP image embedding) and CLIP text embedding vector (CLIP text embedding); A4. Use the image encoding result as the image input of the multimodal diffusion model, the CLIP image embedding vector as the image latent variable input of the multimodal diffusion model, and the CLIP text embedding vector as the prompt word latent variable input of the multimodal diffusion model, perform denoising diffusion decoding, and obtain a reconstructed image; The multimodal diffusion model includes an image encoder, an image decoder, a plurality of residual blocks, a plurality of fully connected residual blocks, a plurality of image cross-attention layers and a plurality of text cross-attention layers; the input of the image encoder is the image encoding result, which is output to the first residual block, the output of the first residual block enters the first image cross-attention layer and the first text cross-attention layer, the output of the first image cross-attention layer and the first text cross-attention layer enters the second residual block, and the output of the first text cross-attention layer also enters the first fully connected residual block, the output of the first fully connected residual block and the second residual block enters the second text cross-attention layer, and the output of the second residual block also enters the second image cross Attention layer, the outputs of the second image cross-attention layer and the second text cross-attention layer enter the third residual block, and the output of the second text cross-attention layer also enters the second fully connected residual block, the outputs of the second fully connected residual block and the third residual block enter the third text cross-attention layer, and the output of the third residual block also enters the third image cross-attention layer, the outputs of the third image cross-attention layer and the third text cross-attention layer enter the fourth residual block, the output of the fourth residual block enters the image decoder, and the image decoder outputs the final reconstructed image; all image cross-attention layers input the CLIP image embedding vector, and all text cross-attention layers input the CLIP text embedding vector.

[0006] The model combines the combination of two input modalities, text and image, and realizes cross-modal information sharing through a shared global layer and exchangeable data layer and context layer. The global layer is shared between all modalities and tasks and is responsible for processing common features. The data layer is responsible for processing data of a specific modality (such as images or text). The context layer is responsible for processing contextual information of the input modality (such as text embedding or image embedding). Through this design, the multimodal diffusion model can share information between different tasks while maintaining the ability to process specific tasks. The multimodal diffusion model uses a cross-attention layer as the context layer, and its structure is similar to the traditional LatentDiffusion: Attention(Q,K,V)=softmax(QK T / d 0.5 )×V; Q = W Q (i) ×φ i (z t ), K = W K (i) ×τ θ (y), V = W V (i) ×τ θ (y); Among them, φi (z t ) represents the intermediate result of U-Net at time point t, τ θ is the embedding vector embedding obtained by projecting the EEG signal after encoding, W Q (i) , W K (i) and W V (i) It is a learnable parameter matrix; secondly, a fully connected residual block is used to convert the original 768-dimensional latent variables output by the text cross-attention layer into 320*4 features as the text data layer, and the latent variables output by the image cross-attention layer are used as the image data layer through residual connection, which helps to reduce the difference between the text and image embedding spaces, thereby accelerating convergence and improving performance.

[0007] A multimodal diffusion model is used for image reconstruction. The input of the diffusion model includes both the image encoding result of the deep variational autoencoder as the base map representing low-level information such as color and contour as the image modality, and the text embedding vector corresponding to the text input as the text modality. It combines the advantages of the two diffusion workflows of text generation and image rewriting to achieve better reconstruction effects.

[0008] Preferably, the objective function during training of the multimodal diffusion model is: ; Among them, x0 is the original image, q is the image after adding noise in the intermediate process, p is the noise added in each forward diffusion; E represents the expectation, which is used to calculate the average value of the random variable, θ represents the learnable model parameter, and T represents the Tth moment of adding noise, that is, x 1:T represents the joint probability distribution of adding noise from the first time to the time T.

[0009] Preferably, the channels selected for the EEG data are O1, Oz, O2, PO7, PO3, POz, PO4, PO8, P7, P5, P3, P1, Pz, P2, P4, P6 and P8 near the occipital and parietal cortices; the preprocessing is to first remove the DC noise with a 1Hz high-pass filter, and then use whitening transformation for normalization, that is, to obtain the preprocessed EEG data. The processed EEG data can reduce the weight of the channel with high noise content and increase the signal-to-noise ratio of the EEG signal.

[0010] Preferably, the EEG feature extraction network includes a three-layer long short-term memory network and a linear fully connected layer, and the training process of the EEG feature extraction network is: The preprocessed EEG data is sent to the EEG feature extraction network to obtain the EEG data features, and then the Triplet-Loss loss of the EEG data features is calculated. The formula is: ; Among them, x a Represents the current data, x p represents data with the same label as the current data (positive sample), x n represents data with different labels from the current data (negative samples), f θ is the learnable neural network parameter, that is, the parameter of the feature extraction network, β is the margin term, which is used to control the distance between positive and negative samples; training is completed by back-propagating the gradient.

[0011] Preferably, after training the EEG feature extraction network, a test set is input and the K-Means clustering accuracy is calculated using the true labels of the images and the features extracted from the corresponding EEG. If the K-Means clustering accuracy is greater than 0.9, the training is completed.

[0012] Preferably, the image coding regression model, the image embedding regression model and the text embedding regression model are all ridge regression models, and the weight coefficients of the image coding regression model, the image embedding regression model and the text embedding regression model are determined by the following method: T101, use the EEG feature extraction network to extract EEG features and standardize them to make their mean value 0 and standard deviation 1; T102. Use a deep variational autoencoder to extract features from images in the training data of the training set to obtain image coding features; use the ridge regression method to perform regression training on EEG features and image coding features, and save the weights to obtain the weight coefficients of the image coding regression model; T103. Use the CLIP model to convert the text in the training data of the training set into CLIP text feature data; use the ridge regression method to perform regression training on the EEG features and CLIP text features, and save the weights to obtain the weight coefficients of the text embedding regression model; the CLIP (Contrastive Language-Image Pre-Training) model uses the pre-trained existing model; T104. Use the CLIP model to convert the images in the training data of the training set into CLIP image feature data; use the ridge regression method to perform regression training on the EEG features and CLIP image features, and save the weights to obtain the weight coefficients of the image embedding regression model.

[0013] Extracting the embedding vector corresponding to the image from the CLIP image is as follows: first convert the image to 512*512 resolution, then convert it to a tensor floating point array, then map the pixel value from the [0,1] interval to [-1,1], and then call the clip_encode_vision method of the Transformer library; With three sets of weights, the EEG features can be converted into the required image encoding results, CLIP image embedding vectors, and CLIP text embedding vectors through the ridge regression model. The Euclidean distance and Pearson correlation coefficient between the predicted results and the test set can be calculated to measure the regression effect.

[0014] The training set includes EEG data, corresponding real pictures (used to obtain real CLIP image features and deep variational autoencoder encoding feature maps), and corresponding texts (used to obtain real CLIP text features).

[0015] Preferably, the deep variational autoencoder comprises two parts, an encoder and a decoder. The encoder encodes the input image into feature maps of different resolutions after multiple downsampling, and sends them to the subsequent decoder in the form of a dictionary, where the key of the dictionary is the resolution and the value is the feature map corresponding to the resolution. The structure of the encoder block is as follows: the first layer starts with a two-dimensional convolutional layer with a 3*3 convolution kernel, 3 input channels, 512 output channels, and a padding and stride of 1 to perform convolution operations on the input image. It then passes through several encoder blocks. Each encoder block consists of four layers of two-dimensional convolutional layer networks. The convolution kernels of the four layers are 1*1, 3*3 or 1*1, 3*3 or 1*1, 1*1 respectively. The convolution kernel size of the middle two layers depends on the pixels of the input feature map. If the pixel is greater than 2, it is 3*3, otherwise it is 1*1. The number of input channels of the four-layer network is The numbers of the output channels are 128, 128, 128, and 512, respectively. The activation functions are all GELU. For the encoder, residual connections are also used to alleviate the gradient vanishing problem. When passing parameters to the encoder block, a residual=True is passed, indicating that the residual connection of the encoder block is enabled. The encoder block then adds the input of the encoder block to the result of the 1*1 convolution. If the encoder block needs to be downsampled, it is finally subjected to two-dimensional average pooling according to the downsampling rate to obtain the downsampled feature map. Based on the aforementioned encoder block structure, the encoder structure is: the resolution of the first to eleventh encoder blocks is 64, and no downsampling is performed; the resolution of the twelfth encoder block is 64, and the downsampling rate is 2; the resolution of the thirteenth to thirty-second encoder blocks is 32, and no downsampling is performed; the resolution of the thirty-third encoder block is 32, and the downsampling rate is 2; the resolution of the thirty-fourth to forty-second encoder blocks is 16, and no downsampling is performed; the resolution of the forty-third encoder block is 16, and the downsampling rate is 2; the resolution of the forty-fourth to fifty-first encoder blocks is 8, and no downsampling is performed; the resolution of the fifty-second encoder block is 8, and the downsampling rate is 2; the resolution of the fifty-third to fifty-ninth encoder blocks is 4, and no downsampling is performed; the resolution of the sixtieth encoder block is 4, and the downsampling rate is 4; the resolution of the sixty-first to sixty-sixth encoder blocks is 1, and no downsampling is performed; and finally the encoder outputs a dictionary, the key of which is the image resolution, and the value is the feature map corresponding to the resolution.

[0016] Preferably, the decoder is responsible for performing a series of feature fusion and upsampling operations on the feature maps at different resolutions output by the encoder to gradually restore the resolution of the image, and output an image feature map, which contains two low-level information: color and contour of the original image; the decoder is composed of a number of decoder blocks, and the decoder block contains three sub-modules. The first sub-module Block1 is used to extract the mean and variance of the latent variables from the feature map of the encoder, inputs the concatenation of the feature map of the current resolution and the feature map of the encoder, and outputs the mean and variance of the latent variables; the second sub-module Block2 is used to calculate the mean and variance of the prior distribution, inputs the feature map of the current resolution, and outputs the prior mean, prior variance and a residual feature of the latent variables; the third sub-module Block3 is a residual module, which is used to further process the feature map.

[0017] The specific decoder structure is as follows: the resolution of the first and second decoder blocks is 1, and no resolution fusion is performed; the third decoder block upsamples the feature map of resolution 1 and fuses it with the feature map of resolution 4; the resolution of the fourth to sixth decoder blocks is 4, and no resolution fusion is performed; the seventh decoder block upsamples the feature map of resolution 4 and fuses it with the feature map of resolution 8; the resolution of the eighth to fourteenth decoder blocks is 8, and no resolution fusion is performed; the fifteenth upsamples the feature map of resolution 8 and fuses it with the feature map of resolution 16; the resolution of the sixteenth to thirtieth decoder blocks is 16, and no resolution fusion is performed; the thirty-first decoder block upsamples the feature map of resolution 16 and fuses it with the feature map of resolution 32; the resolution of the thirty-second to sixty-second decoder blocks is 32, and no resolution fusion is performed; the sixty-third decoder block upsamples the feature map of resolution 32 and fuses it with the feature map of resolution 64; the resolution of the sixty-fourth to seventy-fifth decoder blocks is 64, and no resolution fusion is performed; finally, the image coding features of 64 resolution are obtained through 1*1 two-dimensional convolution.

[0018] By encoding the image using a deep variational autoencoder, we can obtain the underlying features of the image's contour and color.

[0019] Preferably, when training a deep variational autoencoder, in order to make the feature map obtained by image encoding as similar as possible to the original image, a structural similarity metric is used as the loss function: ; Among them, μ x and σ x Represent the mean and variance of the real picture, μ M(x) and σ M(x) They represent the mean and variance of the feature map after being encoded by the deep variational autoencoder, respectively. C1 and C2 are constants for stable calculation. C1 and C2 are empirical values, and C1=6.5025 and C2=58.5225 can generally be taken.

[0020] The substantial effects brought by the present invention are: selecting the EEG channel related to the visual function brain area; establishing a long short-term memory network to extract features, adopting the contrastive learning method for training, selecting Triplet-Loss as the loss function, so that the EEG data corresponding to different samples can be distinguished, and measuring the feature extraction effect through K-Means clustering; encoding the target image through a deep variational autoencoder to extract the underlying features such as color and contour, and using the feature as the image modality input of the multimodal diffusion model; with the help of the multimodal diffusion model and CLIP, the image stimulus and the corresponding text information are converted into vectors, avoiding the problem of traditional generative models such as GAN requiring a large amount of training data and the problem of difficulty in training, and at the same time providing more semantic information than the original diffusion model that only uses CLIP to embed vector input; in the alignment of EEG features with the intermediate parameters of the multimodal diffusion model, the use of linear regression method can also achieve better results and further reduce the calculation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of a method for reconstructing images from brain activities based on a multimodal diffusion model of the present invention; Figure 2 It is a schematic diagram of the structure of a multi-modal diffusion model of the present invention; Figure 3 is a schematic diagram of an encoder block of a deep variational autoencoder of the present invention; Figure 4 It is a decoder block schematic diagram of a decoder of a deep variational autoencoder of the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0023] Embodiment: A method for reconstructing images from brain activity based on a multimodal diffusion model, such as Figure 1 As shown, the following steps are included: A1. Preprocess the collected EEG data. Preprocessing includes channel selection, intercepting EEG signals in a specific time period, and normalization. The input EEG data size is n_channel*time, and the output preprocessed EEG data is n_selected_channel*time. If it is the training stage, the images in the training data also need to be preprocessed, that is, the image resolution size is unified to 512*512. A2. Input the preprocessed EEG data into the EEG feature extraction network to obtain EEG data features. The size of the input EEG data is time*n_channel after the transposition of the preprocessed EEG signal, and the corresponding output feature is a one-dimensional vector of length 128. A3, input the EEG data features into the image coding regression model, the image embedding regression model and the text embedding regression model respectively, and obtain the image coding result, the CLIP image embedding vector and the CLIP text embedding vector; A4. Use the image encoding result as the image input of the multimodal diffusion model, the CLIP image embedding vector as the image latent variable input of the multimodal diffusion model, and the CLIP text embedding vector as the prompt word latent variable input of the multimodal diffusion model, perform denoising diffusion decoding, and obtain a reconstructed image; like Figure 2 As shown, the multimodal diffusion model includes an image encoder, an image decoder, several residual blocks, several fully connected residual blocks, several image cross-attention layers and several text cross-attention layers. The input of the image encoder is the image encoding result, which is output to the first residual block. The output of the first residual block enters the first image cross-attention layer and the first text cross-attention layer. The output of the first image cross-attention layer and the first text cross-attention layer enters the second residual block. At the same time, the output of the first text cross-attention layer also enters the first fully connected residual block. The output of the first fully connected residual block and the second residual block enters the second text cross-attention layer. At the same time, the output of the second residual block also enters the second image cross-attention layer. The output of the second image cross-attention layer and the second text cross-attention layer enters the third residual block. At the same time, the output of the second text cross-attention layer also enters the second fully connected residual block. The output of the second fully connected residual block and the third residual block enters the third text cross-attention layer. At the same time, the output of the third residual block also enters the third image cross-attention layer. The output of the third image cross-attention layer and the third text cross-attention layer enters the fourth residual block. The output of the fourth residual block enters the image decoder, and the image decoder outputs the final reconstructed image; all image cross-attention layers input the CLIP image embedding vector, and all text cross-attention layers input the CLIP text embedding vector.

[0024] The light blue module in the figure is the data layer, the dark blue module is the context layer, and the weight of the white module is fixed and does not participate in the gradient update during training. The image encoder is the AutoKL image encoder, and the image decoder is the AutoKL image decoder. This model combines the combination of two input modalities, text and image, and realizes cross-modal information sharing through a shared global layer and exchangeable data layer and context layer. Among them, the global layer is shared among all modalities and tasks and is responsible for processing common features. The data layer is responsible for processing data of a specific modality (such as images or text), and the context layer is responsible for processing contextual information of the input modality (such as text embedding or image embedding). Through this design, the multimodal diffusion model can share information between different tasks while maintaining the ability to process specific tasks; The multimodal diffusion model uses a cross-attention layer as the context layer, and its structure is similar to the traditional LatentDiffusion: Attention(Q,K,V)=softmax(QK T / d 0.5 )×V; Q = W Q (i) ×φ i (z t ), K = W K (i) ×τ θ (y), V = W V (i) ×τ θ (y); Among them, φ i (z t ) represents the intermediate result of U-Net at time point t, τ θ is the embedding vector embedding obtained by projecting the EEG signal after encoding, W Q (i) , W K (i) and W V (i) It is a learnable parameter matrix; secondly, a fully connected residual block is used to convert the original 768-dimensional latent variables output by the text cross-attention layer into 320*4 features as the text data layer, and the latent variables output by the image cross-attention layer are used as the image data layer through residual connection, which helps to reduce the difference between the text and image embedding spaces, thereby accelerating convergence and improving performance.

[0025] The objective function when training the multimodal diffusion model is: ; Among them, x0 is the original image, q is the image after adding noise in the intermediate process, p is the noise added in each forward diffusion; E represents the expectation, which is used to calculate the average value of the random variable, θ represents the learnable model parameter, and T represents the Tth moment of adding noise, that is, x 1:T represents the joint probability distribution of adding noise from the first time to the time T, x 0:T Represents the joint probability distribution of adding noise from the original data to time T.

[0026] The channels selected for the EEG data are O1, Oz, O2, PO7, PO3, POz, PO4, PO8, P7, P5, P3, P1, Pz, P2, P4, P6, and P8 near the occipital and parietal cortices. The preprocessing is to first remove the DC noise with a 1 Hz high-pass filter and then normalize it using a whitening transformation to obtain the preprocessed EEG data.

[0027] The EEG feature extraction network includes three layers of long short-term memory network and linear fully connected layer. The training process of the EEG feature extraction network is as follows: The preprocessed EEG data is sent to the EEG feature extraction network to obtain the EEG data features, and then the Triplet-Loss loss L of the EEG data features is calculated. triplet , the formula is: ; Among them, x a Represents the current data, x p represents data with the same label as the current data (positive sample), x n represents data with different labels from the current data (negative samples), f θ is the learnable neural network parameter, that is, the parameter of the feature extraction network, β is the margin term, which is used to control the distance between positive and negative samples; training is completed by back-propagating the gradient.

[0028] After training the EEG feature extraction network, input the test set and calculate the K-Means clustering accuracy using the true label of the image and the corresponding EEG extracted features. If the K-Means clustering accuracy is greater than 0.9, the training is completed.

[0029] The image coding regression model, image embedding regression model and text embedding regression model are all ridge regression models. The weight coefficients of the image coding regression model, image embedding regression model and text embedding regression model are determined by the following method: T101, use the EEG feature extraction network to extract EEG features and standardize them to make their mean value 0 and standard deviation 1; T102. Use a deep variational autoencoder to extract features from images in the training data of the training set to obtain image coding features; use the ridge regression method to perform regression training on EEG features and image coding features, and save the weights to obtain the weight coefficients of the image coding regression model; T103. Use the CLIP model to convert the text in the training data of the training set into CLIP text feature data; use the ridge regression method to perform regression training on the EEG features and CLIP text features, and save the weights to obtain the weight coefficients of the text embedding regression model; the CLIP model uses a pre-trained existing model; T104. Use the CLIP model to convert the images in the training data of the training set into CLIP image feature data; use the ridge regression method to perform regression training on the EEG features and CLIP image features, and save the weights to obtain the weight coefficients of the image embedding regression model.

[0030] Extracting the embedding vector corresponding to the image from the CLIP image is as follows: first convert the image to 512*512 resolution, then convert it to a tensor floating point array, then map the pixel value from the [0,1] interval to [-1,1], and then call the clip_encode_vision method of the Transformer library; With three sets of weights, the EEG features can be converted into the required image encoding results, CLIP image embedding vectors, and CLIP text embedding vectors through the ridge regression model. The Euclidean distance and Pearson correlation coefficient between the predicted results and the test set can be calculated to measure the regression effect.

[0031] The deep variational autoencoder consists of two parts: the encoder and the decoder. The encoder encodes the input image into feature maps of different resolutions after multiple downsampling, and sends them to the subsequent decoder in the form of a dictionary. The key of the dictionary is the resolution, and the value is the feature map corresponding to the resolution. like Figure 3 As shown in the figure, the structure of the encoder block is as follows: the first layer starts with a two-dimensional convolutional layer with a 3*3 convolution kernel, 3 input channels, 512 output channels, and a padding and stride of 1 to perform convolution operations on the input image, and then passes through several encoder blocks. Each encoder block consists of four layers of two-dimensional convolutional layer networks. The convolution kernels of the four layers are 1*1, 3*3 or 1*1, 3*3 or 1*1, 1*1 respectively. The convolution kernel size of the middle two layers depends on the pixels of the input feature map. If the pixel is greater than 2, it is 3*3, otherwise it is 1*1. The input channel of the four-layer network is The number of channels is 512, 128, 128, 128, respectively, and the number of output channels is 128, 128, 128, 512, respectively, and the activation function is GELU; for the encoder, residual connection is also used to alleviate the gradient vanishing problem, that is, when passing parameters to the encoder block, a residual=True is passed, indicating that the residual connection of the encoder block is enabled. The encoder block then adds the input of the encoder block to the result of the 1*1 convolution; if the encoder block needs to be downsampled, it is finally subjected to two-dimensional average pooling according to the downsampling rate to obtain the downsampled feature map; Based on the aforementioned encoder block structure, the encoder structure is: the resolution of the first to eleventh encoder blocks is 64, and no downsampling is performed; the resolution of the twelfth encoder block is 64, and the downsampling rate is 2; the resolution of the thirteenth to thirty-second encoder blocks is 32, and no downsampling is performed; the resolution of the thirty-third encoder block is 32, and the downsampling rate is 2; the resolution of the thirty-fourth to forty-second encoder blocks is 16, and no downsampling is performed; the resolution of the forty-third encoder block is 16, and the downsampling rate is 2; the resolution of the forty-fourth to fifty-first encoder blocks is 8, and no downsampling is performed; the resolution of the fifty-second encoder block is 8, and the downsampling rate is 2; the resolution of the fifty-third to fifty-ninth encoder blocks is 4, and no downsampling is performed; the resolution of the sixtieth encoder block is 4, and the downsampling rate is 4; the resolution of the sixty-first to sixty-sixth encoder blocks is 1, and no downsampling is performed; and finally the encoder outputs a dictionary, the key of which is the image resolution, and the value is the feature map corresponding to the resolution.

[0032] The decoder is responsible for performing a series of feature fusion and upsampling operations on the feature maps at different resolutions output by the encoder to gradually restore the resolution of the image, and output the image feature map, which contains the low-level information such as color and contour of the original image; the decoder consists of several decoder blocks, such as Figure 4 As shown in the figure, the decoder block mainly includes three sub-modules. The first sub-module Block1 is used to extract the mean and variance of the latent variables from the feature map of the encoder. It inputs the concatenation of the current resolution feature map and the encoder feature map, and outputs the mean and variance of the latent variables. The second sub-module Block2 is used to calculate the mean and variance of the prior distribution. It inputs the current resolution feature map and outputs the prior mean, prior variance and a residual feature of the latent variable. The third sub-module Block3 is a residual module, which is used to further process the feature map.

[0033] The specific decoder structure is as follows: the resolution of the first and second decoder blocks is 1, and no resolution fusion is performed; the third decoder block upsamples the feature map of resolution 1 and fuses it with the feature map of resolution 4; the resolution of the fourth to sixth decoder blocks is 4, and no resolution fusion is performed; the seventh decoder block upsamples the feature map of resolution 4 and fuses it with the feature map of resolution 8; the resolution of the eighth to fourteenth decoder blocks is 8, and no resolution fusion is performed; the fifteenth upsamples the feature map of resolution 8 and fuses it with the feature map of resolution 16; the resolution of the sixteenth to thirtieth decoder blocks is 16, and no resolution fusion is performed; the thirty-first decoder block upsamples the feature map of resolution 16 and fuses it with the feature map of resolution 32; the resolution of the thirty-second to sixty-second decoder blocks is 32, and no resolution fusion is performed; the sixty-third decoder block upsamples the feature map of resolution 32 and fuses it with the feature map of resolution 64; the resolution of the sixty-fourth to seventy-fifth decoder blocks is 64, and no resolution fusion is performed; finally, the image coding features of 64 resolution are obtained through 1*1 two-dimensional convolution.

[0034] When training a deep variational autoencoder, in order to make the feature map obtained by image encoding as similar as possible to the original image, a structural similarity metric is used as the loss function: ; Among them, μ x and σ x Represent the mean and variance of the real picture, μ M(x) and σ M(x) They represent the mean and variance of the feature map after being encoded by the deep variational autoencoder, respectively. C1 and C2 are constants for stable calculation. C1 and C2 are empirical values, and C1=6.5025 and C2=58.5225 can generally be taken.

[0035] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0036] Although the terminology such as multimodal diffusion model and regression model is used more frequently in this paper, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.

Claims

1. A method for reconstructing images from brain activity based on a multimodal diffusion model, characterized in that: The following steps are involved: A1. Preprocess the collected EEG data; A2, input the preprocessed EEG data into the EEG feature extraction network to obtain EEG data features; A3, input the EEG data features into the image coding regression model, the image embedding regression model and the text embedding regression model respectively, and obtain the image coding result, the CLIP image embedding vector and the CLIP text embedding vector; A4. Use the image encoding result as the image input of the multimodal diffusion model, the CLIP image embedding vector as the image latent variable input of the multimodal diffusion model, and the CLIP text embedding vector as the prompt word latent variable input of the multimodal diffusion model, perform denoising diffusion decoding, and obtain a reconstructed image; The multimodal diffusion model includes an image encoder, an image decoder, a plurality of residual blocks, a plurality of fully connected residual blocks, a plurality of image cross-attention layers and a plurality of text cross-attention layers; the input of the image encoder is the image encoding result, which is output to the first residual block, the output of the first residual block enters the first image cross-attention layer and the first text cross-attention layer, the output of the first image cross-attention layer and the first text cross-attention layer enters the second residual block, and the output of the first text cross-attention layer also enters the first fully connected residual block, the output of the first fully connected residual block and the second residual block enters the second text cross-attention layer, and the output of the second residual block also enters the second image cross Attention layer, the outputs of the second image cross-attention layer and the second text cross-attention layer enter the third residual block, and the output of the second text cross-attention layer also enters the second fully connected residual block, the outputs of the second fully connected residual block and the third residual block enter the third text cross-attention layer, and the output of the third residual block also enters the third image cross-attention layer, the outputs of the third image cross-attention layer and the third text cross-attention layer enter the fourth residual block, the output of the fourth residual block enters the image decoder, and the image decoder outputs the final reconstructed image; all image cross-attention layers input the CLIP image embedding vector, and all text cross-attention layers input the CLIP text embedding vector.

2. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 1, characterized in that: The objective function of the multimodal diffusion model training is: ； Among them, x0 is the original image, q is the image after adding noise in the intermediate process, and p is the noise added in each forward diffusion. The hollow E represents the expectation, which is used to calculate the average value of the random variable. θ represents the learnable model parameter. T represents the moment of adding noise, that is, x 1:T represents the joint probability distribution of adding noise from the first time to the time T, x 0:T Represents the joint probability distribution of adding noise from the original data to time T.

3. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 1, characterized in that: The channels selected for the EEG data are O1, Oz, O2, PO7, PO3, POz, PO4, PO8, P7, P5, P3, P1, Pz, P2, P4, P6, and P8 near the occipital and parietal cortices. The preprocessing is to first remove the DC noise with a 1 Hz high-pass filter and then normalize it using a whitening transformation to obtain the preprocessed EEG data.

4. A method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 1 or 3, characterized in that: The EEG feature extraction network includes a three-layer long short-term memory network and a linear fully connected layer. The training process of the EEG feature extraction network is as follows: The preprocessed EEG data is sent to the EEG feature extraction network to obtain the EEG data features, and then the Triplet-Loss loss of the EEG data features is calculated. The formula is: ； Among them, x a Represents the current data, x p Indicates data with the same label as the current data, x n Indicates data with different labels from the current data, f θ is the learnable neural network parameter, that is, the parameter of the feature extraction network, and β is the margin term.

5. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 4, characterized in that: After training the EEG feature extraction network, input the test set and calculate the K-Means clustering accuracy using the true label of the image and the corresponding EEG extracted features. If the K-Means clustering accuracy is greater than 0.9, the training is completed.

6. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 1, characterized in that: The image coding regression model, image embedding regression model and text embedding regression model are all ridge regression models. The weight coefficients of the image coding regression model, image embedding regression model and text embedding regression model are determined by the following method: T101, use the EEG feature extraction network to extract EEG features and standardize them to make their mean value 0 and standard deviation 1; T102. Use a deep variational autoencoder to extract features from images in the training data of the training set to obtain image coding features; use the ridge regression method to perform regression training on EEG features and image coding features, and save the weights to obtain the weight coefficients of the image coding regression model; T103. Use the CLIP model to convert the text in the training data of the training set into CLIP text feature data; use the ridge regression method to perform regression training on the EEG features and CLIP text features, and save the weights to obtain the weight coefficients of the text embedding regression model; T104. Use the CLIP model to convert the images in the training data of the training set into CLIP image feature data; use the ridge regression method to perform regression training on the EEG features and CLIP image features, and save the weights to obtain the weight coefficients of the image embedding regression model.

7. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 6, characterized in that: The deep variational autoencoder consists of two parts: the encoder and the decoder. The encoder encodes the input image into feature maps of different resolutions after multiple downsampling, and sends them to the subsequent decoder in the form of a dictionary. The key of the dictionary is the resolution, and the value is the feature map corresponding to the resolution. The structure of the encoder block is as follows: the first layer starts with a two-dimensional convolutional layer with a 3*3 convolution kernel, 3 input channels, 512 output channels, and a padding and stride of 1 to perform convolution operations on the input image, and then passes through several encoder blocks. Each encoder block consists of four layers of two-dimensional convolutional layer networks. The convolution kernels of the four layers are 1*1, 3*3 or 1*1, 3*3 or 1*1, 1*1 respectively. The convolution kernel size of the two middle layers depends on the pixels of the input feature map. If the pixel is greater than 2, it is 3*3, otherwise it is 1*1. The input channels of the four layers are 512, 128, 128, 128 respectively, and the output channels are 128, 128, 128, 512 respectively. The activation function is GELU; if the encoder block needs to be downsampled, it will be subjected to two-dimensional average pooling according to the downsampling rate to obtain the downsampled feature map; Based on the aforementioned encoder block structure, the encoder structure is: the resolution of the first to eleventh encoder blocks is 64, and no downsampling is performed; the resolution of the twelfth encoder block is 64, and the downsampling rate is 2; the resolution of the thirteenth to thirty-second encoder blocks is 32, and no downsampling is performed; the resolution of the thirty-third encoder block is 32, and the downsampling rate is 2; the resolution of the thirty-fourth to forty-second encoder blocks is 16, and no downsampling is performed; the resolution of the forty-third encoder block is 16, and the downsampling rate is 2; the resolution of the forty-fourth to fifty-first encoder blocks is 8, and no downsampling is performed; the resolution of the fifty-second encoder block is 8, and the downsampling rate is 2; the resolution of the fifty-third to fifty-ninth encoder blocks is 4, and no downsampling is performed; the resolution of the sixtieth encoder block is 4, and the downsampling rate is 4; the resolution of the sixty-first to sixty-sixth encoder blocks is 1, and no downsampling is performed; and finally the encoder outputs a dictionary, the key of which is the image resolution, and the value is the feature map corresponding to the resolution.

8. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 7, characterized in that: The decoder is responsible for performing a series of feature fusion and upsampling operations on the feature maps at different resolutions output by the encoder to gradually restore the resolution of the image, and output an image feature map containing two low-level information: color and contour of the original image. The decoder consists of several decoder blocks, each of which contains three submodules. The first submodule Block1 is used to extract the mean and variance of the latent variables from the feature map of the encoder. It inputs the concatenation of the current resolution feature map and the encoder feature map, and outputs the mean and variance of the latent variables. The second submodule Block2 is used to calculate the mean and variance of the prior distribution. It inputs the current resolution feature map and outputs the prior mean, prior variance and a residual feature of the latent variables. The third submodule Block3 is a residual module, which is used to further process the feature map; The decoder structure is as follows: the first and second decoder blocks have a resolution of 1, and no resolution fusion is performed; the third decoder block upsamples the feature map of resolution 1 and fuses it with the feature map of resolution 4; the fourth to sixth decoder blocks have a resolution of 4, and no resolution fusion is performed; the seventh decoder block upsamples the feature map of resolution 4 and fuses it with the feature map of resolution 8; The resolution of the eighth to fourteenth decoder blocks is 8, and no resolution fusion is performed; The fifteenth one upsamples the feature map of resolution 8 and merges it with the feature map of resolution 16; The resolution of the 16th to 30th decoder blocks is 16, and no resolution fusion is performed; the 31st decoder block upsamples the feature map of resolution 16 and fuses it with the feature map of resolution 32; The resolution of the 32nd to 62nd decoder blocks is 32, and no resolution fusion is performed; the 63rd decoder block upsamples the feature map of resolution 32 and fuses it with the feature map of resolution 64; the resolution of the 64th to 75th decoder blocks is 64, and no resolution fusion is performed; finally, the image coding features of 64 resolution are obtained through 1*1 two-dimensional convolution.

9. The method for reconstructing images from brain activity based on a multimodal diffusion model according to claim 8, characterized in that: When training a deep variational autoencoder, a structural similarity metric is used as the loss function: ； Among them, μ x and σ x Represent the mean and variance of the real picture, μ M(x) and σ M(x) They respectively represent the mean and variance of the feature map after being encoded by the deep variational autoencoder, and C1 and C2 are constants for stable calculation.

Images