Method for aligning multi-modal fused brain visual neural activity representations

Abstract

The application discloses a multi-modal fusion brain visual nerve activity representation alignment method, and belongs to the cross field of biomedical engineering and pattern recognition. The multi-modal fusion brain visual stimulation representation alignment model is constructed based on functional nuclear magnetic resonance image signal data under the condition of watching a large number of visual perception images, and mainly comprises a visual stimulation semantic fusion module, a VAE model responsible for fusing semantic representation information of a target multi-modal, including images and texts; a brain semantic information encoder, a GRU responsible for extracting semantic information of a main visual cortex of the brain, and the two are aligned to a unified feature space through CLIP type contrast learning. The application solves the alignment ambiguity problem generated when visual activity simultaneously aligns images and texts, is a key to extract semantic enhanced visual features, and provides valuable basis for subsequent brain decoding research by extracting recognition representation with rich semantic information.

Description

Technical Field

[0001] This invention belongs to the fields of biomedical engineering and artificial intelligence technology, and in particular relates to an alignment method for multimodal fusion of brain visual neural activity representations. Background Technology

[0002] The brain can reveal a wide range of information about human cognition, emotion, behavior, and perception. Visual information decoding serves as a method for understanding the brain's performance of visual tasks. Brain visual stimulus representation alignment techniques based on functional magnetic resonance imaging (fMRI) primarily aim to enhance representations at the feature level, thereby extracting more effective semantic representations from brain activity modalities for brain decoding-related tasks and analyses. Initially, most approaches were limited to single-modal (brain activity - visual stimulus image) alignment in the feature space. For example, in 2017, Horikawa et al. proposed a regression method to predict visual features from fMRI data, using brain activity aligned with corresponding visual images. This method not only identifies the categories of seen or imagined objects but also reveals the similarity between human brain vision and machine vision and its potential application in brain information retrieval. Subsequently, more attempts have been made at the feature level, such as introducing the text modality of the stimulus source to induce the model to learn the semantic information of the text in the brain, thereby enhancing the depth of alignment. For example, in 2023, Du et al. proposed BraVL, a multimodal autoencoder variational Bayesian learning framework. It aligns three modalities (brain activity-image-text) using MoPoE (Expert Product Hybrid) and utilizes fMRI signal input by training an SVM classifier on image-text pairs. The results showed that the model using a combination of visual and textual features performed better than the single-modal model, achieving some improvements in results. Similarly, Liu et al.'s later BrainCLIP also used the CLIP (Contrastive Language-Image Pre-training) model to align multimodal representations. This alignment framework achieved a series of improvements in subsequent downstream brain decoding tasks. Even later, Chen et al.'s MindVideo used triples for multimodal alignment, achieving very good results in video decoding tasks related to brain decoding.

[0003] Currently, existing methods for aligning semantic representations of multimodal brain visual stimuli have achieved good results, but an important problem remains: alignment blurring occurs when visual activity simultaneously aligns images and text (see reference). Figure 1 This refers to a problem where, when aligning brain activity in response to visual stimuli with the semantic representation of a target multimodal (image modality, text modality, or other modalities), the alignment is inaccurate or insufficient due to differences in the distribution of the target modality in the feature space. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing a method to align brain activity under visual stimulation with the multimodal semantic representation of images and text from the visual stimulus source. This solves the alignment ambiguity problem in multimodal alignment and improves the alignment effect of semantic information.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for aligning brain visual neural activity representations through multimodal fusion, comprising the following steps;

[0007] Step 1: Use the pre-trained CLIP text encoder to embed the text of the visual stimulus into the CLIP semantic feature space to obtain the text semantic feature representation; use the pre-trained CLIP image encoder to embed the image of the visual stimulus into the CLIP semantic feature space to obtain the image semantic feature representation; that is, obtain the semantic feature representation of the visual stimulus with generalizability based on the CLIP text encoder and CLIP image encoder; wherein, the feature dimension of the text semantic feature representation and the image semantic feature representation are the same.

[0008] Step 2: Use the visual stimulus semantic fusion module to fuse the text semantic feature representation and the image semantic feature representation in the feature space to obtain a multimodal fused semantic information representation.

[0009] Step 3: Use the brain semantic information encoder to convert the response signals of the main visual regions of the specified whole brain into latent feature space vector representations, and add prompt embeddings from the subject information to obtain visual response signal features with semantic information, i.e., brain semantic information representations; enhance the subject generality of the model based on the added prompt embeddings, with the aim of training this encoder to obtain brain representations with semantic information.

[0010] Step 4: Use contrastive learning to align the multimodal fusion semantic information representation from Step 2 with the brain semantic information representation from Step 3 in the feature space.

[0011] Based on the training data set, iterative steps 1-4 train the visual semantic fusion module and the brain semantic information encoder to obtain a visual semantic fusion module and a brain semantic information encoder for aligning the representation of visual neural activity in the brain.

[0012] Furthermore, the present invention also includes step 5, using the trained visual semantic fusion module and the brain semantic information encoder to perform representation category retrieval and classification.

[0013] Furthermore, in step 1, the ViT-L / 14 version of the CLIP image encoder is used, and the CLIP text encoder corresponding to the ViT-L / 14 version is used as the pre-trained CLIP text encoder.

[0014] Furthermore, in step 2, the visual stimulus semantic fusion module uses variational autoencoder (VAE) to fuse text semantic feature representations and image semantic feature representations to obtain a graph-text feature Z (i.e., a multimodal fused semantic information representation) that simultaneously contains image and text semantic information, which can be expressed by the formula:

[0015] Z = VAE(I,T;W) VAE (1)

[0016] Where I and T represent image semantic features and text semantic features, respectively, and W VAE Z represents the weights of all networks in the VAE, and Z represents the multimodal fusion semantic information representation.

[0017] A Variational Autoencoder (VAE) is used to fuse the obtained image and text semantic representations in the latent space. Internally, the VAE trains separate image and text encoders for each modality. Each encoder generates a mean vector μ and a log-variance vector ∑. Using the Mixture of Experts (MoE) model, a general gating network is set up for each image-text pair to dynamically learn the weights for mixing the two modalities. The learned parameters are then linearly mixed with the mean vector μ of the two modalities according to the weights. i With μ t and the variance logarithm vector ∑ i With ∑ t Generate the mixed mean vector μ fusion With the variance logarithm vector ∑ fusion The latent spatial semantic representation Z is generated through VAE reparameterization. fusion Specifically, a random noise ∈ (i.e., ∈ ~ N(0,1)) is sampled from a standard normal distribution, and then this noise is used to calculate Z. fusion The specific calculation formula is as follows:

[0018] Z fusion =μ fusion +∑ fusion ⊙∈ (2)

[0019] Finally, the latent spatial semantic representation Z is obtained through the image decoder and text decoder of VAE. fusion Decoding is performed to reconstruct the image representation and text representation, thereby obtaining the multimodal fusion semantic information representation Z.

[0020] Furthermore, in step 3, the encoding process of the brain's semantic information encoder is specifically as follows:

[0021] In step 3, a gated recurrent unit (GRU) is used to extract visual features from the brain. The specific formula is as follows:

[0022] B = GRU(V1,V2,...,V) T W GRU (3)

[0023] Where B represents visual features; GRU represents gated recurrent units; V1, V2, ..., V T W represents the response signals of visual activity in T main visual regions. GRU Represents the set of parameters of the GRU;

[0024] Step 301: Select the main visual regions in the natural scene dataset. For example, set the main visual regions as 21 visual regions such as: "V1", "MST", "V6", "V2", "V3", "V4", "V8", "V3A", "V7", "IPS1", "FFC", "V3B", "LO1", "LO2";

[0025] The reshape layer is used to unify the response signals of visual activities in different visual regions (referred to as brain region signals) to a specified dimension Num, so as to obtain the response signal feature representation of each major visual region. Num is used to represent the feature dimension of text semantic feature representation and / or image semantic feature representation.

[0026] Then, the response signal features of T main visual regions are spliced ​​together to obtain the original response signal (i.e., the original brain region signal) in T×Num dimensions;

[0027] Step 302: Based on the CLIP text encoder, the source subject information of the original response signal is encoded into a cue embedding, and then the cue embedding is concatenated to the original response signal to obtain (T+1)×Num dimensional visual response signal features; in order to manually enhance the differences between subjects, the model learns the differences between different subjects.

[0028] Step 303: Input the (T+1)×Num dimensional visual response signal features into the GRU neural network, using T+1 as the sequence dimension of the GRU neural network (so as to learn the semantic information of visual stimuli contained in T major visual regions), and take the last 1×Num as the output to obtain a 1×Num visual stimulus semantic representation (brain semantic information representation) of the whole brain visual regions for subsequent alignment.

[0029] Furthermore, step 4 specifically includes:

[0030] Step 401: Pre-train only the VAE from step 3. The loss function L used during VAE pre-training is... vae This includes the VAE reconstruction loss and the KL (Kullback-Leibler divergence) loss, which is the weighted sum of the two. In this invention, the weight of the KL loss can be set to 1, and the corresponding expression is:

[0031]

[0032] The reconstruction loss is:

[0033]

[0034] The KL divergence loss is:

[0035]

[0036] in, p represents the expected value of a mathematical expression. θ (x|z) represents the output of the decoder; p(z) represents the prior distribution of the latent variable z; q φ (z|x) represents the posterior distribution parameterized by the encoder network; J represents the dimension of the latent space z; σ j μ j Let represent the standard deviation and mean of the j-th latent variable, respectively.

[0037] Using loss function L vae The goal is to obtain a variational autoencoder that can extract stable semantic representations;

[0038] Step 402: The 1×Num brain semantic information representation obtained in step 303 is compared with the multimodal fusion semantic information representation obtained in step 2 using a CLIP-style contrastive learning method to perform feature space alignment. The loss function used is the contrastive loss function L. con Its expression is:

[0039]

[0040] Where N represents the total number of samples; n represents the upper limit index; Sim(·) represents the cosine similarity; (A i B i (A) represents a positive sample composed of the multimodal fusion semantic information representation (i.e., image-text features) and the brain semantic information representation (visual features) of the i-th sample; i B j ) represents a negative sample composed of the image and text features of the i-th sample and the visual features of the j-th sample; τ represents the temperature coefficient.

[0041] Step 403: Train VAE and GRU simultaneously. The total loss used during training is set as follows:

[0042] L total =L con +L vae (8)

[0043] That is, the total loss can be set as L. con With L vae The weighted sum, preferably, the weight coefficient can be set to 1.

[0044] The purpose of training VAE and GRU simultaneously is to enhance the semantic information of the semantic representation of visual stimuli in the whole brain's visual regions.

[0045] Furthermore, in step 5, cosine similarity is used as an indicator for category retrieval.

[0046] The technical solution provided by this invention brings at least the following beneficial effects:

[0047] First, this invention uses a pre-trained variational autoencoder to fuse semantic information from different modalities into a unified space. When performing contrastive learning representation alignment, the alignment depth category retrieval performance is significantly improved compared to previous single-modal alignment, with an improvement of 80%, and compared to conventional multimodal alignment methods, with an improvement of 50%, which solves the alignment ambiguity problem in multi-target modal alignment to a certain extent. Second, it uses Prompt Embedding to embed subject information into the input, enabling the model to recognize different subject signals, and proposes a new approach for general decoding. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A diagram illustrating the alignment of fuzzy science problems;

[0050] Figure 2 This is a schematic diagram of the brain visual stimulus representation alignment model structure for modal fusion according to an embodiment of the present invention. In this model, I represents the image features obtained by the image encoder, T represents the text features obtained by the text encoder, E and D are the encoder and decoder of the variational autoencoder, respectively, and the subscripts of E and D are used to distinguish between images and text. u1 and ∑1 represent the image encoder E. I The output mean vector and logarithm vector of variance, u2 and ∑2 represent the text encoder E. T The output mean vector and logarithm vector of variance, Z fus For multimodal fusion semantic information representation, I',T' are the outputs of the corresponding decoders of variational autoencoders.

[0051] Figure 3 This is a schematic diagram comparing specific test results of an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0053] This invention provides a method for aligning brain visual neural activity representations through multimodal fusion. First, this invention establishes a brain visual stimulus representation alignment model based on CLIP by generating textual descriptions of image semantic content and functional magnetic resonance imaging (fMRI) signal data based on viewing a large number of natural images (visual perception images). The proposed model includes two important modules: (1) a visual stimulus semantic fusion module, in which a VAE merges CLIP textual modal semantic representations and CLIP image modal semantic representations into a unified semantic representation space; (2) a brain semantic information encoder, in which a GRU extracts semantic representations from the spliced ​​sequence of signals from major brain regions of the cerebral cortex and the Prompt embeddings of the subject's information. The representations obtained from the two modules are compared and aligned in the semantic feature space to achieve feature layer alignment.

[0054] refer to Figure 2 The model diagram shown illustrates the specific implementation steps of the alignment method for multimodal fusion of brain visual neural activity representations provided in this embodiment of the invention, which includes:

[0055] Step A: Construct a visual stimulus semantic fusion module

[0056] In step A1, the OpenAI open-source ViT-L / 14 model is used as the image encoder, and the corresponding CLIP text encoder is used as the text encoder. Their corresponding feature dimensions are 768. Thus, an m*n visual stimulus image is transformed into a 1*768 image semantic representation after being processed by the image encoder. The text information is first tokenized, which divides the input text string into tokens, and then encoded into a 1*768 text semantic representation.

[0057] Step A2: Use Variational Autoencoder (VAE) to fuse the image semantic representation 1*768 and text semantic representation 1*768 obtained in step A1 in the latent space to obtain fused semantic representation 1*768.

[0058] Step B: Constructing a semantic information encoder for the brain

[0059] Step B1: Select the main visual regions in the NSD dataset (Natural Scene Dataset). In this embodiment, the main visual regions selected include 21 brain visual cortex signals that are present in all 4 subjects, including 'V1', 'MST', 'V6', 'V2', 'V3', 'V4', 'V8', 'V3A', 'V7', 'IPS1', 'FFC', 'V3B', 'LO1', and 'LO2'. Use reshape to unify the signals from different brain regions (1*n) into one dimension, namely the feature dimension 1*768, and then splice the signals from the 21 brain regions to obtain the original brain region signals of 21*768.

[0060] Step B2: Select CLIP's text encoder and encode the source subject information of the 21*768 brain region signal, specifically "Processing on Subject 01", as a Prompt embedding and concatenate it to the end of the 21*768 to become a 22*768 brain signal. The purpose of this is to manually enhance the differences between subjects so that the model can learn the differences between different subjects.

[0061] Step B3: The 22*768 from step B2 is fed into the GRU neural network. 22 is used as the sequence dimension. The purpose is to learn the semantic information of visual stimuli contained in 21 different brain regions. The last 1*768 is taken as the output to obtain a 1*768 semantic representation of visual stimuli of the whole brain visual region for subsequent alignment.

[0062] Step C, Training Phase

[0063] Step C1 first obtains visual activity information from multiple cerebral cortex regions measured by fMRI, including the main categories of stimulus images and manually labeled semantic tags, divided into training and test sets. The cerebral cortex includes 21 brain regions contained in the subjects, such as V1, V2, V3, OFA, PPA, OPA, VWFA, FBA, FFA, and EBA.

[0064] Step C2, the pre-training VAE stage, uses the CLIP representation of the image-text modality of visual stimuli as the input of the VAE; only the visual stimulus semantic fusion module is trained, using the reconstruction loss and KL divergence of the VAE as the loss function, and based on this loss, a representation extractor that extracts fused semantic information in the latent space is trained. Based on the loss function L, the AdamW optimization algorithm is used to update the corresponding weights of the parameters of the entire model.

[0065] The specific optimization parameters for AdamW in step C2 are as follows:

[0066] The learning rate α is initialized to 0.0001; the decay coefficients β1 and β2 of the momentum term are set to 0.88 and 0.98 respectively; the minimum value of zero in the denominator is set to 10. -9 The first and second momentum are initialized as v = 0, u = 0; the time step is initialized as t = 0; each time there are n dataset samples {(x1, y1), (x2, y1)}. 12 ),...,(x n ,y n The following updates will be made:

[0067] Gradient calculation: Where L(f(x;θ),y) represents the loss function. The gradient of the loss function is represented by f(x; θ), which represents the output of the model, i.e., the model's prediction result. x represents the input data of the model, θ represents the model parameters, and y represents the label.

[0068] Time step update: t = t + 1;

[0069] Calculate the first momentum: v = β1 × v + (1 - β1) × g;

[0070] Calculate the second momentum: u = β² × u + (1 - β²) × g 2 ;

[0071] Correction for the first momentum:

[0072] Correction for the second momentum:

[0073] Update parameters: That is, Δθ represents the amount of model parameter update;

[0074] Application update: θ = θ + Δθ.

[0075] Step C3: Align brain activity with multimodal fusion semantic representation through contrastive learning. Align brain activity semantic representation obtained in steps A and B with multimodal fusion semantic representation through feature space contrastive learning. Simultaneously train VAE and GRU, add the loss of contrastive learning to the loss of VAE, and then use AdamW optimization algorithm to update the corresponding weights of the entire model based on the loss function L.

[0076] The specific optimization parameters for AdamW in step C3 are as follows:

[0077] The learning rate α is initialized to 0.0001; the decay coefficients β1 and β2 of the momentum term are set to 0.88 and 0.98 respectively; the minimum value of zero in the denominator is set to 10. -9 The first and second momentum are initialized as v = 0, u = 0; the time step is initialized as t = 0; each time there are n dataset samples {(x1, y1), (x2, y1)}. 12 ),...,(x n ,y n The following updates will be made:

[0078] Gradient calculation: Where L(f(x;θ),y) represents the loss function. The gradient of the loss function is represented by f(x; θ), which represents the output of the model, i.e., the model's prediction result. x represents the input data of the model, θ represents the model parameters, and y represents the label.

[0079] Time step update: t = t + 1;

[0080] Calculate the first momentum: v = β1 × v + (1 - β1) × g;

[0081] Calculate the second momentum: u = β² × u + (1 - β²) × g 2 ;

[0082] Correction for the first momentum:

[0083] Correction for the second momentum:

[0084] Update parameters: That is, Δθ represents the amount of model parameter update;

[0085] Application update: θ = θ + Δθ.

[0086] Step D, Testing Phase

[0087] Step D1 involves collecting test data. Each natural image (the visual stimulus in the experiment) has multiple labels derived from the COCO (Common Objects in COntext) dataset, including one main category, multiple labels, and five text descriptions. The main category is a human-generated annotation corresponding to the natural image. In the COCO dataset's annotation information, the main category includes 12 "super categories," such as "people," "vehicles," "outdoors," etc.

[0088] Step D2 involves testing the trained visual stimulus semantic fusion module with the brain semantic information encoder by performing a "superclass" retrieval on all samples in the test set. The baseline level is equivalent to 12 categories, or 0.08.

[0089] To further verify the accuracy of category retrieval based on the method of this invention, cosine similarity was used as an indicator for category retrieval. There were a total of 12 categories. Category retrieval was performed on all samples in the test set, and the retrieval accuracy was as follows: Figure 3 As shown in the figure, Subj01 to Subj05 represent the subject numbers of the three participants in the NSD experiment. In terms of category retrieval accuracy, the model of this embodiment achieved an average of 0.26 for the three subjects, a significant improvement over the baseline. Furthermore, it showed improvement compared to the single-modal alignment of previous methods (0.19 for image alignment, 0.20 for text alignment), indicating that the multimodal alignment method of this embodiment is superior to single-modal alignment. Moreover, compared to the multimodal alignment methods in previous methods, the method of this embodiment outperforms ordinary multimodal alignment (which updates the model by calculating loss functions for brain activity in image / text modal representations) (0.21 / 0.23) in the sample category retrieval task, demonstrating that the model of this embodiment contributes to solving alignment ambiguity problems and provides value for subsequent brain decoding research.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0091] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for aligning brain visual neural activity representations through multimodal fusion, characterized in that, Includes the following steps: Step 1: Use the pre-trained CLIP text encoder to embed the text of the visual stimulus into the CLIP semantic feature space to obtain the text semantic feature representation; use the pre-trained CLIP image encoder to embed the image of the visual stimulus into the CLIP semantic feature space to obtain the image semantic feature representation. Among them, the feature dimensions of text semantic feature representation and image semantic feature representation are the same; Step 2: Use the visual stimulus semantic fusion module to fuse the text semantic feature representation and the image semantic feature representation in the feature space to obtain a multimodal fused semantic information representation. Step 3: Use a brain semantic information encoder to convert the response signals of the main visual regions of the specified whole brain into latent feature space vector representations, and add cue embeddings from the subject's information to obtain brain semantic information representations. Step 4: Use contrastive learning to align the multimodal fusion semantic information representation from Step 2 with the brain semantic information representation from Step 3 in the feature space. Based on the training data set, iterative steps 1-4 are used to train the visual semantic fusion module and the brain semantic information encoder to obtain the visual semantic fusion module and the brain semantic information encoder for aligning the brain's visual neural activity representation. Specifically, in step 3, the encoding process of the brain semantic information encoder is as follows: Step 301: Based on the selected main visual regions, use a conversion layer to convert the response signals of visual activity in each main visual region. Unifying to a specified dimension Num yields the response signal feature representations for each major visual region, where Num represents the feature dimension of text semantic feature representations and / or image semantic feature representations, and the index of the major visual region. T represents the number of primary visual regions; The original response signal of T×Num dimensional is obtained by splicing the response signals of T main visual regions. Step 302: Based on the CLIP text encoder, the source subject information of the original response signal is encoded into a cue embedding, and then the cue embedding is concatenated to the original response signal to obtain (T+1)×Num dimensional visual response signal features. Step 303: Input the (T+1)×Num dimensional visual response signal features into the GRU neural network, using T+1 as the sequence dimension of the GRU neural network, and take the last 1×Num as the output to obtain a 1×Num brain semantic information representation.

2. The method as described in claim 1, characterized in that, It also includes step 5, which uses the trained visual semantic fusion module and the brain semantic information encoder to perform representation category retrieval and classification.

3. The method as described in claim 2, characterized in that, In step 5, cosine similarity is used as an indicator for category retrieval.

4. The method according to any one of claims 1 to 3, characterized in that, In step 1, the ViT-L / 14 version of the CLIP image encoder is used, and the CLIP text encoder corresponding to the ViT-L / 14 version is used as the pre-trained CLIP text encoder.

5. The method as described in claim 1, characterized in that, In step 2, the visual stimulus semantic fusion module uses a variational autoencoder (VAE) to fuse text semantic feature representations and image semantic feature representations.

6. The method as described in claim 1 or 5, characterized in that, Step 4 specifically includes: Step 401: Pre-train only the VAE from step 3. The loss function used during VAE pre-training is... Including VAE reconstruction loss With KL divergence loss ; Step 402: The 1×Num brain semantic information representation obtained in step 303 is compared with the multimodal fusion semantic information representation obtained in step 2 using a CLIP-style contrastive learning method to perform feature space alignment. The loss function used is the contrastive loss function. ; Step 403: Train VAE and GRU simultaneously, using the total loss during training. for and The weighted sum.

7. The method as described in claim 6, characterized in that, VAE reconstruction loss The expression is: in, Represents the mathematical expectation; This represents the output of the decoder; This represents the prior distribution of the latent variable z; This represents the posterior distribution parameterized by the encoder network.

8. The method as described in claim 6, characterized in that, KL divergence loss The expression is: Where J represents the dimension of the latent variable z; , Let represent the standard deviation and mean of the j-th latent variable, respectively.

9. The method as described in claim 6, characterized in that, Contrast loss function The expression is: Where N represents the total number of samples; n represents the upper limit index; Cosine similarity; Indicates the first A positive sample is composed of multimodal fusion semantic information representation and brain semantic information representation of each sample; Indicates the first The multimodal fusion semantic information representation of the sample and the first The negative sample is composed of the brain semantic information representation of each sample. This represents the temperature coefficient.

Images