Neural decoding method and system based on three-space decoupling and multi-modal mutual information

Abstract

The invention discloses a neural decoding method based on three-space decoupling and multi-mode mutual information. The method comprises the following steps: respectively coding multi-modal data to obtain a private representation, a shared representation and a collaborative representation, and correspondingly converting the private representation, the shared representation and the collaborative representation into distribution parameters of a private space, a shared space and a collaborative space; obtaining a potential representation based on the private representation, the shared representation and the collaborative representation, decoding the potential representation into reconstruction data, and obtaining a distribution parameter of the reconstruction data; constraining private space mutual information, constraining shared space mutual information and constraining collaborative space mutual information; constructing a total loss function to train the private space, the shared space and the collaborative space; and performing visual category neural decoding based on a dynamic feature enhanced attention fusion method. According to the method, the accuracy of fMRI-image cross-modal representation learning and visual category neural decoding can be improved.

Description

Technical Field

[0001] This invention relates to the field of visual category neural decoding technology, and in particular to a neural decoding method, system, storage medium, and electronic device based on three-space decoupling and multimodal mutual information. Background Technology

[0002] Patterns of brain activity contain meaningful information about the perceived world. Neural decoding, which uses recorded brain activity to predict variables of the external world, has been a hot topic in neuroscience. To reveal the workings of the brain, functional magnetic resonance imaging (fMRI), as a non-invasive method, is widely used to decipher the cognitive and perceptual processes of the human brain. Among various fMRI-based neural decoding methods, decoding visual neural representations with rich structural and semantic content is an important and challenging task, and it is of great significance for revealing the mechanisms of visual cognitive processing and developing brain-like intelligent robots. Visual neural representation decoding involves multimodal information from both the visual and cognitive domains. Research shows that multimodal variational autoencoders (VAEs) exhibit good performance in learning decoupled representations and can capture different varying factors in the input data. Building upon information entropy and mutual information, partial information decomposition (PID) is a further extension of information theory, aiming to extend the pairwise relationships that information theory focuses on to complex interactions between multiple variables.

[0003] Traditional variational autoencoders only include private and shared spaces, neglecting the modeling ability of intermodal collaboration to enhance features. Summary of the Invention

[0004] This invention provides a neural decoding method, system, storage medium, and electronic device based on three-space decoupling and multimodal mutual information, which can improve the accuracy of fMRI image cross-modal representation learning and visual category neural decoding.

[0005] This invention provides a neural decoding method based on three-space decoupling and multimodal mutual information, comprising: Acquire multimodal data, which includes fMRI signals and corresponding image data; The multimodal data is encoded into private representation, shared representation and cooperative representation respectively, and then converted into distribution parameters of private space, shared space and cooperative space respectively; The latent representation is obtained based on private representation, shared representation and cooperative representation. The latent representation is then decoded into reconstructed data, and the distribution parameters of the reconstructed data are obtained. Private representation and reconstructed data are input into a private space mutual information discrimination network for discrimination, constraining the private space mutual information; shared representation and reconstructed data are input into a shared space mutual information discrimination network for discrimination, constraining the shared space mutual information; cooperative representation and the joint reconstructed data of the two modal data are input into a cooperative space mutual information discrimination network for discrimination, constraining the cooperative space mutual information. Construct a total loss function and train it on the private space, shared space, and collaborative space; Visual category neural decoding based on dynamic feature-enhanced attention fusion method.

[0006] This invention also provides a neural decoding system based on three-space decoupling and multimodal mutual information, comprising: An acquisition module is used to acquire multimodal data, which includes fMRI signals and corresponding image data; The three-space encoding module is used to encode the multimodal data into private representation, shared representation and cooperative representation respectively, and convert them into distribution parameters of private space, shared space and cooperative space respectively; The decoding and reconstruction module is used to obtain the latent representation based on the private representation, shared representation, and cooperative representation, decode the latent representation into reconstructed data, and obtain the distribution parameters of the reconstructed data; The dynamic constraint module is used to input private representation and reconstructed data into a private space mutual information discrimination network for discrimination, thereby constraining the private space mutual information; input shared representation and reconstructed data into a shared space mutual information discrimination network for discrimination, thereby constraining the shared space mutual information; and input cooperative representation and the joint reconstructed data of the two modal data into a cooperative space mutual information discrimination network for discrimination, thereby constraining the cooperative space mutual information. The training module is used to construct the total loss function and train it on the private space, shared space, and collaborative space. The fusion classification module is used for visual category neural decoding based on the dynamic feature-enhanced attention fusion method.

[0007] The present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described neural decoding methods based on three-space decoupling and multimodal mutual information.

[0008] The present invention also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the neural decoding method based on three-space decoupling and multimodal mutual information described in any of the preceding claims.

[0009] The neural decoding method, system, storage medium, and electronic device based on three-space decoupling and multimodal mutual information provided by this invention have the following beneficial effects: (1) The present invention adopts a three-space latent representation architecture. Through the refined design of private space, shared space and collaborative space, it can more comprehensively decompose and model the information of different levels of multimodal data, and improve the feature expression ability and interpretability. (2) Based on the data characteristics of different modalities, a cross-modal collaborative attention mechanism was designed, which not only considered modality-specific information but also modeled the collaborative interaction relationship between modalities, thereby improving cross-modal understanding ability; (3) By designing the conditional prior distribution of the collaborative space, the dependencies between modes can be modeled more accurately, thereby improving the consistency and quality of cross-modal generation; (4) The present invention uses dynamic feature enhancement attention and dynamically adjusts the fusion weights according to the actual performance of each potential space. Compared with the fixed weight method, the accuracy in zero-shot learning tasks is significantly improved. Attached Figure Description

[0010] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0011] Figure 1 The flowchart illustrates a neural decoding method based on three-space decoupling and multimodal mutual information provided in an embodiment of the present invention.

[0012] Figure 2 This is a schematic diagram of the structure of a neural decoding system based on three-space decoupling and multimodal mutual information provided in an embodiment of the present invention.

[0013] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention provides a neural decoding method, system, storage medium, and electronic device based on three-space decoupling and multimodal mutual information. The neural decoding system based on three-space decoupling and multimodal mutual information provided by this invention can be integrated into an electronic device, such as a terminal or server. The terminal can include tablet computers, laptops, personal computers (PCs), microprocessor boxes, or other devices.

[0016] Please see Figure 1 , Figure 1 The flowchart illustrates a neural decoding method based on three-space decoupling and multimodal mutual information provided in this embodiment of the invention. This method, applied in electronic devices, includes the following steps: S1, acquire multimodal data, which includes fMRI signals and corresponding image data.

[0017] Specifically, fMRI (functional magnetic resonance imaging) signal data was acquired to record the neural activity signals of subjects while performing cognitive tasks. Corresponding image data was obtained, ensuring complete temporal and semantic alignment with the fMRI signals. Feature extraction was then performed on the image data, using a deep learning model to extract high-dimensional visual feature representations.

[0018] S2 encodes the multimodal data into private, shared, and cooperative representations, and converts them into distribution parameters for the private, shared, and cooperative spaces, respectively.

[0019] This step proposes a latent space decomposition strategy, which decomposes the single latent space u of the traditional VAE (Multimodal Variational Autoencoder) into three functionally complementary subspaces to achieve more refined cross-modal representation learning.

[0020] The latent representation is decomposed into a private space Shared Space and collaborative space Private spaces encode modality-specific information, shared spaces encode cross-modal common features, and collaborative spaces encode inter-modal collaborative enhancement features. The complete representation is shown below:

[0021] In one embodiment, a private space coding branch, a shared space coding branch, and a cooperative space coding branch are constructed in advance. Step S2 includes the following steps: S21, input each modal data into the private space coding branch to obtain the private representation, and convert the private representation into the distribution parameters of the private space coding branch.

[0022] The private space coding branch is specifically responsible for extracting the modality-specific information, i.e., the private representation, for each modality. First, for the m-th modality, the private space coding branch... A multi-layer fully connected network architecture is adopted to realize the nonlinear mapping from the original feature space to the private latent space.

[0023] Secondly, based on the distribution parameters of the private space coding branch, a variational posterior distribution of the private space is constructed. Depending on the choice of prior distribution type, a normal prior is adopted, and the variational posterior distribution of the private space is defined as follows:

[0024] in, For the variational posterior distribution of the private space, Indicates encoder parameters, Representing a given latent variable Generate the corresponding normal distribution parameters under the given conditions.

[0025] S22, input each modal data into the shared space coding branch to obtain the shared representation, and convert the shared representation into the distribution parameters of the shared space coding branch.

[0026] The goal of the shared space design is to learn representations that maintain semantic consistency across different modalities. Unlike the independent encoding of the private space, the shared space uses the PoE (Product-of-Experts) method to obtain a shared representation.

[0027] Each modality first extracts candidate representations of the shared space through an independent shared space coding branch. For the m-th modality, the shared space coding branch... The network architecture maintains structural symmetry with the private space branches to ensure that each subspace has considerable learning capabilities.

[0028] The basic idea of ​​Proof-of-Earnings (POE) is to fuse the predictions of various expert models through multiplication. Specifically, POE multiplies the posterior distributions of each modality and multiplies their probability estimates to obtain a final fused result, as shown below:

[0029] Where M represents the total number of modalities, and z represents the shared representation. The table shows the input data for the m-th mode. The shared space posterior distribution after POE fusion. Let be the posterior distribution of the m-th modality, representing the probability distribution of the latent variables given the data of the m-th modality. This represents the parameters of the m-th modal encoder. It indicates a direct proportional relationship.

[0030] S23, input each modal data into the cooperative space coding branch to obtain the cooperative representation, and convert the cooperative representation into the distribution parameters of the cooperative space coding branch.

[0031] Cooperative space coding branch It is responsible for extracting potential collaborative information from unimodal data, providing a basic representation for subsequent attention enhancement.

[0032] First, the private representation of the target modality is mapped to a query vector through a linear transformation. Specifically, the private representation of the target modality... By transforming the matrix Perform a linear transformation to obtain the query matrix. As shown below:

[0033] The private representation of the source modality is transformed into a key vector and a value vector, respectively. Specifically, the private representation of the source modality is transformed... By transforming the matrix and Perform linear transformations separately, and calculate the key matrix K and value matrix V separately, as shown below:

[0034]

[0035] Based on the query, key, and value matrices obtained from the above steps, a scaled dot product is performed to calculate cross-modal relevance weights and weighted feature representations. First, the product of the query matrix and the transpose of the key matrix is ​​calculated, and a scaling factor is applied to obtain the original attention score matrix:

[0036] in, This is the scaling factor.

[0037] Then, the score matrix is ​​normalized using the softmax function, transforming the original scores into an effective probability distribution. :

[0038] Finally, the normalized attention weights are multiplied by the value matrix to complete the weighted aggregation operation and obtain the attention output. :

[0039] Among them, the attention score matrix Each element The correlation strength between the i-th query vector and the j-th key vector is quantified. After softmax normalization, the sum of elements in each row of the attention weight matrix A is 1, forming a standard probability distribution and ensuring the rationality of attention allocation. The final attention output result is obtained. The attention output is a collaborative representation that integrates the interaction information of the private spaces of the source and target modalities.

[0040] To convert the collaborative representation into conditional prior distribution parameters that can be used to guide cross-modal generation, a multilayer perceptron network is designed as a conditional prior parameter predictor. This network employs a two-layer hidden layer structure, each equipped with a ReLU activation function to provide sufficient nonlinear modeling capability. The network outputs the results (collaborative representation) with attention. As input, the output is a complete set of parameters of the co-space Gaussian distribution, including the mean and variance.

[0041]

[0042]

[0043]

[0044] in, These represent the weight matrices of each layer. For the corresponding bias vector, For the hidden layer dimension, and These represent the outputs of the first and second hidden layers, respectively. (Output of the multilayer perceptron) It contains all the necessary parameters for the conditional prior distribution to construct the complete parameter set of the standard Gaussian distribution.

[0045] To achieve cross-modal bidirectional constraints, the roles of the source and target modalities are swapped, and two complete attention computation processes are executed separately. This generates cooperative space prior distribution parameters for each modality based on the private information conditions of the other modality. In other words, a cross-modal attention network is used. It receives private representations from different modalities as input and outputs the conditional prior distribution of the cooperative space.

[0046] Specifically, attention calculation is performed using the private representation of the brain modality as the source and the private representation of the image modality as the target:

[0047] Using image modal private representations as the source and brain modal private representations as the target, attention computation is performed: ( )= ( ) in, This represents the private representation of the Brain modality. The private representation of the Image modality, provided as a key-value pair, carries context. The query generates an attention query, which ultimately yields conditional prior parameters used to guide the modeling of the Image modality co-space. This represents the private representation of the Image modality. As a key, the private representation of the Brain modality As a query, we obtain the conditional prior parameters used to guide the modeling of the Brain modal co-space.

[0048] To support bidirectional cross-modal reconstruction, specific conditional prior distributions are needed for different transition directions. The bidirectional conditional enhancement mechanism dynamically generates corresponding collaborative representations based on the specific cross-modal transition direction. (Cooperative representation from Brain modality to image data modality) and (Collaborative representation from image data modalities to Brain modalities), the posterior distribution of the collaborative space is represented as: The above process can be represented as:

[0049]

[0050] Furthermore, based on the assumption that the three subspaces are independent, the complete latent space posterior distribution is shown below:

[0051] Among them, private space Dependent on a single modality, sharing space By fusing multimodal information through PoE, collaborative space Taking into account both single-modal coding and cross-modal interaction.

[0052] S3 obtains the latent representation based on the private representation, shared representation, and collaborative representation, decodes the latent representation into reconstructed data, and obtains the distribution parameters of the reconstructed data.

[0053] In one embodiment, step S3 includes the following steps: S31, the private representation, shared representation and cooperative representation are concatenated to obtain the complete potential representation.

[0054] The complete potential representation is shown below:

[0055] in, For private representation, For shared representation, For collaborative representation.

[0056] S32, input the latent representation into the decoder to obtain the reconstructed data and the distribution parameters of the reconstructed data.

[0057] Step S32 includes the following steps: S321, Input the latent representation into the first fully connected neural network to perform the transformation from the latent representation to the hidden layer, and obtain the transformation result; S322, the transformation result is input into the second fully connected neural network and mapped to the target mode to obtain the output result; S323 defines the reconstruction distribution of the reconstructed data as a Gaussian distribution, and obtains the distribution parameters of the reconstructed data.

[0058] Specifically, the decoder network adopts a multilayer perceptron architecture, which realizes the mapping from the latent space to the observation space through multilayer nonlinear transformations. First, the complete latent representation is input into the decoder.

[0059] Then, a two-layer fully connected neural network structure is used to implement feature mapping. The network architecture is defined as a nonlinear mapping from the latent space to the target modality feature space.

[0060] The first layer performs the transformation from the latent representation to the hidden layer, and the calculation formula is as follows:

[0061] in, This is the weight matrix. is the bias vector, and LeakyReLU is the activation function, with a negative slope of 0.2 to maintain gradient fluidity.

[0062] The second layer maps the hidden layer representation to the output space of the target modality, and the calculation formula is as follows:

[0063] in, This is the output layer weight matrix. This is the output layer bias vector. The mean parameter is used to reconstruct the distribution. The variance parameter is ensured to be non-negative using the Softplus activation function, as shown below:

[0064] in, It is the numerical stability constant. This is the variance parameter.

[0065] Finally, the reconstruction distribution of mode m is defined as a Gaussian distribution, as shown below:

[0066] in, Indicates a Gaussian distribution. represents the distribution parameters of the reconstructed data for mode m.

[0067] S4. The private representation and reconstructed data are input into the private space mutual information discrimination network for discrimination, constraining the private space mutual information; the shared representation and reconstructed data are input into the shared space mutual information discrimination network for discrimination, constraining the shared space mutual information; the cooperative representation and the joint reconstructed data of the two modal data are input into the cooperative space mutual information discrimination network for discrimination, constraining the cooperative space mutual information.

[0068] The mutual information content of the three latent spaces is controlled by estimating mutual information using neural networks. Specifically, the goal is to ensure that the private space has high mutual information with its corresponding modality's reconstructed data and low mutual information with other modality's reconstructed data; the shared space has high mutual information with all modality's reconstructed data; and the cooperative space has high mutual information with cross-modal joint information. A dedicated neural network discriminator is pre-constructed to estimate the mutual information strength between the private space and its corresponding modality's reconstructed distribution, ensuring that the private space only learns modality-specific features.

[0069] In one embodiment, step S4 includes the following steps: S41, the feature representation obtained after sampling based on the private representation is concatenated with the feature representation of the reconstructed fMRI data and then input into the private space-fMRI mutual information discriminator for discrimination.

[0070] Let the dimensions of fMRI data be... The spatial dimension of W is First, the feature representation obtained by sampling the private representation is concatenated with the feature representation of the reconstructed data after fMRI reconstruction to obtain... The input vector is 3D. The first layer of the network maps the input to the hidden space through a linear transformation, and the second layer maps the hidden representation to scalar discriminant scores.

[0071] The calculation formula for the first layer of the network is as follows:

[0072]

[0073] in, This represents the feature representation after sampling of the private representation of fMRI. This represents the characteristic representation of the reconstructed data from fMRI.

[0074] The formula for calculating the output of the second layer of the network is as follows:

[0075] in, For the obtained discrimination score This is the weight matrix. This is the bias vector.

[0076] S42, the feature representation obtained after sampling based on the private representation is concatenated with the feature representation of the reconstructed image data and then input into the private space-image mutual information discriminator for discrimination.

[0077] Let the image data dimension be . The W-space encoding dimension is The input is concatenated to get Dimensional vector. The network structure is the same as the fMRI discriminator, only the input dimension is different, and the calculation steps are the same as above.

[0078] S43, the feature representation obtained after sampling based on the shared representation is concatenated with the feature representation of the reconstructed fMRI data and then input into the shared space-fMRI mutual information discriminator for discrimination.

[0079] Let the shared space dimension be... fMRI reconstruction data dimensions are The input is concatenated to get 3D vector. The network uses a two-layer fully connected structure, with the first layer containing the ReLU activation function.

[0080] The formula for calculating the input splicing operation is:

[0081] in, This is the result of splicing.

[0082] The formula for calculating the hidden layer is:

[0083] The formula for calculating the output layer is:

[0084] in, To obtain the discrimination score, This is the weight matrix. This is the bias vector.

[0085] S44, the feature representation obtained after sampling based on the shared representation is concatenated with the feature representation of the reconstructed image data and then input into the shared space-image mutual information discriminator for discrimination.

[0086] The specific calculation steps can be referred to the steps above, and will not be elaborated on here.

[0087] In addition, the mutual information discriminator can also be used to evaluate the consistency of shared representations between two modalities. Specifically, the discriminator receives the shared representations of the fMRI modality and the image modality as input to obtain the consistency of the shared representations. The specific calculation steps can be referred to the steps described above.

[0088] S45 combines the reconstructed data of the fMRI signal and the reconstructed data of the image data with the collaborative representation, and then inputs them into the collaborative spatial mutual information discriminant network for discrimination.

[0089] Let the dimension of the collaborative space be... fMRI data dimensions are The image data dimension is First, the two modal data are concatenated to form a joint feature, which is then concatenated with the collaborative representation as input to the discriminator.

[0090] The formula for calculating the joint feature construction is as follows:

[0091] in, This represents the features of fMRI reconstructed data. For combined results.

[0092] The formula for calculating the input of the discriminator is:

[0093] Where c represents the cooperative representation. This is the input for the discriminator.

[0094] The formula for calculating the forward pass of the network is:

[0095] in, To obtain the discrimination score, This is the weight matrix. This is the bias vector.

[0096] Furthermore, this method also includes the following steps: A variational mutual information estimation method is employed to transform mutual information computation into an optimizable lower bound estimation problem. Specifically, the Donsker-Varadhan variational representation theorem is used to estimate the lower bound of the true mutual information through neural network parameterization. The specific steps include: S46, construct positive and negative sample pairs based on distribution parameters and reconstructed data.

[0097] Specifically, based on the preprocessed data obtained in step S1, given a batch of data containing latent representations (latest representations refer to private, shared, or collaborative representations, not complete latent representations) and reconstructed data for B samples, positive and negative sample pairs are constructed by preserving and disrupting the original pairing relationships. Positive sample pairs maintain the original encoding-data pairing relationship, while negative sample pairs disrupt the pairing relationship by randomly permuting the latent encoding, thereby providing contrastive learning samples for variational mutual information estimation.

[0098] Taking the shared space as an example, the representation of the positive sample pair construction is as follows:

[0099] in, Denotes the set of positive sample pairs. This represents the shared representation of the encoder output of the i-th sample in the shared space. This represents the reconstructed data output by the decoder for the i-th sample, where i is the sample index, ranging from 1 to B.

[0100] Negative sample pairs are constructed through random permutations, and the calculation formula is as follows:

[0101]

[0102] in, This represents a random permutation function that shuffles the original index sequence. Represents the i-th after random permutation The potential representation of a sample, This represents the reconstructed data output by the decoder for the i-th sample, where i is the sample index, ranging from 1 to B.

[0103] The formula for constructing the discriminator input matrix is:

[0104]

[0105] in, Represents the positive sample input matrix. The input matrix is ​​for negative samples.

[0106] S47. Input the positive and negative sample pairs into the trained mutual information discriminant network to obtain the corresponding discriminant scores, and calculate the lower bound of mutual information based on the Donsker-Varadhan variational representation.

[0107] Specifically, the constructed positive and negative sample pairs are input into the trained mutual information discriminant network to obtain corresponding discriminant scores. Then, the lower bound of the mutual information is calculated based on the Donsker-Varadhan variational representation. To avoid overflow issues in numerical computation, numerical stabilization techniques are used to process the exponential operation.

[0108] The formula for calculating positive sample scores is:

[0109] in, This represents the positive sample score vector, containing the discrimination scores for all positive sample pairs. The parameter is The discriminant network, This represents the Bth positive sample pair.

[0110] The formula for calculating negative sample scores is:

[0111] in, This represents the negative sample score vector, containing the discrimination scores for all negative sample pairs. The parameter is The discriminant network, This represents the B-th negative sample pair.

[0112] The formula for calculating the expected value of positive samples is:

[0113] in, This represents the expected value of the positive sample scores. As the normalization factor, Let represent the discrimination score of the i-th positive sample.

[0114] The formula for calculating the expected value of the negative sample exponent in numerical stabilization is:

[0115]

[0116] in, This represents the maximum score of the negative sample. Let represent the discrimination score of the i-th negative sample. This represents the negative sample expectation value used for numerical stabilization.

[0117] The formula for estimating the lower bound of variational mutual information is:

[0118] in, This is represented as the next-bound estimate of the variational mutual information of the reconstructed data of the shared space and its corresponding modalities. It is a numerically stable term.

[0119] S5, construct the total loss function and train it on the private space, shared space and collaborative space.

[0120] Step S5 specifically includes the following steps: S51, calculate the VAE (Variational Autoencoder) loss.

[0121] By combining shared, private, and cooperative representations, and to ensure that the shared and cooperative subspaces are learned without falling into shortcuts, an auxiliary distribution is introduced to facilitate the estimation of cross-modal reconstruction. The objective function is as follows:

[0122] in, This is a loss for VAE.

[0123] Derived from the lower bound of evidence (ELBO) of the standard VAE. First from Starting from the variational lower bound, we introduce latent variables. (Shared information) (Unique information) and (Collaborative information), obtained by applying Jensen's inequality. .

[0124] in, This indicates that the joint distribution is decomposed into , It is a priori sharing of latent variables. It is an auxiliary variable prior. It is a priori condition. It is a generative model, variational posterior. Factorize into To achieve scalable reasoning and avoid the leakage of modal private spaces, auxiliary priors are used in cross-modal reconstruction. Force the decoder to rely on shared encoding Perform cross-modal reconstruction. S52, construct a three-space mutual information constraint loss function.

[0125] A three-space mutual information constraint loss function is constructed using the built MINE discriminator network to ensure that each subspace learns the expected information content.

[0126] First, using the outputs of the private space-fMRI mutual information discriminator and the private space-image mutual information discriminator, a private space modality-specific constraint is constructed to ensure that the private space only has high mutual information with the reconstructed data of the corresponding modality. The formula for calculating the private space constraint loss is:

[0127] in, Feature representation after fMRI private space sampling Its reconstructed feature representation Mutual information. Feature representation after sampling of the private space of the image Its reconstructed feature representation Mutual information.

[0128] Secondly, using the output of the shared space mutual information discriminator, cross-modal constraints in the shared space are constructed to ensure high mutual information between the shared space and the reconstructed data of all modalities, and to maintain consistency across different modalities. The formula for calculating the shared space constraint loss is:

[0129] in, Indicates fMRI shared space Its reconstructed feature representation mutual information, Feature representation of an image after shared spatial sampling Rather than rebuilding data Mutual information.

[0130] Finally, using the output of the cross-modal mutual information discriminator in the cooperative space, joint constraints in the cooperative space are constructed to ensure high mutual information between the cooperative space and the cross-modal joint representation. The formula for calculating the cooperative space constraint loss is as follows:

[0131]

[0132] in, This represents the stitched feature representation after reconstruction of the fMRI and image modalities. Feature representation after co-sampling and Mutual information.

[0133] The loss due to complete mutual information constraint is the sum of the three losses mentioned above, calculated as follows:

[0134] in, For the loss due to private space constraints, For the shared space constraint loss, This is due to the loss of collaborative spatial constraints.

[0135] S53 calculates the total loss function based on the three-space ELBO (Evidence Lower Bound, also known as the variational lower bound) loss function and the mutual information constraint loss function. Specifically, the VAE loss and the mutual information constraint loss are combined in a weighted manner, and the formula for calculating the total loss function is as follows:

[0136] in, For the ELBO loss function, For mutual information-constrained loss function, The weight parameter controls the strength of the mutual information constraint.

[0137] S6, Visual Category Neural Decoding Based on Dynamic Feature Enhancement Attention Fusion Method.

[0138] In one embodiment, step S6 includes the following steps: S61, based on the dynamic feature-enhanced attention network, performs bidirectional information fusion on the shared space and the collaborative space to obtain enhanced information of the shared space and enhanced information of the collaborative space.

[0139] Two independent attention modules are used to achieve bidirectional information fusion between the Z-space (shared space) and C-space (cooperative space) within a single modality. Each dynamic feature enhancement attention module contains four core components: a query-key projection network, an offset prediction network, a weight prediction network, and an output projection network.

[0140] In one embodiment, step S61 includes the following steps: S611, the processing procedure of query-key-value projection network.

[0141] The query-key projection network maps features from different spatial dimensions to a unified representation space, ensuring the effectiveness of subsequent attention computation. Feature alignment across different dimensional spaces is achieved through linear transformation.

[0142] Taking the calculation of the Z→C module projection as an example, the query projection, key projection, and value projection are shown below:

[0143]

[0144]

[0145] in, , These are the feature representations sampled from the shared space and the collaborative space, respectively. This indicates a query for the projection weight matrix. This indicates a query for the projection bias vector. This represents the query features after projection. Represents the key projection weight matrix. This represents the key projection bias vector. The value projection weight matrix represents the total weight. The value projection bias vector.

[0146] S612, Processing procedure of the offset prediction network Traditional attention mechanisms use fixed feature representations for all queries, failing to adjust the expression of query features according to the semantic requirements of different query content. Dynamic feature-enhanced attention, through a shift prediction network, dynamically predicts the optimal perturbation direction in the feature space based on the semantic content of the current query features, generating adaptive feature representations, thereby achieving adaptive feature interaction that is aware of query content.

[0147] First, a two-layer fully connected network is designed, taking query features as input and outputting offsets. The first layer of the network performs feature extraction and dimension transformation, while the second layer outputs the original offset values. Then, the offsets are normalized to the [-1, 1] interval using the Tanh function.

[0148] First-level feature extraction:

[0149] in, These are the query features after projection. This is the first layer weight matrix. This is the first layer bias vector. These are features of the intermediate hidden layer.

[0150] Second layer offset output:

[0151] in, This is the weight matrix for the second layer. This is the second layer bias vector. This represents the original offset prediction value.

[0152] S612, the processing procedure of the dynamic weight prediction network.

[0153] Feature variants generated by different perturbation directions contribute differently to the current query, requiring dynamic weight allocation based on the semantic content of the query features. The weight prediction network learns the semantic matching degree between each perturbation feature variant and the current query, achieving adaptive weight allocation based on query content.

[0154] The design is a two-layer fully connected network with a similar structure to the offset prediction network.

[0155]

[0156]

[0157]

[0158] in, This represents the first-level weight matrix for weight prediction. This represents the first-level bias vector. This represents the features of the intermediate hidden layer. This represents the second-layer weight matrix. This represents the second-layer bias vector. This represents the normalized weight of the k-th sampling point of the h-th attention head.

[0159] S614, Implement a feature perturbation sampling mechanism guided by semantic similarity. Existing dynamic feature enhancement attention mechanisms are primarily designed for data with well-defined spatial structures (such as images), lacking clear geometric meaning when directly applied to abstract feature vectors. Therefore, in the abstract feature space, the cosine similarity between the query feature and the target feature is calculated to provide a numerical guidance signal for the perturbation direction, ensuring that feature changes remain consistent with the distribution characteristics of the original features. The calculation steps are as follows:

[0160]

[0161]

[0162] The semantic relevance between the query and the key value is calculated by using the inner product of normalized feature vectors. Cosine similarity is then calculated after normalizing both query and value features, thereby quantifying the directional consistency between the two feature vectors in high-dimensional space. To query the L2 norm of the vector, It is a numerical stability constant to prevent division by zero errors.

[0163] To achieve multi-point sampling in the abstract feature space, K learnable perturbation direction parameters are defined. ,in Let represent the k-th perturbation direction vector. For the k-th sampling point of the h-th attention head, the perturbation intensity is calculated as follows:

[0164] in, It is the predicted offset of the h-th head and the k-th sampling point. ∈ [-1, 1] is used as the directional intensity of the perturbation modulation. Combining the offset and semantic similarity, the feature perturbation sampling is as follows:

[0165] in, It is a feature of the original value. It is the global disturbance intensity control coefficient. This represents the perturbation feature vector of the k-th sampling point.

[0166] S615, Multi-head Attention Convergence and Output Projection Different attention heads can learn different feature interaction patterns, enhancing the model's expressiveness and robustness. Each head focuses on a specific type of feature relationship.

[0167] Specifically, firstly, the sampling point aggregation within a single head is completed by weighting and summing the K perturbation sampling points of each attention head to obtain a single output feature vector for each head; then, multi-head averaging fusion is performed by averaging the output features of H attention heads to obtain a unified multi-head fusion representation; finally, linear transformation and dimensional mapping are performed through the output projection layer to restore the features from the value space dimension to the query space dimension.

[0168]

[0169]

[0170]

[0171] in, This represents the output of the h-th attention head. This represents the normalized weight of the h-th head and the k-th sampling point. Indicates the output projection matrix. Output projection bias vector.

[0172] S62, the first enhanced feature is obtained by enhancing the shared space enhancement information and the original shared space features, the second enhanced feature is obtained by enhancing the collaborative space enhancement information and the original collaborative space features, and the first enhanced feature and the second enhanced feature are concatenated to obtain the fused feature representation.

[0173] Step S62 includes the following steps: S621, Perform residual connection enhancement After obtaining enhanced information through a bidirectional dynamic feature enhancement attention mechanism, it is effectively fused with the original features to form the final enhanced feature representation. A residual connection strategy is employed to ensure that the original feature information is preserved while receiving supplementary attention enhancement.

[0174] First, the original Z-space features and C-space features are input into two symmetrical enhancement paths. In the Z-space enhancement path, the Z features serve as the query vector, and the C features serve as the key-value pair. Enhancement information is calculated using the C→Z dynamic feature enhancement attention module. Simultaneously, in the C-space enhancement path, the C features serve as the query, and the Z features serve as the key-value pair. Enhancement information from the Z-space is obtained through the Z→C dynamic feature enhancement attention module. Next, both paths employ a residual connection strategy, adding the calculated enhancement information to the original features with certain weighting coefficients to form the enhanced feature representation. The specific process is as follows:

[0175]

[0176] in, The residual connectivity coefficients are used to control the degree of contribution to attention enhancement. (·): C→Z attention module function, (·): Z→C attention module function.

[0177] S622, Perform final feature fusion The result and Feature concatenation is performed as follows:

[0178] in, The final fused feature representation contains bidirectional interaction information within the ZC space of the same modality.

[0179] S63, the fused feature representation is input into the trained support vector machine classifier to obtain the visual category neural decoding result.

[0180] In visual category neural decoding applications, the fused feature representation obtained in step S622 is used as input in both the training and testing phases. Specifically, the training phase uses the Z and C space features of the image modality, and obtains the fused features through bidirectional residual connection enhancement and feature concatenation. This feature contains bidirectional interaction information in the Z-C space within the image modality, and is then input into a support vector machine (SVM) classifier for training to learn the mapping relationship from the fused feature to the class label. In the testing phase, Z- and C-space features from the fMRI modality are used, which are also processed through step S62 to obtain the fused feature. Then, the test features are input into the trained SVM classifier for prediction, and the visual category neural decoding results are obtained.

[0181] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of a neural decoding system based on three-space decoupling and multimodal mutual information. Specifically, the neural decoding system based on three-space decoupling and multimodal mutual information can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other device. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0182] Please see Figure 2 , Figure 3 This invention specifically describes a neural decoding system based on three-space decoupling and multimodal mutual information, applied in electronic devices. The neural decoding system based on three-space decoupling and multimodal mutual information may include: An acquisition module is used to acquire multimodal data, which includes fMRI signals and corresponding image data; The three-space encoding module is used to encode the multimodal data into private representation, shared representation and cooperative representation respectively, and convert them into distribution parameters of private space, shared space and cooperative space respectively; The decoding and reconstruction module is used to obtain the latent representation based on the private representation, shared representation, and cooperative representation, decode the latent representation into reconstructed data, and obtain the distribution parameters of the reconstructed data; The dynamic constraint module is used to input private representation and reconstructed data into a private space mutual information discrimination network for discrimination, thereby constraining the private space mutual information; input shared representation and reconstructed data into a shared space mutual information discrimination network for discrimination, thereby constraining the shared space mutual information; and input cooperative representation and the joint reconstructed data of the two modal data into a cooperative space mutual information discrimination network for discrimination, thereby constraining the cooperative space mutual information. The training module is used to construct the total loss function and train it on the private space, shared space, and collaborative space. The fusion classification module is used for visual category neural decoding based on the dynamic feature-enhanced attention fusion method.

[0183] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0184] In addition, this embodiment of the invention also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the neural decoding method based on three-space decoupling and multimodal mutual information provided in this embodiment of the invention. Therefore, it can achieve the beneficial effects achievable by any neural decoding method based on three-space decoupling and multimodal mutual information provided in this embodiment of the invention, as detailed in the preceding embodiments, and will not be repeated here.

[0185] Figure 3 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the neural decoding method based on three-space decoupling and multimodal mutual information provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0186] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control. Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0188] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0189] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0190] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0191] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Acquire multimodal data, which includes fMRI signals and corresponding image data; The multimodal data is encoded into private representation, shared representation and cooperative representation respectively, and then converted into distribution parameters of private space, shared space and cooperative space respectively; The latent representation is obtained based on private representation, shared representation and cooperative representation. The latent representation is then decoded into reconstructed data, and the distribution parameters of the reconstructed data are obtained. Private representation and reconstructed data are input into a private space mutual information discrimination network for discrimination, constraining the private space mutual information; shared representation and reconstructed data are input into a shared space mutual information discrimination network for discrimination, constraining the shared space mutual information; cooperative representation and the joint reconstructed data of the two modal data are input into a cooperative space mutual information discrimination network for discrimination, constraining the cooperative space mutual information. Construct a total loss function and train it on the private space, shared space, and collaborative space; Visual category neural decoding based on dynamic feature-enhanced attention fusion method.

[0192] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0193] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the neural decoding method based on three-space decoupling and multimodal mutual information provided by the present invention.

[0194] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0195] Since the instructions stored in the storage medium can execute the steps in any embodiment of the neural decoding method based on three-space decoupling and multimodal mutual information provided in the embodiments of the present invention, the beneficial effects that any neural decoding method based on three-space decoupling and multimodal mutual information provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0196] The foregoing has provided a detailed description of a neural decoding method, system, storage medium, and electronic device based on three-space decoupling and multimodal mutual information provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A neural decoding method based on three-space decoupling and multimodal mutual information, characterized in that, The method includes: Acquire multimodal data, which includes fMRI signals and corresponding image data; The multimodal data is encoded into private representation, shared representation and cooperative representation respectively, and then converted into distribution parameters of private space, shared space and cooperative space respectively; The latent representation is obtained based on private representation, shared representation and cooperative representation. The latent representation is then decoded into reconstructed data, and the distribution parameters of the reconstructed data are obtained. Private representation and reconstructed data are input into a private space mutual information discrimination network for discrimination, constraining the private space mutual information; shared representation and reconstructed data are input into a shared space mutual information discrimination network for discrimination, constraining the shared space mutual information; cooperative representation and the joint reconstructed data of the two modal data are input into a cooperative space mutual information discrimination network for discrimination, constraining the cooperative space mutual information. Construct a total loss function and train it on the private space, shared space, and collaborative space; Visual category neural decoding based on dynamic feature-enhanced attention fusion method.

2. The neural decoding method based on three-space decoupling and multimodal mutual information according to claim 1, characterized in that, The multimodal data is encoded into private representation, shared representation, and cooperative representation, respectively, and then converted into distribution parameters for the private space, shared space, and cooperative space, respectively, including: Each modal data is input into a private space coding branch to obtain a private representation, and the private representation is converted into the distribution parameters of the private space coding branch; Each modal data is input into the shared space coding branch to obtain a shared representation, and the shared representation is converted into the distribution parameters of the shared space coding branch; Each modal data is input into the cooperative space coding branch to obtain a cooperative representation, and the cooperative representation is converted into the distribution parameters of the cooperative space coding branch.

3. The neural decoding method based on three-space decoupling and multimodal mutual information according to claim 2, characterized in that, The latent representation is obtained from the private representation, shared representation, and collaborative representation. This latent representation is then decoded into reconstructed data, and the distribution parameters of the reconstructed data are obtained, including: By concatenating the private representation, shared representation, and collaborative representation, a complete latent representation is obtained; The latent representation is input into the decoder to obtain the reconstructed data and the distribution parameters of the reconstructed data.

4. The neural decoding method based on three-space decoupling and multimodal mutual information according to claim 3, characterized in that, The decoder comprises a two-layer fully connected neural network. The latent representation is input into the decoder to obtain the reconstructed data and its distribution parameters, including: The latent representation is input into the first fully connected neural network to perform a transformation from the latent representation to the hidden layer, and the transformation result is obtained. The transformation result is input into the second fully connected neural network and mapped to the target mode to obtain the output result; The reconstruction distribution of the reconstructed data is defined as a Gaussian distribution, and the distribution parameters of the reconstructed data are obtained.

5. The neural decoding method based on three-space decoupling and multimodal mutual information according to claim 1, characterized in that, The private space mutual information discriminant network includes a private space-fMRI mutual information discriminant and a private space-image mutual information discriminant, and the shared space mutual information discriminant network includes a shared space-fMRI mutual information discriminant and a shared space-image mutual information discriminant. Private representation and reconstructed data are input into a private space mutual information discrimination network for discrimination, constraining the private space mutual information; shared representation and reconstructed data are input into a shared space mutual information discrimination network for discrimination, constraining the shared space mutual information. The reconstructed data, consisting of the cooperative representation and the joint data from the two modalities, is input into a cooperative space mutual information discriminant network for discrimination. The cooperative space mutual information is constrained, including: The feature representation obtained after sampling based on the private representation is concatenated with the reconstructed data of fMRI and then input into the private space-fMRI mutual information discriminator for discrimination. The feature representation obtained after sampling based on the private representation is concatenated with the feature representation of the reconstructed image data and then input into the private space-image mutual information discriminator for discrimination. The feature representation obtained after sampling based on shared representation is concatenated with the feature representation of the reconstructed fMRI data and then input into the shared space-fMRI mutual information discriminator for discrimination. The feature representation obtained after sampling based on the shared representation is concatenated with the feature representation of the reconstructed image data and then input into the shared space-image mutual information discriminator for discrimination. The reconstructed data of fMRI signals and image data are combined and then concatenated with the cooperative representation, and input into the cooperative spatial mutual information discriminant network for discrimination.

6. The neural decoding method based on three-space decoupling and multimodal mutual information according to claim 5, characterized in that, The method further includes: The variational mutual information estimation method is used to transform mutual information calculation into an optimizable lower bound estimation problem, including: Construct positive and negative sample pairs based on latent representation and reconstructed data; The positive and negative sample pairs are respectively input into the trained mutual information discriminant network to obtain the corresponding discriminant scores, and the lower bound of mutual information is calculated based on the Donsker-Varadhan variational representation.

7. The neural decoding method based on three-space decoupling and multimodal mutual information according to claim 1, characterized in that, Visual neural decoding based on dynamic feature-enhanced attention fusion methods includes: Based on a dynamic feature-enhanced attention network, bidirectional information fusion is performed between the shared space and the collaborative space to obtain enhanced information in the shared space and enhanced information in the collaborative space. The first enhanced feature is obtained by enhancing the shared space information and the original shared space features, and the second enhanced feature is obtained by enhancing the collaborative space information and the original collaborative space features. The first enhanced feature and the second enhanced feature are then concatenated to obtain a fused feature representation. The fused feature representation is input into a trained support vector machine classifier to obtain the visual category neural decoding result.

8. A neural decoding system based on three-space decoupling and multimodal mutual information, characterized in that, include: An acquisition module is used to acquire multimodal data, which includes fMRI signals and corresponding image data; The three-space encoding module is used to encode the multimodal data into private representation, shared representation and cooperative representation respectively, and convert them into distribution parameters of private space, shared space and cooperative space respectively; The decoding and reconstruction module is used to obtain the latent representation based on the private representation, shared representation, and cooperative representation, decode the latent representation into reconstructed data, and obtain the distribution parameters of the reconstructed data; The dynamic constraint module is used to input private representation and reconstructed data into a private space mutual information discrimination network for discrimination, thereby constraining the private space mutual information; input shared representation and reconstructed data into a shared space mutual information discrimination network for discrimination, thereby constraining the shared space mutual information; and input cooperative representation and the joint reconstructed data of the two modal data into a cooperative space mutual information discrimination network for discrimination, thereby constraining the cooperative space mutual information. The training module is used to construct the total loss function and train it on the private space, shared space, and collaborative space. The fusion classification module is used for visual category neural decoding based on the dynamic feature-enhanced attention fusion method.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the neural decoding method based on three-space decoupling and multimodal mutual information as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The device includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps of the neural decoding method based on three-space decoupling and multimodal mutual information as described in any one of claims 1 to 7.

Images