Multimodal fusion method, system and medium for brain magnetic resonance imaging data

Abstract

The present invention discloses a multimodal fusion method, system and medium for brain magnetic resonance imaging data. The present invention includes inputting brain magnetic resonance imaging data to be classified into a pre-trained multimodal fusion classification model to obtain a classification result. The multimodal fusion classification model includes a functional magnetic resonance imaging feature encoding module, a structural magnetic resonance imaging feature encoding module, a dual-branch cross-attention feature fusion module and a classifier module. The functional magnetic resonance imaging feature encoding module is used to encode the functional magnetic resonance imaging data into primary features, the structural magnetic resonance imaging feature encoding module is used to encode into primary features, and the dual-branch cross-attention feature fusion module is used to interactively fuse the two primary features in the same latent space to obtain fusion features. The present invention aims to make full use of the brain structural magnetic resonance imaging data and functional magnetic resonance imaging data of the subject, effectively encode spatiotemporal fusion features, and improve the performance of the multimodal fusion classification model.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification and recognition, and in particular to a multimodal fusion method, system and medium for brain magnetic resonance imaging data. Background Art

[0002] Brain magnetic resonance imaging data can capture the brain structure and functional neural patterns of the subjects, and therefore are widely used in the screening and auxiliary diagnosis of brain abnormality patterns. Among them, brain structural magnetic resonance imaging has high-resolution spatial structural features, while functional magnetic resonance imaging data can capture the pattern of functional changes over time. There are currently many deep learning-based methods to achieve the classification of brain abnormality patterns using brain magnetic resonance imaging data, but almost all methods are built for single-modality brain magnetic resonance imaging data. Although gratifying results can often be achieved in the prediction and classification tasks in single-modality brain magnetic resonance imaging data, a large number of brain abnormality patterns are often coupled with structural and temporal patterns. The encoding model trained only on single-modality data is difficult to effectively integrate spatial structural information and temporal pattern information, which makes the model unable to encode and classify spatiotemporally coupled abnormal patterns.

[0003] Multimodal MRI data fusion methods are expected to achieve spatiotemporal feature fusion and improve model performance. Currently, only a few studies have attempted to fuse structural MRI data and functional MRI data in the feature space, which has further improved the performance of the multimodal fusion classification model to a certain extent. However, due to the difference in the distribution of spatial structural features and temporal pattern signals in the feature space, the alignment method of the encoding features of the two modalities in the latent space becomes crucial. In addition, how to accurately and effectively fuse the spatial structural features with their corresponding temporal patterns and encode effective spatiotemporal fusion features is the key to achieving multimodal MRI data fusion and improving model performance. Therefore, there is an urgent need for a feature encoding and fusion method that can be used for multimodal MRI imaging data, while using the spatial structural features and functional temporal patterns of the subjects' brains to extract spatiotemporal fusion features and improve the classification performance of their abnormal brain patterns. Summary of the invention

[0004] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a multimodal fusion method, system and medium for brain magnetic resonance imaging data are provided. The present invention aims to make full use of the subject's brain structural magnetic resonance imaging data and functional magnetic resonance imaging data, effectively encode spatiotemporal fusion features, and improve the performance of the multimodal fusion classification model.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A multimodal fusion method for brain magnetic resonance imaging data, comprising inputting brain magnetic resonance imaging data to be classified into a pre-trained multimodal fusion classification model to obtain a classification result, wherein the brain magnetic resonance imaging data to be classified comprises functional magnetic resonance imaging data and structural magnetic resonance imaging data, and the multimodal fusion classification model comprises a functional magnetic resonance imaging feature encoding module, a structural magnetic resonance imaging feature encoding module, a dual-branch cross-attention feature fusion module and a classifier module, wherein the functional magnetic resonance imaging feature encoding module is used to encode the functional magnetic resonance imaging data into primary features, the structural magnetic resonance imaging feature encoding module is used to encode the structural magnetic resonance imaging data into primary features, the dual-branch cross-attention feature fusion module is used to interactively fuse the primary features of the two modal data in the same latent space to obtain fusion features, and the classifier module is used to perform classification based on the fusion features to obtain a classification result;

[0007] The primary features output by the functional magnetic resonance imaging feature encoding module include functional correlation features and timing signals in various brain regions The primary features output by the structural magnetic resonance image feature encoding module are the primary features of the structural image. The interactive fusion of the primary features of the two modal data in the same latent space to obtain the fusion features includes: and timing signals in various brain regions The learnable position encoding information and classification information are added respectively to obtain the structural primary attention features and functional primary attention features; the structural primary attention features are subjected to single-modal attention feature extraction to obtain the structural single-modal attention features. , extract the unimodal attention features from the functional primary attention features to obtain the functional unimodal attention features ; A multimodal fusion layer of P layers in series is used to fuse the structural primary attention features and the functional primary attention features in a multimodal manner. The first P / 2 layers use two multi-head attention embedding networks MSA to extract the input structural primary attention features or functional primary attention features respectively, and the latter P / 2 layers perform multimodal feature interactive fusion based on cross-attention features; The features finally obtained by the multimodal fusion layer of P layers in series are combined with the structural single-modal high-order features. , functional single modal high-order features Splicing and functional correlation features After the linear mapping layer The obtained high-level features The fusion feature is obtained by concatenating the dimensions .

[0008] Optionally, encoding the functional magnetic resonance imaging data into primary features comprises: dividing the functional magnetic resonance imaging data into brain regions using a brain atlas, thereby obtaining a time series signal of each brain region. ; Calculate the timing signals of each brain region The Pearson correlation coefficient of the whole brain was obtained , and then the functional correlation matrix of the whole brain is vector flattened to obtain the functional correlation feature and timing signals in various brain regions Primary features as output of the fMRI feature encoding module.

[0009] Optionally, the structural magnetic resonance imaging feature encoding module is a one-dimensional convolutional network, which includes a batch normalization and activation function layer and multiple convolution branches, each convolution branch includes a convolution layer and a linear layer, and encoding the structural magnetic resonance imaging data into primary features includes: dividing the structural magnetic resonance imaging data into brain regions using a brain atlas, inputting the signal of each brain region into the corresponding convolution branch, and then processing the multiple convolution branches through batch normalization and activation function layers to obtain the primary features of the structural image. Primary features as output of the structural MRI feature encoding module.

[0010] Optionally, the structure is like a primary feature and timing signals in various brain regions The function expressions of the structural primary attention features and functional primary attention features obtained by adding the learnable position encoding information and classification information are:

[0011] ,

[0012] ,

[0013] In the above formula, is the structural primary attention feature, is the concatenation operation on the dimension, and is a learnable classification token, For splicing, and Encode information for learnable positions; It is the functional primary attention feature.

[0014] Optionally, the first P / 2 layers all use two multi-head attention embedding networks MSA to extract the structural primary attention features or functional primary attention features of the input, and the function expression is:

[0015] ,

[0016] ,

[0017] In the above formula, is the structural primary attention feature output by the p-th multimodal fusion layer, is the primary attention feature of the function output by the p-th multimodal fusion layer, and For the multi-head attention embedding network MSA, is the structural primary attention feature input to the p-th multimodal fusion layer, is the functional primary attention feature of the p-th multimodal fusion layer input, and the structural primary attention feature of the 1st multimodal fusion layer input To treat structure as primary features Add learnable position encoding information and classification information to obtain structural primary attention features , the primary attention features of the first multimodal fusion layer input To transform the timing signals of various brain regions Functional primary attention features obtained by adding learnable position encoding information and classification information respectively , and there is , is the number of multimodal fusion layers, and the function expression of the multi-head attention embedding network MSA is:

[0018] ,

[0019] In the above formula, is the output of the multi-head attention embedding network MSA for the input vector X, is the attention mechanism, is the query matrix, is the key-value matrix, is the weight matrix. The query matrix, key value matrix and weight matrix are all embedded network models with learnable parameters.

[0020] Optionally, the function expression of the latter P / 2 layer for interactive fusion of multimodal features based on cross-attention features is:

[0021] ,

[0022] ,

[0023] In the above formula, is the structural interaction fusion feature output by the p-th multimodal fusion layer, is the structural primary attention feature output by the p-1th multimodal fusion layer, is the primary attention feature of the function output by the p-1th multimodal fusion layer, For the cross-attention embedding network, is the regularization layer, is the intermediate feature, It is the functional interaction fusion feature output by the p-th multimodal fusion layer, and has:

[0024] ,

[0025] In the above formula, For splicing;

[0026] The functional expression of the cross-attention embedding network is:

[0027] ,

[0028] In the above formula, It is the output of the multi-head attention embedding network MSA for the input vectors X and Y. is the cross attention mechanism, is the query matrix, is the key-value matrix, is the weight matrix, and the query matrix, key matrix and weight matrix are all embedded network models with learnable parameters; the structure interaction fusion feature output by the p-th multimodal fusion layer and functional interaction fusion features It consists of two parts: classification token feature part and cross attention feature part:

[0029] ,

[0030] ,

[0031] In the above formula, and Structural interaction fusion features The classification token feature part and the cross-attention feature part in and Functional interaction fusion features The classification token feature part and the cross attention feature part in the above method; the features obtained by connecting the multimodal fusion layer of the P layer in series and the structural single modal high-order features , functional single modal high-order features Splicing and functional correlation features After the linear mapping layer The obtained high-level features The fusion feature is obtained by concatenating the dimensions The calculation function expression is:

[0032] .

[0033] Optionally, the function expression for obtaining the classification result by performing classification based on the fusion feature is:

[0034] ,

[0035] In the above formula, is the classification result, is the classifier module, The fusion feature is input into the pre-trained multimodal fusion classification model to obtain the classification result, and the multimodal fusion classification model is trained. When training the multimodal fusion classification model, it includes training the multimodal fusion classification model based on the training set. The loss function composed of and cross entropy loss , the gradient descent method is used to update the network parameters of the multimodal fusion model, where the loss function The function expression is:

[0036] ,

[0037] In the above formula, is the loss function, represents the expected value, The training set Functional magnetic resonance imaging data samples, structural magnetic resonance imaging data samples and labels in is the cross entropy loss; the function expression of the network parameter update of the multimodal fusion model using the gradient descent method is:

[0038] ,

[0039] In the above formula, is the updated network parameter, " indicates an update operation, are the network parameters before updating, is the learning rate, is the network parameter The gradient of is the loss function.

[0040] In addition, the present invention also provides a multimodal fusion system for brain magnetic resonance imaging data, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data.

[0041] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data through a processor.

[0042] In addition, the present invention also provides a computer program product, including a computer program or an instruction, which is programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data through a processor.

[0043] Compared with the prior art, the present invention has the following advantages: the method of the present invention includes the simultaneous use of brain structural magnetic resonance imaging data and functional magnetic resonance imaging data of the same subject in the data set, and the use of brain maps to extract spatial structural features and temporal pattern signals at the brain region level, so that the features of the two modalities can be aligned in the latent space of the same dimension; a multimodal fusion module based on a cross-attention mechanism is constructed to effectively encode the features of spatiotemporal fusion; a multi-branch encoding model is constructed to retain the self-attention features of single-modal data while encoding multimodal features, further enriching the feature information in the latent space. The present invention can effectively fuse the spatial structural features of the brain and the functional temporal patterns, realize the encoding of spatiotemporal features, enrich the feature information used for classification, and improve the classification performance of the model in multimodal brain magnetic resonance imaging data for brain abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0045] Figure 2 Schematic diagram of the network structure of the dual-branch cross-attention feature fusion module in an embodiment of the present invention.

[0046] Figure 3 Schematic diagram of the network structure of two feature encoding modules in an embodiment of the present invention.

[0047] Figure 4 Schematic diagram of the training process of the multimodal fusion classification model in an embodiment of the present invention.

[0048] Figure 5 Schematic diagram of comparative experimental results of the fusion experiment in the embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below by taking brain abnormality classification for brain structural magnetic resonance imaging and functional magnetic resonance imaging as an example. It should be noted that the multimodal fusion method for brain magnetic resonance imaging data of the present invention is an information processing method in which all steps are implemented by a device with information processing function such as a computer.

[0050] This embodiment includes building a multimodal fusion classification model that can fuse two modalities of magnetic resonance imaging data for image classification for a data center that has both brain structure magnetic resonance imaging data and functional magnetic resonance imaging data. Figure 1 As shown, this embodiment provides a multimodal fusion method for brain magnetic resonance imaging data, including inputting brain magnetic resonance imaging data to be classified into a pre-trained multimodal fusion classification model to obtain a classification result, the brain magnetic resonance imaging data to be classified includes functional magnetic resonance imaging data and structural magnetic resonance imaging data, the multimodal fusion classification model includes a functional magnetic resonance imaging feature encoding module, a structural magnetic resonance imaging feature encoding module, a dual-branch cross-attention feature fusion module and a classifier module, the functional magnetic resonance imaging feature encoding module is used to encode the functional magnetic resonance imaging data into primary features, and the structural magnetic resonance imaging feature encoding module It is used to encode structural magnetic resonance imaging data into primary features. The dual-branch cross-attention feature fusion module is used to interactively fuse the primary features of two modal data in the same latent space to obtain fused features. The classifier module is used to classify based on the fused features to obtain classification results.

[0051] like Figure 2 As shown, the primary features output by the functional magnetic resonance imaging feature encoding module of this embodiment include functional correlation features and timing signals in various brain regions The primary features output by the structural magnetic resonance image feature encoding module are the primary features of the structural image. The dual-branch cross-attention feature fusion module is to establish an embedding network, the purpose of which is to fuse the spatial structural features and temporal signals at the brain region level in the same latent space using the cross-attention mechanism, thereby obtaining a spatiotemporal representation of the fusion of structural information and temporal information; at the same time, the global correlation feature information obtained above is added to the spatiotemporal representation to obtain the encoding features of multimodal fusion; Figure 2 As shown, in this embodiment, the dual-branch cross-attention feature fusion module interactively fuses the primary features of the two modal data in the same latent space to obtain fused features, including: and timing signals in various brain regions The learnable position encoding information and classification information are added respectively to obtain the structural primary attention features and functional primary attention features; the structural primary attention features are subjected to single-modal attention feature extraction to obtain the structural single-modal attention features. , extract the unimodal attention features from the functional primary attention features to obtain the functional unimodal attention features ; Use P layers in series to form a multimodal fusion layer (multimodal fusion module ) to fuse the structural primary attention features and functional primary attention features in a multi-modal manner. The first P / 2 layers use two multi-head attention embedding networks MSA to extract the input structural primary attention features or functional primary attention features respectively. The latter P / 2 layers perform multi-modal feature interactive fusion based on cross-attention features. The features finally obtained by the multi-modal fusion layer connected in series with the P layers are combined with the structural single-modal high-order features. , functional single modal high-order features Splicing and functional correlation features After the linear mapping layer The obtained high-level features The fusion feature is obtained by concatenating the dimensions .

[0052] In this embodiment, for functional magnetic resonance imaging data, brain region-level time series signals in functional magnetic resonance imaging are extracted based on brain maps. The brain maps divide brain regions into multiple brain regions according to their functions. The purpose of the time series signal extraction process is to reduce the dimensionality of the 4-dimensional functional magnetic resonance imaging from the voxel level to the brain region level, while filtering the noise signal and retaining the original time series pattern; at the same time, the functional correlation matrix can also be obtained, thereby providing global correlation information for subsequent feature encoding. Figure 3 As shown, in this embodiment, the functional magnetic resonance imaging feature encoding module encodes the functional magnetic resonance imaging data into primary features, including: encoding the functional magnetic resonance imaging data into primary features Use brain maps to divide brain regions, so as to obtain the timing signals of each brain region , whose dimensions are: ,in Indicates the length of the time series collected in the original functional magnetic resonance imaging data. Since the timing signal extraction module mainly extracts The temporal variation pattern in , so it is transposed to before entering the feature encoding module ; Calculate the timing signals of each brain region The Pearson correlation coefficient of the whole brain was obtained As global prior knowledge, feature information is further enriched at the classification layer, and there is Then the functional correlation matrix of the whole brain is vector flattened to obtain the functional correlation feature and timing signals in various brain regions As the primary feature output by the functional magnetic resonance imaging feature encoding module, the time series signal The dimension is ,in is the number of brain regions, is the length of time for signal acquisition; functional correlation matrix The dimension is In this embodiment, the brain functional magnetic resonance imaging data uses the BA512 brain atlas to divide the brain regions of the functional magnetic resonance imaging data, and divides the functional magnetic resonance imaging data into 512 brain region-level time series signals. Specifically, the original functional magnetic resonance imaging data The dimension is a four-dimensional structure ,in is the length of time for signal acquisition, is a three-dimensional structure at a collection time point. In this embodiment, , that is, using the signal values ​​of the first 128 points of the time series in each functional magnetic resonance image.

[0053] In this embodiment, for structural magnetic resonance imaging data, a brain region-level spatial feature extraction module based on a brain atlas is constructed. The spatial feature extraction module includes a brain region positioning index based on a brain atlas and a brain region feature extractor. The brain region positioning index is to locate the corresponding brain region in the input brain magnetic resonance image according to the brain region position index using the brain atlas. The brain region feature extractor is to establish an embedded network, perform feature encoding on the indexed different brain regions in turn, and map the spatial features to the latent space at the brain region level. Figure 3 As shown, in this embodiment, the structural magnetic resonance imaging feature encoding module is a one-dimensional convolutional network, which includes a batch normalization and activation function layer and a plurality of convolution branches ( ~ , each convolution branch includes a convolution layer and a linear layer, and encoding the structural magnetic resonance imaging data into primary features includes: dividing the structural magnetic resonance imaging data into brain regions (brain region 1 to brain region N) using a brain atlas, inputting the signal of each brain region into the corresponding convolution branch, and then processing the multiple convolution branches through batch normalization (BN) and an activation function layer (specifically using a ReLU activation function in this embodiment) to obtain the primary features of the structural image. As the primary feature output by the structural magnetic resonance image feature encoding module, the structural image primary feature The dimension is ,in is the number of brain regions, is the dimension of the latent space. The raw data of the structural magnetic resonance imaging data Dimension is a three-dimensional structure By extracting spatial structural features from 512 brain regions one by one and mapping them to Therefore, after the structural magnetic resonance image feature encoding module The brain region-level spatial structure feature size obtained after encoding is In this embodiment, the BA512 brain atlas is also used to divide the brain regions of the functional magnetic resonance imaging data, and the structural magnetic resonance imaging data and the functional magnetic resonance imaging data are divided into 512 brain region-level structural features and time series signals. Specifically, for the functional magnetic resonance imaging data, its original data The dimension is a four-dimensional structure ,in is the length of time for signal acquisition, is a three-dimensional structure at a collection time point. In this embodiment, , that is, using the signal values ​​of the first 128 points of the time series in each functional magnetic resonance image.

[0054] In this embodiment, the structure is like a primary feature and timing signals in various brain regions The function expressions of the structural primary attention features and functional primary attention features obtained by adding the learnable position encoding information and classification information are:

[0055] ,

[0056] ,

[0057] In the above formula, is the structural primary attention feature, is the splicing operation on the dimension, and is a learnable classification token, For splicing, and Encode information for learnable positions; is a functional primary attention feature, and has: , , , , the dimension information obtained is: , .

[0058] Treating structures like primary features and timing signals in various brain regions Add learnable position encoding information and classification information to obtain structural primary attention features and functional primary attention features After that, in this embodiment, the primary attention feature of the structure Perform unimodal attention feature extraction separately to obtain structural unimodal attention features , specifically using a single-modal attention feature extraction module Implementation, unimodal attention feature extraction module Specifically, the Vision Transformer network is used for implementation. In addition, other attention feature extraction networks can also be used as needed. Perform unimodal attention feature extraction to obtain functional unimodal attention features Specifically, a single-modal attention feature extraction module is used Implementation, unimodal attention feature extraction module The Vision Transformer network is also used for this purpose. In addition, other attention feature extraction networks can be used as needed, which can be expressed as follows:

[0059] , ,

[0060] And its dimension is , , the features extracted by the two unimodal attention feature extraction modules are composed of classification tokens and self-attention features:

[0061] , ,

[0062] In the above formula, the structural unimodal attention feature and functional unimodal attention features is the classification token feature part, and It is the self-attention feature part.

[0063] In this embodiment, a multimodal fusion module consisting of a multimodal fusion layer of P layers connected in series is used. To perform multimodal fusion of structural primary attention features and functional primary attention features, the input features of the first layer and They are used to complete the position encoding and classification token concatenation operations respectively. and , the input of each layer from the 2nd layer to the Pth layer is the encoded features of the output of the previous layer. Among them, the first P / 2 layers ( ) Both use two multi-head attention embedding networks MSA to extract the structural primary attention features or functional primary attention features of the input. The functional expression is:

[0064] ,

[0065] ,

[0066] In the above formula, is the structural primary attention feature output by the p-th multimodal fusion layer, It is the primary attention feature of the function output by the p-th multimodal fusion layer, and its dimension is: , , and For the multi-head attention embedding network MSA, is the structural primary attention feature input to the p-th multimodal fusion layer, is the functional primary attention feature of the p-th multimodal fusion layer input, and the structural primary attention feature of the 1st multimodal fusion layer input To treat structure as primary features Add learnable position encoding information and classification information to obtain structural primary attention features , the primary attention features of the first multimodal fusion layer input To transform the timing signals of various brain regions Functional primary attention features obtained by adding learnable position encoding information and classification information respectively , and there is , is the number of multimodal fusion layers, and the function expression of the multi-head attention embedding network MSA is:

[0067] ,

[0068] In the above formula, is the output of the multi-head attention embedding network MSA for the input vector X, is the attention mechanism, is the query matrix, is the key-value matrix, is the weight matrix, and the query matrix, key matrix, and weight matrix are all embedded network models with learnable parameters. The above steps are based on a shallow unimodal attention mechanism. , get the primary attention features of spatial structure and temporal signal primary attention features The attention features of these two modal data are encoded into a brain region-level latent space of the same dimension.

[0069] In this embodiment, the rear P / 2 layer ( )The function expression for interactive fusion of multimodal features based on cross-attention features is:

[0070] ,

[0071] ,

[0072] In the above formula, is the structural interaction fusion feature output by the p-th multimodal fusion layer, is the structural primary attention feature output by the p-1th multimodal fusion layer, is the primary attention feature of the function output by the p-1th multimodal fusion layer, For the cross-attention embedding network, is a regularization layer (used to regularize the features output by different modules), is the intermediate feature, is the functional interaction fusion feature output by the p-th multimodal fusion layer, and its dimension is , , intermediate features For the Spatial structural characteristics of the layer And the timing signal characteristics The concatenated features are used for subsequent encoding of cross-attention features, and their dimensions are , and there are:

[0073] ,

[0074] In the above formula, For splicing;

[0075] The functional expression of the cross-attention embedding network is:

[0076] ,

[0077] In the above formula, It is the output of the multi-head attention embedding network MSA for the input vectors X and Y. is the cross attention mechanism, is the query matrix, is the key-value matrix, is the weight matrix, and the query matrix, key matrix and weight matrix are all embedded network models with learnable parameters; the structure interaction fusion feature output by the p-th multimodal fusion layer and functional interaction fusion features It consists of two parts: classification token feature part and cross attention feature part:

[0078] ,

[0079] ,

[0080] In the above formula, and Structural interaction fusion features The classification token feature part and the cross-attention feature part in and Functional interaction fusion features The classification token feature part and the cross-attention feature part in .

[0081] The features obtained by combining the multimodal fusion layer connected in series with the P layer and the high-order features of the structural single modality , functional single modal high-order features Splicing and functional correlation features After the linear mapping layer The obtained high-level features The fusion feature is obtained by concatenating the dimensions The calculation function expression is:

[0082] .

[0083] In this embodiment, a fusion module with multimodal is constructed for single mode Parallel unimodal attention feature extraction module and , which is used to retain and extract the self-attention features in unimodal data, and construct a linear mapping layer for the prior knowledge of global correlation The prior knowledge of global correlation is used to further enrich the extracted attention features, which can be expressed as:

[0084] ,

[0085] Among them, high-level features The dimension is , Represents a feature based on functional relevance Mapped functional correlation features.

[0086] In this embodiment, the function expression of the classification result obtained by classification based on the fusion feature is:

[0087] ,

[0088] In the above formula, is the classification result, is the classifier module, The fusion feature is input into the pre-trained multimodal fusion classification model to obtain the classification result, and the multimodal fusion classification model is trained. When training the multimodal fusion classification model, it includes training the multimodal fusion classification model based on the training set. The loss function composed of and cross entropy loss , the gradient descent method is used to update the network parameters of the multimodal fusion model, where the loss function The function expression is:

[0089] ,

[0090] In the above formula, is the loss function, represents the expected value, The training set Functional magnetic resonance imaging data samples, structural magnetic resonance imaging data samples and labels in is the cross entropy loss; the function expression of the network parameter update of the multimodal fusion model using the gradient descent method is:

[0091] ,

[0092] In the above formula, is the updated network parameter, " indicates an update operation, are the network parameters before updating, is the learning rate, is the network parameter The gradient of is the loss function.

[0093] like Figure 4 As shown, the steps of training the multimodal fusion classification model in this embodiment include:

[0094] S1, for medical institutions that have both brain structural MRI data and functional MRI data, build a multimodal fusion classification model that can fuse two modalities of MRI data;

[0095] S2, randomly divide the data set into a training set and a test set; the multimodal fusion method for brain magnetic resonance imaging data in this embodiment can extract brain region-level spatial structural features and temporal pattern signals based on brain maps, and use cross attention to achieve accurate fusion of two modal data in latent space, while retaining the self-attention features of single-modal data, and further enriching the feature information used for classification by fusing multi-branch encoding features, thereby improving the classification performance of the model in multimodal brain magnetic resonance imaging data. It should be noted that the data set mentioned in this embodiment will be divided into a training set and a test set. The training set is accessible during model training, and the test set is only accessible during model testing. Specifically, in this embodiment, the medical institutions participating in model training and testing are used to obtain the three-dimensional structural functional magnetic resonance imaging data, four-dimensional functional magnetic resonance imaging and its diagnostic labels of the subjects. All medical institutions are medical institutions with corresponding qualifications, authority and have passed ethical tests, and all subjects are given abnormal labels by authoritative persons. Obtain the structural and functional magnetic resonance imaging (sMRI and fMRI) data of the subjects' brains from medical institutions. The data can be collected on-site using instruments, or pre-collected magnetic resonance imaging data can be input as needed. When collecting the subjects' brain structural and functional magnetic resonance imaging data, the equipment used is a nuclear magnetic resonance scanner, and the scanning parameters are set according to the reasonable needs of each site. Each subject in the data set is required to have structural magnetic resonance imaging data and functional magnetic resonance imaging data as well as the corresponding diagnostic label of the subject. The size of the data set depends on the number of subjects with multimodal magnetic resonance data in the medical institution. The training set and test set are divided using x-fold cross-validation;

[0096] S3, for functional magnetic resonance imaging data, extract the brain region-level temporal signals in the functional magnetic resonance imaging based on the brain atlas , and calculate the functional correlation features ; In this embodiment, the brain functional magnetic resonance imaging data of each subject is divided into 512 brain region-level time series signals using the BA512 brain atlas; specifically, the original data The dimension is a four-dimensional structure ,in is the length of time for signal acquisition, is a three-dimensional structure at a collection time point. In this embodiment, , that is, the signal values ​​of the first 128 points of the time series in each functional magnetic resonance imaging image are used. In this embodiment, the preprocessing process of the functional magnetic resonance imaging data is also included before calculating the brain region time series. The preprocessing is completed using SPM8 software. In this embodiment, for each subject, the first 5 time frames of the scan data are first discarded, and then time correction, head motion correction, spatial standardization, spatial smoothing, time filtering, irrelevant variable regression and other steps are performed in sequence to obtain the four-dimensional brain activity data corresponding to each subject. Then the BA512 brain atlas is used to divide the brain regions, and the average time series of all voxels in each brain region is used as the time series of the brain region. The modal features of the brain region level need to be extracted based on the brain atlas;

[0097] S4, for structural MRI data, extracting primary features of brain region-level structural images based on brain atlas ; In this embodiment, the primary features of the structure image are extracted by calculation The above also includes the preprocessing process of structural magnetic resonance imaging data. The preprocessing is completed using SPM8 software. In this embodiment, for each subject, the gray matter and white matter are separated, and only the three-dimensional gray matter signal of each subject is used for subsequent feature coding and diagnostic classification;

[0098] S5, construct a multi-branch cross-attention feature fusion module to transform the brain region-level structure into primary features and timing signals Use the cross-attention mechanism to fuse in the same latent space, retain the self-attention features of the single modality, and fuse them to obtain multi-modal spatiotemporal representation information;

[0099] S6, by calculating the cross entropy as the classification loss, based on the training set The loss function composed of and cross entropy loss , the gradient descent method is used to update the network parameters of the multimodal fusion model;

[0100] S7, determine whether it is necessary to continue training. If it is necessary to continue training, jump to step S2; end training, and obtain a multimodal fusion classification model that can be used for classification. When determining whether it is necessary to continue training, the required conditions can be selected as needed, such as the number of iterative training times is equal to the preset threshold or a certain indicator of the image classification model (such as accuracy, etc.) is greater than or equal to the preset threshold. The model construction, feature encoding, fusion and classification methods in this embodiment are all implemented based on Pytorch. In this embodiment, the specific parameters of model training are set as follows: the sample batch size of the training set is set to 16, the learning rate of the model is set to 1e-4, and the number of iterations is set to 80. In this embodiment, the gradient descent method is used to implement the model parameter update, which relies on the Pytorch framework. In this embodiment, multimodal magnetic resonance imaging data with diagnostic labels is used to complete the training of the classification model to achieve parameter update, which requires repeated iterations. When the maximum number of iterations is reached, the training process will stop, and a classification model suitable for multimodal magnetic resonance imaging data will be obtained.

[0101] In order to evaluate the application function of the method of this embodiment, the following method is used for verification in this embodiment: I. In this embodiment, the multimodal magnetic resonance data with diagnostic labels collected from medical institutions are first divided into training sets and test sets according to the 5-fold cross-validation method, and a classification model for multimodal magnetic resonance imaging data is constructed, and the spatiotemporal representation of single-modal features and multimodal fusion is encoded respectively. After fusion, the classifier module is used for diagnostic classification, and the cross-entropy classification loss of the model is calculated. The gradient descent method is used to update the model weights. The above model training strategy is repeated until the preset number of iterations is reached, and finally a classification model that can be used for multimodal magnetic resonance imaging data is obtained. The above process is performed 5 times. This embodiment is based on a data set composed of multimodal brain magnetic resonance data on schizophrenia in four medical institutions (AMU, PUTH6, UCLA, COBRE) for experimental verification. Each subject in the data set has brain structural magnetic resonance images, functional magnetic resonance images, and diagnostic labels provided by experienced doctors according to internationally recognized scales. The size of the brain structural magnetic resonance imaging data is , the size of the functional magnetic resonance imaging data is , the diagnostic labels are 0 and 1, the diagnostic label 0 is set as normal, and 1 is set as abnormal. Two verification methods, experimental comparison and literature comparison, are adopted. First, the performance of the classification model of single-modal data is compared with the classification performance after multimodal fusion. Among them, several widely used methods based on structural magnetic resonance imaging data and functional magnetic resonance imaging data are selected to train single-modal classification models. Secondly, by comparing the SOTA method (the most advanced research method) of multimodal fusion in recent years, the performance is compared with the same data in the same medical institution. All experimental result indicators are the average results of three repeated experiments. Finally, the effectiveness of the multimodal fusion method of this embodiment is verified by repeated ablation experiments.

[0102] The method proposed in this embodiment and the SOTA (currently most advanced research method) method for comparison are tested on the schizophrenia dataset for diagnostic classification. Applicable SOTA methods and SOTA methods tested on the corresponding datasets include ResNet-50, Conformer, ROI-CNN, MLP, STAGIN, DGST-Former, multi-layer perceptron MLP, convolutional neural network CNN, Liu 2022, Zou 2020, Kim 2016r and Savio 2015. ResNet-50, see the literature: K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition,” In proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016, doi:10.1109 / CVPR.2016.90. Conformer, see: Z. Peng etal., "Conformer: Local Features Coupling Global Representations for Recognition and Detection," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 9454-9468, Aug. 2023, doi: 10.1109 / TPAMI.2023.3243048. STAGIN, see: BH Kim, JC Ye, J. J, Kim, “LearningDynamic Graph Representation of Brain Connectome with Spatio-TemporalAttention,” In Proceedings of the Advances inneural Information ProcessingSystems, 2021.DGST-Former, see reference: Z. Fan, J. Su, J. Xu, D. Hu and L.-L. Zeng, “DGST-Former: Dynamic Graph with Spatio-Temporal Transpose-Transformer for Brain fMRI Data,” In proceedings of 2023 7th Asian Conference on Artificial Intelligence Technology, pp. 530-535, 2023, doi:10.1109 / ACAIT60137.2023.10528622. Liu 2022, see reference: R. Liu, Z.-A. Huang, Y. Hu, Z. Zhu, K.-C. Wong and K. C. Tan, ”Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI,” in IEEE Transactions on Neural Networks and Learning Systems, 2022, in press, doi:10.1109 / TNNLS.2022.3219551. Zou 2020, see reference: H. Zou and J. Yang, “Multiple functional connectivity networks fusion for schizophrenia diagnosis,” Med. Biol. Eng. Comput., vol. 58, no. 8, pp. 1779–1790, Aug. 2020.Kim 2016, see: J. Kim, VD Calhoun, E. Shim, and J.-H. Lee, “Deep neural network with weight sparsity control and pre-training extracts hierarchical features and enhances performance: Evidence from wholebrain resting-statefunctional connectivity patterns of schizophrenia,” NeuroImage, vol. 124, pp.127–146, Jan. 2016. Savio 2015, see: A. Savio and M. Grana, “Local activity features for computer aided diagnosis of schizophrenia on resting-statefMRI,” Neurocomputing, vol. 164, pp. 154–161, Sep. 2015.

[0103] Table 1 shows a comparison of the experimental results of the method of this embodiment (MBCAT) and some of the SOTA methods mentioned above that are trained using only single-modality magnetic resonance imaging data.

[0104] Table 1: Comparison of model performance on multimodal data and unimodal data

[0105]

[0106] In Table 1, fMRI is functional magnetic resonance imaging data, sMRI is structural magnetic resonance imaging data, ACC is the accuracy of model classification (%), Spe is the specificity of the classification result (%), indicating the ability to correctly classify non-patients; Sen is the sensitivity of the classification result (%), indicating the ability to correctly classify patients; AUC is the area under the receiver operating characteristic curve (ROC), which is usually used to measure the quality of the model; F1 score is an indicator used to measure the accuracy of the binary classification model, which can take into account both the accuracy and recall of the model. The reported variance is in brackets in the table. As shown in Table 1, the method (MBCAT) proposed in this embodiment is superior to the model performance of only using a single modality training in all indicators.

[0107] In comparison with other SOTA methods, the method of this embodiment and the SOTA method both use the same data on the COBRE dataset to train and test the classification model. The model test indicators are shown in Table 2.

[0108] Table 2 Comparison with SOTA methods on COBRE dataset

[0109]

[0110] In Table 2, fMRI is functional magnetic resonance imaging data, sMRI is structural magnetic resonance imaging data, the first row directly uses the layered perceptron MLP to classify functional magnetic resonance imaging data, the second row directly uses the convolutional neural network CNN to classify structural magnetic resonance imaging data sMRI, and the remaining methods all contain corresponding types of classifiers, and the classifier module used by the method (MBCAT) proposed in this embodiment is a piecewise linear classifier DNN. As shown in Table 2, the classification model performance of the method (MBCAT) proposed in this embodiment trained on the COBRE dataset is better than other SOTA methods.

[0111] In addition, an ablation experiment was designed for the method proposed in the embodiment, and the ablation experiment results obtained are as follows: Figure 5 As shown, Figure 5 Among the various indicators: ACC is the accuracy of model classification (%), AUC is the area under the receiver operating characteristic curve (ROC), which is usually used to measure the quality of the model; F1 score is an indicator used to measure the accuracy of the binary classification model, which can take into account both the accuracy and recall rate of the model; Sen is the sensitivity of the classification result (%), which indicates the ability to correctly classify patients; Spe is the specificity of the classification result (%), which indicates the ability to correctly classify non-patients. Figure 5 Each indicator in the above table includes 5 data: The first data multimodal fusion classification model removes the single-modal attention feature extraction module and The test results of the second data multimodal fusion classification model are to remove the linear mapping layer The test results show that the third data multimodal fusion classification model removes the single-modal attention feature extraction module. and and the linear mapping layer The test results of the fourth data multimodal fusion classification model are as follows: The fifth data is the test result of the complete multimodal fusion classification model. Figure 5 It can be seen that the complete multimodal fusion classification model constructed by the method of this embodiment can significantly improve the performance of the classification model, which proves the effectiveness of the method proposed in this embodiment.

[0112] In summary, the multimodal fusion method for brain magnetic resonance imaging data in this embodiment constructs a spatial feature encoding module and a temporal pattern encoding module based on the brain region level as latent space embedding modules, and extracts the spatial representation and temporal pattern at the brain region level based on the brain map; a dual-branch cross-attention feature fusion module based on multi-branch cross-attention is established to fuse the structural features with high spatial resolution and the functional patterns with temporal information in the same latent space to obtain a multimodal representation; the classifier is then used to classify the fused representation, the cross entropy loss is calculated, and the cross entropy classification loss is used for reverse conduction to realize the model parameter update in the multimodal fusion model; the above steps are repeated until the training is completed. Compared with the existing methods, the multimodal fusion method for brain magnetic resonance imaging data in this embodiment can extract brain region-level spatial structure features and temporal pattern signals based on brain maps, and use cross-attention to achieve accurate fusion of two modal data in latent space. It can make full use of the spatial structure information and temporal pattern information in brain magnetic resonance imaging data of different modalities, while retaining the self-attention characteristics of single-modal data. By fully integrating spatial and temporal representations in the latent space to fuse multi-branch encoding features, the feature information used for classification is further enriched, thereby improving the classification performance of the classification model in multimodal brain magnetic resonance imaging data.

[0113] In addition, this embodiment also provides a multimodal fusion system for brain magnetic resonance imaging data, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data.

[0114] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data through a processor.

[0115] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data through a processor.

[0116] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multimodal fusion method for brain magnetic resonance imaging data. It is characterized in that The method comprises inputting brain magnetic resonance imaging data to be classified into a pre-trained multimodal fusion classification model to obtain a classification result, wherein the brain magnetic resonance imaging data to be classified comprises functional magnetic resonance imaging data and structural magnetic resonance imaging data, and the multimodal fusion classification model comprises a functional magnetic resonance imaging feature encoding module, a structural magnetic resonance imaging feature encoding module, a dual-branch cross-attention feature fusion module and a classifier module, wherein the functional magnetic resonance imaging feature encoding module is used to encode the functional magnetic resonance imaging data into primary features, the structural magnetic resonance imaging feature encoding module is used to encode the structural magnetic resonance imaging data into primary features, the dual-branch cross-attention feature fusion module is used to interactively fuse the primary features of the two modal data in the same latent space to obtain fusion features, and the classifier module is used to perform classification based on the fusion features to obtain a classification result; The primary features output by the functional magnetic resonance imaging feature encoding module include functional correlation features and timing signals in various brain regions The primary features output by the structural magnetic resonance image feature encoding module are the primary features of the structural image. ; The interactive fusion of the primary features of the two modal data in the same latent space to obtain the fusion feature includes: and timing signals in various brain regions Add learnable position encoding information and classification information to obtain structural primary attention features and functional primary attention features respectively; The structural primary attention feature is extracted by single-modal attention feature to obtain the structural single-modal attention feature , extract the unimodal attention features from the functional primary attention features to obtain the functional unimodal attention features ; A multimodal fusion layer of P layers in series is used to fuse the structural primary attention features and the functional primary attention features in a multimodal manner. The first P / 2 layers use two multi-head attention embedding networks MSA to extract the input structural primary attention features or functional primary attention features respectively, and the latter P / 2 layers perform multimodal feature interactive fusion based on cross-attention features; The features finally obtained by the multimodal fusion layer of P layers in series are combined with the structural single-modal high-order features. , functional single-modal high-order features Splicing and functional correlation features After the linear mapping layer The obtained high-level features The fusion feature is obtained by concatenating the dimensions .

2. The multimodal fusion method for brain magnetic resonance imaging data according to claim 1, It is characterized in that The encoding of the functional magnetic resonance imaging data into primary features includes: dividing the functional magnetic resonance imaging data into brain regions using a brain atlas, thereby obtaining a time series signal of each brain region. ; Calculate the timing signals of each brain region The Pearson correlation coefficient of the whole brain was obtained , and then the functional correlation matrix of the whole brain is vector flattened to obtain the functional correlation feature and timing signals in various brain regions Primary features as output of the fMRI feature encoding module.

3. The multimodal fusion method for brain magnetic resonance imaging data according to claim 1, It is characterized in that The structural magnetic resonance imaging feature encoding module is a one-dimensional convolutional network, which includes a batch normalization and activation function layer and multiple convolution branches, each of which includes a convolution layer and a linear layer. The encoding of the structural magnetic resonance imaging data into primary features includes: dividing the structural magnetic resonance imaging data into brain regions using a brain atlas, inputting the signal of each brain region into the corresponding convolution branch, and then processing the multiple convolution branches through batch normalization and activation function layers to obtain the primary features of the structural image. Primary features as output of the structural MRI feature encoding module.

4. The multimodal fusion method for brain magnetic resonance imaging data according to claim 1, It is characterized in that The structure is like a primary feature and timing signals in various brain regions The function expressions of the structural primary attention features and functional primary attention features obtained by adding the learnable position encoding information and classification information are: ， ， In the above formula, is the structural primary attention feature, is the splicing operation on the dimension, and is a learnable classification token, For splicing, and Encode information for learnable positions; It is the functional primary attention feature.

5. The multimodal fusion method for brain magnetic resonance imaging data according to claim 1, It is characterized in that The first P / 2 layers all use two multi-head attention embedding networks MSA to extract the structural primary attention features or functional primary attention features of the input. The function expression is: ， ， In the above formula, is the structural primary attention feature output by the p-th multimodal fusion layer, is the primary attention feature of the function output by the p-th multimodal fusion layer, and For the multi-head attention embedding network MSA, is the structural primary attention feature input to the p-th multimodal fusion layer, is the functional primary attention feature of the p-th multimodal fusion layer input, and the structural primary attention feature of the 1st multimodal fusion layer input To treat structure as primary features Add learnable position encoding information and classification information to obtain structural primary attention features , the primary attention features of the first multimodal fusion layer input To transform the timing signals of various brain regions Functional primary attention features obtained by adding learnable position encoding information and classification information respectively , and there is , is the number of multimodal fusion layers, and the function expression of the multi-head attention embedding network MSA is: ， In the above formula, is the output of the multi-head attention embedding network MSA for the input vector X, is the attention mechanism, is the query matrix, is the key-value matrix, is the weight matrix. The query matrix, key value matrix and weight matrix are all embedded network models with learnable parameters.

6. The multimodal fusion method for brain magnetic resonance imaging data according to claim 1, It is characterized in that The functional expression of the latter P / 2 layer for interactive fusion of multimodal features based on cross-attention features is: ， ， In the above formula, is the structural interaction fusion feature output by the p-th multimodal fusion layer, is the structural primary attention feature output by the p-1th multimodal fusion layer, is the primary attention feature of the function output by the p-1th multimodal fusion layer, For the cross-attention embedding network, is the regularization layer, is the intermediate feature, It is the functional interaction fusion feature output by the p-th multimodal fusion layer, and has: ， In the above formula, For splicing; The functional expression of the cross-attention embedding network is: ， In the above formula, It is the output of the multi-head attention embedding network MSA for the input vectors X and Y. is the cross attention mechanism, is the query matrix, is the key-value matrix, is the weight matrix, and the query matrix, key matrix and weight matrix are all embedded network models with learnable parameters; the structure interaction fusion feature output by the p-th multimodal fusion layer and functional interaction fusion features It consists of two parts: classification token feature part and cross attention feature part composition: ， ， In the above formula, and Structural interaction fusion features The classification token feature part and the cross-attention feature part in and Functional interaction fusion features The classification token feature part and the cross attention feature part in the above method; the features obtained by connecting the multimodal fusion layer of the P layer in series and the structural single modal high-order features , functional single-modal high-order features Splicing and functional correlation features After the linear mapping layer The obtained high-level features The fusion feature is obtained by concatenating the dimensions The calculation function expression is: 。 7. The multimodal fusion method for brain magnetic resonance imaging data according to claim 1, It is characterized in that The function expression of the classification result obtained by classification based on the fusion feature is: ， In the above formula, is the classification result, is the classifier module, The fusion feature is input into the pre-trained multimodal fusion classification model to obtain the classification result, and the multimodal fusion classification model is trained. When training the multimodal fusion classification model, it includes training the multimodal fusion classification model based on the training set. The loss function composed of and cross entropy loss , the gradient descent method is used to update the network parameters of the multimodal fusion model, where the loss function The function expression is: ， In the above formula, is the loss function, represents the expected value, The training set Functional magnetic resonance imaging data samples, structural magnetic resonance imaging data samples and labels in is the cross entropy loss; the function expression of the network parameter update of the multimodal fusion model using the gradient descent method is: ， In the above formula, is the updated network parameter, " indicates an update operation, are the network parameters before updating, is the learning rate, is the network parameter The gradient of is the loss function.

8. A multimodal fusion system for brain magnetic resonance imaging data, comprising a microprocessor and a memory connected to each other, It is characterized in that The microprocessor is programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, It is characterized in that The computer program or instruction is programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data as described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, It is characterized in that The computer program or instruction is programmed or configured to execute the multimodal fusion method for brain magnetic resonance imaging data as described in any one of claims 1 to 7 through a processor.

Images