A method for MRI data classification based on a low-rank multi-modal fusion network

Abstract

This invention provides an MRI data classification method based on a low-rank multimodal fusion network. First, raw sMRI and fMRI data are acquired and preprocessed. Second, feature vectors are extracted from both sMRI and fMRI data using a low-rank multimodal fusion network. These extracted feature vectors are then fused using a low-rank tensor. The fused features are input into a support vector machine for classification. The low-rank multimodal fusion network is trained end-to-end based on an overall loss function. Finally, the test set is input into the trained low-rank multimodal fusion network to obtain the output category of the test samples. This invention reduces the heterogeneity between different modalities, enhances the multimodal fusion effect, and improves the accuracy of the multimodal classification model.

Description

Technical Field

[0001] This invention belongs to the field of multimodal fusion and pattern recognition technology, specifically relating to an MRI data classification method based on a low-rank multimodal fusion network. Background Technology

[0002] Magnetic Resonance Imaging (MRI) primarily obtains images of the internal structures of the human body by capturing the electron spin resonance phenomenon within tissues. It boasts advantages such as being non-invasive, radiation-free, and having high resolution, making it widely used in medical diagnosis and research. MRI offers various image modalities, including structural magnetic resonance imaging (sMRI), functional magnetic resonance imaging (fMRI), susceptibility-weighted imaging (SWI), and diffusion-weighted imaging (DWI). These different modalities represent different information and exhibit a certain degree of complementarity. Joint learning using multimodal MRI images can yield comprehensive, reliable, and accurate feature representations of the described object, improving the accuracy of classification results.

[0003] Both sMRI and fMRI are based on magnetic resonance imaging technology to acquire image information. sMRI is mainly used to obtain structural information about human tissues and organs, such as the anatomical structure of the brain, while fMRI is used to capture and analyze dynamic changes in brain function. These two different modalities of sMRI and fMRI reflect different complementary information about the same subject's brain. Therefore, researchers use multimodal fusion methods to fuse image data from sMRI and fMRI to improve the accuracy of MRI image data classification.

[0004] Currently, common methods such as concatenation, weighted summation, and averaging are used to fuse features from multiple modalities. However, the aforementioned multimodal fusion methods ignore the heterogeneity between features from different modalities and fail to fully and effectively fuse complementary information between different modalities, resulting in a decline in the classification performance after fusion. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an MRI data classification method based on low-rank multimodal fusion.

[0006] This invention discloses an MRI data classification method based on low-rank multimodal fusion, comprising the following steps:

[0007] Step 1, Obtain the original multimodal dataset:

[0008] The multimodal raw dataset includes sMRI and fMRI data;

[0009] Step 2: Preprocess the sMRI data obtained in Step 1;

[0010] Step 3: Preprocess the fMRI data obtained in Step 1;

[0011] Step 4: Based on a low-rank multimodal fusion network, feature extraction, feature fusion, and classification are performed on the preprocessed sMRI and fMRI data respectively.

[0012] Step 4.1: Construct 3DCNN and DNN feature extraction networks for sMRI and fMRI data respectively;

[0013] Step 4.2: Use 3DCNN and DNN feature extraction networks to extract the same-dimensional feature vector F from the preprocessed sMRI and fMRI data, respectively. s ,F f ∈R d-1 ;

[0014] Step 4.3, extract the feature vectors F from the sMRI and fMRI modalities. s F f By appending 1 to each vector, we obtain the vector z. s =[F s ,1] T ,z f =[F f ,1] T ,z s ,z f ∈R d ;

[0015] Step 4.4, convert the feature vectors z of the sMRI and fMRI modalities s and z f Perform tensor fusion:

[0016]

[0017] in, Represented as tensor outer product, Represented as a tensor after feature fusion.

[0018] Step 4.5: Fuse the features from Step 4.4 into a representation tensor. Generate a vector h through a linear layer:

[0019]

[0020] Where · represents the tensor dot product, and b represents the bias vector. Represented as a tensor The weight,

[0021]

[0022] Step 4.6, to reduce the computational cost of vector h, first convert the weight tensor... Decompose into r low-rank factors specific to each mode:

[0023]

[0024] in, and Let i be the i-th low-rank weighting factor of sMRI and fMRI, respectively.

[0025] Then the weight tensor The decomposition is introduced into vector h:

[0026]

[0027] in, Represented as element-wise product;

[0028] Step 4.7: Input the fused feature vector h into the support vector machine for classification;

[0029] Step 5: Based on the overall loss function, train the low-rank multimodal fusion network in an end-to-end manner:

[0030] The overall loss function is expressed as:

[0031] L = L Hinge +λL DDM

[0032] Where λ represents the balance coefficient; L Hinge Represented as classification loss, L DDM This is expressed as the loss due to differences in distribution between modes;

[0033] Step 6: Input the test set into the trained low-rank multimodal fusion network to obtain the output category of the test sample.

[0034] Furthermore, step 2 involves preprocessing the sMRI data acquired in step 1, including the following steps:

[0035] Step 2.1: Adjust the inconsistency of brain image angles caused by subject head movement during sMRI data acquisition by correcting the origin of the anterior commissure-posterior commissure.

[0036] Step 2.2: A nonparametric gray-level non-uniformity normalization algorithm is used to correct the gray-level non-uniformity of sMRI images;

[0037] Step 2.3: Map each sMRI brain image to the MNI305 standard space using image registration methods;

[0038] Step 2.4: Use spatial downsampling to standardize all extracted brain images and place them into a sample space with uniform resolution.

[0039] Furthermore, step 3 involves preprocessing the fMRI data acquired in step 1, including the following steps:

[0040] Step 3.1: Obtain brain gray matter images through slice temporal layer correction, head motion correction, spatial standardization, spatial smoothing, elimination of linear trends, temporal bandpass filtering, and covariate regression;

[0041] Step 3.2: Using the physiological brain atlas, the gray matter image of the brain is segmented into multiple brain regions with different functions, and the average time series of all voxels in each brain region is extracted;

[0042] Step 3.3: Calculate the Pearson correlation coefficient between any two brain regions for each sample and construct the whole-brain functional connectivity matrix;

[0043] Step 3.4: Take the upper triangular part of the functional connection matrix and expand it into a one-dimensional vector.

[0044] Furthermore, classification loss L Hinge Represented as:

[0045]

[0046] in, This means predicting the i-th sample as the c-th sample. i The output obtained from the class sample, c i Let represent the true label of the i-th sample. Let N represent the output obtained by predicting the i-th sample as the j-th class, where N represents the number of samples and Δ represents the hyperparameter.

[0047] Furthermore, the distribution difference loss between modes L DDM Represented as:

[0048]

[0049] in, Let F be the feature vector extracted from the sMRI modality of the i-th sample. s The j-th element, Let F be the feature vector extracted from the fMRI modality of the i-th sample.f The j-th element.

[0050] Furthermore, in step 4.1, the 3DCNN consists of two 3D convolutional layers, two batch normalization layers, two max pooling layers, one fully connected layer, and one ReLU activation layer; the DNN consists of one fully connected layer, one Dropout layer, one ReLU activation layer, and two batch normalization layers.

[0051] Beneficial effects:

[0052] 1. The MRI data classification method based on low-rank multimodal fusion network proposed in this invention introduces the distribution difference between modalities into the loss function for end-to-end training, so that the model can reduce the distribution difference between modalities during training, eliminate the heterogeneity between different modal data, capture the inter-modal correlation information, and overcome the shortcomings of traditional multimodal fusion methods that ignore the heterogeneity between different modalities.

[0053] 2. The MRI data classification method based on low-rank multimodal fusion network proposed in this invention effectively fuses the features of two different modalities, sMRI and fMRI, by using a low-rank multimodal fusion network, thereby improving the accuracy of the classification model.

[0054] 3. This invention uses 3DCNN and DNN to represent the complex and nonlinear information in sMRI and fMRI data, respectively, which facilitates the subsequent use of low-rank multimodal fusion methods to fuse different modal data. Attached image description:

[0055] Figure 1 This is a schematic diagram of the structure of the low-rank multimodal fusion network of the present invention;

[0056] Figure 2 This is a schematic diagram of the 3DCNN structure of the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of the DNN of the present invention;

[0058] Figure 4 This is a schematic diagram of the end-to-end training results of the present invention. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0060] Based on a specific theme (e.g., adults and children, high intelligence and low intelligence), participants were categorized into two corresponding classes: Class A and Class B. Assuming there are currently 312 participants in Class A and 334 in Class B, 80% were selected as the training sample and 20% as the test sample. All operations were performed under a Windows operating system. The experimental hardware consisted of: an Intel Xeon(R) Silver 4210 CPU (2.20GHz + 2.19GHz, 2 processors) and an NVIDIA GeForce RTX 3090 graphics card.

[0061] like Figure 1 As shown, an MRI data classification method based on a low-rank multimodal fusion network includes the following steps:

[0062] Step 1, Obtain the original dataset:

[0063] Based on the research topic, the subjects' brain MRI images were divided into Class A and Class B, and sMRI and fMRI data of the same subject were obtained.

[0064] Step 2, preprocessing the sMRI data acquired in Step 1:

[0065] Step 2.1: Adjust the inconsistency of brain image angles caused by subject head movement during sMRI data acquisition by correcting the origin of the anterior commissure-posterior commissure.

[0066] Step 2.2: A nonparametric gray-level non-uniformity normalization algorithm is used to correct the gray-level non-uniformity of sMRI images;

[0067] Step 2.3: Use image registration methods to map individual sMRI brain images into the MNI305 standard space;

[0068] Step 2.4: Spatial downsampling is used to normalize all extracted brain images to a 64×64×64 resolution sample space to eliminate differences between sMRI brain images obtained from different imaging devices.

[0069] Step 3: Preprocess the fMRI data obtained in Step 1:

[0070] Step 3.1: Obtain brain gray matter images through slice temporal layer correction, head motion correction, spatial standardization, spatial smoothing, elimination of linear trends, temporal bandpass filtering, and covariate regression;

[0071] Step 3.2: The brain was divided into 116 functional brain regions using the physiological anatomy autolabelling AAL template, and the average time series of all voxels in each brain region was extracted.

[0072] Step 3.3: Calculate the Pearson correlation coefficient between any two brain region time series to obtain a 116×116 whole-brain functional connectivity matrix;

[0073] Step 3.4: Take the upper triangular part of the functional connection matrix and expand it into a 6670×1-dimensional vector;

[0074] Step 4: Based on a low-rank multimodal fusion network, feature extraction, feature fusion, and classification are performed on the preprocessed sMRI and fMRI data respectively.

[0075] Step 4.1: Construct 3DCNN and DNN feature extraction networks for sMRI and fMRI data respectively;

[0076] The structure of 3DCNN is as follows: Figure 2 As shown, 3DCNN consists of two 3D convolutional layers, two batch normalization layers, two max pooling layers, one fully connected layer, and one ReLU activation layer. The kernel size of the 3D convolutional layers is 1×2×2, and the number of nodes in the fully connected layer is 1000.

[0077] The structure of a DNN is as follows: Figure 3 As shown, the DNN consists of one fully connected layer, one Dropout layer, one ReLU activation layer, and two batch normalization layers. The number of nodes in the fully connected layer is 1000.

[0078] Step 4.2: Use 3DCNN and DNN feature extraction networks to extract the same-dimensional feature vector F from the preprocessed sMRI and fMRI data, respectively. s ,F f ∈R 1000 ;

[0079] Step 4.3, extract the feature vectors F from the sMRI and fMRI modalities. s F f By appending 1 to each vector, we obtain the vector z. s =[F s ,1] T ,z f =[F f ,1] T ,z s ,z f ∈R 1001 ;

[0080] Step 4.4, convert the feature vectors z of the sMRI and fMRI modalities s and z f Perform tensor fusion:

[0081]

[0082] in, Represented as tensor outer product, Represented as a tensor after feature fusion.

[0083] Step 4.5: Fuse the features from Step 3.4 into a representation tensor. Generate a vector h through a linear layer:

[0084]

[0085] Where · represents the tensor dot product, and b represents the bias vector. Here we take d h =128, Represented as a tensor The weight,

[0086] Step 4.6, to reduce the computational cost of vector h, first convert the weight tensor... It is decomposed into r low-rank factors specific to each mode, where r = 4;

[0087]

[0088] in, and Let i be the i-th low-rank weighting factor of sMRI and fMRI, respectively.

[0089] Then the weight tensor By introducing the calculation of h into the decomposition formula, we can obtain:

[0090]

[0091] in, Represented as element-wise product;

[0092] Step 4.7: Input the fused feature vector h into the support vector machine for classification;

[0093] Step 5: Based on the overall loss function, train the low-rank multimodal fusion network in an end-to-end manner, where the overall loss function is expressed as:

[0094] L = L Hinge +λL DDM

[0095] λ represents the balance coefficient, which is taken as λ = 1 here; L Hinge Represented as classification loss, L DDM This is expressed as the loss due to differences in distribution between modes;

[0096]

[0097] in, This means predicting the i-th sample as the c-th sample. i The output obtained from the class sample, c i Let represent the true label of the i-th sample. This represents the output obtained by predicting the i-th sample as the j-th class, where N represents the number of samples and Δ represents the hyperparameter, which is set to 1 by default.

[0098] Intermodal distribution difference loss L DDM Represented as:

[0099]

[0100] in, Let F be the feature vector extracted from the sMRI modality of the i-th sample. s The j-th element, Let F be the feature vector extracted from the fMRI modality of the i-th sample. f The j-th element;

[0101] The model training process is as follows Figure 4 As shown, with the increase of the training step size, the training loss train_loss decreases and eventually tends to converge, while the model training accuracy train_accuracy continues to increase, and the final training accuracy of the model can reach 92%.

[0102] Step 6: Input the test sample into the trained low-rank multimodal fusion network to determine whether the test sample belongs to class A or class B, thus realizing the application of this invention in the classification and recognition of different brain states.

[0103] The experimental results of this invention are shown in Table 1. The evaluation results show that the proposed method achieves a classification accuracy of 78.88%, a sensitivity of 71.79%, and a specificity of 71.79%, which surpasses single-modal methods and some common multimodal fusion methods. Here, sMRI represents the classification model trained using preprocessed sMRI as input to a 3DNN network; fMRI represents the classification model trained using a 6670×1 feature vector extracted from the original fMRI as input to a DNN network. The experimental results in Table 1 show that the multimodal fusion method of this invention is superior to single-modal methods and also superior to multimodal fusion methods such as splicing, DCL, and DCCA, indicating that the proposed method can effectively fuse sMRI and fMRI modalities.

[0104] Table 1

[0105]

[0106] The experimental results used in this embodiment of the invention are shown in Table 2. These results illustrate the test results for two variants of this method, namely (w / o)L. Hinge (w / o)L DDM (w / o)L Hinge The loss function of this invention removes the inter-modal distribution difference loss L. DDM (w / o)L DDM To remove the classification loss L from the loss function of this invention Hinge The experimental results in Table 2 show that the multimodal fusion method of the present invention achieved better results than other variants, indicating that reducing the distribution differences between modes helps to improve the multimodal fusion effect.

[0107] Table 2

[0108]

[0109] In addition, it should be noted that the method in this embodiment can be used not only for binary classification, but also for multiple classifications such as three, four, etc.

[0110] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. A method for classifying MRI data based on a low-rank multimodal fusion network, characterized in that, Includes the following steps: Step 1, Obtain the original multimodal dataset: The multimodal raw dataset includes sMRI and fMRI data; Step 2: Preprocess the sMRI data obtained in Step 1; Step 3: Preprocess the fMRI data obtained in Step 1; Step 4: Based on a low-rank multimodal fusion network, feature extraction, feature fusion, and classification are performed on the preprocessed sMRI and fMRI data respectively. Step 4.1: Construct 3DCNN and DNN feature extraction networks for sMRI and fMRI data respectively; Step 4.2: Use 3DCNN and DNN feature extraction networks to extract the same-dimensional feature vector F from the preprocessed sMRI and fMRI data, respectively. s ,F f ∈R d-1 ; Step 4.3, extract the feature vectors F from the sMRI and fMRI modalities. s F f By appending 1 to each vector, we obtain the vector. Step 4.4, convert the feature vectors z of the sMRI and fMRI modalities s and z f Perform tensor fusion: in, Represented as tensor outer product, Represented as a tensor after feature fusion. Step 4.5: Fuse the features from Step 4.4 into a representation tensor. Generate a vector h through a linear layer: Where · represents the tensor dot product, and b represents the bias vector. Represented as a tensor The weight, Step 4.6, to reduce the computational cost of vector h, first decompose the weight tensor w into r low-rank factors specific to each mode: in, and Let i be the i-th low-rank weighting factor of sMRI and fMRI, respectively. Then, the decomposition formula of the weight tensor w is introduced into the vector h: in, Represented as element-wise product; Step 4.7: Input the fused feature vector h into the support vector machine for classification; Step 5: Based on the overall loss function, train the low-rank multimodal fusion network in an end-to-end manner: The overall loss function is expressed as: L=L Hinge +λL DDM Where λ represents the balance coefficient; L Hinge Represented as classification loss, L DDM This is expressed as the loss due to differences in distribution between modes; Step 6: Input the test set into the trained low-rank multimodal fusion network to obtain the output category of the test sample.

2. The MRI data classification method based on a low-rank multimodal fusion network according to claim 1, characterized in that, Step 2, the preprocessing of the sMRI data acquired in Step 1 includes the following steps: Step 2.1: Adjust the inconsistency of brain image angles caused by subject head movement during sMRI data acquisition by correcting the origin of the anterior commissure-posterior commissure. Step 2.2: A nonparametric gray-level non-uniformity normalization algorithm is used to correct the gray-level non-uniformity of sMRI images; Step 2.3: Map each sMRI brain image to the MNI305 standard space using image registration methods; Step 2.4: Use spatial downsampling to standardize all extracted brain images and place them into a sample space with uniform resolution.

3. The MRI data classification method based on a low-rank multimodal fusion network according to claim 1, characterized in that, Step 3, the preprocessing of the fMRI data acquired in Step 1 includes the following steps: Step 3.1: Obtain brain gray matter images through slice temporal layer correction, head motion correction, spatial standardization, spatial smoothing, elimination of linear trends, temporal bandpass filtering, and covariate regression; Step 3.2: Using the physiological brain atlas, the gray matter image of the brain is segmented into multiple brain regions with different functions, and the average time series of all voxels in each brain region is extracted; Step 3.3: Calculate the Pearson correlation coefficient between any two brain regions for each sample and construct the whole-brain functional connectivity matrix; Step 3.4: Take the upper triangular part of the functional connection matrix and expand it into a one-dimensional vector.

4. The MRI data classification method based on a low-rank multimodal fusion network according to claim 1, characterized in that, Classification loss L Hinge Represented as: in, This means predicting the i-th sample as the c-th sample. i The output obtained from the class sample, c i Let represent the true label of the i-th sample. Let N represent the output obtained by predicting the i-th sample as the j-th class, where N represents the number of samples and Δ represents the hyperparameter.

5. The MRI data classification method based on a low-rank multimodal fusion network according to claim 1, characterized in that, Intermodal distribution difference loss L DDM Represented as: in, Let F be the feature vector extracted from the sMRI modality of the i-th sample. s The j-th element, Let F be the feature vector extracted from the fMRI modality of the i-th sample. f The j-th element.

6. The MRI data classification method based on a low-rank multimodal fusion network according to claim 1, characterized in that, In step 4.1, the 3DCNN consists of two 3D convolutional layers, two batch normalization layers, two max pooling layers, one fully connected layer, and one ReLU activation layer. A DNN consists of one fully connected layer, one Dropout layer, one ReLU activation layer, and two batch normalization layers.

Images