A Multimodal MRI Image Classification and Recognition Method

By optimizing model parameters through low-rank tensor multimodal fusion and gradient dynamic adjustment algorithms, the problems of distribution and convergence differences in sMRI and fMRI data fusion were solved, improving the accuracy and performance of multimodal MRI image classification and recognition.

CN118823440BActive Publication Date: 2026-11-10NANJING TECH UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410804059.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-20
Publication Date
2026-11-10
Estimated Expiration
2044-06-20

Smart Images

  • Figure CN118823440B_ABST
    Figure CN118823440B_ABST
Patent Text Reader

Abstract

This invention provides a multimodal MRI image classification and recognition method. First, a brain multimodal MRI dataset with real labels is acquired as training samples. The acquired sMRI and fMRI data are preprocessed. A multimodal MRI data classification and recognition model is constructed. The preprocessed data is input into the constructed multimodal MRI data classification and recognition model for training. Multimodal MRI data to be identified is acquired and preprocessed. The preprocessed sMRI and fMRI data to be identified are input into the trained multimodal MRI data classification and recognition model to obtain the multimodal MRI data classification and recognition results. This invention's multimodal MRI image classification and recognition method reduces the distribution differences between sMRI and fMRI modalities, while also reducing the differences in convergence speed between different modalities. It effectively alleviates the heterogeneity between different modalities, fully utilizes the complementarity of different modalities, obtains rich feature information, and improves the performance of the multimodal MRI image classification and recognition model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of image classification technology, specifically relating to a multimodal MRI image classification and recognition method. Background Technology

[0002] Magnetic Resonance Imaging (MRI) technology generates images of the human body's internal structures by capturing the electron spin resonance phenomenon within its tissues. MRI technology is widely used in medical diagnostics and research due to its non-invasiveness, lack of radiation, and high resolution. Imaging modalities based on MRI include structural magnetic resonance imaging (sMRI), functional magnetic resonance imaging (fMRI), diffusion tensor imaging (DTI), and magnetic resonance angiography (MRA), each providing different modalities of imagery. Different modalities of brain MRI images provide complementary information about the brain; sMRI reflects the brain's anatomical structure, while fMRI reflects changes in brain function. Combining sMRI and fMRI allows for a more comprehensive understanding of the brain and improves the accuracy of classification and identification. To effectively utilize the complementary information from sMRI and fMRI, researchers typically employ a fusion strategy to combine sMRI and fMRI features, and then use a joint training algorithm with a unified learning objective to train feature extraction networks from different modalities. However, such multimodal fusion methods ignore the distributional differences between sMRI and fMRI data, as well as the differences in convergence speed between different modal feature extraction networks, leading to reduced performance of the multimodal fusion model. Therefore, designing a multimodal MRI classification and recognition method that can effectively fuse information from different modalities has become an urgent technical problem to be solved. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal MRI image classification and recognition method.

[0004] The present invention proposes a multimodal MRI image classification and recognition method, which includes the following specific steps:

[0005] Step 1: Obtain a multimodal MRI dataset of the brain with real labels as training samples, including sMRI data and fMRI data;

[0006] Step 2: Preprocess the acquired training samples;

[0007] Step 3: Construct a multimodal MRI data classification and recognition model;

[0008] Step 4: Input the preprocessed training samples into the constructed multimodal MRI data classification and recognition model for training;

[0009] Step 5: Acquire the multimodal MRI data to be identified and preprocess the multimodal data;

[0010] Step 6: Input the preprocessed sMRI and fMRI data to be identified into the trained multimodal MRI classification and recognition model to obtain the multimodal MRI data classification and recognition results.

[0011] Preferably, the preprocessing of the acquired training samples in step 2 includes:

[0012] The sMRI data were subjected to anterior-posterior commissure origin correction, neck removal, skull removal, image registration, and spatial downsampling to obtain a brain image with uniform resolution in a standardized space as the result of sMRI preprocessing.

[0013] The fMRI data were subjected to slice time-layer correction, head motion correction, spatial standardization, spatial smoothing, elimination of linear trends, time bandpass filtering, covariate regression, functional brain region segmentation, extraction of average time series of all voxels in each brain region, construction of whole brain functional connectivity matrix, extraction and expansion of upper triangular functional connectivity matrix, and the one-dimensional vector representing the functional connectivity between brain regions was obtained as the result of fMRI preprocessing.

[0014] Preferably, the multimodal MRI data classification and recognition model constructed in step 3 includes a feature extraction module, a feature fusion module, and a classification module;

[0015] The feature extraction module uses a 3DCNN as the feature extraction network for sMRI data. Using DNN as the feature extraction network for fMRI data

[0016] The feature fusion module integrates the sMRI feature extraction network. Extracted feature vector Z s and fMRI feature extraction network Extracted feature vector Z f Low-rank tensor multimodal fusion is performed to obtain the fused feature F;

[0017]

[0018] Among them, Z s Z f ∈R d , Let i and n be the i-th low-rank weighting factors of sMRI, respectively. Let be the i-th low-rank weighting factor of fMRI, b be the bias vector, · be the tensor dot product, and ⊙ be the element-wise product.

[0019] The classification module uses a support vector machine as a classifier to classify the fused features F and generate prediction results.

[0020] Furthermore, the training process of the multimodal MRI data classification and recognition model in step 4 specifically includes the following steps:

[0021] Step 4.1, Initialize the sMRI feature extraction network The parameter θ s and fMRI feature extraction network The parameter θ f The number of iterations t = 0;

[0022] Step 4.2, calculate the overall loss L of the multimodal MRI data recognition model;

[0023] Step 4.3, Calculate the sMRI feature extraction network Parameter θ s gradient and fMRI feature extraction network The parameter θ f gradient

[0024]

[0025]

[0026] Step 4.4: Calculate the model parameters θ corresponding to each mode using the ratio of the L2 norm of the model parameter gradients. s θ f gradient difference ratio

[0027]

[0028]

[0029] In the formula, ||·||2 is the L2 norm;

[0030] Step 4.5, calculate the sMRI feature extraction network parameters θ s Regulatory factors fMRI feature extraction network parameters θ f Regulatory factors

[0031]

[0032]

[0033] In the formula, δ is the adjustment coefficient;

[0034] Step 4.6: Update the sMRI feature extraction network using the gradient dynamic adjustment algorithm. The parameter θ s and fMRI feature extraction network The parameter θ f ;

[0035]

[0036]

[0037] In the formula, For the sMRI feature extraction network at the t-th iteration Model parameters, For the fMRI feature extraction network at iteration t... The model parameters are ξ, which is the learning rate, and β1 and β2, which are the first and second exponential decay rates, respectively. t β1 t-j β1 is the power of t and tj, β2 is the power of the first exponential decay rate β1. t β2 t-j The second exponential decay rate β2 is raised to the power of t and tj. Let θ be the parameter for the j-th iteration. s θ f Regulatory factors;

[0038] Step 4.7: If the overall loss L is less than the set threshold or the number of iterations t reaches the set maximum value, then end the training and output the parameter θ. s and θ f Otherwise, t = t + 1, and proceed to step 4.2 of the loop.

[0039] The formula for calculating the total loss L in step 4.2 is as follows:

[0040] L = LOSS class +γLOSS dd

[0041] Among them, LOSS class For classification loss, LOSS dd The loss represents the difference in distribution between modes, and γ is the balance coefficient.

[0042] Furthermore, the classification loss LOSS class for:

[0043]

[0044] Where, k i Let i be the true label of the i-th sample. To predict the i-th sample as the k-th sample i The output obtained from the sample class, The output is the result of predicting the i-th sample as the j-th class, where N is the number of samples and Δ is a hyperparameter.

[0045] Furthermore, the intermodal distribution difference loss LOSS dd for:

[0046]

[0047] in, Let Z be the feature vector extracted from the sMRI modality of the i-th sample. s The value at the j-th position, Let Z be the feature vector extracted from the fMRI modality of the i-th sample. f The value at the j-th position.

[0048] Beneficial effects

[0049] The multimodal MRI image classification and recognition method proposed in this invention can reduce the distribution differences between sMRI and fMRI modalities, while also reducing the differences in convergence speed between different modalities. This effectively alleviates the heterogeneity between different modalities, fully utilizes the complementarity of different modalities, obtains rich feature information, and improves the performance of the multimodal MRI image classification and recognition model. Attached Figure Description

[0050] Figure 1 This is a flowchart of a multimodal MRI image classification and recognition method according to the present invention;

[0051] Figure 2 This is a graph showing the change in training loss of a multimodal MRI image classification and recognition method according to the present invention.

[0052] Figure 3 This is a trend chart showing the test accuracy of the multimodal MRI image classification and recognition method of the present invention.

[0053] Figure 4 This is a distribution map of different modal features in a multimodal MRI image classification and recognition method according to the present invention;

[0054] Figure 5 This is a comparison chart of the classification results of the multimodal MRI image classification and recognition method of the present invention with other methods. Detailed Implementation

[0055] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0056] Participants were categorized into two groups based on a specific theme (e.g., adults and children, high intelligence and low intelligence): Group A and Group B. Assuming there are 312 participants in Group A and 334 in Group B, sMRI and fMRI images were collected from each participant. 80% were selected as the training sample, and 20% as the test sample. All operations were performed under a Windows operating system. The experimental hardware was as follows: Intel Xeon(R) Silver4210 CPU @ 2.20GHz + 2.19GHz (2 processors), and an NVIDIA GeForce RTX 3090 graphics card.

[0057] like Figure 1 As shown, a multimodal MRI image classification and recognition method includes the following steps:

[0058] Step 1: Obtain multimodal MRI images of the brains of 80% of the subjects as training samples, including sMRI data and fMRI data;

[0059] Step 2: Preprocess the acquired training samples;

[0060] Step 3: Construct a classification and recognition model for the multimodal MRI data of the subjects;

[0061] Step 4: Input the preprocessed subject training samples into the constructed multimodal MRI data classification and recognition model for training;

[0062] Step 5: Obtain multimodal MRI images of the remaining 20% ​​of the subjects to be identified, and preprocess the multimodal data;

[0063] Step 6: Input the preprocessed sMRI and fMRI data to be identified into the trained multimodal MRI classification and recognition model to obtain the multimodal MRI data classification and recognition results.

[0064] Preferably, the acquired training samples are preprocessed, including the following specific steps:

[0065] The sMRI data were preprocessed, and anterior commissure-posterior commissure origin correction was performed to adjust for inconsistencies in brain image angles caused by subject head movements during sMRI data acquisition. A nonparametric gray-level non-uniformity normalization algorithm was used to correct gray-level non-uniformity in the sMRI images. Image registration was used to map individual sMRI brain images to the MNI305 standard space. Spatial downsampling was used to normalize all extracted brain images to a 64×64×64 resolution sample space to eliminate differences between sMRI brain images obtained from different imaging devices.

[0066] The fMRI data was preprocessed, including slice time-layer correction, head motion correction, spatial standardization, spatial smoothing, linear trend elimination, temporal bandpass filtering, and covariate regression to obtain brain gray matter images. The brain was divided into 116 functional regions using an automatically labeled physiological anatomy (AAL) template, and the average time series of all voxels in each region was extracted. The Pearson correlation coefficient between any two brain region time series was calculated, resulting in a 116×116 whole-brain functional connectivity matrix. The upper triangular portion of the functional connectivity matrix was then extracted and expanded into a 6670×1-dimensional vector.

[0067] Preferably, the multimodal MRI data classification and recognition model constructed in step 3 includes a feature extraction module, a feature fusion module, and a classification module;

[0068] The feature extraction module uses a 3DCNN as the feature extraction network for sMRI data. Using DNN as the feature extraction network for fMRI data 3DCNN consists of two 3D convolutional layers, two batch normalization layers, two max pooling layers, one fully connected layer, and one ReLU activation layer. The kernel size of the 3D convolutional layers is 1×2×2, and the number of nodes in the fully connected layer is 1000. DNN consists of one fully connected layer, one Dropout layer, one ReLU activation layer, and two batch normalization layers. The number of nodes in the fully connected layer is 1000.

[0069] The feature fusion module integrates the sMRI feature extraction network. Extracted feature vector Z s and fMRI feature extraction network Extracted feature vector Z f Low-rank tensor multimodal fusion is performed to obtain the fused feature F:

[0070]

[0071] Among them, Z s Z f ∈R 1001 , Let i and n be the i-th low-rank weighting factors of sMRI, respectively. Let m = 4 be the i-th low-rank weighting factor of fMRI, and b be the bias vector, where b ∈ R. 128 ,· represents tensor dot product, and ⊙ represents element-wise product;

[0072] The classification module uses a support vector machine as a classifier to classify the fused features F and generate prediction results.

[0073] Furthermore, the training process of the multimodal MRI data classification and recognition model in step 4 specifically includes the following steps:

[0074] Step 4.1, Initialize the sMRI feature extraction network The parameter θ s and fMRI feature extraction network The parameter θ f The number of iterations t = 0;

[0075] Step 4.2, calculate the overall loss L of the multimodal MRI data recognition model;

[0076] Step 4.3, Calculate the sMRI feature extraction network Parameter θ s gradient and fMRI feature extraction network The parameter θ f gradient

[0077]

[0078]

[0079] Step 4.4: Calculate the model parameters θ corresponding to each mode using the ratio of the L2 norm of the model parameter gradients. s θ f gradient difference ratio

[0080]

[0081]

[0082] In the formula, ||·||2 is the L2 norm;

[0083] Step 4.5, calculate the sMRI feature extraction network parameters θ s Regulatory factors fMRI feature extraction network parameters θ f Regulatory factors

[0084]

[0085]

[0086] In the formula, δ is the adjustment coefficient, which is taken as δ = 0.6;

[0087] Step 4.6: Update the sMRI feature extraction network using the gradient dynamic adjustment algorithm. The parameter θ s and fMRI feature extraction network The parameter θ f ;

[0088]

[0089]

[0090] In the formula, For the sMRI feature extraction network at the t-th iteration Model parameters, For the fMRI feature extraction network at iteration t... The model parameters are: ξ is the learning rate, which is set to 0.01; the first exponential decay rate β1 and the second exponential decay rate β2 are set to 0.9 and 0.999 respectively; β1 t β1 t-j β1 is the power of t and tj, β2 is the power of the first exponential decay rate β1. t β2 t-j The second exponential decay rate β2 is raised to the power of t and tj. Let θ be the parameter for the j-th iteration. s θ f Regulatory factors;

[0091] Step 4.7: If the overall loss L is less than the set threshold or the number of iterations t reaches the set maximum value, then end the training and output the parameter θ. s and θ f Otherwise, t = t + 1, and proceed to step 4.2 of the loop.

[0092] The formula for calculating the total loss L in step 4.2 is as follows:

[0093] L = LOSS class +γLOSS dd

[0094] Among them, LOSS class For classification loss, LOSS dd The loss represents the difference in distribution between modes, and γ is the balance coefficient.

[0095] Furthermore, the classification loss LOSSclass for:

[0096]

[0097] Where, k i Let i be the true label of the i-th sample. To predict the i-th sample as the k-th sample i The output obtained from the sample class, The output is the result of predicting the i-th sample as the j-th class, where N is the number of samples and Δ is a hyperparameter.

[0098] Furthermore, the intermodal distribution difference loss LOSS dd for:

[0099]

[0100] in, Let Z be the feature vector extracted from the sMRI modality of the i-th sample. s The value at the j-th position, Let Z be the feature vector extracted from the fMRI modality of the i-th sample. f The value at the j-th position.

[0101] The loss change trend during training in this embodiment of the invention is as follows: Figure 2 The gray curve represents the training loss curve when using the method employed in this example, i.e., sMRI and fMRI modalities as inputs. The red curve represents the training loss obtained by feeding the output of the sMRI feature extraction network in the multimodal model into the classifier, and the blue curve represents the training loss obtained by feeding the output of the fMRI feature extraction network in the multimodal model into the classifier. The red and blue curves are used to evaluate the convergence speed of the single-modal feature extraction network in the multimodal model. Figure 2 As can be seen, the training curve of the method used in this example eventually converges, and the difference in loss between different modalities shows a decreasing trend. The trend of accuracy change on the test set during training is shown below. Figure 3 As shown, the gray curve represents the test accuracy curve of the method used in this example, i.e., using sMRI and fMRI modalities as inputs; the red curve represents the test accuracy obtained by feeding the output of the sMRI feature extraction network in the multimodal model into the classifier on the test dataset; and the blue curve represents the training loss obtained by feeding the output of the fMRI feature extraction network in the multimodal model into the classifier on the test dataset. Figure 3 As can be seen, with the increase of training iterations, the test accuracy curve of the method used in this example shows an upward trend, and the differences in classification accuracy among different modalities on the test set all decrease. The method used in this example effectively alleviates the optimization imbalance among different modalities.

[0102] During training, feature vectors of the fMRI and sMRI modalities are extracted for the current training round. Then, the fMRI and sMRI modal feature vectors are reduced to 2 dimensions using t-distributed random neighborhood embedding. The changes in the feature distribution of the two modalities during training are then visually represented graphically. The training feature distribution diagram of this embodiment is shown below. Figure 4 As shown in the figure, blue represents the fMRI modality, and red represents the sMRI modality. (From...) Figure 4 As can be seen, the red and blue feature distribution points are close, which indicates that the method used in this example can effectively bring the sMRI and fMRI modal feature distributions closer together and improve the correlation between different multimodal features.

[0103] The experimental results used in the embodiments of the present invention are as follows: Figure 5 As shown, the results demonstrate that, compared to other methods, the proposed method exhibits higher test accuracy, as well as higher sensitivity and specificity values. This indicates that the method of this invention can effectively fuse complementary information between different modalities, thereby improving the overall performance of the multimodal MRI classification and recognition model.

[0104] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the claims and their equivalents.

[0105] References

[0106] [1] Sun Y, Mai S, Hu H.Learning to balance the learning rates between various modalities via adaptive tracking factor[J]. IEEE Signal Processing Letters, 2021, 28: 1650-1654.

[0107] [2]Witten D M,Tibshirani R J.Extensions of sparse canonicalcorrelation analysis with applications to genomic data[J].Statisticalapplications in genetics and molecular biology,2009,8(1).

[0108] [3]Hu W,Cai B,Zhang A,et al.Deep collaborative learning withapplication to the study of multimodal brain development[J].IEEE Transactionson Biomedical Engineering,2019,66(12):3346-3359.

[0109] [4]Andrew G,Arora R,Bilmes J,et al.Deep canonical correlationanalysis[C] / / International conference on machine learning.PMLR,2013:1247-1255.

[0110] [5]Liu Z,Shen Y,Lakshminarasimhan V B,et al. Efficient Low-rankMultimodal Fusion With Modality-Specific Factors[C] / / Proceedings of the 56thAnnual Meeting of the Association for Computational Linguistics(Volume 1:LongPapers).2018:2247-2256.

Claims

1. A multimodal MRI image classification and recognition method, characterized in that, Includes the following steps: Step 1: Obtain a multimodal MRI dataset of the brain with real labels as training samples, including sMRI data and fMRI data; Step 2: Preprocess the acquired training samples; Step 3: Construct a multimodal MRI data classification and recognition model; Step 4: Input the preprocessed training samples into the constructed multimodal MRI data classification and recognition model for training, wherein the training process includes: Step 4.1, Initialize the sMRI feature extraction network parameters and fMRI feature extraction network parameters The number of iterations t=0; Step 4.2, calculate the overall loss L of the multimodal MRI data recognition model; Step 4.3, Calculate the sMRI feature extraction network parameter gradient and fMRI feature extraction network parameters gradient : Step 4.4: Calculate the model parameters corresponding to each mode using the ratio of the L2 norm of the model parameter gradient. gradient difference ratio : In the formula, It is an L2 norm; Step 4.5: Calculate the parameters of the sMRI feature extraction network. Regulatory factors fMRI feature extraction network parameters Regulatory factors : In the formula, This is the adjustment coefficient; Step 4.6: Update the sMRI feature extraction network using the gradient dynamic adjustment algorithm. parameters and fMRI feature extraction network parameters : In the formula, For the first sMRI feature extraction network at the next iteration Model parameters, For the first fMRI feature extraction network at the next iteration Model parameters, For learning rate, and These are the first exponential decay rate and the second exponential decay rate, respectively. The first exponential decay rate of Power of 1 The second exponential decay rate of Power of 1 , For the first Next iteration parameters Regulatory factors; Step 4.7: If the overall loss L is less than the set threshold or the number of iterations t reaches the set maximum value, then end the training and output the parameters. and Otherwise, t = t + 1, and proceed to step 4.2 in the loop; Step 5: Acquire the multimodal MRI data to be identified and preprocess the multimodal MRI data; Step 6: Input the preprocessed sMRI and fMRI data to be identified into the trained multimodal MRI classification and recognition model to obtain the multimodal MRI data classification and recognition results.

2. The multimodal MRI image classification and recognition method according to claim 1, characterized in that, Step 2, the preprocessing of the acquired training samples includes: The sMRI data were subjected to anterior-posterior commissure origin correction, neck removal, skull removal, image registration, and spatial downsampling to obtain a brain image with uniform resolution in a standardized space as the result of sMRI preprocessing. The fMRI data underwent slice time-layer correction, head motion correction, spatial standardization, spatial smoothing, elimination of linear trends, temporal bandpass filtering, covariate regression, functional brain region segmentation, extraction of the average time series of all voxels in each brain region, construction of the whole brain functional connectivity matrix, extraction and expansion of the upper triangular part of the functional connectivity matrix, and the resulting one-dimensional vector representing the functional connectivity between brain regions was used as the result of fMRI preprocessing.

3. The multimodal MRI image classification and recognition method according to claim 1, characterized in that... Step 3 involves constructing a multimodal MRI data classification and recognition model, which includes a feature extraction module, a feature fusion module, and a classification module. The feature extraction module uses a 3DCNN as the feature extraction network for sMRI data. DNN was used as the feature extraction network for fMRI data. ; The feature fusion module integrates the sMRI feature extraction network. Extracted feature vectors and fMRI feature extraction network Extracted feature vectors Low-rank tensor multimodal fusion is performed to obtain the fused feature F: in, , Let i and n be the i-th low-rank weighting factors of sMRI, respectively. Let be the i-th low-rank weighting factor of fMRI, b be the bias vector, ⦁ be the tensor dot product, and ⊙ be the element-wise product; The classification module uses a support vector machine as a classifier to classify the fused features F and generate prediction results.

4. The multimodal MRI image classification and recognition method according to claim 1, characterized in that, The formula for calculating the total loss L in step 4.2 is as follows: in, For classifying losses, For the loss due to differences in distribution between modes, This is the balance coefficient; Furthermore, the classification loss for: in, For the first The true label of each sample To make the first The predicted value for the nth sample is... The output obtained from the sample class, To make the first The predicted value for the nth sample is... The output obtained by the class For the number of samples, For hyperparameters; Furthermore, the intermodal distribution difference loss for: in, It is represented as the feature vector extracted from the sMRI modality of the i-th sample. The value at the j-th position, It is represented as the feature vector extracted from the fMRI modality of the i-th sample. The value at the j-th position.

Citation Information

Patent Citations

  • MRI (Magnetic Resonance Imaging) data classification method based on low-rank multi-modal fusion network

    CN116958683A