Alzheimer's disease diagnosis method based on multi-modal brain network feature fusion

Abstract

The invention provides an Alzheimer's disease diagnosis method based on multi-modal brain network feature fusion, which is based on a multi-modal brain image, a neural network and an attention mechanism, and has the main innovation points that: 1) effective feature combination of three types of multi-modal data of MRI, DTI and fMRI can reflect all feature information of a patient sample; and 2) aiming at different image data, a targeted feature extractor is used, the three-dimensional convolution is more suitable for feature learning of a three-dimensional MRI whole brain image, the Mama model is more suitable for learning of fMRI time sequence features, and the graph convolution network is more suitable for learning of DTI fiber number structure brain network features. And 3) the innovative mixed attention mechanism is fused with three types of multi-modal features, position coding and a Transform structure are used at the same time, the mixing effect and the classification accuracy of the multi-modal features are improved, and compared with a convolution and full connection layer structure, richer information can be learned.

Description

Technical Field

[0001] The present invention relates to the field of bioinformatics, focusing on neuroimaging and computational neuroscience. It involves the use of artificial intelligence algorithms to analyze brain structure and functional connection patterns. This field combines the knowledge of computer science, neuroscience and clinical medicine, and specifically relates to an Alzheimer's disease diagnosis method based on the fusion of multimodal brain network features. Background Art

[0002] Alzheimer's disease (AD) is the most common form of dementia. It progresses slowly and worsens over time, seriously affecting the quality of life. The causes of AD are extremely complex. Most patients have already progressed to the late stage of the disease when they are discovered, missing the golden opportunity for treatment. Therefore, early diagnosis and rapid detection of the patient's condition are critical to dealing with AD [1].

[0003] In recent years, magnetic resonance imaging technology has become the main means of AD diagnosis research, such as structural MRI, functional MRI (fMRI), diffusion tensor imaging (DTI), etc., and is widely used in clinical detection, diagnosis and treatment of various brain diseases. At the same time, with the strong rise of deep learning technology, AD neuroimaging diagnosis models based on deep learning have excellent performance [2]. Researchers use deep learning models to fully learn the rich features and hidden representations of brain magnetic resonance imaging data. Even with fewer samples, they can still effectively identify and classify Alzheimer's patients, significantly improving the efficiency and reliability of early diagnosis of AD.

[0004] In previous AD deep learning diagnosis research, most researchers focused on single-modality methods. Structural MRI can provide structural information of the brain. Islam, J. et al. [3] extracted slices from 3D MRI according to three axes (axial plane, coronal plane, and sagittal plane) to increase the number of training samples, and then input them into 2D CNN. The model diagnostic accuracy was 93% on the OASIS dataset. Fan et al. [4] divided the entire brain image into multiple local areas, extracted some 3D patches from each area, clustered and grouped the patches, and then constructed a DenseNet network to learn patch features for classification, retaining the three-dimensional spatial information of the image. The accuracy of AD vs. HC classification was 89.5%, and the accuracy of MCI vs. HC classification was 73.8%. With the significant improvement of GPU computing resources, the method of using whole-brain data for classification is becoming more and more common. S. Korolev et al. [5] directly used residuals and ordinary 3D convolutional neural networks to diagnose 3D MRI images, with an accuracy of 80% in the AD vs. HC binary classification. Although these studies have achieved good results in diagnosing AD, structural MRI has limited features and its expression is restricted to the information of a single data source. It cannot fully capture the rich connotation of the object, and it is difficult to achieve an accurate diagnostic rate when the sample size is insufficient.

[0005] In contrast, the introduction of multimodal features can integrate information from different data sources, resulting in richer and more meaningful feature representations, reducing information uncertainty and improving model performance [6]. Juan Son et al. [7] fused the gray matter tissue regions of MRI and FDG-PET images and used 3D CNN to diagnose AD, with an AD vs. HC classification accuracy of 94.11% and a MCI vs. HC classification accuracy of 88.48%. However, their method requires the fusion of the two modal data in advance, and the model can only process images of a single dimension.

[0006] In addition, there are currently two Chinese patents related to Alzheimer's disease diagnosis based on multimodal neuroimaging: "Pre-diagnosis method of Alzheimer's disease based on 3D convolutional neural network" applied by Sichuan University (application number: CN202011551593.2). And "Comprehensive diagnosis system of Alzheimer's disease based on multimodal brain imaging" applied by Hangzhou Ranman Intelligent Technology Co., Ltd., Teigong (Hangzhou) Technology Industry Co., Ltd. and Hangzhou Hairui Boyan Technology Co., Ltd. (application number: CN202011329714.9).

[0007] Patent 1: "Pre-diagnosis method for Alzheimer's disease based on 3D convolutional neural network" (application number: CN202011551593.2). Dataset: ADNI database, including MRI and DTI images. Network architecture: 3D CNN, including 3D convolution layer, batch normalization, linear rectifier activation function (ReLU), 3D pooling layer. Method structure: The input layer receives MRI and DTI images. Multiple 3D convolution layers and pooling layers are stacked in sequence. The batch normalization layer is used to accelerate the training process. The output layer produces classification results. Method process: S1. Select the data set. S2. Data preprocessing. S3. Apply 3D CNN for feature extraction. S4. Build the network architecture. S5. Implementation and performance evaluation.

[0008] Patent 2: "Comprehensive Diagnostic System for Alzheimer's Disease Based on Multimodal Brain Imaging" (Application No.: CN202011329714.9), a deep learning system based on multimodal neuroimaging and a neural network based on domain adversarial. The multimodal neuroimaging system is used for cognitive state classification. The domain adversarial neural network fuses MRI and PET data. Method structure: MRI and PET data are input into different network branches respectively. The domain adversarial neural network is responsible for fusing the information of the two modalities. The output layer produces the final diagnosis result. Method process: Data acquisition - obtain MRI and PET images. Data preprocessing: standardize and normalize the images. Feature extraction - apply deep learning models to extract features. Fusion and classification: fuse features and perform classification through domain adversarial networks.

[0009] Disadvantages:

[0010] Other imaging modalities such as fMRI are underutilized, as fMRI provides information not only on brain structure but also on brain function.[8] PET scans are expensive and require radioactive tracers.[9]

[0011] For different types of modal data, each modality usually has its own unique characteristics, and using a single type of feature extractor may not be able to effectively capture these features.

[0012] In neuroimaging analysis, brain networks have important research value, as they reflect the connection relationship between different brain regions or neurons in the brain. Neurodegenerative diseases, especially AD, can seriously damage the brain and lead to cognitive impairment. DTI brain connection networks can be used to further enhance multimodal representation

[10]

[11] .

[0013] Multimodal feature extraction and fusion only rely on common convolutional networks, and the information that convolutional and fully connected layer structures can learn is limited.

[0014] References:

[0015] [1]“2019Alzheimer’s disease facts and figures,”Alzheimer’s&Dementia,vol.15,no.3,pp.321–387,Mar.2019,doi:10.1016 / j.jalz.2019.01.010.

[0016] [2]M.A.Ebrahimighahnavieh,S.Luo,and R.Chiong,“Deep learning to detectAlzheimer’s disease from neuroimaging:A systematic literature review,”Computer Methods and Programs in Biomedicine,vol.187,p.105242,Apr.2020,doi:10.1016 / j.cmpb.2019.105242.

[0017] [3]J.Islam and Y.Zhang,“Brain MRI analysis for Alzheimer’s diseasediagnosis using an ensemble system of deep convolutional neural networks,”Brain Inf.,vol.5,no.2,p.2,Dec.2018,doi:10.1186 / s40708-018-0080-3.

[0018] [4]F.Li and M.Liu,“Alzheimer’s disease diagnosis based on multiplecluster dense convolutional networks,”Computerized Medical Imaging andGraphics,vol.70,pp.101–110,Dec.2018,doi:10.1016 / j.compmedimag.2018.09.009.

[0019] [5]S.Korolev,A.Safiullin,M.Belyaev,and Y.Dodonova,“Residual and plainconvolutional neural networks for 3D brain MRI classification,”2017IEEE 14thInternational Symposium on Biomedical Imaging(ISBI 2017),pp.835–838,Apr.2017,doi:10.1109 / ISBI.2017.7950647.

[0020] [6]F.Behrad and M.Saniee Abadeh,“An overview of deep learning methodsfor multimodal medical data mining,”Expert Systems with Applications,vol.200,p.117006,Aug.2022,doi:10.1016 / j.eswa.2022.117006.

[0021] [7]J.Song,J.Zheng,P.Li,X.Lu,G.Zhu,and P.Shen,“An Effective MultimodalImage Fusion Method Using MRI and PET for Alzheimer’s Disease Diagnosis,”Front.Digit.Health,vol.3,p.637386,Feb.2021,doi:10.3389 / fdgth.2021.637386.

[0022] [8]Warren,Samuel L,and Ahmed A Moustafa.“Functional magneticresonance imaging,deep learning,and Alzheimer's disease:A systematic review.”Journal of neuroimaging:official journal of the American Society ofNeuroimaging vol.33,1(2023):5-18.doi:10.1111 / jon.13063.

[0023] [9]Silverman,Daniel H S.“Brain 18F-FDG PET in the diagnosis ofneurodegenerative dementias:comparison with perfusion SPECT and with clinicalevaluations lacking nuclear imaging.”Journal of nuclear medicine:officialpublication,Society of Nuclear Medicine vol.45,4(2004):594-607.

[0024]

[10] F.Palesi,G.Castellazzi,L.Casiraghi,E.Sinforiani,P.Vitali,C.A.M.Gandini Wheeler-Kingshott,Exploring Patterns of Alteration inAlzheimer’s Disease Brain Networks:A Combined Structural and FunctionalConnectomics Analysis,Front.Neurosci.10(2016).https: / / doi.org / 10.3389 / fnins.2016.00380.

[0025]

[11] H.Cai, Z.Zhou, D.Yang, G.Wu, J.Chen, Discovering Brain NetworkDysfunction in Alzheimer's Disease Using Brain Hypergraph Neural Network, in: H.Greenspan, A.Madabhushi, P.Mousavi, S.Salcudean, J.Duncan, T.Syeda-Mahmood, R.Taylor (Eds.), Medical Image Computing and Computer Assisted Intervention–MICCAI 2023, Springer Nature Switzerland, Cham, 2023: pp.230–240. https: / / doi.org / 10.1007 / 978-3-031-43904-9_23. Summary of the invention

[0026] The purpose of the technical solution of the present invention is to propose a pre-diagnosis method for Alzheimer's disease based on multimodal brain imaging, neural networks and attention mechanisms, to solve the problems of underutilization of other imaging methods, the inability to effectively capture the corresponding unique characteristics of different types of modal data using a single type of feature extractor, neurodegenerative diseases seriously damaging the brain and causing cognitive impairment affecting the research value of brain networks, and the limited information that can be learned by convolutional and fully connected layer structures.

[0027] In order to solve the above technical problems, the technical solution of the present invention provides an Alzheimer's disease diagnosis method based on multimodal brain network feature fusion, comprising the following steps:

[0028] Using the ADNI database, select MRI image data, DTI image data, and fMRI image data;

[0029] The MRI image data were linearly aligned, format converted, and z-score voxel normalized, intensity outliers were trimmed, and background removed. The DTI image data were preprocessed and the fiber quantity structure brain network was extracted. The fMRI image data were functionally realigned and dewarped, slice time corrected, outliers were identified and removed, and MNI space normalized. The time series of each region of interest were extracted to obtain the preprocessed MRI image data, fiber quantity structure brain network, and time series of each region of interest.

[0030] Symptom binary classification experiments were performed on the preprocessed MRI image data, fiber quantity structure brain network and time series of each region of interest to obtain AD vs. HC image datasets with Alzheimer's patients and healthy people, and MCI vs. HC image datasets with mild cognitive impairment and healthy people. According to the Alzheimer's patients, mild cognitive impairment and healthy people, AD vs. HC image annotation datasets and MCI vs. HC image annotation datasets were obtained.

[0031] The AD vs. HC attempted image dataset, the MCI vs. HC attempted image dataset, the AD vs. HC attempted image annotation dataset, and the MCI vs. HC attempted image annotation dataset are divided into a training set and a validation set according to a preset ratio;

[0032] The pre-processed MRI image data, fiber quantity structure brain network and time series data of each region of interest in the training set are simultaneously input into a neural network and attention mechanism model with MRI data feature extractors, fMRI data feature extractors and DTI data feature extractors for feature training;

[0033] The MRI data feature extractor is configured with a first preset number of fully connected layer nodes, and the three-dimensional DenseNet169 model is modified according to the preprocessed MRI image data to obtain MRI data features corresponding to Alzheimer's patients, mild cognitive impairment and healthy people. After the DTI data feature extractor is configured with a second preset number of fully connected layer nodes, feature extraction is performed on the fiber quantity structure brain network to obtain DTI data features corresponding to Alzheimer's patients, mild cognitive impairment and healthy people. After the fMRI data feature extractor is configured with a third preset number of fully connected layer nodes, feature extraction is performed on the time series of each region of interest to obtain fMRI data features corresponding to Alzheimer's patients, mild cognitive impairment and healthy people; the values ​​of the first preset number of fully connected layer nodes, the second preset number of fully connected layer nodes and the third preset number of fully connected layer nodes are the same;

[0034] The MRI data features, DTI data features and fMRI data features are dimensionally transformed, and the query Q, key K and value V are initialized by linear transformation according to the product of the numerically converted data and the weight data. The attention of the query Q, key K and value V is calculated according to the key K dimension, and the output of the attention is weighted and summed to obtain the MRI fusion feature data corresponding to Alzheimer's patients, mild cognitive impairment and healthy people, the DTI fusion feature data corresponding to Alzheimer's patients, mild cognitive impairment and healthy people, and the fMRI fusion feature data corresponding to Alzheimer's patients, mild cognitive impairment and healthy people, and splice them to obtain Multimodal feature vector, generate the row position and column position of the multimodal feature vector through sine and cosine functions, use Transformer encoder to use the row position and column position as the position encoding matrix through addition operation to fuse the feature vector with the multimodal feature vector, perform the first convolution, ReLU activation and pooling on the output after the feature vector fusion to obtain the first feature, add a self-attention module in the middle to obtain the second feature, and perform the second convolution, ReLU activation and pooling to obtain the third feature, and obtain the Alzheimer's disease diagnosis result and the trained Alzheimer's disease diagnosis model through the fully connected layer and softmax function;

[0035] The pre-processed MRI image data, fiber quantity structure brain network and time series data of each region of interest in the validation set are simultaneously input into the trained Alzheimer's disease diagnosis model for verification and debugging to obtain the final Alzheimer's disease diagnosis model, and Alzheimer's disease patients are diagnosed according to the final Alzheimer's disease diagnosis model.

[0036] Preferably, the Alzheimer's disease diagnosis result includes a probability numerical feature vector of an Alzheimer's disease patient, a probability numerical feature vector of a mild cognitive impairment patient, a probability numerical feature vector of a healthy person, and a diagnosis result with the highest probability.

[0037] Preferably, according to the preprocessed MRI image data, a Squeeze-and-Excitation attention mechanism is followed after 4 DenseBlocks to modify the three-dimensional DenseNet169 model.

[0038] Preferably, the DTI data feature extractor structure is a graph convolutional network layer, ReLU, a graph convolutional network layer, a pooling layer, and a fully connected layer.

[0039] Preferably, the fMRI data feature extractor comprises adding a fully connected layer after the Mamba model.

[0040] The technical solution of the present invention proposes a method for diagnosing Alzheimer's disease based on multimodal brain network feature fusion, using the public database ADNI (Alzheimer's Disease Neuroimaging Initiative, Alzheimer's disease neuroimaging plan), including structural magnetic resonance imaging (MRI), functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI). Network structure: Corresponding feature extraction network modules are constructed for the three types of modalities. For MRI, the Squeeze-and-Excitation attention mechanism and the DenseNet network are used to construct the feature extraction module; for fMRI, the Mamba model is combined to learn the time series data features of each region of interest (ROI); for DTI, the graph convolutional network (GCN) is used to learn the fiber quantity structure connection network construction graph features. For the three types of features learned by the feature extractor module, the cross attention mechanism is used for fusion to obtain a multimodal representation vector, and the feature position information is added using position encoding. The diagnostic network module structure includes Transformer encoder, one-dimensional convolution layer, linear rectifier activation function (ReLU), pooling layer, self-attention mechanism layer, and fully connected layer to produce classification results. Method process: S1. Select the dataset. S2. Data preprocessing. S3. Construct the classification diagnosis network. S4. Implementation and performance evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the overall framework corresponding to an Alzheimer's disease diagnosis method based on multimodal brain network feature fusion is provided for an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the network structure of an MRI feature extraction module corresponding to an Alzheimer's disease diagnosis method based on multimodal brain network feature fusion is provided for an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the network structure of an fMRI feature extraction module corresponding to an Alzheimer's disease diagnosis method based on multimodal brain network feature fusion is provided for an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a DTI feature extraction module network structure corresponding to an Alzheimer's disease diagnosis method based on multimodal brain network feature fusion is provided for an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a classification network structure corresponding to an Alzheimer's disease diagnosis method based on multimodal brain network feature fusion is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0047] like Figure 1 As shown, the embodiment of the present invention provides an Alzheimer's disease diagnosis method based on multimodal brain network feature fusion, and the specific steps are as follows:

[0048] S1: Select dataset: Use the ADNI database and select MRI, DTI, and fMRI image data.

[0049] S2: Data preprocessing: perform corresponding preprocessing on the selected MRI, DTI, and fMRI image data;

[0050] S2.1: MRI preprocessing: The FMRIB software library (FSL) was used for linear registration, format conversion, z-score voxel normalization, trimming of intensity outliers, and background removal.

[0051] fMRI preprocessing: The CONN toolbox (Functional connectivity toolbox, CONN) was used. The data preprocessing process used in the experiment included: functional realignment and dewarping, slice time correction, outlier identification and removal, and MNI spatial normalization (Montreal Neurological Institute). The python nilearn library was used to extract the time series of each region of interest from the preprocessed fMRI data.

[0052] DTI preprocessing: Use the pipeline toolbox PANDA to preprocess and extract the fiber quantity structure brain network.

[0053] S2.2: Initialize the training set and test set for the Alzheimer's disease diagnosis task:

[0054] Two binary classification experiments were conducted: AD (Alzheimer's disease patients) vs. HC (healthy people) and MCI (mild cognitive impairment) vs. HC.

[0055] The dataset for the AD vs. HC task includes AD and HC subject image data. Each subject data contains pre-processed MRI, fMRI, and DTI modality data. The image data is labeled to obtain labeled image data. AD patient data is labeled with 1 (indicating positive), and HC data is labeled with 0 (indicating negative).

[0056] The dataset of the MCI vs. HC task includes the image data of MCI and HC subjects. Similarly, each subject’s data contains pre-processed MRI, fMRI, and DTI modality data, and the image data is labeled to obtain labeled image data. The MCI patient data is labeled with 1 (indicating positive), and the HC data is labeled with 0 (indicating negative).

[0057] The labeled image data is divided into a training set and a validation set according to a preset ratio. The training set is used to train the diagnostic model, and the test set is used to validate the diagnostic model to obtain the diagnostic model.

[0058] S3: Deep learning method based on neural network and attention mechanism. The specific steps are as follows:

[0059] S3.1: Model training (training using training set data):

[0060] Data input and model learning:

[0061] In the diagnostic experiment, the deep learning neural network model learned the numerical differences in image data between Alzheimer's patients and healthy people on the training set. During this process, the various parts of the model continuously adjusted parameters during the back propagation process so that the predicted annotation labels can be consistent with the input data annotations.

[0062] The innovation of this part is that MRI, fMRI, and DTI data are sent to the network synchronously. Each subject and each type of modality corresponds to its own feature extractor. The data of all modalities are trained synchronously in the same model, and the parameters are updated together during backpropagation. Finally, multimodal prediction is an integrated process.

[0063] MRI feature extraction: 3D MRI images can be viewed as a three-dimensional matrix, in which each voxel (voxel, i.e., volume element) represents the signal intensity of a point in the image. The three-dimensional convolution layer is the core component of the 3D CNN. Unlike two-dimensional convolution, the three-dimensional convolution kernel slides in three dimensions (width, height, and depth), so that the spatial relationship between voxels can be captured. Each convolution operation generates a new feature map, which together constitute an abstract representation of the input Alzheimer's patient MRI data. Modify the three-dimensional DenseNet169 model and follow the Squeeze-and-Excitation attention mechanism after 4 DenseBlocks. The number of nodes in the fully connected layer in the last part of the network is fixed to 1280, and the structure is as follows: Figure 2 The function of the fully connected layer is to further integrate the numerical features learned by the previous network layer to form a fixed-length feature vector. This feature vector condenses the key information of the input Alzheimer's MRI data. The feature representation of fMRI and DTI is similar. In this part, we represent the features learned from MRI as X MRI :

[0064] X MRI =SEDenseNet(MRI)(1)

[0065] fMRI feature extraction: Input the preprocessed fMRI region of interest time series data, add a fully connected layer (FC) after the Mamba model, and the number of nodes is fixed to 1280, such as Figure 3 Similarly, the features learned from fMRI are represented as X FMRI :

[0066] X fMRI =MambaFC(fMRI)(2)

[0067] DTI feature extraction: Input the DTI fiber number structure brain network obtained by preprocessing. The extractor structure is graph convolutional network layer, ReLU, graph convolutional network layer, pooling layer, and fully connected layer. The number of nodes in the fully connected layer is fixed to 1280, such as Figure 3 As shown. Figure 4 Similarly, the features learned from DTI are represented as X DTI :

[0068] X DTI =GCNFC(DTI)(3)

[0069] S3.2: This part innovatively uses the cross-attention mechanism to perform multimodal fusion of the X obtained in step S3.1 MRI , XfMRI , X DTI :

[0070] First, we calculate the cross attention. At the beginning, we first MRI , X fMRI , X DTI Perform dimension conversion. Since the number of nodes in the fully connected layer is set to 1280 in step S3.1, the dimensions of the three feature data are the same. For the convenience of calculation, each data is converted to 10x128. After conversion, it is still used as X. MRI , X fMRI , X DTI To express.

[0071] Then use X MRI , X fMRI , X DTI To initialize three sets of queries Q(Q MRI , Q DTI , Q fMRI ), key K(K MRI , K DTI , K fMRI ) and value V(V MRI 、V DTI 、V fMRI ) data, specifically using X MRI , X fMRI , X DTI This is achieved by multiplying the weight data using linear transformation, see formula (4)(5)(6):

[0072]

[0073] Among them is the trainable weight data, the size is 128x128, these weights are randomly generated by the computer at the beginning, and updated during the back propagation process of neural network training; query Q(Q MRI , Q DTI , Q fMRI ), key K(K MRI , K DTI , K fMRI ) and value V(V MRI 、V DTI 、V fMRI ) Data is what you get.

[0074] Then for each pair of data, the attention is calculated, taking formula (7) as an example, where d k =128, representing the key K dimension:

[0075]

[0076] in, Calculate Q first MRI and K DTI The dot product of k After scaling, the attention weight is obtained through the softmax function and then multiplied by V DTI Weighted sum.

[0077] Similarly, according to the logic of formula (7), calculate other attention (Q MRI ,K fMRI ,V fMRI ), Attention(Q fMRI ,K MRI ,V MRI ), Attention(Q fMRI ,K DTI ,V DTI ), Attention(Q DTI ,K MRI ,V MRI ), Attention(Q DTI ,K fMRI ,V fMRI ), and perform weighted summation on these attention outputs to obtain the fusion feature data F MRI , F fMRI , F DTI , as shown in formulas (8)(9)(10).

[0078] F MRI =α1·Attention(Q MRI ,K DTI ,V DTI )+α2·Attention(Q MRI ,K fMRI ,V fMRI )(8)

[0079] F fMRI =β1·Attention(Q fMRI ,K MRI ,V MRI )+β2·Attention(Q fMRI ,K DTI ,V DTI )(9)

[0080] F DTI =γ1·Attention(Q DTI ,K MRI ,V MRI )+γ2·Attention(Q DTI ,K fMRI ,V fMRI)(10)

[0081] Among them, α1, α2, β1, β2, γ1, γ2 are defined as trainable parameters, which are randomly initialized by the computer and updated during the back propagation process of neural network training. MRI , F fMRI , F DTI , and then splice to obtain a multimodal feature vector X that integrates the information between MRI, fMRI, and DTI. multi (size is 30x128), as shown in formula (11).

[0082] X multi =concat(F MRI ,F fMRI ,F DTI )(11)

[0083] Next, for the multimodal feature data X multi Perform position encoding and add the position information of each value in the feature. Generate through sine and cosine functions, for X multi The position p in X multi The position of the middle row, ranging from 1 to 30) and dimension i (representing X multi The position of the column in the range from 1 to 64), the encoding definition is shown in formula (12) (13), d = 128. The position encoding matrix PE is added to X multi The feature vectors are fused, as shown in formula (14).

[0084]

[0085] S3.3: Diagnostic network classification structure: Transformer encoder is introduced, followed by one-dimensional convolution layer, linear rectifier activation function (ReLU), pooling layer, self-attention mechanism layer, one-dimensional convolution layer, linear rectifier activation function (ReLU), pooling layer, and fully connected layer. The overall structure is as follows: Figure 5 shown.

[0086] After the Transformer encoder, the network structure includes two convolutions, ReLU, and pooling, with a self-attention module added in between. The process can be expressed by formulas (15), (16), and (17), where X M express The output obtained after the Transformer encoder, z1 is the feature obtained after the first convolution, ReLU activation and pooling, z2 is the output after the self-attention module, and z3 is the representation obtained after the final convolution, ReLU, and pooling.

[0087] z1=MaxPool(ReLU(Conv1(X M )))(15)

[0088] z2=SAttention(z1)(16)

[0089] z3=MaxPool(ReLU(Conv2(z2)))(17)

[0090] Finally, the output result of the model is obtained through the fully connected layer and the softmax function, see formula (18), and finally used.

[0091] Classification Results=softmax(FC(z3))(18)

[0092] The classification results obtained by the softmax function are the probability of each Alzheimer's disease subject being labeled in each category as predicted by the model. The label with the highest probability is the diagnosis result. The subject label predicted by the model is compared with its true label and evaluated using the machine learning confusion matrix.

[0093] S4: Model implementation and evaluation:

[0094] The model is evaluated by its prediction performance in the test set. In machine learning, four results can be obtained based on whether the sample prediction label matches the actual label: true positive (TP; false positive (FP), false negative (FN), and true negative (TN). Accuracy [Formula (19)] and AUC (Area Under the Curve) are calculated to evaluate model performance. AUC usually refers to the area under the ROC curve (Receiver Operating Characteristic Curve). The ROC curve is drawn by changing the classification threshold, with the horizontal axis being the false positive rate (False Positive Rate, FPR) [Formula (20)] and the vertical axis being the true positive rate (True Positive Rate, TPR) [Formula (21)].

[0095]

[0096] Implementation: The GPU used was 1 NVIDIA GeForce RTX 3090, and the deep learning framework used was Pytorch v2.1.2. Image preprocessing tools included CONN toolbox, v.22.a and PANDA pipeline toolbox v1.3.1. In the training and testing of the model, the ADNII Alzheimer's disease subjects were arranged in ascending order according to the number, and a 4-fold cross-validation method was used. During the training process, it was ensured that the subjects in the training set and the test set would not overlap, and the subjects in each of the four training sets did not overlap. The total number of training rounds was set to 100, the batch size was 2, the initial learning rate was set to 0.0001, and the learning rate decayed by 3% every 10 epochs. The loss function used CrossEntropyLoss, the optimizer was Adam, and the dropout was set to 0.1.

[0097] In the AD vs. HC and MCI vs. HC binary classification experiments, the public dataset ADNI was used to evaluate the performance of the proposed model. The experimental results show that the average accuracy of AD vs. HC is 95.19%, which is 1%-3% higher than the advanced model; the average accuracy of MCI vs. HC is 92.71%, which is 1%-5% higher than the advanced model. The proposed model can accurately diagnose early AD and promote early intervention treatment. At the same time, some studies published in recent years that use multimodal brain imaging to diagnose AD are summarized, including the models used, datasets, modality types, sample sizes, and classification accuracy of patients and healthy people. The comparison results are shown in Table 1. In these different research methods, there are differences in the number of subjects selected, the type of modality, the model structure, and the training method. By comparison, it can be seen that in AD vs. HC, the accuracy of the proposed method is 95.19%, which is slightly lower than the 96.75% of Q. Zhu et al.

[18] and the 95.27% of HAN et al. [7], but it still achieves a good diagnostic effect. In the diagnosis of MCI vs. HC, even though the number of samples was less than that of Zhang L et al.

[32] , our accuracy still achieved the best experimental results. This indirectly proves that the model integrating convolution and attention mechanisms can fully learn and mine multimodal features, improving the early diagnosis effect of Alzheimer's patients.

[0098]

[0099] Table 1 Comparison with other multimodal brain imaging studies for AD diagnosis

[0100] The brain imaging data used in the above embodiment is the public data set ADNI, including 56 AD, 48 HC, and 48 MCI. The imaging data of each subject consists of T1MRI, DTI and fMRI. The GPU used is 1 NVIDIAGeForce RTX 3090, and the deep learning framework used is Pytorch v2.1.2. Image preprocessing tools include CONNtoolbox, v.22.a and PANDA pipeline toolbox v1.3.1. In the training and testing of the model, the subjects are arranged in ascending order according to the subject number, and the 4-fold cross-validation method is used. During the training process, it is ensured that the subjects of the training set and the test set do not overlap, and the subjects of each test set in the 4 trainings do not overlap. The total number of training is set to 100 rounds, the batch size is 2, the initial learning rate is set to 0.0001, and the learning rate decays by 3% every 10 epochs. The loss function uses CrossEntropyLoss, the optimizer is Adam, and the dropout is set to 0.1. The prediction results of the classification model are summarized using the confusion matrix. The accuracy and AUC are calculated on the test set to evaluate the model performance.

[0101] The embodiment of the present invention provides an Alzheimer's disease diagnosis method based on multimodal brain network feature fusion, based on multimodal brain imaging, neural networks and attention mechanisms. The main innovations are: 1) The effective feature combination of three types of multimodal data, MRI, DTI and fMRI, can reflect all the feature information of the patient sample. 2) For different imaging data, targeted feature extractors are used. Three-dimensional convolution is more suitable for feature learning of three-dimensional MRI whole brain images, Mamba model is more suitable for learning fMRI time series features, and graph convolution network is more suitable for learning DTI fiber quantity structure brain network features. 3) The innovative hybrid attention mechanism fuses three types of multimodal features, and uses position encoding and Transformer structure at the same time to improve the mixing effect and classification accuracy of multimodal features. Compared with convolution and fully connected layer structures, richer information can be learned.

Claims

1. A method for diagnosing Alzheimer's disease based on multimodal brain network feature fusion, characterized in that: The following steps are involved: Using the ADNI database, select MRI image data, DTI image data, and fMRI image data; The MRI image data were linearly aligned, format converted, and z-score voxel normalized, intensity outliers were trimmed, and background removed. The DTI image data were preprocessed and the fiber quantity structure brain network was extracted. The fMRI image data were functionally realigned and dewarped, slice time corrected, outliers were identified and removed, and MNI space normalized. The time series of each region of interest were extracted to obtain the preprocessed MRI image data, fiber quantity structure brain network, and time series of each region of interest. Symptom binary classification experiments were performed on the preprocessed MRI image data, fiber quantity structure brain network and time series of each region of interest to obtain AD vs. HC datasets with Alzheimer's patients and healthy people, and MCI vs. HC datasets with mild cognitive impairment and healthy people. The datasets were labeled according to Alzheimer's patients, mild cognitive impairment and healthy people to obtain AD vs. HC labeled datasets and MCI vs. HC labeled datasets. Divide the AD vs. HC attempted image dataset, the MCI vs. HC attempted image dataset, the AD vs. HC attempted image annotation dataset, and the MCI vs. HC attempted image annotation dataset into a training set and a validation set according to a preset ratio; The pre-processed MRI image data, fiber quantity structure brain network and time series data of each region of interest in the training set are simultaneously input into a neural network and attention mechanism model with MRI data feature extractors, fMRI data feature extractors and DTI data feature extractors for feature training; The MRI data feature extractor is configured with a first preset number of fully connected layer nodes, and the three-dimensional DenseNet169 model is modified according to the preprocessed MRI image data to obtain MRI data features corresponding to Alzheimer's patients, mild cognitive impairment, and healthy people. After the DTI data feature extractor is configured with a second preset number of fully connected layer nodes, feature extraction is performed on the fiber quantity structure brain network to obtain DTI data features corresponding to Alzheimer's patients, mild cognitive impairment, and healthy people. After the fMRI data feature extractor is configured with a third preset number of fully connected layer nodes, feature extraction is performed on the time series of each region of interest to obtain fMRI data features corresponding to Alzheimer's patients, mild cognitive impairment, and healthy people. The values ​​of the first preset number of fully connected layer nodes, the second preset number of fully connected layer nodes, and the third preset number of fully connected layer nodes are the same; The MRI data features, DTI data features and fMRI data features are dimensionally transformed, and the query Q, key K and value V are initialized by linear transformation according to the product of the numerically converted data and the weight data. The attention of the query Q, key K and value V is calculated according to the key K dimension, and the output of the attention is weighted and summed to obtain the MRI fusion feature data corresponding to Alzheimer's patients, mild cognitive impairment and healthy people, the DTI fusion feature data corresponding to Alzheimer's patients, mild cognitive impairment and healthy people, and the fMRI fusion feature data corresponding to Alzheimer's patients, mild cognitive impairment and healthy people, and splice them to obtain Multimodal feature vector, generate the row position and column position of the multimodal feature vector through sine and cosine functions, use Transformer encoder to use the row position and column position as the position encoding matrix through addition operation to fuse the feature vector with the multimodal feature vector, perform the first convolution, ReLU activation and pooling on the output after the feature vector fusion to obtain the first feature, add a self-attention module in the middle to obtain the second feature, and perform the second convolution, ReLU activation and pooling to obtain the third feature, and obtain the Alzheimer's disease diagnosis result and the trained Alzheimer's disease diagnosis model through the fully connected layer and softmax function; The pre-processed MRI image data, fiber quantity structure brain network and time series data of each region of interest in the validation set are simultaneously input into the trained Alzheimer's disease diagnosis model for verification and debugging to obtain the final Alzheimer's disease diagnosis model, and Alzheimer's disease patients are diagnosed according to the final Alzheimer's disease diagnosis model.

2. The method for diagnosing Alzheimer's disease based on multimodal brain network feature fusion as claimed in claim 1, characterized in that: The Alzheimer's disease diagnosis result includes a probability numerical feature vector of an Alzheimer's disease patient, a probability numerical feature vector of a mild cognitive impairment, a probability numerical feature vector of a healthy person, and a diagnosis result with the highest probability.

3. The method for diagnosing Alzheimer's disease based on multimodal brain network feature fusion as claimed in claim 1, characterized in that: According to the preprocessed MRI image data, a Squeeze-and-Excitation attention mechanism is followed after 4 DenseBlocks to modify the three-dimensional DenseNet169 model.

4. The method for diagnosing Alzheimer's disease based on multimodal brain network feature fusion according to claim 1, characterized in that: The DTI data feature extractor structure is a graph convolutional network layer, ReLU, a graph convolutional network layer, a pooling layer, and a fully connected layer.

5. The method for diagnosing Alzheimer's disease based on multimodal brain network feature fusion as claimed in claim 1, characterized in that: The fMRI data feature extractor consists of adding fully connected layers after the Mamba model.

Images