Alzheimer disease classification method based on multi-modal image feature fusion

Abstract

The invention discloses an Alzheimer's disease classification method based on multi-modal image feature fusion. The Alzheimer's disease classification method is used for auxiliary diagnosis of Alzheimer's disease. The method comprises the following steps: firstly, preprocessing brain sMRI and PET image data, and extracting sMRI grey matter tissues; secondly, sending the sMRI grey matter image and the PET image into two sub-residual networks in the multi-modal network for feature extraction and feature fusion, and acting on the fused features by adopting spatial pyramid pooling; and finally, the two sub-networks respectively learn respective characteristics of the multi-modal image through a non-shared parameter training mode, and then a prediction result is output through full connection and a softmax layer. The characteristics of a deep learning method and brain images are combined, more remarkable diagnosis performance is achieved through effective fusion of multi-mode characteristics, the diagnosis performance is further improved compared with the diagnosis performance of two single modes, and the method has a certain application prospect in the aspect of auxiliary diagnosis of the Alzheimer's disease.

Description

Technical Field

[0001] The present invention relates to an Alzheimer's disease classification method based on multimodal image feature fusion, which realizes feature extraction of sMRI gray matter and PET multimodal images, and effectively fuses multimodal features to further improve the auxiliary diagnosis performance of Alzheimer's disease, and relates to the fields of medical imaging and deep learning. Background Art

[0002] Alzheimer's disease (AD) is a degenerative disease of the central nervous system, mainly characterized by memory, language, cognition and even emotional disorders, which can affect the patient's daily life and is the most common type of dementia. With the trend of global aging, the number of patients with Alzheimer's disease is increasing year by year. Early detection and accurate diagnosis of AD are crucial to delay the progression of the disease and improve the quality of life of patients. At present, the clinical diagnosis of AD requires a comprehensive evaluation based on cognitive behavioral assessment and imaging and other auxiliary examinations. Imaging examination is a relatively convenient and reliable diagnostic aid, which plays an important role in the detection and diagnosis of Alzheimer's disease. The results of imaging examination can assist doctors to improve the accuracy of Alzheimer's disease diagnosis to a certain extent. With the development of imaging technology, a variety of neuroimaging technologies have emerged, including structural imaging sMRI and functional imaging PET. As a non-invasive, radiation-free structural imaging, sMRI is currently the most widely used imaging tool in clinical practice. Doctors can observe some obvious structural changes in the brain through sMRI to evaluate AD. In terms of functional imaging, 18F-FDG PET is a molecular examination method that observes glucose metabolism by injecting radioactive elements into the body and analyzes the brain's glucose metabolism function to evaluate AD. The above two imaging technologies provide more basis for the screening and early diagnosis of AD. To this end, using current deep learning technology, how to effectively use these two imaging features to improve AD diagnostic performance has been a research hotspot in recent years. Summary of the invention

[0003] In order to solve the above problems, the present invention provides an Alzheimer's disease classification method based on multimodal image feature fusion, realizes AD auxiliary diagnosis based on multimodal sMRI gray matter and PET images, and improves the classification accuracy of AD. First, considering the characteristics of brain images, in order to reduce the loss of detail information in convolution feature extraction, low-level features and high-level features are integrated in the fully connected layer. Secondly, in order to effectively fuse the relevant features of two different modal images, we realize feature extraction and feature learning of multimodal images through a network training method based on non-shared parameters, and further use spatial pyramid pooling to reduce the dimensionality of multimodal image features. Finally, the final classification prediction is obtained through the fully connected layer and the softmax layer, which improves the reliability of classification and auxiliary diagnosis.

[0004] The ADNI dataset is a publicly available dataset used to study the diagnosis of Alzheimer's disease. We selected sMRI images and PET images of 461 subjects in the database as sub-datasets. In order to avoid data leakage, the training set, validation set and test set were divided according to the subject's ID. The subjects are divided into: cognitively normal (Normal Control, NC) and AD subjects. Our invention achieved good test results on the ADNI sub-dataset, and the rationality and effectiveness of the invention were verified through ablation experiments.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] 1. In step 1, a data set is constructed. The public data set used for AD auxiliary diagnosis is ADNI. The data subset of the present invention patent uses sMRI and PET images of 461 subjects in the ADNI database, including 246 NC subjects and 215 AD subjects. We divide the data set according to the number of subjects at a ratio of 70%, 15% and 15% by subject number. The first n-1 numbered subjects are used for network training, half of the data after n is used for verification, and the other half is used for testing. All subjects are not from the same person, which avoids inaccurate test results due to data leakage. The two modality images are preprocessed using the Clinica software platform. After deviation correction, template registration and pixel normalization, standard images are obtained. The resolution size of the two preprocessed images is 121×145×121.

[0007] 2. In step 2, the present invention proposes the implementation of feature extraction of two modal images. First, the two preprocessed images are respectively subjected to feature extraction through two 3D ResNet10 sub-networks, and then the channel attention mechanism is implemented on the feature maps obtained in the four convolutional layers. Finally, the multimodal feature maps in each convolutional layer are spliced ​​to obtain the fusion features of four groups of multimodal images. This method realizes effective feature extraction of two brain images, and multimodal images provide better feature specificity through complementary information characteristics.

[0008] 3. In step 3, the present invention proposes the dimension reduction of fusion features by spatial pyramid pooling. Spatial pyramid pooling is a method for increasing the receptive field, and also solves the problem of loss of some detail information caused by the pooling effect. After the spatial pyramid pooling module acts on the fusion features, a multi-scale feature map is obtained, and the feature map scales are 4×4×4, 2×2×2 and 1×1×1 respectively. Then the multi-scale feature map is flattened and spliced, and finally 1-dimensional convolution is used to reduce the dimensions of 4 groups of multimodal features respectively. The feature dimensions after dimension reduction are 128, 256, 512, and 1024 respectively. Then, the 4 groups of reduced-dimensional features are spliced ​​in the fully connected layer to obtain a 1-dimensional feature vector of 1960 dimensions. Since the spatial pyramid pooling strategy considers multi-scale information in the fusion feature representation stage, its pooling effect makes the sampling features have better robustness. These reduced-dimensional features contain multi-scale information of multimodal features, which is conducive to further improvement of network classification performance.

[0009] 4. In step 4, the present invention proposes a training method for non-shared parameters of multimodal sub-networks. The spliced ​​1960-dimensional feature vector is passed through a fully connected layer and softmax to output the final prediction result. The two sub-networks in the multimodal network learn their respective image features through non-shared parameters during training, and the network uses cross entropy loss for optimization to achieve classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram of the sMRI and PET data preprocessing process of the present invention.

[0011] Figure 2 It is a principle block diagram of the multimodal network structure of the present invention.

[0012] Figure 3 It is a schematic diagram of the pyramid space pooling principle of the present invention. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with the accompanying drawings:

[0014] Figure 1 The following is a schematic diagram of the sMRI and PET data preprocessing process, which includes the following steps:

[0015] The present invention uses the following method to perform sMRI and PET image data preprocessing, including anterior commissure AC correction, registration and normalization; using the neuroimaging data analysis software package SPM tool on the Clinica software platform to perform anterior commissure correction on the image, and performing affine registration in the MNI152 space to align the image with the standardized template to achieve registration of the sMRI and PET images; then using the CAT12 tool to extract gray matter tissue from the registered sMRI image, and processing of the two images is performed in the MNI space.

[0016] Figure 2 It is a block diagram of the multimodal network structure, which includes the following steps:

[0017] The preprocessed sMRI and PET image data are used as the input of the two sub-networks in the multimodal network for training and learning; the two modal features are fused after the four convolutional layers of their respective networks, and the four groups of multimodal features are acted on through the spatial pyramid pooling module, and finally the classification prediction results are output through the fully connected layer and softmax.

[0018] In the multimodal network structure, each ResNet10 sub-network structure includes four convolutional residual layers; the first convolutional layer contains 2 groups of 64-channel 3×3×3 convolutional kernels; the second convolutional layer contains 2 groups of 128-channel 3×3×3 convolutional kernels; the third convolutional layer contains 2 groups of 256-channel 3×3×3 convolutional kernels; the fourth convolutional layer contains 2 groups of 512-channel 3×3×3 convolutional kernels; after Block4 are the fully connected layer and the softmax layer.

[0019] Figure 3 It is the principle diagram of spatial pyramid pooling, which includes the following steps:

[0020] The principle of spatial pyramid pooling (SPP) is to first perform global average pooling on the multimodal image features, and obtain three scale feature maps after pooling, with scales of 4×4×4, 2×2×2, and 1×1×1, respectively. Then, the three feature maps are flattened and concatenated in sequence, and finally, the flattened and concatenated feature dimensions are reduced to the dimension of the output feature map in the corresponding convolutional layer through 1D convolution, which are 128, 256, 512, and 1024, respectively. The dimension calculation comes from the cascade features of the two sub-networks.

[0021] During the test, in order to evaluate the classification performance of the network for AD patients and cognitively normal people (NC), three evaluation indicators were introduced according to practical applications: accuracy, sensitivity, and specificity, as shown in the following formula:

[0022] Accuracy=(TP+TN) / (TP+TN+FP+FN) (1)

[0023] Sensitivity=TP / (TP+FN) (2)

[0024] Specificity=TN / (FP+TN) (3)

[0025] Among them, true positive (True Positive, TP) is the number of AD samples predicted as AD, false positive (False Positive, FP) is the number of NC samples predicted as AD, false negative (False Negative, FN) is the number of AD samples predicted as NC, and true negative (True Negative, TN) is the number of NC samples predicted as NC.

[0026] Accuracy refers to the percentage of correctly classified subjects in the total number of subjects in the test set, and is used to evaluate the overall discrimination ability of the model for image classification of AD and NC subjects; sensitivity is a measure of missed diagnosis; specificity is an important indicator for diagnosing diseases. Table 1 shows the comparison results between the network of the present invention and the 3D ResNet10 classification network based on sMRI and PET single modality and the multi-modal network Multi-modal ResNet10 (MResNet10) with shared parameter training method, which shows that the method of the present invention has obvious advantages in the diagnostic performance of Alzheimer's disease.

[0027] Table 1 Comparison of diagnostic performance of the method of the present invention with other network methods

[0028]

Claims

1. A classification method for Alzheimer's disease based on multimodal image feature fusion, characterized in that The following steps are involved: Step 1: Based on the public data set ADNI, sMRI and PET images from the same subject are selected to form the data set used in the present invention, a non-leakage partitioned data set is formulated, and the two images are preprocessed to obtain standard images; Step 2: Extract features from the two images. The two images preprocessed in step 1 are respectively subjected to the attention mechanism residual network for feature extraction. Then, the two image features obtained after the four convolutional layers in the residual network are concatenated to obtain four sets of multimodal fusion features. Step 3: Perform spatial pyramid pooling on the fused features, and then reduce the dimension through 1D convolution. The dimension is equal to the number of convolution kernels in the convolution layer. Then, the 4 sets of reduced-dimensional features are concatenated and fused as the input of the final fully connected layer. Step 4: The two sub-networks of the multimodal network are trained and learned by non-shared parameters, and the cross entropy loss is used to optimize the network to finally achieve classification prediction.

2. The Alzheimer's disease classification method based on multimodal image feature fusion according to claim 1 is characterized in that The preparation and preprocessing of the data set in step 1 are constructed as follows: based on the public data set ADNI, sMRI and PET images of the same subject are selected to form the data set of the present invention, and the training set, validation set and test set are divided in a non-leaking manner; the sMRI and PET images are preprocessed, including bias correction, template alignment and normalization, and gray matter tissue is extracted from the sMRI. The resolution of the preprocessed sMRI gray matter images and PET images is 121×145×121.

3. The Alzheimer's disease classification method based on multimodal image feature fusion according to claim 1 is characterized in that The extraction and fusion of multimodal features in step 2 are constructed as follows: the sMRI gray matter and PET images in the step 1 data set are input into the 3DResNet10 residual network to obtain multimodal image features and perform feature fusion; first, the two residual sub-networks extract features from the multimodal images respectively, and then perform channel attention on the feature maps obtained by the four convolutional layers of the residual network, and then the feature maps after the action are spliced ​​in the corresponding convolutional layers to obtain four groups of fused multimodal features.

4. The Alzheimer's disease classification method based on multimodal image feature fusion according to claim 1, characterized in that The dimension reduction of the multimodal fusion features in step three is constructed as follows: first, spatial pyramid pooling is used on the multimodal features obtained in step two to obtain multi-scale feature maps with scales of 4×4×4, 2×2×2, and 1×1×1, respectively. The multi-scale feature maps are then flattened and concatenated. Finally, one-dimensional convolution is used for dimensionality reduction. The feature dimensions of the four groups of multimodal features after dimensionality reduction are 128, 256, 512, and 1024, respectively. They are then fused in the fully connected layer to obtain a one-dimensional feature vector of 1920 dimensions.

5. The Alzheimer's disease classification method based on multimodal image feature fusion according to claim 1 is characterized in that The learning method of the non-shared parameters of the multimodal sub-network in step 4 is constructed as follows: the 1920-dimensional 1D vector obtained in step 3 is passed through the fully connected layer and the softmax layer to obtain the final prediction result. The multimodal network is optimized using cross-entropy loss. The two sub-networks can better learn the characteristics of each modality image through the non-shared parameter training method, further improving the performance of Alzheimer's disease auxiliary diagnosis based on multimodal image feature fusion.

Images