Alzheimer's disease intelligent classification method based on multi-modal feature fusion

Abstract

The invention discloses an Alzheimer's disease classification method based on multi-modal data to solve the problems that in an Alzheimer's disease auxiliary classification technology, single-modal data is adopted, the clinical AD diagnosis requirement is difficult to meet, and extracted feature information is too single. Comprising the following steps: firstly, performing feature extraction on MRI image data through an improved ResNet, and introducing a coordinate attention mechanism to enable a network to pay attention to more spatial position information in consideration that the network should pay attention to a lesion position of an MRI image; and secondly, according to non-image data characteristics such as scales, biomarkers, genes and the like, a multi-layer perceptron is adopted to extract features of the non-image data, and finally, the image data and the non-image data are subjected to feature fusion to realize Alzheimer's disease classification. According to the method provided by the invention, the Alzheimer's disease can be diagnosed by analyzing from the perspective of various modal data, the diagnosis time of a doctor is shortened, the shortage of medical resources is relieved, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to an intelligent classification method for Alzheimer's disease based on multimodal feature fusion, and belongs to the field of computer vision and intelligent information technology. Background Art

[0002] Alzheimer's disease (AD), commonly known as senile dementia, is one of the most difficult geriatric diseases to cure. As the disease worsens, AD patients will experience a series of brain function damages such as gradual memory loss, mobility difficulties, decreased language expression ability, and cognitive difficulties. Clinically, the development of AD is usually divided into three stages: normal subjects (Cognitively Normal, CN), patients with mild cognitive impairment (MCI), and patients with Alzheimer's disease. Among them, MCI is an intermediate state between CN and AD. At this stage, the patient's cognitive function will decline slightly, but the daily living ability is not significantly disturbed. According to studies, the annual conversion rate of MCI to AD is as high as 10%-15%. Therefore, being able to accurately distinguish the different stages of patients, especially screening out patients in the MCI stage, plays a vital role in doctors delaying the progression of AD patients through drug treatment.

[0003] At present, the auxiliary diagnosis technology of Alzheimer's disease mainly uses 3D convolutional neural network (CNN) to study the whole brain three-dimensional MRI images. Compared with two-dimensional slices, the whole brain three-dimensional images can enrich the pixel information and better extract the spatial position features. Since the information of single-modality image data is too single, multimodal data can combine various modal data to achieve information complementarity, which is closer to the clinical diagnosis method of Alzheimer's disease by doctors. It can be seen from the existing research methods of auxiliary diagnosis of Alzheimer's disease that compared with single-modality diagnosis, multimodal diagnosis has obvious advantages, effectively improves the accuracy of classification, and is also the research trend of computer-assisted diagnosis of Alzheimer's disease. Summary of the invention

[0004] The present invention proposes an intelligent classification method for Alzheimer's disease based on multimodal feature fusion, which uses MRI images, scales, and genetic biomarkers to classify Alzheimer's disease. The feature information of the image data MRI is extracted by an improved ResNet, and the scale, gene, and biomarker non-image data are extracted using a multi-layer perceptron. Finally, the image data features and non-image data features are fused to achieve classification.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] (1) The improved ResNet is used to extract features from MRI images, and the feature information of MRI is obtained after the coordinate attention module (CA) and the four-layer residual module.

[0007] (2) Use multi-layer perceptron to extract features from non-imaging data such as scales, biomarkers, and genes.

[0008] (3) The MRI image features and non-image features are fused to obtain multimodal fusion feature information to ultimately achieve the classification of Alzheimer's disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a network structure diagram of the Alzheimer's disease intelligent classification method based on multimodal feature fusion of the present invention.

[0010] Figure 2 Improved coordinate attention module structure diagram for the present invention. DETAILED DESCRIPTION

[0011] The present invention will be further described below in conjunction with the accompanying drawings:

[0012] The method of constructing a multimodal network is as follows:

[0013] The multimodal classification diagram designed by the present invention is as follows Figure 1 As shown in the figure, the network mainly consists of two parts: feature extraction and fusion classification module. According to the characteristics of multimodal data, the feature extraction module can be divided into two feature extraction networks: image data and non-image data to realize feature extraction. The image data is MRI images, and the non-image data includes clinical scales, cerebrospinal fluid and genetic data.

[0014] In view of the brain MRI characteristics of AD, the feature extraction network of image data is based on ResNet, combined with the attention mechanism module. The network consists of an input module, an attention module, and a residual module. The improved image feature extraction network based on ResNet can not only combine the low-dimensional features and high-dimensional features in MRI images, but also pay attention to the spatial position relationship in the image through the attention mechanism module, so that the network pays attention to the lesion location of AD, thereby extracting better feature information.

[0015] Since non-imaging data are all numerical data and have a relatively obvious mapping relationship with Alzheimer's disease, it is not possible to use a more complex network structure or a model with a deeper number of network layers for feature extraction, which will cause overfitting problems and waste computing resources. The network structure of the multilayer perceptron is simple, it can fit any nonlinear mapping, and has good generalization performance. Therefore, this paper uses a multilayer perceptron to extract features from non-imaging data. The feature extraction network for non-imaging data consists of two fully connected layers, and the neurons increase from 64 dimensions to 1024 dimensions in sequence, so that the network can fully learn the characteristics of each modality and improve the classification accuracy. Dropout random inactivation neurons are used in the network to prevent network overfitting, and the activation function uses Leaky ReLu. Leaky ReLu can solve the "death problem" of neurons. Compared with the ReLU activation function, some negative axis values ​​are retained, which alleviates the problem that the activation value is too small and the neuron parameters cannot be updated.

[0016] The method of constructing the CA attention module is as follows:

[0017] The structure of the CA attention module is shown in the figure Figure 2 As shown in the figure: In the present invention, the three-dimensional MRI image is used, so this paper improves the CA attention module into a three-dimensional structure. First, the feature map X with a size of C×D×H×W is c Use pooling kernels (D, 1, 1), (1, H, 1) and (1, 1, W) to encode features in the horizontal, vertical and front-back directions, and generate feature maps Z in three directions with sizes of C×D×1×1, C×1×H×1 and C×1×1×W respectively. d , Z h , Z w Compared with the compression method of global pooling, this can capture long-distance relationships in a single direction while retaining spatial information in other directions, helping the network to more accurately locate the lesions of Alzheimer's disease. The specific formula is as follows:

[0018]

[0019]

[0020]

[0021] in It represents the feature map of the c-th channel from the D direction, H direction, and W direction respectively. After the three feature maps obtained above are transformed to the same dimension, they are concatenated, and then a 1×1×1 convolution kernel and activation function are introduced to reduce the dimension to generate the feature map f. The formula is as follows.

[0022] f=δ(F1([Z d , Z h, Z w ])) (4)

[0023] in, The intermediate features contain spatial information in depth, height, and width. r is the reduction factor, F1 represents the 1×1×1 convolution operation and BN normalization operation, δ represents the ReLU activation function, and then the feature map f is split along the spatial dimension into independent features in three directions. Then, the 1×1×1 convolution is used to increase the dimension, and the activation function is introduced to transform the features to obtain the final attention weight vector g. d , g h and g w The specific formula is as follows:

[0024] g d =σ(F d (f d )) (5)

[0025] g h =σ(F h (f h )) (6)

[0026] g w =σ(F w (f w )) (7)

[0027] where g d , g h , g w is the weight vector in the spatial direction, F d 、F h 、F w Indicates that 1×1×1 convolution operations are performed in three directions respectively, σ represents the sigmoid activation function, They represent the three independent spatial features of f decomposition. Finally, the attention weight vector g d , g h , g w After merging into the weight matrix, it is combined with the original feature map X c Calculate and get the output Y c , the formula is as follows.

[0028] Y c (d, h, w) = X c (d, h, w)×g d (d)×g h (h)×g w (w) (8)

[0029] The method for constructing an MRI image feature extraction network based on the improved ResNet is as follows:

[0030] The improved structure diagram based on ResNet is as follows Figure 1 As shown in the image data feature extraction module: In the present invention, the improved network based on ResNet is mainly composed of an input module, a CA attention module and a residual module. Compared with ResNet, the improved network replaces the 7×7×7 convolution in the input module of the original model with a 3×3×3 convolution, and introduces a coordinate attention mechanism after the input module, and uses the Mish activation function to replace the ReLu activation function in the residual module. In view of the need to pay more attention to detail features in the MRI feature extraction process, the input module uses 3×3×3 convolution to extract shallow features to avoid losing effective information. MRI images can observe the location of AD lesions, so the network model should focus on the spatial position feature information in the MRI images, use the CA attention module to obtain the relationship between the spatial positions, guide the network to pay attention to the lesion area, and reduce the interference of redundant information in the space on the network and suppress spatial features that have little effect on classification, thereby helping the network to extract more accurate feature areas. After the CA attention module, it passes through the four layers of BasicBlock (B1, B2, B3, B4) in ResNet for deeper feature extraction. The number of channels of the four-layer residual module is 64, 128, 256, and 512 respectively. The increasing number of channels can improve the feature extraction ability of the model. The residual module can fully combine the low-dimensional and high-dimensional features of MRI images through its short-link structure, effectively prevent gradient explosion, and improve the network's ability to fit high-dimensional functions. The residual module is mainly composed of convolutional layers and short connections. Its calculation formula is shown in the formula:

[0031]

[0032] Among them, x i and i represents the input and output of the i-th layer residual module, W i Represents the convolution operation parameters in the residual module. Indicates the convolution operation process. The Mish function is selected as the activation function in the residual module, and the curve is as follows Figure 2 As shown in the figure, the Mish function is smooth everywhere on the curve and can select representative feature information to improve the generalization ability of the model. Its expression is:

[0033] y=x·tan h(ln(1+e x )) (10)

[0034] The experimental data of the present invention is derived from the ADNI public database, and a multi-source multimodal data set is constructed, totaling 2128 data including 625 CN samples, 1178 MCI samples, and 325 AD samples, and research is carried out on two-classification AD / CN and three-classification AD / MCI / CN. First, on the multimodal data set constructed above, the results of the Alzheimer's disease classification method designed by the present invention based on multimodal data are compared with the relevant research results, wherein MRI represents MRI images, CS represents cognitive test scale (Cognitive Scale), CSF represents cerebrospinal fluid data, GD represents genetic data (Genetic Data), AGE represents age data, PET represents PET neuroimaging data, NP represents neuropsychological data, SNP represents single nucleotide polymorphism, which belongs to a kind of genetic data, D represents demographic information (Demographics), such as basic information such as age, gender, education level, marital status, etc., MRI+PET+CSF+GD+AGE represents network input as MRI, PET, CSF, GD and AGE, and the experimental results are shown in Table 3. According to the experimental results, the accuracy of the three-classification of the present invention reached 75.77%, and the accuracy of the two-classification was 92.89%. Due to the different types of modal data and data sets used in the relevant multimodal data research, only a rough comparison can be made. In addition, the improved ResNet in the present invention also has a good classification effect for the classification of Alzheimer's disease in MRI images in a single mode. The experimental results are shown in Table 2. The accuracy of the three-classification reached 62.26%, and the accuracy of the two-classification reached 84.49%. In summary, the experimental results of the present invention have a more obvious advantage over the experimental results of the above-mentioned method, indicating that the Alzheimer's disease classification method based on multimodal data proposed by the present invention has certain advancement.

[0035] Table 1 Comparison of the accuracy of cutting-edge methods for Alzheimer's disease classification based on multimodal data (%)

[0036]

[0037] Table 2 Comparison of accuracy, sensitivity and specificity of different network algorithms based on MRI images (%)

[0038]

Claims

1. An intelligent classification method for Alzheimer's disease based on multimodal feature fusion, characterized by The following steps are involved: (1) The improved ResNet network is used to extract features from the preprocessed MRI images. The coordinate attention mechanism (CA) is embedded into ResNet to make the network pay more attention to the characteristic information of the lesion area in the MRI image. (2) Use multi-layer perceptron to extract features from non-imaging data such as scales, biomarkers, and genes; (3) The MRI image features and non-image features are fused to obtain multimodal fusion feature information to ultimately achieve the classification of Alzheimer's disease.

2. The method according to claim 1, characterized in that In (1), a network based on the improved ResNet is constructed to extract features from MRI images. The construction method is as follows: The network is mainly composed of an input layer, a CA attention module, and a residual module; the input layer is a 3×3×3 convolutional layer, an activation function (ReLu), and a pooling function to extract shallow features to obtain X c After the input layer, the CA attention module is introduced to extract the spatial position relationship information in the MRI image so that the network can pay more attention to the lesion area. The CA attention module first needs to encode the three directions in space to obtain the feature map Z d , Z h , Z w , the formula is as follows: In the formula It represents the feature maps of the c-th channel from the D direction, H direction, and W direction respectively. After transforming the three feature maps obtained above to the same dimension, they are concat-concatenated, and then a 1×1×1 convolution kernel and activation function are introduced to reduce the dimension to generate the feature map f. The formula is as follows: f=δ(F1([Z d ,z h ,Z w ])) (4) In the formula, f contains the intermediate features of spatial information in depth, height, and width, r is the reduction factor, F1 represents the 1×1×1 convolution operation and BN normalization operation, δ represents the ReLU activation function, and then the feature map f is split along the spatial dimension into independent features in three directions, and then the 1×1×1 convolution is used for dimensionality increase, and the activation function is introduced for feature transformation to obtain the final attention weight vector g d , g h and g w , the formula is as follows: g d =σ(F d f d )) (5) g h =σ(F h (f h )) (6) g w =σ(F w (f w )) (7) Where g d , g h , g w is the weight vector in the spatial direction, F d 、F h 、F w represents a 1×1×1 convolution operation in three directions respectively, σ represents the sigmoid activation function, and the obtained attention weight vector g d , g h , g w After merging into the weight matrix, it is combined with the original feature map X c Calculate and get the output Y c , the formula is as follows: Y c (d,h,w)=X c (d,h,w)×g d (d)×g h (h)×g w (w) (8) Afterwards, the four-layer residual modules B1, B2, B3, and B4 in ResNet18 fully combine the high-dimensional features and low-dimensional features in the MRI image and replace the activation function in the residual module with the Mish function, finally obtaining the feature information F1 of the MRI image data.

3. The method according to claim 1, characterized in that In (2), the multi-layer perceptron is used to extract features from non-image data such as biomarkers and genes of the scale. The implementation method is as follows: The non-imaging data consists of scales, biomarkers, and genes. The scales include MMSE, FAQ, CDR, and CDR-SOB; the biomarkers include Tua protein, P-tua protein, and Aβ; the gene data is APOE; since non-imaging data are all numerical data and have a relatively obvious mapping relationship with Alzheimer's disease, a multi-layer perceptron is used to extract features from non-imaging data. The input non-imaging data passes through two fully connected layers, and the neurons are increased from 64 dimensions to 1024 dimensions. Then, dropout is used to randomly inactivate neurons to prevent network overfitting, and finally the Leaky ReLu activation function is used to obtain the feature information F2 of the non-imaging data.

4. The method according to claim 1, characterized in that (3) The MRI image features and non-image features are fused and classified to finally classify Alzheimer's disease. The fusion classification method is as follows: After feature extraction, the MRI image feature F1 and non-image feature F2 both have a dimension of 1024. Concat is used to concatenate and fuse the two features to 2048 dimensions, and then the 2048-dimensional feature vector is reduced to 1024 dimensions for feature reorganization to obtain the final multimodal feature vector. Finally, Softmax is used to classify the features after multimodal data fusion to achieve the classification diagnosis of Alzheimer's disease.

Images