Alzheimer disease classification method based on image and cognitive score fusion

Abstract

The invention discloses an Alzheimer's disease classification method based on image and cognitive score fusion. The Alzheimer's disease classification method is used for Alzheimer's disease auxiliary diagnosis. The method comprises the following steps: firstly, preprocessing sMRI or PET brain images; secondly, obtaining image features of the sMRI grey matter image or the PET image through a pre-training model; and finally, realizing classification prediction through a graph neural network constructed by fusing image features and cognitive scores. In combination with a deep learning method, classification of the Alzheimer's disease and a normal control group is realized in a mode of combining single-mode images with cognitive scores, and it is proved that the method can significantly improve the classification accuracy on two modes of images, and the method has a certain application prospect in the aspect of auxiliary diagnosis of the Alzheimer's disease.

Description

TECHNICAL FIELD

[0001] The application relates to an Alzheimer's disease classification method based on image and cognitive score fusion, realizes image and cognitive score fusion technology, and improves the auxiliary diagnosis performance of Alzheimer's disease, and relates to the fields of medical images and deep learning. BACKGROUND

[0002] Alzheimer's disease (AD) is a degenerative disease of the central nervous system, mainly characterized by memory, language, cognition and even emotional disorders, and can affect the daily life of patients. It is the most common type of dementia. With the global trend of aging, the number of Alzheimer's disease patients is increasing year by year. Realizing accurate diagnosis of AD is crucial to delay disease progression and improve the quality of life of patients. At present, the clinical diagnosis of AD needs to be comprehensively evaluated by cognitive assessment and imaging examination. Imaging examination is a relatively reliable diagnostic aid, such as sMRI and PET imaging technology, which plays an important role in the diagnosis and exclusion of Alzheimer's disease. In cognitive assessment, the clinical scale score can reflect the degree of cognitive impairment of patients. Therefore, combining cognitive score information can help to refine imaging classification and further improve the diagnostic accuracy of AD. Therefore, using current deep learning technology, effectively combining image features and cognitive scores to improve the diagnostic performance of AD is the focus of this field. SUMMARY

[0003] The application provides an Alzheimer's disease classification method based on image and cognitive score fusion, realizes AD auxiliary diagnosis by combining single-modality image and cognitive score, and significantly improves the AD classification accuracy. First, sMRI and PET images are used to obtain features. Second, the image features and cognitive scores are effectively fused. The advantages of graph neural networks in processing structured data and non-Euclidean space data are used to represent the subjects as nodes in the graph neural network and associate them with image features. The cognitive score information is fused by constructing an adjacency matrix to assign edge weights between subjects. Finally, a two-layer graph convolutional neural network with multi-head attention is used to realize classification output and improve the accuracy of disease classification.

[0004] The ADNI dataset is a publicly available dataset for studying the diagnosis of Alzheimer's disease. The application selects the sMRI images and PET images of the subjects in the database and the cognitive score information of the subjects as the dataset. The subjects are divided into: normal control group (NC) and AD subjects. The application achieves good test results on the two images of ADNI, and verifies the effectiveness of the application through the ablation experiment.

[0005] The application achieves the above-mentioned purposes through the following technical solutions.

[0006] 1. In step one, the data set is constructed, and the public data set for the ADvs.NC classification task is ADNI. The data set of the present application uses sMRI and PET images of 447 subjects and the cognitive scores of the subjects, including MMSE scale scores, CDR scale scores, and ADAS scale scores, wherein the NC subjects are 234, and the AD subjects are 213. According to the number of subjects, the data set is divided into 70%, 15%, and 15% according to the subject number, the data of the first n-1 numbered subjects is used as the training set, half of the data of the nth subject is used for verification, and the other half is used for testing, and all the subject data does not come from the same sample to avoid data leakage. Then, the two kinds of modal images are preprocessed through the Clinica software platform, and after bias correction, template registration and pixel normalization, the standard brain image is obtained, and the resolution of the preprocessed sMRI gray matter image and PET image is 121x145x121.

[0007] 2. In step two, the present application proposes the implementation of brain image feature extraction. The preprocessed two kinds of images are respectively subjected to feature extraction through a multi-layer attention residual network, the network uses a 3DResNet18 as the backbone, wherein the feature maps coded by four convolutional layers are subjected to global flat pooling, the four groups of features are concatenated and spliced as the input of the full connection layer, and the output is subjected to two full connections and a softmax layer. The extracted image features are the input features of the full connection layer, and the dimension is the sum of 64, 128, 256, and 512, that is, 960.

[0008] 3. In step three, the present application proposes a construction method of an adjacency matrix for a graph neural network to associate the image features with multiple cognitive scores. First, a subject edge weight matrix is constructed according to the multiple cognitive scores of the subject, and if there are M cognitive scores, M edge weight matrices are constructed; then the M edge weight matrices are added to calculate an average edge weight matrix, and the coefficient of each edge weight matrix is 1 / M; finally, the correlation matrix of the subject image features is calculated, and is subjected to element point multiplication with the optimized average edge weight matrix to obtain the adjacency matrix of the graph neural network. Each element of the adjacency matrix reflects the connection strength between different subjects, and the connection associates the cognitive information and the image features.

[0009] 4、Step four, the application proposes a two-layer graph convolutional neural network with multi-head attention to realize classification decision. The multi-head attention module enhances the feature representation of the graph neural network. The multi-head attention mechanism allows the model to focus on different parts of the input simultaneously and learn feature representations separately in each head, which helps to extract richer and more global features. After applying multi-head attention to the graph convolution model of each layer, the model's understanding and representation ability of the relationship between nodes can be further improved. In the graph neural network, the subject image features are associated through nodes, and the edge weights of the subjects are enhanced through cognitive scores in the adjacency matrix. The network is optimized using cross-entropy loss, and finally outputs the disease classification probability of each subject through the softmax layer. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is the sMRI and PET image preprocessing process schematic diagram of the application.

[0011] Figure 2 is the pre-training model network structure diagram of the application.

[0012] Figure 3 is the adjacency matrix construction principle diagram of the application.

[0013] Figure 4 is the overall structure diagram of the graph neural network of the application. DETAILED DESCRIPTION

[0014] The application will be further described below in conjunction with the drawings:

[0015] Figure 1 is the sMRI and PET image preprocessing process schematic diagram, including the following steps:

[0016] The application uses the following method for sMRI and PET image data preprocessing, including AC correction, registration and normalization. The SPM tool of the neural imaging data analysis software package is used on the Clinica software platform to perform AC correction on the image, and affine registration in MNI152 space is performed to align the image with the standardized template, realizing the registration of sMRI and PET images respectively, and extracting the gray matter from the registered sMRI. The processing of the two images is carried out in the MNI standard space.

[0017] Figure 2 is the pre-training model network structure diagram, including the following steps:

[0018] The preprocessed sMRI or PET image is input into a multi-layer attention residual network to train the network on the training set data for the AD vs. NC task, and a pre-trained model is obtained to extract the feature tensor of the corresponding image; the multi-layer attention residual network is a network with 3DResNet18 as the backbone, and the network structure includes four convolutional residual layers; the first convolutional layer includes 64-channel 3x3x3 convolutional kernels; the second convolutional layer includes 128-channel 3x3x3 convolutional kernels; the third convolutional layer includes 256-channel 3x3x3 convolutional kernels; and the fourth convolutional layer includes 512-channel 3x3x3 convolutional kernels; spatial channel attention and global average pooling are performed on each layer of features, and feature splicing is performed after the fourth convolutional layer to obtain the final image features.

[0019] Figure 3 is a schematic diagram of the construction principle of an adjacency matrix in a graph neural network, including the following steps:

[0020] First, the edge weight matrix of the subject is constructed using multiple cognitive score information, and the calculation expressions are as follows:

[0021]

[0022]

[0023]

[0024] where r M (M v ,M u ) represents the edge weight matrix of the MMSE cognitive score, M v ,M u is the MMSE cognitive score of the subject v and u; r C (C v ,C u ) represents the edge weight matrix of the CDR cognitive score, C v ,C u is the CDR cognitive score of the subject v and u; r D (D v ,D u ) represents the edge weight matrix of the ADAS cognitive score, D v ,D u is the ADAS cognitive score of the subject v and u. In the calculation process of formula (1), formula (2), and formula (3), if the cognitive scores of the subjects represented by two nodes are not more than 1 apart, the weight between the nodes will be increased and set to 1; otherwise, the edge weight between the nodes will be set to 0. Based on each phenotype information, an N x N dimensional edge weight matrix can be obtained for N subjects.

[0025] Then, the correlation distance is used to estimate the similarity S(·) between subjects v and u, and as the correlation matrix of the subject population, the expression is calculated as (4):

[0026]

[0027] Where ρ(·) is the correlation distance, σ is the width of the kernel, F v and F u are the image feature tensors of subject v and subject u.

[0028] Finally, the edge weight matrix is element-wise multiplied with the correlation matrix S(·) obtained according to equations (1), (2), (3), and the adjacency matrix A of the graph neural network is obtained, and the calculation expression is as (5):

[0029]

[0030] Where K is the number of cognitive score categories used, that is, the mean of the cognitive score categories is taken as the coefficient of different edge weight matrices, and K in the present application is 3.

[0031] Figure 4 is the overall structure of the graph neural network, including the following steps:

[0032] The overall structure of the graph neural network includes two graph convolution GCN models, first, the pre-trained model is used to extract the image features of the subjects from the corresponding modalities, and the image features are associated to the nodes in the graph; second, the adjacency matrix integrates the image features and the cognitive score information; finally, in the two-layer graph convolution neural network with multi-head attention, the prediction probability of the subject population is output through the softmax layer, and the final AD classification of the subject is realized.

[0033] A multi-head self-attention mechanism is used after each graph convolution layer in the graph neural network structure to enhance the expression of long-distance information. First, for each channel from the feature sequence after the graph convolution, the attention weight is calculated to represent the attention degree to the position channel. By multiplying the input matrix X with three learnable parameter matrices (query matrix W Q , key matrix W K , and value matrix W V ), the query Q, key K, and value V are obtained. Then, for each query q i , the correlation with all keys k j is calculated, the attention score is calculated by dot product, and finally the attention weight is obtained by softmax normalization. The outputs of the multi-head mechanism are spliced and merged through another learnable linear transformation to generate the final multi-head self-attention output. In the calculation process, assuming that the input matrix X is given, the calculation expressions of the query Q, key K, and value V are defined as Q=XW Q, K = XW K , V = XW V . Then, the attention score is calculated as follows:

[0034]

[0035] where d k is the dimension of the key K.

[0036] The output representation of multi-head self-attention is obtained by weighted sum of the value matrix V through attention weights:

[0037] MHA(Q, K, V) = Concat(head1,..., head h )W o (7)

[0038] where MHA(·) denotes multi-head attention, Concat(·) denotes feature concatenation, W o is the linear layer weight matrix, head i = Attention(QW i Q , KW i K , VW i V is the i-th scaled dot-product attention, i = 1, 2,..., h, and h = 4 is the number of heads.

[0039] In the testing process, in order to evaluate the classification performance of the network for Alzheimer's disease (AD) and normal control group (NC), three evaluation indexes are introduced according to the actual application, including accuracy (Accuracy), sensitivity (Sensitivity) and specificity (Specificity), and their calculation expressions are shown in (8), (9) and (10):

[0040] Accuracy = (TP + TN) / (TP + TN + FP + FN) (8)

[0041] Sensitivity = TP / (TP + FN) (9)

[0042] Specificity = TN / (FP + TN) (10)

[0043] where True Positive (TP) is the number of AD samples predicted as AD, False Positive (FP) is the number of NC samples predicted as AD, False Negative (FN) is the number of AD samples predicted as NC, and True Negative (TN) is the number of NC samples predicted as NC.

[0044] The accuracy rate refers to the percentage of the subjects classified correctly in the total number of subjects in the test set, which is used to evaluate the overall discriminant ability of the model in classifying AD and NC, the sensitivity is a measure of missed diagnosis, and the specificity is a measure of misdiagnosis. Table 1 is the classification comparison result of the method of the present application and the pre-training model method without fusion of cognitive scores. It can be seen that the method of the present application shows more obvious performance advantage in the classification and diagnosis of Alzheimer's disease on two different modal images.

[0045] Table 1 Comparison of diagnostic performance of the method of the present application and other network methods

[0046]

Claims

1. A method for classifying Alzheimer's disease based on fusion of images and cognitive scores, characterized in that Comprising the following steps: Step one: based on the public data set ADNI selection sMRI and PET image and image subject corresponding cognitive scale score information to construct the data set used in the application, formulate the non-leakage division data set and pre-process the brain image; Step two: feature acquisition is performed on the brain image, and the pre-processed image in step one is subjected to feature extraction by an attention residual network pre-training model, so as to obtain a 960-dimensional brain image feature tensor for each brain image; Step three: the edge weight matrix of the subject is constructed according to the cognitive score, and the adjacency matrix in the graph neural network is calculated by using the edge weight matrix and the image correlation matrix; Step four: the two-layer graph convolutional neural network with multi-head attention is used to realize the classification of Alzheimer's disease and normal control group, wherein the nodes in the graph neural network are associated with the image features of the subjects, and the network is optimized by using cross-entropy loss.

2. The Alzheimer's disease classification method based on fusion of images and cognitive scores according to claim 1, characterized in that The construction process of the data set and pre-processing in step one is as follows: the public database ADNI baseline data is used, and sMRI and PET images and the corresponding cognitive scores of the image subjects are selected to construct the data set of the application, the cognitive scores including MMSE scale score, CDR scale score and ADAS scale score; the training set, the validation set and the test set are divided by a non-leakage method, wherein the subjects in the test set cannot appear in the training set and the validation set; The sMRI and PET images are pre-processed, including bias correction, template registration and normalization, and the gray matter of the sMRI is extracted. 3.The Alzheimer's disease classification method based on fusion of images and cognitive scores according to claim 1, characterized in that In step two, the brain image is subjected to feature acquisition, and the construction process is as follows: the pre-processed sMRI gray matter image or PET image data in step one is subjected to feature extraction by a pre-training model; first, the residual network with multi-layer attention is used to train the network for the AD vs.NC task, and the pre-training model is obtained; then, the brain image is subjected to feature extraction by the pre-training model. 4.The Alzheimer's disease classification method based on fusion of images and cognitive scores according to claim 1, characterized in that In step three, the adjacency matrix is constructed, and the construction process is as follows: first, the edge weight matrix of the subject is constructed by using multiple cognitive scores, wherein the mean value of the cognitive scores is used as the coefficient of the different score edge weight matrix; then, the corresponding correlation matrix is calculated by using the image features of the subjects; finally, the edge weight matrix and the correlation coefficient matrix are subjected to element point multiplication, so as to obtain the adjacency matrix of the graph neural network.

5. The Alzheimer's disease classification method based on fusion of images and cognitive scores according to claim 1, characterized in that In step four, the two-layer graph convolutional neural network with multi-head attention is used to realize the classification decision, and the construction process is as follows: the feature tensor obtained in step two is associated with the nodes in the graph neural network, the graph neural network adopts a two-layer graph convolutional model and is subjected to multi-head attention after each layer, the network is optimized by using cross-entropy loss, wherein the training set and the validation set have labels in the network optimization, and the labels of the test set are covered by a mask.

Images