A method for interpretable, individual and group visualization of Alzheimer's disease based on multi-source heterogeneous data
Through the convolutional neural network of multi-source heterogeneous data and stacking integration algorithm, combined with the three-dimensional image interpretation module, the problems of low accuracy and insufficient interpretation of Alzheimer's disease diagnosis are solved, and efficient early warning and risk assessment are achieved. Improves the diagnostic accuracy and preventive ability of Alzheimer's disease.
Patent Information
- Application Number
- CN202410775752.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-17
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-06-17
AI Technical Summary
The existing Alzheimer's diagnosis methods have low accuracy, lack of early warning function and insufficient interpretation, making it difficult to effectively prevent disease progression.
A convolutional neural network model based on multi-source heterogeneous data is adopted, combined with a three-dimensional image interpretation module and a stacking integration algorithm, and individual and group risk assessment of early warning and visualization of Alzheimer's disease is achieved through transfer learning and feature fusion.
It improves the accuracy and interpretability of early warnings for Alzheimer's disease, can quantify the risk contribution at the three-dimensional pixel level, provide high-risk points and key points in the brain of individuals and groups, and enhances the generalization ability and prediction performance of the model.
Smart Images

Figure CN118553434B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an interpretable, individual visualization and group visualization method for Alzheimer's disease based on multi-source heterogeneous data, belonging to the technical field of interpretable artificial intelligence. Background Art
[0002] Currently, clinical methods are mainly used to complete the screening of Alzheimer's disease. The main screening methods include: cerebrospinal fluid examination, electroencephalogram examination, head CT examination, head magnetic resonance imaging and biomarker examination. These clinical methods have been verified in theory and practice for a long time and have advantages in pathological interpretability. However, the clinical methods for diagnosing Alzheimer's disease have the following problems: (1) The disease is difficult to diagnose. The accuracy of the doctor's subjective decision is related to his professional skills and experience. Even for experts with rich experience, the diagnostic accuracy is generally lower than that of other diseases. (2) There is no intervention time to avoid Alzheimer's disease. Once the clinical method screens out that the patient is a patient with Alzheimer's disease, this will become an established fact. For Alzheimer's disease, which is known as an incurable disease, prevention is more meaningful than treatment. Therefore, it is imperative to develop a data-driven decision-making algorithm with high prediction accuracy, early warning function of Alzheimer's disease and strong interpretability.
[0003] With the continuous iteration and upgrading of deep learning algorithms and (Graphics Processing Unit) GPU, the development of early warning of Alzheimer's disease using brain (Magnetic Resonance Imaging) MRI has become a reality. Given the hidden nature and complexity of potential Alzheimer's disease, multiple MRI images are usually required to characterize potential three-dimensional lesions, so three-dimensional deep learning algorithms are more suitable for the development of early warning models for Alzheimer's disease. The document Neuroimage Clin (Automated classification of Alzheimer's disease and mild cognitive impairment using a single MRI and deep neural networks) uses an end-to-end three-dimensional convolutional neural network to complete early warning of Alzheimer's disease. This invention will follow the strategy of extracting spatial features by three-dimensional convolution in the above document, but use brain sub-blocks as samples to expand the number of samples and improve the resolution of brain lesion visualization.
[0004] It is also noted that deep learning algorithms involving image classification, detection, and segmentation widely use transfer algorithms. The Alzheimer's disease warning models developed based on MRI data from different countries, races, and eras have differences and universality. The essence of the transfer algorithm is to use the features or (and) model parameters of the existing model as the initial features or (and) model parameters of the model to be developed in the new scenario, and then use a small number of samples in the new scenario to train a new warning model. This ensures the universality between scenarios and reflects the differences between scenarios, improving the generalization ability of the new model. The document Front Bioeng Biotechnol (Alzheimer's Disease Detection Through Whole-Brain 3D-CNN MRI) uses a feature-based migration algorithm for the (Computer-Aided Diagnosis of Dementia, Computer-Assisted Diagnosis of Alzheimer's Disease) CADDementia competition. The document first develops a deep learning algorithm for Alzheimer's disease warning in the (Alzheimer's Disease Neuroimaging Initiative, Alzheimer's Disease Neuroimaging Initiative) ADNI dataset, then uses logistic regression to replace the output layer of the deep learning algorithm, and freezes the weights of all layers before the logistic regression. Finally, the training set of the CADDementia competition is used as input to train the logistic regression. This invention will continue to use the idea of feature-based migration learning, but will add additional feature information such as age and (Mini-Mental State Examination) MMSE to the high-risk brain features screened out, so that multi-source heterogeneous data can be used in one model for model development. This mode of using images, audio, tabular data, and text data for single model development is usually called model fusion. Since the invention adds multiple tabular data, the conventional logistic regression algorithm has a weak ability to extract complex information of the newly added features. If an integrated algorithm is used to replace logistic regression, the model will have stronger generalization ability. The literature Sensors (Basel) (Diagnosis of Alzheimer's Disease with Ensemble Learning Classifier and 3D Convolutional Neural Network) uses a (bagging) integrated algorithm to take the weighted sum of multiple classifiers as the final prediction result of Alzheimer's disease.The invention will use a stacking ensemble algorithm, which has higher prediction performance than bagging algorithms. Specifically, it combines algorithms such as logistic regression, random forest, and (eXtreme Gradient Boosting) XGBoost into a super algorithm to achieve the goal of overall prediction performance surpassing local performance. The invention will use the stacking ensemble algorithm to complete the binary classification of the amplified feature vector, and ultimately achieve early warning of Alzheimer's disease.
[0005] For a long time, deep learning models for disease diagnosis have been difficult to apply clinically. Despite their high prediction accuracy, they lack the interpretability of prediction results that doctors can understand and trust. The early warning models developed in the above literature focus on prediction performance, but do not quantify the risk contribution of three-dimensional pixel levels that lead to Alzheimer's disease in individuals, nor do they qualitatively provide the bilateral temporal lobes, hippocampus, corpus callosum, parietal lobe, and frontal lobe areas where Alzheimer's patients are prone to lesions. The literature InCVPR (Learning Deep Features for Discriminative Localization) proposes a method that can visualize the area corresponding to the prediction result on a two-dimensional input image. However, this method is only applicable to simple networks without fully connected layers and requires retraining to obtain visualization graphs, which limits the development of high-performance neural networks and is not friendly to deep neural networks with expensive training costs. Summary of the invention
[0006] The purpose of the present invention is to provide an interpretable, individual visualization and group visualization method for Alzheimer's disease based on multi-source heterogeneous data. The model adopts a feature-based transfer learning structure and can simultaneously process the prediction and interpretation of multi-source heterogeneous sample data.
[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions:
[0008] A convolutional neural network model is built and trained based on the brain sub-blocks to mine highly specific semantics for distinguishing between normal cognition and Alzheimer's disease. The structure and weights of the trained convolutional neural network model are frozen to seal the learned general knowledge that can distinguish between normal cognition and Alzheimer's disease, ensuring that the general knowledge does not change.
[0009] Send the big brain block data of the current scene into the convolutional neural network with frozen weights and structures, perform forward calculations, and obtain corresponding general knowledge;
[0010] The 3D image interpretation module was used to quantitatively obtain the risk value of each pixel when the prediction results of the brain sub-blocks were normal cognition and Alzheimer's disease;
[0011] The risk values are arranged in descending order. According to the Pareto principle, the prediction results of the large brain block data of the current scene are taken as the front of the normal cognitive class and the Alzheimer's disease class. The high risk values and the tabular data of the predictors are fed into the stacking model as sample features, and the stacking model is trained;
[0012] The brain sub-block data of the test-taker is collected and sent to the trained convolutional neural network to quantify the voxel contribution values of the group and individual based on the three-dimensional image. The tabular data of the test-taker and the high-risk features obtained by the convolutional neural network are collected and sent to the stacking model to quantify the feature contribution values of individuals and groups based on the tabular data.
[0013] Preferably, the convolutional neural network model includes an input layer, five Block layers and an output layer in sequence, the first Block layer, the second Block layer, the third Block layer, and the fourth Block layer include a 3D convolution layer, a 3D maximum pooling layer, a 3D normalization layer, a LeakyRelu layer, and a Dropout layer in sequence, and the fifth Block layer includes a flatten layer, a fully connected layer, a LeakyRelu layer, and a fully connected layer in sequence, wherein the hyperparameters of the 3D convolution layer are determined by a genetic algorithm.
[0014] Preferably, the three-dimensional image interpretation module is as follows:
[0015] Get the feature atlas output by the LeakyRelu layer of the third Block layer ;
[0016] Process the brain sub-blocks through the trained convolutional neural network model to obtain their values before passing through the softmax function , Values indicating classification as cognitively normal; represents the value classified as Alzheimer's disease;
[0017] The feature map The weights of all pixel levels are averaged to obtain the feature map in the feature map set Importance ;
[0018] The linearized feature atlas The weighted sum and activation result is the predicted category: The activated feature map of
[0019] Using trilinear interpolation Upsample to the size of a three-dimensional brain block to obtain a risk feature map of the brain sub-block.
[0020] Preferably, the tabular data include age, education level, socioeconomic status, Mini-Mental State Examination, Clinical Dementia Rating Score, total brain volume, and white matter volume.
[0021] Preferably, the stacked model is divided into three layers, the first layer is 4 learners, the second layer is 3 learners, and the third layer is 1 learner; wherein, the prediction results of learner 1 and learner 2 in the first layer are used as input to learner 5, the prediction results of learner 2 and learner 3 are used as input to learner 6, and the prediction results of learner 3 and learner 4 are used as input to learner 7; the prediction results of learner 5, learner 6 and learner 7 in the second layer are used as input to learner 8; and the prediction result of learner 8 is the final prediction result of the stacked model.
[0022] Preferably, the learner 1 is the lightGNM algorithm, the learner 2 is the AdaBoosting algorithm, the learner 3 is the random forest algorithm, the learner 4 is the XGBoost algorithm, the learner 5 is the NaiveBayes algorithm, the learner 6 is the decision tree algorithm, the learner 7 is the KNN algorithm, and the learner 8 is the logistic regression algorithm.
[0023] Preferably, the stacking model uses the KernelSHAP algorithm to quantify the importance of individuals and groups.
[0024] A method for visualizing individual Alzheimer's disease based on multi-source heterogeneous data, including:
[0025] Using the three-dimensional image interpretation module to obtain the risk value of each pixel of the individual brain sub-block;
[0026] Take the risk mean of all brain sub-blocks of the individual at the pixel level;
[0027] The risk value of the individual whole-brain pixel level is calculated and the lesions are segmented to obtain an individual whole-brain risk visualization map.
[0028] A method for visualizing the Alzheimer's disease population based on multi-source heterogeneous data, including:
[0029] Using the three-dimensional image interpretation module to obtain the risk value of each pixel of the group brain sub-block;
[0030] Take the risk mean of all brain sub-blocks of each individual in the group at the pixel level;
[0031] Obtain the predicted results and actual results of each individual in the group by using the convolutional neural network model;
[0032] Calculate the pixel-level Matthew correlation coefficient for each pixel in the population , obtain the Matthew correlation coefficient of the whole brain pixels, obtain the key points of the group three-dimensional whole brain according to the coefficient size, and visualize it;
[0033] The Matthew correlation coefficient formula is as follows:
[0034] ,
[0035] in, Indicates that the predicted result is a patient with Alzheimer's disease, and the predicted result is consistent with the actual result; Indicates that the predicted result is a patient with Alzheimer's disease, and the predicted result is opposite to the actual result; It means that the predicted result is normal cognition and the predicted result is consistent with the actual result; It means that the predicted result is normal cognition and the predicted result is opposite to the actual result.
[0036] The advantages of the present invention are: the present invention provides a three-dimensional image interpretation method that is suitable for any complex three-dimensional convolutional neural network and does not require retraining to complete the visualization of the lesion area of the three-dimensional image of the early warning of Alzheimer's disease of the present invention. The image interpretation method can obtain the high-risk points and key points of the brain of individuals and groups, and combine the high-risk points of the brain of individuals with the age of potential patients, brain volume, etc. into tabular data and send them into the stacking integration algorithm to complete the final early warning of Alzheimer's disease. For tabular data, the present invention uses Kernel SHapley Additive exPlanations (kernel Shapley additive interpretation algorithm with kernel) KernelSHAP to quantify the characteristic contributions of individuals and groups of tabular data. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0038] Figure 1 A schematic diagram of the structure of an interpretable method for Alzheimer's disease based on multi-source heterogeneous data provided in an embodiment of the present application.
[0039] Figure 2 A flow chart of a genetic algorithm for determining hyperparameters of a convolutional neural network provided in an embodiment of the present application.
[0040] Figure 3 The prediction result provided in the embodiment of the present application is a visualization of the whole-brain three-dimensional risk and lesion segmentation of an individual with Alzheimer's disease.
[0041] Figure 4 A visualization of the importance of whole-brain pixel level for a group provided in an embodiment of the present application.
[0042] Figure 5A diagram of a learner deployment scheme for stacked integration provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] Example 1
[0045] like Figure 1 As shown in the figure, a method for interpretable, individual visualization and group visualization of Alzheimer's disease based on multi-source heterogeneous data includes a convolutional neural network module, a three-dimensional image interpretation module and a transfer learning module, with two training stages:
[0046] Phase 1: The invention develops a convolutional neural network based on brain blocks. The network has sufficient depth to mine highly specific semantics for distinguishing between cognitively normal and Alzheimer's disease classes. The data set used in the first phase is usually high-quality data with a large amount of data, strong data versatility, and has been tested in theory and application. The convolutional neural network trained based on this type of data has the advantages of high depth, low risk of overfitting, and strong model generalization ability. After training, the structure and weights of the trained convolutional neural network are frozen, and the general knowledge that has been learned to distinguish between cognitively normal and Alzheimer's disease classes is sealed to ensure that the general knowledge does not change.
[0047] The second stage: train another stacking model for the recognition of cognitive normal class and Alzheimer's disease class in the current scenario. The training data and test data used to develop the stacking model come from the current business scenario. The typical feature of the data is the small sample size. Using small sample data to develop a deep learning model will inevitably lead to high bias in the model, and it is difficult to mine the general knowledge that distinguishes the cognitive normal class from the Alzheimer's disease class. In the first stage, a convolutional neural network that preserves general knowledge has been trained. As long as the training set samples of the current business are sent to the convolutional neural network, the corresponding general knowledge can be obtained. Here, the invention uses a new three-dimensional graphics interpretation algorithm to mine the pixel-level risk value of the current business sample. The top 20% high-risk values of the sample prediction results for the cognitive normal class and the Alzheimer's disease class, as well as the predicted age, brain volume, education level, etc. are taken as features and sent to the stacking model, and the stacking model is trained. Then, the stacking model not only learns general knowledge but also learns the unique knowledge of the current scenario. The sample data of the stacking model is tabular data. The invention uses the KernelSHAP algorithm to quantify the contribution value of individual and group characteristics. At this point, the invention completes the development of an interpretable deep learning model based on multi-source heterogeneous data (image data and tabular data).
[0048] Specifically include:
[0049] S1: Build and train a convolutional neural network model based on brain sub-blocks, mine highly specific semantics for distinguishing between normal cognition and Alzheimer's disease, freeze the structure and weights of the trained convolutional neural network model, and seal the learned general knowledge that can distinguish between normal cognition and Alzheimer's disease to ensure that the general knowledge does not change.
[0050] S2: Send the big brain block data of the current scene into the convolutional neural network with frozen weights and structures, perform forward calculations, and obtain the corresponding general knowledge.
[0051] S3: Use the 3D image interpretation module to quantitatively obtain the risk value of each pixel when the brain sub-block prediction results are normal cognition and Alzheimer's disease.
[0052] S4: Arrange the risk values in reverse order, and according to the Pareto principle, take the top 20% high-risk values of the cognitive normal class and Alzheimer's disease class predicted by the big brain block data of the current scene and the tabular data of the predictor as sample features and send them into the stacking model, and train the stacking model.
[0053] S5: Collect the sub-block data of the brain of the test-taker and send it to the trained convolutional neural network to quantify the voxel contribution value of the group and individual based on the three-dimensional image. Collect the tabular data of the test-taker and the high-risk features obtained by the convolutional neural network and send them to the stacking model to quantify the feature contribution value of the individual and group based on the tabular data.
[0054] As a refinement of the above steps, the higher the resolution of the three-dimensional brain imaging, the more detailed and accurate the information of potential lesions is expressed. If the whole brain image is used as a sample, this will greatly increase the training cost of a single sample and easily cause the model to overfit. The usual practice is to compress (downsample) the three-dimensional brain image, but the compressed three-dimensional brain map is likely to lose key pixel information, and the prediction accuracy is weakened. The present invention adopts the method of randomly extracting brain sub-blocks to complete the development of the convolutional neural network module.
[0055] Its advantages are: (1) No need to compress the 3D image, the lesion information is not lost, and the prediction accuracy is not weakened. (2) The 3D image of the whole brain of an individual is randomly extracted into several sub-blocks, which is equivalent to expanding the number of samples, reducing the risk of overfitting in the simulation, and enhancing the generalization ability of the model. (3) The time and cost of sample training and testing are reduced, and high-resolution lesion structure visualization images can be provided to developers and users in real time.
[0056] Specifically, the convolutional neural network model training sample construction process in this embodiment is as follows: the whole brain MRI is preprocessed and grown into , width He Gaowei Pixel-based 3D brain maps, using Represents the brain map. A pixel point is randomly selected from the brain map as the center point of the sub-block, and a size of If the sub-block exceeds the range of the three-dimensional brain map, the sub-block is discarded and the above steps are repeated until the sub-block is extracted. If there are A three-dimensional brain map of an individual produces Sub-blocks, sample size expanded times.
[0057] Table 1 shows the structure of the convolutional neural network module. The convolutional neural network is an end-to-end structure, and its forward calculation order is input layer → Block1 → Block2 → Block3 → Block4 → Block5 → output layer. The forward calculation order in each Blocki (i=1,2,3,4) is 3D convolution layer → 3D maximum pooling layer → 3D normalization layer → LeakyRelu layer → Dropout layer. The forward calculation order in Blocki (i=5) is flatten layer → fully connected layer → LeakyRelu layer → fully connected layer. The invention sets eps=1e-05, momentum=0.1, and affine=Ture for the 3D normalization layer. The function of the parameter "eps" is to prevent the normalized value from being infinite; the function of the parameter "momentum" is to help accelerate convergence; the function of the parameter "affine" equal to "True" is to adjust the output back to the nonlinear region of the activation function. The invention sets the parameter "Negativeslope=0.01" of the LeakyRelu layer. The role of "Negativeslope" is to set the slope of negative numbers to 0.01, which helps to enhance the model's ability to learn negative responses. In the "Input / Output Size" column, Indicates the number of channels , length, width and height are ; express Round up to an integer; and so on for the rest.
[0058] Hyperparameters in bold italics in Table 1 C1 , C2 , C3 , C4 , N1 and N2 Determined by genetic algorithm. The genetic algorithm is as follows Figure 2 shown.
[0059] Table 1 Structure of convolutional neural network module
[0060]
[0061]
[0062]
[0063]
[0064]
[0065] As a refinement of the above steps, after completing the training and optimization of the convolutional neural network module, the doctor can use the convolutional neural network alone to predict whether the predictor is an Alzheimer's patient. Previous studies have focused more on improving the prediction performance, but the interpretability of the prediction results is more important. The importance of interpreting image data is reflected in the following aspects: (1) The prediction result is correct and there is no bias in the image interpretability. The doctor can formulate a clinical intervention plan based on the lesion area of the three-dimensional image and the quantified voxel risk value. (2) The prediction result is correct, but there is a bias in the image interpretability. The developer needs to re-check the training data and whether the model is biased, and re-develop the model. (3) The prediction result is wrong. The developer and the doctor can give a reasonable explanation through image interpretation. The three-dimensional image interpretation module designed by the invention can quantitatively calculate the risk value at the voxel level and can qualitatively segment the lesion structure.
[0066] Specifically, the specific processing process of the three-dimensional image interpretation module in this embodiment is as follows:
[0067] S3-1: Generally, the three-dimensional feature atlas closer to the output end has higher semantics, and it is more accurate to generate a pixel-level classifiable visualization based on this feature atlas. The invention uses the penultimate feature atlas (the output of the LeakyRelu layer of Block3 in Table 1), whose size is .
[0068] S3-2: A large brain block sample is input into the trained convolutional neural network and forward calculated. The value before passing the softmax function is recorded as , The corresponding values are classified as cognitively normal; The corresponding value is classified as Alzheimer's disease.
[0069] S3-3: The feature map in the penultimate feature map set is , the size of each feature map is . The importance of a feature map is :
[0070] ;
[0071] in, express Relative to the feature map The number of voxels The gradient of , which is calculated by back-propagation via the chain rule; Indicates that the feature map The weights of all pixel levels are averaged, and the average value is the feature map The importance of.
[0072] S3-4: Weighted sum and activation of the penultimate feature atlas:
[0073] ;
[0074] in It means that the linearized feature atlas set is weighted and summed into an inactivated feature map; the Relu function activates the inactivated feature map from the voxel level; Indicates the predicted category is The activated feature map. The value of the feature map activated by the Relu function is non-zero, and the greater the strength, the greater the probability of the predicted class, so it can be easily obtained from The high-risk areas are distinguished through numerical values and visualization maps.
[0075] S3-5: Use (Trilinear Interpolation) TI to Upsample to the size of a 3D brain patch.
[0076] ;
[0077] The size is , using python3 and the third-party library seaborn, the visualization of brain sub-blocks can be realized. In order to obtain a three-dimensional visualization of the whole brain of a predictor, the invention first creates a The whole brain risk feature map is empty, and then the predictor The brain sub-blocks correspond to The values are filled into the corresponding positions of the whole brain. If there are multiple values to fill a certain pixel point in the whole brain, the average is taken and then filled. For the pixels that are not assigned values, they are filled by three-line interpolation method based on the values that have been filled in the whole brain.
[0078] As a refinement of the above steps, the present invention also includes a transfer learning module, the essence of which is to apply the knowledge learned at other time and space points to the prediction of the current scene. The process of transfer learning in the present invention is:
[0079] (1) First, use the brain block data at other time and space points to train a convolutional neural network.
[0080] (2) Freeze the weights and structure of the convolutional neural network to preserve the knowledge of normal cognition and Alzheimer's disease learned from other points in time and space.
[0081] (3) Send a small amount of brain block data of the current scene into the convolutional neural network with frozen weights and structure, perform forward calculations, and obtain the penultimate feature atlas and The 3D image interpretation module is used to quantitatively obtain the risk value of each pixel when the prediction results of the brain sub-blocks are normal cognition and Alzheimer's disease, and the risk values are arranged in reverse order.
[0082] (4) Select the risk values of the top 20% pixels of the cognitive normal class and the Alzheimer's disease class respectively; concatenate the selected high-risk values with the predictor's tabular data into a row of data (still tabular data), and send it as a sample to the stacking model for training and testing.
[0083] The features of the predictor's tabular data used in this invention are consistent with the open source dataset (https: / / www.kaggle.com / datasets / shashwatwork / dementia-prediction-dataset), including Age, EDUC, SES, MMSE, CDR, TIV, and WBV. The literature Mendeley Data (Datafor: MACHINE LEARNING IN MEDICINE: CLASSIFICATION AND PREDICTION OF DEMENTIABY SUPPORT VECTOR MACHINES (SVM)) only uses the above 7 tabular data features, with an accuracy of 68.75%. This invention continues to use these 7 features, and additionally adds the top 20% high-risk features of the cognitive normal class and Alzheimer's disease class extracted from the brain, and then uses a stacked model with stronger generalization ability to learn the assembled tabular data information. The stacking model is divided into three layers, the first layer is learners 1-4, the second layer is learners 5-7, and the third layer is learner 8; the prediction results of learners 1-2 are used as the input of learner 5; the prediction results of learners 2-3 are used as the input of learner 6; the prediction results of learners 3-4 are used as the input of learner 7; the prediction results of learners 5-7 are used as the input of learner 8; the prediction results of learner 8 are the final prediction results of the transfer learning model. There are already mature interpretation methods for tabular data. This invention uses the KernelSHAP algorithm to quantify the feature contribution values of individuals and groups.
[0084] It should be noted that:
[0085] (1) The invention first trains a random block convolutional neural network on the ADNI dataset, combines the neural network with the stacking algorithm into a feature-based transfer learning method, and then uses the CADDementia dataset to train only the stacking algorithm. The invention can use the knowledge learned in different time and space to predict the current scene, which can not only ensure the ability to identify the common characteristics of potential Alzheimer's disease, but also take into account the characteristics of the lesions that cause potential Alzheimer's disease in the current scene.
[0086] (2) This invention uses multi-source heterogeneous data (three-dimensional brain images and tables) as input, and cleverly achieves model fusion by splicing feature vectors, greatly improving the predictive performance of the model.
[0087] (3) The invention interprets multi-source heterogeneous data. For three-dimensional brain maps, a three-dimensional image interpretation method is used to quantify the risk of individuals and groups at the voxel level, and a weakly supervised algorithm is used to segment the three-dimensional diseased tissue. For tabular data, the KernelSHAP algorithm is used to quantify the feature contribution values of groups and individuals.
[0088] Example 2
[0089] A method for visualizing individual Alzheimer's disease based on multi-source heterogeneous data, including:
[0090] S1: Build and train a convolutional neural network model based on brain sub-blocks, mine highly specific semantics for distinguishing between normal cognition and Alzheimer's disease, freeze the structure and weights of the trained convolutional neural network model, and seal the learned general knowledge that can distinguish between normal cognition and Alzheimer's disease to ensure that the general knowledge does not change.
[0091] S2: Send the big brain block data of the current scene into the convolutional neural network with frozen weights and structures, perform forward calculations, and obtain the corresponding general knowledge.
[0092] S3: Use the 3D image interpretation module to quantitatively obtain the risk value of each pixel when the prediction results of individual brain sub-blocks are normal cognition and Alzheimer's disease.
[0093] S4: Take the risk average of all brain sub-blocks of the individual at the pixel level.
[0094] S5: Calculate the risk value of the individual whole brain pixel level and segment the lesions to obtain an individual whole brain risk visualization map.
[0095] Example 3
[0096] A method for visualizing the Alzheimer's disease population based on multi-source heterogeneous data, including:
[0097] S1: Build and train a convolutional neural network model based on brain sub-blocks, mine highly specific semantics for distinguishing between normal cognition and Alzheimer's disease, freeze the structure and weights of the trained convolutional neural network model, and seal the learned general knowledge that can distinguish between normal cognition and Alzheimer's disease to ensure that the general knowledge does not change.
[0098] S2: Send the big brain block data of the current scene into the convolutional neural network with frozen weights and structures, perform forward calculations, and obtain the corresponding general knowledge.
[0099] S3: Use the 3D image interpretation module to quantitatively obtain the risk value of each pixel when the prediction results of the brain sub-blocks of the group are normal cognition and Alzheimer's disease.
[0100] S4: Take the risk mean of all brain sub-blocks of each individual in the group at the pixel level.
[0101] S5: Obtain the predicted results and the actual results of each individual in the group by using the convolutional neural network model;
[0102] S6: Calculate the pixel-level Matthew correlation coefficient for each pixel in the population , calculate the Matthew correlation coefficient of the whole brain pixels, obtain the key points of the group three-dimensional whole brain according to the coefficient size, and visualize it.
[0103] Considering the situation where TP*TN*FP*FN=0 and the MCC value is -1, the Matthew correlation coefficient formula is as follows:
[0104] ,
[0105] in, Indicates that the predicted result is a patient with Alzheimer's disease, and the predicted result is consistent with the actual result; Indicates that the predicted result is a patient with Alzheimer's disease, and the predicted result is opposite to the actual result; It means that the predicted result is normal cognition and the predicted result is consistent with the actual result; It means that the predicted result is normal cognition and the predicted result is opposite to the actual result.
[0106] Total The whole brain data of potential patients are randomly extracted from each whole brain brain blocks, a total of For each pixel in the whole brain in three dimensions, the corresponding TP, FP, TN and FP can be calculated, and the MCC value of the pixel can be obtained by substituting it into the MCC formula. By calculating the MCC value of all brain pixels, the importance of each pixel can be determined.
[0107] Example 4
[0108] This application is further elaborated based on the ADNI dataset.
[0109] (Alzheimer's Disease Neuroimaging Initiative) ADNI dataset is a global open dataset. Its main goals include:
[0110] (1) Early detection: Through neuroimaging, early signs of AD can be identified before patients show obvious clinical symptoms, thereby facilitating early intervention and treatment.
[0111] (2) Tracking disease progression: By conducting regular neuroimaging examinations on patients, the development and changes of the disease in the brain are recorded, providing researchers with detailed information about the progression of AD.
[0112] The invention develops a convolutional neural network based on the ADNI dataset. The inclusion criteria for the participants are:
[0113] People aged ≥50 years and diagnosed with normal cognition or Alzheimer's disease; people who have performed T1-weighted 3D brain mapping using a magnetic resonance imaging device with a magnetic field strength of 1.5 Tesla; the time between the 3D brain mapping scan and the diagnosis of normal cognition or Alzheimer's disease is 1 to 2 years.
[0114] Exclusion criteria for the participants were: patients with Alzheimer's disease with mixed dementia, non-Alzheimer's disease dementia, history of severe traumatic brain injury, severe depression, stroke and brain tumor, and patients with sudden onset of major systemic illness.
[0115] According to the inclusion and exclusion criteria, a total of 500 subjects with full brain images that met the requirements were selected, namely , of which 265 were female and 235 were male. 125 of the 500 people had Alzheimer's disease and 375 had normal cognition. After preprocessing, the size of each whole brain map is ,Right now The invention randomly extracts 1000 brain blocks from each whole brain, namely , each brain block is , , then the total number of brain block samples is 500,000. These 500,000 samples are divided into training set and test set according to 4:1. Based on the training set, the genetic algorithm with five-fold cross validation is used to optimize the hyperparameters in Table 1. The genetic algorithm process is detailed in Figure 2In the genetic algorithm, the population size is set to 100, the mutation probability is 0.1, the crossover probability is 0.1, and the number of genetic evolution generations is 5000. Finally, the hyperparameters of the convolutional neural network are determined as follows: C1 =150, C2 =120, C3 =100, C4 =100, N1 =500 and N2 =200.
[0116] Using the 3D image interpretation method, the interpretability of all brain images of a patient predicted to have Alzheimer's disease was integrated on the ADNI test set to construct a whole-brain risk visualization map and lesion area segmentation map of the patient predicted to have Alzheimer's disease ( Figure 3 ). Figure 3 The areas designated by 1-1, 2-1, and 3-1 represent the axial, coronal, and sagittal brain slices of the potential patient, respectively. Taking "2-1" as an example, 2 represents rows and 1 represents columns. 1-2, 2-2, and 3-2 are visualizations of the risk level at the voxel level corresponding to these three brain slices, respectively. The darker the color, the higher the risk (except for the marginal non-brain part). 1-3, 2-3, and 3-3 are the segmentations of the risk areas corresponding to these three brain slices, respectively. It is easy to see that the high-risk areas for the patient to develop Alzheimer's disease include the bilaterally asymmetric temporal lobes and right hippocampus, cingulate cortex, corpus callosum, part of the parietal lobe, and frontal lobe. Of course, the lesion is a three-dimensional irregular structure. Figure 3 The location of the lesion and the risk level are only shown through slices of the brain at three different planes. In addition, the output of the convolutional neural network is a binary output of softmax, that is, the output result also includes the cognitive normal class. According to the above visualization steps, the non-risk value of the whole brain pixel level and the segmentation of the normal area of the cognitive normal class can also be obtained.
[0117] Based on all the data of ADNI, using the three-dimensional image interpretation module and based on the MCC index, we can further obtain the importance of the whole brain pixel level of the group ( Figure 4 ). Figure 4 The left picture shows the MCC visualization results of the whole brain images of all individuals in the axial direction. Of course, this axial direction includes multiple slices. The white area corresponds to a larger MCC value, reflecting that this area is more effective in distinguishing between normal cognition and Alzheimer's disease. Figure 4 The middle and right images are the MCC visualization results of all individual whole-brain images in the axial and sagittal planes, respectively.
[0118] In order to further improve the generalization ability of the convolutional neural network, the invention uses all the data of ADNI and the determined hyperparameters to re-complete the training of the convolutional neural network. The trained convolutional neural network will be used for transfer learning in new business scenarios.
[0119] The invention has obtained the whole-brain data of 50 potential patients from a tertiary hospital in China, as well as tabular data such as brain volume, age, education level, brain trauma, and MMSE. The inclusion and exclusion criteria, the time of full brain image acquisition and the time interval between diagnoses, as well as the size of the whole brain and brain blocks and the preprocessing methods are exactly the same as the ADNI data set. A total of 50*1000=50,000 brain blocks were obtained. These 50,000 sub-blocks were split into training and test sets at a ratio of 4:1. The training set is used to train the stacking model. In order to determine the deployment of each learner in the stacking model, the invention uses an exhaustive method and ultimately determines the deployment plan with the highest predictive performance ( Figure 5 ).
[0120] In order to further verify the performance of the interpretable deep learning of the invention, the invention has designed the following benchmark models (learners). Benchmark model 1: The stacked integration part is replaced with lightGBM, and the rest is the same as the interpretable deep learning model of the invention; Benchmark model 2: The stacked integration part is replaced with logistic regression, and the rest is the same as the interpretable deep learning model of the invention; Benchmark model 3: Based on the trained convolutional neural network, the weights are allowed to change, that is, the training set data of a tertiary hospital is used to fine-tune the trained convolutional neural network. Benchmark model 4: The idea of transfer learning is not used, that is, the convolutional neural network trained on ADNI is not used for knowledge transfer, and only the data set of the tertiary hospital itself is used to develop an interpretable deep learning model. The performance of each model on the test set of a tertiary hospital is shown in Table 2.
[0121] Table 2 Performance comparison of the interpretable deep learning model of the invention and the benchmark model
[0122]
[0123] Obviously, the interpretable deep learning model of this invention has the highest predictive performance and the strongest generalization ability.
[0124] As the performance of the model of the invention is higher than that of the baseline model 1, it can be seen that the performance of the stacked ensemble exceeds that of the local learner lightGBM. As the performance of the baseline model 1 is higher than that of the baseline model 2, it can be seen that the ability of lightGBM to distinguish category differences is higher than that of logistic regression. As the performance of the model of the invention is higher than that of the baseline model 3, it can be seen that the invention uses a feature-based transfer learning approach that can be combined with tabular data such as age and brain volume to achieve further performance improvement; while parameter-based fine-tuning does not use tabular data, and its performance is relatively low. As the performance of the model of the invention is higher than that of the baseline model 4, it can be seen that the use of transfer learning can greatly improve the predictive performance of the model, and without using transfer learning, it is impossible to obtain good predictive performance on small samples.
[0125] For stacked models, we can use the KernelSHAP algorithm to quantify the importance of individuals and groups.
[0126] The features with higher calculated importance are (mini-mental status examination) MMSE, age, etc. Here, the invention provides a feature contribution value table corresponding to a patient in a tertiary hospital whose prediction result is Alzheimer's disease.
[0127] Table 3. The feature contribution value corresponding to the prediction result of a patient with Alzheimer's disease
[0128]
[0129] h1, i1, j1 are the voxel positions of the whole brain where the high risk value 1 is located; h8, i8, j8 are the voxel positions of the whole brain where the high risk value 8 is located.
[0130] Example 5
[0131] The disclosed embodiment also provides an Alzheimer's disease interpretable device based on multi-source heterogeneous data, including a processor and a memory. Optionally, the device may also include a communication interface and a bus. The processor, the communication interface, and the memory may communicate with each other through the bus. The communication interface may be used for information transmission. The processor may call the logic instructions in the memory to execute the Alzheimer's disease interpretable method based on multi-source heterogeneous data of the above embodiment.
[0132] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.
[0133] The memory, as a computer-readable storage medium, can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes the function application and data processing by running the program instructions / modules stored in the memory, that is, the Alzheimer's disease interpretable method based on multi-source heterogeneous data in the above embodiment is implemented.
[0134] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory.
[0135] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An interpretable method for Alzheimer's disease based on multi-source heterogeneous data, characterized in that: include: A convolutional neural network model is built and trained based on the brain sub-blocks to mine highly specific semantics for distinguishing between normal cognition and Alzheimer's disease. The structure and weights of the trained convolutional neural network model are frozen to seal the learned general knowledge that can distinguish between normal cognition and Alzheimer's disease, ensuring that the general knowledge does not change. Send the big brain block data of the current scene into the convolutional neural network with frozen weights and structures, perform forward calculations, and obtain corresponding general knowledge; The 3D image interpretation module was used to quantitatively obtain the risk value of each pixel when the prediction results of the brain sub-blocks were normal cognition and Alzheimer's disease; Arrange the risk values in descending order, and take the prediction results of the large brain block data of the current scene as the front of the normal cognitive class and the Alzheimer's disease class. The high risk values and the current tabular data of the predictor are fed into the stacking model as sample features, and the stacking model is trained; Collect the sub-block data of the test subject's brain and send it to the trained convolutional neural network to quantify the voxel contribution values of the group and individual based on the three-dimensional image; collect the tabular data of the test subject and the high-risk features obtained by the convolutional neural network and send them to the stacking model to quantify the feature contribution values of the individual and group based on the tabular data; Tabular data included age, education level, socioeconomic status, Mini-Mental State Examination, Clinical Dementia Rating Score, total brain volume, and white matter volume; The stacking model is divided into three layers. The first layer is 4 learners, the second layer is 3 learners, and the third layer is 1 learner. Among them, the prediction results of learners 1 and 2 in the first layer are used as the input of learner 5, the prediction results of learners 2 and 3 are used as the input of learner 6, and the prediction results of learners 3 and 4 are used as the input of learner 7. The prediction results of learners 5, 6 and 7 in the second layer are used as the input of learner 8. The prediction result of learner 8 is the final prediction result of the stacking model. The learner 1 is the lightGNM algorithm, the learner 2 is the AdaBoosting algorithm, the learner 3 is the random forest algorithm, the learner 4 is the XGBoost algorithm, the learner 5 is the NaiveBayes algorithm, the learner 6 is the decision tree algorithm, the learner 7 is the KNN algorithm, and the learner 8 is the logistic regression algorithm; The stacking model uses the KernelSHAP algorithm to quantify the importance of individuals and groups.
2. The Alzheimer's disease interpretable method based on multi-source heterogeneous data according to claim 1, characterized in that: The convolutional neural network model includes an input layer, five Block layers and an output layer in sequence. The first Block layer, the second Block layer, the third Block layer, and the fourth Block layer include a 3D convolution layer, a 3D maximum pooling layer, a 3D normalization layer, a LeakyRelu layer, and a Dropout layer in sequence. The fifth Block layer includes a flatten layer, a fully connected layer, a LeakyRelu layer, and a fully connected layer in sequence. The hyperparameters of the 3D convolution layer are determined by a genetic algorithm.
3. The Alzheimer's disease interpretable method based on multi-source heterogeneous data according to claim 2, characterized in that: The three-dimensional image interpretation module is as follows: Get the feature atlas output by the LeakyRelu layer of the third Block layer ; Process the brain sub-blocks through the trained convolutional neural network model to obtain their values before passing through the softmax function , Values indicating classification as cognitively normal; represents the value classified as Alzheimer's disease; The feature map The weights of all pixel levels are averaged to obtain the feature map in the feature map set Importance ; The linearized feature atlas The weighted sum and activation result is the predicted category: The activated feature map of Using trilinear interpolation Upsample to the size of a three-dimensional brain block to obtain a risk feature map of the brain sub-block.
4. A method for visualizing individual Alzheimer's disease based on multi-source heterogeneous data, characterized in that: include: Using a three-dimensional image interpretation module of any method described in claims 1 to 3 to obtain a risk value for each pixel of an individual brain sub-block; Take the risk mean of all brain sub-blocks of the individual at the pixel level; The risk value of the individual whole-brain pixel level is calculated and the lesions are segmented to obtain an individual whole-brain risk visualization map.
5. A method for visualizing Alzheimer's disease population based on multi-source heterogeneous data, characterized in that: include: Using a three-dimensional image interpretation module of any of the methods described in claims 1-3 to obtain the risk value of each pixel of the group brain sub-block; Take the risk mean of all brain sub-blocks of each individual in the group at the pixel level; Obtain the predicted results and actual results of each individual in the group by using the convolutional neural network model of any one of the methods of claims 1-3; Calculate the pixel-level Matthew correlation coefficient for each pixel in the population , obtain the Matthew correlation coefficient of the whole brain pixels, obtain the key points of the group three-dimensional whole brain according to the coefficient size, and visualize it; The Matthew correlation coefficient formula is as follows: , in, Indicates that the predicted result is a patient with Alzheimer's disease, and the predicted result is consistent with the actual result; Indicates that the predicted result is a patient with Alzheimer's disease, and the predicted result is opposite to the actual result; It means that the predicted result is normal cognition and the predicted result is consistent with the actual result; It means that the predicted result is normal cognition and the predicted result is opposite to the actual result.
6. An Alzheimer's disease interpretable device based on multi-source heterogeneous data, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the Alzheimer's disease interpretable method based on multi-source heterogeneous data as described in any one of claims 1-3 when running the program instructions.
Citation Information
Patent Citations
Medical image processing device and method employing convolutional neural network
CN107145727A
Alzheimer's disease classification method based on hierarchical integrated learning
CN109859839A
