Alzheimer's disease development trajectory prediction system based on multi-modal feature fusion

Abstract

The invention discloses an Alzheimer's disease development trajectory prediction system based on multi-modal feature fusion, and belongs to the technical field of development trajectory prediction systems. A medical image preprocessing module is used for performing registration on head MRI image data of an Alzheimer's disease patient through an image obtained after skull extraction to obtain a brain image; registering to a unified image space to obtain preprocessed image data; the system comprises a clinical information preprocessing module for screening various clinical data of a patient to obtain important clinical indexes of multiple stages, and a disease trajectory prediction module for carrying out multi-modal fusion on an image and an index data set and carrying out training and prediction through a clustering model and an LSTM (Long Short Term Memory) network. Therefore, the data can be efficiently utilized, the model has good trajectory prediction precision, and the method has important significance for reducing the incidence rate of the Alzheimer's disease.

Description

Technical Field

[0001] The present invention relates to a development trajectory prediction system, in particular to an Alzheimer's disease development trajectory prediction system based on multimodal feature fusion, and belongs to the technical field of development trajectory prediction systems. Summary of the invention

[0002] The main purpose of the present invention is to provide an Alzheimer's disease development trajectory prediction system based on multimodal feature fusion.

[0003] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0004] The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion includes a preprocessing module, which is used to perform skull extraction and registration operations on the original head MRI images to obtain trainable images, and to screen and merge the clinical information of patients at different stages of the disease to obtain complete data;

[0005] A non-negative matrix factorization module is used to obtain the data of the preprocessing module and reduce the high-dimensional multimodal fusion features to two-dimensional and three-dimensional, and simultaneously simulate the two-dimensional patient memory and cognitive space;

[0006] The disease trajectory prediction module is used to perform multimodal fusion processing on the image and clinical information features transmitted through the non-negative matrix decomposition module and train the model to obtain the prediction results.

[0007] Preferably, the preprocessing module includes a medical image preprocessing module and a clinical information preprocessing module;

[0008] Medical image preprocessing module, used to perform skull extraction and registration operations on the original head MRI images to obtain trainable images;

[0009] The clinical information preprocessing module is used to screen and merge the clinical information of patients at different stages of illness to obtain complete data.

[0010] Preferably, the preprocessing module uses FSL tools to import the patient's head MRI image, perform skull extraction and align the image to the MNI152 space.

[0011] Preferably, the method of using the FSL tool to extract the skull and register the image to the MNI152 space comprises the following steps:

[0012] Step 1: Use central mass estimation to determine the center position of the brain and generate an initial spherical network model from the center position;

[0013] Step 2: Identify the boundary of brain tissue based on the gradient information of the MRI image and apply a dynamic contour model to deform the spherical network to gradually approach the brain boundary;

[0014] Step 3: Image registration: Align images acquired at different times, by different individuals or by different imaging devices to the same space. First, select a sample image in the MNI152 standard space, and preliminarily estimate the alignment parameters of the patient image and the sample image through centroid alignment and principal component analysis. Then, use a similarity metric function to evaluate the similarity of the two images after alignment. Then, use a multi-resolution pyramid structure to gradually optimize from low resolution to high resolution, using a gradient descent algorithm.

[0015] Step 4: Find the parameters that maximize the similarity metric function in the parameter space, and finally calculate the optimal linear transformation matrix to transform the patient image into the MNI152 space of the example image.

[0016] Preferably, the skull extraction uses BET in the FSL tool to distinguish the skull from the brain tissue by calculating the intensity threshold of each voxel, thereby realizing automatic removal of the skull;

[0017] The MNI152 standard space template was selected for image registration to ensure that brain images of different individuals can be compared and analyzed in a unified coordinate system. The intensity threshold formula of the voxels used is as follows:

[0018]

[0019] H(I) is the number of voxels with intensity value I;

[0020] is the gradient of the histogram.

[0021] Preferably, the clinical information preprocessing module screens out patients with follow-up records at the initial stage, 6th month, 12th month and 24th month, and uses their clinical indicators as part of the multimodal features.

[0022] Preferably, the data dimension is reduced by a non-negative matrix decomposition module, and the multimodal features are decomposed into a disease progression vector matrix and a disease progression indicator matrix, wherein the disease progression vector is used to predict the patient's disease progression stage at 24 months and 48 months, and the progression indicator matrix is ​​used to construct a simulation space for memory decline and cognitive function decline.

[0023] Preferably, the disease trajectory prediction module includes a Gaussian mixture model and a long short-term memory neural network LSTM model.

[0024] Preferably, a Gaussian mixture model is used to perform self-supervised clustering of the multimodal fusion data and fit into high, medium, and low progression rate patient groups to provide a basis for decision making;

[0025] The long short-term memory neural network LSTM model controls the transmission of information through the forget gate, input gate and output gate, and uses 5-fold cross-validation for evaluation. The optimal model is selected to fit the patient's actual disease development trajectory.

[0026] Beneficial technical effects of the present invention:

[0027] The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion provided by the present invention has the following advantages and positive effects; the present invention not only uses a deep learning neural network model to pre-process the MRI images and multiple clinical indicator data of Alzheimer's patients, but also places special emphasis on multimodal data fusion.

[0028] By efficiently fusing data features from different modalities, the performance of the model and the accuracy of prediction are further improved.

[0029] The advantage of multimodal fusion is that it can integrate data from different sources and extract richer information, thereby forming a more comprehensive and detailed description of the patient's health status.

[0030] This method not only improves the accuracy of identifying Alzheimer's disease, but also reduces the errors and biases that may be introduced by a single data source.

[0031] Through multimodal fusion, not only a more comprehensive disease analysis is provided, but also the reliability and robustness of the model are improved by cross-validation of multimodal data, which provides stronger support for clinical medicine and further optimizes the utilization of medical resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the overall block diagram of the Alzheimer's disease development trajectory prediction system based on multimodal feature fusion of the present invention

[0033] Figure 2 This is the basic framework diagram of the medical image preprocessing module of the present invention.

[0034] Figure 3 This is the basic framework diagram of the disease trajectory prediction module in the present invention. DETAILED DESCRIPTION

[0035] In order to make the technical solution of the present invention more clear and specific to those skilled in the art, the present invention is further described in detail below in conjunction with embodiments and drawings, but the implementation manner of the present invention is not limited thereto.

[0036] like Figure 1-Figure 3As shown, the Alzheimer's disease development trajectory prediction system based on multimodal feature fusion provided by this embodiment has three main modules in its overall framework. First, the patient's MRI images and clinical indicators are input as original inputs, and then the feature vectors obtained through their respective preprocessing processes are merged into the entire vector, and then the total features are decomposed into a progress vector matrix and a progress indicator matrix through the non-negative matrix decomposition part, and then the two matrices are input into the Gaussian mixture model and the LSTM network model for downstream task processing.

[0037] The present invention discloses an Alzheimer's disease development trajectory prediction system based on multimodal feature fusion, which uses multimodal feature fusion technology and deep learning neural network model to extract and learn features from the imaging data and clinical indicator data of Alzheimer's patients to predict the disease trajectory, including:

[0038] 1) Medical image preprocessing module, combined with Figure 2, showing the head MRI image processing and feature extraction process of the present invention. First, the original MRI image is imported through the FSL (FMRIB Software Library) tool. In order to eliminate the anatomical differences between individuals, it is uniformly registered to the MNI152 standard space. Then FSL uses edge detection and region growing methods to identify and peel off tissues other than the cerebral cortex, such as skin, skull, etc., and automatically determines the brain boundary by analyzing the intensity and spatial distribution characteristics of the image. Through iterative regional expansion and contraction, the brain boundary is gradually refined, and finally a pure brain image is extracted to achieve skull extraction; then it is registered to the MNI152 standard space again, and the transformation parameters are optimized by minimizing the difference between images (such as mutual information, cross-correlation, etc.) to achieve alignment of different individual brains. In order to realize the feature extraction of 3D MRI images, the present invention uses MedicalNet based on ResNet10 for extraction. MedicalNet is a 3D-based convolutional neural network. First, the voxels of the 3D MRI image are input into a convolutional layer (kernel size = 7×7×7, step size = 2, number of channels = 64), followed by a maximum pooling layer (kernel size = 3×3×3, step size = 2). Next, 8 residual blocks are applied, each of which contains 1 convolutional layer. The residual block parameters are as follows: Block 1 and 2: kernel size = 3 × 3 × 3, number of channels = 64, Block 3 and 4: kernel size = 3 × 3 × 3, number of channels = 128, Block 5 and 6: kernel size = 3 × 3 × 3, number of channels = 256, Block 7 and 8: kernel size = 3 × 3 × 3, number of channels = 512, where the convolutional layer stride of block 3 is 2, the convolutional layer expansion rate of block 5 is 2, and the convolutional layer expansion rate of block 7 is 4. Finally, global average pooling is applied, and a 512-dimensional fully connected layer is added to output 512-dimensional image features. This operation is performed for the initial stage, 6 months, and 12 months of MRI images, respectively.

[0039] 2) Clinical information preprocessing module. The present invention combines the common clinical evaluation indicators of Alzheimer's disease and selects the Montreal Cognitive Assessment, Clinical Dementia Rating Scale, Neuropsychiatric Symptom Questionnaire, Neuropsychological Test, Simple Mental State Examination Form, Geriatric Depression Scale, Daily Cognition-Research Partner Table, Daily Cognition-Subject Table, Functional Assessment Questionnaire, and Alzheimer's Disease Cognitive Assessment Scale. Subsequently, the present invention formulates a data processing strategy to process missing data records using the time information in the data, and fills in missing values ​​by linear interpolation based on feature past access records, thereby avoiding the influence of other observations during data interpolation. After interpolation, the amount of data was reduced by about 7%. One-hot encoding was used for categorical variables. For some numerical variables, in order to avoid the influence of certain features on other features, it is necessary to scale continuous features to the same range, so min-max normalization is used, and min-max normalization does not affect categorical features.

[0040] 3) Disease trajectory prediction module, combined with Figure 3 , showing the process of disease trajectory prediction by the present invention. The progress vector matrix and the progress indicator matrix after non-negative matrix decomposition are uniformly input into the long short-term memory neural network LSTM model. LSTM is a special recurrent neural network structure that can better learn and maintain long-term dependencies and is suitable for tasks such as trajectory prediction. LSTM consists of several key components, including cell state, forget gate, input gate, output gate and hidden state. The cell state is the core of LSTM and is responsible for transmitting information. The forget gate determines whether the previous information should be retained or forgotten. The input gate determines what new information should be added to the cell state. The output gate determines what kind of hidden state should be output at the current moment. The hidden state is the output of LSTM, which integrates the current input and cell state, can capture the long-term dependencies in the sequence, and is passed to the next time step to participate in the subsequent gated calculation. Through this carefully designed gating mechanism, LSTM can selectively remember and forget information, thereby performing well in various sequence modeling tasks. Subsequently, a 5-fold cross-validation method was used for evaluation. The optimal model parameters were a single hidden layer, 128 hidden units, a learning rate of 0.001, and a dropout probability of 0.2. The accuracy was 100% on the training set and 85% on the test set.

[0041] The present invention constructs a multimodal feature fusion method, using non-negative matrix decomposition technology. Non-negative matrix decomposition is a commonly used unsupervised dimensionality reduction technology, which attempts to convert the original high-dimensional data matrix, that is, the total matrix of multimodal features, which is as high as 2921 dimensions. In order to reduce computational overhead, dimensionality reduction is very necessary. NMF decomposes the large matrix into the product of two non-negative matrices W and H: X≈WH, where W is a k×m matrix, H is an m×n matrix, and k is the dimension after dimensionality reduction. By iteratively optimizing W and H, the reconstruction error ||X-WH||^2 is minimized, thereby obtaining the optimal dimensionality reduction mapping. This decomposition method ensures that the elements in W and H are non-negative, which makes the features learned by NMF interpretable. The present invention names the two matrices as the disease progression vector matrix and the disease progression indicator matrix, wherein the progression vector is used to construct a two-dimensional regression space, wherein the progression indicator matrix can map the original data into the regression space, and the modeling space of memory and cognitive function decline can be simulated.

[0042] The present invention constructs an unsupervised clustering method for multimodal features, using a Gaussian mixture model to define potential subtypes in patients with mild cognitive impairment and Alzheimer's disease. The Gaussian mixture model is an expectation maximization algorithm that maximizes the likelihood of observed data given distribution parameters, and then uses the Bayesian information criterion to select the optimal number of potential clusters for the Gaussian mixture model. The Bayesian information criterion is a maximum likelihood estimate that attempts to select the best model from a given set of candidate models. For all patients, the present invention inputs the two matrices after non-negative matrix decomposition into the Gaussian mixture model, determines the three optimal cluster numbers by selecting the minimum Bayesian information criterion value, and clusters them into high, medium and low disease progression rate clusters to assist doctors in making decisions under progression trajectory prediction.

[0043] 0041 The model experiment results show that the present invention effectively utilizes the head MRI image data and clinical indicator data of Alzheimer's patients under multimodal feature fusion, and fits the disease progression trajectory of the patients with higher accuracy. The corresponding characteristics of the images and indicators also ensure the matching degree between the data. In actual clinical applications, the trajectory fitted by the present invention can also help doctors discover the changes in the patient's condition earlier, intervene in the development of the disease as early as possible, and improve the patient's quality of life and the doctor's diagnostic ability.

[0044] The medical image preprocessing module of the present invention first imports the head MRI image data of the patient during the examination through the FSL (FMRIB Software Library) tool, performs skull extraction based on morphological contraction and expansion operations, and extracts brain tissue. Then, according to the intensity distribution of the brain tissue, the optimal translation, rotation and scaling parameters are found, and the brain MRI image is aligned to a standard template space MNI152, which can eliminate the anatomical differences between individuals and provide a standardization for subsequent analysis. The clinical information preprocessing module of the present invention first screens out complete clinical indicators within 24 or 48 months, collects the total scores and sub-item scores of cognitive, functional and longitudinal clinical data elements commonly used in clinical practice, and considers a total of 147 clinical variables for analysis, and all indicators match the patient's images.

[0045] The disease development trajectory prediction module of the present invention includes: constructing a non-negative matrix decomposition part to reduce the high-dimensional multimodal fusion features to 2 dimensions and 3 dimensions, which reduces the processing overhead and simulates the 2-dimensional patient memory and cognitive space; constructing a Gaussian mixture model part to perform unsupervised clustering of multimodal fusion features to find potential subtypes of AD, and divide them into high, medium and low risk groups to provide a basis for subsequent decision-making; constructing a long short-term memory neural network LSTM, and testing it in a 5-fold cross-validation manner, by optimizing parameters, selecting the best model to fit the patient's disease development trajectory.

[0046] The present invention constructs a non-negative matrix decomposition method, which fuses multimodal data and reduces them to a two-dimensional space, simulates a patient distribution space with cognitive decline on the horizontal axis and memory decline on the vertical axis, and reveals the overall distribution of patients.

[0047] Montreal Cognitive Assessment, which assesses overall cognitive function

[0048] The Clinical Dementia Rating Scale was used to assess global cognitive and functional aspects.

[0049] The Neuropsychiatric Symptom Questionnaire was used to assess psychiatric and behavioral symptoms.

[0050] Neuropsychological testing for a detailed neuropsychological assessment.

[0051] The Easy Mental State Examination was used to assess cognitive function.

[0052] The Annual Depression Scale was used to assess depressive symptoms.

[0053] The Everyday Cognition-Study Partner Form is used to assess everyday cognitive functioning, as assessed by the caregiver.

[0054] The Daily Cognition-Subject Form is used to assess daily cognitive function and is self-assessed by the patients.

[0055] Functional Assessment Questionnaire, used to assess daily living function.

[0056] The Alzheimer's Disease Cognitive Assessment Scale is used to assess cognitive function related to Alzheimer's disease.

[0057] The present invention constructs an LSTM network, whose architecture has a single-layer bidirectional LSTM layer connected to a fully connected layer, and uses a cross entropy loss function that combines the logarithmic and negative log-likelihood loss functions of softmax in the output layer. After adjusting the parameter optimization, the best 128 single hidden layer model structures are selected for trajectory fitting.

[0058] The above description is only a further embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and concepts of the present invention within the scope disclosed by the present invention, which belong to the protection scope of the present invention.

Claims

1. Alzheimer's disease development trajectory prediction system based on multimodal feature fusion, characterized by: It includes a preprocessing module, which is used to perform skull extraction and registration operations on the original head MRI images to obtain trainable images, and to screen and merge the clinical information of patients at different stages of illness to obtain complete data; A non-negative matrix factorization module is used to obtain the data of the preprocessing module and reduce the high-dimensional multimodal fusion features to two-dimensional and three-dimensional, and simultaneously simulate the two-dimensional patient memory and cognitive space; The disease trajectory prediction module is used to perform multimodal fusion processing on the image and clinical information features transmitted through the non-negative matrix decomposition module and train the model to obtain the prediction results.

2. The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion according to claim 1, characterized in that: The preprocessing module includes a medical image preprocessing module and a clinical information preprocessing module; Medical image preprocessing module, used to perform skull extraction and registration operations on the original head MRI images to obtain trainable images; The clinical information preprocessing module is used to screen and merge the clinical information of patients at different stages of illness to obtain complete data.

3. The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion according to claim 1, characterized in that: The preprocessing module uses the FSL tool to import the patient's head MRI images, extract the skull, and align the images to the MNI152 space.

4. The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion according to claim 3, characterized in that: The method of using FSL tools to extract skull bones and register images to MNI152 space includes the following steps: Step 1: Use central mass estimation to determine the center position of the brain and generate an initial spherical network model from the center position; Step 2: Identify the boundary of brain tissue based on the gradient information of the MRI image and apply a dynamic contour model to deform the spherical network to gradually approach the brain boundary; Step 3: Image registration: Align images acquired at different times, by different individuals or by different imaging devices to the same space. First, select a sample image in the MNI152 standard space, and preliminarily estimate the alignment parameters of the patient image and the sample image through centroid alignment and principal component analysis. Then, use a similarity metric function to evaluate the similarity of the two images after alignment. Then, use a multi-resolution pyramid structure to gradually optimize from low resolution to high resolution, using a gradient descent algorithm. Step 4: Find the parameters that maximize the similarity metric function in the parameter space, and finally calculate the optimal linear transformation matrix to transform the patient image into the MNI152 space of the example image.

5. The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion according to claim 4, characterized in that: The skull extraction uses the BET in the FSL tool to distinguish the skull from the brain tissue by calculating the intensity threshold of each voxel, thus achieving automatic removal of the skull. The MNI152 standard space template was selected for image registration to ensure that brain images of different individuals can be compared and analyzed in a unified coordinate system. The intensity threshold formula of the voxels used is as follows: H(I) is the number of voxels with intensity value I; is the gradient of the histogram.

6. The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion according to claim 2, characterized in that: The clinical information preprocessing module screens out patients with follow-up records at the initial stage, 6th month, 12th month, and 24th month, and uses their clinical indicators as part of the multimodal features.

7. The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion according to claim 1, characterized in that: The data dimension was reduced through the non-negative matrix factorization module, and the multimodal features were decomposed into a disease progression vector matrix and a disease progression indicator matrix. The disease progression vector was used to predict the disease progression stage of patients at 24 months and 48 months, and the progression indicator matrix was used to construct a simulation space for memory decline and cognitive function decline.

8. The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion according to claim 1, characterized in that: The disease trajectory prediction module includes a Gaussian mixture model and a long short-term memory neural network LSTM model.

9. The Alzheimer's disease development trajectory prediction system based on multimodal feature fusion according to claim 8, characterized in that: Gaussian mixture models were used to perform self-supervised clustering of multimodal fusion data and fit them into high, medium, and low progression rate patient groups to provide a basis for decision making; The long short-term memory neural network LSTM model controls the transmission of information through the forget gate, input gate and output gate, and uses 5-fold cross-validation for evaluation. The optimal model is selected to fit the patient's actual disease development trajectory.

Images