GFE-Mama neural network-based interpretable Alzheimer's progress classification method
Through the GFE-Mamba neural network, combined with 3D GAN-Vit and Vision transformer, the pixel-level mutual attention mechanism is used to solve the problem of insufficient accuracy of AD classification in the existing technology, and more efficient feature extraction and diagnosis capabilities are achieved. The multimodal feature acquisition process is simplified.
Patent Information
- Application Number
- CN202410943009.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-15
AI Technical Summary
The existing Alzheimer's disease (AD) classification methods have insufficient prognostic interpretability and accuracy, especially the acquisition of PET images is time-consuming and difficult, and traditional machine learning and CNN methods cannot effectively extract images and scales. Pathological characteristics lead to insufficient accuracy of AD classification.
Using a GFE-Mamba neural network-based method, combined with 3D GAN-Vit, Vision transformer (VIT) bottleneck layer and multimodal mamba classifier, the pathological features in MRI images are extracted and PET is generated through the pixel-level mutual attention mechanism. Images, combined with scale information, use the mamba block backbone network to realize AD classification.
It improves the accuracy and feature extraction ability of AD classification, enhances clinicians' diagnostic ability of AD and other brain diseases, simplifies the diagnostic process, and reduces the difficulty of obtaining multimodal features.
Smart Images

Figure CN118918363A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to an interpretable Alzheimer's progression classification method based on a GFE-Mamba neural network. Background Art
[0002] At present, the methods for classifying the prognosis of aMCI patients and explaining the progression of Alzheimer's disease mainly include neuroimaging, cognitive scale scores, and biomarker detection. Knowledge scales are widely used in clinical practice for preliminary diagnosis, providing an effective solution for doctors to conduct preliminary screening of patients. However, it still faces many challenges in practical application, such as large individual differences and insufficient classification accuracy, so it can often only be used for preliminary diagnosis. In terms of neuroimaging, magnetic resonance imaging (MRI) and positron emission tomography (PET) are the two main technical means. MRI can provide detailed images of brain structure, helping doctors observe changes in brain volume and cortical atrophy, while PET can show brain metabolic activity and deposition of β-amyloid protein, which are usually important signs of early AD. In clinical diagnosis, PE shooting is time-consuming, expensive, and technically difficult, but its value in the early diagnosis of Alzheimer's disease (AD) cannot be ignored compared with MRI. PET technology can accurately reveal subtle changes in brain metabolic activity, especially for early pathological signs of AD such as abnormal deposition of β-amyloid protein in the brain, with extremely high sensitivity and specificity. This ability makes PET a powerful tool for classifying the risk of AD and the progression of the disease. However, the complexity and high requirements of PET technology limit its wide application. Its imaging relies on specific radioactive tracers and requires high-precision detectors and professional image reconstruction technology, which increases the difficulty and cost of its operation. In addition, PET examinations require high skills and experience from professionals, which further increases its threshold. Despite this, PET's unique value in providing key classification information and disease monitoring for AD is still irreplaceable. In summary, the classification of aMCI progression to AD requires consideration of the combined effects of multiple risk factors.
[0003] In addition, when diagnosing AD, clinicians need to use MRI and PET images and scale data at the same time. However, the acquisition of PET images is time-consuming and difficult, and the fusion of the multimodal features of the three types of data also has many areas for improvement. Therefore, the above-mentioned traditional machine learning AD classification method and the CNN-based multimodal AD classification method cannot achieve AD classification well. The required modalities are difficult to obtain and have major drawbacks. The traditional AD classification method process is relatively cumbersome, the extracted features are not significant enough, and pathological information cannot be accurately obtained. The CNN-based AD classification method cannot learn the long-distance dependencies of images due to the receptive field limitations caused by the convolution algorithm, and the reduction of the feature map size by the pooling layer of CNN may cause the loss of important pathological feature information. Therefore, it is impossible to better extract the pathological features of the image and scale, and then accurately classify AD. Summary of the invention
[0004] In view of the deficiencies of the prior art, the present invention proposes an interpretable Alzheimer's progression classification method based on the GFE-Mamba neural network. The model can effectively extract pathological features in MRI images and use the GFE module to extract MRI information to generate PET images (GFE) by integrating 3D GAN-Vit, Vision transformer (VIT) bottleneck layer, and mamba block backbone network, pixel level cross-attention, and extract both information, combine scale information and use the mamba block backbone network to achieve AD classification. This method can not only provide important reference standards for clinicians to diagnose and treat major dangerous diseases, but also further enhance clinicians' ability to diagnose AD and other major brain diseases, which is of great significance for clinical diagnosis and AD disease prevention.
[0005] In order to solve the above technical problems, the technical solution of the present invention is:
[0006] An interpretable Alzheimer's progression classification method based on GFE-Mamba neural network, comprising the following steps:
[0007] Step 1: construct a GFE-Manba neural network model, which includes a 3D GAN-Vit module, a multimodal mamba classifier, and pixel-level mutual attention.
[0008] The 3D GAN-Vit module uses 3D GAN as the backbone network, and uses the VIT bottleneck layer to replace the middle module in the 3D GAN; the multimodal Mamba classifier includes a time interval extraction module, a scale preprocessing module, and a Mamba backbone network module; the pixel-level mutual attention includes a query matrix, a key matrix, and a value matrix;
[0009] Step 2: Create MRI and PET image datasets and divide them into training sets and test sets, and use the MRI and PET image training sets and test sets to pre-train the 3D GAN-Vit module;
[0010] Step 3: Establish a multimodal Alzheimer's dataset combining MRI and scale data, and divide it into a training set and a test set;
[0011] Step 4: Use the multimodal Alzheimer's training set combining MRI and scale data to train the pre-trained GFE-Mamba neural network model and perform parameter tuning;
[0012] Step 5: Use the multimodal Alzheimer's test set that combines MRI and scale data to test and evaluate the pre-trained GFE-Mamba neural network model, which can finally explain the multimodal Alzheimer's classification.
[0013] Preferably, the 3D GAN-Vit module includes a generator and a discriminator, wherein the generator includes an encoder and a decoder based on a convolution module and an intermediate layer based on a VIT bottleneck layer, so that while performing generative feature extraction (GFE), 3D GAN-Vit can optimize the sequence processing network fusion with the mamba module.
[0014] Preferably, the encoder includes three downsampling modules, each layer of the downsampling module includes a maximum pooling layer, a group normalization layer, a convolution layer and a relay activation layer, and the channels of the three downsampling modules are 64, 128, and 256 respectively.
[0015] Preferably, the decoder comprises three upsampling modules, each of which comprises a group normalization layer, a transposed convolution layer and a relu activation layer, the number of channels of the upsampling module is mirrored to that of the downsampling module, and the filter sizes of the transposed convolution layers are 3x3x3 except for the first and last convolution layers which are 7x7x7.
[0016] Preferably, the discriminator structure and the encoder structure are the same.
[0017] Preferably, the input of the generator is:
[0018] MRI(iex M ), after passing through the encoder and decoder, PET(iey P ), PET(iey P ) as the input of the discriminator, for the PET data X PET ∈R H×W×D×C , generating a The generator loss is calculated using the feature map of
[0019] As a preferred method, the backbone network of the Classifier is a sequence processing module based on the Mamba module, and the scale information of the class sequence module is to be fused with the image information. Therefore, if the latent space features rich in spatial information are flattened directly and passed to the backbone network, not only will the original spatial information be lost, but the sequence information after flattening will also be very scattered. Therefore, the Vision transformer is introduced to perform mutual attention on the image after the vector in the latent space is flattened.
[0020] 1. After passing through the encoder, MRI is extracted into a latent space, and a feature map is flattened into 2D, which is then passed to the VIT bottleneck layer. In the VIT bottleneck layer, the feature map is split into a series of image patch sequences through Patch Embedding. Where p is the patch size and N is After the 3D feature map is converted into a sequence form, it can be passed into the transformer encoder. The transformer encoder consists of 4 transformer modules. After the transformer encoder, we get Then resize back to x LP ∈R H×W×D×C Then generate PET through Decoder. Preferably, the time interval extraction module adopts a dynamic Δt strategy, which is to obtain the time interval Δt between two diagnoses of the same patient and integrate it into the model as part of the training data. The value of the dynamic Δt adopts the average value of the training set. The method of integrating the dynamic Δt into the model is: merging the value of the dynamic Δt with the category value in the scale information and using it as the input of the scale preprocessing module. The scale information is obtained by converting the category value into a unique hot encoding, and after standardizing the digital value, it is embedded using a linear transformation, and then spliced together with the image features to form a multimodal input, which is passed to the classification network to classify whether the patient will change from MCI to AD after a specific time interval.
[0021] Preferably, the meter preprocessing module divides the meter information into discrete category values and continuously distributed numerical values.
[0022] The method for obtaining the discrete category values is as follows: first, the category values will be converted into one-hot encoding In order to avoid duplication of one-hot encoding in different rows, the value of the latter column will increase the maximum number of categories of the values of all previous columns during one-hot encoding. The expression is as follows:
[0023]
[0024] After getting the new value, you can use a linear matrix to embed it:
[0025]
[0026] in is the embedding representation of the i-th discrete category value, is the i-th feature bias, and is the i-th look up table;
[0027] The numerical value of the continuous distribution The method of obtaining is: standardization is required, that is, the mean of each column in the scale information is calculated and standard deviation And calculate the standardized value, that is: Then use linear transformation to embed it:
[0028]
[0029] in is the embedded representation of the i-th discrete category value, and then the processed discrete category value and the tabular information of the continuously distributed numerical value are spliced together, and also spliced together with the image features:
[0030]
[0031] Where T is the embedding concatenation result of discrete category values and continuous values, n and m are the number of rows of discrete category values and continuous distribution values, and x LMP , x LPP They represent the characteristics of MRI and PET in the latent space in the above-mentioned generative network, respectively, and d is the embedding size.
[0032] Preferably, the Mamba backbone network module includes 6 Mamba modules, which are classified as follows:
[0033] After processing, x will be passed to the classification network to classify whether the patient becomes AD after Δt. The Mamba backbone network module uses the state space model in it. It can classify the next state of the system according to the previous data through the hidden state and input sequence. After processing the table information and merging it with the image information, this sequence will be input to the classifier. After passing through 6 Mamba modules, the average and linear layer are calculated to obtain the final classification result.
[0034] Preferably, the classification method of the Mamba module is:
[0035] For each Mamba module, the input is first RMS normalized, which is achieved by the root mean square value of the input activation, effectively preventing the gradient explosion problem in deep networks, and then added to the residual of the input itself after passing through the Mamba module:
[0036] x i+1 =Mamba(RMSNorm(x i ))+x i
[0037] The input features will first pass through the linear layer and be split into two: x, z = split(linear(x)), where x will first pass through a 1d convolution and be passed into the Selective Scan Model after activation: y = SSM(Conv(x)), and z, as the gate vector, will be element-wise multiplied with y after activation, and then y will pass through a linear layer to obtain the final result of this module.
[0038] Preferably, the implementation method of the pixel-level mutual attention module is:
[0039] Get the output of the last Mamba module, marked as Before the final classification, the mutual attention with MRI and PET is added, where MRI and PET are respectively recorded as Flatten it into The specific process of mutual attention is as follows:
[0040] Q y =W q y
[0041] K x =W k x M
[0042] V x =W v x M
[0043]
[0044] Where Q y It is the query matrix, W q is the weight of the query matrix, K x is the key matrix, W k is the weight of the key matrix, x M is the MRI feature, V x is the value matrix, W v is the weight of the value matrix, d k The dimension of the key matrix.
[0045] The query matrix is obtained by linearly changing the output of the Classifier, while the key matrix and the value matrix are obtained by linearly changing the features of the MRI sequence. For PET, the same mutual attention process is performed again. The features after the mutual attention calculation are residually added to y, and after the feedback layer and layer normalization operation, the residual is added to the original y again.
[0046] Preferably, the training method of the GFE-Mamba neural network model is:
[0047] When training the 3D GAN-Vit module, the resolution of MRI and PET was adjusted to 160, 160, 96. The 3D GAN-Vit module was trained on two NVIDIA GeForce RTX4090Ti GPUs for 200 rounds of distributed training with a batch size of 2, which took 52.5 hours.
[0048] When training the multimodal mamba classifier, a pair of data is randomly selected from the table, including the table information and the patient's name and diagnosis time. The corresponding MRI is found based on the name and diagnosis time. The data is randomly cut into 160, 160, and 100 rounds of training are performed on an NVIDIA GeForce RTX4090 with a batch size of 8, which takes 22 hours;
[0049] The optimization of the 3D GAN-Vit module and the multimodal mamba classifier both adopts the Adam algorithm, and the learning rate and betas are 0.9 and 0.999 respectively.
[0050] Preferably, the evaluation method of the GFE-Mamba neural network model is:
[0051] The five indicators of accuracy, precision, recall, F1-score and balance are used for evaluation. The expressions are as follows:
[0052]
[0053] Among them, Accuracy is accuracy, the overall proportion of correct classification; Precision is precision, the proportion of true positives in positive classification; Recall is recall, the proportion of correctly identified positive instances; F1-score is F1-score, the harmonic mean of precision and recall; MCC is a balanced indicator to measure the quality of binary classification, TP and TN represent real samples in data classified as true or false, FP and FN represent false samples in data classified as true or false.
[0054] The present invention has the following characteristics and beneficial effects:
[0055] Using the above technical solution, clinicians need to use MRI and PET images and scale data at the same time when diagnosing AD, but the acquisition of PET images is time-consuming and difficult, and the fusion of multimodal features of the three data also has many areas that can be improved. Therefore, the above-mentioned traditional machine learning AD classification method and the multimodal AD classification method based on CNN cannot achieve AD classification well, and the required modality is difficult to obtain and has great disadvantages. The traditional AD classification method process is relatively cumbersome, the extracted features are not significant enough, and pathological information cannot be accurately obtained. The AD classification method based on CNN, due to the limitation of the receptive field caused by the convolution algorithm, cannot learn the long-distance dependency of the image, and the reduction of the feature map size by the pooling layer of CNN may cause the loss of important pathological feature information, so it is impossible to better extract the pathological features of the image and the scale, and then accurately classify AD. However, the present invention uses GFE to extract important MRI information using pre-trained MRI to PETGAN, and uses the mutual attention mechanism to process the vector of the latent space to avoid information loss and sequence confusion, and uses the Mamba backbone network to effectively process the dependency between long sequences to improve the feature extraction efficiency, further improving the accuracy of AD classification.
[0056] Therefore, in the invention, focusing on the theory of pattern recognition and medical image and scale data processing, combined with the actual characteristics of Alzheimer's MRI image data and scale data and the actual process and diagnosis method of Alzheimer's diagnosis by psychiatrists, targeted research on the Alzheimer's progression classification algorithm is carried out, and finally a multimodal Alzheimer's progression classification method based on the GFE-Mamba neural network composed of 3D GAN-Vit, Vision transformer (VIT) bottleneck layer, and multimodal mamba classifier, and pixel level cross-attention is proposed. The GFE-Mamba model solves the problems of limited feature extraction ability and parameter redundancy caused by traditional CNN by introducing 3D GAN-Vit and multimodal Mamba classifier. At the same time, our GFE-Mamba model makes up for the high computational complexity and insufficient global information capture of the 3D CNN model when processing high-dimensional neural imaging data by introducing the pixel-level cross-attention mechanism. Therefore, GFE-Mamba has been significantly improved in feature expression ability and model interpretability.
[0057] The multimodal Alzheimer's progression classification method (GFE-Mamba) proposed in the present invention only needs to input MRI and scale data to realize automatic and intelligent multimodal Alzheimer's progression classification function. The present invention makes it possible to establish an Alzheimer's progression classification model and assist doctors in diagnosis and treatment, which is of great significance in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0059] Figure 1 The overall structure diagram of the GFE-Mamba neural network model of the present invention;
[0060] Figure 2 This is a schematic diagram of the 3D GAN-Vit module of the present invention;
[0061] Figure 3 This is a schematic diagram of the multimodal Mamba classifier module of the present invention;
[0062] Figure 4 Schematic diagram of the pixel-level mutual attention module of the present invention;
[0063] Figure 5Schematic diagram of the process of constructing the MCI-AD dataset from the ADNI dataset in the present invention. DETAILED DESCRIPTION
[0064] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0065] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.
[0066] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.
[0067] The present invention provides a multimodal Alzheimer's progression classification method based on GFE-Mamba neural network, such as Figure 1 As shown, the multimodal Alzheimer's progression classification method based on the GFE-Mamba neural network consists of 3D GAN-Vit, a Vision transformer (VIT) bottleneck layer, a multimodal mamba classifier, and pixel level cross-attention.
[0068] The method comprises the following steps:
[0069] Step 1: Build Figure 1 A GFE-Mamba neural network model for classifying Alzheimer's disease, wherein the GFE-Mamba neural network model consists of a 3D GAN-Vit, a Vision transformer (VIT) bottleneck layer, a multimodal mamba classifier, and pixel level cross-attention.
[0070] Specifically, Figure 2 As shown, the 3D GAN-Vit module consists of an encoder and decoder based on a convolutional module and an intermediate layer based on a Vision transformer architecture.
[0071] The encoder contains three down-sampling modules, each layer contains: maximum pooling layer, group normalization layer, convolution layer and relay activation layer, and the channels of the three modules are: 64, 128, 256. The decoder contains three up-sampling modules, each layer contains: group normalization layer, transposed convolution layer and relu activation layer, and the number of channels is consistent with the down-sampling module. The filter size of the convolution layer is 3x3x3 except for the first and last convolution layers, which are 7x7x7.
[0072] MRI(iex M ) is passed into the generator as input and generates PET(iey P ), PET is used as the input of the discriminator. The discriminator uses the same three downsampling modules as before, for X PET ∈R H×W×D×C , generating a The feature map of the 3D GAN-Vit module is divided into two parts: the generator loss and the discriminator loss.
[0073] The loss of the generator is:
[0074]
[0075] Among them, the first term refers to the MSE reconstruction loss between MRI and the generated PET, the second term refers to the adversarial loss of the generator, and the third term refers to the perceptual loss extracted by the vgg19 network.
[0076] The loss of the discriminator is:
[0077]
[0078] Among them, the first and second terms refer to the adversarial loss of the discriminator on the real PET and the generated PET.
[0079] Furthermore, based on the original 3DGAN network, the middle module, which was originally composed of the Resnet module, was replaced with the Vision transformer. This is because the backbone network of the Classifier is a sequence processing module based on the Mamba module, and the scale information of the sequence-like module is to be fused with the image information. Therefore, if the latent space features rich in spatial information are flattened directly and passed to the backbone network, not only will the original spatial information be lost, but the sequence information after flattening will also be very scattered. Therefore, we introduced the Vision transformer to perform mutual attention on the image after the vector in the latent space is flattened. The specific structure of the model is as follows: Figure 1As shown in the figure, MRI is extracted as a latent space after passing through the encoder. This 3D feature map is first flattened into a 2D feature map and then passed into the Vision transformer. In the Vision transformer, the feature map is split into a series of image patch sequences through Patch Embedding. Where p is the patch size and N is After the 3D feature map is converted into a sequence form, it can be passed into the transformer encoder. The transformer encoder consists of 4 transformer modules. After the transformer encoder, we get Then resize back to x LP ∈R H×W×D×C Then the decoder generates PET. After pre-training, and can be considered to represent the information of MRI and PET in the latent space respectively. They will be collected and passed to the classifier as image modality information for the next part of information fusion.
[0080] Further, such as Figure 3 As shown, the multimodal Mamba classifier consists of a time interval extraction module, a scale preprocessing module, and a Mamba backbone network module.
[0081] Specifically, the time interval extraction module adopts a dynamic Δt strategy: that is, the time interval Δt between two diagnoses of the same patient is obtained (excluding abnormal values, such as the time interval between two diagnoses is one day) and integrated into the model as part of the training data. During inference, the value of Δt is inferred using the average value of the training set. As for the way to integrate Δt into the model, it is merged with the category value in the scale information and forward propagated as part of the table data. The scale preprocessing module divides the scale information into discrete category values (category values) and continuously distributed numerical values (number values).
[0082] For discrete category values: First, the category values will be converted into one-hot encoding To avoid duplication of unique hot encodings in different rows, the value of the latter column will increase the maximum number of categories of the values of all previous columns: After getting the new value, you can use a linear matrix to embed it:
[0083]
[0084] in is the embedding representation of the i-th discrete category value, is the i-th feature deviation, and is the i-th lookup table.
[0085] For numeric values For example, standardization is required, that is, the mean of each column is calculated. and standard deviation And calculate the standardized value, that is:
[0086] Then use linear transformation to embed it:
[0087]
[0088] Then the processed categorical and numerical values are concatenated together and concatenated with the image features:
[0089]
[0090] Where n and m are the number of rows of categorical and numerical values, and x LMP , x LPP They represent the characteristics of MRI and PET in the latent space in the above-mentioned generative network, respectively, and d is the size of embedding.
[0091] After processing, x will be passed to the Mamba backbone network module to classify whether the patient becomes AD after Δt. The Mamba backbone network module uses the state space model therein, which can classify the next state of the system according to the previous data through the hidden state and input sequence. After processing the table information and merging it with the image information, this sequence will be input to the classifier, and its backbone network consists of 6 Mamba modules. After the input sequence passes through 6 Mamba modules, the average value and linear layer are calculated to obtain the final classification result. For each Mamba module, the input will first be RMS normalized, and the normalization is achieved by the root mean square value of the input activation, which effectively prevents the gradient explosion problem of the deep network, and then it is added to the residual of the input itself after passing through the Mamba module:
[0092] x i+1 =Mamba(RMSNorm(x i ))+x i
[0093] The structure of each Mamba module is shown in the figure. The input feature will first pass through the linear layer and be divided into two: x, z = split (linear (x)), where x will first pass through a 1d convolution and be passed into the SelectiveScan Model after activation: y = SSM (Conv (x)). As a gate vector, z will be element-wise multiplied with y after activation. Then y passes through a linear layer to get the final result of this module. After cross attention is performed on awdad and PET respectively, it passes through a linear layer to get the final binary classification result.
[0094] Further, such as Figure 4 As shown, the pixel-level mutual attention module obtains the output of the last Mamba module in the classifier, marked as Before the final classification, the mutual attention with MRI and PET was added. Flatten it into That is, and. For MRI, the specific process of mutual attention is as follows:
[0095] Q y =W q y
[0096] K x =W k x M
[0097] V x =W v x M
[0098]
[0099] Where Q y It is the query matrix, W q is the weight of the query matrix, K x is the key matrix, W k is the weight of the key matrix, x M is the MRI feature, V x is the value matrix, W v is the weight of the value matrix, d k The dimension of the key matrix.
[0100] The query matrix is obtained by linearly changing the output of the Classifier, while the key matrix and the value matrix are obtained by linearly changing the features of the MRI sequence. For PET, the same mutual attention process is performed again. The features after the mutual attention calculation are residually added to y, and after the feedback layer and layer normalization operation, the residual is added to the original y again.
[0101] Step 2: Create MRI and PET image datasets, which are divided into training sets and test sets for pre-training the 3D GAN-Vit module. Then, create a multimodal Alzheimer's dataset combining MRI and scale data, which is divided into training sets and test sets.
[0102] In this embodiment, the Alzheimer's progression classification dataset is a paired MRI and PET dataset (MRI-PET dataset) and a dataset for Classifier to determine whether it has turned into AD after Δt (MCI-AD dataset), which is obtained as follows: Figure 5 . The MRI and PET datasets consist of 2843 pairs of MRI and PET data, divided into 2274 pairs of training sets and 569 pairs of validation sets. The MCI-AD dataset is divided into a 1-year dataset and a 3-year dataset, which are data within one year and data within three years respectively. The 1-year dataset is divided into a training set of 242 samples and a test set of 60 samples, and the 1-year dataset is divided into a training set of 281 samples and a test set of 70 samples. The MCI-AD dataset consists of 136 positive samples and 155 negative samples, and the samples have scale data and MRI data.
[0103] Step 3: Use the training set to train the GFE-Mamba neural network model. The training of the 3D GAN-Vit module and the multimodal mamba classifier are both performed on Pytorch 2.0cuda 11. The library for reading NIfTI format 3D images is monai, and the library for reading tables and converting them into training data is pandas.
[0104] Specifically, when training 3D GAN-Vit, the resolution of MRI and PET will be adjusted to (160, 160, 96). 3DGAN-Vit performed 200 rounds of distributed training on two NVIDIA GeForce RTX4090Ti with a batch size of 2, which took 52.5 hours. When training the classifier, a pair of data will be randomly selected from the table, including the table information and the patient's name and diagnosis time. The corresponding MRI is found based on the name and diagnosis time, and randomly cut to (160,160).
[0105] The training of the multimodal mamba classifier took 22 hours for 100 epochs on an NVIDIA GeForce RTX4090 with a batch size of 8. Both models were optimized using the Adam algorithm with learning rates and betas of (0.9, 0.999) respectively.
[0106] Step 4: Use the test set to test and evaluate the obtained network model, and finally realize the automatic and intelligent multimodal Alzheimer's progression classification function.
[0107] In order to evaluate the model's performance in image segmentation, GFE-Mamba is compared with other classification models on the ADNI-derived dataset. In this embodiment, five standard indicators are used to evaluate the classification performance. They are: Accuracy is accuracy, the overall proportion of correct classification; Precision is accuracy, the proportion of true positives in positive classification; Recall is recall, the proportion of correctly identified positive instances; F1-score is the F1-score, the harmonic mean of precision and recall; MCC is a balanced indicator to measure the quality of binary classification. Their formulas are as follows
[0108]
[0109] Among them, TP and TN represent real samples in the data classified as true or false, and FP and FN represent false samples in the data classified as true or false.
[0110] In this embodiment, focusing on the theory of pattern recognition and medical image and scale data processing, combined with the actual characteristics of Alzheimer's MRI image data and scale data and the actual process and diagnosis method of psychiatrists diagnosing Alzheimer's, targeted research on the Alzheimer's progression classification algorithm is carried out, and finally a multimodal interpretable Alzheimer's progression classification method based on the GFE-Mamba neural network composed of 3D GAN-Vit, Visiontransformer (VIT) bottleneck layer, and multimodal mamba classifier, and pixel-level cross-attention is proposed. The GFE-Mamba model solves the problems of limited feature extraction capability and parameter redundancy caused by traditional CNN by introducing 3D GAN-Vit and multimodal Mamba classifier. At the same time, the GFE-Mamba model makes up for the problem of high computational complexity and insufficient global information capture of the 3D CNN model when processing high-dimensional neural imaging data by introducing a pixel-level cross-attention mechanism. Therefore, GFE-Mamba has been significantly improved in terms of feature expression ability and model interpretability.
[0111] The multimodal Alzheimer's progression classification method (GFE-Mamba) proposed by the present invention can realize automatic and intelligent multimodal Alzheimer's classification function by only inputting MRI and scale data. The present invention makes it possible to establish an Alzheimer's classification model and assist doctors in diagnosis and treatment, which is of great significance in practical applications.
[0112] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.
Claims
1. An interpretable Alzheimer's progression classification method based on GFE-Mamba neural network, characterized in that: The steps include: Step 1: construct a GFE-Manba neural network model, which includes a 3D GAN-Vit module, a multimodal mamba classifier, and pixel-level mutual attention. The 3D GAN-Vit module uses 3D GAN as the backbone network, and uses the VIT bottleneck layer to replace the middle module in the 3D GAN; the multimodal Mamba classifier includes a time interval extraction module, a scale preprocessing module, and a Mamba backbone network module; the pixel-level mutual attention includes a query matrix, a key matrix, and a value matrix; Step 2: Create MRI and PET image datasets and divide them into training sets and test sets, and use the MRI and PET image training sets and test sets to pre-train the 3D GAN-Vit module; Step 3: Establish a multimodal Alzheimer's dataset combining MRI and scale data, and divide it into a training set and a test set; Step 4: Use the multimodal Alzheimer's training set combining MRI and scale data to train the pre-trained GFE-Mamba neural network model and perform parameter tuning; Step 5: Use the multimodal Alzheimer's test set that combines MRI and scale data to test and evaluate the pre-trained GFE-Mamba neural network model, which can finally explain the multimodal Alzheimer's classification.
2. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 1, characterized in that: The 3D GAN-Vit module includes a generator and a discriminator. The generator includes an encoder and a decoder based on a convolution module and an intermediate layer based on a VIT bottleneck layer.
3. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 2, characterized in that: The encoder includes three downsampling modules, each of which includes a maximum pooling layer, a group normalization layer, a convolution layer and a relay activation layer, and the channels of the three downsampling modules are 64, 128, and 256 respectively.
4. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 3, characterized in that: The decoder includes three upsampling modules, each of which includes a group normalization layer, a transposed convolution layer and a relu activation layer. The number of channels of the upsampling module is consistent with that of the downsampling module. The filter size of the transposed convolution layer is 3x3x3 except for the first and last convolution layers, which are 7x7x7.
5. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 4, characterized in that: The discriminator structure is the same as the encoder structure.
6. According to the method for interpretable Alzheimer's progression classification based on GFE-Mamba neural network in claim 4, the input of the generator is MRI (iex M ), after passing through the encoder and decoder, PET(iey P ), PET(iey P ) as the input of the discriminator, for the PET data X PET ∈R H×W×D×C , generate a The generator loss is calculated using the feature map of 7. According to the method for interpreting Alzheimer's progression classification based on GFE-Mamba neural network in claim 1, the working principle of the VIT bottleneck layer is: After passing through the encoder, MRI is extracted into a latent space, and a feature map is flattened into 2D, which is then passed to the VIT bottleneck layer. In the VIT bottleneck layer, the feature map is split into a series of image patch sequences through Patch Embedding. Where p is the patch size and N is After the 3D feature map is converted into a sequence form, it can be passed into the transformer encoder. The transformer encoder consists of 4 transformer modules. After the transformer encoder, we get Then resize back to x LP ∈R H×W×D×c Then generate PET through Decoder.
8. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 1, characterized in that: The time interval extraction module adopts a dynamic Δt strategy, which is to obtain the time interval Δt between two diagnoses of the same patient and integrate it into the GFE-Mamba model as part of the scale data. The value of the dynamic Δt adopts the average value of the training set. The method of integrating the dynamic Δt into the model is: merging the value of the dynamic Δt with the category value in the scale information and using it as the input of the scale preprocessing module.
9. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 8, characterized in that: The meter preprocessing module divides the meter information into discrete category values and continuously distributed numerical values. The method for obtaining the discrete category values is as follows: first, the category values will be converted into one-hot encoding When one-hot encoding is used, the value of the next column will increase the maximum number of categories of the values of all previous columns. The expression is as follows: After getting the new value, you can use a linear matrix to embed it: in is the embedding representation of the i-th discrete category value, is the i-th feature deviation, and is the i-th lookup table; The method for obtaining the continuous distribution value is: calculate the mean of each column in the scale information and standard deviation And calculate the standardized value, that is: Then use linear transformation to embed it: in is the embedded representation of the i-th discrete category value, and then the processed discrete category value and the tabular information of the continuously distributed numerical value are spliced together, and also spliced together with the image features: Where T is the embedding concatenation result of discrete category values and continuous values, n and m are the number of rows of discrete category values and continuous distribution values, and x LMP , x LPP They represent the characteristics of MRI and PET in the latent space in the above-mentioned generative network, respectively, and d is the embedding size.
10. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 9, characterized in that: The Mamba backbone network module includes 6 Mamba modules, which are classified as follows: The output x of the scale preprocessing module is taken as input, and after passing through 6 Mamba modules, the average value and linear layer are calculated to obtain the final classification result.
11. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 10, characterized in that: Classification method of the Mamba modules: For each Mamba module, the input is first RMS normalized by the root mean square value of the input activation, and then added to the input residual after passing through the Mamba module: x i+1 =Mamba(RMSNorm(x i ))+x i The input features will first pass through the linear layer and be split into two: x, z = split(linear(x)), where x will first pass through a 1d convolution and be passed into the Selective Scan Model after activation: y = SSM(Conv(x)), and z, as the gate vector, will be element-wise multiplied with y after activation, and then y will pass through a linear layer to obtain the final result of this module.
12. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 11, characterized in that: The implementation method of the pixel-level mutual attention module is: Get the output of the last Mamba module, marked as Before the final classification, the mutual attention with MRI and PET is added, where MRI and PET are respectively recorded as Flatten it into The specific process of mutual attention is as follows: Q y =W q y K x =W k x M V x =W v x M Where Q y It is the query matrix, W q is the weight of the query matrix, Kx 为 key matrix, W k is the weight of the key matrix, x M is the MRI feature, V x is the value matrix, W v is the weight of the value matrix, d k The dimension of the key matrix.
13. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 1, characterized in that: The training method of the GFE-Mamba neural network model is: When training the 3D GAN-Vit module, the resolution of MRI and PET was adjusted to 160, 160, 96. The 3D GAN-Vit module was trained on two NVIDIA GeForce RTX4090Ti GPUs for 200 rounds of distributed training with a batch size of 2, which took 52.5 hours. When training the multimodal mamba classifier, a pair of data is randomly selected from the table, including the table information and the patient's name and diagnosis time. The corresponding MRI is found based on the name and diagnosis time. The data is randomly cut into 160, 160, and 100 rounds of training are performed on an NVIDIA GeForce RTX4090 with a batch size of 8, which takes 22 hours; The optimization of the 3D GAN-Vit module and the multimodal mamba classifier both adopts the Adam algorithm, and the learning rate and betas are 0.9 and 0.999 respectively.
14. The interpretable Alzheimer's progression classification method based on GFE-Mamba neural network according to claim 1, characterized in that: The evaluation method of the GFE-Mamba neural network model is: The five indicators of accuracy, precision, recall, F1-score and balance are used for evaluation. The expressions are as follows: Among them, Accuracy is accuracy, the overall proportion of correct classification; Precision is precision, the proportion of true positives in positive classification; Recall is recall, the proportion of correctly identified positive instances; F1-score is F1-score, the harmonic mean of precision and recall; MCC is a balanced indicator to measure the quality of binary classification, TP and TN represent real samples in data classified as true or false, FP and FN represent false samples in data classified as true or false.
Citation Information
Patent Citations
Welding anomaly detection method fusing VIT and GAN
CN115810013A
Explanatable depressive emotion risk assessment method for social media users
CN116153512A
Neural image synthesis and disease prediction system for Alzheimer's disease
CN117495696A
A method and a system using multi-modal neuroimaging for early diagnosis of alzheimer's disease
IN201911019667A
Cited By
Double-encoder semi-supervised heart MRI double-ventricle segmentation method based on improved SAM
CN119624999A
Chemical exchange saturation transfer image anomaly detection method
CN119919371A
Spine 3D model reconstruction method based on multi-scale symmetric architecture learning
CN119919579A
Cancer image processing method and system based on Mamba structure, and medium
CN120070355A
AD early diagnosis method and system based on deep learning, medium and equipment
CN120107250A
