Alzheimer's disease classification and explainability analysis method based on cross-modal attention fusion

Abstract

The application relates to an Alzheimer's disease classification and explainability analysis method based on cross-modal attention fusion, and belongs to the technical field of medical image processing. The sMRI data and peripheral blood gene expression data are used to extract modal features through a double encoder of a Transformer and a multilayer MLP, and a TransformerMask network is constructed to realize cross-modal attention fusion. The attention weight matrix generated by the model is used for dynamically matching the brain region and gene features, the spatiotemporal topological alignment of the two modalities is realized, and the diagnostic accuracy and explainability are improved. In the classification stage, Hadamard product fusion features are adopted and input into a three-layer MLP network to classify Alzheimer's disease. The application effectively overcomes the dimension mismatch and 'black box' problem of the traditional single-modal model, and provides a high-precision, explainable multi-modal analysis method for early screening and pathological mechanism research of Alzheimer's disease.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology and relates to a method for Alzheimer's disease classification and interpretability analysis based on cross-modal attention fusion. Background Technology

[0002] The diagnosis of Alzheimer's disease is a multi-stage, multimodal comprehensive assessment process. The pathological mechanisms involve multi-level interactions of gene mutations, brain atrophy, and cognitive decline. Single-modal models struggle to quantify this cross-scale association; their average area under the curve for early Alzheimer's (AD) identification is only 0.82-0.86, with a misdiagnosis rate as high as 30%. Multimodal deep learning models, by integrating heterogeneous data from multiple sources and simulating the decision-making process of multidisciplinary consultations, have become a key path to overcome the bottlenecks in medical diagnosis. The modern medical environment generates massive amounts of heterogeneous data daily, including medical images (CT / MRI / pathological slides), electronic medical record texts, genomic data, physiological signals, and doctor-patient dialogue recordings. Studies show that top-tier hospitals generate over 50TB of data daily, 80% of which is unstructured or semi-structured. Traditional single-modal machine learning models (such as pure image analysis or gene data classifiers) can only capture local features of the disease, leading to fragmentation of crucial information.

[0003] Compared to unimodal models, multimodal models demonstrate significant performance advantages in the medical field. Studies show that for the identification and diagnosis of Alzheimer's disease and related cognitive impairments, the area under the diagnostic model curve of multimodal models is on average 6 percentage points higher than that of unimodal models. The technological development of multimodal deep learning models has gone through three stages: early expert systems relied on rule-driven approaches, the machine learning era focused on feature engineering, and currently, multimodal models based on Transformers have become the mainstream.

[0004] However, due to the "black box" nature of deep learning models, the interpretability of current multimodal fusion processes is poor (e.g., the distribution of attention weights is opaque), making it difficult for doctors to trace the basis of model decisions. When genetic data conflicts with MRI conclusions, the model cannot clearly explain why it adopts the dominant signal of a certain modality, increasing the risk of misdiagnosis. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method for Alzheimer's disease classification and interpretability analysis based on cross-modal attention fusion. This method integrates multimodal features, matches the number of gene modalities with the number of nodes in MRI brain regions through spatiotemporal topology adaptation, establishes a biological mapping relationship between brain regions and genes, solves the association break caused by dimensional mismatch in traditional methods, and dynamically adjusts the mapping relationship between gene features and brain region features through the attention weight matrix generated by Transformer, thereby improving the accuracy and interpretability of the Alzheimer's disease diagnostic model.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for Alzheimer's disease classification and interpretability analysis based on cross-modal attention fusion, the method comprising: T1-weighted magnetization was used to prepare sMRI data and peripheral blood gene expression data for fast gradient echo imaging; Preprocessing was performed on sMRI data and genetic data separately; A multimodal model was constructed to process sMRI and genetic data to complete the classification task.

[0007] Furthermore, the sMRI data preprocessing included using FreeSurfer software to preprocess the sMRI data to obtain seven statistical indicators at the brain region level, including cortical thickness, gray matter volume, cortical surface area, intrinsic curvature, mean curvature, folding index, and Gaussian curvature.

[0008] Furthermore, the preprocessing of the gene data includes background noise correction, standardization, and batch noise correction using robust multiarray averaging and alternative variable analysis methods, followed by selection of gene data features based on prior knowledge.

[0009] Furthermore, the multimodal model includes an encoder network, a TransformerMask network, and a classification layer; the encoder network is used to extract features from sMRI data and genetic data; the TransformerMask network generates an attention weight matrix in the brain region dimension based on sMRI features and genetic features; the classification layer is used to perform a classification task and output class labels to obtain classification results.

[0010] The encoder network includes a Transformer encoder and an MLP encoder; the Transformer encoder is used to extract sMRI features, and the MLP encoder is used to extract gene features. The TransformerMask network includes a position encoder and a Transformer encoder-decoder structure. The position encoder is used to generate positional encoding information based on genetic features. The Transformer encoder-decoder structure receives positional encoding information, genetic features, and sMRI features, generates content encoding information, and then fuses the positional encoding information and content encoding information to output an attention weight matrix in the brain region dimension. The classification layer uses an MLP network. First, the attention weight matrix of the brain region dimension is integrated with the sMRI features using the Hadaman product to obtain weighted fusion features. Then, the weighted fusion features are input into the MLP network for classification and output class labels.

[0011] Furthermore, the MLP network consists of three layers, each of which includes a linear layer, a regularization layer, a ReLU layer, and a Dropout layer connected in sequence.

[0012] Furthermore, the method also includes training the multimodal model before applying it, setting a loss function during training, and optimizing the multimodal model by minimizing the loss function; the loss function is expressed as:

[0013] In the formula, Let be the hyperparameter of the cross-entropy loss in the MLP network. For the hyperparameters of the reconstruction loss, For the hyperparameters of the sparsity constraint; For the cross-entropy loss of the MLP network, The reconstruction loss is used in the process of learning the target sequence by projecting gene features. This is a sparsity constraint used to ensure that the generated sMRI-like modalities possess the sparsity characteristics of that modality. Modal consistency constraints are used to reduce intermodal differences.

[0014] The beneficial effects of this invention are as follows: (1) This invention integrates peripheral blood gene expression data and sMRI statistical data (morphological features) and applies them to a cross-modal deep model, which improves the performance of the model and provides a new prospect for the early classification of Alzheimer's disease and related mild cognitive impairment diseases.

[0015] (2) The TransformerMask network was used to align the spatial dimensions of peripheral blood gene expression data and sMRI data, which have different data structures. This provides an interpretable visualization interface for this cross-modal fusion field. By visualizing the distribution of attention weights, the association between genetic factors and imaging features can be further identified.

[0016] (3) The gradient weighted mapping method was used to calculate the classification contribution rate of individual genes and individual brain regions in gene modality and sMRI modality. From the perspective of the ex post-interpretability of deep learning models, some key genes and brain regions of Alzheimer's disease were revealed, providing a feasible channel for screening genes based on deep learning image data and providing new ideas for multimodal research on Alzheimer's disease.

[0017] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a framework diagram of the multimodal model; Figure 2 A flowchart illustrating the classification and interpretability analysis methodology for Alzheimer's disease; Figure 3 A visualization of the attention weight matrix generated for the TransformerMask network (CN-AD classification task); Figure 4 A visualization of the brain regions with high classification contribution rates after completing the classification task. Detailed Implementation

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0021] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0022] Please see Figure 1 and Figure 2 An embodiment of the present invention provides a method for Alzheimer's disease classification and interpretability analysis based on cross-modal attention fusion, the method comprising: 1. T1-weighted magnetized sMRI data and peripheral blood gene expression data prepared for fast gradient echo imaging were selected from the ADNI database, including 75 Alzheimer's disease patients, 114 patients with progressive mild cognitive impairment (pMCI), 177 patients with stable mild cognitive impairment (sMCI), and 145 cognitively normal (CN) subjects, as shown in Table 1: Table 1

[0023] The mild cognitive impairment group was further divided into stable and progressive types based on whether the MCI progressed to dementia during the 30-month follow-up period.

[0024] 2. Preprocess the sMRI data and gene data.

[0025] (1) sMRI data were preprocessed using FreeSurfer software.

[0026] FreeSurfer first performs motion correction and averaging to generate a single image with a higher signal-to-noise ratio. Then, it normalizes the image intensity to reduce the impact of differences in scanner and scanning parameters, making subsequent segmentation more stable. FreeSurfer uses its unique hybrid method (combining atlas-based registration, morphological operations, and edge detection) to perform skull dissection, generating an image containing only brain tissue.

[0027] Next, using intensity information based on a Gaussian mixture model, the brain tissue images were segmented into gray matter, white matter, and cerebrospinal fluid. After generating the gray matter surface and registering it to a standard space, FreeSurfer, based on the individual subject's geometric features (sulcus and gyri pattern) and the information registered to the template, applied a predefined DK-308 atlas (a smaller-resolution segmentation of the Desikan-Killiany atlas using a backtracking algorithm, comprising 308 brain regions) to label each vertex on the cortical surface as belonging to a specific anatomical region.

[0028] Finally, for each vertex, the thickness, volume and other information of the cortex and subcortical structures are accurately segmented and quantified. Seven important statistical indicators are extracted from the processing results: cortical thickness, gray matter volume, cortical surface area, intrinsic curvature, mean curvature, folding index and Gaussian curvature.

[0029] (2) Peripheral blood gene expression data were processed by robust multiarray averaging and alternative variable analysis methods for background noise correction, standardization and batch (site) noise correction.

[0030] To reduce dimensionality and remove irrelevant noise, probes with low expression or low variance were filtered. Since the data was at the probe level, probe expression values ​​needed to be aggregated to the gene level. Common methods include selecting the median or highest value; in this embodiment, probes with the median expression value were used. Subsequently, the current peripheral blood gene expression data were filtered based on lists of genes significantly associated with AD or human brain developmental degeneration provided by the AlzGene database and the Allen Human Brain Genome Atlas database.

[0031] 3. Construct a multimodal model to process MRI and genetic data. The model structure is as follows: Figure 1 As shown.

[0032] (1) Use two encoder networks, Transformer and MLP, to extract sMRI features and gene features respectively.

[0033] Gene data input is used to reduce the dimensionality of MLP with a three-layer linear mapping to obtain 308-dimensional latent spatial features. sMRI statistical indicators input Then, the Transformer encoder is used to process the multi-index sequence (308) 7) The output is mapped to a 308-dimensional MRI feature vector. To maintain consistency in the output dimensions so that subsequent fusion can be carried out while providing a certain degree of interpretability.

[0034]

[0035]

[0036] (2) Extracted sMRI features With genetic characteristics Input into the TransformerMask network.

[0037] The TransformerMask network consists of a location encoding module for generating location-encoded information and a Transformer encoder-decoder structure with two layers and six attention heads. The Transformer encoder-decoder structure receives location-encoded information, genetic features, and sMRI features to generate content-encoded information; the Transformer decoder fuses the location-encoded information and content-encoded information to output an attention weight matrix in the brain region dimension.

[0038] In the TransformerMask network, genetic features and sMRI features are projected onto the channel dimension respectively:

[0039]

[0040] The projection S of sMRI features is used to calculate a 308-dimensional dynamic location code, incorporating region information. Then, a learnable content embedding matrix C is set to obtain the target sequence tgt, and the gene features are fused with the dynamic position encoding results as the decoding input.

[0041]

[0042]

[0043] in, The learnable parameter matrix in the linear transformation layer. It is a bias vector.

[0044] The Transformer decoding process is as follows:

[0045]

[0046] Where tgt is the target sequence of the Transformer decoder. This is an attention weight matrix based on brain region dimensions. L This refers to the number of layers in the Transformer decoder. This is the learnable parameter matrix of the decoder. This is the bias vector. In this TransformerMask network, a Transformer encoder-decoder network with 2 layers and 6 attention heads is used to extract, align, and fuse positional and genetic information.

[0047] The attention weight matrix at the brain region level can be visualized to observe the specific attention areas of the TransformerMask network. Because the output dimension of the weight matrix is ​​set to 308 in the Transformer decoder, cross-modal spatial alignment is achieved between the two modalities. Therefore, the attentional brain regions can be intuitively observed on the brain template, such as... Figure 3 This study visualizes brain regions representing the average attention weight matrix of samples from the CN-AD classification task. Notable findings include significant involvement of the posterior cingulate cortex (PCC) and the adjacent precuneus. The PCC / precuneus is one of the earliest cortical regions in Alzheimer's disease to deposit β-amyloid (Aβ) and abnormal Tau protein. Since sMRI structural information is generally not readily apparent in its direct reflection of amyloid deposition, valuable information is inferred from gene expression data directly related to these proteins. This region is a core hub of the default mode network, and its abnormalities directly lead to typical AD symptoms—episodic memory impairment and disorientation. Significant differences in this region are a radiographic marker distinguishing AD from other types of dementia (such as frontotemporal dementia). Furthermore, hippocampal and medial temporal lobe involvement is also observed, with significant involvement in the deeper hippocampus and entorhinal cortex. The hippocampus is a key structure for forming new memories; its atrophy and Tau protein pathology are evidence of early structural types of AD, directly leading to short-term memory loss. Tau protein tangles typically spread from the entorhinal cortex to the hippocampus and PCC, confirming the pathological progression of AD.

[0048] (3) Hadaman product is used in combination with attention weight matrix and sMRI features to introduce modality fusion information, which is then input into MLP network for classification task and output class label.

[0049] The attention weight matrix in the brain region dimension output by the Transformer decoder. As an attention mask, the Hadaman product is used in conjunction with sMRI features. The fusion yields weighted features Used for classifying tasks:

[0050] The weighted feature F is input into a three-layer MLP network. Each layer consists of a linear sequence followed by regularization, ReLU, and Dropout, with dimensions increasing sequentially from 308 to 128 to 64, and finally, the number of categories is increased. The final output is the classification result. The class labels include Alzheimer's disease patients (AD), stable mild cognitive impairment patients (sMCI), developmental mild cognitive impairment patients (pMCI), and cognitively normal subjects (CN).

[0051] Since the TransformerMask network contains multiple modules and its loss consists of multiple loss terms, the optimization objective is to reduce the total loss composed of these multiple loss terms, with the participation of the bias term, thereby optimizing the model parameters. Its structure is as follows: For an MLP network used for classification, its loss consists of cross-entropy and Focal Loss:

[0052]

[0053] in, For the true one-hot label of sample i, This represents the predicted probability of the multimodal model for category c. For the true class index of sample i, Focusing parameters (default) =2).

[0054] The process of learning the target sequence by projecting gene features can be viewed as a process of reconstructing sMRI modalities from gene modality data based on the location information encoded by sMRI modality. During this process, gene modality information is fused under the constraint of the location information encoded by sMRI modality. The reconstruction loss is expressed as:

[0055] in, For the target sMRI modality. Effective sample index set. for:

[0056] To ensure that the generated sMRI-like modalities possess the sparsity characteristics of that modality, the following sparsity constraints are set:

[0057] in, The set of spatial attention vector elements is: To integrate weights, For fusion features j , This represents the total number of elements.

[0058] Due to the differences in data structure between sMRI and gene expression modalities, constraints need to be set to minimize the instability caused by these differences. The modal consistency constraints are as follows:

[0059] In summary, the total loss calculation structure, consisting of multiple loss terms, is as follows:

[0060] in, Let be the hyperparameter of the cross-entropy loss in the MLP network. For the hyperparameters of the reconstruction loss, is the hyperparameter for sparsity constraints.

[0061] The classification performance of this model and the comparison model is evaluated using the Area Under the Curve (AUC), a widely used metric in medical diagnostic models. This involves randomly selecting a positive sample and a negative sample, and determining whether the model predicts the positive sample as positive more likely than predicting the negative sample as positive. The calculation formula is as follows:

[0062] in, The score for the i-th positive sample being predicted as positive by the model. Let be the score at which the j-th negative sample is predicted as positive by the model, and m and n be the number of positive and negative samples, respectively. I(x) represents the indicator function, which is 1 when condition x is true and 0 when condition x is false.

[0063] The method proposed in this invention is compared with several other methods that utilize multimodal or unimodal data from the ADNI database to perform the same identification and diagnosis task. Table 2 lists the area under the curve (AUC) performance of different classification methods under two identification and diagnosis tasks. The results show that the method proposed in this invention is superior to the other methods.

[0064] Table 2 Comparison of Area Under the Classification Curve for Each Model

[0065] The following examples of ex post-hoc interpretability analysis results are all derived from the five-fold cross-validation results of the CN-AD classification task.

[0066] This includes calculating the cumulative gradient of each gene involved in the classification using a gradient-weighted mapping method, thereby calculating the classification contribution rate of each gene. The top 20 genes with the highest ranking are shown in Table 3. For example, YTHDC1 is a "reader" of RNA m^6A, involved in regulating m^6A-modified RNA splicing and stabilization. In AD patients, the m^6A methyl group in the frontal cortex undergoes reprogramming, affecting neuronal survival and excitotoxicity tolerance. CNNM3 is a magnesium (Mg^2+) transmembrane transport mediator. It has been confirmed in rat AD models that increasing Mg^2+ levels in the brain can reverse synaptic loss and cognitive deficits. CNNM3 may protect synaptic function and delay the pathological progression of AD by maintaining neuronal Mg^2+ homeostasis. CEBPA is a member of the C / EBP transcription factor family, regulating inflammatory factors and glial cell activation. Microglial-mediated neuroinflammation is a core component of AD pathology. CEBPA affects the intensity and course of neuroinflammation by regulating the expression of pro-inflammatory genes. PSMB6 has been listed as a gene associated with Alzheimer's disease in GeneCards. Its dysregulation can lead to decreased proteasome activity and promote the accumulation of misfolded proteins such as Aβ and Tau.

[0067] Table 3. Top 20 Gene Classification Contribution Rates

[0068] In addition, it also includes calculating the cumulative gradient of each brain region under each index involved in classification using a gradient-weighted mapping method, thereby calculating the classification contribution rate, such as... Figure 4 Table 4 shows the top ten brain regions in terms of classification contribution rate for each hemisphere under the gray matter volume index. High-contribution areas are concentrated in the default mode network (e.g., precuneus, posterior cingulate cortex, inferior parietal lobe), higher visual processing areas (e.g., fusiform gyrus, lingual gyrus, lateral occipital lobe, cuneate gyrus), and language and executive networks (e.g., superior temporal lobe, middle temporal lobe, prefrontal lobe, supramarginal gyrus). These regions all exhibit varying degrees of gray matter atrophy and Aβ / Tau deposition in Alzheimer's disease pathology and are highly correlated with clinical manifestations such as cognitive memory, language, and neuropsychiatric disorders. Furthermore, gradient-weighted mapping also reflects the concept of "desymmetry" in brain region degeneration in patients. For example, the left hemisphere emphasizes language / memory (superior temporal lobe / supramarginal gyrus) and somatosensory functions; while the right hemisphere focuses on visuospatial (e.g., fusiform gyrus, lingual gyrus, lateral occipital lobe) and compensatory executive networks, suggesting that future multimodal interventions for Alzheimer's disease should be individually designed based on hemispheric lateralization characteristics.

[0069] Table 4. Top ten brain regions in terms of the classification contribution rate of gray matter volume index

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for classifying and interpretability analysis of Alzheimer's disease based on cross-modal attention fusion, characterized in that, This method includes: selecting T1-weighted magnetization to prepare sMRI data for fast gradient echo imaging and peripheral blood gene expression data; Preprocessing was performed on sMRI data and genetic data separately; A multimodal model was constructed to process sMRI and genetic data to complete the classification task.

2. The method according to claim 1, characterized in that, Preprocessing of sMRI data involved using FreeSurfer software to obtain seven statistical indicators at the brain region level, including cortical thickness, gray matter volume, cortical surface area, intrinsic curvature, mean curvature, folding index, and Gaussian curvature.

3. The method according to claim 1, characterized in that, Preprocessing of the gene data includes background noise correction, standardization, and batch noise correction using robust multiarray averaging and alternative variable analysis methods, followed by selection of gene data features based on prior knowledge.

4. The method according to claim 1, characterized in that, The multimodal model includes an encoder network, a TransformerMask network, and a classification layer; the encoder network is used to extract features from sMRI data and genetic data. The TransformerMask network generates an attention weight matrix based on sMRI and genetic features to represent brain regions; the classification layer performs a classification task and outputs class labels to obtain the classification result.

5. The method according to claim 4, characterized in that, The encoder network includes a Transformer encoder and an MLP encoder; the Transformer encoder is used to extract sMRI features, and the MLP encoder is used to extract gene features. The TransformerMask network includes a position encoder and a Transformer encoder-decoder structure. The position encoder is used to generate positional encoding information based on genetic features. The Transformer encoder-decoder structure receives positional encoding information, genetic features, and sMRI features, generates content encoding information, and then fuses the positional encoding information and content encoding information to output an attention weight matrix in the brain region dimension. The classification layer uses an MLP network. First, the attention weight matrix of the brain region dimension is integrated with the sMRI features using the Hadaman product to obtain weighted fusion features. Then, the weighted fusion features are input into the MLP network for classification and output class labels.

6. The method according to claim 5, characterized in that, The MLP network consists of three layers, each of which includes a linear layer, a regularization layer, a ReLU layer, and a Dropout layer connected in sequence.

7. The method according to claim 5, characterized in that, The method also includes training the multimodal model before applying it, setting a loss function during training, and optimizing the multimodal model by minimizing the loss function; the loss function is expressed as: In the formula, Let be the hyperparameter of the cross-entropy loss in the MLP network. For the hyperparameters of the reconstruction loss, For the hyperparameters of the sparsity constraint; For the cross-entropy loss of the MLP network, The reconstruction loss is used in the process of learning the target sequence by projecting gene features. This is a sparsity constraint used to ensure that the generated sMRI-like modalities possess the sparsity characteristics of that modality. Modal consistency constraints are used to reduce intermodal differences.

Images