Alzheimer disease prediction method based on asymmetric cross-modal cross attention
By constructing an asymmetric cross-modal cross-attention prediction network, the complex interdependencies between imaging, genetic, and clinical data are effectively captured, solving the problem of insufficient fusion of multi-omics data and achieving high-precision Alzheimer's disease prediction.
Patent Information
- Application Number
- CN202610077764.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-21
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies struggle to effectively capture complementary information between multi-omics and multi-modal data, resulting in suboptimal Alzheimer's disease prediction performance. Traditional methods suffer from the loss of key discriminative features and failure to capture cross-modal feature dependencies when dealing with single data domains or simple data concatenation.
We employ an Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention. By constructing multimodal medical data and combining shallow feature extraction, multimodal attention fusion, deep feature extraction and MLP classification modules, we build an asymmetric cross-modal cross-attention prediction network to achieve deep fusion of images and structured data.
It significantly enhanced the interaction and synergy at the cross-omics feature level, improved the accuracy of Alzheimer's disease prediction results, and achieved a classification accuracy of 94.88%, with all evaluation indicators leading traditional methods.
Smart Images
Figure CN122067747A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer science and technology, and in particular to a method for predicting Alzheimer's disease based on asymmetric cross-modal cross-attention. Background Technology
[0002] Alzheimer's disease (AD) is an irreversible neurodegenerative disease characterized by progressive cognitive decline. In current research using deep learning techniques to predict AD, traditional methods such as convolutional neural networks and simple feature concatenation often have significant limitations: they struggle to fully mine and utilize the inherent complementary information within multi-omics and multimodal data. This leads to suboptimal fusion results from data from different sources and may result in the loss of key discriminative features. Therefore, related technologies suffer from the following problems: 1) Traditional machine learning models that utilize numerical features of scales (such as SVM and RF) have limited performance when applied to a single data domain; 2) Traditional deep learning models that utilize image features (such as CNN and RNN) for MRI or PET image recognition can extract image structural features but lack feature understanding of scale numerical types; 3) The multimodal simple fusion model predicts AD by simply stitching together MRI images and scale numerical features, and cannot capture the dependencies between cross-modal features. Summary of the Invention
[0003] To address the aforementioned technical problems, the purpose of this invention is to provide a method for predicting Alzheimer's disease based on asymmetric cross-modal cross-attention, which can more effectively capture the interdependencies between different modalities, thereby improving the accuracy of Alzheimer's disease prediction results.
[0004] The first technical solution adopted in this invention is: an Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention, comprising the following steps: Multimodal medical data was constructed by combining PET imaging data, MRI imaging data, clinical scale data, and genetic data, and the data was preprocessed to obtain preprocessed multimodal medical data. An asymmetric cross-modal cross-attention prediction network is constructed by combining a shallow feature extraction module, a multimodal attention fusion module, a deep feature extraction module, and an MLP classification module. Based on an asymmetric cross-modal cross-attention prediction network, Alzheimer's disease prediction is performed on preprocessed multimodal medical data, and the prediction results are obtained.
[0005] Furthermore, the asymmetric cross-modal cross-attention prediction network specifically includes a shallow feature extraction module, a multimodal attention fusion module, a deep feature extraction module, and an MLP classification module. The shallow feature extraction module, the multimodal attention fusion module, the deep feature extraction module, and the MLP classification module are connected sequentially, wherein: The shallow feature extraction module includes a pre-trained ResNet50 model and a deep neural network model; The deep feature extraction module includes a self-attention module and a Fourier module; The MLP classification module includes a fully connected layer and a ReLU activation function.
[0006] Furthermore, the step of using an asymmetric cross-modal cross-attention prediction network to predict Alzheimer's disease from preprocessed multimodal medical data and obtain the Alzheimer's disease prediction results specifically includes: The preprocessed multimodal medical data is input into an asymmetric cross-modal cross-attention prediction network; A shallow feature extraction module based on an asymmetric cross-modal cross-attention prediction network is used to perform shallow feature extraction on preprocessed multimodal medical data to obtain multimodal medical shallow feature data. A multimodal attention fusion module based on an asymmetric cross-modal cross-attention prediction network is used to perform feature fusion processing on shallow feature data of multimodal medicine to obtain multimodal medical fused feature data. A deep feature extraction module based on an asymmetric cross-modal cross-attention prediction network is used to perform deep feature extraction processing on multimodal medical fusion feature data to obtain multimodal medical deep feature data. The MLP classification module based on the asymmetric cross-modal cross-attention prediction network is used to predict and classify multimodal medical deep feature data to obtain Alzheimer's disease prediction results.
[0007] Furthermore, the shallow feature extraction module based on the asymmetric cross-modal cross-attention prediction network performs shallow feature extraction processing on the preprocessed multimodal medical data to obtain multimodal medical shallow feature data. This step specifically includes: The preprocessed multimodal medical data is input into the shallow feature extraction module of the asymmetric cross-modal cross-attention prediction network; A pre-trained ResNet50 model based on a shallow feature extraction module is used to perform shallow feature extraction on PET and MRI image data respectively, resulting in shallow feature data of PET and MRI images. Based on the deep neural network model of the shallow feature extraction module, shallow feature extraction is performed on clinical scale data and gene data respectively to obtain shallow feature data of clinical scale and shallow feature data of genes. By combining superficial feature data from PET images, superficial feature data from MRI images, superficial feature data from clinical scales, and superficial feature data from genes, multimodal medical superficial feature data is obtained.
[0008] Furthermore, the multimodal attention fusion module based on the asymmetric cross-modal cross-attention prediction network performs feature fusion processing on the multimodal medical shallow feature data to obtain multimodal medical fused feature data. This step specifically includes: Multimodal medical shallow feature data is input into the multimodal attention fusion module of an asymmetric cross-modal cross-attention prediction network; The superficial feature data of clinical scales and superficial feature data of genes are linearly transformed into query vectors, and the superficial feature data of MRI images and superficial feature data of PET images are mapped into key vectors and value vectors, respectively. Calculate the dot product between the query vector and the key vector, divide by the square root of the vector dimension, and convert it into a probability vector using the Softmax function to obtain the fused features of the MRI image shallow feature data and the clinical scale shallow feature data. The fusion features and value vectors of MRI image features and clinical data features are calculated using the Softmax function to obtain the fusion features of PET image shallow feature data and gene shallow feature data. Multimodal medical fusion feature data is obtained by splicing together the fusion features of MRI image superficial feature data and clinical scale superficial feature data, the fusion features of clinical scale superficial feature data, PET image superficial feature data and gene superficial feature data, and gene superficial feature data.
[0009] Furthermore, the calculation process for the fusion features of the MRI image superficial feature data and the clinical scale superficial feature data is as follows: ; In the above formula, This represents the fusion characteristics of superficial feature data from MRI images and superficial feature data from clinical scales. This represents the query vector obtained by linearly transforming the shallow feature data of the clinical scale. This represents the key vector obtained by mapping shallow feature data from MRI images. This represents the value vector obtained by mapping shallow feature data from MRI images. This represents the Softmax function. This represents a cross-modal attention mechanism. This represents the dimension of the key vector.
[0010] Furthermore, the calculation process for the fusion features of the PET image shallow feature data and the gene shallow feature data is as follows: ; In the above formula, This represents the fusion characteristics of PET image shallow feature data and genetic shallow feature data. This represents the query vector obtained by linearly transforming shallow gene feature data. This represents the key vector obtained by mapping shallow feature data from PET images. This represents the value vector obtained by mapping shallow feature data from PET images. This represents the Softmax function. This represents a cross-modal attention mechanism. This represents the dimension of the key vector.
[0011] Furthermore, the deep feature extraction module based on the asymmetric cross-modal cross-attention prediction network performs deep feature extraction processing on the multimodal medical fusion feature data to obtain multimodal medical deep feature data. This step specifically includes: Multimodal medical fusion feature data is input into the deep feature extraction module of an asymmetric cross-modal cross-attention prediction network; The self-attention module based on the deep feature extraction module performs Fourier transform processing on the multimodal medical fusion feature data to obtain the Fourier transformed multimodal medical fusion feature data. The Fourier module based on the deep feature extraction module performs global relation capture processing on the multimodal medical fusion feature data after Fourier transform to obtain multimodal medical deep feature data.
[0012] Furthermore, the step of using the MLP classification module based on the asymmetric cross-modal cross-attention prediction network to predict and classify multimodal medical deep feature data to obtain Alzheimer's disease prediction results specifically includes: Multimodal medical deep feature data is input into the MLP classification module of an asymmetric cross-modal cross-attention prediction network; A fully connected layer based on the MLP classification module is used to perform linear transformation on the multimodal medical deep feature data to obtain the transformed multimodal medical deep feature data. Based on the ReLU activation function of the MLP classification module, the transformed multimodal medical deep feature data is classified and predicted to obtain Alzheimer's disease prediction results.
[0013] The beneficial effects of the method of this invention are as follows: This invention constructs multimodal medical data by combining PET image data, MRI image data, clinical scale data, and gene data, and performs data preprocessing to obtain preprocessed multimodal medical data; then, it constructs an asymmetric cross-modal cross-attention prediction network by combining a shallow feature extraction module, a multimodal attention fusion module, a deep feature extraction module, and an MLP classification module. The asymmetric cross-modal cross-attention mechanism can effectively achieve deep fusion of images and structured data; finally, based on the asymmetric cross-modal cross-attention prediction network, it performs Alzheimer's disease prediction on the preprocessed multimodal medical data to obtain Alzheimer's disease prediction results. This can more effectively capture the complex interdependencies between different modalities (such as between images and genes, clinical data), thereby significantly enhancing the interaction and synergy at the cross-omics feature level and improving the accuracy of Alzheimer's disease prediction results. Attached Figure Description
[0014] Figure 1 This is a flowchart of the steps of the Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention in this invention; Figure 2 This is a schematic diagram of an asymmetric cross-modal cross-attention prediction network provided in a specific embodiment of the present invention; Figure 3 This is a schematic diagram of the multimodal attention fusion module provided in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of the GRAD-CAM comparison visualization results of three diagnostic categories generated based on the ACMCA model, provided in a specific embodiment of the present invention. Detailed Implementation
[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
[0016] First, it should be noted that this invention proposes a novel network architecture called Asymmetric Cross-Modal Cross-Attention (ACMCCA), designed to achieve multi-omics data integration and accurate prediction for Alzheimer's disease (AD). This mechanism can more effectively capture the complex interdependencies between different modalities (such as between imaging and genetic / clinical data), thereby significantly enhancing the interaction and synergy at the cross-omics feature level. The effectiveness of the model was verified through systematic experiments. The results show that compared with traditional single-modal baseline methods and conventional multi-modal fusion models, the ACMCA model exhibits significant performance advantages in AD classification tasks. On independent test datasets, its classification accuracy reached 94.88%, and all evaluation metrics (such as recall and F1 score) were leading.
[0017] Based on this, refer to Figure 1 This invention provides a method for predicting Alzheimer's disease based on asymmetric cross-modal cross-attention, which includes the following steps: S100. Multimodal medical data is constructed by combining PET image data, MRI image data, clinical scale data and genetic data, and data preprocessing is performed to obtain preprocessed multimodal medical data. Specifically, it combines four different types of medical multi-perspective data: PET images, MRI images, clinical scale data, and genetic data.
[0018] S200, combining shallow feature extraction module, multimodal attention fusion module, deep feature extraction module and MLP classification module, constructs an asymmetric cross-modal cross-attention prediction network; Specifically, such as Figure 2 As shown, the asymmetric cross-modal cross-attention prediction network specifically includes a shallow feature extraction module, a multimodal attention fusion module, a deep feature extraction module, and an MLP classification module. The shallow feature extraction module, the multimodal attention fusion module, the deep feature extraction module, and the MLP classification module are connected sequentially. The shallow feature extraction module includes a pre-trained ResNet50 model and a deep neural network model; the deep feature extraction module includes a self-attention module and a Fourier module; and the MLP classification module includes a fully connected layer and a ReLU activation function.
[0019] S300, based on an asymmetric cross-modal cross-attention prediction network, performs Alzheimer's disease prediction on preprocessed multimodal medical data, and obtains Alzheimer's disease prediction results.
[0020] S310. Input the preprocessed multimodal medical data into the asymmetric cross-modal cross-attention prediction network; S320, a shallow feature extraction module based on an asymmetric cross-modal cross-attention prediction network, performs shallow feature extraction on the preprocessed multimodal medical data to obtain multimodal medical shallow feature data; Specifically, the preprocessed multimodal medical data is input into the shallow feature extraction module of the asymmetric cross-modal cross-attention prediction network; based on the pre-trained ResNet50 model of the shallow feature extraction module, shallow feature extraction is performed on PET image data and MRI image data respectively to obtain shallow feature data of PET image data and shallow feature data of MRI image data; based on the deep neural network model of the shallow feature extraction module, shallow feature extraction is performed on clinical scale data and gene data respectively to obtain shallow feature data of clinical scale data and shallow feature data of genes; combining the shallow feature data of PET image data, shallow feature data of MRI image data, shallow feature data of clinical scale data and shallow feature data of genes, multimodal medical shallow feature data is obtained.
[0021] In this embodiment, a pre-trained ResNet50 model is used in the shallow feature extraction module to extract features from image modalities to obtain rich semantic information. Since MRI and PET images are both medical imaging data, ResNet50 can effectively extract high-level image features from them. For gene data, the three genotypes (risk-free allele, single-risk allele, and double-risk allele) are converted into an input format that can be processed by a deep neural network (DNN), represented as encoded vectors, and dimensionality reduction is performed simultaneously. For clinical scale data (containing 7 detection indicators), DNN is used to extract features from the encoded clinical scale data, and dimensionality increase is performed simultaneously. This series of processes ensures that the feature dimensions of clinical scale data, gene data, MRI images, and PET images remain consistent. This unified feature dimension ensures that these features have similar representations in the feature space, laying the foundation for the subsequent feature fusion module.
[0022] S330, a multimodal attention fusion module based on an asymmetric cross-modal cross-attention prediction network, performs feature fusion processing on multimodal medical shallow feature data to obtain multimodal medical fused feature data; Specifically, multimodal medical shallow feature data is input into the multimodal attention fusion module of an asymmetric cross-modal cross-attention prediction network; clinical scale shallow feature data and genetic shallow feature data are linearly transformed into query vectors, and MRI image shallow feature data and PET image shallow feature data are mapped into key vectors and value vectors, respectively; the dot product between the query vector and the key vector is calculated, divided by the square root of the vector dimension, and converted into a probability vector using the Softmax function to obtain the fusion features of MRI image shallow feature data and clinical scale shallow feature data; the fusion features of MRI image features and clinical data features are calculated with the value vector using the Softmax function to obtain the fusion features of PET image shallow feature data and genetic shallow feature data; the fusion features of MRI image shallow feature data and clinical scale shallow feature data, the fusion features of clinical scale shallow feature data, PET image shallow feature data and genetic shallow feature data are concatenated with the genetic shallow feature data to obtain multimodal medical fusion feature data.
[0023] In this embodiment, the fusion of clinical scale data, genetic data, MRI images, and PET images during the multimodal fusion process of the ACMCA model is referred to as the "image-numerical fusion branch." This fusion branch operates based on specific principles and formulas. First, the location-encoded clinical data features and genetic data features are processed into query vectors, the specific expressions of which are shown below: ; ; MRI and PET features are mapped to corresponding key vectors and value vectors, respectively, as shown in the following expressions: ; ; Next, for each query vector, its dot product with all key vectors is calculated, divided by the square root of the vector dimension, and then the result is converted into a probability vector using the Softmax function. This operation yields the correlation matrix between features at different locations. Finally, this correlation matrix and the value vectors are used to calculate the image representation processed by the attention mechanism, as shown in the following expression: ; ; In the above formula, This represents a cross-modal attention mechanism used to fuse information from genetic data and PET imaging data. This represents the dimension of the key vector, used to adjust the scale of the dot product calculation to ensure numerical stability.
[0024] In the multimodal fusion module of this embodiment, the features output by the clinical data feature extraction module are denoted as... This includes information such as the subjects' clinical data, neuropsychological test results, and demographic data. Furthermore, the features extracted from the gene data in this module are denoted as... , Features extracted from MRI images This indicates the features acquired from PET images. , , and The feature dimensions are all set to 100, i.e., they are all 100-dimensional vectors, in order to facilitate subsequent feature fusion.
[0025] During the ACMCA model training process The transformation matrix, which can be iteratively optimized, plays a crucial role during the model training phase. The matrix values are continuously adjusted to achieve linear transformations of different modal features. Within the attention mechanism framework, and Through and right and The query vector obtained by linear transformation is used to learn relevant feature information from MRI and PET image features, and to help the model obtain key cross-modal data information.
[0026] MRI imaging features and PET image features Converted into key vectors using transformation matrices respectively , Sum value vector , The key vector is used to calculate the similarity between query vectors to determine the degree of association between features at different locations; the value vector is then weighted and summed based on the similarity calculation results to ultimately generate multimodal fusion features. These multimodal fusion features contain association information between different modalities.
[0027] in, These features are obtained by fusing MRI imaging characteristics and clinical data characteristics. These are features obtained by fusing PET image features and genetic data features. After a series of complex and ordered calculations, the output of this module is represented as: ; This fusion method based on asymmetric cross-modal cross-attention guides the model to focus on learning the alignment relationship between medical images and numerical data, thereby efficiently understanding the intrinsic connection between image data blocks and clinical data. The hybrid features generated by this fusion method are defined as multimodal features, with a tensor form of (batch size, number of channels, feature dimension), specifically (32, 3, 100). For details... Figure 3 As shown.
[0028] S340. A deep feature extraction module based on an asymmetric cross-modal cross-attention prediction network is used to perform deep feature extraction processing on multimodal medical fusion feature data to obtain multimodal medical deep feature data. Specifically, the multimodal medical fusion feature data is input into the deep feature extraction module of the asymmetric cross-modal cross-attention prediction network; the self-attention module based on the deep feature extraction module performs Fourier transform processing on the multimodal medical fusion feature data to obtain the Fourier transformed multimodal medical fusion feature data; the Fourier module based on the deep feature extraction module performs global relation capture processing on the Fourier transformed multimodal medical fusion feature data to obtain the multimodal medical deep feature data.
[0029] In this embodiment, the deep feature extraction module is responsible for deep extraction of the fused features output by the multimodal attention module. Its function is to mine effective correlation information within the multimodal features, thereby achieving better interaction between features at different locations. This module consists of a self-attention module and a Fourier module. The Fourier module adopts an Fnet structure, applying Discrete Fourier Transform (DFT) in the Fourier layers to transmit the internal information of the multimodal features. For a sequence of length N, the formula for the Fast Fourier Transform is: ; For each feature location k, the Fnet module applies a Fourier transform along the feature dimension to integrate information from other locations, capturing global relationships in the sequence. The frequency domain representation is then passed through a fully connected feedforward network, with residual connections and layer normalization applied after both the Fourier and feedforward layers.
[0030] S350, an MLP classification module based on an asymmetric cross-modal cross-attention prediction network, performs prediction and classification on multimodal medical deep feature data to obtain Alzheimer's disease prediction results.
[0031] Specifically, multimodal medical deep feature data is input into the MLP classification module of an asymmetric cross-modal cross-attention prediction network; based on the fully connected layer of the MLP classification module, the multimodal medical deep feature data is linearly transformed to obtain the transformed multimodal medical deep feature data; based on the ReLU activation function of the MLP classification module, the transformed multimodal medical deep feature data is classified and predicted to obtain the Alzheimer's disease prediction results.
[0032] In this embodiment, deep features are input into a four-layer MLP classification module. The fully connected layer first performs a linear transformation, then passes through a ReLU activation function to generate three outputs: CN (Cognitive Normal), MCI (Mild Cognitive Impairment), and AD (Alzheimer's Disease). The model is trained using cross-entropy loss. ; in It is the sample size. Corresponding to three diagnostic categories, Indicates the true label, This represents the predicted probability. Minimizing this loss optimizes the model parameters and improves classification performance.
[0033] Finally, experimental analysis was conducted on the embodiments of the present invention. To evaluate the performance of the ACMCA model in the Alzheimer's disease multi-class prediction task, the present invention conducted experimental comparisons based on MRI images, PET images, genetic data, and clinical data from the ADNI database, and compared them with other baseline models. For performance evaluation, four typical indicators were selected: accuracy, recall, specificity, and F1 score.
[0034] 1) Multimodal effectiveness experiment; To evaluate the performance of the ACMCA model in Alzheimer's disease diagnosis, this study systematically compared it with representative multimodal learning methods from recent years. The core methods of each baseline model are summarized below: Model 1 (Feature Stitching): Venugopalan et al. integrated image and non-image features through direct stitching and used a random forest classifier to complete a multi-classification task.
[0035] Model 2 (MCAD): A fusion architecture based on cross-attention mechanism that optimizes multimodal feature representation through alignment loss function.
[0036] Model 3 (Dual Attention): A cascaded attention network that sequentially interacts with the spatial attention module through channels.
[0037] Model 4 (MADDi): A hierarchical attention architecture that first extracts intramodal features through a self-attention module, and then captures intermodal relationships through cross-attention.
[0038] To analyze the contributions of the model components in depth, this study designed the following comparison scheme: Model 5 (MADDi-ACM): Introduces an asymmetric attention mechanism into the MADDi framework. Model 6 (ACMCA-CM): Replaces the asymmetric modules in the ACMCA model with symmetric structures. Model 7 (ACMCA-MCAD): Employs a hybrid architecture of feature concatenation followed by cross-attention in the ACMCA model. Model 8 (ACMCA): The complete ACMCA model architecture proposed in this embodiment. Detailed performance comparisons are shown in Table 1.
[0039] Table 1 Performance Comparison Experiment Results ; Analysis of the experimental data in Table 1 shows that the attention mechanism significantly improves the performance of multi-classification tasks. In the benchmark comparison, the proposed Model 1 uses a simple feature concatenation strategy, which is difficult to effectively capture the deep correlations between multimodal features. In contrast, the MADDi model, the constructed Model 3, and the ACMCA model proposed in this embodiment all introduce an attention mechanism for multimodal feature processing. Experimental data show that these models improve classification accuracy by 7.9%, 6.1%, and 16.8% respectively compared to Model 1. This result fully verifies the effectiveness of the attention mechanism in uncovering the intrinsic correlations in multimodal data.
[0040] Further analysis reveals that the asymmetric cross-modal cross-attention mechanism employed by the ACMCA model exhibits unique advantages in fusing image and non-image features. In contrast, the MADDi model employs a strategy of directly stitching together bimodal features after processing them with symmetric cross-attention. When the symmetric mechanism is replaced with the asymmetric design of this invention, the model's ability to capture the correlation between image and non-image features is significantly enhanced, improving classification accuracy by 2.6%. Furthermore, the proposed Model 2 addresses modal heterogeneity by first stitching image features and then performing symmetric cross-attention with non-image features. After replacing the asymmetric mechanism in the ACMCA model with this symmetric processing method, a significant decrease in model performance was observed. This comparative result further confirms the superiority of the asymmetric architecture in processing heterogeneous modal data.
[0041] 2) Ablation experiment; To evaluate the effectiveness of each component of the model, this invention designed an ablation experiment for the system. On a unified dataset, the performance of the complete ACMCA model in AD tri-class classification was compared with that of four variants with specific modules removed: The ACMCA-WCM version removes the cross-attention module and directly passes image and non-image features to the downstream network to verify the key role of the cross-modal interaction mechanism. The ACMCA-WDE variant skips the deep feature extraction stage and directly inputs the fused features into the classifier to evaluate the necessity of deep feature transformation. The ACMCA-WFnet model, while keeping the backbone architecture unchanged, masks the Fourier transform component, aiming to explore the contribution of frequency domain analysis to feature learning. The ACMCA-WT variant specifically examines the effectiveness of the self-attention mechanism in feature extraction by removing the self-attention module but retaining other components.
[0042] This hierarchical ablation design can accurately quantify the marginal contribution of each sub-module to the overall system performance, providing data support for model structure optimization, as shown in Table 2.
[0043] Table 2 Ablation Experiment Results ; Table 2 shows the results of the ablation experiments, indicating that each core module in the ACMCA model significantly contributes to performance. Removing the asymmetric cross-modal attention module resulted in a substantial drop in model performance, confirming its crucial role in uncovering the intrinsic correlations within multimodal data and improving classification performance.
[0044] The deep feature extraction module is equally crucial. Removing it entirely leads to a significant performance degradation. Further decomposition experiments show that disabling either the Fnet (Fourier) module or the Transformer (self-attention) module individually results in similar performance losses (AUC decreases of 2.2% and 1.0%, respectively). This phenomenon indicates that the frequency domain transformer plays a similar important role in the model as the self-attention mechanism.
[0045] 3) Analysis of actual visualization results; Figure 4 This presentation showcases the GRAD-CAM comparative visualization results for three diagnostic categories generated based on the ACMCA model. Figure 4 (a) and Figure 4 (b)). The GRAD-CAM (Gradient Weighted Class Activation Mapping) technique generates heatmaps that highlight the key brain regions that the model focuses on during classification, thereby revealing the attention mechanism behind its decision-making.
[0046] Visualization results showed that the model exhibited significant high-intensity local activation in specific brain regions (such as the entorhinal cortex)—a region widely considered an early biomarker for Alzheimer's disease (AD). Simultaneously, significant activation was also observed in the hippocampus and ventricles, reflecting local structural changes associated with brain atrophy.
[0047] These experiments demonstrate that the ACMCA model proposed in this embodiment can effectively focus on key brain regions related to Alzheimer's disease and successfully identify potential biomarkers with clinical significance, thereby verifying the interpretability and medical relevance of the model.
[0048] The main technical features that distinguish this invention from the prior art are as follows: 1) In the feature fusion module, an asymmetric cross-modal cross-attention mechanism deep learning network model is proposed to enhance the association learning ability between features of different modalities.
[0049] 2) In the deep feature extraction module, the Transformer architecture and frequency domain converter are combined to improve the feature extraction effect by utilizing frequency domain analysis.
[0050] 3) Innovatively combining four different types and modalities of medical multi-view data—PET images, MRI images, clinical data, and genetic data—to train a predictive model for AD.
[0051] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A method for predicting Alzheimer's disease based on asymmetric cross-modal cross-attention, characterized in that, Includes the following steps: Multimodal medical data was constructed by combining PET imaging data, MRI imaging data, clinical scale data, and genetic data, and the data was preprocessed to obtain preprocessed multimodal medical data. An asymmetric cross-modal cross-attention prediction network is constructed by combining a shallow feature extraction module, a multimodal attention fusion module, a deep feature extraction module, and an MLP classification module. Based on an asymmetric cross-modal cross-attention prediction network, Alzheimer's disease prediction is performed on preprocessed multimodal medical data, and the prediction results are obtained.
2. The Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention as described in claim 1, characterized in that, The asymmetric cross-modal cross-attention prediction network specifically includes a shallow feature extraction module, a multimodal attention fusion module, a deep feature extraction module, and an MLP classification module. The shallow feature extraction module, the multimodal attention fusion module, the deep feature extraction module, and the MLP classification module are connected sequentially, wherein: The shallow feature extraction module includes a pre-trained ResNet50 model and a deep neural network model; The deep feature extraction module includes a self-attention module and a Fourier module; The MLP classification module includes a fully connected layer and a ReLU activation function.
3. The Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention according to claim 2, characterized in that, The step of using an asymmetric cross-modal cross-attention prediction network to predict Alzheimer's disease from preprocessed multimodal medical data and obtain the prediction results specifically includes: The preprocessed multimodal medical data is input into an asymmetric cross-modal cross-attention prediction network; A shallow feature extraction module based on an asymmetric cross-modal cross-attention prediction network is used to perform shallow feature extraction on preprocessed multimodal medical data to obtain multimodal medical shallow feature data. A multimodal attention fusion module based on an asymmetric cross-modal cross-attention prediction network is used to perform feature fusion processing on shallow feature data of multimodal medicine to obtain multimodal medical fused feature data. A deep feature extraction module based on an asymmetric cross-modal cross-attention prediction network is used to perform deep feature extraction processing on multimodal medical fusion feature data to obtain multimodal medical deep feature data. The MLP classification module based on the asymmetric cross-modal cross-attention prediction network is used to predict and classify multimodal medical deep feature data to obtain Alzheimer's disease prediction results.
4. The Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention according to claim 3, characterized in that, The shallow feature extraction module based on the asymmetric cross-modal cross-attention prediction network performs shallow feature extraction processing on the preprocessed multimodal medical data to obtain multimodal medical shallow feature data. This step specifically includes: The preprocessed multimodal medical data is input into the shallow feature extraction module of the asymmetric cross-modal cross-attention prediction network; A pre-trained ResNet50 model based on a shallow feature extraction module is used to perform shallow feature extraction on PET and MRI image data respectively, resulting in shallow feature data of PET and MRI images. Based on the deep neural network model of the shallow feature extraction module, shallow feature extraction is performed on clinical scale data and gene data respectively to obtain shallow feature data of clinical scale and shallow feature data of genes. By combining superficial feature data from PET images, superficial feature data from MRI images, superficial feature data from clinical scales, and superficial feature data from genes, multimodal medical superficial feature data is obtained.
5. The Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention according to claim 4, characterized in that, The multimodal attention fusion module based on the asymmetric cross-modal cross-attention prediction network performs feature fusion processing on the shallow feature data of multimodal medicine to obtain multimodal medical fused feature data. This step specifically includes: Multimodal medical shallow feature data is input into the multimodal attention fusion module of an asymmetric cross-modal cross-attention prediction network; The superficial feature data of clinical scales and superficial feature data of genes are linearly transformed into query vectors, and the superficial feature data of MRI images and superficial feature data of PET images are mapped into key vectors and value vectors, respectively. Calculate the dot product between the query vector and the key vector, divide by the square root of the vector dimension, and convert it into a probability vector using the Softmax function to obtain the fused features of the MRI image shallow feature data and the clinical scale shallow feature data. The fusion features and value vectors of MRI image features and clinical data features are calculated using the Softmax function to obtain the fusion features of PET image shallow feature data and gene shallow feature data. Multimodal medical fusion feature data is obtained by splicing together the fusion features of MRI image superficial feature data and clinical scale superficial feature data, the fusion features of clinical scale superficial feature data, PET image superficial feature data and gene superficial feature data, and gene superficial feature data.
6. The Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention according to claim 5, characterized in that, The calculation process for the fusion features of the MRI image superficial feature data and the clinical scale superficial feature data is as follows: ； In the above formula, This represents the fusion characteristics of superficial feature data from MRI images and superficial feature data from clinical scales. This represents the query vector obtained by linearly transforming the shallow feature data of the clinical scale. This represents the key vector obtained by mapping shallow feature data from MRI images. This represents the value vector obtained by mapping shallow feature data from MRI images. This represents the Softmax function. This represents a cross-modal attention mechanism. This represents the dimension of the key vector.
7. The Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention according to claim 6, characterized in that, The calculation process for the fusion features of the PET image shallow feature data and the gene shallow feature data is as follows: ； In the above formula, This represents the fusion characteristics of PET image shallow feature data and genetic shallow feature data. This represents the query vector obtained by linearly transforming shallow gene feature data. This represents the key vector obtained by mapping shallow feature data from PET images. This represents the value vector obtained by mapping shallow feature data from PET images. This represents the Softmax function. This represents a cross-modal attention mechanism. This represents the dimension of the key vector.
8. The Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention according to claim 7, characterized in that, The deep feature extraction module based on the asymmetric cross-modal cross-attention prediction network performs deep feature extraction processing on the multimodal medical fusion feature data to obtain multimodal medical deep feature data. This step specifically includes: Multimodal medical fusion feature data is input into the deep feature extraction module of an asymmetric cross-modal cross-attention prediction network; The self-attention module based on the deep feature extraction module performs Fourier transform processing on the multimodal medical fusion feature data to obtain the Fourier transformed multimodal medical fusion feature data. The Fourier module based on the deep feature extraction module performs global relation capture processing on the multimodal medical fusion feature data after Fourier transform to obtain multimodal medical deep feature data.
9. The Alzheimer's disease prediction method based on asymmetric cross-modal cross-attention according to claim 8, characterized in that, The MLP classification module based on an asymmetric cross-modal cross-attention prediction network performs prediction and classification on multimodal medical deep feature data to obtain Alzheimer's disease prediction results. This step specifically includes: Multimodal medical deep feature data is input into the MLP classification module of an asymmetric cross-modal cross-attention prediction network; A fully connected layer based on the MLP classification module is used to perform linear transformation on the multimodal medical deep feature data to obtain the transformed multimodal medical deep feature data. Based on the ReLU activation function of the MLP classification module, the transformed multimodal medical deep feature data is classified and predicted to obtain Alzheimer's disease prediction results.