ADHD classification diagnosis method based on multi-modal attention fusion network

Abstract

The invention relates to an ADHD classification diagnosis method based on a multi-modal attention fusion network. The ADHD classification diagnosis method comprises the steps of data preprocessing, feature extraction, cross-modal attention fusion, Mama module application, feature fusion and classification and loss function and model optimization. The present invention improves the diagnostic accuracy of ADHD by integrating functional magnetic resonance imaging (fMRI), structural magnetic resonance imaging (sMRI), and phenotypic data (including age, gender, average intelligence quotient, etc.

Description

Technical Field

[0001] The invention relates to an ADHD classification diagnosis method based on a multimodal attention fusion network, and belongs to the technical field of computer vision. Background Art

[0002] Attention-Deficit Hyperactivity Disorder (ADHD) is a common developmental mental disorder in children, characterized by difficulty concentrating, hyperactivity, and impulsive behavior. Global data show that ADHD affects approximately 7.6% of children aged 3 to 12 and 5.6% of adolescents. The overall prevalence among Chinese children is approximately 5.7%, and has been on the rise in recent years. The diagnosis of ADHD relies on behavioral observations and feedback from parents and teachers, based on standards such as DSM-5 or ICD-10 / 11. The symptoms are complex and varied, and the diagnostic process usually requires the exclusion of diseases with similar symptoms such as emotional disorders and learning disabilities.

[0003] In recent years, with the development of medical imaging technology, methods such as functional magnetic resonance imaging (fMRI) and structural magnetic resonance imaging (sMRI) have gradually been applied to the diagnosis of ADHD in order to improve the accuracy of diagnosis. In addition, with the advancement of machine learning and deep learning technologies, researchers have begun to try to use models such as convolutional neural networks (CNN) and multi-layer perceptrons (MLP) to automatically diagnose ADHD. However, the current ADHD diagnostic model mainly focuses on a single imaging modality and lacks effective integration of multimodal data. Existing studies have shown that the combined application of multimodal data such as fMRI and sMRI can provide more comprehensive diagnostic information and help improve the accuracy and reliability of ADHD diagnosis.

[0004] Although several studies have attempted to use deep learning techniques in ADHD diagnosis, current multimodal fusion methods have many shortcomings when dealing with heterogeneous data (such as fMRI, sMRI, and phenotypic data):

[0005] 1. Lack of cross-modal fusion methods: Many existing studies only analyze single-modality data, or use simple multi-modal combinations, failing to fully utilize the complementary information between fMRI, sMRI, and phenotypic data. This fusion method lacks an effective mechanism to promote feature interaction between modalities, resulting in incomplete diagnostic information and waste of modality information.

[0006] 2. Fusion methods do not adequately handle heterogeneous data: Existing multimodal fusion methods are usually unable to effectively solve the heterogeneity problem between different modal data. For example, fMRI and sMRI are image data, while phenotypic data are text data. How to efficiently integrate these heterogeneous data types to support clinical diagnosis remains a challenge.

[0007] 3. Insufficient model accuracy and interpretability: In medical applications, high accuracy and strong interpretability of diagnostic models are crucial. However, existing models still have much room for improvement in the accuracy of ADHD diagnosis. Due to the lack of an effective fusion mechanism, traditional methods have limitations in feature expression and classification accuracy, and poor interpretability, making it difficult to provide clear diagnostic evidence in actual clinical applications. Summary of the invention

[0008] In order to overcome the defects of the existing technology that it is difficult to effectively integrate the information contained in multimodal data (such as fMRI, sMRI and phenotypic data), resulting in insufficient diagnostic accuracy and reliability, the present invention provides a trimodal fusion module that can efficiently fuse multiple modalities and a Mamba self-attention mechanism module, thereby significantly improving the performance and interpretability of the ADHD diagnostic model. The ADHD classification diagnosis method based on the multimodal attention fusion network aims to solve the shortcomings of the existing ADHD diagnostic technology in multimodal data fusion.

[0009] The present invention proposes an ADHD classification model based on a multimodal attention fusion network (MMAFN), which aims to improve the diagnostic accuracy of ADHD by integrating functional magnetic resonance imaging (fMRI), structural magnetic resonance imaging (sMRI) and phenotypic data (including age, gender, average IQ, etc.). The model structure mainly includes a feature extraction module, a multimodal interaction module and a classification module, and the self-attention mechanism of the Mamba module is used to enhance the extraction and fusion of cross-modal features. First, feature extraction is performed on data of different modalities through a pre-trained model, and then fusion is performed using a cross-modal attention mechanism to capture the deep correlation between fMRI, sMRI and phenotypic data, and finally ADHD is accurately classified through a DenseNet classifier. The present invention optimizes the fusion effect of multimodal features through a specific adversarial cross entropy loss function, thereby significantly improving the classification performance.

[0010] A classification and diagnosis method for ADHD based on a multimodal attention fusion network comprises the following steps:

[0011] Step 1: Data preprocessing: Standardize and clean the multimodal data to ensure the quality and consistency of the input data; apply noise filtering to the fMRI and sMRI data after standardization to ensure the quality of functional and structural imaging data. In addition, the fMRI data is subjected to average pooling to reduce the impact of noise on the model input. For phenotypic data, data from multiple sources can be integrated to construct phenotypic information such as age, gender, average IQ, and handedness as text data for subsequent processing;

[0012] Step 2: Feature extraction: Use independent encoders to extract features from fMRI, sMRI and phenotypic data. For fMRI and sMRI features, the 3D visual features of fMRI and sMRI data can be extracted through a pre-trained visual encoder. For phenotypic features, the phenotypic data can be encoded using a text encoder. BioBERT is a language model designed specifically for the biomedical field that can extract key feature information from biomedical texts. In step 2, the text encoder uses the publicly available pre-trained BioClinicalBERT model, and the visual encoder uses a pre-trained 3D ResNet-50.

[0013] Step 3: Cross-modal attention fusion: The cross-modal attention mechanism is used to achieve deep fusion of multimodal features, promote the extraction of associations between different data modalities, and calculate the query matrix Q, key matrix K and value matrix V for fMRI, sMRI and text data respectively to perform cross-modal feature interaction. The attention weight is calculated based on the similarity between the query matrix Q and the key matrix K, and then applied to the value matrix V to form a fused cross-modal feature; Fusion process: The features of fMRI, sMRI and phenotypic data are fused through the attention mechanism to construct a global feature representation containing trimodal information;

[0014] Step 4: Application of Mamba module: The Mamba module is used to fuse features to capture more complex cross-modal feature relationships. The Mamba module uses local convolution and state expansion techniques to simulate autoregressive processes in long sequences, obtain long-range dependencies in features, and generate weights for each input feature matrix through the self-attention mechanism, emphasizing features with higher information content and weakening irrelevant or repeated features. This module is based on the self-attention mechanism and has the characteristic of capturing long-range dependencies, making it suitable for long sequence data in medical imaging data.

[0015] Local convolution and state expansion: The Mamba module uses local convolution and state expansion techniques to simulate autoregressive processes in long sequences to capture long-range dependencies in features.

[0016] Self-attention calculation: The self-attention mechanism generates weights for each input feature matrix, thereby emphasizing features with higher information content and weakening irrelevant or repeated features.

[0017] Step 5: Feature fusion and classification: After integrating the multimodal features into a unified global feature representation, the final classification of ADHD is performed through the classification module;

[0018] Step 6: Loss function and model optimization: Design an adversarial cross entropy loss function to optimize the feature fusion effect. Adversarial cross entropy loss combines cross entropy loss and generative adversarial loss. By adjusting the hyperparameters, the contribution of these two parts of loss is balanced to enhance the model's learning ability in classification tasks. The adversarial loss effect helps the model learn the distribution characteristics of samples and promotes deeper fusion of cross-modal features, so that the model can effectively capture the intrinsic correlation of each modal data.

[0019] The step 2 specifically includes: extracting features from fMRI, sMRI image data and phenotypic data respectively through a visual encoder and a text encoder to form a visual feature representation and a text feature representation.

[0020] The step three specifically includes:

[0021] 3.1 Linear transformation: Data features F for each mode i Perform linear transformation to generate the corresponding query matrix Q, key matrix K and value matrix V. The formula is as follows:

[0022]

[0023] in, is the linear transformation matrix;

[0024] 3.2 Calculate similarity: By calculating the query matrix Q i and key matrix K j The dot product of modality i and modality j is used to obtain the similarity score α ij , the formula is:

[0025]

[0026] Among them, d k is the dimension of the feature space, used to scale the similarity score;

[0027] 3.3 Cross-modal feature fusion: Normalize the similarity scores and apply them to the value matrix V j , generate cross-modal fusion features F cross , the calculation formula is:

[0028]

[0029] Through calculation, the cross-modal attention mechanism can effectively fuse information from different modalities and generate rich global feature representations.

[0030] The step six specifically includes:

[0031] Cross entropy loss L CE: Cross entropy loss is used for supervised classification tasks, calculating the difference between the model's predicted output and the true label, helping the model to distinguish categories;

[0032] Generate adversarial loss L GAN : Generate adversarial loss through adversarial learning mechanism, encouraging the model to capture richer multimodal feature distribution in feature learning;

[0033] The final adversarial cross entropy loss function is:

[0034] L=αL CE +βL GAN

[0035] Among them, α and β are hyperparameters used to balance the contribution of cross entropy loss and generative adversarial loss. By adjusting the two parameters, flexible optimization for different task requirements can be achieved.

[0036] The step five specifically includes:

[0037] Global feature construction: connect the fused phenotypic features and imaging features into a unified global representation to form a feature set suitable for classification tasks;

[0038] Classification module: Use DenseNet as a classifier and input the final fused features into the DenseNet classification head. The DenseNet structure can maintain feature fluidity during classification, which helps to retain information of each modality and improve classification accuracy.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention relates to an ADHD classification diagnosis method based on a multimodal attention fusion network, which includes the following innovative features:

[0041] 1. Multimodal data fusion: The present invention combines multimodal imaging data (including structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI)) with phenotypic data (such as age, gender, IQ, etc.) to improve the accuracy of ADHD classification and the interpretability of the model. Through the effective integration of multimodal data, the present invention can provide richer information support and make the diagnostic results more clinically valuable.

[0042] 2. Trimodal fusion component: A multimodal attention fusion network (MMAFN) was developed, which can achieve effective cross-modal integration between different data types and overcome the heterogeneity problem between multimodal data. This fusion component realizes the deep fusion of imaging data and phenotypic data, which helps to extract more comprehensive feature representation, thereby enhancing the overall performance of the model.

[0043] 3. Introduction of Mamba module: The Mamba module is embedded in the multimodal attention fusion module, which further improves the efficiency of feature extraction and combination. The Mamba module adopts the self-attention mechanism, which can effectively capture the long-range dependencies between different modalities, helping the model to better understand the complex medical data structure, thereby improving classification performance and data interpretability.

[0044] 4. Joint loss function design: We designed an adversarial cross entropy loss function that combines cross entropy loss and generative adversarial loss to enhance the generalization ability of the model through adversarial learning of multimodal features. This loss function not only optimizes the classification performance, but also improves the effect of multimodal feature fusion, making the model more robust and accurate in complex ADHD diagnosis tasks.

[0045] The experimental results in the specific embodiments show that the MMAFN model proposed in the present invention is superior to the current mainstream ADHD diagnostic method in all performance indicators, achieving a classification accuracy of 83.51%, and indicators such as precision, recall rate, F1 score and AUC value have all been significantly improved, with accuracy and precision increased by 2.22% and 4.38% respectively. The present invention provides a new idea for the application of multimodal fusion in the diagnosis of mental illness, provides effective support for the diagnosis of ADHD in clinical practice, and has the potential to be further promoted and applied to the diagnosis of other mental disorders. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 This is a diagram of the architecture of a multimodal attention fusion network (MMAFN) in an ADHD classification diagnosis method based on a multimodal attention fusion network of the present invention. From left to right, the algorithm module uses pre-trained 3D ResNet-50 and BioBERT models to extract deep features from fMRI / sMRI and phenotypic text data, the attention module uses a cross-modal attention mechanism for multimodal fusion, and integrates the Mamba module to capture long-range dependencies, the extracted features are classified through the DenseNet architecture, and the model performance is optimized through a joint loss function;

[0048] Figure 2This is an architecture diagram of a multimodal attention fusion module of an ADHD classification diagnosis method based on a multimodal attention fusion network of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] A classification diagnosis method for ADHD based on a multimodal attention fusion network comprises the following steps:

[0051] Step 1: Data preprocessing: Standardize and clean the multimodal data to ensure the quality and consistency of the input data; apply noise filtering to the fMRI and sMRI data after standardization to ensure the quality of functional and structural imaging data. In addition, the fMRI data is subjected to average pooling to reduce the impact of noise on the model input. For phenotypic data, data from multiple sources can be integrated to construct phenotypic information such as age, gender, average IQ, and handedness as text data for subsequent processing;

[0052] Step 2: Feature extraction: Use independent encoders to extract features from fMRI, sMRI and phenotypic data. For fMRI and sMRI features, the 3D visual features of fMRI and sMRI data can be extracted through a pre-trained visual encoder. For phenotypic features, the phenotypic data can be encoded using a text encoder. BioBERT is a language model designed specifically for the biomedical field that can extract key feature information from biomedical texts. In step 2, the text encoder uses the publicly available pre-trained BioClinicalBERT model, and the visual encoder uses a pre-trained 3D ResNet-50.

[0053] Step 3: Cross-modal attention fusion: The cross-modal attention mechanism is used to achieve deep fusion of multimodal features, promote the extraction of associations between different data modalities, and calculate the query matrix Q, key matrix K and value matrix V for fMRI, sMRI and text data respectively to perform cross-modal feature interaction. The attention weight is calculated based on the similarity between the query matrix Q and the key matrix K, and then applied to the value matrix V to form a fused cross-modal feature; Fusion process: The features of fMRI, sMRI and phenotypic data are fused through the attention mechanism to construct a global feature representation containing trimodal information;

[0054] Step 4: Application of Mamba module: The Mamba module is used to fuse features to capture more complex cross-modal feature relationships. The Mamba module uses local convolution and state expansion techniques to simulate autoregressive processes in long sequences, obtain long-range dependencies in features, and generate weights for each input feature matrix through the self-attention mechanism, emphasizing features with higher information content and weakening irrelevant or repeated features. This module is based on the self-attention mechanism and has the characteristic of capturing long-range dependencies, making it suitable for long sequence data in medical imaging data.

[0055] Local convolution and state expansion: The Mamba module uses local convolution and state expansion techniques to simulate autoregressive processes in long sequences to capture long-range dependencies in features.

[0056] Self-attention calculation: The self-attention mechanism generates weights for each input feature matrix, thereby emphasizing features with higher information content and weakening irrelevant or repeated features.

[0057] Step 5: Feature fusion and classification: After integrating the multimodal features into a unified global feature representation, the final classification of ADHD is performed through the classification module;

[0058] Step 6: Loss function and model optimization: Design an adversarial cross entropy loss function to optimize the feature fusion effect. Adversarial cross entropy loss combines cross entropy loss and generative adversarial loss. By adjusting the hyperparameters, the contribution of these two parts of loss is balanced to enhance the learning ability of the model in classification tasks. The adversarial loss effect helps the model learn the distribution characteristics of samples and promotes deeper fusion of cross-modal features, so that the model can effectively capture the intrinsic correlation of each modal data.

[0059] The step 2 specifically includes: extracting features from fMRI, sMRI image data and phenotypic data respectively through a visual encoder and a text encoder to form a visual feature representation and a text feature representation.

[0060] Model structure diagram Figure 1 The present invention mainly includes three core modules: feature extraction module, cross-modal attention fusion module and classification module. Among them, each module has a clear division of labor and jointly completes the ADHD diagnosis task.

[0061] In the feature extraction module, the model extracts features from fMRI, sMRI image data and phenotypic data through visual encoders and text encoders respectively to form visual feature representations and text feature representations. In the cross-modal attention fusion module, the model uses a cross-modal attention mechanism to fuse visual features and text features to generate a global feature representation containing multimodal information. In order to further optimize the fusion effect, the model embeds the Mamba module in the fusion block to enhance the feature association between multiple modalities. In the final classification module, the fused global features are input into the classifier, and the classification result of ADHD is obtained through the DenseNet classification head.

[0062] The structure of the cross-modal attention block is as follows Figure 2 As shown. The main function of this module is to realize the feature interaction between fMRI, sMRI and phenotypic data through the cross-modal attention mechanism to enhance the fusion effect of multimodal data. In this process, the model calculates the query (Q), key (K) and value (V) matrices in turn to generate cross-modal attention features. The specific process is as follows:

[0063] Linear transformation: for each modality data feature F i Perform linear transformation to generate the corresponding query matrix Q, key matrix K and value matrix V. The formula is as follows:

[0064]

[0065] in, is the linear transformation matrix;

[0066] Calculate similarity: By calculating the query matrix Q i and key matrix K j The dot product of modality i and modality j is used to obtain the similarity score α ij , the formula is:

[0067]

[0068] Among them, d k is the dimension of the feature space, used to scale the similarity score;

[0069] Cross-modal feature fusion: Normalize the similarity scores and apply them to the value matrix V j , generate cross-modal fusion features F cross , the calculation formula is:

[0070]

[0071] Through calculation, the cross-modal attention mechanism can effectively fuse information from different modalities and generate rich global feature representations.

[0072] Mamba module: In order to further optimize the fusion effect, the present invention introduces the Mamba module after the cross-modal attention mechanism. This module is based on the self-attention mechanism and uses local convolution and state expansion to capture long-range dependencies. Compared with the traditional self-attention method, the Mamba module is more suitable for processing long sequence features, which can effectively improve the fusion effect of multimodal data and help the model better understand complex medical imaging data.

[0073] Finally, after being processed by the cross-modal attention block and the Mamba module, the model is able to generate more accurate and comprehensive cross-modal fusion features, providing a solid foundation for the classification of ADHD.

[0074] In order to further improve the classification performance of the model of the present invention under multimodal fusion, an adversarial cross entropy loss function is designed, which combines the cross entropy loss and the generative adversarial loss to enhance the learning and classification effect of the model on multimodal data.

[0075] Cross entropy loss L CE : Cross entropy loss is used for supervised classification tasks, calculating the difference between the model's predicted output and the true label, helping the model to distinguish categories;

[0076] Generate adversarial loss L GAN : Generate adversarial loss through adversarial learning mechanism, encouraging the model to capture richer multimodal feature distribution in feature learning;

[0077] The final adversarial cross entropy loss function is:

[0078] L=αL CE +βL GAN

[0079] Among them, α and β are hyperparameters used to balance the contribution of cross entropy loss and generative adversarial loss. By adjusting the two parameters, flexible optimization for different task requirements can be achieved.

[0080] The step five specifically includes:

[0081] Global feature construction: connect the fused phenotypic features and imaging features into a unified global representation to form a feature set suitable for classification tasks;

[0082] Classification module: Use DenseNet as a classifier and input the final fused features into the DenseNet classification head. The DenseNet structure can maintain feature fluidity during classification, which helps to retain information of each modality and improve classification accuracy.

[0083] This paper proposes an ADHD classification method based on a multimodal attention fusion network (MMAFN), which improves the accuracy and robustness of multimodal diagnosis by combining fMRI, sMRI and phenotypic data. The model design of the present invention realizes deep feature fusion of different modal data through the introduction of cross-modal attention mechanism and Mamba module, and uses adversarial cross entropy loss function to optimize the feature learning and classification performance of the model.

[0084] Experimental results show that the MMAFN model outperforms existing mainstream methods in key indicators such as classification accuracy, precision, recall rate, and F1 score. Through the effective fusion of cross-modal features, the present invention can achieve rich feature expression in complex data backgrounds and improve the recognition ability of ADHD-related neurobehavioral patterns. At the same time, the method of the present invention has good scalability and can be further applied to the diagnosis of other mental illnesses.

[0085] In summary, this invention demonstrates the application potential of multimodal deep learning technology in the diagnosis of mental illness and provides an effective technical solution for the automated classification of ADHD, which has significant clinical application value and potential for further promotion.

[0086] experiment:

[0087] 4.1 Dataset

[0088] The present invention uses the public ADHD-200 dataset, which contains multimodal data for attention deficit hyperactivity disorder (ADHD) research, including fMRI (functional magnetic resonance imaging), sMRI (structural magnetic resonance imaging) and phenotypic information (such as age, gender, average IQ and hand shape, etc.). These data come from a wide range of sources, covering participants from different age groups and geographical regions, with good representativeness and diversity, and are suitable for multimodal model training for ADHD diagnosis.

[0089] 4.2 Baselines and Evaluation Metrics

[0090] In order to comprehensively evaluate the effectiveness of the MMAFN model proposed in the present invention in the ADHD classification task, the present invention sets up multiple baseline models and uses a series of evaluation indicators for performance comparison.

[0091] Baseline model: To verify the improvement of classification performance by multimodal data, the present invention conducted comparative experiments with the following baseline models:

[0092] (1) fMRI-only model: Only fMRI image data are used for classification to evaluate the performance of functional magnetic resonance imaging in a single-modality case.

[0093] (2) sMRI-only model: Only sMRI image data are used for classification to evaluate the classification ability of structural magnetic resonance imaging.

[0094] (3) Text-only model: Only phenotypic data (such as age, gender, average IQ, etc.) are used for classification to evaluate the effectiveness of phenotypic data in a unimodal situation.

[0095] (4) fMRI+sMRI model: combines fMRI and sMRI data, but does not include phenotypic information, to test the fusion effect of multimodal imaging data.

[0096] (5) fMRI+Text model and sMRI+Text model: fMRI or sMRI is combined with phenotypic data for classification to test the fusion effect between imaging data and phenotypic data.

[0097] Evaluation indicators: The present invention evaluates the performance of the model through the following indicators:

[0098] (1) Accuracy: It indicates the proportion of correct classification of the model in all test samples, and measures the overall classification ability of the model.

[0099] (2) Precision: It indicates the proportion of samples that are actually ADHD among the samples predicted by the model as ADHD, which measures the accuracy of the model's positive sample prediction.

[0100] (3) Recall: It indicates the proportion of samples that are actually ADHD that are correctly identified by the model, and measures the model's ability to identify positive samples.

[0101] (4) F1 Score: The harmonic average of precision and recall, which comprehensively evaluates the model's performance in terms of accuracy and completeness.

[0102] (5) AUC (Area Under the Curve): The area under the ROC curve measures the model's ability to distinguish between positive and negative samples.

[0103] 4.3 Implementation Details

[0104] The experiment of the present invention is implemented on the PyTorch framework, using the Python programming language, and trained on a single Nvidia A100 GPU. To ensure the convergence and stability of the model, the experiment sets a unified training strategy and hyperparameter configuration. The specific implementation details are as follows:

[0105] (1) Training Epochs: The model training cycle is set to 300 epochs to ensure that the model fully learns the data features.

[0106] (2) Batch Size: The batch size is set to 2 to accommodate the memory requirements of multimodal data and ensure the diversity of each batch of data.

[0107] (3) Optimizer: The Adam optimizer is used to update the model parameters. This optimizer has strong convergence ability and is suitable for the training of deep neural networks.

[0108] (4) Learning Rate: The initial learning rate is set to 0.0001. This learning rate is used to control the convergence speed of the model to prevent too fast or too slow convergence.

[0109] (5) Loss function: The adversarial cross entropy loss function is used, which combines the cross entropy loss and the generative adversarial loss. It can enhance the fusion effect of multimodal features while maintaining the classification accuracy.

[0110] During the training process, the present invention monitors the performance of the validation set through the early stopping strategy. When the validation error of the model does not decrease significantly for multiple consecutive rounds, the training is stopped in advance to prevent overfitting. At the same time, a weight regularization term is added during the model training process to improve the generalization ability of the model.

[0111] The above implementation details ensure the training stability and convergence of the model of the present invention in the ADHD classification task, and provide a reliable experimental basis for the final multimodal feature fusion and classification performance.

[0112] 4.4 Experimental Results

[0113] 4.4.1 Quantitative analysis

[0114] This paper quantitatively analyzes the classification effect of the MMAFN model by comparing the performance indicators of different models. The experimental results show that the present invention is superior to the baseline model in all key indicators, which fully verifies the effectiveness of multimodal fusion and cross-modal attention mechanism. The following are the main quantitative analysis results of each model on the test set:

[0115] (1) Single-modal model performance:

[0116] fMRI-only model: Using only fMRI data, the model achieved 78.24% accuracy, but its performance in identifying ADHD features was relatively limited.

[0117] sMRI-only model: When only sMRI data were used, the accuracy was 74.15%, which was slightly lower than that of the fMRI-only model, indicating that single structural imaging data has limited information in ADHD classification.

[0118] Text-only model: When using only phenotypic data for classification, the model’s accuracy was 69.88%, the lowest among the unimodal models, indicating that it is difficult to accurately distinguish ADHD by relying solely on phenotypic data.

[0119] (2) Multimodal fusion model performance:

[0120] fMRI+sMRI model: The classification accuracy was improved to 80.45% by combining fMRI and sMRI data, indicating that the fusion of imaging data can enhance feature expression.

[0121] 6. fMRI+Text model and sMRI+Text model: When the imaging data are combined with the phenotypic data, the accuracy rates reach 81.32% and 80.22% respectively, showing the complementary role of phenotypic data in multimodal fusion.

[0122] (3) MMAFN model performance:

[0123] Trimodal fusion: The MMAFN model proposed in this paper achieved a classification accuracy of 83.51% by fusing fMRI, sMRI and phenotypic data, which is significantly higher than other baseline models. The MMAFN model also showed significant advantages in precision, recall, F1 score and AUC indicators, with improvements of 4.38%, 6.94%, 0.0574 and 0.0222 respectively. This shows that the MMAFN model can effectively capture the association between multimodal features through the multimodal attention fusion mechanism and the Mamba module, greatly improving the classification performance of ADHD.

[0124] The quantitative analysis results of the present invention verify the superiority of the MMAFN model in multimodal fusion and feature extraction, and provide strong technical support for the application of multimodal data in ADHD diagnosis.

[0125] 4.4.2 Qualitative analysis

[0126] This paper further verifies the effectiveness of the MMAFN model in ADHD diagnosis tasks through qualitative analysis, focusing on the advantages of the model in multimodal data fusion and cross-modal feature extraction. By analyzing the classification results and feature attention distribution of specific samples, we can intuitively see how the model uses multimodal features to improve the accuracy and interpretability of diagnosis.

[0127] (1) Complementarity of multimodal data:

[0128] In the diagnostic process of the MMAFN model, fMRI, sMRI and phenotypic data work together, and each modality is deeply integrated through a cross-modal attention mechanism. For example, for samples with complex symptoms or unclear boundaries, the model can combine the functional information of fMRI, the structural characteristics of sMRI, and the demographic information in the phenotypic data to more comprehensively identify ADHD characteristics. This shows that the joint use of multimodal data can effectively make up for the shortcomings of single-modal data, thereby providing more accurate classification results.

[0129] (2) Mamba module’s long-range dependency capture capability:

[0130] The Mamba module introduces long-range dependencies in the fused features, which can effectively capture the complex associations between fMRI, sMRI, and phenotypic data. Through the visualization analysis of the feature focus areas, it can be seen that the Mamba module helps the model focus on areas with more prominent ADHD features and reduces attention to irrelevant information. This mechanism makes the MMAFN model more stable and reliable in identifying complex neurobehavioral patterns of ADHD.

[0131] (3) Enhanced model interpretability:

[0132] Due to the sensitivity and accuracy requirements of medical diagnosis, the interpretability of the model is crucial. By visualizing the model's focus area, we can intuitively understand the degree to which the model relies on different features during the diagnosis process. For example, in some typical ADHD samples, the MMAFN model pays more attention to the functional activity (provided by fMRI) and structural information (provided by sMRI) of specific brain regions, and combined with the characteristics of phenotypic data, the diagnostic results are more interpretable. This not only improves the credibility of the model in actual clinical applications, but also provides clinicians with a reliable basis for auxiliary diagnosis.

[0133] Table 1 shows the results of the experiment.

[0134]

[0135] In summary, the qualitative analysis of the present invention further demonstrates the significant advantages of the MMAFN model in multimodal fusion and cross-modal feature extraction, and proves the good diagnostic performance and explanatory ability of the model in the ADHD classification task.

[0136] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A classification and diagnosis method for ADHD based on a multimodal attention fusion network, characterized by: The following steps are involved: Step 1: Data preprocessing: Standardize and clean the multimodal data to ensure the quality and consistency of the input data; apply noise filtering after standardization of fMRI and sMRI data; for phenotypic data, integrate data from multiple sources and construct the phenotypic information of age, gender, average IQ and handedness into text data for subsequent processing; Step 2: Feature extraction: Use independent encoders to extract features from fMRI, sMRI, and phenotypic data. For fMRI and sMRI features, use pre-trained visual encoders to extract 3D visual features of fMRI and sMRI data. Phenotypic features: phenotypic data can be encoded using a text encoder; Step 3: Cross-modal attention fusion: The cross-modal attention mechanism is used to achieve deep fusion of multimodal features. The query matrix Q, key matrix K and value matrix V are calculated for fMRI, sMRI and text data respectively, and cross-modal feature interaction is performed. The similarity between the query matrix Q and the key matrix K is calculated and then applied to the value matrix V to form a fused cross-modal feature. The features of fMRI, sMRI and phenotypic data are fused through the attention mechanism to construct a global feature representation containing trimodal information. Step 4: Mamba module application: The Mamba module applies fusion features to capture more complex cross-modal feature relationships. The Mamba module uses local convolution and state expansion techniques to simulate autoregressive processes in long sequences, obtain long-range dependencies in features, and generate weights for each input feature matrix through the self-attention mechanism, emphasizing features with higher information content and weakening irrelevant or repeated features. Step 5: Feature fusion and classification: After integrating the multimodal features into a unified global feature representation, the final classification of ADHD is performed through the classification module; Step 6: Loss function and model optimization: Design an adversarial cross entropy loss function to optimize the feature fusion effect. Adversarial cross entropy loss combines cross entropy loss and generative adversarial loss. By adjusting the hyperparameters, the contribution of these two parts of loss is balanced to enhance the model's learning ability in classification tasks. The adversarial loss effect helps the model learn the distribution characteristics of samples and promotes deeper fusion of cross-modal features. The model can effectively capture the intrinsic correlation of each modal data.

2. The ADHD classification diagnosis method based on a multimodal attention fusion network according to claim 1, characterized in that: The step 2 specifically includes: extracting features from fMRI, sMRI image data and phenotypic data respectively through a visual encoder and a text encoder to form a visual feature representation and a text feature representation.

3. The ADHD classification diagnosis method based on a multimodal attention fusion network according to claim 1 or 2, characterized in that: In step 2, the text encoder uses the publicly available pre-trained BioClinicalBERT model, and the visual encoder uses the pre-trained 3D ResNet-50.

4. The ADHD classification diagnosis method based on a multimodal attention fusion network according to claim 1, characterized in that: The step three specifically includes: 3.1 Linear transformation: Data features F for each mode i Perform linear transformation to generate the corresponding query matrix Q, key matrix K and value matrix V. The formula is as follows: Q i =F i W Qi ,K i =F i W Ki ,V i =F i W Vi Among them, W Qi , W Ki , W Vi is the linear transformation matrix; 3.2 Calculate similarity: By calculating the query matrix Q i and key matrix K j The dot product of modality i and modality j is used to obtain the similarity score α ij , the formula is: Among them, d k is the dimension of the feature space, used to scale the similarity score; 3.3 Cross-modal feature fusion: Normalize the similarity scores and apply them to the value matrix V j , generate cross-modal fusion features F cross , the calculation formula is: Through calculation, the cross-modal attention mechanism can effectively fuse information from different modalities and generate rich global feature representations.

5. The ADHD classification diagnosis method based on a multimodal attention fusion network according to claim 1, characterized in that: The step five is specifically include: Global feature construction: connect the fused phenotypic features and imaging features into a unified global representation to form a feature set suitable for classification tasks; Classification module: Use DenseNet as the classifier and input the final fusion features into the DenseNet classification head.

6. The ADHD classification diagnosis method based on a multimodal attention fusion network according to claim 1, characterized in that: The step six specifically includes: Cross entropy loss L CE : Cross entropy loss is used for supervised classification tasks, calculating the difference between the model's predicted output and the true label, helping the model to distinguish categories; Generate adversarial loss L GAN : Generate adversarial loss through adversarial learning mechanism, encouraging the model to capture richer multimodal feature distribution in feature learning; The final adversarial cross entropy loss function is: L=αL CE +βL GAN Among them, α and β are hyperparameters used to balance the contribution of cross entropy loss and generative adversarial loss. By adjusting the two parameters, flexible optimization for different task requirements can be achieved.

Images