Alzheimer's disease early-stage auxiliary diagnosis system based on multi-mode Transform

Abstract

The invention, which belongs to the technical field of medical artificial intelligence, discloses a multi-modal Transform-based early auxiliary diagnosis system for Alzheimer's disease, comprising a data acquisition module, a preprocessing module, a medical priori knowledge guiding module, a hierarchical feature extraction module, a hierarchical diagnosis module, a lightweight reasoning module and a diagnosis output module. According to the system, multi-modal data such as MRI images, cognitive tests and voice features are deeply fused through a hierarchical Transform architecture, a medical priori knowledge guidance model is introduced to pay attention to tiny changes of key brain areas such as hippocampus, ventricle and temporal lobe, and a lightweight reasoning module is adopted to support efficient operation of a basic medical terminal. A visual thermodynamic diagram is generated through the interpretability analysis module to enhance the interpretability of diagnosis, the accuracy of early detection of Alzheimer's disease is improved to 92%, the reasoning speed is improved by 3-5 times, and a key time window is provided for early intervention and treatment.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence technology, and specifically relates to an early auxiliary diagnostic system for Alzheimer's disease (AD) based on multimodal Transformer. This system integrates multi-source data such as MRI images, cognitive tests and speech features, and adopts a hierarchical Transformer deep learning architecture to achieve accurate early detection of cognitive impairment. Background Technology

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease that primarily affects cognitive function and memory in older adults. With the increasing aging of the global population, the incidence of Alzheimer's disease continues to rise, placing a heavy burden on society and families. Early diagnosis and intervention are crucial for slowing disease progression and improving patients' quality of life.

[0003] Alzheimer's disease is a progressive neurodegenerative disease that primarily affects cognitive function and memory in older adults. With the increasing aging of the global population, the incidence of Alzheimer's disease continues to rise, placing a heavy burden on society and families. Early diagnosis and intervention are crucial for slowing disease progression and improving patients' quality of life.

[0004] Currently, the diagnosis of Alzheimer's disease primarily relies on clinical cognitive assessments, neuroimaging examinations, and biomarker testing. Magnetic resonance imaging (MRI) can clearly show structural changes in the brain, such as hippocampal atrophy and enlarged ventricles—typical pathological features. Cognitive tests and speech analysis have also been shown to reflect a patient's cognitive function. However, effectively integrating these multi-source, heterogeneous data to improve diagnostic accuracy remains a technical challenge.

[0005] Chinese patent application CN118044785A discloses a multimodal Alzheimer's disease feature fusion analysis method and terminal. This method uses a 1D CNN model combined with an attention mechanism to extract features from electroencephalogram (EEG) signals, and a ViT model to extract features from MRI images. Then, it fuses the features from the two modalities using tensor fusion, and finally uses a support vector machine for classification and diagnosis. Although this method achieves the fusion of multimodal data, it has the following shortcomings:

[0006] First, the 1D CNN model used in this method is primarily designed for EEG signals and has limited processing capabilities for other modalities such as MRI images, failing to fully capture the complex correlations between different modalities. Second, the tensor fusion method used in this method is a simple feature concatenation strategy, prone to the curse of feature dimensionality, and the fused features may contain a large amount of redundant information, affecting diagnostic efficiency. Third, this method does not consider medical prior knowledge in Alzheimer's disease diagnosis, such as the importance of key brain regions like the hippocampus and temporal lobe, resulting in a lack of specificity in feature extraction. Fourth, this method uses a support vector machine as a classifier, resulting in high model complexity and difficulty in efficient operation on resource-constrained primary healthcare equipment, limiting its application in early screening. Fifth, the diagnostic process of this method lacks interpretability; doctors find it difficult to understand the model's decision-making basis, affecting the credibility of its clinical application.

[0007] Therefore, a new technological solution is needed to more effectively integrate multimodal medical data, make full use of prior medical knowledge to guide feature extraction, and achieve high-precision, high-efficiency, and interpretable early auxiliary diagnosis of Alzheimer's disease. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides an early auxiliary diagnostic system for Alzheimer's disease based on multimodal Transformer. Through a hierarchical Transformer architecture, it achieves deep fusion of multi-source data such as MRI images, cognitive tests, and speech features. It introduces medical prior knowledge to guide the model to focus on subtle changes in key brain regions and designs a lightweight inference module to support early screening applications in primary healthcare institutions. Improvements in early diagnostic accuracy, efficiency, and interpretability were observed in the embodiments.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer includes a data acquisition module, a preprocessing module, a medical prior knowledge guidance module, a hierarchical feature extraction module, a cross-modal Transformer fusion module, a hierarchical diagnosis module, a lightweight inference module, a diagnostic output module, an interpretable analysis module, a continuous learning module, and a remote diagnosis module.

[0011] The data acquisition module is used to acquire the patient's MRI image data, cognitive test data, and speech feature data. In one possible implementation, the MRI image data includes T1-weighted and T2-weighted images, the cognitive test data includes Mini-Mental State Examination (MMSE) scores, Montreal Cognitive Assessment (MoCA) scores, etc., and the speech feature data includes the acoustic features, prosodic features, and semantic features of the speech.

[0012] The preprocessing module is used to preprocess the acquired multimodal data. For MRI image data, if the resolution of the MRI image data meets the first target range (exemplarily, 256×256 to 512×512 pixels), the MRI image data is normalized to a standard resolution (preferably 512×512 pixels). If the noise level of the MRI image data does not meet the second target threshold range (exemplarily, signal-to-noise ratio below 25dB), the MRI image data is denoised, which can be done using Gaussian filtering, median filtering, or deep learning denoising methods. For cognitive test data and speech feature data, standardization is performed to normalize the values ​​to the range of 0 to 1.

[0013] The medical prior knowledge guidance module is used to determine key brain region information based on the pathological features of Alzheimer's disease. Typical pathological features of Alzheimer's disease include hippocampal atrophy, ventricular enlargement, and thinning of the temporal lobe cortex. Based on these pathological features, the medical prior knowledge guidance module determines key brain region information, including the hippocampus, ventricular regions, and temporal lobe regions, and assigns corresponding attention weight parameters to each key brain region. In one possible implementation, the preferred range for the attention weight parameter of the hippocampus region is 0.4-0.6 (exemplarily set to 0.5), the preferred range for the attention weight parameter of the temporal lobe region is 0.2-0.4 (exemplarily set to 0.3), and the preferred range for the attention weight parameter of the ventricular regions is 0.1-0.3 (exemplarily set to 0.2). Alternatively, these weight parameters can also be automatically learned using meta-learning methods, or set as learnable parameters that can be adaptively optimized through gradient descent.

[0014] The hierarchical feature extraction module includes a single-modal feature extraction unit and a cross-modal Transformer fusion unit. The single-modal feature extraction unit extracts features from each preprocessed modal data using a Transformer encoder. Specifically, for MRI image data, the image is divided into several image patches, each typically 16×16 pixels in size. Each patch is then embedded and encoded, converted into a fixed-dimensional vector representation (preferably 768-dimensional). Feature extraction is then performed using a multi-layer Transformer encoder (preferably 12 layers) to obtain the MRI image feature vector. For cognitive test data, various test indicators are numerically encoded, and then cognitive features are extracted using a one-dimensional Transformer encoder to obtain the cognitive test feature vector. For speech feature data, spectral analysis is performed to extract features such as Mel-frequency cepstral coefficients (MFCCs), and then speech features are extracted using a temporal Transformer encoder to obtain the speech feature vector.

[0015] The cross-modal Transformer fusion unit is used to weight the features of corresponding key brain regions in the MRI image feature vector based on key brain region information and attention weight parameters provided by the medical prior knowledge guidance module. Specifically, firstly, based on the spatial location information of the key brain regions, the corresponding feature representations are located from the MRI image feature vectors, and then multiplied by the corresponding attention weight parameters for weighting. The weighted MRI image feature vectors, cognitive test feature vectors, and speech feature vectors are fused through the cross-modal Transformer fusion layer. The cross-modal Transformer fusion layer employs a multi-head cross-attention mechanism to calculate the correlation between features from different modalities, achieving deep fusion and obtaining a fused feature representation. In one possible implementation, the dimension of the fused feature representation is preferably 1024.

[0016] The hierarchical diagnostic module is used to perform hierarchical judgment on the fused feature representation. First, the confidence level of the fused feature representation is calculated. This confidence level can be based on the variance of the feature vector, entropy value, or through an additional confidence estimation network. If the confidence level of the fused feature representation meets the third target threshold range (for example, confidence level greater than 0.8), the fused feature representation is input into the diagnostic classifier for classification to obtain a diagnostic result. The diagnostic classifier can be a multilayer perceptron (MLP) or a lightweight classification network. If the confidence level of the fused feature representation does not meet the third target threshold range, feature enhancement processing is performed on the fused feature representation. For example, the discriminative power of the features is enhanced through contrastive learning or self-supervised learning methods, and then it is re-input into the diagnostic classifier.

[0017] The lightweight inference module is used to invoke the lightweight Transformer model for inference after the diagnostic classifier outputs a diagnostic result. If the computing resources of the currently running device are detected to be below the fourth target threshold (exemplarily, GPU memory less than 4GB or CPU computing power less than 10 GFLOPS), the lightweight Transformer model is invoked. The lightweight Transformer model extracts knowledge from the standard Transformer model (teacher model) through knowledge distillation, employing a student-teacher architecture. The preferred distillation temperature parameter is 2.0-4.0 (exemplarily set to 3.0), and the preferred loss weight coefficient is 0.3-0.7 (the ratio of hard label loss to soft label loss, exemplary set to 0.5:0.5). Alternatively, feature distillation can be used for loss, performing feature alignment in the intermediate layer, or combining KL divergence and mean squared error loss. In the embodiments, it was observed that the lightweight model has approximately 20%-30% of the parameters of the standard model, and the inference speed is improved by about 3-5 times, enabling the system to run efficiently on ordinary medical terminals.

[0018] The diagnostic output module outputs diagnostic results, including health status, mild cognitive impairment, and Alzheimer's disease status. In addition to outputting the diagnostic category, the module also outputs a confidence score for the diagnostic result. If the confidence score meets the 13th target threshold range (for example, a confidence score greater than 0.9), the diagnostic result is marked as high confidence; if the confidence score does not meet the 13th target threshold range, the diagnostic result is marked as low confidence, and further investigation is recommended for the patient. The diagnostic output module also generates a detailed diagnostic report, including abnormal indicators for each modality of data, assessment of the severity of lesions in key brain regions, and diagnostic recommendations, providing decision support for clinicians.

[0019] The interpretability analysis module generates a visual heatmap based on the attention weight matrix of the Transformer model, identifying the brain regions and feature dimensions that contribute most to the diagnostic results. Through this visual heatmap, physicians can intuitively understand the model's decision-making process, such as whether the model makes a diagnostic judgment based on significant hippocampal atrophy or temporal lobe metabolic abnormalities, thereby enhancing the system's interpretability and clinical credibility.

[0020] The continuous learning module is used to incrementally update the model after acquiring new diagnostic data. If the distribution characteristics of the new data do not conform to the twelfth objective range (for example, the data distribution deviates significantly from the training set, MMD>0.1), the Transformer model is incrementally trained to update the model parameters to adapt to the new data distribution. If the distribution characteristics of the new data conform to the twelfth objective range, the model parameters are kept unchanged to avoid unnecessary training overhead.

[0021] The data security module encrypts and stores patient medical data during transmission. It employs federated learning during model training, ensuring data remains within the local healthcare facility; only model parameter updates are transmitted, thus protecting patient privacy. The remote diagnosis module allows primary healthcare institutions to upload patient data to a cloud server via network connection. A high-performance Transformer model in the cloud performs the diagnosis, and the results are then returned to the primary healthcare institution, enabling resource-constrained institutions to access high-quality diagnostic services.

[0022] The beneficial effects of this invention are as follows:

[0023] This invention employs a hierarchical Transformer architecture, first extracting deep features from single-modal data, and then capturing the complex correlations between different biomarkers through a cross-modal fusion layer. Compared to the simple feature splicing strategies of existing technologies, this approach can more effectively integrate multimodal information, avoiding the curse of feature dimensionality and redundant information problems. In tests on the ADNI dataset in this embodiment, the early detection accuracy of the system for Alzheimer's disease reached 92.3%, an improvement of approximately 8 percentage points compared to the method in comparative document CN118044785A (approximately 84.5%), and an improvement of approximately 12-17 percentage points compared to the accuracy achieved by clinicians in the early diagnosis stage (approximately 75%-80%). Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention;

[0025] Figure 2 This is a schematic diagram of the hierarchical feature extraction module of the present invention;

[0026] Figure 3 This is a schematic diagram of the medical prior knowledge guidance mechanism of the present invention;

[0027] Figure 4 This is a flowchart illustrating the workflow of the lightweight inference module of the present invention.

[0028] Figure 5 This is a schematic diagram of the diagnostic process of the system of the present invention. Detailed Implementation

[0029] Please refer to Figures 1-5 The present invention will now be described in further detail with reference to the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0030] like Figure 1 As shown, the Alzheimer's disease early auxiliary diagnosis system based on multimodal Transformer of the present invention includes a data acquisition module 1, a first preprocessing module 2, a second preprocessing module 3, a medical prior knowledge guidance module 4, a hierarchical feature extraction module 5, a cross-modal Transformer fusion module 6, a hierarchical diagnosis module 7, a lightweight inference module 8, a diagnosis output module 9, an interpretability analysis module 10, a continuous learning module 11, and a remote diagnosis module 12.

[0031] Data acquisition module 1 acquires the patient's MRI image data, cognitive test data, and speech feature data from medical devices. In this embodiment, the MRI image data is T1-weighted images acquired by a 3T MRI scanner, with an image resolution of 512×512 pixels and a slice thickness of 1mm. Cognitive test data includes indicators such as the MMSE score (out of 30), MoCA score (out of 30), and Clinical Dementia Rating Scale (CDR). Speech feature data is obtained by having the patient read a standard text and recording it, then extracting acoustic features such as MFCC features, fundamental frequency, and formants, as well as prosodic features such as speech rate and pauses.

[0032] The first preprocessing module 2 preprocesses the MRI image data. First, it checks the resolution of the MRI image data. If the resolution meets the first target range (256×256 to 512×512 pixels in this embodiment), the MRI image data is normalized to 512×512 pixels. If the original image resolution is lower than 256×256 or higher than 512×512, it is adjusted to the standard resolution using interpolation or downsampling methods. Next, the noise level of the image is evaluated, and the signal-to-noise ratio (SNR) is calculated. If the SNR does not meet the second target threshold range (SNR lower than 25dB in this embodiment), the MRI image data is denoised. This embodiment uses an adaptive denoising filter, combined with Gaussian filtering (kernel size 5×5, standard deviation σ=1.5) and non-local mean denoising methods, effectively removing image noise while preserving edge details.

[0033] The first preprocessing module 2 also performs slice selection on the MRI image data. For three-dimensional MRI data, the information entropy value of each slice is calculated. :

[0034] ,

[0035] in, grayscale value The probability of appearing in an image. This represents the grayscale value of a pixel, ranging from 0 to 255. If the information entropy value of an MRI image slice conforms to the fifth target range (in this embodiment, it is...) If the information entropy value does not meet the fifth target range, the slice is retained; if the information entropy value does not meet the target range, the slice is discarded. Information entropy filtering allows for the selection of slices containing rich brain structural information and the removal of slices with less information, improving subsequent processing efficiency. In this embodiment, the 32 slices with the highest information entropy are selected as key slices.

[0036] The first preprocessing module 2 further enhances the contrast of the retained slices, particularly in the hippocampus, ventricles, and temporal lobe regions. This embodiment employs Adaptive Histogram Equalization (CLAHE), dividing the image into an 8×8 grid and performing histogram equalization on each grid individually, limiting the contrast enhancement amplitude to 3.0 to avoid noise amplification due to over-enhancement. After contrast enhancement, the boundaries of key brain regions such as the hippocampus become clearer, facilitating subsequent feature extraction.

[0037] The second preprocessing module 3 standardizes the cognitive test data and speech feature data. For the cognitive test data, the Z-score standardization method is used:

[0038] ,

[0039] in, For the original test score, This is the mean of the score in the training set. Standard deviation, This is the standardized score. The standardized score has a mean of 0 and a standard deviation of 1, eliminating the influence of different test scale units.

[0040] For the speech feature data, 13-dimensional MFCC features were first extracted at a sampling rate of 16kHz, a frame length of 25ms, a frame shift of 10ms, and 26 Mel filter banks. Then, the first and second order differences were calculated to obtain a 39-dimensional acoustic feature vector. Next, these 39-dimensional features were normalized, with the value of each dimension normalized to the range of 0 to 1.

[0041] ,

[0042] in, These are the original eigenvalues. and These are the minimum and maximum values ​​of this feature dimension in the training set, respectively. These are the normalized eigenvalues.

[0043] like Figure 3 As shown, the medical prior knowledge guidance module 4 determines key brain region information based on the pathological characteristics of Alzheimer's disease and assigns attention weight parameters to each key brain region.

[0044] Typical pathological features of Alzheimer's disease include: hippocampal atrophy (reduced by 15% to 25% compared to normal elderly individuals), ventricular enlargement (increased lateral ventricular volume by 20% to 30%), thinning of the temporal lobe cortex (reduction of 10% to 20% in the thickness of the medial temporal lobe cortex), and decreased frontal lobe metabolism. Based on these pathological features, the medical prior knowledge guidance module 4 identified three key brain regions: the hippocampus, the ventricles, and the temporal lobe.

[0045] In this embodiment, based on existing medical research and clinical data statistics, initial attention weight parameters are assigned to these three key brain regions. The hippocampus is one of the earliest affected regions in Alzheimer's disease, and its volume shrinkage is highly correlated with cognitive decline. Therefore, the preferred range for the attention weight parameter of the hippocampus is 0.4-0.6, and in this example, it is set to... The temporal lobe region includes the medial temporal cortex and the lateral temporal cortex. Structural changes and functional metabolic abnormalities in this region are important biomarkers of Alzheimer's disease. The preferred range for the attention weight parameter in the temporal lobe region is 0.2-0.4; in this example, it is set to... Ventricular enlargement reflects the overall degree of brain tissue atrophy, but its diagnostic specificity is slightly lower compared to changes in the hippocampus and temporal lobe. Therefore, the preferred range for the attention weighting parameter in the ventricular region is 0.1-0.3, which is set to [value missing] in this example. These three weight parameters satisfy the normalization condition: .

[0046] As an alternative implementation, these attention weight parameters can also be automatically learned from multiple patient datasets using meta-learning methods, or set as learnable parameters in the network and adaptively optimized through backpropagation and gradient descent. In the learnable parameter scheme, the weights are initialized to the example values ​​described above and updated according to the gradient of the loss function during training, eventually converging to the optimal configuration.

[0047] The medical prior knowledge guidance module 4 also dynamically adjusts the attention weight parameters based on the patient's age. Research shows that the pathological characteristics of Alzheimer's disease differ among patients of different ages. If the patient's age falls within the sixth target range (in this example, early-aged patients aged 65 to 75), the attention weight parameter for the hippocampus region is increased by 0.1, setting it to... Accordingly, the attention weight parameter for the ventricular region is reduced by 0.1 and set to [value missing]. The weights for the temporal lobe region are kept constant. This is because hippocampal atrophy is often the most significant pathological change in early-aged patients, while ventricular enlargement is relatively less noticeable. For elderly patients (over 75 years of age), the initial weight parameters are kept unchanged.

[0048] The medical prior knowledge guidance module 4 also dynamically adjusts the attention weight parameters based on the patient's cognitive test score. If the cognitive test score falls within the seventh target range (in this embodiment, an MMSE score between 20 and 24, indicating mild cognitive impairment), the attention weight parameter for the hippocampus region is increased and set to... This is because hippocampal dysfunction is the primary pathological mechanism in the mild cognitive impairment stage. If the cognitive test score does not meet the seventh target range (for example, an MMSE score below 20 indicates moderate to severe cognitive impairment), the attention weight parameter for the temporal lobe region is increased and set to... Reduce the weight of the hippocampus to This is because as the disease progresses, the lesions in the temporal lobe cortex gradually worsen, becoming a more important diagnostic feature.

[0049] Guided by this prior medical knowledge, the model can focus on the brain regions most relevant to the patient's condition during feature extraction. In the implementation examples, an improvement in diagnostic specificity and accuracy was observed, which is an important innovation of this invention compared to the prior art.

[0050] like Figure 2 As shown, the hierarchical feature extraction module 5 includes a single-modal feature extraction unit and a cross-modal Transformer fusion unit 6. The single-modal feature extraction unit extracts features from each of the preprocessed modal data separately, while the cross-modal Transformer fusion unit 6 fuses the extracted features.

[0051] The single-modal feature extraction unit includes an MRI feature extraction subunit, a cognitive test feature extraction subunit, and a speech feature extraction subunit.

[0052] The MRI feature extraction subunit adopts a Vision Transformer-based architecture. First, the 512×512 pixel MRI image is divided into several image patches. In this embodiment, each patch is 16×16 pixels in size; therefore, a 512×512 image is divided into... Each patch is embedded and encoded using a linear projection layer (fully connected layer), transforming it into a 768-dimensional vector representation. Specifically, a 16×16 pixel patch is flattened into a 256-dimensional vector, and then projected through a fully connected layer with 256 inputs and 768 outputs.

[0053] ,

[0054] in, For the first The pixel vector of each patch The embedding matrix has dimensions of . , The position encoding matrix has a dimension of . , The embedded encoded feature sequence has a dimension of , This is the total number of patches in this embodiment. .

[0055] The location encoding uses a learnable location embedding method, assigning a location vector to each patch, enabling the model to perceive the spatial relationship between different patches in the image.

[0056] The embedded and encoded feature sequence is input into a multi-layer Transformer encoder for feature extraction. This embodiment uses a 12-layer Transformer encoder, each layer comprising two sub-modules: a multi-head self-attention mechanism (MHSA) and a feed-forward network (FFN). Layer normalization (LN) and residual connections are applied after each sub-module.

[0057] ,

[0058] ,

[0059] in, For the first The output features of the layer For the first The output features of the layer The layer index ranges from 1 to 12. MHSA and FFN represent multi-head self-attention mechanism and feedforward neural network, respectively. It is an intermediate feature.

[0060] The multi-head self-attention mechanism projects the input features into three matrices: query, key, and value, and then calculates the attention score.

[0061] ,

[0062] in, , , These are query, key, and value matrices, respectively. In this embodiment, the dimension of the key vector is... The softmax function is used to normalize the attention score.

[0063] This embodiment uses 8 attention heads (heads=8), each head having a dimension of 768 / 8=96. The multi-head mechanism allows the model to learn different aspects of features in different representation subspaces, improving feature extraction capabilities.

[0064] The feedforward neural network is a two-layer fully connected network, using the GELU activation function in between:

[0065] ,

[0066] in, and Here is the weight matrix, with dimensions as follows: and , and For bias vectors, The input features are used. The hidden layer dimension of the feedforward network is 3072, which is 4 times the embedding dimension of 768. This is the standard setting for the Transformer architecture.

[0067] After passing through 12 Transformer encoder layers, the final feature sequence is obtained. , dimension To obtain fixed-dimensional MRI image feature vectors, the following steps are taken: Perform global average pooling:

[0068] ,

[0069] in, For the first The feature vector corresponding to each patch This is an MRI image feature vector with a dimension of 768.

[0070] The cognitive test feature extraction subunit extracts features from the cognitive test data. In this embodiment, the cognitive test data includes eight indicators such as MMSE score, MoCA score, and CDR score, forming an 8-dimensional vector. First, this 8-dimensional vector is projected to a 768-dimensional vector through a fully connected layer, and then features are extracted through a 3-layer Transformer encoder. Since the cognitive test data has a low dimensionality, a 12-layer deep network is not required. Finally, global average pooling is performed on the encoder output to obtain the cognitive test feature vector. The dimension is 768.

[0071] The speech feature extraction subunit extracts features from the speech feature data. In this embodiment, the speech feature data is a 39-dimensional MFCC feature sequence with a sequence length of [length missing]. The number of frames (depending on the speech duration, typically 100 to 300 frames) is calculated. First, the 39-dimensional MFCC features are projected into a 768-dimensional vector through a fully connected layer. Then, a 6-layer temporal Transformer encoder is used for feature extraction. The temporal Transformer encoder is similar to the Vision Transformer, but its input is a time series sequence instead of an image patch sequence. After the encoder output, global average pooling is performed on the time dimension to obtain the speech feature vector. The dimension is 768.

[0072] The single-modal feature extraction unit converts the raw data of the three modalities into feature vector representations of a unified dimension (768 dimensions), laying the foundation for subsequent cross-modal fusion.

[0073] The cross-modal Transformer fusion unit 6 fuses the MRI image feature vector, cognitive test feature vector, and speech feature vector output by the single-modal feature extraction unit to obtain a fused feature representation.

[0074] The cross-modal Transformer fusion unit 6 first weights the features of the corresponding key brain regions in the MRI image feature vector based on the key brain region information and attention weight parameters provided by the medical prior knowledge guidance module 4. The specific implementation method is as follows:

[0075] During MRI feature extraction, the feature vector corresponding to each patch is retained. , dimension Based on the spatial location of the patches in the image, it is determined which patches correspond to the hippocampus, ventricles, and temporal lobe regions. In this embodiment, the brain region to which each patch belongs can be determined using pre-annotated brain region segmentation templates or automatic segmentation algorithms. Let the hippocampus region contain a set of patches. The ventricular region contains a set of patches. The temporal lobe region contains a set of patches. .

[0076] For patches belonging to key brain regions, a weighted average is applied by multiplying by the corresponding attention weight parameter:

[0077] ,

[0078] in, , , The attention weight parameters for the hippocampus, temporal lobe, and ventricles are provided by the medical prior knowledge guidance module 4. This is the weighted eigenvector.

[0079] After weighting, global average pooling is performed on the feature vectors of all patches to obtain the weighted MRI image feature vectors:

[0080] ,

[0081] This weighting method gives greater weight to features of key brain regions in the feature vector, making the model pay more attention to changes in these regions and improving the specificity of the diagnosis.

[0082] Next, the cross-modal Transformer fusion unit 6 fuses the weighted MRI image feature vector, cognitive test feature vector, and speech feature vector through the cross-modal Transformer fusion layer.

[0083] The cross-modal Transformer fusion layer employs a multi-head cross-attention mechanism. It concatenates the feature vectors from the three modalities into a single feature sequence.

[0084] ,

[0085] in, The concatenated feature sequence has a dimension of . .

[0086] Then, the features are fused using a cross-modal Transformer encoder. The cross-modal encoder consists of four layers, each including a multi-head cross-attention mechanism and a feedforward network. Unlike the single-modal Transformer, the attention mechanism of the cross-modal Transformer allows features from different modalities to interact with each other. When calculating the attention score, features from one modality can be used as the query, and features from other modalities can be used as the key and value.

[0087] ,

[0088] in, For modality The query matrix, and For modality The key and value matrix, and It can be any two of the three modalities: MRI, Cog, or Speech.

[0089] Through multi-head cross-attention mechanisms, the model can learn the correlations between different modalities. For example, hippocampal atrophy shown on MRI images may be highly correlated with a decline in memory scores on cognitive tests, and slowed speech rate in speech features may be correlated with reduced frontal lobe metabolism on MRI images. Cross-modal attention mechanisms can capture these cross-modal correlation patterns, achieving deep fusion.

[0090] After passing through 4 layers of cross-modal Transformer encoders, the fused feature sequence is obtained. , dimension .right Global average pooling is performed to obtain the final fused feature representation:

[0091] ,

[0092] in, For the first The fusion feature vector corresponding to each modality The fusion feature representation has a dimension of 768.

[0093] In one possible implementation, the cross-modal Transformer fusion unit 6 further calculates cross-modal attention scores between different modal features to evaluate the contribution of each modality to the diagnostic results. If the attention score of a certain modality with other modalities falls within the eighth target threshold range (exemplarily, the attention score is greater than 0.6), it indicates that the modal feature is highly correlated with other modalities, and the modal feature is enhanced by multiplying it by an enhancement coefficient (preferably 1.2). If the attention score of a certain modality with other modalities does not fall within the eighth target threshold range (exemplarily, the attention score is less than 0.3), it indicates that the modal feature is weakly correlated with other modalities and may contain noise or anomalous information, and the modal feature is suppressed by multiplying it by a suppression coefficient (preferably 0.8). This adaptive modality weighting strategy further improves the quality of the fused features.

[0094] The hierarchical diagnosis module 7 performs hierarchical judgment on the fused feature representation and determines the diagnosis strategy based on the credibility of the features.

[0095] First, the credibility of the fused feature representation is calculated. This embodiment employs a credibility estimation method based on feature variance. Specifically, for the fused feature vector... Calculate the variance for each dimension:

[0096] ,

[0097] in, For the fused feature vector One dimension, The mean of all dimensions. For feature dimension, Variance. Larger variance indicates a more dispersed feature distribution and lower reliability; smaller variance indicates a more concentrated feature distribution and higher reliability.

[0098] Credibility is defined as the reciprocal of the variance after normalization:

[0099] ,

[0100] in, For reliability, the value ranges from 0 to 1. To adjust the parameters, this embodiment is set to... .

[0101] If the credibility of the fused feature representation meets the third target threshold range (in this embodiment, it is...) If the feature quality is high, it can be directly input into the diagnostic classifier for classification. The diagnostic classifier is a three-layer fully connected network with a structure of 768→256→64→3. The last layer outputs the probability distributions of the three categories, corresponding to healthy state, mild cognitive impairment state, and Alzheimer's disease state, respectively. The classifier uses the softmax function for normalization.

[0102] ,

[0103] in, For category The predicted probability, and The weights and biases of the classifier, For category indexing, The category with the highest probability is selected as the diagnostic result.

[0104] If the credibility of the fused feature representation does not meet the third target threshold range (i.e.) If the sample does not meet the criteria for feature quality, then feature enhancement is considered necessary. Feature enhancement can employ contrastive learning methods, comparing the fused features with both positive and negative samples to improve their discriminative power. Specifically, positive sample features of the same category as the current sample are sampled from the training set. and features of negative samples of different categories Calculate the contrast loss:

[0105] ,

[0106] in, The cosine similarity function is used. For temperature parameters, in this embodiment , To represent the number of negative samples, in this embodiment... By minimizing the contrastive loss, the fused features can be adjusted to be more similar to positive samples and less similar to negative samples, thereby enhancing the discriminative power of the features. The enhanced features are then re-input into the diagnostic classifier for classification.

[0107] In another possible implementation, the hierarchical diagnostic module 7 also performs uncertainty estimation. Uncertainty estimation can be achieved using the Monte Carlo Dropout method. During inference, the Dropout layer remains active, and multiple forward propagations (preferably 10 times) are performed on the same sample to obtain multiple prediction results. The variance of the prediction results is then calculated as the uncertainty estimate. If the uncertainty estimate falls within the ninth target range (in this embodiment, variance less than 0.05), the model's prediction is considered relatively stable, and the diagnostic result is directly output. If the uncertainty estimate does not fall within the ninth target range (i.e., variance greater than 0.05), a multi-model ensemble mechanism is initiated.

[0108] The multi-model ensemble mechanism invokes multiple Transformer models with different configurations for diagnosis. These models can have different numbers of layers, different numbers of attention heads, or different training strategies, resulting in model diversity. This embodiment uses three different models: Model 1 is a standard 12-layer Transformer, Model 2 is a lightweight 6-layer Transformer, and Model 3 is a 12-layer Transformer trained with different initialization seeds. The fused features are input into these three models respectively, resulting in three predictions, which are then voted on. If two or more of the three models predict the same category, that category becomes the final diagnosis; if the predictions from the three models are different, the category with the highest average confidence is selected as the final diagnosis. Multi-model ensemble can improve the robustness of diagnosis and reduce the risk of misjudgment by a single model.

[0109] like Figure 4 As shown, the lightweight inference module 8 is used for efficient inference on resource-constrained devices. The lightweight inference module 8 first detects the computing resources of the currently running device. Computing resources include metrics such as GPU memory and CPU computing power. If the detected computing resources of the currently running device are below the fourth target threshold (in this embodiment, GPU memory is less than 4GB or CPU computing power is less than 10 GFLOPS), then the lightweight Transformer model is invoked for inference.

[0110] The lightweight Transformer model extracts knowledge from the standard Transformer model (teacher model) through knowledge distillation. Knowledge distillation employs a student-teacher architecture, where the teacher model is a standard 12-layer Transformer, and the student model is a lightweight 4-layer Transformer. The core of the distillation process is to enable the student model to learn the output distribution and feature representation of the teacher model.

[0111] The lightweight Transformer model in this embodiment reduces the standard model's 12-layer encoder to 4 layers, the number of attention heads from 8 to 4, and the hidden layer dimension of the feedforward network from 3072 to 1024. Through these prunings, the lightweight model has approximately 25% of the parameters of the standard model. In this embodiment, it was observed that through knowledge distillation training, the model accuracy decreased by only about 2%, while the inference speed increased by about 4 times.

[0112] The loss function for knowledge distillation consists of two parts: hard label loss and soft label loss. Hard label loss is the cross-entropy loss between the student model's predictions and the true labels.

[0113] ,

[0114] in, One-hot encoding of the real label. This represents the predicted probability for the student model.

[0115] The soft-label loss is the KL divergence between the predictions of the student model and the predictions of the teacher model.

[0116] ,

[0117] in, and The teacher model and the student model were respectively located at temperature. Softening probability distribution under temperature parameters The preferred range is 2.0-4.0, and in this example it is set to... The softening probability distribution is achieved by introducing a temperature parameter into the softmax function:

[0118] ,

[0119] in, This is logits (the output of the last layer of the classifier, without softmax normalization).

[0120] The total loss is the weighted sum of the hard-label loss and the soft-label loss:

[0121]

[0122] in, This is the weighting coefficient, preferably in the range of 0.3-0.7, and is set to [value missing] in this example. .

[0123] As an alternative, feature distillation can also be used to align features in the intermediate layers of the Transformer, achieving knowledge transfer by minimizing the mean squared error (MSE) of features in the intermediate layers of the student and teacher models. Furthermore, a hybrid distillation strategy can be formed by combining KL divergence and mean squared error loss to further improve the performance of lightweight models.

[0124] By minimizing the distillation loss, the student model (lightweight Transformer model) can learn from the teacher model (standard Transformer model), maintaining high diagnostic performance with a significant reduction in the number of parameters.

[0125] In one possible implementation, the lightweight inference module 8 further selects different optimization strategies based on the priority of the diagnostic task. If the priority of the diagnostic task falls within the tenth target range (exemplarily marked as a high-priority task), a quantization compression strategy is used to compress the parameters of the Transformer model. Quantization compression compresses the model's 32-bit floating-point parameters into 8-bit integers, reducing the model size to 1 / 4 of its original size and further improving inference speed. If the priority of the diagnostic task does not fall within the tenth target range (exemplarily marked as a normal-priority task), a dynamic pruning strategy is used to prune the number of layers in the Transformer model, dynamically deciding whether to use a 4-layer, 6-layer, or 8-layer model based on computing resources.

[0126] During inference, if the inference time exceeds the eleventh target threshold (2 seconds in this embodiment), an early stopping mechanism is activated to output the diagnostic results of intermediate layers ahead of schedule. For example, if the standard model has 12 layers, the output of each layer is monitored during inference. If the output of the 8th layer can determine the diagnostic category with a high confidence level (preferably greater than 0.9), the inference is terminated early, and the result of the 8th layer is output, eliminating the need to calculate the subsequent four layers and thus saving computation time. The early stopping mechanism further improves inference efficiency while ensuring diagnostic accuracy.

[0127] The system of the present invention also includes an interpretability analysis module 10. The interpretability analysis module 10 generates a visual heatmap based on the attention weight matrix of the Transformer model, identifying the brain regions and feature dimensions that contribute the most to the diagnostic results.

[0128] In the Transformer model, the attention weight matrix records the degree of attention the model pays to different locations when calculating features. For MRI image feature extraction, the attention weight matrix... The dimension is This represents the attention relationships among the 1024 patches. By analyzing the attention weights, it is possible to determine which patches contribute the most to the final diagnostic results.

[0129] The specific implementation is as follows: After the diagnostic classifier outputs the diagnostic result, backtrack to the last layer of the Transformer encoder and extract the attention weight matrix. For each patch Calculate the cumulative value of its attention weights:

[0130] ,

[0131] in, For Patch For Patch Attention weights , For Patch The cumulative attention value. The higher the cumulative attention value, the greater the contribution of the patch to the diagnostic results.

[0132] The accumulated attention values ​​are mapped back to the spatial location of the original image to generate an attention heatmap. In the heatmap, darker areas (closer to red) indicate more concentrated attention, meaning a higher contribution to the diagnostic results; lighter areas (closer to blue) indicate weaker attention, meaning a lower contribution to the diagnostic results.

[0133] By using visualized heatmaps, doctors can intuitively see which brain regions the model prioritizes when making a diagnosis. For example, if the heatmap shows the model highly focuses on the hippocampus, it indicates the model's diagnosis of Alzheimer's disease is based on hippocampal atrophy; if the heatmap shows the model highly focuses on the temporal lobe, it indicates the model's diagnosis is based on temporal lobe lesions. This interpretability analysis enhances doctors' trust in the model's decisions and also helps identify potential problems with the model.

[0134] In one possible implementation, the interpretability analysis module 10 also generates a contribution analysis graph of the feature dimensions. The fused feature vector contains 768 dimensions, each representing a certain abstract feature representation. The contribution of each dimension to the diagnostic result can be calculated using gradient analysis methods (such as Integrated Gradients). Dimensions with higher contributions are identified, and attempts are made to interpret the biological significance represented by these dimensions; for example, some dimensions may correspond to hippocampal volume features, while others may correspond to temporal lobe metabolic features, etc.

[0135] The diagnostic output module 9 outputs the diagnostic results and related information. The diagnostic results include three categories: healthy status, mild cognitive impairment, and Alzheimer's disease status. In addition to outputting the diagnostic category, the diagnostic output module 9 also outputs the confidence score of the diagnostic result. The confidence score is determined by the maximum probability value of the classifier.

[0136] ,

[0137] in, The confidence score ranges from 0 to 1.

[0138] If the confidence score meets the thirteenth target threshold range (in this example, ...), If the confidence score does not meet the thirteenth target threshold (i.e., ...), the diagnosis is marked as highly reliable, and doctors can refer to the diagnosis with greater confidence. If the result is negative, the diagnosis is marked as low confidence, and further examinations are recommended for the patient, such as PET scans and cerebrospinal fluid biomarker testing, to obtain a more accurate diagnosis.

[0139] The diagnostic output module 9 also generates a detailed diagnostic report. The diagnostic report includes the following:

[0140] Patient basic information: name, age, gender, medical history, etc.

[0141] Abnormal indicators in each modality of data: For example, MRI images showed a hippocampal volume of 2.8 cm³, which is 20% lower than the average of 3.5 cm³ for healthy individuals of the same age; the MMSE cognitive test score was 22, indicating mild cognitive impairment; speech characteristics showed a speech rate of 110 words per minute, which is lower than the normal speech rate of 150 words per minute.

[0142] Assessment of the severity of lesions in key brain regions: Based on the analysis of medical prior knowledge-guided module 4, the attention weight of the hippocampus region was 0.5, indicating significant atrophy; the attention weight of the temporal lobe region was 0.3, indicating moderate cortical thinning; and the attention weight of the ventricles region was 0.2, indicating mild enlargement.

[0143] Diagnosis and confidence level: The diagnosis is mild cognitive impairment with a confidence level of 0.92, which is considered a high-confidence diagnosis.

[0144] Diagnostic recommendations: It is recommended that the patient have regular follow-up examinations (preferably an MRI scan and cognitive test every 6 months) to monitor changes in the condition; it is recommended that the patient undergo cognitive training and lifestyle interventions (e.g., increasing social activities, engaging in aerobic exercise, and maintaining a healthy diet); if the condition progresses, it is recommended to consider drug treatment.

[0145] Interpretability analysis results: An attention heatmap is attached, showing the brain regions that the model focuses on, to help doctors understand the basis of diagnosis.

[0146] Diagnostic reports provide doctors with comprehensive decision support and patients with a clear description of their condition.

[0147] like Figure 5 As shown, the complete diagnostic process of the system of the present invention is as follows:

[0148] Step 1: The patient goes to a medical institution for examination, and the data acquisition module 1 acquires the patient's MRI image data, cognitive test data, and speech feature data.

[0149] Step 2: The preprocessing modules (first preprocessing module 2 and second preprocessing module 3) preprocess the acquired data, including denoising, normalization, slice selection and contrast enhancement of MRI images, as well as standardization of cognitive test data and speech feature data.

[0150] Step 3: Medical Prior Knowledge Guidance Module 4 determines the attention weight parameters of key brain regions based on the patient's age and cognitive test scores.

[0151] Step 4: The single-modal feature extraction unit of the hierarchical feature extraction module 5 extracts features from each modality of data to obtain MRI image feature vectors, cognitive test feature vectors, and speech feature vectors.

[0152] Step 5: The cross-modal Transformer fusion module 6 weights the MRI features based on prior medical knowledge, and then performs deep fusion of the three modal features through the cross-modal Transformer fusion layer to obtain the fused feature representation.

[0153] Step 6: The hierarchical diagnosis module 7 calculates the credibility of the fused features. If the credibility is high, it is directly input into the diagnosis classifier for classification; if the credibility is low, feature enhancement processing is performed first, and then it is input into the diagnosis classifier.

[0154] Step 7: If the computing resources of the running device are limited, the lightweight inference module 8 calls the lightweight Transformer model for inference to improve inference efficiency.

[0155] Step 8: The diagnostic classifier outputs diagnostic results and confidence scores, and the interpretability analysis module 10 generates an attention heatmap.

[0156] Step 9: Diagnostic output module 9 outputs diagnostic results, confidence scores, and detailed diagnostic reports to doctors and patients.

[0157] The entire diagnostic process, from data acquisition to result output, is typically completed within 5 minutes (excluding MRI scan time), enabling rapid, accurate, and interpretable early diagnosis.

[0158] To verify the performance of the system of this invention, experiments were conducted on the publicly available Alzheimer's disease dataset ADNI (Alzheimer's Disease Neuroimaging Initiative). The ADNI dataset contains data from more than 1,200 patients from multiple medical centers, including healthy controls, mild cognitive impairment groups, and Alzheimer's disease groups.

[0159] Experimental setup: The dataset was divided into a training set (840 cases), a validation set (180 cases), and a test set (180 cases) in a 7:1.5:1.5 ratio. Stratified random sampling was used to ensure consistent proportions for each class. The model was trained using the Adam optimizer with an initial learning rate of 1×10⁻⁶. -4 A cosine annealing learning rate scheduling strategy is adopted, with a minimum learning rate of 1×10. -6 The batch size was 32, and the training epochs were 100. The early stopping criterion was set to stop training if the validation set loss did not improve for 10 consecutive epochs. The loss function used was weighted cross-entropy loss, with class weights set to [1.0, 1.2, 1.5] based on the inverse of the training set sample distribution. Data augmentation strategies included: random rotation of MRI slices (-10° to 10°), random translation (-5% to 5%), and random contrast adjustment (0.9x to 1.1x). The random seed was set to 42 to ensure experimental reproducibility. All experiments were performed on an NVIDIA RTX 3080 GPU using the PyTorch 1.10 framework.

[0160] Evaluation metrics: Accuracy, Precision, Recall, F1 score, and Area Under the Receiver Operating Characteristic (AUC) were used as evaluation metrics. Paired t-tests were used to test statistical significance, with a confidence interval of 95% (p < 0.05 was considered statistically significant).

[0161] Experimental results: The performance of the system of the present invention on the test set is as follows:

[0162] Overall accuracy: 92.3% (95% confidence interval: 89.7%-94.9%);

[0163] Precision of health status detection: 94.5%, recall: 93.2%, F1 score: 93.8%;

[0164] Precision rate for mild cognitive impairment detection: 89.7%, recall rate: 90.8%, F1 score: 90.2%;

[0165] The accuracy rate of Alzheimer's disease detection was 92.8%, the recall rate was 91.5%, and the F1 score was 92.1%.

[0166] AUC: 0.95 (95% confidence interval: 0.93-0.97).

[0167] Comparative experiment: The system of the present invention is compared with the method in prior art document CN118044785A and other existing technologies:

[0168] In the embodiments, it was observed that the system of the present invention, compared with the method in prior art CN118044785A (accuracy approximately 84.5%, AUC approximately 0.88), achieved an accuracy improvement of approximately 8 percentage points and an AUC improvement of 0.07. Paired t-tests showed that the difference was statistically significant (p<0.01). Compared with the average accuracy of clinicians in the early diagnostic stage (approximately 75%-80%), the system of the present invention improved by approximately 12-17 percentage points, providing more reliable diagnostic support for early intervention.

[0169] Ablation experiments: Ablation experiments were conducted to verify the effectiveness of each module.

[0170] In the embodiments, it was observed that the medical prior knowledge guidance module and the cross-modal Transformer fusion module significantly contributed to system performance. Removing the medical prior knowledge guidance module resulted in a 3.2 percentage point decrease in accuracy; removing the cross-modal fusion module resulted in a 4.7 percentage point decrease in accuracy. The interpretability analysis module did not affect diagnostic accuracy but significantly improved the system's interpretability. The lightweight inference module achieved approximately a 4x increase in inference speed while only decreasing accuracy by 0.2 percentage points, achieving a good balance between accuracy and efficiency.

[0171] Efficiency Assessment: On a standard GPU (NVIDIA RTX 3080), the inference time for a single sample using the system of this invention is approximately 0.8 seconds; with the lightweight model, the inference time on a typical CPU (Intel i7-10700) is approximately 3.2 seconds, meeting the requirements for real-time diagnostics. In comparison, the method in the prior art document CN118044785A has an inference time of approximately 8.5 seconds on the same CPU, and the efficiency improvement of the system of this invention was observed to be approximately 2.7 times in the examples.

[0172] Sensitivity for early detection: The system of the present invention achieves an accuracy of 90.2% and an F1 score of 90.2% in the stage of mild cognitive impairment (MCI). This is a significant improvement compared to the method in prior art CN118044785A (accuracy of approximately 82.5%) and the accuracy of clinicians (accuracy of approximately 70%). In the examples, the system of the present invention is observed to have advantages in early detection, providing a critical time window for early intervention and treatment.

[0173] The system of the present invention also includes a continuous learning module 11 and a remote diagnosis module 12, which support continuous system optimization and remote medical applications.

[0174] The continuous learning module 11 is used to incrementally update the model after acquiring new diagnostic data. In practical applications, medical institutions continuously accumulate new patient data, which may come from different devices, different populations, or different regions, causing changes in data distribution. To enable the system to adapt to new data distributions, the continuous learning module 11 periodically performs incremental training on the model.

[0175] The specific implementation is as follows: After accumulating a certain amount of new data (preferably 100 new samples), the continuous learning module 11 first analyzes the distribution characteristics of the new data and calculates the distribution distance between the new data and the training set data. The distribution distance can be measured using metrics such as maximum mean difference (MMD) or KL divergence.

[0176] ,

[0177] in, For the distribution of training set data, For the new data distribution, For feature mapping function, and These represent the number of samples in the training set and the number of samples in the new data, respectively.

[0178] If the distribution characteristics of the new data do not conform to the twelfth target range (in this embodiment, ...), If the new data distribution deviates significantly from the training set, then incremental training of the Transformer model is performed. Incremental training uses a small learning rate (preferably 1×10⁻⁶). -5 The model is trained for 10 epochs on new data, and the model parameters are updated to adapt to the new data distribution. To avoid catastrophic forgetting (the model forgets old knowledge when learning new knowledge), incremental training is performed by mixing a portion of the training set data to maintain the memory of the old data.

[0179] If the distribution characteristics of the new data conform to the twelfth objective range (i.e.) If the distribution of the new data is similar to that of the training set, then the model parameters should be kept unchanged to avoid unnecessary training overhead. The new data can be added to the database for future model updates.

[0180] Through a continuous learning mechanism, the system can continuously optimize itself as data accumulates, adapt to different application scenarios and population characteristics, and maintain long-term diagnostic performance.

[0181] The remote diagnosis module 12 allows primary healthcare institutions to upload patient data to a cloud server via network connection. The high-performance Transformer model in the cloud performs the diagnosis, and the results are then returned to the primary healthcare institution. This remote diagnosis model enables resource-constrained primary healthcare institutions to enjoy high-quality diagnostic services, which is beneficial for the early screening and prevention of Alzheimer's disease.

[0182] In summary, through the detailed description of the embodiments, this invention provides a complete implementation scheme for an early auxiliary diagnostic system for Alzheimer's disease based on multimodal Transformer. This system achieves deep fusion of multimodal data through a hierarchical Transformer architecture, introduces prior medical knowledge to guide feature extraction, designs a lightweight inference module to support primary healthcare applications, provides interpretable analysis to enhance clinical credibility, and supports continuous learning and remote diagnosis. It possesses advantages such as high accuracy, high efficiency, interpretability, and scalability, providing an innovative technical solution for the early auxiliary diagnosis of Alzheimer's disease.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An early auxiliary diagnostic system for Alzheimer's disease based on multimodal Transformer, characterized in that, include: The data acquisition module is used to acquire the patient's MRI image data, cognitive test data, and speech feature data; The preprocessing module is used to preprocess MRI image data and standardize cognitive test data and speech feature data; The medical prior knowledge guidance module is used to identify key brain region information based on the pathological characteristics of Alzheimer's disease and assign attention weight parameters to each key brain region. The hierarchical feature extraction module includes a single-modal feature extraction unit and a cross-modal fusion unit. The single-modal feature extraction unit is used to extract features from each modality of data based on the Transformer encoder to obtain feature vectors. The cross-modal fusion unit is used to weight the MRI image feature vectors based on the attention weight parameters and fuse the weighted feature vectors through the cross-modal Transformer fusion layer to obtain a fused feature representation. The hierarchical diagnosis module is used to perform hierarchical judgment on the fused feature representation, and decide whether to perform feature enhancement processing based on the confidence level before inputting it into the diagnostic classifier to obtain the diagnostic result. The lightweight inference module is used to invoke a lightweight Transformer model for inference when computing resources are below a target threshold. The diagnostic output module is used to output diagnostic results.

2. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, The first preprocessing module is also used for: Slices are selected from MRI image data. If the information entropy value of an MRI image slice meets the fifth target range, the slice is retained; otherwise, the slice is discarded. The preserved slices were subjected to contrast enhancement processing to improve the contrast of the hippocampus, ventricle, and temporal lobe regions.

3. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, The medical prior knowledge guidance module is also used for: The priority order of key brain regions is determined based on the patient's age information. If the patient's age falls within the sixth target range, the attention weight parameter of the hippocampus region is set as the first weight value, the attention weight parameter of the temporal lobe region is set as the second weight value, and the attention weight parameter of the ventricle region is set as the third weight value, where the first weight value is greater than the second weight value, and the second weight value is greater than the third weight value. The attention weight parameters are dynamically adjusted based on the patient's cognitive test score. If the cognitive test score is within the seventh target range, the attention weight parameters of the hippocampus region are increased; if the cognitive test score is not within the seventh target range, the attention weight parameters of the temporal lobe region are increased.

4. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, The single-modal feature extraction unit includes: The MRI feature extraction subunit is used to divide MRI image data into several image blocks, embed and encode each image block, and extract spatial features through a multi-layer Transformer encoder to obtain MRI image feature vectors. The cognitive test feature extraction subunit is used to numerically encode the cognitive test data and extract cognitive features through a one-dimensional Transformer encoder to obtain a cognitive test feature vector. The speech feature extraction subunit is used to perform spectral analysis on speech feature data and extract speech features through a time-series Transformer encoder to obtain a speech feature vector.

5. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, The cross-modal fusion unit is also used for: Calculate the cross-modal attention score among MRI image feature vectors, cognitive test feature vectors, and speech feature vectors; If the attention score of a certain modality is within the range of the eighth target threshold, then the feature of that modality is enhanced; if the attention score of a certain modality is not within the range of the eighth target threshold, then the feature of that modality is suppressed. Based on the attention score, the feature vectors of each modality are adaptively weighted and fused to obtain the fused feature representation.

6. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, The hierarchical diagnostic module is also used for: Uncertainty estimation is performed on the fusion feature representation. If the uncertainty estimate is within the range of the ninth target, the diagnostic result is directly output. If the uncertainty estimate is not within the range of the ninth target, the multi-model integration mechanism is activated. The multi-model integration mechanism includes calling multiple Transformer models with different configurations to perform diagnosis, and then voting on the multiple diagnosis results to obtain the final diagnosis result.

7. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, The lightweight inference module is also used for: When computing resources are detected to be below the fourth target threshold, if the priority of the diagnostic task is within the tenth target range, a quantization compression strategy is used to compress the parameters of the Transformer model; if the priority of the diagnostic task is not within the tenth target range, a dynamic pruning strategy is used to prune the number of layers in the Transformer model. If the inference time exceeds the eleventh target threshold during the inference process, the early stop mechanism is activated to output the diagnostic results of the intermediate layer ahead of time.

8. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, Also includes: The interpretability analysis module is used to generate a visual heatmap based on the attention weight matrix, which identifies the brain regions and feature dimensions that contribute the most to the diagnostic results. The continuous learning module is used to incrementally train the Transformer model and update the model parameters if the distribution characteristics of the new data do not conform to the twelfth target range after acquiring new diagnostic data; otherwise, the model parameters remain unchanged.

9. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, The diagnostic output module is also used for: Output the confidence score of the diagnostic result. If the confidence score meets the thirteenth target threshold range, the diagnostic result is marked as high confidence. If the confidence score does not meet the thirteenth target threshold range, the diagnostic result is marked as low confidence and further examination is recommended. A diagnostic report is generated, which includes abnormal indicators of each modality of data, assessment of the degree of lesions in key brain regions, and diagnostic recommendations.

10. The Alzheimer's disease early auxiliary diagnostic system based on multimodal Transformer according to claim 1, characterized in that, The system also includes: The data security module is used to encrypt and store and transmit patients' medical data, and adopts a federated learning approach to protect patient privacy during model training. The remote diagnosis module supports primary healthcare institutions in uploading patient data to a cloud server via network connection. The cloud-based Transformer model then performs the diagnosis and returns the results to the primary healthcare institution.

Images