Cognitive disorder progress trend prediction method based on multi-modal data fusion

Abstract

The invention discloses a cognitive disorder progress trend prediction method based on multi-modal data fusion, which comprises the following steps: firstly, acquiring MR I images, PET images, EEG and scale information which comprises clinical cognitive score data, patient age, intelligence quotient score, gender and education degree, and carrying out standardization processing; a feature extraction network composed of a generator and a discriminator is constructed, the MRI image serves as input, high-quality PET image features are generated through the generator, the generated PET image features and real PET image features are discriminated through the discriminator, and then the generator is optimized; constructing an enhanced multi-modal classification model, and performing deep fusion on the MR I image, the generated PET image features, the EEG signals and the clinical cognitive scores to obtain multi-modal features; and 4, inputting multi-modal features, carrying out time sequence modeling by adopting a Mamba model, and predicting the cognitive decline trend of the patient in a future time period.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and intelligent medical treatment, and particularly to a method for predicting the progression trend of cognitive impairment based on multi-modal data fusion. Background Art

[0002] At present, the clinical diagnosis of Alzheimer's disease mainly relies on neuroimaging examinations, cerebrospinal fluid biomarker detection, and cognitive scale assessment. However, traditional methods have obvious limitations in application. Although MRI can reflect brain structure changes, it is difficult to capture early functional metabolic abnormalities; although PET can detect amyloid deposition, its high cost, radioactive exposure, and equipment scarcity limit its wide application; cognitive scales are easily interfered by subjective factors and have insufficient sensitivity and specificity. In addition, single-modal data is difficult to comprehensively reflect the complex pathological characteristics of AD, resulting in low early diagnosis accuracy and inability to meet the needs of precision medicine.

[0003] In recent years, multi-modal data fusion technology based on machine learning has gradually become a research hotspot in AD. Existing methods integrate MRI, PET, EEG, and clinical data, and use deep learning models for feature extraction and classification, significantly improving the diagnostic performance. However, the heterogeneity of multi-modal data leads to difficulties in data alignment, affecting the generalization ability of the model. At the same time, the scarcity of PET data has become a key bottleneck in model training. Existing generative adversarial networks have problems such as detail loss and insufficient medical interpretability when generating high-quality PET images. In addition, traditional time series modeling methods are difficult to effectively capture the long-term dependence relationship of electroencephalogram signals, limiting the accuracy of disease course prediction.

[0004] In the prior art, although multi-modal fusion models optimize feature interaction through the attention mechanism, the pixel-level fusion of cross-modal information is still insufficient, and the balance problem between data scarcity and computational efficiency has not been solved. And although the real-time diagnosis system based on the cloud computing platform has improved accessibility, there are still challenges in privacy protection and multi-center data adaptability. Therefore, an intelligent diagnosis technology that can deeply fuse multi-modal data, efficiently generate alternative features, and take into account security is needed to break through the existing bottlenecks and promote the clinical application of early diagnosis and personalized intervention of AD. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and propose a method for predicting the progression trend of cognitive impairment based on multimodal data fusion. By integrating multi-source medical data, combining the feature generation capabilities of generative adversarial networks and vision transformers, and introducing a pixel-level bidirectional cross-attention mechanism, it can achieve high-precision and low-cost early diagnosis and disease course prediction. This method not only solves the limitations of traditional single-modal diagnosis but also further optimizes the allocation of medical resources through a cloud-based real-time diagnosis platform, which has important clinical significance for improving the efficiency of early intervention in Alzheimer's disease and reducing medical costs.

[0006] To achieve the above object, the specific technical solution adopted by the present invention is as follows:

[0007] A method for predicting the progression trend of cognitive impairment based on multimodal data fusion, comprising the following steps:

[0008] Step 1: Obtain MRI images, PET images, EEG, and scale information, where the scale information includes clinical cognitive score data, patient age, IQ score, gender, and education level, and perform standardization processing;

[0009] Step 2: Construct a feature extraction network composed of a generator and a discriminator. Taking the MRI image as the input, extract latent space features through the generator, capture global spatial information, and finally generate high-quality PET image features. The discriminator discriminates the generated PET image features from the real PET image features to optimize the generator;

[0010] Step 3: Construct an enhanced multimodal classification model (MSTNet). The enhanced multimodal classification model includes a tabular feature encoder, a temporal feature encoder, and a cross-modal aggregation encoder. Introduce a multimodal spatio-temporal attention mechanism in the cross-modal aggregation encoder to deeply fuse the MRI image, the generated PET image features, the EEG signal, and the clinical cognitive score to obtain multimodal features;

[0011] Step 4: Input the multimodal features and use the Mamba model for temporal modeling to predict the cognitive decline trend of the patient in the future time period.

[0012] Preferably, in Step 1, the standardization processing method is: Use a spatial registration algorithm to achieve spatial alignment of the MRI image and the PET image, and perform time heterogeneity elimination processing on the obtained multimodal data through a time synchronization strategy.

[0013] Preferably, the generator includes an encoder based on a convolutional module, a decoder, and an intermediate layer with a ViT bottleneck layer architecture. Taking MRI images as input, the encoder extracts latent space features, the ViT bottleneck layer captures global spatial information, and finally the decoder generates high-quality PET image features.

[0014] Preferably, the decoder includes three downsampling modules and three upsampling modules. Each downsampling module includes a max pooling layer, a group normalization layer, a convolutional layer, and a relay activation layer. Each upsampling module includes a group normalization layer, a transposed convolutional layer, and a ReLU activation layer. The filter size of the transposed convolutional layer is 7×7×7 for the first and last convolutional layers, and 3×3×3 for the others.

[0015] Preferably, the channels of the downsampling modules are 64, 128, and 256 respectively, and the number of channels of the upsampling modules is mirror-consistent with that of the downsampling modules.

[0016] Preferably, in step 3, the scale information is used as input. The tabular feature encoder converts numerical features and categorical features into embedding vectors through a feature tokenizer, and captures the complex relationships between features through a multi-head self-attention mechanism and a feed-forward network to obtain a first feature vector. The temporal feature encoder performs multi-period feature extraction on EEG signals through a temporal block module, decomposes time series data using fast Fourier transform, and captures periodic changes through multi-scale two-dimensional convolutional kernels to obtain a second feature vector. The cross-modal aggregation encoder fuses MRI and EEG features through a cross-attention mechanism to obtain a third feature vector. Finally, the first feature vector, the second feature vector, and the third feature vector are concatenated and fused to obtain multi-modal features.

[0017] Preferably, the Mamba model includes 6 Mamba modules, processes the fused multi-modal features through a state space model, and then obtains a classification result through a linear layer.

[0018] Preferably, for multiple diagnosis cases, record the time interval Δt between the previous and the current diagnoses of the same patient, and incorporate the time interval Δt between the previous and the current diagnoses into the scale information.

[0019] The present invention has the following characteristics and beneficial effects:

[0020] The present invention strictly preprocesses the MRI, PET, EEG, and clinical cognitive scale data through standardization to ensure the unity of each modality data in terms of space, time, and numerical distribution; and combines advanced multi-modal feature extraction and fusion technologies to make full use of the advantages of each modality data, realizing the organic integration of spatial and temporal information, thereby providing a solid data foundation and technical guarantee for early diagnosis and disease progression prediction.

[0021] Therefore, in the present invention, around the optimization of multi-modal data fusion and deep learning models, generative adversarial networks and vision Transformer modules are comprehensively used to strengthen the expression of the latent space features of MRI images, making up for the deficiencies of scarce PET data and lost spatial information in traditional diagnosis; at the same time, through the cross-modal attention mechanism, deep fusion of EEG and MRI data is achieved to ensure sufficient information interaction between different modalities and further eliminate feature redundancy.

[0022] To verify the effectiveness of this technical solution, the present invention conducts comprehensive multi-index evaluation and carefully verifies the contributions of each key module through ablation experiments. Therefore, the present invention proposes a method for early diagnosis of Alzheimer's disease and prediction of MCI progression with high precision and low cost, significantly reducing the misdiagnosis risk and data mismatch problems caused by single-modal diagnosis, improving the overall diagnostic accuracy, robustness and clinical applicability, and providing a more scientific and reliable basis for clinical intervention and patient management. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the overall structure of a method for predicting the progression trend of cognitive impairment based on multi-modal data fusion according to an embodiment of the present invention;

[0024] Figure 2 It is a schematic diagram of the table feature encoder module of the present invention;

[0025] Figure 3 It is a schematic diagram of the time feature encoder module of the present invention;

[0026] Figure 4 It is a schematic diagram of the overall architecture of GFE-Mamba of the present invention;

[0027] Figure 5 It is a schematic diagram of the 3DGAN-ViT network structure of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0029] Embodiment 1

[0030] A method for predicting the progression trend of cognitive impairment based on multi-modal data fusion, as Figure 1 shown, includes the following steps:

[0031] Step 1: Obtain MRI images, PET images, EEG, and clinical cognitive scoring data, and perform standardization processing.

[0032] Specifically, through a data preprocessing module, a spatial registration algorithm is used to achieve the spatial alignment of MRI and PET images, and a time synchronization strategy is adopted to solve the temporal heterogeneity of multimodal data, eliminating the influence of device differences on the data distribution, so as to provide consistent and high-quality input data for subsequent modules. The results of this module serve as the basis for subsequent multimodal feature extraction and fusion, ensuring the stable and standardized input quality of each modality data, thereby improving the robustness and prediction accuracy of the overall system.

[0033] Step 2: Construct a feature extraction network composed of a generator and a discriminator. Taking the MRI image as the input, extract latent space features through the generator, capture global spatial information, and finally generate high-quality PET image features. The discriminator discriminates between the generated PET image features and the real PET image features to optimize the generator.

[0034] In this embodiment, as Figure 5 shown, the feature extraction network is a 3DGAN-ViT network composed of a generator and a discriminator. Latent space features are extracted through the encoder, and the global spatial information is captured using the ViT bottleneck layer. The decoder generates high-quality PET image features, breaking through the bottleneck of the scarcity of actual PET data and providing supplementary information for subsequent fusion.

[0035] Specifically, the 3DGAN-ViT generation network is used to solve the problem of scarce PET data. By converting MRI images into high-quality PET representations, it provides supplementary information for the multimodal classification model. This module consists of two parts: a generator and a discriminator. The generator is composed of an encoder, a ViT bottleneck layer, and a decoder. The encoder part consists of three downsampling modules, each module containing a max pooling layer, a group normalization layer, a convolutional layer, and a ReLU activation layer, with the number of channels being 64, 128, and 256 in sequence. The decoder part includes three upsampling modules, each module consisting of a group normalization layer, a transposed convolutional layer, and a ReLU activation layer, with the number of channels mirroring the downsampling modules. The middle layer uses a Vision Transformer (ViT) module to replace the original ResNet middle block. The 3D feature map is flattened into a sequence of 2D image patches through PatchEmbedding, and then processed by the Transformer encoder and resized for the decoder to generate PET images. The discriminator structure is the same as the encoder, extracts features from the generated PET images, and generates feature maps for loss calculation. The loss function of the 3DGANGAN-ViT generation network is divided into generator loss and discriminator loss, and its generator loss is defined as:

[0036]

[0037] Among them, ||G(x M ) - y p ||₂ represents the mean square error reconstruction loss between the real PET image and the generated PET image, log(1 - D(G(x M ))) represents the adversarial loss of the generator, ||VGG(x M ) - VGG(y p )||₂ represents the perceptual loss extracted by VGG19

[28] . x M refers to the real PET image, and y p is the generated PET image.

[0038] The discriminator loss is defined as:

[0039]

[0040] Among them, log(1 - D(y p )) represents the adversarial loss of the discriminator on the real PET image, and log(D(G(x M ))) represents the adversarial loss on the generated PET image.

[0041] Furthermore, the middle module of the Vision Transformer (ViT) is used to retain the global spatial information of the MRI image in the latent space and avoid the loss of spatial information caused by direct flattening. The specific steps include: 1. After the MRI image is processed by the encoder, a latent space representation is obtained, and this feature map is first flattened into a 2D feature map; 2. Subsequently, the 2D feature map is split into a series of image patch sequences through the PatchEmbedding operation; 3. The obtained image patch sequence enters the encoder composed of 4 Transformer modules, and the global dependency relationship is captured through the mutual attention mechanism; 4. After the processed sequence is resized, it is passed to the decoder to generate the PET image.

[0042] Step 3: Construct an enhanced multi-modal classification model. The enhanced multi-modal classification model includes a tabular feature encoder, a temporal feature encoder, and a cross-modal aggregation encoder. A multi-modal spatio-temporal attention mechanism is introduced into the cross-modal aggregation encoder to deeply fuse the MRI image, the generated PET image features, EEG signals, and clinical cognitive scores to obtain multi-modal features.

[0043] Specifically, as Figure 2 shown, for the tabular feature encoder, this module uses a feature tokenizer to convert digital features such as patient age, total MMSE score, etc., and categorical features such as gender, education level into embedding vectors to obtain the first feature vector. For a given feature, its embedding calculation can be described as:

[0044] T j = b j + f j (x j ) ∈ ℝ d f j : → ℝ d

[0045] where b j is the deviation of the j-th feature. For numerical features, is implemented as an element-wise multiplication with the vector W (num) ∈ ℝ d . For categorical features, is implemented as a lookup table Specifically expressed as:

[0046]

[0047] It should be noted that the tabular encoder architecture processes the embedding vectors generated by the feature tokenizer. It uses the multi-head self-attention mechanism and the feed-forward network to capture the complex relationships between features. The final output of the tabular encoder comes from the processed output of the transformer layer. In the output of the transformer, CLS summarizes the entire input sequence. This token is passed to the prediction layer, where it is combined with the aggregated features from the image and EEG modalities through cross-modal attention. Then the resulting features are fed into a linear layer for classification.

[0048] As Figure 3 shown, for the time feature encoder, aiming at the complex time information contained in the EEG data, this module first preprocesses the one-dimensional time series, decomposes the time series data using the fast Fourier transform, and captures the periodic variations through multi-scale two-dimensional convolutional kernels to obtain the second feature vector, with the formula as follows:

[0049] A = Avg(Amp(FFT(X 1D )))

[0050]

[0051] where A ∈ ℝ T represents the intensity of each resolved frequency component in X 1D . The k frequencies {f1,..., f k} with the maximum intensity correspond to the most significant period lengths {p1,..., p k}. This process extracts the key cyclic features and filters out the noise. Next, fold the original one-dimensional time series X 1D :

[0052]

[0053] Among them, Padding(·) extends the time series by adding zeros along the time dimension. This ensures that the sequence length is divisible by p i and the result is a set of two-dimensional quantities Each tensor corresponds to a specific time variation. These two-dimensional tensors exhibit locality, which can be captured by two-dimensional convolution. A multi-scale two-dimensional convolutional kernel is used to process the transformed two-dimensional features and capture the intra-cycle variations (columns) and inter-cycle variations (rows) within the EEG signal:

[0054]

[0055] Among them, the initial block contains multiple two-dimensional convolutional kernels of different scales. This multi-scale convolution captures various feature patterns in the time series data, enhancing the model's ability to understand and process EEG signals. After processing, the two-dimensional features are transformed into one-dimensional features and weighted according to the amplitude values of each cycle:

[0056]

[0057] In the formula, is the amplitude corresponding to the frequency, which is normalized by Softmax. This weighted aggregation ensures the importance of each cycle feature in the final feature representation. To ensure robust training, residual connections are applied:

[0058] X l，1D = TimesBlock(X l-1，1D ) + X l-1，1D

[0059] Cross-modal aggregation encoder, this module aims to fuse the spatial information of MRI images and the time information of EEG signals. First, the 3D MRI images are preprocessed with 3×3×3 convolution, batch normalization (BN), and ReLU activation, and the brain structure features are extracted using multiple residual dense connection layers and spatial average pooling; at the same time, the EEG features extracted by the time feature encoder are processed into three-dimensional features through appropriate scaling. Then, a cross-modal attention mechanism is used to fuse the processed MRI features and EEG features to obtain the third feature vector, and its cross-attention calculation formula is:

[0060] Q = XW Q , K = XW K , V = XW V

[0061]

[0062] Attention(Q, K, V) = cos(Q, K) × V

[0063] Among them, W Q , W K and W V ∈R d×d . Here, Q represents query information, K is the matching vector (or index), and V is the content vector. Q is derived from the electroencephalogram feature vector sequence X EEG ∈R N×d , while K and V are derived from the MRI feature vector sequence X MRI ∈R N×d . Then, the high-level features obtained through cross-modal attention are combined with the simple concatenation features in the aggregation module. This combination integrates high-level and low-level features, creating a richer and more comprehensive feature representation.

[0064] Finally, the first feature vector T, the second feature vector X l，1D and the third feature vector attention(Q, K, V) are concatenated and fused to obtain multi-modal features.

[0065] Step 4: Input the multi-modal features and use the Mamba model for temporal modeling to predict the cognitive decline trend of the patient in the future time period.

[0066] It can be understood that the prediction model composed of Steps 1 - 4 is defined as the GFE-Manba model. Specifically, as Figure 4 shown, the present invention uses the overall architecture of GFE-Mamba after multi-modal fusion to achieve temporal modeling and prediction of the future cognitive decline risk of the patient. Specifically, the fused high-dimensional features are transmitted to the temporal modeling module, which combines the Transformer encoder and the Mamba neural network. In this embodiment, the cognitive decline risk within 6 to 24 months is predicted through continuous state updates, and a personalized disease course progress report is generated to provide a scientific basis for clinical intervention.

[0067] For the above technical solution, in this embodiment, a variety of evaluation indicators are used to comprehensively evaluate the overall performance of the system, and the indicators include accuracy, precision, recall rate, F1 score, and Matthews correlation coefficient. The specific indicators and calculation formulas are as follows:

[0068]

[0069] Among them, Accuracy represents the proportion of correctly predicted cases in all classifications; Precision represents the proportion of truly positive cases among all samples predicted as positive; Recall refers to the proportion of correctly identified cases among all truly positive samples; F1-score is the harmonic mean of precision and recall; and MCC is a balance index for measuring the overall performance of a binary classifier, where TP and TN represent the number of samples correctly predicted as positive and negative respectively, and FP and FN represent the number of samples of the positive and negative classes that are mispredicted.

[0070] Finally, in practical applications, an intelligent diagnosis platform based on cloud computing and 5G communication is constructed to achieve rapid upload, real-time analysis of multi-modal data, and automatic generation of diagnostic reports, support multi-terminal access for medical institutions, patients, and research institutions, and improve diagnostic efficiency and accessibility.

[0071] Specifically, the intelligent diagnosis platform is based on cloud computing and 5G communication technologies, supports rapid upload, real-time analysis of multi-modal data, and automatic generation of diagnostic reports; the front end of the platform integrates Ajax technology to achieve asynchronous data exchange, improving the user experience and enabling the web page to be updated in real time without reloading; uses the Chart.js library to build intuitive and easy-to-understand charts based on HTML5, facilitating users to understand complex medical data; the back end uses the SpringBoot framework to simplify the configuration work of the application and improve development efficiency; Redis is used as a caching solution to improve data processing speed and concurrency; for algorithm processing on the GPU server side, the Flask framework is used for encapsulation, and API interfaces are opened for the front end to call, realizing the efficient operation of the algorithm and the flexible docking of business logic, supporting multi-terminal access, and ensuring that users can obtain diagnostic results anytime and anywhere.

[0072] In this embodiment, aiming at the key problems in the diagnosis process of Alzheimer's disease and mild cognitive impairment, such as insufficient multi-modal data processing ability, limitations in single-modal diagnosis, and heterogeneity in the collection and fusion of different data sources, multi-source data such as MRI, PET, EEG, and clinical cognitive scales are combined, and standardized preprocessing methods are used to ensure data consistency, and through multi-modal feature extraction and fusion strategies, efficient integration of cross-modal information is achieved. In addition, deep learning technology is introduced to optimize the model architecture and enhance the data feature expression ability to more accurately identify early AD patients and the progression of MCI. Finally, a method for early diagnosis of Alzheimer's disease and prediction of MCI progression based on multi-modal data fusion and deep learning is proposed. This method not only effectively reduces problems such as inconsistent data distribution, separation of spatial and temporal information, and feature redundancy, but also provides strong technical support in terms of diagnostic accuracy, stability, and interpretability, providing a reliable basis for early screening and disease progression prediction in clinical practice.

[0073] The technical solution proposed by the present invention integrates a generative adversarial network, a vision Transformer module, and a cross-modal attention mechanism, which can fully exploit and utilize the key information in multi-modal data to achieve in-depth feature extraction and adaptive fusion of the patient's brain structure, function, and biomarkers. The cross-modal attention mechanism is used to enhance the collaborative analysis ability of different modal data such as EEG and MRI, thereby improving the recognition accuracy and generalization ability of Alzheimer's disease. Compared with traditional diagnostic methods based on single-modal or shallow feature extraction, this method performs excellently in multiple key indicators such as accuracy, recall rate, F1-Score, and MCC, significantly reducing the misdiagnosis rate and at the same time enhancing the prediction ability of the disease progression of MCl patients. In addition, this method also shows good adaptability in terms of computational efficiency, data utilization, and scalability, making it possible for low-cost and high-precision early screening of Alzheimer's disease, providing a new technical direction for future precision medicine and intelligent assisted diagnosis, and having important application value and social significance for the early detection, intervention, and clinical decision-making of Alzheimer's disease.

[0074] Table 1 shows the comparison data between the embodiments of the present invention and the prior art solutions

[0075] Method Precision Recall F1 Acc MCC HTCF 87.78 87.74 87.76 88.22 75.52 JD-CNN 88.34 87.84 88.33 88.32 85.79 Table Transformer 88.10 88.73 87.92 89.10 43.52 MSTNet 88.25 88.89 88.57 90.00 86.16

[0076] The experiment of this embodiment was conducted on a private dataset, which included EEG, MRI, and scale data from 100 subjects (average age: 72.4 years; age range: 56 - 93 years; 56 female; 22 married). For each patient, the dataset included 180 - second continuous artifact - free EEG segments recorded at a sampling frequency of 256 Hz, and 72 MRI slices with a size of 256×256 pixels. In addition, the scale data included MMSE and MoCA scores, and the detailed MMSE results provided the total score and individual item scores. The dataset was divided into 80 samples for training and 20 samples for evaluation. In this embodiment, the MSTNet network model was implemented using the PyTorch framework and trained on an NVIDIA Tesla V100 GPU. During the training process, the dropout rate was set to 0.1, the batch size was set to 100, and the learning rate was set to 0.0001. The training process lasted for 24 hours. As can be seen from Table 1, compared with the ResNet series models, although ResNet effectively solved the problem of gradient disappearance in deep networks through residual connectivity, its ability to capture pathological features when processing MRI images was still insufficient. MSTNet significantly improved the classification accuracy and robustness through a multi - scale feature extraction and fusion mechanism. Therefore, MSTNet was superior to ResNet in terms of accuracy, recall, F1 - score, MCC, and precision by 34.44%, 33.33%, 43.72%, and 50.65% respectively.

[0077] Compared with HTCF, MSTNet improved in precision, recall, F1 - score, and accuracy. Especially in terms of the Matthews correlation coefficient (MCC), MSTNet's score was 10.64% higher than that of HTCF, showing a significant advantage of MSTNet in classification performance, especially when dealing with imbalanced datasets. Compared with JD - CNN, MSTNet showed better performance in all five metrics. Although JD - CNN was slightly higher in precision by 0.09%, the superior performance of MSTNet in other key metrics indicated that it was more effective and reliable in diagnosing Alzheimer's disease. Compared with TableTransformer, MSTNet was 0.90% and 42.64% higher in accuracy and MCC respectively. This showed that MSTNet far exceeded TableTransformer in overall prediction accuracy and the ability to handle imbalanced datasets, providing more reliable diagnostic results.

[0078] Example 2

[0079] The difference between this embodiment and Embodiment 1 lies in that for multiple diagnosis situations, the time interval Δt between two consecutive diagnoses of the same patient is recorded, and the time interval Δt between two consecutive diagnoses is incorporated into the scale information, significantly improving the accuracy of diagnosis and prediction. The introduction of Δt enables the system to more comprehensively capture the dynamic changes in the patient's condition, thereby improving the accuracy of diagnosis. The addition of time features not only enhances the ability to predict the progression of the disease, helps predict possible future changes in the patient's condition, but also optimizes the effect of multi-modal data fusion, enabling the system to combine information in the time dimension and more accurately evaluate the rate and trend of changes in the patient's condition.

[0080] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A method for predicting the progression trend of cognitive impairment based on multi-modal data fusion, characterized in that, It includes the following steps: Step 1: Obtain MRI images, PET images, EEG, and scale information. The scale information includes clinical cognitive score data, patient age, IQ score, gender, and education level, and perform standardization processing; Step 2: Construct a feature extraction network composed of a generator and a discriminator. Taking the MRI image as the input, extract latent space features through the generator, capture global spatial information, and finally generate high-quality PET image features. The discriminator discriminates the generated PET image features from the real PET image features to optimize the generator; Step 3: Construct an enhanced multi-modal classification model. The enhanced multi-modal classification model includes a tabular feature encoder, a temporal feature encoder, and a cross-modal aggregation encoder. Introduce a multi-modal spatio-temporal attention mechanism in the cross-modal aggregation encoder to deeply fuse the MRI image, the generated PET image features, the EEG signal, and the clinical cognitive score to obtain multi-modal features; Step 4: Input the multi-modal features and use the Mamba model for temporal modeling to predict the cognitive decline trend of the patient in the future time period.

2. The cognitive impairment progression trend prediction method based on multi-modal data fusion according to claim 1, characterized in that In the above Step 1, the standardization processing method is: Use a spatial registration algorithm to achieve spatial alignment of the MRI image and the PET image, and perform time heterogeneity elimination processing on the acquired multi-modal data through a time synchronization strategy.

3. A method for predicting the progression trend of cognitive impairment based on multi-modal data fusion according to claim 1, characterized in that, The generator includes an encoder based on a convolutional module, a decoder, and an intermediate layer with a ViT bottleneck layer architecture. Taking the MRI image as the input, extract latent space features through the encoder, capture global spatial information using the ViT bottleneck layer, and finally generate high-quality PET image features through the decoder.

4. A method for predicting the progression trend of cognitive impairment based on multi-modal data fusion according to claim 1, characterized in that, The decoder includes three downsampling modules and three upsampling modules. Each downsampling module includes a max pooling layer, a group normalization layer, a convolutional layer, and a relay activation layer. Each upsampling module includes a group normalization layer, a transposed convolutional layer, and a ReLU activation layer. The filter size of the transposed convolutional layer is 7×7×7 for the first and last convolutional layers, and 3×3×3 for the others.

5. A method for predicting the progression trend of cognitive impairment based on multimodal data fusion according to claim 4, characterized in that, The channels of the downsampling modules are 64, 128, and 256 respectively, and the number of channels of the upsampling modules is the mirror image of that of the downsampling modules.

6. A method for predicting the progression trend of cognitive impairment based on multimodal data fusion according to claim 1, wherein In the above Step 3, taking the scale information as the input, the tabular feature encoder converts the numerical features and categorical features into embedding vectors through a feature tokenizer, and captures the complex relationships between the features through a multi-head self-attention mechanism and a feed-forward network to obtain a first feature vector; the temporal feature encoder performs multi-period feature extraction on the EEG signal through a temporal block module, decomposes the time series data using a fast Fourier transform, and captures the periodic changes through a multi-scale two-dimensional convolutional kernel to obtain a second feature vector; the cross-modal aggregation encoder fuses the MRI and EEG features through a cross-attention mechanism to obtain a third feature vector. Finally, the first feature vector, the second feature vector, and the third feature vector are concatenated and fused to obtain multi-modal features.

7. A method for predicting the progression trend of cognitive impairment based on multi-modal data fusion according to claim 1, characterized in that, The Mamba model includes 6 Mamba modules, processes the fused multi-modal features through a state space model, and then obtains the classification result through a linear layer.

8. A method for predicting the progression trend of cognitive impairment based on multi-modal data fusion according to claim 1, wherein, For multiple diagnosis cases, record the time interval Δt between the previous and the subsequent diagnoses of the same patient, and incorporate the time interval Δt between the previous and the subsequent diagnoses into the scale information.

Images