Mild cognitive impairment conversion prediction method based on incomplete three-mode common-attention fusion

Abstract

The invention discloses a mild cognitive impairment conversion classification method based on incomplete three-mode common-attention fusion. The method comprises the following steps: acquiring a data set; step 2, applying a pre-trained missing mode generation module, inputting MRI data, and generating missing PET data in a cross-mode conversion mode; step 3, applying a feature extraction module, obtaining table feature representation through a table encoder, and obtaining visual feature representation through a visual encoder; 4, performing intra-modal feature aggregation on the extracted multi-modal features through a multi-head self-attention module to obtain three-modal features; step 5, inputting the trimodal features after feature alignment into a common attention fusion module for feature fusion so as to obtain multi-modal fusion features; and step 6, applying a pre-trained classification head, and inputting the obtained multi-modal fusion features into the classification head to realize final classification. According to the method, the accuracy of MCI conversion prediction is improved by using richer multi-modal features.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method for predicting mild cognitive impairment conversion based on incomplete trimodal co-attention fusion. Background Art

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease with a very high mortality rate, which leads to permanent cognitive damage. Mild cognitive impairment (MCI) is one of the early stages of AD. Some MCI patients will develop AD within three years, which is called progressive MCI (pMCI), while stable MCI (sMCI) patients remain cognitively stable during this period. Accurate prediction of the progression of pMCI is crucial for early intervention, because early intervention helps to delay the development of AD. Clinically, brain imaging methods such as magnetic resonance imaging (MRI) and positron emission tomography (PET) are used to assess the patient's brain state and assist doctors in diagnosis. However, this process relies on the doctor's experience and subjective judgment, resulting in greater uncertainty and time cost in the diagnosis results. However, PET data are often missing due to the high cost and the use of radioactive tracers, which limits the application of multimodal methods. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention proposes a mild cognitive impairment conversion prediction method based on incomplete trimodal co-attention fusion, which solves the problem of PET missing and combines MRI, PET and clinical table data to improve the accuracy of MCI conversion prediction with richer multimodal features.

[0004] In order to solve the above technical problems, the technical solution of the present invention is:

[0005] A conversion classification method for mild cognitive impairment based on incomplete trimodal co-attention fusion, comprising the following steps:

[0006] Step 1: Acquire a data set, wherein the data set includes MRI data, PET data, and clinical table data;

[0007] Step 2: Apply a pre-trained missing modality generation module to generate missing PET data by inputting MRI data through cross-modality conversion;

[0008] Step 3, applying a feature extraction module, obtaining a table feature representation through a table encoder, and obtaining a visual feature representation through a visual encoder, wherein the visual feature representation includes an MRI feature representation and a PET feature representation;

[0009] Step 4: The extracted multimodal features are aggregated through a multi-head self-attention module to obtain trimodal features;

[0010] Step 5: Input the trimodal features into a co-attention fusion module for feature fusion to obtain multimodal fusion features;

[0011] Step 6: Apply a pre-trained classification head and input the obtained multimodal fusion features into the classification head to achieve the final classification.

[0012] Preferably, the missing modality generation module adopts a vector quantization generative adversarial network.

[0013] Preferably, the method for generating missing PET data is: taking the MRI data as input, firstly passing it through a 3D convolutional encoder to obtain a feature vector Then initialize the code book vector Z and in Z Find the nearest code for each code position and get the discrete variable z at the same latitude q , which means the intermediate feature Further discretization encoding is performed, and the expression is as follows:

[0014]

[0015] Finally, the discrete variable z q Decoding by 3D convolutional decoder to generate missing PET data The expression is as follows:

[0016]

[0017] Preferably, a mixed loss function is used when training the missing modality generation module, and the mixed loss function includes L1 loss Quantifying the loss Perceived loss and combat loss The hybrid loss function is defined as follows:

[0018]

[0019] Among them, λ L1 , Qua , Per and λ Adv are the weight parameters of each loss term.

[0020] Preferably, the visual encoder adopts a pre-trained 3D ResNet-50.

[0021] Preferably, in step 3, the MRI data is used as input to obtain MRI feature representation through a visual encoder, and the original PET data and the generated missing PET data are used as input to obtain PET feature representation through a visual encoder.

[0022] Preferably, in step 5, the method for implementing feature alignment using multimodal similarity matching loss is:

[0023]

[0024] in, For MRI data and clinical table data, For PET data and clinical table data, For MRI data and PET data, λ is the multimodal balance weight, which is set to 0.5 in this method.

[0025] Preferably, in step 6, the method for performing feature fusion through the common attention fusion module is:

[0026] First, each modal feature is processed through a linear layer to obtain the corresponding key matrix K i Sum value matrix V i At the same time, by connecting all modal features and mapping them to the linear layer, a multimodal query vector Q is generated multi , the attention score of each modality is calculated by the following formula:

[0027]

[0028] Among them, d i Represents vector K i The dimension of Represents the input features of each modality and uses softmax as the activation function.

[0029] Preferably, a joint loss function is used to optimize the entire network, combining the task-specific loss with the multimodal similarity matching loss. This overall loss function is expressed as a weighted sum:

[0030]

[0031] in, Refers to mission-specific losses.

[0032] Preferably, the classification head uses a pre-trained 1D DenseNet-121 as a classifier.

[0033] The present invention has the following characteristics and beneficial effects:

[0034] The above technical scheme is adopted, based on the incomplete trimodal co-attention fusion network, which is used for MCI conversion prediction. The model solves the problem of missing PET and combines MRI, PET and clinical table data to improve the accuracy of MCI conversion prediction with richer multimodal features. It includes a missing modality generation module and a co-attention fusion module, which are used to generate missing PET and trimodal feature fusion, respectively. In order to enhance the alignment of heterogeneous modal features, a loss function, trimodal similarity matching (multimodal similarity matching) loss, is introduced. The key advantages of the method of the present invention are summarized as follows: (1) The missing modality generation module synthesizes the missing PET images, solving the problem of multimodal data availability. (2) The co-attention fusion module can effectively learn and integrate multimodal imaging PET and MRI data with clinical table data, thereby enhancing the model's understanding of the task. (3) A new loss function is designed to alleviate the data imbalance problem and align feature representations between imaging and non-imaging data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 Schematic diagram of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0038] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0039] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0040] The meanings of the special terms and English abbreviations involved in this embodiment

[0041]

[0042]

[0043] The lack of demographic and clinical information and the inefficient cross-modal fusion mechanism may hinder the effective extraction of cross-modal fusion features. To solve this problem, a mild cognitive impairment conversion classification method based on incomplete trimodal co-attention fusion is proposed in the embodiment for the prognosis classification of cerebral hemorrhage, such as Figure 1 As shown, the specific steps are as follows:

[0044] Step 1: Acquire a data set, wherein the data set includes MRI data, PET data, and clinical table data.

[0045] Specifically, in this embodiment, the datasets are from the Alzheimer's Disease Neuroimaging Initiative (ADNI), specifically ADNI-1 and ADNI-2. To prevent duplication, subjects present in both datasets were deleted in ADNI-2. We selected T1-weighted structural MRI, FDG-PET, and clinical table data and divided them into four groups: normal control (NC), sMCI, pMCI, and AD. The demographic information of the dataset is shown in Table \ref{dataset}. In addition, PET data for 82 pMCI cases and 95 sMCI cases in ADNI-1 were missing, and PET data for 1 pMCI case and 30 sMCI cases in ADNI-2 were missing.

[0046] It is understandable that in order to ensure the accuracy of the classification results, the data set needs to be preprocessed in this embodiment.

[0047] All MRI images underwent preprocessing steps including intensity normalization, skull stripping, and normalization to MNI space. FDG-PET images were intensity normalized, normalized to MNI space, and co-registered with MRI.

[0048] For clinical table data, seven attributes were selected in this example: age, gender, education, ApoE4 status, P-tau 181, T-tau, and summary measurements of 18F-FDG PET imaging. The demographic information of the data used in the experiment can be found in Table 1.

[0049] Table 1: Experimental data demographic information statistics

[0050]

[0051] Step 2: Apply a pre-trained missing modality generation module and input MRI data to generate missing PET data through cross-modality conversion.

[0052] Specifically, the missing mode generation module structure is shown in Figure 1 (b), this module uses a simplified vector quantization generative adversarial network (VQ-GAN). Specifically, the MRI data is processed by a 3D convolutional encoder (CNN encoder) to obtain the feature vector Then, initialize Z and in Z Find the nearest code for each code position of and get the discrete variable z of the same dimension q , which means the intermediate feature Further discretization coding is performed.

[0053] Among them, Z is the codebook in VQGAN, which contains a set of discrete vectors.

[0054] Specifically, this process can be described as follows:

[0055]

[0056] Next, the variable z q Decoded by 3D convolutional decoder (CNN decoder) to generate PET images The process is as follows:

[0057]

[0058] By introducing vector quantization technology, the uncertainty in the generation process is effectively reduced, thereby improving the stability and quality of the generated data.

[0059] In order to optimize the training of the module, a hybrid loss function is given in this embodiment, which combines multiple loss terms: L1 loss Quantifying the loss Perceived loss and combat loss These loss terms work together to enhance the local authenticity of the generated PET images. The hybrid loss function is defined as follows:

[0060]

[0061] Among them, λ L1 , Qua , Per and λ Adv are the weight parameters of each loss term. Through the reasonably designed encoding, decoding process and hybrid loss function, the problem of missing PET data is effectively solved, and the utilization rate of multimodal data and the reliability of generated images are improved.

[0062] Step 3: Apply a feature extraction module to obtain a table feature representation through a table encoder, and obtain a visual feature representation through a visual encoder, wherein the visual feature representation includes an MRI feature representation and a PET feature representation. In this embodiment, the visual encoder uses a pre-trained 3D ResNet-50.

[0063] Specifically, the MRI data is used as input to obtain the MRI feature representation through a visual encoder, and the original PET data and the generated missing PET data are used as input to obtain the PET feature representation through a visual encoder.

[0064] Step 4: The extracted multimodal features are aggregated through a multi-head self-attention module to obtain trimodal features;

[0065] Step 5: Input the trimodal features into a co-attention fusion module for feature fusion to obtain multimodal fusion features.

[0066] Specifically, the structure diagram of the common attention fusion module can be found in Figure 1 (c) This module processes each modal feature through a linear layer to obtain the corresponding K i and V i , and generate a multimodal query vector Q by connecting all modal features and mapping them to a linear layer multi The attention score of each modality is calculated by the following formula:

[0067]

[0068] Among them, d i Represents vector K i The dimension of Represents the input features of each modality and uses softmax as the activation function.

[0069] It should be noted that K i and V i Represents the K matrix and V matrix required for calculation of mode i. The three modal features will have three sets of corresponding K and V matrices.

[0070] In order to fully integrate the intrinsic information of MRI and PET, a cross-connection method is used to connect them with clinical characteristics to generate the final output. The calculation formula is as follows:

[0071]

[0072] Step 6: Apply a pre-trained classification head and input the obtained multimodal fusion features into the classification head to achieve the final classification.

[0073] Finally, the pre-trained 1D DenseNet-121 is used as the classifier. The use of pre-trained parameters speeds up the convergence of the model and reduces the risk of overfitting.

[0074] During the training process of the model back propagation, the trimodal features are aligned through the multimodal similarity matching loss. Specifically, in order to alleviate the impact of data imbalance, the focal loss is used as the task loss. In addition, the multimodal similarity matching (SDM) loss is used to align the trimodal features. For MRI and tabular data, For PET and tabular data, For MRI and PET, λ is the multimodal balance weight, which is set to 0.5 in this method. The loss is defined as follows:

[0075]

[0076] A joint loss function is used to optimize the entire network, combining the task-specific loss with the multimodal alignment loss. This overall loss function is expressed as a weighted sum:

[0077]

[0078] Comparative Example

[0079] In this embodiment, a variety of indicators are used to evaluate the effectiveness and robustness of the ITCFN model, including accuracy (ACC), sensitivity (SEN), specificity (SPE), area under the curve (AUC) and F1 score (F1).

[0080] To ensure that the results are reproducible and comparable, 5-fold cross validation was used in all experiments to verify the stability and generalization ability of the model, while maintaining a consistent random seed for data splitting. We used the PyTorch 2.0 framework for experiments, using a single NVIDIAA100 80GB GPU for computational efficiency. The model was trained from scratch in two different stages, each containing 200 epochs with a batch size of 8 to manage the data efficiently. The Adam algorithm was used to optimize the model parameters, and the learning rate was set to 0.0001 to ensure precise adjustments during training.

[0081] Comparative experimental results

[0082]

[0083] Table 2 shows the comparison results of this embodiment with other different methods. The HOPE method achieved certain recognition performance by utilizing a single modality (MRI) feature associated with the progression of Alzheimer's disease (AD). Although ResNet performed well, it had difficulties in combining multimodal data, which limited its performance under limited and unbalanced data. The JSRL method solved the problem of missing modalities, but performed poorly on small data sets, probably due to its direct splicing of modalities. In contrast, the VPAL method enhances feature representation by integrating heterogeneous data. Finally, HFBSurv uses a factorized bilinear model to effectively fuse trimodal features, allowing the gradual fusion of multimodal data, thereby achieving superior performance.

[0084] The embodiment method of the present invention (ITCHN) outperforms other methods in all indicators. On the ADNI1 dataset, the framework of the present invention achieved an ACC of 0.947, an SPE of 0.949, a SEN of 0.944, an AUC of 0.946, and an F1 of 0.944, which were 0.026, 0.045, 0.007, 0.026, and 0.028 higher than the best results of other methods, respectively. Similarly, on the ADNI2 dataset, the method of the present invention achieved an ACC of 0.960, an SPE of 0.992, a SEN of 0.937, an AUC of 0.965, and an F1 of 0.960, which were 0.006, 0.015, 0.028, 0.022, and 0.028 higher than the best results of other methods, respectively.

[0085] Table 3: Ablation experiment results

[0086]

[0087]

[0088] In this embodiment, the effectiveness of the missing modality generation module and the co-attention fusion module in solving the problems of missing data and modality fusion was evaluated. In the ADNI1 dataset, missing data is an important challenge. The missing modality generation module enhances the model's ability to recognize pMCI, increasing the sensitivity SEN by 0.014. The co-attention fusion module effectively integrates information from different modalities, improving ACC by 0.043 and AUC by 0.027. The data imbalance problem in the ADNI2 dataset is serious. The missing modality generation module generated 30 cases of sMCI, but only 1 case of pMCI, which aggravated the imbalance. The co-attention fusion module achieved the best performance when used alone, increasing ACC by 0.019 and AUC by 0.033. The ablation experiment results are shown in Table 3.

[0089] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A conversion classification method for mild cognitive impairment based on incomplete trimodal co-attention fusion, characterized in that: The steps include: Step 1: Acquire a data set, wherein the data set includes MRI data, PET data, and clinical table data; Step 2: Apply a pre-trained missing modality generation module to generate missing PET data by inputting MRI data through cross-modality conversion; Step 3, applying a feature extraction module, obtaining a table feature representation through a table encoder, and obtaining a visual feature representation through a visual encoder, wherein the visual feature representation includes an MRI feature representation and a PET feature representation; Step 4: The extracted multimodal features are aggregated through a multi-head self-attention module to obtain trimodal features; Step 5: Input the trimodal features into a co-attention fusion module for feature fusion to obtain multimodal fusion features; Step 6: Apply a pre-trained classification head and input the obtained multimodal fusion features into the classification head to achieve the final classification.

2. According to claim 1, the mild cognitive impairment conversion classification method based on incomplete trimodal co-attention fusion is characterized in that: The missing mode generation module adopts a vector quantization generative adversarial network.

3. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 2 is characterized in that: The method for generating missing PET data is as follows: taking MRI data as input, firstly passing it through a 3D convolutional encoder to obtain a feature vector Then initialize the code book vector Z and in Z Find the nearest code for each code position and get the discrete variable z at the same latitude q , which means the intermediate feature Further discretization encoding is performed, and the expression is as follows: Finally, the discrete variable z q Decoding by 3D convolutional decoder to generate missing PET data The expression is as follows:

4. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 2 or 3, characterized in that: When the missing modality generation module is trained, a mixed loss function is used, and the mixed loss function includes L1 loss Quantifying the loss Perceived loss and combat loss The hybrid loss function is defined as follows: Among them, λ L1 , Qua , Per and λ Adv are the weight parameters of each loss term.

5. According to claim 1, the method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion is characterized in that: The visual encoder uses a pre-trained 3D ResNet-50.

6. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 5 is characterized in that: In step 3, the MRI data is used as input to obtain MRI feature representation through a visual encoder, and the original PET data and the generated missing PET data are used as input to obtain PET feature representation through a visual encoder.

7. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 1, characterized in that: In step 5, before feature fusion, the method of using multimodal similarity matching loss to achieve feature alignment is: in, For MRI data and clinical table data, For PET data and clinical table data, For MRI data and PET data, λ is the multimodal balance weight.

8. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 1, characterized in that: In step 6, the method for performing feature fusion through the common attention fusion module is: First, the features of each modality are processed through a linear layer to obtain the key matrix K required for attention calculation i Sum value matrix V i , and generate a multimodal query vector Q by connecting all modal features and mapping them to a linear layer multi , the attention score of each modality is calculated by the following formula: Among them, d i Represents vector K i The dimension of Represents the input features of each modality and uses softmax as the activation function.

9. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 7 is characterized in that: The joint loss function is used to optimize the overall network composed of steps 3-6, combining the specific task loss with the multimodal similarity matching loss. This overall loss function is expressed as a weighted sum: in, Refers to task-specific losses.

10. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 1, characterized in that: The classification head uses the pre-trained 1D DenseNet-121 as the classifier.

Images