Mild cognitive impairment conversion prediction method based on incomplete three-modal co-attention fusion

Abstract

The application discloses a mild cognitive impairment conversion classification method based on incomplete three-modal co-attention fusion, comprising the following steps: acquiring a data set; step 2, applying a pre-trained missing modality generation module, inputting MRI data to generate missing PET data through cross-modal conversion; step 3, applying a feature extraction module, obtaining table feature representation through a table encoder, and obtaining visual feature representation through a visual encoder; step 4, the extracted multi-modal features are aggregated through a multi-head self-attention module to obtain three-modal features; step 5, the three-modal features after feature alignment are input into a co-attention fusion module for feature fusion, thereby obtaining multi-modal fusion features; step 6, applying a pre-trained classification head, inputting the obtained multi-modal fusion features into the classification head to realize final classification. The method uses richer multi-modal features to improve the accuracy of MCI conversion prediction.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for predicting mild cognitive impairment conversion based on incomplete trimodal co-attention fusion. Background Art

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease with a very high mortality rate that leads to permanent cognitive impairment. Mild cognitive impairment (MCI) is one of the early stages of AD. Some MCI patients will develop AD within three years, which is called progressive MCI (pMCI), while patients with stable MCI (sMCI) remain cognitively stable during this period. Accurately predicting the progression of pMCI is crucial for early intervention, as early intervention can help delay the development of AD. Clinically, brain imaging methods such as magnetic resonance imaging (MRI) and positron emission tomography (PET) are used to assess the patient's brain status and assist doctors in diagnosis. However, this process relies on the doctor's experience and subjective judgment, resulting in high uncertainty and time-consuming diagnostic results. However, PET data are often missing due to the high cost and the use of radioactive tracers, limiting the application of multimodal methods. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, the present invention proposes a mild cognitive impairment conversion prediction method based on incomplete trimodal co-attention fusion, which solves the problem of PET missing and combines MRI, PET and clinical table data to improve the accuracy of MCI conversion prediction with richer multimodal features.

[0004] In order to solve the above technical problems, the technical solution of the present invention is:

[0005] A conversion classification method for mild cognitive impairment based on incomplete trimodal co-attention fusion includes the following steps:

[0006] Step 1: Acquire a data set, wherein the data set includes MRI data, PET data, and clinical table data;

[0007] Step 2: Apply a pre-trained missing modality generation module to generate missing PET data by inputting MRI data through cross-modality conversion;

[0008] Step 3: applying a feature extraction module to obtain a table feature representation through a table encoder and a visual feature representation through a visual encoder, wherein the visual feature representation includes an MRI feature representation and a PET feature representation;

[0009] Step 4: The extracted multimodal features are aggregated into trimodal features through a multi-head self-attention module;

[0010] Step 5, inputting the three-modal feature into a co-attention fusion module for feature fusion to obtain a multi-modal fusion feature;

[0011] Step 6, applying a pre-trained classification head to input the obtained multi-modal fusion feature into the classification head to realize final classification.

[0012] Preferably, the missing modal generation module adopts a vector quantization generative adversarial network.

[0013] Preferably, the method for generating missing PET data is: taking MRI data as input, first passing through a 3D convolutional encoder to obtain a feature vector Then, initializing a codebook vector Z, and finding the nearest neighbor code for each encoding position of to obtain the same latitude discrete variable z q , which represents further discrete coding of the intermediate feature , and the expression is as follows:

[0014]

[0015] Finally, the obtained discrete variable z q is decoded by a 3D convolutional decoder to generate missing PET data , and the expression is as follows:

[0016]

[0017] Preferably, when the missing modal generation module is trained, a hybrid loss function is adopted, and the hybrid loss function includes L1 loss quantization loss perception loss and adversarial loss The hybrid loss function is defined as follows:

[0018]

[0019] Wherein, λ L1 , λ Qua , λ Per and λ Adv are weight parameters of each loss term.

[0020] Preferably, the visual encoder adopts a pre-trained 3D ResNet-50.

[0021] Preferably, in step 3, the MRI data is taken as input to obtain an MRI feature representation through a visual encoder, and the original PET data and the generated missing PET data are taken as input to obtain a PET feature representation through a visual encoder.

[0022] As preferred, in step 5, the method for feature alignment using multi-modal similarity matching loss is as follows:

[0023]

[0024] Wherein, For MRI data and clinical table data, For PET data and clinical table data, For MRI data and PET data, λ is the multi-modal balance weight, which is set to 0.5 in this method.

[0025] As preferred, in step 6, the method for feature fusion through the co-attention fusion module is as follows:

[0026] First, the features of each modality are processed through a linear layer to obtain the corresponding key matrix K i and value matrix V i At the same time, by connecting all the modality features and mapping them to a linear layer, a multi-modal query vector Q multi is generated, and the attention score of each modality is calculated by the following formula:

[0027]

[0028] Wherein, d i represents the dimension of the vector K i , represents the input features of each modality, and softmax is used as the activation function.

[0029] As preferred, the joint loss function is used to optimize the entire network, combining the specific task loss with the multi-modal similarity matching loss, and the overall loss function is represented as a weighted sum:

[0030]

[0031] Wherein, refers to the specific task loss.

[0032] As preferred, the classification head uses a pre-trained 1D DenseNet-121 as the classifier.

[0033] The present application has the following characteristics and beneficial effects:

[0034] The above technical solution is adopted, based on an incomplete trimodal co-attention fusion network, which is used for MCI conversion prediction. The model solves the problem of missing PET and combines MRI, PET and clinical table data to improve the accuracy of MCI conversion prediction with richer multimodal features. It includes a missing modality generation module and a co-attention fusion module, which are used to generate missing PET and trimodal feature fusion, respectively. In order to enhance the alignment of heterogeneous modal features, a loss function, trimodal similarity matching (multimodal similarity matching) loss, is introduced. The key advantages of the method of the present invention are summarized as follows: (1) The missing modality generation module synthesizes the missing PET images, solving the problem of multimodal data availability. (2) The co-attention fusion module can effectively learn and integrate multimodal imaging PET and MRI data with clinical table data, thereby enhancing the model's understanding of the task. (3) A new loss function is designed to alleviate the data imbalance problem and align feature representations between imaging and non-imaging data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 Schematic diagram of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0038] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0039] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0040] The meanings of the special nouns and English abbreviations involved in this embodiment

[0041]

[0042]

[0043] The lack of demographic and clinical information and the inefficient cross-modal fusion mechanism may hinder the effective extraction of cross-modal fusion features. To solve this problem, a conversion classification method for mild cognitive impairment based on incomplete trimodal co-attention fusion is proposed in the embodiment. Figure 1 The specific steps are as follows:

[0044] Step 1: Acquire a data set, which includes MRI data, PET data, and clinical table data.

[0045] Specifically, in this example, the datasets were from the Alzheimer's Disease Neuroimaging Initiative (ADNI), specifically ADNI-1 and ADNI-2. To prevent duplication, subjects present in both datasets were deleted from ADNI-2. We selected T1-weighted structural MRI, FDG-PET, and clinical table data, which were divided into four groups: normal control (NC), sMCI, pMCI, and AD. The demographic information of the datasets is shown in Table \ref{dataset}. In addition, PET data were missing for 82 pMCI cases and 95 sMCI cases in ADNI-1, and for 1 pMCI case and 30 sMCI cases in ADNI-2.

[0046] It is understandable that in order to ensure the accuracy of the classification results, the data set needs to be preprocessed in this embodiment.

[0047] All MRI images underwent preprocessing steps, including intensity normalization, skull stripping, and normalization to MNI space. FDG-PET images were intensity normalized, normalized to MNI space, and co-registered with MRI.

[0048] For the clinical table data, seven attributes were selected in this example: age, sex, education, ApoE4 status, P-tau 181, T-tau, and summary measurements of 18F-FDG PET imaging. The demographic information of the data used in the experiment can be found in Table 1.

[0049] Table 1: Experimental data demographic information statistics

[0050]

[0051] Step 2: Apply a pre-trained missing modality generation module and input MRI data to generate missing PET data through cross-modality conversion.

[0052] Specifically, the missing mode generation module structure is shown in Figure 1 (b), this module uses a simplified vector quantization generative adversarial network (VQ-GAN). Specifically, the MRI data is processed by a 3D convolutional encoder (CNN encoder) to obtain the feature vector Then, initialize Z and in Z Each encoding position of finds its nearest neighboring encoding and obtains the discrete variable z of the same dimension q , which represents the intermediate features Further discretization coding is performed.

[0053] Among them, Z is the codebook in VQGAN, which contains a set of discrete vectors.

[0054] Specifically, this process can be described as follows:

[0055]

[0056] Next, the variable z q Decoded by a 3D convolutional decoder (CNN decoder) to generate PET images The process is as follows:

[0057]

[0058] By introducing vector quantization technology, the uncertainty in the generation process is effectively reduced, thereby improving the stability and quality of the generated data.

[0059] To optimize the training of the module, a hybrid loss function is given in this embodiment, which combines multiple loss terms: L1 loss Quantifying Losses Perceptual loss and combat losses These loss terms work together to enhance the local realism of the generated PET images. The hybrid loss function is defined as follows:

[0060]

[0061] Among them, λ L1 ,λ Qua ,λ Per and λ Adv are the weight parameters of each loss term. Through the rational design of encoding and decoding processes and hybrid loss functions, the problem of missing PET data is effectively solved, and the utilization rate of multimodal data and the reliability of generated images are improved.

[0062] Step 3: Apply a feature extraction module to obtain a table feature representation through a table encoder and obtain a visual feature representation through a visual encoder, wherein the visual feature representation includes an MRI feature representation and a PET feature representation. In this embodiment, the visual encoder uses a pre-trained 3D ResNet-50.

[0063] Specifically, the MRI data is used as input to obtain MRI feature representation through a visual encoder, and the original PET data and the generated missing PET data are used as input to obtain PET feature representation through a visual encoder.

[0064] Step 4: The extracted multimodal features are aggregated into trimodal features through a multi-head self-attention module;

[0065] Step 5: Input the trimodal features into a co-attention fusion module for feature fusion to obtain multimodal fusion features.

[0066] Specifically, the structure diagram of the common attention fusion module is shown in Figure 1 (c) This module processes each modal feature through a linear layer to obtain the corresponding K i and V i , and generate a multimodal query vector Q by connecting all modal features and mapping them to a linear layer multi The attention score of each modality is calculated using the following formula:

[0067]

[0068] Among them, d i Represents vector K i Dimensions, Represents the input features of each modality and uses softmax as the activation function.

[0069] It should be noted that K i and V i Represents the K matrix and V matrix required for calculation of mode i. The three modal features will have three sets of corresponding K and V matrices.

[0070] In order to fully integrate the intrinsic information of MRI and PET, a cross-connection method is used to connect them with clinical features to generate the final output. The calculation formula is as follows:

[0071]

[0072] Step 6: Apply a pre-trained classification head and input the obtained multimodal fusion features into the classification head to achieve the final classification.

[0073] Finally, a pre-trained 1D DenseNet-121 is used as the classifier. The use of pre-trained parameters speeds up the convergence of the model and reduces the risk of overfitting.

[0074] During the back-propagation training of the model, the trimodal features are aligned using the multimodal similarity matching loss. Specifically, in order to alleviate the impact of data imbalance, the focus loss is used as the task loss. In addition, the multimodal similarity matching (SDM) loss is used to align the trimodal features. For MRI and tabular data, For PET and tabular data, For MRI and PET, λ is the multimodal balance weight, which is set to 0.5 in this method. The loss is defined as follows:

[0075]

[0076] A joint loss function is used to optimize the entire network, combining the task-specific loss with the multimodal alignment loss. This overall loss function is expressed as a weighted sum:

[0077]

[0078] Comparative Example

[0079] In this example, multiple indicators were used to evaluate the effectiveness and robustness of the ITCFN model, including accuracy (ACC), sensitivity (SEN), specificity (SPE), area under the curve (AUC), and F1 score (F1).

[0080] To ensure reproducible and comparable results, 5-fold cross-validation was employed across all experiments to verify model stability and generalization capabilities, while maintaining a consistent random seed for data splitting. We used the PyTorch 2.0 framework for our experiments, utilizing a single NVIDIA A100 80GB GPU for computational efficiency. The model was trained from scratch in two distinct phases, each consisting of 200 epochs, with a batch size of 8 to efficiently manage the data. Model parameters were optimized using the Adam algorithm, with a learning rate set to 0.0001 to ensure precise adjustments during training.

[0081] Comparative experimental results

[0082]

[0083] Table 2 shows the comparison results of this embodiment with other different methods. The HOPE method achieved certain recognition performance by utilizing a single modality (MRI) feature related to the progression of Alzheimer's disease (AD). Although ResNet performed well, it had difficulties in combining multimodal data, which limited its performance under limited and unbalanced data. The JSRL method solved the problem of missing modalities, but performed poorly on small datasets, probably due to its direct splicing of modalities. In contrast, the VPAL method enhanced feature representation by integrating heterogeneous data. Finally, HFBSurv used a factorized bilinear model to effectively fuse trimodal features, allowing the gradual fusion of multimodal data, thereby achieving superior performance.

[0084] The embodiment method of the present invention (ITCHN) outperforms other methods in all indicators. On the ADNI1 dataset, the framework of the present invention achieved an ACC of 0.947, an SPE of 0.949, a SEN of 0.944, an AUC of 0.946, and an F1 of 0.944, which were 0.026, 0.045, 0.007, 0.026, and 0.028 higher than the best results of other methods, respectively. Similarly, on the ADNI2 dataset, the method of the present invention achieved an ACC of 0.960, an SPE of 0.992, a SEN of 0.937, an AUC of 0.965, and an F1 of 0.960, which were 0.006, 0.015, 0.028, 0.022, and 0.028 higher than the best results of other methods, respectively.

[0085] Table 3: Ablation experiment results

[0086]

[0087] In this embodiment, the effectiveness of the missing modality generation module and the co-attention fusion module in solving the problems of missing data and modality fusion was evaluated. In the ADNI1 dataset, missing data is an important challenge. The missing modality generation module enhances the model's recognition ability for pMCI, increasing the sensitivity SEN by 0.014. The co-attention fusion module effectively integrates information from different modalities, improving ACC by 0.043 and AUC by 0.027. The data imbalance problem in the ADNI2 dataset is serious. The missing modality generation module generated 30 cases of sMCI, but only 1 case of pMCI, which exacerbated the imbalance. The co-attention fusion module achieved the best performance when used alone, increasing ACC by 0.019 and AUC by 0.033. The ablation experiment results are shown in Table 3.

[0088] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A conversion classification method for mild cognitive impairment based on incomplete trimodal co-attention fusion, characterized by: The steps include: Step 1: Acquire a data set, wherein the data set includes MRI data, PET data, and clinical table data; Step 2: Apply a pre-trained missing modality generation module to generate missing PET data by inputting MRI data through cross-modality conversion; The missing modality generation module adopts a simplified vector quantization generative adversarial network; The method for generating missing PET data is as follows: taking MRI data as input, first passing it through a 3D convolutional encoder to obtain a feature vector Then initialize the code book vector Z and in Z For each coding position of , find its nearest neighboring coding and get the discrete variable z at the same latitude. q , which represents the intermediate features Further discretization coding is performed, and the expression is as follows: Finally, the discrete variable z q Decoding is performed through a 3D convolutional decoder to generate the missing PET data The expression is as follows: When the missing modality generation module is trained, a hybrid loss function is used, which includes L1 loss Quantifying Losses Perceptual loss and combat losses The hybrid loss function is defined as follows: Among them, λ L1 ,λ Qua ,λ Per and λ Adv are the weight parameters of each loss term respectively; Step 3: applying a feature extraction module to obtain a table feature representation through a table encoder and a visual feature representation through a visual encoder, wherein the visual feature representation includes an MRI feature representation and a PET feature representation; Step 4: The extracted multimodal features are aggregated into trimodal features through a multi-head self-attention module; Step 5: Input the trimodal features into a co-attention fusion module for feature fusion to obtain multimodal fusion features; The method of feature fusion through the common attention fusion module is: First, the features of each modality are processed through the linear layer to obtain the key matrix K required for attention calculation i Sum matrix V i , and generate a multimodal query vector Q by connecting all modal features and mapping them to a linear layer multi , the attention score of each modality is calculated by the following formula: Among them, d i Represents vector K i Dimensions, Represents the input features of each modality, using softmax as the activation function; In order to fully integrate the intrinsic information of MRI and PET, a cross-connection method is used to connect them with clinical features to generate the final output. The calculation formula is as follows: Step 6: Apply a pre-trained classification head and input the obtained multimodal fusion features into the classification head to achieve the final classification.

2. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 1 is characterized in that: The visual encoder uses a pre-trained 3D ResNet-50.

3. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 2, characterized in that: In step 3, the MRI data is used as input to obtain MRI feature representation through a visual encoder, and the original PET data and the generated missing PET data are used as input to obtain PET feature representation through a visual encoder.

4. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 1 is characterized in that: In step 5, before performing feature fusion, the method for achieving feature alignment using multimodal similarity matching loss is as follows: in, For MRI data and clinical table data, For PET data and clinical table data, For MRI data and PET data, λ is the multimodal balance weight.

5. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 4 is characterized in that: The joint loss function is used to optimize the overall network composed of steps 3-6, combining the specific task loss with the multimodal similarity matching loss. The overall loss function is expressed as a weighted sum: in, Refers to mission-specific losses.

6. The method for converting and classifying mild cognitive impairment based on incomplete trimodal co-attention fusion according to claim 1, characterized in that: The classification head uses the pre-trained 1D DenseNet-121 as the classifier.

Images