Method for constructing Alzheimer's disease classification model based on multi-modal feature fusion

Abstract

The invention discloses a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion, and relates to the field of medical image processing. The problems that in the prior art, an Alzheimer's disease diagnosis method based on deep learning is insufficient in multi-modal data utilization, high in model complexity, high in medical cost and the like are solved. According to the method, sMRI images and clinical scale data serve as input data, the 3D MLP-Mixer obtains image detail information and space information, the GRU is used for extracting behavior cognition information, the ACF module is used for fusing the behavior cognition information and the image information extracted by the 3D MLP-Mixer, and therefore the patient recognition capacity and the cross-dataset generalization performance of the model are improved. The method is also suitable for the field of Alzheimer's disease image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion. BACKGROUND

[0002] Alzheimer's disease is a typical neurodegenerative disease with characteristics of insidious onset and irreversibility. Current clinical diagnosis is highly dependent on brain magnetic resonance imaging, such as structural MRI, and cognitive scale scores. Traditional diagnostic methods have the problems of strong subjectivity, low efficiency, high cost, etc. With the development of deep learning, multi-modal data fusion has become an important means to improve the accuracy and efficiency of diagnosis, especially the combination of structural MRI images and cognitive behavior scale information for joint modeling. The existing methods still have problems such as complex model, high reasoning cost, and insufficient information utilization in multi-modal fusion. SUMMARY

[0003] The present application proposes an end-to-end lightweight and efficient deep learning model, namely GA-GMixer model, to solve the problems of insufficient utilization of multi-modal data, high model complexity, and high medical cost in the prior art Alzheimer's disease diagnosis method based on deep learning. To solve the above technical problems, the present application is realized by the following technical scheme: Scheme one, the present application proposes a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion, the method comprises the following steps: Step 1, obtaining the data in sMRI image, social demographic characteristics, mini-mental state scale MMSE and clinical dementia rating scale CDR from the Alzheimer's disease neuroimaging dataset, and preprocessing, based on the preprocessed data, constructing the multi-modal dataset required for the Alzheimer's disease classification model, the Alzheimer's disease neuroimaging dataset includes ADNI-1 dataset, ADNI-2 dataset and ADNI-3 dataset; Step 2, dividing the sMRI image obtained in step 1 into a training set and a test set; Step 3, constructing a 3D MLP-Mixer model based on the sMRI image divided in step 2, for extracting global detail information and spatial neighborhood information of the sMRI image; Step 4, constructing a gated recurrent unit GRU based on the data in social demographic characteristics, mini-mental state scale MMSE and clinical dementia rating scale CDR obtained in step 1, for capturing the association information and complex cross-modal relationship between different scale items or scores; Step 5, constructing an attention-guided co-occurrence fusion module ACF, obtaining the final fusion features based on the 3D MLP-Mixer model of step 3 and the gated recurrent unit GRU of step 4. Step 6, constructing a GA-GMixer model, combining the 3D MLP-Mixer model of step 3, the gated recurrent unit GRU of step 4, and the fusion module ACF of step 5, and completing two classification tasks of AD vs. NC and sMCI vs. pMCI through a classifier; Step 7, training and verifying the GA-GMixer model on the Alzheimer's disease neuroimaging dataset, and performing generalization verification on the ADNI-2 dataset and the ADNI-3 dataset, and finally obtaining an Alzheimer's disease classification model.

[0004] Further, a preferred embodiment is provided, wherein the preprocessing in step 1 comprises the steps of converting the data format before preprocessing into the NIfTI format, AC-PC commissure, intensity correction, skull stripping, registration to the MNI space, and resampling to 121x145x121 pixels.

[0005] Further, a preferred embodiment is provided, wherein the method for dividing the sMRI image obtained in step 1 into a training set and a test set in step 2 is that: the sample data of the ADNI-1 dataset is processed by using a five-fold cross-validation method, and the ratio of the training set to the test set is 8:2.

[0006] Further, a preferred embodiment is provided, wherein the method for constructing the 3D MLP-Mixer model in step 3 is that: The 3D sMRI image with a size of 121x145x121 after preprocessing is uniformly divided into patches with a size of 25x25x25, a linear layer is used to embed the features of each input patch into a 2D feature matrix, and the obtained feature matrix is placed into the GNN-Mixer module and the Mixer Layers module for transformation.

[0007] Further, a preferred embodiment is provided, wherein the method for placing the obtained feature matrix into the GNN-Mixer module and the Mixer Layers module for transformation is that: the GNN-Mixer module is added to the GNN Layer module to combine it with the token-mixer module in the Mixer Layer module, and the characteristics of the message passing mechanism of the GNN are used to enhance the interaction of the model to local information.

[0008] Further, a preferred embodiment is provided, wherein the GNN-Mixer module and the Mixer Layers module are further provided with a skip connection for transmitting low-level basic features to a deep layer.

[0009] Further, a preferred embodiment is provided, and the method for constructing the gated recurrent unit GRU in step 4 is: controlling the information flow through the gating mechanism realized by the reset gate and the update gate, extracting the time series features of the data in the social demographic characteristics, the mini-mental state examination MMSE and the clinical dementia rating CDR row coding, and generating a hidden state vector representing global time dependence.

[0010] Further, a preferred embodiment is provided, and the method for constructing the attention-guided co-occurrence fusion module ACF in step 5 is: Step 5.1, import subunit: used to construct the basic library of the multi-modal fusion module, including torch, torch.nn, torch.nn.functional, and import UMixer and gated recurrent unit GRU Model class as subnetworks to realize feature acquisition of two modalities of image and scale; Step 5.2, define the FusionModel class, which combines the UMixer image branch and the gated recurrent unit GRU scale branch when initialized, and defines the full connection classification layer after fusion; Step 5.3, in the forward propagation, the MRI image is input into the UMixer to obtain the image features of (batch, 256), and the scale is input into the gated recurrent unit GRU Model to obtain the scale features of (batch, 256); Step 5.4, based on the image and scale features obtained in step 5.3, initialize a zero matrix of (256, 256) size as a co-occurrence matrix, and traverse the samples in the batch, and add the outer product of the image and scale features to the co-occurrence matrix to count the feature co-occurrence relationship between the two modalities; Step 5.5, based on the co-occurrence matrix generated in step 5.4, construct a trainable attention weight matrix, and use softmax in the column dimension to normalize to obtain the attention distribution aligned between modalities; Step 5.6, use the attention weight obtained in step 5.5 to weight the image features and map them to the scale feature space; at the same time, weight the scale features and map them to the image feature space to realize bidirectional feature interaction.

[0011] Step 5.7, concatenate the two sets of weighted features obtained in step 5.6 in the channel dimension to form (batch, 512), input the full connection layer to map to the class number dimension 2, and output the final classification result.

[0012] Scheme two, a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the steps of the method of scheme one.

[0013] A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program stored in the memory, the processor executes the method of the first aspect.

[0014] The present application has the advantages of: The construction method of the Alzheimer's disease classification model based on multi-modal feature fusion combines the 3DMLP-Mixer model with the GNN-Mixer module and the skip connection, can utilize the MLP-Mixer module to capture long-distance dependence to extract global information, and can enhance the detail information and spatial neighborhood information through the GNN, and comprehensively excavate the biological information of the sMRI image.

[0015] The gating recurrent unit GRU can effectively capture the time dependence relationship between different scale items or scores through the gating mechanism of the reset gate and the update gate, and can capture deeper associated information and complex cross-modal relationships after combining image features.

[0016] The co-occurrence fusion module ACF fuses the co-occurrence matrix and the attention mechanism, can statically represent the inter-modal dependence relationship through the co-occurrence matrix, and dynamically adjust the feature weight by means of the attention mechanism, realizes flexible and efficient fusion of multi-modal features, and improves the model diagnosis performance.

[0017] The GA-GMixer model proposed in the present application takes the sMRI and scale data of the ADNI data set as input, realizes high-accuracy recognition in the Alzheimer's disease classification task, has low parameter quantity and calculation amount, strong generalization ability, and takes into account high accuracy and low cost, and has significant clinical application value.

[0018] The present application is suitable for the field of early diagnosis of Alzheimer's disease and mild cognitive impairment conversion prediction. BRIEF DESCRIPTION OF DRAWINGS

[0019] Fig. 1 The flowchart of the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion in the eleventh embodiment.

[0020] Fig. 2 The flowchart of the preprocessing in the eleventh embodiment.

[0021] Fig. 3 The flowchart of constructing the co-occurrence fusion module ACF based on attention guidance in the eleventh embodiment. DETAILED DESCRIPTION

[0022] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application.

[0023] Embodiment one, the embodiment proposes a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion, the method comprising the following steps: Step 1, obtaining data in sMRI images, social demographic characteristics, mini-mental state quantity MMSE and clinical dementia rating scale CDR from an Alzheimer's disease neuroimaging dataset, and preprocessing, based on the preprocessed data, constructing a multi-modal dataset required for an Alzheimer's disease classification model, the Alzheimer's disease neuroimaging dataset comprising an ADNI-1 dataset, an ADNI-2 dataset and an ADNI-3 dataset; Step 2, dividing the sMRI images obtained in step 1 into a training set and a test set; Step 3, constructing a 3D MLP-Mixer model based on the sMRI images divided in step 2, for extracting global detail information and spatial neighborhood information of the sMRI images; Step 4, constructing a gated recurrent unit GRU based on the data in the social demographic characteristics, the mini-mental state quantity MMSE and the clinical dementia rating scale CDR obtained in step 1, for capturing associated information and complex cross-modal relationships between different scale items or scores; Step 5, constructing an attention-guided co-occurrence fusion module ACF, obtaining final fusion features based on the 3D MLP-Mixer model in step 3 and the gated recurrent unit GRU in step 4; Step 6, constructing a GA-GMixer model, combining the 3D MLP-Mixer model in step 3, the gated recurrent unit GRU in step 4 and the fusion module ACF in step 5, and completing two classification tasks of AD vs. NC and sMCI vs. pMCI through a classifier; Step 7, training and verifying the GA-GMixer model on the Alzheimer's disease neuroimaging dataset, and verifying the generalizability on the ADNI-2 dataset and the ADNI-3 dataset, and finally obtaining an Alzheimer's disease classification model.

[0024] Embodiment two, the embodiment is further limited to the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment one, the preprocessing in step 1 includes the steps of converting the data format before preprocessing into NIfTI format, AC-PC commissure, intensity correction, skull stripping, registration to MNI space and resampling to 121x145x121 pixels.

[0025] Embodiment three, the embodiment is further limited to the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment one, the method for dividing the sMRI image obtained in step 1 into a training set and a test set in step 2 is: using five-fold cross-validation to process the sample data of the ADNI-1 dataset, and the ratio of the training set to the test set is 8:2.

[0026] Embodiment four, the embodiment is further limited to the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment one, the method for constructing a 3D MLP-Mixer model in step 3 is: The 3D sMRI image with a size of 121x145x121 after preprocessing is uniformly divided into patches with a size of 25x25x25, a linear layer is used to embed the features of each input patch into a 2D feature matrix, and the obtained feature matrix is put into the GNN-Mixer module and the Mixer Layers module for transformation.

[0027] Embodiment five, the embodiment is further limited to the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment four, the method for transforming the obtained feature matrix in the GNN-Mixer module and the Mixer Layers module is: adding the GNN-Mixer module to the GNN Layer module to combine it with the token-mixer module in the MixerLayer module, and using the characteristics of the message passing mechanism of GNN to enhance the interaction of local information by the model.

[0028] Embodiment six, the embodiment is further limited to the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment four, the GNN-Mixer module and the Mixer Layers module are also provided with a skip connection for transmitting low-level basic features to deep layers.

[0029] Embodiment seven, the embodiment is further limited to the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment one, the method for constructing the gated recurrent unit GRU in step 4 is: controlling the information flow through the gating mechanism realized by the reset gate and the update gate, extracting the time sequence features of the data row coding in the social demographic characteristics, the mini-mental state MMSE and the clinical dementia rating scale CDR, and generating a hidden state vector representing global time dependence.

[0030] Embodiment eight, the embodiment is further limited to the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment one, the method for constructing the co-occurrence fusion module ACF based on attention guidance in step 5 is: Step 5.1, import subunit: used to construct the basic library of the multi-modal fusion module, including torch, torch.nn, torch.nn.functional, and import UMixer and gated recurrent unit GRU Model class as subnetworks to realize feature acquisition of two modalities of image and scale; Step 5.2, define the FusionModel class, combine the UMixer image branch and the gated recurrent unit GRU scale branch during initialization, and define the full connection classification layer after fusion; Step 5.3, in the forward propagation, input the MRI image into the UMixer to obtain the image features of (batch, 256), and input the scale into the gated recurrent unit GRU Model to obtain the scale features of (batch, 256); Step 5.4, based on the image and scale features obtained in step 5.3, initialize a zero matrix of (256, 256) size as a co-occurrence matrix, and traverse the samples in the batch, and add the outer product of the image and scale features to the co-occurrence matrix to count the feature co-occurrence relationship between the two modalities; Step 5.5, based on the co-occurrence matrix generated in step 5.4, construct a trainable attention weight matrix, and use softmax to normalize in the column dimension to obtain the attention distribution aligned between modalities; Step 5.6, use the attention weight obtained in step 5.5 to weight the image features and map them to the scale feature space; at the same time, weight the scale features and map them to the image feature space to realize bidirectional feature interaction.

[0031] Step 5.7, concatenate the two sets of weighted features obtained in step 5.6 in the channel dimension to form (batch, 512), input the full connection layer to map to the class number dimension 2, and output the final classification result.

[0032] Embodiment nine, the embodiment proposes a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize the steps of the method in any one of embodiments one to eight. Embodiment ten, the embodiment proposes a computer device, comprising a memory and a processor, the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the method in any one of embodiments one to eight.

[0033] Embodiment eleven, the embodiment proposes an embodiment for explaining the above-mentioned embodiments one to ten, which is specifically: Reference is made to Figs. 1-3 The embodiment is described with reference to Fig. 1 It is shown that the embodiment provides a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion, and the specific steps are as follows: Step 1, obtaining sMRI images, social demographic characteristics, mini-mental state quantity MMSE and data in clinical dementia rating scale CDR from Alzheimer's disease neuroimaging dataset ADNI, preprocessing and constructing multi-modal dataset.

[0034] Step 1.1, downloading 3D T1 weighted sMRI images of AD patients, NC (normal control), sMCI and pMCI subjects, and corresponding age, gender, MMSE score, CDR score and other clinical scale data from ADNI-1, ADNI-2 and ADNI-3 datasets.

[0035] Step 1.2, sMRI image preprocessing includes the following steps: Step 1.2.1, the first part of preprocessing, using SPM12 toolbox to convert DICOM format to NIfTI format; Step 1.2.2, the second part of preprocessing, AC-PC joint, head motion correction is performed on the image to eliminate the influence of subject head motion during data acquisition; Step 1.2.3, the third part of preprocessing, using CAT12 toolbox to remove non-brain tissue parts such as skull, scalp and eyeball; Step 1.2.4, the fourth part of preprocessing, spatial normalization, aligning the image to MNI152 standard template through CAT12 toolbox, and unifying to the same spatial coordinates; Step 1.2.5, the fifth part of preprocessing, using SPM12 toolbox to resample the image to a whole brain image of 121x145x121 pixels to ensure consistent input size.

[0036] Step 2, based on the pre-processed data set in step 1, the ADNI-1 data set is divided into five-fold cross-validation, 4 data as the training set, accounting for 80%, 1 data as the validation set, accounting for 20%; ADNI-2 and ADNI-3 are used to verify the generalization ability of the model across data sets; Step 3, based on the image data set divided in step 2, a 3D MLP-Mixer model is constructed to process sMRI data, extracting global information, detailed information and spatial neighborhood information of the image. The construction steps of the model are as follows: Step 3.1, import subunit: import the basic libraries required for constructing the image feature extraction module, including torch, torch.nn, torch.nn.functional and einops library, and introduce custom GNNLayer, FeedForward, MixerBlock and other submodules.

[0037] Step 3.2, define the GNNLayer submodule, which is used for message passing of graph structure at the patch level. It receives node features with shape (batch, patch number, channel number) (for example, batch=4, patch number=80, channel number=256) and adjacency matrix with shape (patch number, patch number), and realizes feature aggregation between nodes through matrix multiplication, then uses linear mapping to adjust the channel dimension, so as to explicitly model the spatial adjacency relationship in patch features.

[0038] Step 3.3, define the FeedForward submodule, which is used for token-mix or channel-mix operation in MLP-Mixer. It is composed of two fully connected layers and GELU activation, and has residual connection function, which can introduce nonlinear transformation to enhance the feature expression ability while keeping the original information.

[0039] Step 3.4, based on steps 3.2 and 3.3, 7 MixerBlock series modules are constructed. Each module first calls GNNLayer to realize information propagation between patches, then uses FeedForward for token mixing, then transposes the dimensions and uses FeedForward again for channel mixing, realizing bidirectional feature interaction across patches and channels.

[0040] Step 3.5, the UMixer main structure is defined, and to_patch_embedding is constructed in initialization, using three-dimensional convolution (Conv3d) with patch_size as the convolution kernel and stride to divide the input MRI body data (such as voxel data with a size of 121x145x121) into non-overlapping three-dimensional patches (patch number = 80). Then, through the Rearrange operation, the convolution output is rearranged into a patch sequence form of (batch, patch number, channel number) for subsequent encoder processing.

[0041] Step 3.6, based on the patch features obtained in step 3.5, three GNN-Mixer encoder layers (encoder_layer1, encoder_layer2, encoder_layer3) are constructed. The input channel dimension of these encoders is 256, and the patch number can be expanded to 512 in different layers. Adjacency matrix and GNNLayer are used to model spatial relationships, and FeedForward is combined for feature mixing, gradually improving global expression ability.

[0042] Step 3.7, based on the high-level features output by the encoder, a bottleneck2 bottleneck layer is added. This layer further integrates cross-patch information in the global range through the GNN-Mixer structure, so that the features have sufficient global perception ability before entering the decoding stage.

[0043] Step 3.8, based on the results of steps 3.6 and 3.7, three decoders (decoder_layer1, decoder_layer2, decoder_layer3) are constructed. Each decoder not only processes the input features, but also adds the output of the corresponding encoder layer to realize skip connection, in order to fuse multi-scale information and restore spatial details.

[0044] Step 3.9, based on the features output by the decoder, LayerNorm is applied for normalization, and mean pooling is performed in the patch dimension to obtain a global image feature vector with a shape of (batch, 256), which is the final output of the image branch.

[0045] Step 4, based on the social demographic characteristics and neurocognitive test scale data obtained in step 1, a gated recurrent unit (GRU) is constructed to process sequence data. The specific steps are as follows: Step 4.1, import the subunit to build the basic library of the scale feature extraction module, including torch and torch.nn.

[0046] Step 4.2, a gated recurrent unit GRU Model class is defined for temporal encoding of scale or clinical information. In the initialization, the input dimension (scale feature number 15), hidden layer dimension (256), output dimension (256), and the number of gated recurrent unit GRU layers (1 layer) are set, and the nn. Gated recurrent unit GRU instance is created, and batch_first=True ensures that the batch dimension of the input data is at the beginning.

[0047] Step 4.3, based on step 4.2, add a fully connected mapping layer (nn. Linear) to map the hidden state vector output by the gated recurrent unit GRU to the specified output dimension, so as to align the dimensions with the features of the image branch.

[0048] Step 4.4, in the forward propagation, the input scale data (shape batch x 1 x 15) is input into the gated recurrent unit GRU network, the hidden state of the last time step (shape batch x 256) is extracted, and the scale feature vector (batch, 256) is obtained through the fully connected layer as the output of the scale branch.

[0049] Step 5, based on the 3D MLP-Mixer model of step 3 and the gated recurrent unit GRU of step 4, an attention-guided co-occurrence fusion module ACF is constructed, the specific steps are as follows: Step 5.1, import subunit: the basic library for constructing multi-modal fusion module, including torch, torch.nn, torch.nn.functional, and introduce UMixer and gated recurrent unit GRU Model class as subnetworks to realize feature acquisition of image and scale two modalities.

[0050] Step 5.2, define the FusionModel class, which combines the UMixer image branch (output (batch, 256) image feature vector) and the gated recurrent unit GRU scale branch (output (batch, 256) scale feature vector) in initialization, and defines the fully connected classification layer after fusion (input dimension 512, output dimension is the number of categories 2).

[0051] Step 5.3, in the forward propagation, the MRI image is input into the UMixer to obtain the (batch, 256) image feature, and the scale is input into the gated recurrent unit GRU Model to obtain the (batch, 256) scale feature.

[0052] Step 5.4, based on the image obtained in step 5.3 and the scale features, initialize a (256, 256) size zero matrix as a co-occurrence matrix, and traverse the samples in the batch, and add the outer product of the image and the scale features to the co-occurrence matrix, thereby counting the feature co-occurrence relationship between the two modalities.

[0053] Step 5.5, based on the co-occurrence matrix generated in step 5.4, construct a trainable attention weight matrix, and use softmax for normalization in the column dimension to obtain the inter-modal alignment attention distribution.

[0054] Step 5.6, use the attention weight obtained in step 5.5 to weight the image features and map them to the scale feature space; at the same time, weight the scale features and map them to the image feature space to realize bidirectional feature interaction.

[0055] Step 5.7, concatenate the two sets of weighted features obtained in step 5.6 in the channel dimension to form (batch, 512), input the full connection layer to map to the class number dimension 2, and output the final classification result.

[0056] Step 6, based on the model constructed by the module group constructed in steps 3 to 5, five-fold cross-validation is used as the training set and validation set on the ADNI-1 dataset; ADNI-2 and ADNI-3 are used to verify the generalization ability of the model across datasets.

[0057] Step 6.1, import subunit: the basic library for training and verification, including torch, torch.nn, torch.utils.data, numpy, pandas, nibabel, etc., and introduce the FusionModel class as the core model for training.

[0058] Step 6.2, define the NiiDataset dataset class, which is used to load MRI images and Excel scale data simultaneously. In the initialization, nibabel.nib.load() reads the MRI image (NIfTI format, 121x145x121 voxels) and converts it to a numpy array using.get_fdata(), pandas.read_excel() reads the scale file and performs.dropna() to delete missing values, and LabelEncoder() is used to implement class encoding, pd.to_datetime() is used for time feature splitting to ensure that all columns are numerical types.

[0059] Step 6.3. In the getitem function, add a channel dimension to the MRI data and adjust the dimension order to match the Conv3D input format (final shape (1, D, H, W)), convert the scale data to a float32 tensor, and return the corresponding label (AD = 1, NC = 0).

[0060] Step 6.4. Split the dataset into training set (80%) and validation set (20%) in proportion, and use DataLoader to batch load data with batch_size = 4 and shuffle = True.

[0061] Step 6.5. Initialize the FusionModel, combine modules 1-3 into a complete network, and pass in image size (121, 145, 121), patch size (25, 25, 25), hidden dimension 256, etc.

[0062] Step 6.6. Define all model training using cross-entropy loss function, optimizer Adam, learning rate 0.0001, and 100 epochs. The depth of 3D MLP-Mixer (the number of Mixer Layers), dim, token_dim, and channel_dim are set to 7, 256, 256, and 2048, respectively.

[0063] Step 6.7. In the training phase, perform forward propagation to calculate loss for each batch, update parameters through backpropagation, and calculate training accuracy and average loss.

[0064] Step 6.8. In the validation phase, calculate loss and prediction results through forward propagation, and calculate accuracy, specificity, sensitivity, and AUC value as evaluation indicators.

[0065] Step 6.9. If the validation accuracy improves, save the current model weights and record the predicted labels and probabilities for subsequent analysis and visualization.

[0066] Step 6.10. Use cross-dataset validation method to verify the generalization ability of the model. Perform generalization experiments on ADNI2 and ADNI3. The parameter settings are consistent with the main experiment, and finally, calculate and output the relevant evaluation indicators to evaluate the generalization ability of the model.

[0067] The present application uses sMRI images and clinical scale data as input data, 3D MLP-Mixer to obtain image detail information and spatial information, uses GRU to extract behavioral cognitive information, and uses ACF to fuse behavioral cognitive information and image information extracted by 3D MLP-Mixer, thereby improving the recognition ability of the model for patients and the generalization performance across datasets.

[0068] The technical solutions of the present application and their technical effects are further described below in combination with specific experimental data.

[0069] To verify the effectiveness of each component in the proposed model GA-GMixer, we conducted an ablation experiment on the ADNI-1 dataset. As shown in Table 1, the accuracy of the final model GA-GMixer reached 96.96% and 82.27%, respectively.

[0070] To comprehensively evaluate the feasibility and economy of the model described in the present embodiment in practical application, the present embodiment investigates the computational complexity of the GA-GMixer model. The number of parameters (Params) and the number of floating point operations per second (floating point operations per second) of the model are counted. And GA-GMixer is compared with three-dimensional classification methods 3D ResNet-50, 3D DenseNet121 and ViT-B widely used in early diagnosis of AD, and the comparison results are shown in Table 2.

[0071] To verify the generalization ability of the GA-GMixer model, the cross-dataset evaluation method is used to evaluate the AD vs. NC prediction experiment on the ADNI-1, ADNI-2 and ADNI-3 datasets. This section designs two groups of experiments: the first group of experiments takes the ADNI-1 dataset as the training set, and takes the ADNI-1, ADNI-2 and ADNI-3 as the test set respectively; the second group of experiments takes the ADNI-2 dataset as the training set, and takes the ADNI-3 as the test set. Table 3 shows the evaluation indexes of the generalization performance experiment of the GA-GMixer model.

[0072] Table 1 Ablation experiment data of the effectiveness of each component module of the GA-GMixer model

[0073] Table 2 Statistical data of the computational complexity of the GA-GMixer model in two classification tasks on the ADNI-1 dataset

[0074] Table 3 Evaluation indexes of the generalization performance experiment of the GA-GMixer model

[0075] Those skilled in the art can understand that the above description is only the preferred embodiment of the present application, and the features described in various embodiments and / or technical solutions of the present disclosure can be combined or combined, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, and those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements for part of the technical features, any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0076] Although preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various modifications and changes to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and changes of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include these modifications and changes.

Claims

1. A method for constructing an Alzheimer's disease classification model based on multi-modal feature fusion, characterized in that, The method comprises the following steps: Step 1, obtaining data in sMRI images, social demographic characteristics, mini-mental state examination (MMSE) and clinical dementia rating (CDR) from an Alzheimer's disease neuroimaging dataset, and preprocessing, and constructing a multi-modal dataset required for an Alzheimer's disease classification model based on the preprocessed data, wherein the Alzheimer's disease neuroimaging dataset comprises ADNI-1 dataset, ADNI-2 dataset and ADNI-3 dataset; Step 2, dividing the sMRI images obtained in step 1 into a training set and a test set; Step 3, constructing a 3D MLP-Mixer model based on the sMRI images divided in step 2, for extracting global detail information and spatial neighborhood information of the sMRI images; Step 4, constructing a gated recurrent unit (GRU) based on the data in the social demographic characteristics, MMSE and CDR obtained in step 1, for capturing associated information and complex cross-modal relationships between different scale items or scores; Step 5, constructing an attention-guided co-occurrence fusion module (ACF) based on the 3D MLP-Mixer model in step 3 and the GRU in step 4 to obtain final fusion features; Step 6, constructing a GA-GMixer model by combining the 3D MLP-Mixer model in step 3, the GRU in step 4 and the fusion module ACF in step 5, and completing two classification tasks of AD vs. NC and sMCI vs. pMCI through a classifier; Step 7, training and verifying the GA-GMixer model on the Alzheimer's disease neuroimaging dataset, and verifying the generalization on the ADNI-2 dataset and the ADNI-3 dataset, and finally obtaining an Alzheimer's disease classification model.

2. The method of claim 1, wherein the method is characterized by: The preprocessing in step 1 comprises the steps of converting the data format before preprocessing into NIfTI format, AC-PC commissure, intensity correction, skull stripping, registration to MNI space and resampling to 121x145x121 pixels. 3.The method of claim 1, wherein, In step 2, the sMRI images obtained in step 1 are divided into a training set and a test set by using a five-fold cross-validation method to process sample data of the ADNI-1 dataset, and the ratio of the training set to the test set is 8:

2.

4. The method of claim 1, wherein the method further comprises: determining a plurality of features of the plurality of modalities; and determining a plurality of weights of the plurality of modalities. In step 3, the method for constructing the 3D MLP-Mixer model is as follows: The preprocessed 3D sMRI image with a size of 121x145x121 is uniformly divided into patches with a size of 25x25x25, a linear layer is used to embed the features of each input patch into a 2D feature matrix, and the obtained feature matrix is put into a GNN-Mixer module and a Mixer Layers module for transformation.

5. The method of claim 4, wherein the method further comprises: The method for transforming the obtained feature matrix into the GNN-Mixer module and the Mixer Layers module is: adding the GNN-Mixer module to the GNN Layer module, combining it with the token-mixer module in the Mixer Layer module, and using the characteristics of the message passing mechanism of the GNN to enhance the interaction of local information of the model.

6. The method of claim 4, wherein the method further comprises: The GNN-Mixer module and the Mixer Layers module are also provided with a skip connection for transmitting low-level basic features to a deep layer.

7. The method of claim 1, wherein the method further comprises: determining a plurality of features of the plurality of modalities; and determining a plurality of weights of the plurality of modalities. The method for constructing the gating recurrent unit GRU in step 4 is: controlling the information flow through the gating mechanism realized by the reset gate and the update gate, extracting the time series features of the data rows in the social demographic characteristics, the mini-mental state examination MMSE and the clinical dementia rating scale CDR, and generating a hidden state vector representing global time dependence. 8.The method of claim 1, wherein, The method for constructing the co-occurrence fusion module ACF based on attention guidance in step 5 is: Step 5.1, import subunit: used to construct the basic library of the multi-modal fusion module, including torch, torch.nn, torch.nn.functional, and import UMixer and GRUModel classes as subnetworks to realize feature acquisition of two modalities of images and scales; Step 5.2, the FusionModel class is defined, which is combined with the UMixer image branch and the GRU scale branch during initialization, and a full connection classification layer after fusion is defined; Step 5.3, in the forward propagation, the MRI image is input into the UMixer to obtain the image features of (batch, 256), and the scale is input into the GRUModel to obtain the scale features of (batch, 256); Step 5.4, based on the image and scale features obtained in step 5.3, a zero matrix of (256, 256) size is initialized as a co-occurrence matrix, and the samples in the batch are traversed, and the outer product of the image and scale features is calculated and added to the co-occurrence matrix, which is used to count the feature co-occurrence relationship between the two modalities; Step 5.5, based on the co-occurrence matrix generated in step 5.4, a trainable attention weight matrix is constructed, and softmax is used for normalization in the column dimension to obtain the attention distribution aligned between the modalities; Step 5.6, the attention weight obtained in step 5.5 is used to weight the image features and map them to the scale feature space; at the same time, the scale features are weighted and mapped to the image feature space, realizing bidirectional feature interaction; Step 5.7, the two sets of weighted features obtained in step 5.6 are concatenated in the channel dimension to form (batch, 512), which is input into the full connection layer to map to the class number dimension 2, and the final classification result is output.

9. A computer storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the method of any one of claims 1-8.

10. A computer device, comprising: Comprise: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the method of any one of claims 1-8.

Images