Construction method of alzheimer's disease classification model based on multi-modal feature fusion

Abstract

The application relates to a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion, and relates to the field of medical image processing.The method solves the problems of insufficient utilization of multi-modal data, high model complexity and high medical cost in the prior art Alzheimer's disease diagnosis method based on deep learning; the sMRI image and the clinical scale data are used as input data, the 3D MLP-Mixer is used to acquire image detail information and spatial information, the gated recurrent unit (GRU) is used to extract behavior cognitive information, the ACF module is used to fuse the behavior cognitive information and the image information extracted by the 3D MLP-Mixer, so that the recognition ability of the model to patients and the generalization performance of the model across data sets are improved.The application is also suitable for the field of Alzheimer's disease image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion. BACKGROUND

[0002] Alzheimer's disease is a typical neurodegenerative disease with the characteristics of insidious onset and irreversibility. At present, the clinical diagnosis is highly dependent on brain magnetic resonance imaging, such as structural MRI, and cognitive scale score information. The traditional diagnosis method has the problems of strong subjectivity, low efficiency, high cost and the like. With the development of deep learning, multi-modal data fusion has become an important means to improve the diagnosis accuracy and efficiency, especially the combination of structural MRI images and cognitive behavior scale information for joint modeling. The existing method still has problems of complex model, high reasoning cost, insufficient information utilization and the like in multi-modal fusion. SUMMARY

[0003] The present application proposes an end-to-end lightweight and efficient deep learning model, namely GA-GMixer model, to solve the problems of insufficient utilization of multi-modal data, high model complexity and high medical cost in the prior art Alzheimer's disease diagnosis method based on deep learning. To solve the above technical problems, the present application is realized by the following technical scheme:

[0004] Scheme one, the present application proposes a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion, which comprises the following steps:

[0005] Step 1, obtaining the data in sMRI images, social demographic characteristics, mini-mental state scale MMSE and clinical dementia rating scale CDR from an Alzheimer's disease neuroimaging dataset, and preprocessing, based on the preprocessed data, constructing a multi-modal dataset required for an Alzheimer's disease classification model, wherein the Alzheimer's disease neuroimaging dataset comprises ADNI-1 dataset, ADNI-2 dataset and ADNI-3 dataset;

[0006] Step 2, dividing the sMRI images obtained in step 1 into a training set and a test set;

[0007] Step 3, constructing a 3D MLP-Mixer model based on the sMRI images divided in step 2, for extracting global detail information and spatial neighborhood information of the sMRI images;

[0008] Step 4, constructing a gated recurrent unit GRU based on the data in social demographic characteristics, mini-mental state scale MMSE and clinical dementia rating scale CDR obtained in step 1, for capturing the associated information and complex cross-modal relationship between different scale items or scores;

[0009] Step 5, constructing an attention guide-based co-occurrence fusion module ACF, based on the 3D MLP-Mixer model of step 3 and the gated recurrent unit GRU of step 4 to obtain the final fusion features;

[0010] Step 6, constructing a GA-GMixer model, combining the 3D MLP-Mixer model of step 3, the gated recurrent unit GRU of step 4, and the fusion module ACF of step 5, and completing the two classification tasks of AD vs. NC and sMCI vs. pMCI through a classifier;

[0011] Step 7, training and verifying the GA-GMixer model on the Alzheimer's disease neuroimaging dataset, and performing generalization verification on the ADNI-2 dataset and the ADNI-3 dataset, and finally obtaining an Alzheimer's disease classification model.

[0012] Further, a preferred embodiment is provided, wherein the preprocessing in step 1 includes the steps of converting the data format before preprocessing to NIfTI format, AC-PC commissure, intensity correction, skull stripping, registration to MNI space, and resampling to 121x145x121 pixels.

[0013] Further, a preferred embodiment is provided, wherein the method for dividing the sMRI image obtained in step 1 into a training set and a test set in step 2 is to process the sample data of the ADNI-1 dataset using a five-fold cross-validation method, and the ratio of the training set to the test set is 8:2.

[0014] Further, a preferred embodiment is provided, wherein the method for constructing a 3D MLP-Mixer model in step 3 is:

[0015] The preprocessed 3D sMRI image with a size of 121x145x121 is uniformly divided into patches with a size of 25x25x25, a linear layer is used to embed the features of each input patch into a 2D feature matrix, and the obtained feature matrix is placed into the GNN-Mixer module and the Mixer Layers module for transformation.

[0016] Further, a preferred embodiment is provided, wherein the method for placing the obtained feature matrix into the GNN-Mixer module and the Mixer Layers module for transformation is to add the GNN-Mixer module to the GNN Layer module to combine it with the token-mixer module in the Mixer Layer module, and to enhance the interaction of the model with local information by using the characteristics of the message passing mechanism of GNN.

[0017] Further, a preferred embodiment is provided, wherein the GNN-Mixer module is further provided with a skip connection in the Mixer Layers module, for transmitting low-level basic features to deep layers.

[0018] Further, a preferred embodiment is provided, wherein the method for constructing the gated recurrent unit GRU in step 4 is as follows: the information flow is controlled through the gating mechanism realized by the reset gate and the update gate, the time series features are extracted by encoding the data rows in the social demographic characteristics, the mini-mental state examination MMSE and the clinical dementia rating scale CDR, and a hidden state vector representing global time dependence is generated.

[0019] Further, a preferred embodiment is provided, wherein the method for constructing the co-occurrence fusion module ACF based on attention guidance in step 5 is as follows:

[0020] Step 5.1, import subunit: used to construct the basic library of the multi-modal fusion module, including torch, torch.nn, torch.nn.functional, and import UMixer and gated recurrent unit GRU Model class as subnetworks to realize feature acquisition of two modalities of image and scale;

[0021] Step 5.2, define the FusionModel class, which combines the UMixer image branch and the gated recurrent unit GRU scale branch during initialization, and defines the full connection classification layer after fusion;

[0022] Step 5.3, in the forward propagation, the MRI image is input into the UMixer to obtain the image features of (batch, 256), and the scale is input into the gated recurrent unit GRU Model to obtain the scale features of (batch, 256);

[0023] Step 5.4, based on the image and scale features obtained in step 5.3, a zero matrix of (256, 256) size is initialized as a co-occurrence matrix, and the samples in the batch are traversed, and the outer product of the image and scale features is calculated and added to the co-occurrence matrix, which is used to count the feature co-occurrence relationship between the two modalities;

[0024] Step 5.5, based on the co-occurrence matrix generated in step 5.4, a trainable attention weight matrix is constructed, and softmax is used for normalization in the column dimension to obtain the attention distribution aligned between the modalities;

[0025] Step 5.6, the attention weight obtained in step 5.5 is used to weight the image features and map them to the scale feature space; at the same time, the scale features are weighted and mapped to the image feature space, realizing bidirectional feature interaction.

[0026] Step 5.7, the two groups of weighted features obtained in step 5.6 are spliced in the channel dimension to form (batch, 512), and input into a fully connected layer to map to a category number dimension 2, and output a final classification result.

[0027] Option two, a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize the steps of the method of option one.

[0028] Option three, a computer device, comprising a memory and a processor, the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the method of option one.

[0029] The present application has the advantages of:

[0030] The construction method of the Alzheimer's disease classification model based on multi-modal feature fusion disclosed by the application combines a 3DMLP-Mixer model with a GNN-Mixer module and a skip connection, can capture long-distance dependencies to extract global information through the MLP-Mixer module, and can enhance detail information and spatial neighborhood information through the GNN, and comprehensively excavate biological information of sMRI images.

[0031] The gated recurrent unit GRU disclosed by the application can effectively capture the time dependence between different scale items or scores through the gating mechanism of the reset gate and the update gate, and can capture deeper associated information and complex cross-modal relationships after combining image features.

[0032] The co-occurrence fusion module ACF disclosed by the application fuses a co-occurrence matrix and an attention mechanism, can statically represent the inter-modal dependence relationship through the co-occurrence matrix, and dynamically adjust the feature weight through the attention mechanism, realizes flexible and efficient fusion of multi-modal features, and improves the model diagnosis performance.

[0033] The GA-GMixer model proposed by the application takes sMRI and scale data of an ADNI data set as input, realizes high-accuracy recognition in the Alzheimer's disease classification task, has low parameter quantity and calculation amount, has strong generalization ability, balances high accuracy and low cost, and has significant clinical application value.

[0034] The application is suitable for the field of early diagnosis of Alzheimer's disease and mild cognitive impairment conversion prediction. BRIEF DESCRIPTION OF DRAWINGS

[0035] Fig. 1 The flowchart of the construction method of the Alzheimer's disease classification model based on multi-modal feature fusion in embodiment eleven.

[0036] Fig. 2 Flowchart for preprocessing in embodiment eleven.

[0037] Fig. 3 Flowchart for constructing the attention-guided co-occurrence fusion module ACF in embodiment eleven. DETAILED DESCRIPTION

[0038] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application.

[0039] Embodiment one, the embodiment proposes a method for constructing an Alzheimer's disease classification model based on multi-modal feature fusion, the method comprising the following steps:

[0040] Step 1, obtaining data in sMRI images, sociodemographic characteristics, mini-mental state quantity MMSE and clinical dementia rating scale CDR from an Alzheimer's disease neuroimaging dataset, and preprocessing, constructing a multi-modal dataset required for an Alzheimer's disease classification model based on the preprocessed data, the Alzheimer's disease neuroimaging dataset comprising ADNI-1 dataset, ADNI-2 dataset, ADNI-3 dataset;

[0041] Step 2, dividing the sMRI images obtained in step 1 into a training set and a test set;

[0042] Step 3, constructing a 3D MLP-Mixer model based on the sMRI images divided in step 2, for extracting global detail information and spatial neighborhood information of sMRI images;

[0043] Step 4, constructing a gated recurrent unit GRU based on the data in sociodemographic characteristics, mini-mental state quantity MMSE and clinical dementia rating scale CDR obtained in step 1, for capturing associated information and complex cross-modal relationships between different scale items or scores;

[0044] Step 5, constructing an attention-guided co-occurrence fusion module ACF, obtaining final fusion features based on the 3D MLP-Mixer model in step 3 and the gated recurrent unit GRU in step 4;

[0045] Step 6, constructing a GA-GMixer model, combining the 3D MLP-Mixer model in step 3, the gated recurrent unit GRU in step 4 and the fusion module ACF in step 5, and completing two classification tasks of AD vs. NC and sMCI vs. pMCI through a classifier;

[0046] Step 7, training and verifying the GA-GMixer model on the Alzheimer's disease neuroimaging dataset, and performing generalization verification on the ADNI-2 dataset and the ADNI-3 dataset, and finally obtaining an Alzheimer's disease classification model.

[0047] Embodiment II, the embodiment is a further limitation of the method for constructing an Alzheimer's disease classification model based on multi-modal feature fusion according to embodiment I, the preprocessing in step 1 includes the steps of converting the data format before preprocessing to NIfTI format, AC-PC commissure, intensity correction, skull stripping, registration to MNI space, and resampling to 121x145x121 pixels.

[0048] Embodiment III, the embodiment is a further limitation of the method for constructing an Alzheimer's disease classification model based on multi-modal feature fusion according to embodiment I, the method for dividing the sMRI image obtained in step 1 into a training set and a test set in step 2 is: using a five-fold cross-validation method to process the sample data of the ADNI-1 dataset, and the ratio of the training set to the test set is 8:2.

[0049] Embodiment IV, the embodiment is a further limitation of the method for constructing an Alzheimer's disease classification model based on multi-modal feature fusion according to embodiment I, the method for constructing a 3D MLP-Mixer model in step 3 is:

[0050] The preprocessed 3D sMRI image with a size of 121x145x121 is uniformly divided into patches with a size of 25x25x25, a linear layer is used to embed the features of each input patch into a 2D feature matrix, and the obtained feature matrix is placed into the GNN-Mixer module and the Mixer Layers module for transformation.

[0051] Embodiment V, the embodiment is a further limitation of the method for constructing an Alzheimer's disease classification model based on multi-modal feature fusion according to embodiment IV, the method for placing the obtained feature matrix into the GNN-Mixer module and the Mixer Layers module for transformation is: adding the GNN-Mixer module to the GNN Layer module to combine it with the token-mixer module in the MixerLayer module, and using the message passing mechanism of GNN to enhance the interaction of local information by the model.

[0052] Embodiment VI, the embodiment is a further limitation of the method for constructing an Alzheimer's disease classification model based on multi-modal feature fusion according to embodiment IV, the GNN-Mixer module and the Mixer Layers module are also provided with a skip connection for transmitting low-level basic features to deep layers.

[0053] Embodiment seven, the embodiment is further limited to the method for constructing the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment one, the method for constructing the gated recurrent unit GRU in step 4 is: controlling the information flow through the gating mechanism realized by the reset gate and the update gate, extracting the time series features of the data row coding in the social demographic characteristics, the mini-mental state MMSE and the clinical dementia rating scale CDR, and generating a hidden state vector representing global time dependence.

[0054] Embodiment eight, the embodiment is further limited to the method for constructing the Alzheimer's disease classification model based on multi-modal feature fusion described in embodiment one, the method for constructing the co-occurrence fusion module ACF based on attention guidance in step 5 is:

[0055] Step 5.1, import subunit: used to construct the basic library of the multi-modal fusion module, including torch, torch.nn, torch.nn.functional, and introduce UMixer and gated recurrent unit GRU Model class as subnetworks to realize feature acquisition of two modalities of image and scale;

[0056] Step 5.2, define the FusionModel class, which combines the UMixer image branch and the gated recurrent unit GRU scale branch when initialized, and defines the full connection classification layer after fusion;

[0057] Step 5.3, in the forward propagation, the MRI image is input into the UMixer to obtain the image features of (batch, 256), and the scale is input into the gated recurrent unit GRU Model to obtain the scale features of (batch, 256);

[0058] Step 5.4, based on the image and scale features obtained in step 5.3, initialize a zero matrix of size (256, 256) as a co-occurrence matrix, and traverse the samples in the batch, and add the outer product of the image and scale features to the co-occurrence matrix to count the feature co-occurrence relationship between the two modalities;

[0059] Step 5.5, based on the co-occurrence matrix generated in step 5.4, construct a trainable attention weight matrix, and use softmax to normalize in the column dimension to obtain the attention distribution aligned between the modalities;

[0060] Step 5.6, using the attention weight obtained in step 5.5 to weight the image features and map them to the scale feature space; at the same time, the scale features are weighted and mapped to the image feature space to realize bidirectional feature interaction.

[0061] Step 5.7, the two groups of weighted features obtained in step 5.6 are spliced in the channel dimension to form (batch, 512), and input into a fully connected layer to map to a category number dimension 2, and output a final classification result.

[0062] Embodiment Nine, this embodiment provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize the steps of the method in any one of the embodiments one to eight.

[0063] Embodiment Ten, this embodiment provides a computer device, comprising a memory and a processor, the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the method in any one of the embodiments one to eight.

[0064] Embodiment Eleven, this embodiment provides an example for explaining the above-mentioned embodiments one to ten, and the example is specifically:

[0065] Reference Figs. 1-3 This embodiment is described with reference to Fig. 1 It is shown that the embodiment provides a construction method of an Alzheimer's disease classification model based on multi-modal feature fusion, and the specific steps are as follows:

[0066] Step 1, obtaining sMRI images, social demographic characteristics, mini-mental state quantity MMSE and data in clinical dementia rating scale CDR from Alzheimer's disease neuroimaging dataset ADNI, preprocessing and constructing a multi-modal dataset.

[0067] Step 1.1, downloading 3D T1 weighted sMRI images of AD patients, NC (normal control), sMCI and pMCI subjects, and corresponding age, gender, MMSE score, CDR score and other clinical scale data from ADNI-1, ADNI-2 and ADNI-3 datasets.

[0068] Step 1.2, sMRI image preprocessing includes the following steps:

[0069] Step 1.2.1, the first part of preprocessing, using SPM12 toolbox to convert DICOM format to NIfTI format;

[0070] Step 1.2.2, the second part of preprocessing, AC-PC joint, head motion correction is performed on the image to eliminate the influence of subject head motion during data acquisition;

[0071] Step 1.2.3, the third part of preprocessing, using CAT12 toolbox to remove non-brain tissue parts such as skull, scalp and eyeball;

[0072] Step 1.2.4, Preprocessing the fourth part, spatial normalization, registering the image to the MNI152 standard template through the CAT12 toolbox, and uniform to the same spatial coordinates;

[0073] Step 1.2.5, Preprocessing the fifth part, resampling the image to a whole brain image of 121x145x121 pixels using the SPM12 toolbox, ensuring consistent input size.

[0074] Step 2, based on the preprocessed data set in step 1, divide the data set, use five-fold cross-validation for ADNI-1 data set, 4 parts of data as training set, accounting for 80%, 1 part as validation set, accounting for 20%; ADNI-2 and ADNI-3 are used to verify the generalization ability of the model across data sets;

[0075] Step 3, based on the image data set divided in step 2, construct a 3D MLP-Mixer model to process sMRI data, extract image global information, detail information and spatial neighborhood information. The construction steps of the model are as follows:

[0076] Step 3.1, import subunit: import the required basic libraries for constructing image feature extraction module, including torch, torch.nn, torch.nn.functional and einops library, and introduce custom GNNLayer, FeedForward, MixerBlock and other submodules.

[0077] Step 3.2, define the GNNLayer submodule, which is used for message passing of graph structure at patch level. It receives node features with shape (batch, patch number, channel number) (for example, batch=4, patch number=80, channel number=256) and adjacency matrix with shape (patch number, patch number), and realizes feature aggregation between nodes through matrix multiplication, then uses linear mapping to adjust channel dimension, so as to explicitly model the spatial adjacency relationship in patch features.

[0078] Step 3.3, define the FeedForward submodule, which is used for token-mix or channel-mix operation in MLP-Mixer. It is composed of two fully connected layers and GELU activation, and has residual connection function, which can introduce nonlinear transformation to enhance feature expression ability while keeping the original information.

[0079] Step 3.4. Based on steps 3.2 and 3.3, seven MixerBlock series modules were constructed. Each module first calls GNNLayer to realize inter-patch information propagation, then uses FeedForward for token mixing, and then transposes the dimensions and uses FeedForward again for channel mixing, realizing bidirectional feature interaction across patches and channels.

[0080] Step 3.5. The UMixer main structure was defined, and to_patch_embedding was constructed in initialization. Three-dimensional convolution (Conv3d) was used with patch_size as the convolution kernel and step to divide the input MRI body data (such as voxel data with a size of 121x145x121) into non-overlapping three-dimensional patches (patch number = 80). Subsequently, through the Rearrange operation, the convolution output was rearranged into a patch sequence form of (batch, patch number, channel number) for subsequent encoder processing.

[0081] Step 3.6. Based on the patch features obtained in step 3.5, three GNN-Mixer encoder layers (encoder_layer1, encoder_layer2, encoder_layer3) were constructed. The input channel dimension of these encoders is 256, and the patch number can be expanded to 512 in different layers. Adjacency matrix and GNNLayer are used to realize spatial relationship modeling, and FeedForward is used for feature mixing to gradually improve the global expression ability.

[0082] Step 3.7. Based on the high-level features output by the encoder, a bottleneck2 bottleneck layer was added. This layer further integrates cross-patch information in the global range through the GNN-Mixer structure, so that the feature has sufficient global perception ability before entering the decoding stage.

[0083] Step 3.8. Based on the results of steps 3.6 and 3.7, three-level decoders (decoder_layer1, decoder_layer2, decoder_layer3) were constructed. Each decoder not only processes the input features, but also adds the output of the corresponding encoder layer to realize skip connection, in order to fuse multi-scale information and restore spatial details.

[0084] Step 3.9. Apply LayerNorm to normalize the features output by the decoder and perform mean pooling on the patch dimension to obtain a global image feature vector of shape (batch, 256) as the final output of the image branch.

[0085] Step 4. Based on the sociodemographic characteristics and neurocognitive test scale data obtained in step 1, a gated recurrent unit (GRU) is used to process sequence data. The specific steps are as follows:

[0086] Step 4.1. Import the subunit to build the base library for the scale feature extraction module, including torch and torch.nn.

[0087] Step 4.2. Define a gated recurrent unit (GRU) Model class for time series encoding of scale or clinical information. In the initialization, set the input dimension (scale feature number 15), hidden layer dimension (256), output dimension (256), and GRU layer number (1 layer), and create an nn.GRU instance with batch_first=True to ensure that the batch dimension of the input data is at the beginning.

[0088] Step 4.3. Based on step 4.2, add a fully connected mapping layer (nn.Linear) to map the hidden state vector output by the GRU to the specified output dimension, so that the features of the image branch can be aligned in dimension.

[0089] Step 4.4. In the forward propagation, input the scale data (shape batch x 1 x 15) into the GRU network, extract the hidden state at the last time step (shape batch x 256), and pass it through the fully connected layer to obtain a scale feature vector of shape (batch, 256) as the output of the scale branch.

[0090] Step 5. Based on the 3D MLP-Mixer model in step 3 and the gated recurrent unit (GRU) in step 4, construct an attention-guided co-occurrence fusion module (ACF). The specific steps are as follows:

[0091] Step 5.1. Import the subunit: the base library for building the multi-modal fusion module, including torch, torch.nn, torch.nn.functional, and introduce the UMixer and GRU Model classes as subnetworks to realize feature acquisition of image and scale modalities.

[0092] Step 5.2, define the FusionModel class, initialize the UMixer image branch (output (batch, 256) image feature vector) and the gated recurrent unit GRU scale branch (output (batch, 256) scale feature vector) combined, and define the full connection classification layer after fusion (input dimension 512, output dimension is the number of categories 2).

[0093] Step 5.3, in the forward propagation, input the MRI image into the UMixer to get the (batch, 256) image feature, and input the scale into the gated recurrent unit GRU Model to get the (batch, 256) scale feature.

[0094] Step 5.4, based on the image and scale features obtained in step 5.3, initialize a zero matrix of size (256, 256) as the co-occurrence matrix, and traverse the samples in the batch, and add the outer product of the image and scale features to the co-occurrence matrix, so as to count the feature co-occurrence relationship between the two modalities.

[0095] Step 5.5, based on the co-occurrence matrix generated in step 5.4, construct a trainable attention weight matrix, and use softmax to normalize in the column dimension to get the attention distribution between modalities.

[0096] Step 5.6, use the attention weight obtained in step 5.5 to weight the image features and map them to the scale feature space; at the same time, weight the scale features and map them to the image feature space, realizing bidirectional feature interaction.

[0097] Step 5.7, concatenate the two sets of weighted features obtained in step 5.6 in the channel dimension to form (batch, 512), input the full connection layer to map to the category number dimension 2, and output the final classification result.

[0098] Step 6, based on the model constructed by the module group in steps 3 to 5, use five-fold cross-validation as the training set and validation set on the ADNI-1 dataset; ADNI-2 and ADNI-3 are used to verify the generalization ability of the model across datasets.

[0099] Step 6.1, import subunit: the basic library for training and verification, including torch, torch.nn, torch.utils.data, numpy, pandas, nibabel, etc., and introduce the FusionModel class as the core model for training.

[0100] Step 6.2, Define the NiiDataset dataset class for loading MRI images and Excel scale data simultaneously. In initialization, nibabel.nib.load() reads the MRI image (NIfTI format, 121x145x121 voxels) and converts it to a numpy array with.get_fdata(), pandas.read_excel() reads the scale file and performs.dropna() to remove missing values, LabelEncoder() is used for category encoding, and pd.to_datetime() is used for time feature splitting to ensure all columns are numerical types.

[0101] Step 6.3, In the getitem method, add a channel dimension to the MRI data and adjust the dimension order to match the Conv3D input format (final shape (1, D, H, W)), convert the scale data to a float32 tensor, and return the corresponding label (AD=1, NC=0).

[0102] Step 6.4, Split the dataset into training set (80%) and validation set (20%) in proportion, use DataLoader to batch load data, set batch_size=4 and shuffle=True.

[0103] Step 6.5, Initialize FusionModel, combine modules 1-3 into a complete network, and pass in image size (121, 145, 121), patch size (25, 25, 25), hidden dimension 256, etc.

[0104] Step 6.6, Define all model training using cross-entropy loss function, optimizer selection Adam, learning rate set to 0.0001, 100 epochs. The depth of 3D MLP-Mixer (number of Mixer Layers), dim, token_dim, channel_dim are set to 7, 256, 256, 2048 respectively.

[0105] Step 6.7, In the training phase, perform forward propagation to calculate loss for each batch, update parameters through backpropagation, and calculate training accuracy and average loss.

[0106] Step 6.8, In the validation phase, calculate loss and prediction results through forward propagation, and calculate accuracy, specificity, sensitivity, and AUC value as evaluation indicators.

[0107] Step 6.9, If the validation accuracy improves, save the current model weights and record the predicted labels and probabilities for subsequent analysis and visualization.

[0108] Step 6.10, the cross-dataset validation method is used to verify the generalization ability of the model. The generalization experiment is carried out in ADNI2 and ADNI3. The parameter settings are consistent with the main experiment, finally, the relevant evaluation indexes are calculated and output, which evaluates the generalization ability of the overall performance of the model.

[0109] The present application takes sMRI images and clinical scale data as input data, 3D MLP-Mixer obtains image detail information and spatial information, uses a gated recurrent unit GRU to extract behavioral cognitive information, and uses a co-occurrence fusion module ACF to fuse the behavioral cognitive information and the image information extracted by the 3D MLP-Mixer, thereby improving the recognition ability of the model to patients and the generalization performance across data sets.

[0110] The technical solutions and technical effects of the present application will be further described below in combination with specific experimental data.

[0111] To verify the effectiveness of each component in the proposed model GA-GMixer, we conducted an ablation experiment on the ADNI-1 dataset. As shown in Table 1, the accuracy of the final model GA-GMixer reached 96.96% and 82.27%, respectively.

[0112] To comprehensively evaluate the feasibility and economy of the model described in the embodiment in actual application, the computational complexity of the GA-GMixer model is investigated. The number of parameters (Params) and the number of floating point operations per second (floating point operations per second) are counted. And the GA-GMixer is compared with the three-dimensional classification method 3D ResNet-50, 3D DenseNet121 and ViT-B widely used in AD early diagnosis, and the comparison results are shown in Table 2.

[0113] To verify the generalization ability of the GA-GMixer model, the cross-dataset evaluation method is used to perform AD vs. NC prediction experiments on the ADNI-1, ADNI-2 and ADNI-3 datasets. This section designs two groups of experiments: the first group of experiments takes the ADNI-1 dataset as the training set, and takes the ADNI-1, ADNI-2 and ADNI-3 as the test set respectively; the second group of experiments takes the ADNI-2 dataset as the training set, and takes the ADNI-3 as the test set. Table 3 shows the evaluation indexes of the generalization performance experiment of the GA-GMixer model.

[0114] Table 1 Ablation experiment data of the effectiveness of each component module of the GA-GMixer model

[0115]

[0116] Table 2 Statistics of computational complexity of GA-GMixer model in two classification tasks on ADNI-1 dataset

[0117]

[0118] Table 3 Evaluation metrics of GA-GMixer model generalization performance experiment

[0119]

[0120] Those skilled in the art can understand that the above description is merely preferred embodiments of the present application, and the features described in various embodiments and / or technical solutions of the present disclosure can be combined or combined, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, and those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements for part of the technical features, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0121] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended technical solutions are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various modifications and changes to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and changes of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include these modifications and changes.

Claims

1. A method for constructing an Alzheimer's disease classification model based on multimodal feature fusion, characterized in that, The method includes the following steps: Step 1: Obtain sMRI images, sociodemographic features, Mini-Mental State Examination (MMSE), and Clinical Dementia Rating Scale (CDR) data from the Alzheimer's disease neuroimaging dataset, and perform preprocessing. Based on the preprocessed data, construct the multimodal dataset required for the Alzheimer's disease classification model. The Alzheimer's disease neuroimaging dataset includes the ADNI-1 dataset, ADNI-2 dataset, and ADNI-3 dataset. Step 2: Divide the sMRI images obtained in Step 1 into training and test sets; Step 3: Construct a 3D MLP-Mixer model based on the sMRI images segmented in Step 2, which is used to extract global detail information and spatial neighborhood information of the sMRI images; Step 4: Based on the sociodemographic characteristics, Mini-Mental State Examination (MMSE), and Clinical Dementia Rating Scale (CDR) data obtained in Step 1, construct a gated recurrent unit (GRU) to capture the correlation information and complex cross-modal relationships between different scale items or scores. Step 5: Construct the attention-guided co-occurrence fusion module ACF, and obtain the final fusion features based on the 3D MLP-Mixer model in Step 3 and the gated recurrent unit GRU in Step 4. Step 6: Construct the GA-GMixer model, combining the 3D MLP-Mixer model from Step 3, the Gated Recurrent Unit (GRU) from Step 4, and the Co-occurrence Fusion Module (ACF) from Step 5, and complete the classification task through the classifier. Step 7: Train and validate the GA-GMixer model on the Alzheimer's disease neuroimaging dataset, and perform generalization validation on the ADNI-2 and ADNI-3 datasets to finally obtain the Alzheimer's disease classification model. The method for constructing the 3D MLP-Mixer model in step 3 is as follows: The preprocessed 3D sMRI image with a size of 121×145×121 is uniformly divided into patches of size 25×25×25. A linear layer is used to embed the features of each input patch into a 2D feature matrix. The resulting feature matrix is ​​then fed into the GNN-Mixer module and the Mixer Layers module for transformation. The method for transforming the obtained feature matrix into the GNN-Mixer module and Mixer Layers module is as follows: add the GNN-Mixer module into the GNN Layer module and combine it with the token-mixer module in the Mixer Layer module to enhance the model's interaction with local information by utilizing the message passing mechanism of GNN. The GNN-Mixer module and Mixer Layers module also have skip connections to pass low-level basic features to deeper layers. A three-level GNN-Mixer encoder layer is constructed, consisting of three GNNLayer sub-modules. The input channel dimension of the GNN-Mixer encoder is 256, and the number of patches is extended to 512 in different layers. Spatial relationship modeling is achieved using the adjacency matrix and GNNLayer sub-modules, and feature mixing is performed in conjunction with FeedForward. The GNNLayer submodule is used for message passing of graph structures at the patch level. It receives node features of shape (batch, number of patches, number of channels) and an adjacency matrix of shape (number of patches, number of patches). It aggregates features between nodes through matrix multiplication and then uses linear mapping to adjust the channel dimension, thereby explicitly modeling spatial adjacency relationships in the patch features. That is, each module first calls GNNLayer to realize information propagation between patches, then uses FeedForward to mix tokens, then transposes the dimensions and mixes channels again through FeedForward, realizing bidirectional feature interaction across patches and across channels; Based on the high-level features output by the encoder, a bottleneck layer 2 is added. This bottleneck layer further fuses cross-patch information at the global level through the GNN-Mixer module, so that the high-level features have sufficient global perception capabilities before entering the decoding stage. A three-level decoder is constructed. Each decoder not only processes the input features, but also adds them to the output of the corresponding encoder layer to achieve skip connections, so as to fuse multi-scale information and restore spatial details. The method for constructing the attention-guided co-occurrence fusion module (ACF) in step 5 is as follows: Step 5.1, Import Sub-units: The basic library used to build the multimodal fusion module, including torch, torch.nn, torch.nn.functional, and introduces UMixer and GRUModel classes as sub-networks to realize feature acquisition of both image and scale modalities; Step 5.2: The FusionModel class is defined. During initialization, the UMixer image branch and the GRU scale branch are combined, and the fused fully connected classification layer is defined. Step 5.3: In forward propagation, input the MRI images into UMixer to obtain image features of (batch, 256), and input the scale into GRUModel to obtain scale features of (batch, 256). Step 5.4: Based on the image and scale features obtained in Step 5.3, initialize a zero matrix of size (256, 256) as a co-occurrence matrix, and iterate through the samples in the batch, adding the extrapolation of image and scale features to the co-occurrence matrix to statistically analyze the feature co-occurrence relationship between the two modalities. Step 5.5: Based on the co-occurrence matrix generated in Step 5.4, construct a trainable attention weight matrix and normalize it using softmax in the column dimension to obtain the intermodal aligned attention distribution; Step 5.6: Use the attention weights obtained in Step 5.5 to weight the image features and map them to the scale feature space; at the same time, weight the scale features and map them to the image feature space to achieve bidirectional feature interaction. Step 5.7: Concatenate the two sets of weighted features obtained in Step 5.6 in the channel dimension to form (batch, 512), input the fully connected layer to map to the number of categories dimension 2, and output the final classification result.

2. The method for constructing an Alzheimer's disease classification model based on multimodal feature fusion according to claim 1, characterized in that, The preprocessing described in step 1 includes converting the data format before preprocessing to NIfTI format, AC-PC merging, intensity correction, skull dissection, registration to MNI space, and resampling to 121×145×121 pixels.

3. The method for constructing an Alzheimer's disease classification model based on multimodal feature fusion according to claim 1, characterized in that, In step 2, the method for dividing the sMRI images obtained in step 1 into training and test sets is as follows: the sample data of the ADNI-1 dataset is processed using a five-fold cross-validation method, and the ratio of the training set to the test set is 8:

2.

4. The method for constructing an Alzheimer's disease classification model based on multimodal feature fusion according to claim 1, characterized in that, The method for constructing the gated recurrent unit (GRU) in step 4 is as follows: information flow is controlled by a gating mechanism jointly implemented by a reset gate and an update gate. The time series features are extracted by encoding the data rows in the sociodemographic features, the Mini-Mental State Examination (MMSE), and the Clinical Dementia Rating Scale (CDR), and a hidden state vector representing global time dependence is generated.

5. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-4.

6. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-4.

Images