Alzheimer's disease image diagnosis method based on multi-modal middle-deep state space model

Abstract

This invention discloses an image diagnosis method for Alzheimer's disease based on a multimodal deep state-space model, belonging to the fields of medical image analysis and artificial intelligence. Addressing the limitations of existing methods in fully extracting complementary information between structural and functional images and their high computational complexity, this invention extracts initial features from structural magnetic resonance imaging (MRI) and positron emission tomography (PET) scans using a voxel flow backbone. Multi-scale modeling is then performed using a hierarchical feature extraction backbone network. A mid-depth scanning mechanism enables content-aware adaptive scanning path optimization, and a multimodal interaction module achieves deep interactive fusion of structural and metabolic features through a bidirectional cross-modal attention mechanism, ultimately outputting a disease classification prediction. This invention effectively integrates complementary information from multimodal images, improving diagnostic accuracy while maintaining linear computational complexity, and can be used for early auxiliary diagnosis of Alzheimer's disease.

Description

Technical Field

[0001] This invention relates to the fields of medical image analysis and artificial intelligence technology, specifically to a multimodal automated diagnostic method for Alzheimer's disease that integrates structural magnetic resonance imaging and positron emission tomography. Background Technology

[0002] Alzheimer's disease is an irreversible neurodegenerative disease, and early diagnosis and intervention are crucial for slowing disease progression and improving patients' quality of life. Structural magnetic resonance imaging (MRI) provides high-resolution information on brain anatomy, sensitively capturing atrophy patterns in key brain regions such as the hippocampus and entorhinal cortex; while fluorodeoxyglucose positron emission tomography (PET) reveals abnormalities in brain glucose metabolism, detecting functional abnormalities that often precede structural changes. These two modalities provide complementary information on the pathology of Alzheimer's disease from both structural and functional perspectives.

[0003] Existing deep learning-based multimodal fusion methods, such as multimodal cross-attention networks and Alzheimer's disease Transformers, mostly employ simple cascading or summation operations, lacking deep modeling of complex cross-modal interactions and failing to fully exploit the nonlinear correlations between structural and metabolic features. Furthermore, the quadratic computational complexity of the Transformer architecture's self-attention mechanism limits its end-to-end processing capability for complete 3D images. While state-space models proposed in recent years, such as Mamba, have shown potential in long-range modeling with linear complexity, existing methods primarily focus on unimodal analysis and have not yet been specifically optimized for adaptive scanning and deep interactive fusion of multimodal neuroimages. Summary of the Invention

[0004] This invention provides an imaging diagnostic method for Alzheimer's disease based on a multimodal deep state space model, aiming to achieve high-precision automated diagnosis of Alzheimer's disease and related disease processes by deeply fusing structural magnetic resonance imaging and positron emission tomography images.

[0005] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0006] S1: Acquire and preprocess multimodal image data. Acquire raw structural magnetic resonance imaging (MRI) and fluorodeoxyglucose positron emission tomography (PET) images of the subjects. Perform spatial registration, skull removal, Gaussian smoothing, and intensity normalization on the two modal images respectively, and register them to standard brain space to obtain a standardized multimodal image dataset.

[0007] S2: Construct and train a multimodal mid-to-deep state space model. The model includes a dual-branch voxel flow backbone module, a hierarchical feature extraction backbone network based on mid-to-deep state space model blocks, and a multimodal interaction module.

[0008] S3: Initial feature extraction of the voxel flow backbone. The preprocessed structural magnetic resonance imaging and positron emission tomography images are respectively input into two structurally identical voxel flow backbone modules. Each voxel flow backbone is sequentially passed through a 3D convolutional layer, batch normalization, activation function, depthwise separable convolution, and adaptive pooling to extract the initial feature map. The feature maps output by the two modal branches are concatenated along the channel dimension to form a unified multimodal initial feature representation.

[0009] S4: Hierarchical Feature Extraction Backbone Network Modeling. The concatenated multimodal features are input into a hierarchical feature extraction backbone network consisting of multiple stages. Each stage contains several multimodal mid-deep state space model blocks and downsampling layers, gradually reducing the spatial resolution of the feature map and increasing the number of channels. The multimodal mid-deep state space model blocks adopt a pre-normalized residual design and include mid-deep state space model sub-modules and feedforward neural network sub-modules.

[0010] S5: Mid-level deep scanning mechanism for adaptive feature extraction. In the mid-deep state space model submodule, a mid-deep scanning mechanism is executed. This mechanism first predicts the spatial offset and scan index offset of each voxel using a lightweight offset network, which includes deep convolutional layers and channel attention layers. Then, trilinear interpolation sampling is performed on the original feature map based on the spatial offset to obtain a deformed feature map. Next, the flattened deformed feature sequence is reordered based on the index offset, allowing the network to prioritize voxels corresponding to key diagnostic brain regions. Finally, the reordered sequence is input into a selective state space model for global context modeling.

[0011] S6: The multimodal interaction module enables cross-modal feature fusion. The feature map output from the final stage of the hierarchical feature extraction backbone network is modally split into a structural magnetic resonance imaging (SMRI) branch and a positron emission tomography (PET) branch, which are then input into the multimodal interaction module. The multimodal interaction module contains two parallel and symmetrical paths: a metabolic feedback path uses PET features as queries and SMRI features as keys and values, and generates metabolically enhanced structural features through cross-modal attention calculation; a structure-guided path uses SMRI features as queries and PET features as keys and values, and generates structurally enhanced metabolic features. The enhanced feature maps output from the two paths are concatenated along the channel dimension and dimensionality reduced through three-dimensional convolution to obtain a fused feature map.

[0012] S7: Predict the probability of disease category using a classifier to obtain the image classification result. Global average pooling and flattening operations are sequentially performed on the fused feature map, and the classification prediction probability of Alzheimer's disease, mild cognitive impairment, or normal control is output through a fully connected layer.

[0013] Preferably, in the mid-depth scanning mechanism of step S5, the spatial offset prediction branch creates a reference grid, obtains deformable sampling points by adding predicted spatial offsets, and samples feature values ​​from the original feature map using trilinear interpolation; the scan index offset prediction branch flattens the three-dimensional feature map into a one-dimensional sequence, generates a content-aware scanning order by adding index offsets and reordering; the gradient of the sorting operation is approximately propagated by averaging the loss gradient along the sequence dimension and copying it to the index offset.

[0014] Preferably, in step S6, the multimodal interaction module employs a multi-head cross-modal attention mechanism for both its metabolic feedback path and structural guidance path, with 8 attention heads and the feature dimensions remaining the same as the input features. Figure 1 The metabolic feedback pathway and the structural guidance pathway use independent learnable weight matrices.

[0015] Preferably, the hierarchical feature extraction backbone network in step S4 includes four stages, and the number of deep state space model blocks in the multimodal mode in each stage is 2, 2, 9, and 2 respectively. Spatial downsampling is performed between stages through three-dimensional convolutional layers with a kernel size of 2.

[0016] Preferably, in step S3, the voxel flow backbone module uses depthwise separable convolution to decompose standard 3D convolution into two independent steps: depthwise convolution and pointwise convolution. This reduces the number of parameters compared to standard 3D convolution by approximately [amount missing]. times, of which Number of output channels This represents the kernel size.

[0017] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0018] (1) Improved diagnostic accuracy: This invention extracts bimodal initial features through a voxel flow backbone module, performs long-range dependency modeling through a hierarchical feature extraction backbone network, and then achieves deep interactive fusion of structural and metabolic features through a bidirectional cross-modal attention mechanism, significantly improving the classification accuracy of Alzheimer's disease and mild cognitive impairment. Experiments show that the accuracy of this invention reaches 97.62% and the area under the curve reaches 98.74% in the classification task of Alzheimer's disease and normal controls; the sensitivity reaches 93.34% in the classification task of normal controls and mild cognitive impairment.

[0019] (2) Enhanced sensitivity to early pathological features: The mid-depth scanning mechanism proposed in this invention, through a dual-branch structure of spatial offset and index offset, enables the network to dynamically adjust the scanning path according to the image content, adaptively focus on key brain regions for diagnosis such as the hippocampus and amygdala, and effectively capture early and subtle pathological changes.

[0020] (3) Maintaining linear computational complexity: The present invention constructs a backbone network based on a state-space model, which has the advantage of linear time complexity compared to the Transformer architecture; at the same time, the design of depth-separable convolution in the voxel flow backbone significantly reduces the number of parameters and computational overhead, making it suitable for resource-constrained clinical deployment scenarios.

[0021] (4) Good interpretability: The brain regions that the model focuses on in this invention are highly consistent with the clinical pathological markers of Alzheimer's disease, providing a reliable basis for clinical auxiliary diagnosis. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a specific implementation of the Alzheimer's disease imaging diagnostic method based on a multimodal deep state space model according to the present invention.

[0023] Figure 2 This is a diagram illustrating the overall architecture of the Alzheimer's disease imaging diagnostic method based on a multimodal deep state space model, as described in this invention.

[0024] Figure 3 This is a schematic diagram of the voxel flow backbone module structure of the present invention;

[0025] Figure 4 This is a schematic diagram of the deep state space model block structure in the multimodal mode of the present invention;

[0026] Figure 5 This is a schematic diagram of the mid-layer depth scanning mechanism of the present invention;

[0027] Figure 6 This is a schematic diagram of the multimodal interaction module structure of the present invention; Detailed Implementation

[0028] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. To better illustrate this embodiment, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It will be understood by those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0029] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following description is provided in conjunction with the appendix. Figure 2 The following is a detailed description of the implementation method, structure, features and efficacy of an Alzheimer's disease imaging diagnostic method based on a multimodal deep state space model proposed in accordance with the present invention, and its preferred embodiments.

[0030] Example

[0031] This embodiment provides an imaging diagnostic method for Alzheimer's disease based on a multimodal deep state-space model. The following is in conjunction with the appendix... Figure 1Each step is explained in detail.

[0032] S1: Multimodal Image Data Acquisition and Preprocessing. The structural magnetic resonance imaging and fluorodeoxyglucose positron emission tomography (PET) image data used in this embodiment were both obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) public database. Image data from 1252 subjects were screened from ADNI-1 and ADNI-2 phases, covering three population groups: normal controls (NC), mild cognitive impairment (MCI), and Alzheimer's disease (AD).

[0033] The preprocessing workflow was completed in the Statistical Parametric Mapping Toolbox (SPM12) of the MATLAB platform, including: converting the original DICOM format to Analyze format; performing head motion correction; registering the images to the Montreal Neuroscience Institute (MNI) standard brain space through affine and nonlinear transformations, unifying the image dimensions to 121×145×121 and the voxel spatial resolution to 1.5mm×1.5mm×1.5mm; removing the skull and retaining only the brain tissue region; smoothing the images using a Gaussian kernel with a full width at half maximum (FWHM) of 8mm to improve the signal-to-noise ratio; correcting for intensity inhomogeneities in structural magnetic resonance imaging (MRI); and normalizing the intensity of positron emission tomography (PET) scans.

[0034] S2: Data Splitting and Augmentation. The 210 participants from ADNI-2 were used as an independent test set, while the remaining 1042 participants from ADNI-1 were used for model development. A five-fold cross-validation strategy was employed for both training and validation. In each fold, 80% of the participants were used for training, and 20% for validation. During the training phase, online data augmentation strategies were used, applying the following operations to the loaded raw data with a 50% probability: randomly rotating the image within a range of -30° to +30°; performing non-linear brightness adjustment using random gamma coefficients; and randomly selecting three 10×10×10 voxel cubic regions in the image for pixel-zeroing occlusion.

[0035] S3: Initial feature extraction of the voxel flow backbone. For example... Figure 3 As shown, preprocessed structural magnetic resonance imaging (SMRI) and positron emission tomography (PET) images are input into two identical voxel stream backbone modules. The processing flow for each voxel stream backbone is as follows: First, a 3×3×3 kernel-sized 3D convolutional layer is used to capture local spatial patterns, with 64 output channels; then, a batch normalization layer stabilizes the training process; a ReLU activation function is used to introduce nonlinearity; next, depthwise separable convolutions are used to reduce computational complexity, decomposing the standard 3D convolution into independent spatial and pointwise convolutions; finally, adaptive pooling generates a fixed-size feature representation. The feature maps output from the two modal branches are concatenated along the channel dimension to form a unified multimodal initial feature representation with 128 channels. The structure of the voxel stream backbone is shown below. Figure 3 As shown.

[0036] S4: Hierarchical Feature Extraction Backbone Network Modeling. The concatenated multimodal features are input into the hierarchical feature extraction backbone network. This backbone network consists of four stages, with the number of deep state space model blocks in the multimodal features in each stage being 2, 2, 9, and 2 respectively. Spatial downsampling is performed between stages using 3D convolutional layers with a kernel size of 2 and a stride of 2, gradually reducing the spatial resolution of the feature maps, while the number of channels is successively multiplied to 64, 128, 256, and 512. Each deep state space model block (MDSSM Block) in the multimodal features adopts a pre-normalized residual design, with the structure as shown below. Figure 4 As shown, the input features are first normalized by layers and then fed into the mid-deep state space model submodule (MDSSM). The output is added to the original input through residual connections. Subsequently, it is normalized again by layers and then fed into the feedforward neural network submodule (FFN), where it is also stacked through residual connections. The feedforward neural network submodule contains two fully connected layers with the GeLU activation function in between, and the hidden layer dimension is expanded to four times the input dimension.

[0037] S5: Middle Deep Scanning (MDS) mechanism enables adaptive feature extraction. Within the middle-deep state space model submodule, the middle deep scanning mechanism (MDS) is executed, and its process is as follows: Figure 5 As shown. Given a 3D feature map. Lightweight migration network predicts migration tensor The first three channels represent spatial offset. The fourth channel represents the scan index offset. The offset network contains deep convolutional layers and channel attention layers to aggregate global contextual information. For the spatial offset branch, a regular reference grid is created. The coordinates are normalized to [-1, 1]. The deformed sampling points are obtained by adding the spatial offsets: .because Including decimal coordinates, feature values ​​are sampled from X using trilinear interpolation to obtain the deformed feature map. For the index offset branch, the 3D volume is flattened into a 1D sequence. Let... Original raster index Normalized to the [-1, 1] interval. Index offset. Flattened The deformed index is calculated as .right Sort to get a new scan order Deformation feature map Flattened ,according to Rearrangement: The resulting sequence prioritizes voxels that the network deems more important. Since the sorting operation is non-differentiable, the loss gradient is averaged along the sequence dimension and copied to... This approximates gradient propagation. Finally, the structure-aware sequence input is used to selectively model the state-space context for global context modeling, and the output is reconstructed back to its original 3D dimensions and propagated through residual connections.

[0038] S6: The multimodal interaction module enables cross-modal feature fusion. It splits the feature map output from the final stage of the hierarchical feature extraction backbone network into structural magnetic resonance imaging feature maps according to modality. Positron emission tomography (PET) feature map The inputs are respectively the Multimodal Interaction Module (MI), whose structure is as follows: Figure 6 As shown. The multimodal interaction module contains two parallel and symmetrical paths: (1) Metabolic feedback path: using positron emission tomography features as queries and structural magnetic resonance imaging features as keys and values. First, projection is performed through a learnable linear transformation: , , The cross-modal attention map is calculated as follows: ,in The feature dimension is used for the structural features of metabolic enhancement. (2) Structural guidance path: using structural magnetic resonance imaging features as queries and positron emission tomography (PET) features as keys and values: , , , , Both paths employ a multi-head attention mechanism, with 8 attention heads. This results in two enhanced feature maps. and Then, they are concatenated along the channel dimension and reduced in dimensionality by 1×1×1 three-dimensional convolution to obtain the fused feature map. .

[0039] S7: Classification prediction. This involves analyzing the fused feature maps. Global average pooling and flattening operations are performed sequentially to obtain feature vectors. Classification is performed using a two-layer fully connected network, with the hidden layer dimension set to 256 and the output dimension set to 2 based on the classification task. The model parameters are optimized using the cross-entropy loss function.

[0040] Experimental results

[0041] To verify the effectiveness of this invention, ablation experiments and comparative experiments were conducted on the ADNI dataset. Evaluation metrics included accuracy (ACC), sensitivity (SEN), specificity (SPE), and area under the curve (AUC).

[0042] Table 1 presents the ablation experiment results on three classification tasks. A simple two-stream 3D CNN with directly cascaded bimodal features followed by a fully connected layer is used as the baseline. Experimental results show that the Voxel Flow Backbone (VFS), the Mid-Deep State Space Model (MDSSM), and the Multimodal Interaction Module (MI) each make significant contributions to performance, and the complete model achieves the best performance on all tasks.

[0043] Table 1. Ablation experimental results of the MMDSSM framework on three classification tasks.

[0044] Classification tasks method ACC(%) SEN(%) SPE (%) AUC(%) AD vs. NC Baseline 89.30 88.20 90.10 91.50 +VFS 91.80 90.50 92.70 93.80 +VFS+MDSSM 95.20 93.80 96.40 96.90 Complete model 97.62 96.50 98.60 98.74 AD vs. MCI Baseline 72.50 71.80 73.20 74.60 +VFS 75.30 74.50 76.10 77.20 +VFS+MDSSM 78.60 77.90 79.30 80.50 Complete model 85.63 81.50 93.54 85.50 NC vs. MCI Baseline 76.80 74.50 78.20 79.10 +VFS 79.50 77.80 80.40 81.60 +VFS+MDSSM 83.70 82.90 84.10 84.90 Complete model 86.52 93.34 81.70 86.20

[0045] Table 2 presents the comparative experimental results of this invention with existing mainstream multimodal methods. This invention achieves optimal performance on all three tasks: AD vs. NC, AD vs. MCI, and NC vs. MCI.

[0046] Table 2 Performance comparison with existing methods

[0047] method AD vs. NC ACC (%) AD vs. NC AUC(%) AD vs. MCI ACC(%) NC vs. MCI ACC(%) MMMSC 91.42 94.85 81.63 80.35 MCAD 91.07 94.07 73.10 71.26 AD-TF 96.00 98.50 74.60 72.80 3DCNN-IF 93.21 92.85 80.20 82.50 This invention 97.62 98.74 85.63 86.52

[0048] In summary, the Alzheimer's disease imaging diagnostic method based on a multimodal mid-deep state space model proposed in this invention effectively integrates complementary information from structural magnetic resonance imaging and positron emission tomography through the synergistic effect of the voxel flow backbone, mid-deep scanning mechanism, and bidirectional cross-modal attention interaction module. While maintaining linear computational complexity, it significantly improves diagnostic performance and provides a reliable technical solution for the early and accurate diagnosis of Alzheimer's disease.

[0049] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any equivalent embodiments made by those skilled in the art using the disclosed technical content without departing from the spirit and scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multi-modal mid-deep state space model-based Alzheimer's disease image diagnosis method, characterized in that, Includes the following steps: S1: Acquire and preprocess the subject's structural magnetic resonance imaging and positron emission tomography image data to obtain a standardized multimodal image dataset; S2: Input the two preprocessed modal images into the dual-branch voxel flow backbone module respectively, extract the initial feature map and stitch it along the channel dimension to form a unified multimodal initial feature representation; S3: Input the spliced ​​multimodal features into a hierarchical feature extraction backbone network consisting of multiple stages. Each stage contains several deep state space model blocks and downsampling layers in the multimodal data to gradually extract multi-scale semantic features. S4: In the mid-deep state space model submodule of the multimodal mid-deep state space model block, a mid-deep scanning mechanism is executed to extract structure-aware feature sequences through content-aware adaptive scanning path optimization. S5: The feature map output from the final stage of the hierarchical feature extraction backbone network is split into two branches according to modality and input into the multimodal interaction module. Through the bidirectional cross-modal attention mechanism, the deep interaction and fusion of structural features and metabolic features are realized to obtain the fused feature map. S6: Perform global average pooling and flattening operations on the fused feature map in sequence, and output the classification prediction probability of Alzheimer's disease, mild cognitive impairment or normal control through a fully connected layer.

2. The method of claim 1, wherein, The preprocessing procedure of S1 includes: registering the original images to the standard brain space of the Montreal Neuroscience Institute, unifying the image dimensions to 121×145×121, and the voxel spatial resolution to 1.5mm×1.5mm×1.5mm; removing skull and non-brain tissues; smoothing using a Gaussian kernel with a full width at half maximum (FWHM) of 8mm; correcting for intensity inhomogeneity in structural magnetic resonance imaging (MRI); and normalizing the intensity of positron emission tomography (PET) scans.

3. The method of claim 1, wherein, The voxel flow backbone module of S2 sequentially extracts initial features through a 3D convolutional layer, a batch normalization layer, a ReLU activation function, a depthwise separable convolutional layer, and an adaptive pooling layer; the feature maps output by the two modal branches are spliced ​​along the channel dimension to form a unified multimodal initial feature representation.

4. The method according to claim 3, characterized in that, The depthwise separable convolution decomposes the standard 3D convolution into two independent steps: depthwise convolution and pointwise convolution. This reduces the number of parameters by approximately [amount missing] compared to the standard 3D convolution. times, of which Number of output channels This represents the kernel size.

5. The method according to claim 1, characterized in that, The hierarchical feature extraction backbone network of S3 consists of four stages. The number of deep state space model blocks in the multimodal mode in each stage is 2, 2, 9, and 2 respectively. Spatial downsampling is performed between stages through three-dimensional convolutional layers with a kernel size of 2 and a stride of 2, and the number of feature map channels increases by a factor of 2.

6. The method according to claim 1, characterized in that, The multimodal deep state space model block adopts a pre-normalized residual design. The input features are first normalized by the layer and then sent to the deep state space model sub-module. The output is added to the original input through the residual connection. The output is then fed into the feedforward neural network submodule again through layer normalization, and the output is superimposed through residual connections. The feedforward neural network submodule contains two fully connected layers with the GeLU activation function in between, and the hidden layer dimension is expanded to 4 times the input dimension.

7. The method according to claim 1, characterized in that, The mid-depth scanning mechanism of S4 includes the following steps: S41: Predict the spatial offset and scan index offset of each voxel using a lightweight offset network, which includes deep convolutional layers and channel attention layers; S42: Perform trilinear interpolation sampling on the original feature map based on the spatial offset to obtain the deformed feature map; S43: Flatten the 3D feature map into a 1D sequence, and reorder the flattened deformable feature sequence according to the index offset to generate a content-aware scanning order; S44: Input the reordered sequence into the selective state-space model for global context modeling, and reshape the output back to the original three-dimensional size and pass it through residual connections.

8. The method according to claim 7, characterized in that, The gradient of the sorting operation in S43 is approximately propagated by averaging the loss gradient along the sequence dimension and copying it to the index offset.

9. The method according to claim 1, characterized in that, The S5 multimodal interaction module contains two parallel and symmetrical paths: Metabolic feedback pathway: Using positron emission tomography features as queries and structural magnetic resonance imaging features as keys and values, metabolically enhanced structural features are generated through cross-modal attention computation. Structure-guided path: Using structural magnetic resonance imaging features as queries and positron emission tomography (PET) features as keys and values, metabolic features that enhance structure are generated; The enhanced feature maps output from the two paths are concatenated along the channel dimension and then dimensionality reduced by 1×1×1 three-dimensional convolution to obtain the fused feature map.

10. The method according to claim 9, characterized in that, Both the metabolic feedback path and the structure guidance path employ a multi-head cross-modal attention mechanism, with 8 attention heads and the feature dimensions remaining consistent with the input feature map. The metabolic feedback path and the structure guidance path use independent learnable weight matrices.

Images