Medical image classification method based on multi-space multi-modal fusion network guided by brain region learning

Abstract

The application provides a brain region learning guided multi-space multi-modal fusion network for a medical image classification task, and belongs to the field of medical image classification.The application comprises the following steps: acquiring multi-modal medical images without data preprocessing; extracting initial local deep features through a 3D ResNet; sending the local deep features into a deep space learning module to capture global deep fusion features of the multi-modal data; sending the local deep features into a breadth space learning module to capture global generalized fusion features of the multi-modal data; sending the local deep features into an intrinsic space learning module to capture key invariant features between the multi-modal data; splicing the global deep fusion features, the global generalized fusion features and the key invariant features to obtain multi-modal fusion features; and sending the multi-modal fusion features into a brain region learning module to explicitly integrate prior knowledge of a brain region structure into a model learning process.The application improves the multi-modal medical image classification precision and algorithm interpretability.

Description

Technical Field

[0001] This invention proposes a brain region learning-guided multi-spatial multimodal fusion network for medical image classification tasks, belonging to the field of medical image classification. Background Technology

[0002] On the one hand, existing models still have limitations in multi-level feature extraction. Specifically, most deep learning models tend to capture deep features only from a single network layer, ignoring the complementarity between different layers, or focusing only on local or global feature extraction, failing to achieve coordinated integration of local and global features, as well as the fusion of deep and broad features. Furthermore, in cross-modal fusion, effectively capturing key shared features and stable invariant features between different modalities is crucial for improving the model's generalization ability. Fully utilizing these intrinsically invariant features not only helps reduce noise and redundancy in multimodal data but also improves the accuracy and robustness of medical image classification. However, current research has not effectively utilized such invariant features. On the other hand, most current methods still rely on simply connecting features extracted from different modalities or employing only a single-level fusion strategy to achieve cross-modal information interaction. These methods often ignore the inherent heterogeneity and complex nonlinear relationships between multimodal data, making it difficult for models to fully explore the complex coupling relationships embedded within them. Due to this limitation, models struggle to capture effective multimodal coupling features, thus affecting the accuracy of early diagnosis in medical image classification. Therefore, to more effectively capture the complex coupling relationships between multimodal data, it is necessary to extract features from multiple levels and integrate diverse fusion strategies. Furthermore, most existing machine learning methods still suffer from shortcomings in interpretability. Models with interpretable mechanisms not only help elucidate underlying pathological processes but also provide valuable evidence for clinical decision-making. Although deep learning models have made significant progress in medical image classification tasks, how to improve model interpretability while maintaining high diagnostic performance remains a critical issue to be addressed. Moreover, existing interpretability methods typically provide post-hoc interpretations by visualizing the activation patterns of network layers. The heatmaps generated by these methods often lack clear anatomical boundaries, making it difficult to establish a direct correspondence with the brain's intrinsic functional regions. Their interpretation mechanisms mainly rely on features automatically learned by the model, failing to explicitly introduce prior knowledge of brain regions to enhance interpretability, and neglecting statistical significance testing. Therefore, the resulting interpretations may deviate from the actual anatomical distribution of the brain, thus limiting their credibility and applicability in clinical neuroscience research. Therefore, a key challenge facing medical image classification is developing models that can simultaneously achieve high diagnostic accuracy and strong interpretability. Based on the above, this invention proposes a medical image classification method based on a multi-spatial, multi-modal fusion network guided by brain region learning. Summary of the Invention

[0003] The technical problem solved by this invention is: This invention proposes a medical image classification method based on a brain region learning-guided multi-space multimodal fusion network, which is used to solve the problems of insufficient multi-level feature extraction, single multimodal fusion mode, and poor interpretability in previous multimodal medical image classification methods, thereby improving the accuracy and interpretability of multimodal medical image classification.

[0004] The technical solution of this invention is: a medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network, the method comprising:

[0005] Step 1: Acquire multimodal medical images without data preprocessing, mainly including two types of data: structural magnetic resonance imaging (sMRI) and positron emission tomography (PET). Perform data preprocessing on the two types of data to extract the white and gray matter images corresponding to the sMRI, as well as the preprocessed PET images.

[0006] Step 2: Extract initial local depth features using 3D ResNet;

[0007] Step 3: Feed the local deep features into the deep space learning module to capture the global deep fusion features of multimodal data. ;

[0008] Step 4: Feed the local deep features into the wide-space learning module to capture the global generalized fusion features of the multimodal data. ;

[0009] Step 5: Feed the local deep features into the intrinsic space learning module to capture key invariant features among multimodal data. ;

[0010] Step 6, splicing , and To obtain multimodal fusion features ;

[0011] Step 7, The brain region learning module explicitly integrates prior knowledge of brain region structure into the model learning process, actively guiding the model to learn discriminative features consistent with anatomically defined brain regions.

[0012] Further, in Step 1, image preprocessing is performed using the SPM and CAT toolboxes. Following a standardized pipeline, the sMRI data is processed using the CAT12 toolbox, with the following steps: converting the sMRI data to NIfTI format, anterior-posterior commissure alignment, N4 bias field correction, skull dissection, image spatial alignment, tissue segmentation, and spatial normalization. Then, gray and white matter images are extracted, and the regional gray matter volume is calculated based on the AAL116 template. The PET data is batch-processed using the SPM toolbox, including converting the PET data to NIfTI format, anterior-posterior commissure alignment, spatial normalization, image registration, skull dissection, and smoothing. The size of the preprocessed image is [size missing]. These preprocessing steps ensure the consistency of spatial structure and signal intensity throughout the imaging data, providing a reliable foundation for subsequent multimodal data fusion and brain region feature analysis.

[0013] Furthermore, Step 2 includes:

[0014] Let the gray matter image (GM), white matter image (WM), and PET image from sMRI be represented as follows in the input multimodal data: , , These input data are fed into a 3D ResNet network to extract initial multimodal local depth features. The 3D ResNet processing flow is as follows: First, the input data is upsampled to improve spatial resolution and unify dimensionality; then, initial feature extraction is completed through 3D convolutional layers; next, the generated feature maps are input into three cascaded 3D ResNet blocks for deep encoding. This structure effectively fuses information from different local receptive fields using a residual learning mechanism, thereby obtaining initial multimodal local depth features.

[0015] Furthermore, Step 3 includes:

[0016] First, this module extracts global long-range dependency features of three modalities using a ViT network, and then adaptively fuses the three modalities based on a sliding window attention mechanism. This sliding window attention mechanism can capture local information within a modality while reducing the number of parameters. Specifically, the initially extracted local depth features are... Positional encoding is added to it to obtain initial information with spatial representation. The initial feature representation after adding positional encoding is as follows: Secondly, through a depth of The ViT network extracts global depth features for each modality. Let the first... The input to each Transformer module is Obtain the first by random mapping The input to the self-attention mechanism corresponding to each Transformer module is represented as follows: , , , and , Next, the self-attention mechanism was used to... The inputs of the three modal self-attention mechanisms in each Transformer module are processed to generate corresponding self-attention outputs. , , :

[0017] in , These represent gray matter, white matter, and PET, respectively. First, residual connections are used, followed by processing through a multilayer perceptron to obtain the... The final output of the first Transformer block; The output of the Transformer module is the first one. The input to each Transformer module is then used to obtain global depth features. , , ,in It is the depth of the ViT model.

[0018] Finally, a sliding window attention mechanism is introduced, based on Adaptively learn feature fusion weights Sliding window attention computes associations only within a fixed-size local window, reducing its computational complexity for long sequences from quadratic to linear while maintaining the model's effective perception of local information structure. Subsequently, global deep features are integrated... , , Adaptive weight matrix between this mode Global deep multimodal fusion features are obtained. As shown below:

[0019]

[0020] MLP refers to a multilayer perceptron consisting of linear layers, Gelu activation, and Dropout layers. It achieves deep fusion of global features. Global average pooling and shape adjustment are performed to obtain the final global deep fusion features. .

[0021] Furthermore, Step 4 includes:

[0022] The wide-space learning module first aggregates feature maps from different depths through wide connections to promote the fusion of complementary information across layers. Then, a wide-space attention mechanism is applied to the aggregated features to further enhance their representational power. Specifically, the inputs of the self-attention mechanisms corresponding to each modality in all Transformer layers are first concatenated to obtain the inputs required for the wide-space attention mechanism. , , , , , Its form is as follows:

[0023]

[0024] Then, the global generalized features Bg, Bw, and Bp corresponding to the three modalities are extracted using a generalized attention mechanism, as shown below:

[0025]

[0026]

[0027]

[0028] in , , For the corresponding , , Dimensions.

[0029] Then, a chain-like cross-modal attention mechanism is used to fuse multimodal features to generate initial global generalized features. :

[0030]

[0031]

[0032] MLP refers to a multilayer perceptron consisting of linear layers, Gelu activation, and Dropout layers. At this point, the initial global generalized fusion features can be obtained. ,right Shape adjustments are performed and global average pooling is applied to obtain the global generalized fusion feature. .

[0033] Furthermore, Step 5 and Step 6 include:

[0034] The wide-space learning module utilizes average pooling and max pooling to construct cross-modal interaction features and extract key intrinsic information to generate multimodal collaborative features. Specifically, it first performs average pooling (AP) and max pooling (MP) on the initial features, effectively reducing feature dimensionality while extracting richer local and global information.

[0035]

[0036]

[0037]

[0038] Subsequently, the features of the three modalities are added pairwise to obtain paired complementary information between modalities, thereby capturing the potential correlations and complementary features between different modalities:

[0039]

[0040]

[0041]

[0042] Next, intrinsic microstate decomposition is applied to the paired interaction features to extract the intrinsic microstates corresponding to the largest eigenvalues ​​and sum them as key intermodal collaborative information. Subsequently, on Perform summation pooling and Normalization process yields key invariant features based on intrinsic space learning. .

[0043] Finally, splicing , and To obtain multimodal fusion features .

[0044] Furthermore, Step 7 includes:

[0045] Based on multimodal fusion features First, linear mapping is applied to obtain classification features. This feature was further predicted as a hidden feature associated with brain regions. Meanwhile, from Another linear mapping directly generates initial brain region features. In subsequent optimization processes, Providing task-related feature information enables the model to identify which brain regions contribute most to classification decisions; The model preserves the intrinsic representations of brain regions, reflecting their inherent structural or functional characteristics. Through a bidirectional cross-attention mechanism and GRU updates, these two types of features interact and complement each other, enabling the model to highlight brain regions relevant to the target task while preserving the intrinsic information of the brain regions. The inputs required to define the bidirectional cross-attention mechanism are as follows:

[0046]

[0047]

[0048] in, , , , , , Represent the weights corresponding to the query (q), key (k), and value (v) required for the bidirectional cross-attention mechanism; calculate the bidirectional cross-attention. , Specifically, on the one hand, it focuses on the characteristics of task-related brain regions. As a query of attention mechanisms, initial brain region features As keys and values, they capture task information to guide brain region features; on the other hand, they use... As a query, with As keys and values, structural priors of brain regions are introduced to constrain task features:

[0049]

[0050]

[0051] The outputs in both directions are iteratively updated using GRU to obtain a brain region weight matrix that balances task relevance with intrinsic brain region features. and :

[0052]

[0053]

[0054] in, .

[0055] Based on the above application of the bidirectional cross-attention mechanism and GRU weight update, the brain region weight matrix is ​​gradually optimized to obtain the final brain region feature representation. = Apply a linear mapping to this feature, and then... Dropout processing is performed to enhance feature representation and prevent overfitting, generating the final brain region features. The obtained features are input into the classification layer to generate brain region-based classification features. To fully integrate brain region features with multimodal fusion features, brain region classification features were analyzed. Multimodal fusion classification features An adaptive weighted combination is performed to obtain the final features used for AD classification. .

[0056] Furthermore, Step 7 also includes:

[0057] Define the weighted sum of classification cross-entropy loss and brain region loss as the loss function:

[0058]

[0059] in Indicates predicted label Indicates the true label, Indicates predicted brain region features, This refers to the volume of gray matter in the brain. and These are adjustable weights used to adaptively balance the importance of the multimodal integrated spatial learning module and the brain region learning module.

[0060] The beneficial effects of this invention are:

[0061] 1) This invention proposes a medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network. This method mainly consists of a multimodal integrated spatial learning module and a brain region learning module.

[0062] 2) The multimodal integrated spatial learning module effectively integrates deep features, broad features, and intrinsic invariant features through the deep spatial learning module, the broad spatial learning module, and the intrinsic spatial learning module;

[0063] 3) By incorporating brain region constraints into a composite loss function with adaptive weights, the brain region learning module is guided to extract features consistent with anatomically defined brain regions, thereby identifying key discriminative brain regions and improving the interpretability of the model.

[0064] 4) Extensive experiments on public datasets demonstrate that the proposed method outperforms existing methods in multiple classification tasks, validating its effectiveness in medical image classification. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the process structure of the present invention;

[0067] Figure 2 This is a schematic diagram of the deep spatial learning module of the present invention;

[0068] Figure 3 This is a schematic diagram of the extensive spatial learning module of the present invention;

[0069] Figure 4 This is a schematic diagram of the intrinsic space learning module of the present invention;

[0070] Figure 5 This is a schematic diagram of the brain region learning module of the present invention. Detailed Implementation

[0072] Example 1: As Figures 1-5 As shown, a registration-free medical image fusion method based on state selection feature aggregation is described, the method comprising:

[0073] Step 1: Acquire unprocessed multimodal medical images, primarily including structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) data. Perform data preprocessing on both types of data, extracting the corresponding white and gray matter images from the sMRI, and the preprocessed PET images. Specifically: Image preprocessing is performed using the SPM and CAT toolboxes. Following a standardized pipeline, the CAT12 toolbox is used to process the sMRI data. The specific steps are as follows: converting the sMRI data to NIfTI format, anterior-posterior commissure alignment, N4 bias field correction, skull dissection, image spatial alignment, tissue segmentation, and spatial normalization. Then, gray and white matter images are extracted, and the regional gray matter volume is calculated based on the AAL116 template. The SPM toolbox is used for batch processing of the PET data, including converting the PET data to NIfTI format, anterior-posterior commissure alignment, spatial normalization, image registration, skull dissection, and smoothing. The size of the preprocessed image is [size missing]. These preprocessing steps ensure the consistency of spatial structure and signal intensity throughout the imaging data, providing a reliable foundation for subsequent multimodal data fusion and brain region feature analysis.

[0074] Step 2: Extract initial local depth features using 3D ResNet. Specifically: Let the gray matter image (GM), white matter image (WM), and PET image from the input multimodal data be represented as follows: , , These input data are fed into a 3D ResNet network to extract initial multimodal local depth features. The 3D ResNet processing flow is as follows: First, the input data is upsampled to improve spatial resolution and unify dimensionality; then, initial feature extraction is completed through 3D convolutional layers; next, the generated feature maps are input into three cascaded 3D ResNet blocks for deep encoding. This structure effectively fuses information from different local receptive fields using a residual learning mechanism, thereby obtaining initial multimodal local depth features.

[0075] Step 3: Feed the local deep features into the deep space learning module to capture the global deep fusion features of multimodal data. The specific implementation process is as follows: Figure 2 As shown. Specifically, the module first extracts global long-range dependency features of the three modalities using the ViT network, and then adaptively fuses the three modalities based on a sliding window attention mechanism. This sliding window attention mechanism can reduce the number of parameters while capturing local information within the modality. Specifically, the initially extracted local depth features are... Positional encoding is added to it to obtain initial information with spatial representation. The initial feature representation after adding positional encoding is as follows: Secondly, through a depth of The ViT network extracts global depth features for each modality. Let the first... The input to each Transformer module is Obtain the first by random mapping The input to the self-attention mechanism corresponding to each Transformer module is represented as follows: , , , and , Next, the self-attention mechanism was used to... The inputs of the three modal self-attention mechanisms in each Transformer module are processed to generate corresponding self-attention outputs. , , :

[0076]

[0077] in , These represent gray matter, white matter, and PET, respectively. First, residual connections are used, followed by processing through a multilayer perceptron to obtain the... The final output of the Transformer block:

[0078]

[0079] in , These represent gray matter, white matter, and PET, respectively. MLP refers to a multilayer perceptron consisting of linear layers, Gelu activation layers, and Dropout layers. The output of the Transformer module is the first one.

[0080] The input to each Transformer module is then used to obtain global depth features. , , ,in It is the depth of the ViT model.

[0081] Finally, a sliding window attention mechanism is introduced, based on Adaptive learning of feature fusion weights Sliding window attention computes associations only within a fixed-size local window, reducing its computational complexity for long sequences from quadratic to linear while maintaining the model's effective perception of local information structure. Subsequently, global deep features are integrated... , , Adaptive weight matrix between this mode Global deep multimodal fusion features are obtained. As shown below:

[0082]

[0083] MLP refers to a multilayer perceptron consisting of linear layers, Gelu activation, and Dropout layers. It achieves deep fusion of global features. Global average pooling and shape adjustment are performed to obtain the final global deep fusion features. .

[0084] Step 4: Feed the local deep features into the wide-space learning module to capture the global generalized fusion features of the multimodal data. The specific implementation process is as follows: Figure 3As shown. Specifically, the wide-space learning module first aggregates feature maps from different depths through wide connections to promote the fusion of complementary information across layers. Then, a wide-attention mechanism is applied to the aggregated features to further enhance their representational power. Specifically, the inputs of the self-attention mechanisms corresponding to each modality in all Transformer layers are first concatenated to obtain the inputs required for the wide-attention mechanism: , , , , , Its form is as follows:

[0085]

[0086] Then, the global generalized features Bg, Bw, and Bp corresponding to the three modalities are extracted using a generalized attention mechanism, as shown below:

[0087]

[0088]

[0089]

[0090] in , , For the corresponding , , Dimensions.

[0091] Then, a chain-like cross-modal attention mechanism is used to fuse multimodal features to generate initial global generalized features. :

[0092]

[0093]

[0094] MLP refers to a multilayer perceptron consisting of linear layers, Gelu activation, and Dropout layers. At this point, the initial global generalized fusion features can be obtained. ,right Shape adjustments are performed and global average pooling is applied to obtain the global generalized fusion feature. .

[0095] Step 5: Feed the local deep features into the intrinsic space learning module to capture key invariant features among multimodal data. The specific implementation process is as follows: Figure 4As shown. Specifically, the wide-space learning module utilizes average pooling and max pooling to construct cross-modal interaction features and extract key intrinsic information to generate multimodal collaborative features. Specifically, it first performs average pooling (AP) and max pooling (MP) on the initial features, effectively reducing feature dimensionality while extracting richer local and global information:

[0096]

[0097]

[0098]

[0099] Subsequently, the features of the three modalities are added pairwise to obtain paired complementary information between modalities, thereby capturing the potential correlations and complementary features between different modalities:

[0100]

[0101]

[0102]

[0103] Next, intrinsic microstate decomposition is applied to the paired interaction features to extract the intrinsic microstates corresponding to the largest eigenvalues ​​and sum them as key intermodal collaborative information. Subsequently, on Perform summation pooling and Normalization process yields key invariant features based on intrinsic space learning. .

[0104] Step 6, splicing , and To obtain multimodal fusion features .

[0105] Step 7, The data is fed into the brain region learning module, where prior knowledge of brain region structures is explicitly integrated into the model learning process. This actively guides the model to learn discriminative features consistent with anatomically defined brain regions. The specific implementation process is as follows: Figure 5 As shown. Specifically, based on multimodal fusion features. First, linear mapping is applied to obtain classification features. This feature was further predicted as a hidden feature associated with brain regions. Meanwhile, from Another linear mapping directly generates initial brain region features. In subsequent optimization processes, Providing task-related feature information enables the model to identify which brain regions contribute most to classification decisions; The model preserves the intrinsic representations of brain regions, reflecting their inherent structural or functional characteristics. Through a bidirectional cross-attention mechanism and GRU updates, these two types of features interact and complement each other, enabling the model to highlight brain regions relevant to the target task while preserving the intrinsic information of the brain regions. The inputs required to define the bidirectional cross-attention mechanism are as follows:

[0106]

[0107]

[0108] in, , , , , , The weights corresponding to the query (q), key (k), and value (v) required for the bidirectional cross-attention mechanism are represented; based on the above, the bidirectional cross-attention is calculated. , Specifically, on the one hand, it focuses on the characteristics of task-related brain regions. As a query of attention mechanisms, initial brain region features As keys and values, they capture task information to guide brain region features; on the other hand, they use... As a query, with As keys and values, structural priors of brain regions are introduced to constrain task features:

[0109]

[0110]

[0111] The outputs in both directions are iteratively updated using GRU to obtain a brain region weight matrix that balances task relevance with intrinsic brain region features. and :

[0112]

[0113]

[0114] in, .

[0115] Based on the above application of the bidirectional cross-attention mechanism and GRU weight update, the brain region weight matrix is ​​gradually optimized to obtain the final brain region feature representation. = Apply a linear mapping to this feature, and then... Dropout processing is performed to enhance feature representation and prevent overfitting, generating the final brain region features. The obtained features are input into the classification layer to generate brain region-based classification features. To fully integrate brain region features with multimodal fusion features, brain region classification features were analyzed. Multimodal fusion classification features An adaptive weighted combination is performed to obtain the final features used for AD classification. .

[0116] Furthermore, the weighted sum of the classification cross-entropy loss and the brain region loss is defined as the loss function:

[0117]

[0118] in Indicates predicted label Indicates the true label, Indicates predicted brain region features, This refers to the volume of gray matter in the brain. and These are adjustable weights used to adaptively balance the importance of the multimodal integrated spatial learning module and the brain region learning module.

[0119] To verify the effectiveness of the method of the present invention, this chapter of the present invention is based on sMRI and PET data of ADNI1, ADNI2 and ADNI3 in the ADNI database

[194] . The ADNI dataset has been continuously collecting multimodal imaging, clinical assessment and biological sample data since its establishment in 2003. sMRI consists of 3D T1-weighted images and PET consists of 18F-FDG PET images. The data used in this study includes 1027 participants: 315 CN, 421 MCI and 291 AD, and the detailed demographics are as follows: ADNI1 (CN: 91, MCI: 183, AD: 83), ADNI2 (CN: 209, MCI: 21, AD: 145) and ADNI3 (CN: 15, MCI: 217, AD: 63). This experiment uses the following parameter configuration: the SGD optimizer is used, with an initial learning rate of 0.001, a weight decay factor of 0.01, a momentum of 0.90, and gradient clipping (threshold of 2.0) to stabilize the training process. The learning rate scheduler uses a cosine annealing hot restart strategy, setting the initial period T0 to 10 and the period multiplication factor... The minimum learning rate is 3. The value is 0.00001. The training batch size is set to 16, with a total of 60 training epochs, and 4 data loading workflows are provided. To ensure fairness in the comparison, the available parameters and input data of all comparison methods in the overall performance evaluation are consistent with those of the proposed model.

[0120] To evaluate the effectiveness of the proposed model, three binary classification tasks were performed on the ADNI dataset: CN vs. MCI, CN vs. AD, and MCI vs. AD. Five-fold cross-validation was used to ensure the reliability of the evaluation, and the final performance metric was the average of all folds. Core evaluation metrics included: accuracy, sensitivity, specificity, F1 score, balance, and area under the curve (AUC). Balance is the average of sensitivity and specificity, and it gives equal weight to both positive and negative classes. In medical diagnosis, avoiding false positives and false negatives are equally important, and therefore it is often considered a core comprehensive metric for medical diagnostic models.

[0121] Furthermore, the experimental results of this invention were compared with those of existing unregistered multimodal image fusion methods MADDi, ViT, MFNet, SSFTT, and MDL-Net. The comparison results of the fusion results of this method with those of MADDi, ViT, MFNet, SSFTT, and MDL-Net are shown in Table 1.

[0122] Table 1. Performance comparison of different methods in three classification tasks.

[0123]

[0124] As shown in Table 1, the method proposed in this invention is effective and outperforms the comparative methods in almost all evaluation metrics. Specifically, in the CN vs. MCI classification task, this method achieves improvements of at least 7.47%, 4.55%, 0.44%, 10.13%, and 9.65% in accuracy, sensitivity, specificity, F1 score, AUC, and balance, respectively. In the CN vs. AD classification task, it shows advantages in all six core metrics, with minimum improvements of 4.14% (accuracy), 4.77% (sensitivity), 3.58% (specificity), 5.58% (F1 score), 3.53% (AUC), and 4.18% (balance), respectively. In the MCI vs. AD classification task, it achieves improvements of at least 5.42%, 8.10%, 0.10%, 6.58%, and 3.19% in accuracy, specificity, F1 score, AUC, and balance, respectively. These results fully demonstrate the superior performance of the proposed method in medical image classification tasks. Comparisons of different methods in terms of model complexity and inference efficiency show that although the proposed method integrates the 3DResNet encoder and multi-layer ViT modules, resulting in slightly higher complexity than some lightweight models, it remains comparable to most classic and state-of-the-art methods. Furthermore, with a similar computational budget, the proposed method achieves superior performance, demonstrating a good balance between performance and complexity. This indicates that the proposed model maintains competitive performance while improving overall performance.

[0125] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network, characterized by: The method includes: Step 1: Acquire multimodal medical images, including two types of data: structural magnetic resonance images (sMRI) and PET images. Perform data preprocessing on the two types of data to extract the white and gray matter images corresponding to the structural magnetic resonance images, as well as the preprocessed PET images. Step 2: Extract initial local depth features using 3D ResNet; Step 3: Feed the local deep features into the deep space learning module to capture the global deep fusion features of multimodal data; Step 4: Feed the local deep features into the wide-space learning module to capture the global generalized fusion features of multimodal data; Step 5: Feed the local deep features into the intrinsic space learning module to capture key invariant features among multimodal data; Step 6: Combine the global deep fusion features, global generalized fusion features, and key invariant features to obtain multimodal fusion features; Step 7: Feed the multimodal fusion features into the brain region learning module, and explicitly integrate the prior knowledge of brain region structure into the model learning process to actively guide the model to learn discriminative features that are consistent with the anatomically defined brain regions.

2. The medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network according to claim 1, characterized in that: Step 1 includes: Image preprocessing was performed using the SPM and CAT toolboxes. Following the standardized pipeline, the CAT12 toolbox was used to process the structural magnetic resonance images (sMRI). The specific steps were as follows: converting the sMRI data to NIfTI format, anterior-posterior commissure alignment, N4 bias field correction, skull dissection, image spatial alignment, tissue segmentation, and spatial normalization. Then, gray matter and white matter images were extracted, and the regional gray matter volume was calculated based on the AAL116 template. Batch processing of PET data was performed using the SPM toolbox, including converting PET data to NIfTI format, anterior-posterior commissure alignment, spatial normalization, image registration, skull dissection, and smoothing; the preprocessed image size was [size missing]. .

3. The medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network according to claim 1, characterized in that: Step 2 includes: Let the gray matter image (GM), white matter image (WM), and PET image from sMRI be represented as follows in the input multimodal data: , , These input data are fed into a 3D ResNet network to extract initial local depth features; The processing flow of 3D ResNet is as follows: First, the input data is upsampled and preprocessed to improve spatial resolution and unify dimensions; then, initial feature extraction is completed through three-dimensional convolutional layers; next, the generated feature map is input into three cascaded 3D ResNet blocks for deep encoding. This structure effectively integrates information from different local receptive fields by means of residual learning mechanism, thereby obtaining initial local depth features.

4. The medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network according to claim 1, characterized in that: Step 3 includes: First, the deep spatial learning module extracts global long-range dependency features of the three modalities through the ViT network, and then adaptively fuses the three modalities based on the sliding window attention mechanism; this sliding window attention mechanism can capture local information within the modality while reducing the number of parameters; Specifically, the initial local depth features extracted are: Positional encoding is added to it to obtain initial information with spatial representation. The initial feature representation after adding positional encoding is as follows: Secondly, through a depth of The ViT network extracts global depth features for each modality; let the first... The input to each Transformer module is Obtain the first by random mapping The input to the self-attention mechanism corresponding to each Transformer module is represented as follows: , , , and , Following this, the self-attention mechanism was used to... The inputs of the three modal self-attention mechanisms in each Transformer module are processed to generate corresponding self-attention outputs. , , : ； in , Representing gray matter, white matter, and PET respectively; first, residual connections are used, and then the result is obtained through multilayer perceptron processing. The final output of the Transformer block: ； in, , Representing gray matter, white matter, and PET respectively, MLP refers to a multilayer perceptron consisting of linear layers, Gelu activation layers, and Dropout layers; The output of the Transformer module is the first one. The input to each Transformer module is then used to obtain global depth features. , , ,in It is the depth of the ViT model; Finally, a sliding window attention mechanism is introduced, based on Adaptively learn feature fusion weights Subsequently, by integrating global deep features , , Adaptive weights between this mode Global deep multimodal fusion features are obtained. As shown below: ； By analyzing global deep fusion features Global average pooling and shape adjustment are performed to obtain the final global deep fusion features. .

5. The medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network according to claim 1, characterized in that: Step 4 includes: The breadth-space learning module first aggregates feature maps from different depths through breadth-width connections; then, it applies a breadth-width attention mechanism to the aggregated features. Specifically, the inputs of the self-attention mechanisms for each modality in all Transformer layers are first concatenated to obtain the inputs required for the breadth-of-attention mechanism: , , , , , ; Then, the global generalized features Bg, Bw, and Bp corresponding to the three modalities are extracted using a generalized attention mechanism, as shown below: ； ； ； in , , For the corresponding , , The dimension; Then, a chain-like cross-modal attention mechanism is used to fuse multimodal features to generate initial global generalized features. : ； ； Here, MLP refers to a multilayer perceptron consisting of linear layers, Gelu activation, and Dropout layers. Thus, the initial global generalized fusion features are obtained. ,right Shape adjustments are performed and global average pooling is applied to obtain the final global generalized fusion feature. .

6. The medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network according to claim 1, characterized in that: Step 5 and Step 6 include: The wide-space learning module utilizes average pooling and max pooling to construct cross-modal interaction features and extract key intrinsic information to generate multimodal collaborative features; Specifically, the initial features are first subjected to average pooling (AP) and max pooling (MP); Then, the features of the three modalities are added together in pairs; Next, intrinsic microstate decomposition is applied to the paired interaction features to extract the intrinsic microstates corresponding to the largest eigenvalues ​​and sum them as key intermodal collaborative information. ; then on Perform summation pooling and Normalization process yields key invariant features based on intrinsic space learning. ; Finally, the global deep fusion features are spliced ​​together. Global generalized fusion features and To obtain multimodal fusion features .

7. The medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network according to claim 1, characterized in that: Step 7 includes: Based on multimodal fusion features First, linear mapping is applied to obtain classification features. This feature was further predicted as a hidden feature associated with brain regions. Meanwhile, from Another linear mapping directly generates initial brain region features. The input required for the bidirectional cross-attention mechanism is defined as follows, using a bidirectional cross-attention mechanism and GRU updates: ； ； in, , , , , , The weights corresponding to the query (q), key (k), and value (v) required for the bidirectional cross-attention mechanism are represented. Calculate bidirectional cross attention , ; Specifically, on the one hand, based on the characteristics of task-related brain regions As a query of attention mechanisms, initial brain region features As keys and values, they capture task information to guide brain region features; on the other hand, they use... As a query, with As keys and values, structural priors of brain regions are introduced to constrain task features: ； ； The outputs in both directions are iteratively updated using GRU to obtain a brain region weight matrix that balances task relevance with intrinsic brain region features. and : ； ； in, ; Based on the above application of the bidirectional cross-attention mechanism and GRU weight update, the brain region weight matrix is ​​gradually optimized to obtain the final brain region feature representation. = Apply a linear mapping to this feature, and then... Dropout processing is performed to enhance feature representation and prevent overfitting, generating the final brain region features. The obtained features are input into the classification layer to generate brain region-based classification features. To fully integrate brain region features with multimodal fusion features, brain region classification features are analyzed. Multimodal fusion classification features An adaptive weighted combination is performed to obtain the final features used for AD classification. .

8. The medical image classification method based on a brain region learning-guided multi-spatial multimodal fusion network according to claim 1, characterized in that: Step 7 also includes: Define the weighted sum of classification cross-entropy loss and brain region loss as the loss function: ； in Indicates predicted label Indicates the true label, Indicates the predicted brain region features, The volume of gray matter in the brain. and These are adjustable weights used to adaptively balance the importance of the multimodal integrated spatial learning module and the brain region learning module.

Images