Alzheimer's disease picture classification method and system based on bimodal iterative cross attention fusion

Abstract

The application discloses an Alzheimer's disease picture classification method based on a bimodal iterative cross attention fusion, relates to the technical field of Alzheimer's disease medical image classification, and comprises the following steps: acquiring sMRI and PET image data of AD and preprocessing; cutting and data augmenting the preprocessed data; constructing a bimodal iterative cross attention fusion model as a base classifier, and training the base classifier by using the cut images; in the training, inputting the cut images into a residual network module to extract features, using a spatial feature contraction module to reduce redundant information of the features, and using a bimodal iterative cross attention module to extract complementary information in combination with an iterative learning strategy to enhance the features and fusion; inputting a verification set into the trained base classifier for classification, and selecting a base classifier with a classification accuracy greater than a preset threshold; constructing a meta-classifier, classifying the selected base classifier for the cut images to be classified, inputting the fusion features into the meta-classifier for classification, and obtaining a classification result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Alzheimer's disease medical image classification, and more particularly to an Alzheimer's disease picture classification method and system based on double-modal iterative cross attention fusion. BACKGROUND

[0002] Alzheimer's disease (AD) is a type of irreversible brain neurodegenerative disease that destroys memory and causes cognitive thinking ability to deteriorate, and its prevalence rate increases significantly with age. Classification of AD-related pictures is of great significance for AD drug research and development, AD prevention and treatment, and delay of disease progression.

[0003] Most of the current AD-related picture classification methods only analyze MRI single-modal data and usually require manual feature extraction based on expert knowledge, which has certain limitations. Although MRI single-modal data can contain a large amount of AD-related feature information, traditional single-modal image data classification methods have single feature source, lack complementary information, weak generalization ability, and high risk of overfitting, while multi-modal images based on MRI and PET can help understand the anatomical and neural changes related to AD, and studies have shown that the fusion of multi-modal features can improve the classification performance of AD. In addition, deep learning methods have made great progress in recent years, especially in extracting information features for computer vision and medical image analysis. Through data-driven automatic extraction of image features, subjective factors and labor costs caused by manual feature extraction can be reduced, so there is an urgent need for an Alzheimer's disease picture classification method based on double-modal iterative cross attention fusion. SUMMARY

[0004] The purpose of the present application is to overcome the defects of traditional single-modal picture data classification methods in the prior art, such as single feature source, lack of complementary information, weak generalization ability, and high risk of overfitting, and to provide an Alzheimer's disease picture classification method and system based on double-modal iterative cross attention fusion.

[0005] To achieve the above-mentioned purpose, the technical solutions of the present application are as follows:

[0006] In a first aspect, the present application provides an Alzheimer's disease picture classification method based on double-modal iterative cross attention fusion, comprising the following steps:

[0007] Obtaining structural magnetic resonance imaging sMRI and positron emission tomography PET image data of Alzheimer's disease and performing preprocessing;

[0008] Slicing the pre-processed sMRI and PET image data respectively to obtain an equal number of sMRI and PET sliced ​​images and randomly selecting a preset percentage of the sliced ​​images for data augmentation;

[0009] Constructing a bimodal iterative cross-attention fusion model including a residual network module, a spatial feature shrinkage module, and a bimodal iterative cross-attention module, using the bimodal iterative cross-attention fusion model as a base classifier, and using a number of the sliced ​​images after data augmentation to train a corresponding number of the base classifiers;

[0010] During the training process of each base classifier, the sliced ​​images of the sMRI and PET are input into a residual network module to extract sMRI features and PET features, a spatial feature shrinkage module is used to reduce redundant information in the sMRI features and PET features, and then a bimodal iterative cross attention module is used in combination with an iterative learning strategy to extract complementary information to enhance the sMRI features and PET features, and then the enhanced sMRI features and PET features are fused;

[0011] A validation set is preset and input into the trained base classifier for classification, and a base classifier whose classification accuracy in the validation set is greater than a preset threshold is selected;

[0012] A meta-classifier is constructed, and the selected base classifier is used to classify the sMRI and PET slice images to be classified. The fusion features obtained in the classification process are input into the meta-classifier for classification to obtain the final classification result.

[0013] Preferably, the pretreatment includes:

[0014] The sMRI images were de-skulled, registered to the Montreal Neurological Institute standard space, and smoothed. Subsequently, grayscale normalization was performed so that the pixel value of each sMRI image data was between 0 and 1.

[0015] The PET images were registered to the PET template, affine-registered to the corresponding sMRI template, and the voxels outside the brain in the image data were deleted, intensity normalized, and converted to images with uniform isotropic resolution.

[0016] After preprocessing, the sMRI and PET images have the same size and voxel size.

[0017] Preferably, the step of slicing the pre-processed sMRI and PET image data to obtain an equal number of sMRI and PET sliced ​​images and randomly selecting a preset percentage of the sliced ​​images for data augmentation comprises:

[0018] The sMRI and PET image data were padded with zeros on both sides of each dimension, and then a triple loop was used to traverse the padded image data, taking one cubic block at a time, and finally obtaining an equal number of sMRI and PET sliced ​​images. 50% of the sliced ​​images were randomly flipped along the X-axis, Y-axis, or Z-axis for data augmentation.

[0019] Preferably, the step of inputting the sliced ​​sMRI and PET images into a residual network module to extract sMRI features and PET features, using a spatial feature shrinkage module to reduce redundant information in the sMRI features and PET features, then using a bimodal iterative cross-attention module in combination with an iterative learning strategy to extract complementary information to enhance the sMRI features and PET features, and then fusing the enhanced sMRI features and PET features comprises the following steps:

[0020] The preprocessed sMRI and PET block images are respectively used as inputs of the residual network module for feature extraction to obtain the three-dimensional feature vectors of sMRI and PET, which are then respectively subjected to feature shrinkage by the spatial feature shrinkage module to obtain the two-dimensional feature vectors of sMRI and PET, which are respectively input into the bimodal iterative cross-attention module for feature enhancement to obtain enhanced two-dimensional feature vectors of sMRI and PET, respectively. The enhanced two-dimensional feature vector of sMRI is concatenated element-by-element with the three-dimensional feature vector of sMRI to obtain the enhanced three-dimensional feature vector of sMRI. At the same time, the enhanced two-dimensional feature vector of PET is concatenated element-by-element with the three-dimensional feature vector of PET to obtain the enhanced three-dimensional feature vector of PET, the three-dimensional feature vector of sMRI, the enhanced three-dimensional feature vector of sMRI, the three-dimensional feature vector of PET, and the enhanced three-dimensional feature vector of PET are connected channel-by-channel and output to the fully connected layer for classification, and the classification result is output.

[0021] Preferably, the residual network module includes an sMRI branch and a PET branch, and both branches include a three-dimensional convolution layer, a layer normalization layer, a batch normalization layer, a ReLU activation function layer, a maximum pooling operation layer and three residual blocks.

[0022] Preferably, the output of the previous residual block among the three residual blocks serves as the input of the next residual block, and the number of feature channels of the three residual blocks is 64, 128, and 256, respectively.

[0023] Preferably, the spatial feature shrinkage module includes a pooling operation and a convolution operation;

[0024] For the convolution operation, a dimensionality reduction method based on convolution operation is used, and the expression of this method is as follows:

[0025] Fconv = conv 1×1 (Reshape(F))

[0026] wherein F represents a feature map formed by a three-dimensional feature vector of sMRI or PET output by a residual network module, and conv represents compression of the feature map by 1x1 convolution, specifically, conversion of spatial information of the feature into a channel dimension by reshaping the dimension of the feature map, and then compression of the channel dimension by 1x1 convolution operation;

[0027] For the pooling operation, an adaptive aggregation method of average pooling and maximum pooling is adopted, which enhances the feature expression ability of the network by adaptively aggregating average pooling and maximum pooling while retaining significant features and global background information. The formula of the adaptive aggregation method is as follows:

[0028] F a = AvgPooling(F, S)

[0029] F m = MaxPooling(F, S)

[0030] F o = a · F a + (1-a) · F m

[0031] wherein S represents a scale factor of the feature map, F a and F m represent the feature maps compressed by average pooling AvgPooling(·) and maximum pooling MaxPooling(·) respectively, a is a weight for adaptively aggregating the feature maps after average pooling and maximum pooling, a is between 0 and 1, and is a learnable parameter.

[0032] Preferably, the dual-modality iterative cross-attention module includes an sMRI branch and a PET branch, each branch applies a learnable coefficient, and each branch includes two cross-attention modules, each cross-attention module includes a multi-head attention module, a layer normalization layer, and a feedforward neural network, and complementary information across modalities is extracted by the two cross-attention modules of the sMRI branch and the PET branch respectively to enhance the features of sMRI and PET.

[0033] Preferably, the dual-modality iterative cross-attention module combines the iterative learning strategy with the cross-attention module, gradually deepens the network depth by multiple iterations and sharing parameters, and gradually optimizes the complementary information across modalities without increasing the number of parameters. The expression of the iterative learning strategy is as follows:

[0034]

[0035] in, represents the output of the bimodal iterative cross attention module after n iterations, {V M , V P} represents the input of the bimodal iterative cross attention module, f ICA (·) represents a bimodal iterative crisscross attention module. The output of each iterative operation is used as the input of the next iterative operation, and parameters are shared between each iterative operation. In addition, at the output, the bimodal iterative crisscross attention module converts the output sequence and into a feature map, which is then recalibrated to the original size of the feature map through bilinear interpolation.

[0036] In a second aspect, the present invention provides an Alzheimer's disease image classification system based on bimodal iterative cross-attention fusion, applying the Alzheimer's disease image classification method based on bimodal iterative cross-attention fusion described in the above technical solution, the system comprises:

[0037] Data acquisition and preprocessing module, used to acquire structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) imaging data of Alzheimer's disease and perform preprocessing;

[0038] An image slicing and data augmentation module, configured to slice the pre-processed sMRI and PET image data into blocks, obtain an equal number of sMRI and PET sliced ​​images, and randomly select a preset percentage of the sliced ​​images for data augmentation;

[0039] A model construction and training module is used to construct a bimodal iterative cross-attention fusion model including a residual network module, a spatial feature shrinkage module, and a bimodal iterative cross-attention module, and use the bimodal iterative cross-attention fusion model as a base classifier and use a number of the sliced ​​images after data augmentation to train a corresponding number of the base classifiers;

[0040] a feature processing module for inputting the sliced ​​sMRI and PET images into a residual network module during the training process of each base classifier to extract sMRI features and PET features, using a spatial feature shrinkage module to reduce redundant information in the sMRI features and PET features, then using a bimodal iterative cross-attention module in combination with an iterative learning strategy to extract complementary information to enhance the sMRI features and PET features, and then fusing the enhanced sMRI features and PET features;

[0041] A base classifier selection module is used to preset a verification set and input it into the trained base classifier for classification, and select a base classifier whose classification accuracy in the verification set is greater than a preset threshold;

[0042] The classification result acquisition module is used to construct a meta-classifier, use the selected base classifier to classify the sMRI and PET slice images to be classified, input the fusion features obtained in the classification process into the meta-classifier for classification, and obtain the final classification result.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention constructs and trains a dual-modal iterative cross-attention fusion model as a base classifier, combines iterative learning strategies to avoid overfitting, and more efficiently and accurately learns to extract complementary information between and within modalities and enhances the features of sMRI and PET images. A meta-classifier is used to classify the enhanced fusion features of sMRI and PET images obtained by the selected base classifier to obtain classification results. This effectively solves the defects of traditional single-modal image data classification methods, such as a single feature source, lack of complementary information, weak generalization ability, and high overfitting risk, and improves the accuracy of Alzheimer's disease image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a schematic diagram of the steps of the Alzheimer's disease image classification method based on bimodal iterative cross-attention fusion according to Example 1 of the present application;

[0046] Figure 2 Schematic diagram of the steps for preprocessing the sMRI image data of Example 1 of the present application;

[0047] Figure 3 Schematic diagram of the Bimodal Iterative Cross-Attention Fusion Integration Framework (BICAFE) provided in Example 1 of the present application;

[0048] Figure 4 A schematic diagram of the bimodal iterative cross attention fusion model (BICAF) provided in Example 1 of the present application;

[0049] Figure 5 A schematic diagram of a residual block provided in Example 1 of the present application;

[0050] Figure 6 A schematic diagram of a bimodal iterative cross-attention module provided in Example 1 of the present application;

[0051] Figure 7 This is a result diagram of the visual analysis of the experiment using the method of the present invention provided in Example 3 of the present application. DETAILED DESCRIPTION

[0052] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0053] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0054] Example 1

[0055] See also Figure 1 Embodiment 1 of the present invention provides an Alzheimer's disease image classification method based on bimodal iterative cross-attention fusion, comprising the following steps S1-S6:

[0056] S1: Acquire structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) imaging data for Alzheimer's disease and perform preprocessing.

[0057] In a preferred embodiment, the structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) imaging data acquired in step S1 are derived from the publicly available Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu), the dataset used in the embodiments of the present invention being available on this website (www.loni.ucla.edu / ADNI). The primary purpose of the ADNI is to examine whether a series of magnetic resonance imaging (MRI), positron emission tomography (PET), other biomarkers, and clinical and neuropsychological assessments can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer's disease (AD).

[0058] Since the raw sMRI and PET image data downloaded directly from the ADNI database cannot be used directly, they need to be preprocessed to make the data from different modalities more similar.

[0059] In this embodiment 1, two preprocessing methods are used to preprocess sMRI and PET data respectively. For the preprocessing of sMRI image data, please refer to Figure 2The sMRI image is skull-stripped, registered to the Montreal Neurological Institute (MNI) standard space and smoothed, and then normalized to have pixel values between 0 and 1. In a specific embodiment, the original image data in DICOM format can be converted to NII format using mricron software, and the NII format MRI can be skull-stripped, registered to the MNI standard space, smoothed, and the like using the CAT12 toolbox in MATLAB software, and then each sMRI data is normalized to have pixel values between 0 and 1.

[0060] Regarding the preprocessing of PET image data, after the PET original image data is converted to hdr format, the hdr format PET image is registered to the PET template using the spm12 toolbox in MATLAB software, and then affine registration is performed to the corresponding sMRI, while deleting the voxels outside the brain in the image data, and performing intensity normalization and conversion to an image with a uniform isotropic resolution of 8mm FWHM (Full Width at Half Maximum), and the image is smoothed to the same degree in all directions. The size of the sMRI and PET image after preprocessing is 121mm x 145mm x 121mm, and the voxel size is 1.5.

[0061] S2: The preprocessed sMRI and PET image data are cut into blocks respectively to obtain an equal number of sMRI and PET block images, and a preset percentage of the block images are randomly taken for data augmentation.

[0062] For the preprocessed sMRI and PET image data, in order to reduce the difficulty of training the neural network model and the consumption of computing resources, and to train multiple base classifiers simultaneously, Example 1 of the present invention cuts the three-dimensional images of the two modalities into 150 small blocks. To obtain cubic blocks and reduce the loss of edge information caused by the blocks, the image of size 121mm×145mm×121mm is first padded with zeros on both sides of each dimension to obtain an image of size 125mm×150mm×125mm. Then, a triple loop is used to traverse the padded image data, taking a cubic block of size 25mm×25mm×25mm each time, and finally obtaining 150 blocks for sMRI and PET respectively. Since the present invention subsequently uses a base classifier with a dual-branch structure, the data of the two modalities need to be stored separately, and each block must also be stored separately. The storage method is as follows: For the modalities sMRI and PET, 150 folders are created and named block0 to block149 respectively. Block 0 corresponds to the first block, and the first blocks of all samples in the same modality are also stored in block 0. Similarly, block 149 corresponds to the 150th block of all samples in the same modality. This storage method not only facilitates data reading for the base classifier but also ensures that the input of each sample is the same block, avoiding training problems caused by data confusion.

[0063] In addition, in order to increase data diversity, Example 1 of the present invention performs data augmentation on the sliced ​​images. The data augmentation method is to randomly select 50% of the slices and randomly flip them along the X-axis, Y-axis or Z-axis.

[0064] S3: Construct a bimodal iterative cross-attention fusion model including a residual network module, a spatial feature shrinkage module and a bimodal iterative cross-attention module, use the bimodal iterative cross-attention fusion model as a base classifier, and use a number of the sliced ​​images after data augmentation to train a corresponding number of the base classifiers.

[0065] See also Figure 4 , Figure 4 Schematic diagram of the bimodal iterative cross-attention fusion model (BICAF) provided in Example 1 of the present application. In this Example 1, the bimodal iterative cross-attention fusion model (BICAF) is used as a base classifier, including a residual network module, a spatial feature contraction module and a bimodal iterative cross-attention module, and also includes a fully connected layer FC. BICAF is based on 150 pre-processed sMRI and PET slice images. 150 base classifiers are trained as input. First, feature extraction is performed through the residual network module to obtain a three-dimensional feature vector Then, the feature shrinking is performed through the spatial feature shrinking module SFS to obtain a two-dimensional feature vector Subsequently, the two-dimensional feature vectors of the sMRI and the PET are input into the bimodal iterative cross attention module ICA to extract intra-modal and inter-modal complementarity for feature enhancement and the three-dimensional feature vectors of the sMRI and the PET Element-wise concatenation is performed to obtain the enhanced three-dimensional feature vector Finally, the enhanced three-dimensional feature vector is concatenated with the three-dimensional feature vector Channel-wise dimension concatenation is performed, and then a full connection layer FC is used for classification to obtain a classification result

[0066] Please refer to Figure 4 The residual network module (ResNet) used for feature extraction in the bimodal iterative cross attention fusion model in Embodiment 1 is a 3DCNN based on the ResNet-18 architecture. Compared with the standard 3D ResNet-18, the network is pruned and optimized in Embodiment 1, so that it is more suitable for small-scale patch data sets, effectively reduces the computational overhead of the model while maintaining sufficient feature capacity, and prevents overfitting. The residual network module ResNet in Embodiment 1 includes an sMRI branch and a PET branch. Both branches include a three-dimensional convolution layer (Conv 3D), a layer normalization layer (LN), a ReLU activation function layer, a maximum pooling operation layer (Max Pooling), and three residual blocks (ResidualBlock) connected in sequence. Taking the sMRI branch as an example, the input of the three-dimensional convolution layer is wherein 25x25x25 is the spatial dimension HxWxD of the sMRI, and 1 represents the number of channels. First, a three-dimensional convolution operation with 64 filters, a kernel size of 7x7x7, a stride of 2, and a padding of same can be represented as:

[0067] H0 = ReLU(LN(Conv3D(P M ,64,7,2,same)))

[0068] wherein LN represents layer normalization; then, a maximum pooling operation is performed with a pooling window of 3x3x3 and a stride of 2, and the maximum pooling operation can be represented as:

[0069] H I = MaxPool3D(H0,3,2,same)

[0070] Then, three Residual Blocks are passed through, and each residual block can be represented as frcs (H, F, s). Where H is the input feature vector, F is the number of output feature channels, and s is the stride of the convolution.

[0071] See also Figure 5 , Figure 5 Schematic diagram of the residual block provided in Example 1 of the present application. In the residual block, H first passes through a three-dimensional convolution layer, whose output feature channels are F, the kernel size is 3×3×3, and the stride is s. It then passes through batch normalization (BN) and ReLU activation function, and then passes through a three-dimensional convolution layer whose output feature channels are F, the kernel size is 3×3×3, and the stride is 1. It then passes through another BN to obtain the feature vector Y. Finally, a residual connection is performed to obtain the final output:

[0072] H out =ReLU(Y+S)

[0073] Where S is defined as:

[0074]

[0075] In order to enable ResNet to gradually increase its feature expression capability from the initial stage of feature extraction to the aggregation of high-order features, the three Residual Blocks use different F and s, which are expressed as follows:

[0076] H2=f res (H1,64,1)

[0077] H3=f res (H2,128,2)

[0078] F M =f res (H3,256,2)

[0079] Among them, the output of the previous residual block in the three residual blocks is used as the input of the next residual block and the number of feature channels of the three residual blocks is 64, 128, and 256 respectively. H2 is the output of the first Residual Block, H3 is the output of the second Residual Block, and F M is the final output of the sMRI branch in the residual network module ResNet. Similarly, after passing through the PET branch in the residual network module ResNet, the PET feature vector F P .

[0080] To further reduce redundant information in features, highlight key features, and reduce subsequent computational costs, a spatial feature shrinkage module (SFS) is applied before the ICA module. SFS consists of two parts: pooling and convolution.

[0081] For the convolution operation, a dimensionality reduction method based on convolution operation is first used, as shown in the following formula.

[0082] F conv =conv 1×1 (Reshape(F))

[0083] Among them, F represents the feature map formed by the three-dimensional feature vector of sMRI or PET output by the residual network module, and conv represents the compression of the feature map by 1×1 convolution. Specifically, the spatial information of the feature is converted into the channel dimension by reshaping the dimension of the feature map, and then the channel dimension is compressed by 1×1 convolution operation.

[0084] For pooling operations, traditional average pooling and maximum pooling methods each have their own advantages and disadvantages. Maximum pooling focuses more on local salient features, such as edges and textures, but may lose background information; average pooling can retain overall information more smoothly, but is prone to losing details. Therefore, the SFS module of Example 1 of the present invention adopts a method of adaptively aggregating average pooling and maximum pooling. By adaptively aggregating average pooling and maximum pooling, salient features and global background information can be retained at the same time, thereby enhancing the feature expression ability of the network. The formula of the adaptive aggregation method is as follows:

[0085] F a =AvgPooling(F,S)

[0086] F m =MaxPooling(F,S)

[0087] F o =α·F a +(1-α)·F m

[0088] Among them, S represents the scale factor of the feature map, F a and F m They represent the feature maps compressed by average pooling (AvgPooling) and maximum pooling (MaxPooling), respectively. α is the weight used to adaptively aggregate the feature maps after average pooling and maximum pooling. The value of α is between 0 and 1 and is a learnable parameter.

[0089] The ICA module combines a novel iterative learning strategy with a cross-attention module (CA) to extract complementary information from inter-modality and intra-modality features to enhance the features of sMRI and PET, further improving the performance of the model. Unlike previous studies that capture local features of different modalities, the CA proposed in Example 1 of the present invention enables a single modality to learn more complementary information from the auxiliary modality (i.e., another modality) from a global perspective.Figure 6 , Figure 6 The schematic diagram of the bimodal iterative cross-attention module CA provided for Embodiment 1 of the application, CA is composed of a multi-head attention module, a layer normalization layer (Layer Norma, LN) and a feed-forward neural network (Feed-Forward Network, FFN).

[0090] S4: In the training process of each base classifier, the sMRI and PET patch images are input into the residual network module to extract sMRI features and PET features, the spatial feature contraction module is used to reduce redundant information in the sMRI features and the PET features, and then the bimodal iterative cross-attention module is used to extract complementary information to enhance the sMRI features and the PET features using an iterative learning strategy, and then the enhanced sMRI features and the PET features are fused.

[0091] S5: A preset verification set is input into the trained base classifier for classification, and a base classifier with a classification result accuracy greater than a preset threshold in the verification set is selected.

[0092] In Embodiment 1 of the application, 150 sMRI and PET patch images are used to train 150 base classifiers respectively, a preset verification set is input into the trained base classifier for classification, and a base classifier with an accuracy greater than 75% on the verification set is selected.

[0093] S6: A meta-classifier is constructed, the selected base classifier is used to classify the sMRI and PET patch images to be classified, the fusion features obtained in the classification process are input into the meta-classifier for classification, and a final classification result is obtained.

[0094] In Embodiment 1 of the application, the meta-classifier is composed of a convolutional layer and a fully connected layer, the input of which is the fusion features obtained in the classification process of the sMRI and PET patch images to be classified by the base classifier with an accuracy greater than 75%, and the final classification result is obtained in the meta-classifier under the guidance of the label.

[0095] Please refer to Figure 3 , Figure 3 The schematic diagram of the bimodal iterative cross-attention fusion integrated framework (BICAFE) formed by the above steps of Embodiment 1 of the application, wherein the Selection module is used for selecting the base classifier in step S5, and the Meta-classifier module represents the meta-classifier.

[0096] The bimodal iterative cross-attention module ICA and the iterative learning strategy are further described as follows:

[0097] The dual-modal iterative cross-attention module includes an sMRI branch and a PET branch, each of which applies a learnable coefficient and each of which includes two cross-attention modules, each of which includes a multi-head attention module, a layer normalization layer, and a feedforward neural network, cross-modal complementary information is extracted through the two cross-attention modules of the sMRI branch and the PET branch respectively to enhance the features of sMRI and PET.

[0098] Given the input feature map F M and The present application first flattens each feature map into a set of tokens and adds a learnable position embedding, which is a trainable parameter with a dimension of HWDxC, to encode the spatial information between different tokens. Subsequently, a set of tokens of sMRI and PET with position embedding is obtained: V M and V P , which are taken as the input of the CA module. Since there are usually differences in spatial expression between sMRI and PET features, both the sMRI branch and the PET branch in the ICA module use dual CA to extract intra-modal and inter-modal complementary information to enhance sMRI and PET features. The CA module of the sMRI branch is taken as an example for description as follows:

[0099] For the CA module of the sMRI branch, first, the tokens V M of sMRI are projected into two separate matrices The tokens V P of PET are projected into two separate matrices to perform multi-head attention calculation to obtain a feature vector C M , as shown in the following formula:

[0100] C M =Concat(head1,...,head h )W O

[0101] Wherein, the attention calculation formula of the i-th head is:

[0102]

[0103] Then, the feature vector C M is re-projected back to the original space through a nonlinear transformation and added to the input sequence through a residual connection, as shown in the following formula:

[0104] V′ M =α·V M +β·C M W O

[0105] Wherein, The output weight matrix before LN is represented. LN is then passed. Finally, the FFN of the same two-layer fully connected layer as in the standard Transformer is used to further refine the global information to improve the robustness and accuracy of the model, and output the enhanced features The formula is as follows:

[0106]

[0107] In order to adaptively learn data from different branches to achieve performance improvement. The present application applies a learnable coefficient on each branch of the residual connection, and in the formula, alpha, beta, gamma and delta are learnable parameters, which are initialized to 1 during the training process.

[0108] Similarly, another CA module is used to enhance the features of the PET branch, but contrary to the sMRI branch, V P Q, V M Then project K, V to achieve more comprehensive capture of the correlation between modalities, thereby improving the performance of the model.

[0109] Traditional methods usually improve performance by stacking multiple modules, but this strategy of increasing model depth not only greatly increases the number of parameters, but also may cause overfitting. In order to overcome the shortcomings of traditional methods, the iterative learning strategy used by the present application gradually deepens the network depth through multiple iterations and shares parameters, while gradually optimizing the complementary information across modalities without increasing the number of parameters, dynamically allocating weights to sMRI and PET features to achieve efficient fusion. The formula of this adaptive aggregation method is as follows:

[0110]

[0111] Wherein, The output of the ICA module after n iterations is represented, {V M ,V P} represents the input of ICA. f ICA (·) represents the ICA module proposed by the present application, which integrates two CA modules of the sMRI branch and the PET branch respectively. The output of each iteration operation is used as the input of the next iteration operation, and the parameters are shared between each iteration operation. In addition, the output sequence of the ICA module is converted into a feature map, and then recalibrated to the original size of the feature map through bilinear interpolation.

[0112] Embodiment 2

[0113] The embodiment 2 based on the embodiment 1 provides an Alzheimer's picture classification system based on a bimodal iterative cross-attention fusion, applies the Alzheimer's picture classification method based on the bimodal iterative cross-attention fusion in the embodiment 1, and the system comprises:

[0114] A data acquisition and preprocessing module is configured to acquire structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) image data of Alzheimer's disease and perform preprocessing.

[0115] An image slicing and data augmentation module is configured to slice the preprocessed sMRI and PET image data respectively, to obtain a same number of sMRI and PET sliced images, and to randomly take a preset percentage of the sliced images for data augmentation.

[0116] A model construction and training module is configured to construct a bimodal iterative cross-attention fusion model comprising a residual network module, a spatial feature shrinking module and a bimodal iterative cross-attention module, to use the bimodal iterative cross-attention fusion model as a base classifier, and to train a corresponding number of base classifiers using a number of data-augmented sliced images.

[0117] A feature processing module is configured to, in the training process of each base classifier, input the sMRI and PET sliced images into the residual network module to extract sMRI features and PET features, use the spatial feature shrinking module to reduce redundant information in the sMRI features and the PET features, and then use the bimodal iterative cross-attention module to extract complementary information to enhance the sMRI features and the PET features using an iterative learning strategy, and then fuse the enhanced sMRI features and the PET features.

[0118] A base classifier selection module is configured to preset a verification set and input the verification set into the trained base classifiers for classification, and to select a base classifier with a classification result accuracy greater than a preset threshold in the verification set.

[0119] A classification result acquisition module is configured to construct a meta-classifier, to use the selected base classifier to classify sMRI and PET sliced images to be classified, to input fused features obtained in the classification process into the meta-classifier for classification, and to obtain a final classification result.

[0120] Other steps or technical details of the embodiment 2 are the same as those of the embodiment 1, and will not be described here.

[0121] Embodiment 3

[0122] In Example 3 of the present invention, based on Example 1 and Example 2, to verify the effectiveness and advancement of the present invention, imaging data of subjects with both sMRI and PET image data were randomly selected for experiments and visual analysis was performed, as follows:

[0123] In Example 3 of the present invention, a total of 418 subjects with both sMRI and PET image data were selected from the ADNI database. The subjects were divided into two groups (AD and HC) (192 AD and 226 HC). The AD group consisted of subjects with clinically diagnosed AD symptoms, and the HC group consisted of subjects without any cognitive impairment. The experiment used five classification indicators: classification accuracy (ACC), area under the receiver operating characteristic (ROC) curve (AUC), sensitivity (SEN), specificity (SPE), and Matthews correlation coefficient (MCC) to measure model performance.

[0124] A 5-fold cross-validation strategy was used to evaluate the classification performance.

[0125]

[0126] Table 1 Comparative experimental results

[0127] Table 1 shows the comparative experimental results of the proposed method compared with current mainstream multimodal fusion methods. The comparison methods include the 2DCNN+BGRU method proposed by Huang et al., the dual-branch 3D convolutional neural network model proposed by Sharma et al., the MCAD framework proposed by Zhang et al., the MADDi method proposed by Golovanevsky et al., and the DMFF method proposed by Shen et al. As shown in Table 1, the proposed method outperforms the other methods in terms of classification performance. Notably, using the same dataset, the proposed method achieves ACCs that are 4.9%, 4.9%, 2.4%, 0.8%, and 1.5% higher than those of 2DCNN+BGRU, Sharma, MCAD, MADDi, and DMFF, respectively. The AUCs are 2.2%, 1%, 5.7%, 0.7%, and 0.7% higher, and the MCCs are 9.9%, 9.5%, 4.7%, 0.8%, and 3% higher than those of DMFF. This demonstrates that the proposed method better captures important features in the data, thereby improving classification accuracy. Furthermore, the improved AUC further reflects the model's classification capabilities at different thresholds, demonstrating its reliability in practical applications. The improvement in MCC, particularly the significant 9.9% increase, demonstrates the robustness of our proposed method in dealing with class imbalance. It also demonstrates the effectiveness of our method in handling complex datasets, further demonstrating its value in real-world applications.

[0128]

[0129] Table 2 Experimental comparison results of single-modal and dual-modal fusion

[0130] Table 2 shows an experimental comparison of single-modality and dual-modality fusion in Example 3 of the present invention. In the AD vs. HC experiment, the PET modality performed better than the sMRI modality overall, and the classification performance of dual-modality fusion was superior to that of either single sMRI or single PET modalities. The ACC improved by 6.6% and 1% compared to single sMRI and single PET, respectively. The AUC improved by 2.9% and 0.6%, and the MCC improved by 13% and 2%, respectively. Furthermore, dual-modality fusion had a lower standard deviation. This demonstrates that the dual-modality fusion method proposed in the present invention can effectively combine the structural information provided by sMRI with the functional and metabolic information provided by PET, thereby achieving information complementarity and improving the accuracy of AD image classification.

[0131] In order to better illustrate how different modalities affect network performance and make the model more interpretable, Example 3 of the present invention provides a visual result analysis, using Grad-CAM to display the areas of interest of the single-modality and bimodal fusion models.

[0132] See also Figure 7 , Figure 7The results of the visualization analysis of the experiment using the method of the present invention provided in Example 3 of the present invention are shown in FIG. Figure 7 As shown in the figure, the results show that the single-modality model pays attention to different areas of sMRI and PET, which indicates that the model extracts different brain area information for different modalities. The areas of interest of the dual-modality fusion model overlap with those of the single-modality model, and also present more comprehensive information. In order to further display the brain areas of interest of the single-modality model and the dual-modality fusion model, the present invention uses Brainnetome Atlas to annotate Patch106 brain areas and compares the specific brain areas of interest of the models under different modalities, as shown in Table 3 below.

[0133]

[0134] Table 3 Specific brain regions focused by the model under different modalities

[0135] Table 3 shows that the main brain regions that sMRI focuses on include those related to brain morphology and structural changes, such as the superior temporal gyrus (73), prefrontal cortex (77, 79), and areas related to memory and emotional processing, such as the hippocampus (215). PET, on the other hand, tends to focus on brain regions related to metabolic activity, blood flow, and neurotransmitter distribution, such as the ventral insular cortex (165) and dorsal insular cortex (167), which are related to cognitive and emotional regulation of the brain. The brain regions that the dual-modality fusion model focuses on not only include the significant regions of sMRI and PET, but also cover the intersection of multiple brain regions and have a wider range of brain functional connections. This shows that by fusing sMRI and PET data, the dual-modality fusion model can capture more comprehensive brain region characteristics and improve the ability to identify and classify complex brain lesions. Compared with a single modality, the dual-modality fusion model can better reflect the interactions between different brain regions and may reveal subtle differences that cannot be identified by a single modality.

[0136] The other steps of this embodiment 3 are the same as those of embodiment 1 or embodiment 2 and will not be repeated here.

[0137] In summary, the present invention constructs and trains a bimodal iterative cross-attention fusion model as a base classifier, combines iterative learning strategies to avoid overfitting, and more efficiently and accurately learns to extract complementary information between and within modalities and enhances the features of sMRI and PET images. A meta-classifier is used to classify the enhanced fusion features of sMRI and PET images obtained by the selected base classifier to obtain classification results, which effectively solves the defects of traditional single-modal image data classification methods, such as single feature source, lack of complementary information, weak generalization ability and high risk of overfitting, and improves the accuracy of Alzheimer's disease image classification. The present invention proposes a new iterative cross-attention fusion strategy and combines it with a bimodal cross-attention fusion module to strengthen the memory of complementary information from inter-modal and intra-modal features, thereby more efficiently and accurately learning complementary information of different modalities and performing AD image classification. Experimental results prove the effectiveness of this method, with an accuracy of 94.3% in AD and HC. Compared with other methods, it has been greatly improved. Visual analysis confirmed that the method of the present invention extracted different brain region information for different modalities, and the dual-modal fusion model complemented the brain region information of different modalities, thereby improving the accuracy of AD image classification and recognition.

Claims

1. Alzheimer's disease image classification method based on bimodal iterative cross attention fusion, characterized by: The following steps are involved: Acquire structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) imaging data of Alzheimer's disease and perform preprocessing; Slicing the pre-processed sMRI and PET image data respectively to obtain an equal number of sMRI and PET sliced ​​images and randomly selecting a preset percentage of the sliced ​​images for data augmentation; Constructing a bimodal iterative cross-attention fusion model including a residual network module, a spatial feature shrinkage module, and a bimodal iterative cross-attention module, using the bimodal iterative cross-attention fusion model as a base classifier, and using a number of the sliced ​​images after data augmentation to train a corresponding number of the base classifiers; During the training process of each base classifier, the sliced ​​images of the sMRI and PET are input into a residual network module to extract sMRI features and PET features, a spatial feature shrinkage module is used to reduce redundant information in the sMRI features and PET features, and then a bimodal iterative cross attention module is used to extract complementary information to enhance the sMRI features and PET features, and then the enhanced sMRI features and PET features are fused; A validation set is preset and input into the trained base classifier for classification, and a base classifier whose classification accuracy in the validation set is greater than a preset threshold is selected; Constructing a meta-classifier, using the selected base classifier to classify the sMRI and PET slice images to be classified, inputting the fusion features obtained in the classification process into the meta-classifier for classification, and obtaining a final classification result; The bimodal iterative cross-attention module includes an sMRI branch and a PET branch, each branch applies a learnable coefficient and each branch includes two cross-attention modules, each cross-attention module includes a multi-head attention module, a layer normalization layer and a feedforward neural network, and extracts cross-modal complementary information through the two cross-attention modules of the sMRI branch and the PET branch to enhance the features of sMRI and PET; The bimodal iterative cross-attention module combines the iterative learning strategy with the cross-attention module, gradually deepens the network depth through multiple iterations and shared parameters, and gradually optimizes the cross-modal complementary information without increasing the number of parameters.

2. The Alzheimer's disease image classification method based on bimodal iterative cross attention fusion according to claim 1 is characterized in that: The pretreatment includes: The sMRI images were de-skulled, registered to the Montreal Neurological Institute standard space, and smoothed. Subsequently, grayscale normalization was performed so that the pixel value of each sMRI image data was between 0 and 1. The PET images were registered to the PET template, affine-registered to the corresponding sMRI template, and the voxels outside the brain in the image data were deleted, intensity normalized, and converted to images with uniform isotropic resolution. After preprocessing, the sMRI and PET images have the same size and voxel size.

3. The Alzheimer's disease image classification method based on bimodal iterative cross attention fusion according to claim 2 is characterized in that: The steps of slicing the pre-processed sMRI and PET image data to obtain equal numbers of sMRI and PET sliced ​​images and randomly selecting a preset percentage of the sliced ​​images for data augmentation include: The sMRI and PET image data were padded with zeros on both sides of each dimension, and then a triple loop was used to traverse the padded image data, taking one cubic block at a time, and finally obtaining an equal number of sMRI and PET sliced ​​images. 50% of the sliced ​​images were randomly flipped along the X-axis, Y-axis, or Z-axis for data augmentation.

4. The Alzheimer's disease image classification method based on bimodal iterative cross attention fusion according to claim 1 is characterized in that: The method inputs the sliced ​​images of the sMRI and PET into a residual network module to extract sMRI features and PET features, uses a spatial feature shrinkage module to reduce redundant information in the sMRI features and PET features, then uses a bimodal iterative cross attention module to extract complementary information to enhance the sMRI features and PET features, and then fuses the enhanced sMRI features and PET features, including the following steps: The preprocessed sMRI and PET block images are used as inputs of the residual network module for feature extraction to obtain three-dimensional feature vectors of sMRI and PET, which are then subjected to feature shrinkage by the spatial feature shrinkage module to obtain two-dimensional feature vectors of sMRI and PET, respectively. The two-dimensional feature vectors of sMRI and PET are input into the bimodal iterative cross-attention module for feature enhancement to obtain enhanced two-dimensional feature vectors of sMRI and PET, respectively. The enhanced two-dimensional feature vector of sMRI is concatenated element-by-element with the three-dimensional feature vector of sMRI to obtain an enhanced three-dimensional feature vector of sMRI. At the same time, the enhanced two-dimensional feature vector of PET is concatenated element-by-element with the three-dimensional feature vector of PET to obtain an enhanced three-dimensional feature vector of PET. The three-dimensional feature vector of sMRI, the enhanced three-dimensional feature vector of sMRI, the three-dimensional feature vector of PET, and the enhanced three-dimensional feature vector of PET are concatenated channel-by-channel and output to the fully connected layer for classification, and the classification result is output.

5. The Alzheimer's disease image classification method based on bimodal iterative cross attention fusion according to claim 4 is characterized in that: The residual network module includes an sMRI branch and a PET branch, and both branches include a three-dimensional convolution layer, a layer normalization layer, a ReLU activation function layer, a maximum pooling operation layer and three residual blocks.

6. The Alzheimer's disease image classification method based on bimodal iterative cross attention fusion according to claim 5 is characterized in that: The output of the previous residual block in the three residual blocks serves as the input of the next residual block, and the number of feature channels of the three residual blocks is 64, 128, and 256, respectively.

7. The Alzheimer's disease image classification method based on bimodal iterative cross attention fusion according to claim 4 is characterized in that: The spatial feature shrinkage module includes a pooling operation and a convolution operation; For the convolution operation, a dimensionality reduction method based on convolution operation is used, and the expression of this method is as follows: in, Represents the feature map formed by the three-dimensional feature vector of sMRI or PET output by the residual network module, conv represents the feature map formed by Convolution compresses the feature map. Specifically, the spatial information of the feature is converted into the channel dimension by reshaping the dimension of the feature map, and then the channel dimension is compressed using a 1×1 convolution operation; For the pooling operation, a method of adaptively aggregating average pooling and maximum pooling is adopted. By adaptively aggregating average pooling and maximum pooling, significant features and global background information are retained at the same time, thereby enhancing the feature expression ability of the network. The formula of the method of adaptively aggregating average pooling and maximum pooling is as follows: in, represents the scale factor of the feature map, and Respectively represent the average pooling and max pooling Compressed feature maps, is the weight used to adaptively aggregate the feature maps after average pooling and maximum pooling, The value of is between 0 and 1 and is a learnable parameter.

8. The Alzheimer's disease image classification method based on bimodal iterative cross attention fusion according to claim 1 is characterized in that: The expression of the iterative learning strategy is as follows: in, Indicates passing The output of the bimodal iterative cross attention module after iterations, represents the input of the bimodal iterative crisscross attention module, Represents a bimodal iterative crisscross attention module. The output of each iterative operation is used as the input of the next iterative operation, and parameters are shared between each iterative operation. In addition, the bimodal iterative crisscross attention module converts the output sequence into a feature map at the output, and then recalibrates it to the original size of the feature map through bilinear interpolation.

9. An Alzheimer's disease image classification system based on bimodal iterative cross-attention fusion, applying the Alzheimer's disease image classification method based on bimodal iterative cross-attention fusion according to any one of claims 1 to 8, characterized in that: The system comprises: Data acquisition and preprocessing module, used to acquire structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) imaging data of Alzheimer's disease and perform preprocessing; An image slicing and data augmentation module, configured to slice the pre-processed sMRI and PET image data into blocks, obtain an equal number of sMRI and PET sliced ​​images, and randomly select a preset percentage of the sliced ​​images for data augmentation; A model construction and training module is used to construct a bimodal iterative cross-attention fusion model including a residual network module, a spatial feature shrinkage module, and a bimodal iterative cross-attention module, and use the bimodal iterative cross-attention fusion model as a base classifier and use a number of the sliced ​​images after data augmentation to train a corresponding number of the base classifiers; a feature processing module, configured to, during the training process of each base classifier, input the sliced ​​sMRI and PET images into a residual network module to extract sMRI features and PET features, use a spatial feature shrinkage module to reduce redundant information in the sMRI features and PET features, then use a bimodal iterative cross-attention module to extract complementary information to enhance the sMRI features and PET features, and then fuse the enhanced sMRI features and PET features; A base classifier selection module is used to preset a verification set and input it into the trained base classifier for classification, and select a base classifier whose classification accuracy in the verification set is greater than a preset threshold; The classification result acquisition module is used to construct a meta-classifier, use the selected base classifier to classify the sMRI and PET slice images to be classified, input the fusion features obtained in the classification process into the meta-classifier for classification, and obtain the final classification result.

Images