Alzheimer's disease classification method based on multi-view mixed attention network

Abstract

The invention provides an Alzheimer's disease classification method based on a multi-view mixed attention network. The Alzheimer's disease classification method comprises the steps that an sMRI data set is acquired and preprocessed; establishing an Alzheimer's disease classification model, wherein the Alzheimer's disease classification model comprises a feature extraction backbone network, a Transform encoder and an MMSE (Minimum Mean Square Error) fractional auxiliary classifier which are connected in sequence; inputting the preprocessed sMRI data set into an Alzheimer's disease classification model for iterative training; obtaining sMRI image data of a subject to be classified and inputting the sMRI image data into the trained Alzheimer's disease classification model for classification; according to the method, a full-image-level method is used, two deep learning network frameworks with complementary advantages are combined, strategies such as multi-view, attention mechanism and multi-scale feature fusion are used at the same time, MMSE clinical examination scores are introduced to serve as auxiliary supervision information, and the reliability of computer-aided diagnosis and classification of the Alzheimer's disease can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and structural Magnetic Resonance Imaging (sMRI)-assisted diagnosis of Alzheimer's disease, and more specifically, to an Alzheimer's disease classification method based on a multi-view hybrid attention network. Background Art

[0002] Alzheimer's disease (AD) is an irreversible neurodegenerative disease, which is particularly common in the elderly population over 65 years old. The clinical manifestations of AD include progressive degeneration of functions such as language, memory and emotion, and the neuropathological changes are manifested as gradually developing brain atrophy, which is also one of the main causes of dementia. Studies have found that there is a progressive prodromal disease before the onset of AD, namely mild cognitive impairment (MCI), which manifests as a mild decline in patients' memory, cognition and sensory functions, but the impact on patients' daily life is not obvious, so it is not easy for patients to notice. Among MCI patients, MCI can be divided into progressive type (pMCI) and stable type (sMCI) according to whether cognitive function remains relatively stable over a certain period of time. Studies have shown that pMCI patients will transform into AD patients several years after diagnosis. However, there is still no effective treatment for Alzheimer's disease (AD). Therefore, early preventive diagnosis and timely intervention measures are crucial to slow down the progression of Alzheimer's disease. Structural MRI technology shows high sensitivity in capturing image structural changes caused by brain atrophy. This technology can perform non-invasive imaging examinations under non-invasive conditions and present brain pathological changes in an intuitive manner. It has become an important auxiliary tool in many medical fields.

[0003] Machine learning (ML) and deep learning (DL) have been widely used in sMRI image analysis, especially in computer-aided diagnosis of early Alzheimer's disease (AD). Traditional AD-assisted diagnosis methods first require experts to manually design and extract useful features, and then use machine learning methods for feature learning. For example, Kloppel et al. extracted features from the gray matter tissue of the whole brain, mapped them to a high-dimensional space, and then used a support vector machine (SVM) to classify the subjects. Another method is that Silveira et al. divided the whole brain into 116 anatomical regions of interest (ROIs) and then used a boosting classifier for classification. However, traditional methods have problems such as high requirements for expert knowledge, subjective design of features, and time-consuming and labor-intensive processes, which limits the improvement of AD diagnostic performance. In contrast, deep learning (DL) uses an end-to-end approach to automatically learn low-level to high-level potential features from sMRI images, avoiding the complex process of manually designing features. In addition, deep learning methods based on sMRI images are more flexible to apply. Usually, according to the different data input methods of the model, feature extraction methods are divided into four types: slice level (2D slice-level), 3D block level (3D patch-level), ROI (Region-of-Interest) and full image level (subject-level). In recent years, convolutional neural networks (CNN) have achieved great success in AD diagnosis based on sMRI images. For example, Lim et al. used an axial 2D slice dataset of 3D sMRI images, constructed a feature extractor using Resnet-50 and VGG-16, and then combined it with a fully connected layer for AD classification. However, this method obtains limited lesion information in 2D slices and ignores structural related information between slices. On the other hand, Zhu et al. first performed a significance analysis on the 3D-patch of the AD group and the NC group, and obtained the AD diagnosis-related large discriminant brain region as the classification basis of the multi-instance deep learning model. Although this method can reduce computational complexity and inference time, it also loses spatial information between patches, and the selection of patch size remains a challenge. The CAM-CNN proposed by Zhang et al. takes the entire sMRI image as input, extends the convolution operation to 3D to better capture the spatial and image details of the sMRI, and applies a connected attention mechanism to integrate more compact high-level features. However, this method only uses 3D convolution to capture local information, making it difficult to effectively model the global dependencies between pathological regions in the 3D sMRI image space.The Vision Transformer (VIT) captures long-range dependencies by dividing the original image into blocks, mapping these blocks into vectors, and then using the self-attention mechanism in the encoder for global modeling, which enables the model to better focus on important areas. For example, Hoang et al. studied the classification performance of different versions of the VIT model on a 2D slice dataset based on sMRI. However, the pure VIT model is not as good as CNN in capturing local features and has higher requirements on the size of the dataset, which limits its convergence and accuracy.

[0004] An existing patent document discloses an Alzheimer's disease classification method based on CNN and attention mechanism, including: obtaining a brain magnetic resonance imaging (MRI) image to be classified and slice images corresponding to three sections; obtaining local fine-grained features of the brain magnetic resonance imaging (MRI) image to be classified; increasing the channel features of the input feature map through a channel attention mechanism; adding the position information of each slice; capturing the long-distance dependencies between slices through a multi-head attention mechanism; using one-dimensional convolution to enhance the local continuity of features between two adjacent slices, and globally averaging the fused feature vector and sending it to the classifier to obtain the classification result of the brain magnetic resonance imaging (MRI) image to be classified; although the prior art solution can combine the advantages of convolutional neural networks and attention mechanisms, and utilize the interaction between local lesion features and global information to improve classification accuracy, it is still difficult to comprehensively and accurately consider the relevant brain atrophy structural information contained in sMRI images and the correlation between them, thereby losing spatial and semantic information and limiting the diagnostic performance of the model. Summary of the invention

[0005] In order to overcome the defects of the above-mentioned prior art in insufficient generalization ability and limited performance of atrophic lesion areas related to AD in sMRI images, the present invention provides an Alzheimer's disease classification method based on a multi-view hybrid attention network, which uses a full-image-level research method, combines two deep learning network frameworks with complementary advantages, and uses multi-view, attention mechanism and multi-scale feature fusion strategies, and introduces MMSE clinical examination scores (Mini mental status examination) as auxiliary supervision information, which can effectively improve the reliability of computer-aided diagnosis and classification of Alzheimer's disease.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A method for classifying Alzheimer's disease based on a multi-view hybrid attention network, comprising the following steps:

[0008] S1: Acquire sMRI dataset and perform preprocessing;

[0009] The sMRI data set includes three-dimensional sMRI image data of several subjects;

[0010] S2: Establish a classification model for Alzheimer's disease;

[0011] The Alzheimer's disease classification model includes: a feature extraction backbone network, a Transformer encoder, and an MMSE score auxiliary classifier connected in sequence;

[0012] The feature extraction backbone network also includes: a feature rough extraction subnetwork, a plurality of multi-view feature fusion modules connected in sequence, and a cascade multi-scale feature fusion subnetwork; the output of each multi-view feature fusion module is connected to the input of the cascade multi-scale feature fusion subnetwork;

[0013] S3: input the preprocessed sMRI dataset into the Alzheimer's disease classification model for iterative training to obtain a trained Alzheimer's disease classification model;

[0014] S4: Obtain sMRI image data of the subject to be classified, input the sMRI image data of the subject to be classified into a trained Alzheimer's disease classification model for classification, and complete the classification of Alzheimer's disease.

[0015] Preferably, the preprocessing in step S1 includes:

[0016] The three-dimensional sMRI image data of each subject is preprocessed by performing skull stripping, spatial registration, smoothing and denoising, blank background clipping, resampling, format conversion, data enhancement and data partitioning operations.

[0017] Preferably, the feature rough extraction subnetwork in step S2 includes: a 3D convolution block, a Convnext feature extraction block 1 and a Convnext feature extraction block 2 connected in sequence;

[0018] The structure of the 3D convolution block includes: a 4×4×4 convolution layer, a batch normalization BN layer, and a ReLU activation layer connected in sequence;

[0019] The structure of each of the Convnext feature extraction blocks includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext feature extraction block also form a residual sum connection through a bypass.

[0020] Preferably, the multi-view feature fusion module in step S2 includes branches 1, 2 and 3 arranged in parallel, and branches 1, 2 and 3 correspond to the sagittal plane view, the coronal plane view and the axial plane view respectively;

[0021] The structure of the branch 1 includes: a convolution block 1 and a mixed attention block 1 connected in sequence; the structure of the branch 2 includes: a convolution block 2 and a mixed attention block 2 connected in sequence; the structure of the branch 3 includes: a convolution block 3 and a mixed attention block 3 connected in sequence;

[0022] Each of the convolution blocks has the same structure, and includes: a 4×4×4 convolution layer, a batch normalization BN layer, a ReLU activation layer, and a 1×1×1 convolution layer connected in sequence;

[0023] Each of the hybrid attention blocks has the same structure, and includes: a spatial domain attention branch and a channel domain attention branch arranged in parallel; the spatial domain attention branch includes: a 1×1×1 convolution layer, a 3×3×3 convolution layer, a 3×3×3 convolution layer, a 1×1×1 convolution layer, and a batch normalization BN layer connected in sequence; the channel domain attention branch includes: an adaptive pooling layer and a one-dimensional convolution layer connected in sequence;

[0024] The outputs of the spatial domain attention branch and the channel domain attention branch are added and normalized by a Sigmoid activation function to obtain a weighted attention score of the mixed domain, the weighted attention score of the mixed domain is weighted multiplied by the input of the mixed attention block, and the weighted multiplication result is used as the output of the mixed attention block;

[0025] The outputs of branches 1, 2 and 3 are spliced ​​and fused as the output of the multi-view feature fusion module.

[0026] Preferably, in step S2, a transition layer is provided before each of the multi-view feature fusion modules, and the transition layer includes: a 1×1×1 convolution layer, a batch normalization BN layer, a ReLU activation layer, a maximum pooling downsampling layer and a Convnext block connected in sequence;

[0027] The Convnext block includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext block also form a residual sum connection through a bypass.

[0028] Preferably, in step S2, the outputs of each multi-view feature fusion module are recorded as: F1, F2, F3, ..., F n , n is a positive integer, indicating the number of multi-view feature fusion modules;

[0029] In the cascaded multi-scale feature fusion sub-network, F n After 1×1×1 convolutional layer and upsampling layer, n-1 Perform feature splicing and fusion to obtain the intermediate fusion feature map M1;

[0030] After the intermediate fusion feature map M1 passes through a 1×1×1 convolution layer and an upsampling layer, it is combined with F n-2 Perform feature splicing and fusion to obtain the intermediate fusion feature map M2;

[0031] Repeat the above fusion steps recursively to obtain the intermediate fusion feature maps M1, M2, ..., M n-1 ;

[0032] The intermediate fusion feature map M n-1 After convolution operation and average pooling downsampling operation, it is fused with the intermediate feature map M n-2 Perform secondary fusion to obtain the intermediate fusion feature map P1;

[0033] After the convolution operation and average pooling downsampling operation, the intermediate fusion feature map P1 is combined with the intermediate fusion feature map M n-3 Perform secondary fusion to obtain the intermediate fusion feature map P2;

[0034] Repeat the above secondary fusion steps recursively to obtain the intermediate fusion feature map P of the last secondary fusion n-2 , the intermediate fusion feature map P n-2 With F n An addition operation is performed, and the addition result is used as the output of the cascaded multi-scale feature fusion sub-network.

[0035] Preferably, the Transformer encoder in step S2 comprises: a flat linear layer and a plurality of Transformer blocks connected in sequence;

[0036] Each of the Transformer blocks has the same structure, and includes: a layer-normalized LN layer 1, a multi-head self-attention layer, a layer-normalized LN layer 2, and a multi-layer perceptron layer connected in sequence; the input of the layer-normalized LN layer 1 also forms a residual sum connection with the output of the multi-head self-attention layer; the input of the layer-normalized LN layer 2 also forms a residual sum connection with the output of the multi-layer perceptron layer.

[0037] Preferably, the MMSE score auxiliary classifier in step S2 includes: an MMSE score prediction branch and an Alzheimer's disease classification branch arranged in parallel;

[0038] The MMSE score prediction branch includes a linear layer FC1 and a linear layer FC2 connected in sequence; the Alzheimer's disease classification branch includes a linear layer FC3;

[0039] The output of the Transformer encoder is subjected to matrix slicing processing to obtain a classification projection vector c after global information mapping and a prediction projection vector m that additionally introduces an MMSE score;

[0040] The predicted projection vector m is input into the MMSE score prediction branch for nonlinear mapping to obtain the predicted value of the MMSE score; the classified projection vector c is input into the Alzheimer's disease classification branch to obtain the predicted value of the class probability of Alzheimer's disease.

[0041] Preferably, the Alzheimer's disease classification model is also provided with a mixed loss function L Hybird , the mixed loss function L Hybird Including MMSE score loss function L MMSE And the classification loss function L Class , specifically:

[0042] L Hybird =L Class +αL MMSE

[0043]

[0044]

[0045]

[0046] Where N is the total number of batch samples; s i ,p i are the MMSE prediction score and category prediction probability of the i-th sample, respectively, i ,y i are the MMSE true score and true category label of the i-th sample, represents the true category label of the i-th sample after smoothing; ε represents the label smoothing parameter; α is a hyperparameter.

[0047] The present invention also provides an Alzheimer's disease classification system based on a multi-view hybrid attention network, which applies the above-mentioned Alzheimer's disease classification method based on a multi-view hybrid attention network, including:

[0048] Data acquisition and preprocessing unit: used to acquire sMRI data sets and perform preprocessing;

[0049] The sMRI data set includes three-dimensional sMRI image data of several subjects;

[0050] Model building unit: used to build Alzheimer's disease classification model;

[0051] The Alzheimer's disease classification model includes: a feature extraction backbone network, a Transformer encoder, and an MMSE score auxiliary classifier connected in sequence;

[0052] The feature extraction backbone network also includes: a feature rough extraction subnetwork, a plurality of multi-view feature fusion modules connected in sequence, and a cascade multi-scale feature fusion subnetwork; the output of each multi-view feature fusion module is connected to the input of the cascade multi-scale feature fusion subnetwork;

[0053] Model training unit: used to input the preprocessed sMRI data set into the Alzheimer's disease classification model for iterative training to obtain a trained Alzheimer's disease classification model;

[0054] Alzheimer's disease classification unit: used to obtain sMRI image data of subjects to be classified, input the sMRI image data of subjects to be classified into the trained Alzheimer's disease classification model for classification, and complete the classification of Alzheimer's disease.

[0055] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0056] The present invention provides an Alzheimer's disease classification method based on a multi-view hybrid attention network, constructs a backbone network based on three-dimensional convolution to extract sMRI local features, and combines it with a Transformer encoder to summarize the global information of the brain; at the same time, an efficient spatial and channel dimension weighted hybrid attention module is designed to fuse the spatial structural features of different view angles (coronal plane, sagittal plane, axial plane) of sMRI images; and a cascaded multi-scale feature fusion strategy is adopted to merge information at different granularity levels to further improve the representation ability and robustness of the overall model; in addition, by introducing the MMSE clinical diagnosis score as additional supervision information, and by designing a hybrid loss function, the model is guided to learn more efficiently and accurately; the present invention uses the subject's sMRI full image information and MMSE clinical score, and utilizes the complementarity between different view angle information in the subject's image and the correlation between pathological areas to model, and then performs early diagnosis of AD and MCI conversion prediction, realizing the fusion of two modal information of clinical diagnosis numerical features and image features, thereby enhancing the model's learning ability for identification features related to AD;

[0057] The Alzheimer's disease classification model in the present invention also has the following advantages:

[0058] 1) The multi-view feature fusion module in the present invention is based on a hybrid attention mechanism, including channel attention and spatial attention mechanism, to effectively capture the features of the atrophic structure under different views and fuse them into a multi-view feature map, thereby generating a richer feature representation;

[0059] 2) The present invention constructs a cascade-structured multi-scale feature fusion module on the feature extraction backbone network, which is more computationally efficient when processing three-dimensional sMRI images and can also provide a more hierarchical multi-scale feature representation; specifically, the module fuses feature maps of adjacent scales layer by layer in a descending order according to the fine-grained level of the feature map to achieve multi-scale information fusion of sMRI images, thereby further improving the feature representation capability of the model;

[0060] 3) The present invention designs an auxiliary classifier at the top level of the model, introduces the MMSE score as auxiliary supervisory information, and uses it together with the category label information of sMRI for supervised learning of the model; based on the characteristics of these two types of supervisory information, a hybrid loss function is also designed to control the weight ratio of auxiliary information in training by introducing hyperparameters, optimize the gradient distribution of model learning, and help improve the accuracy and reliability of AD diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A flow chart of an Alzheimer's disease classification method based on a multi-view hybrid attention network provided in Example 1.

[0062] Figure 2 This is a partial pretreatment operation flow chart provided in Example 2.

[0063] Figure 3 This is a structural diagram of the Alzheimer's disease classification model provided in Example 2.

[0064] Figure 4 This is a structural diagram of the feature rough extraction subnetwork provided in Example 2.

[0065] Figure 5 This is a structural diagram of the Convnext feature extraction block provided in Example 2.

[0066] Figure 6 This is a structural diagram of the multi-view feature fusion module provided in Example 2.

[0067] Figure 7 This is a structural diagram of the hybrid attention block provided in Example 2.

[0068] Figure 8 This is a structural diagram of the transition layer provided in Example 2.

[0069] Fig. 9 This is a structural diagram of the cascaded multi-scale feature fusion sub-network provided in Example 2.

[0070] Fig.10 This is a structural diagram of the Transformer encoder provided in Example 2.

[0071] Fig.11 This is a structural diagram of the MMSE score auxiliary classifier provided in Example 2. DETAILED DESCRIPTION

[0072] The drawings are for illustrative purposes only and are not to be construed as limiting the present patent;

[0073] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0074] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0075] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0076] Example 1

[0077] like Figure 1 As shown, this embodiment provides an Alzheimer's disease classification method based on a multi-view hybrid attention network, comprising the following steps:

[0078] S1: Acquire sMRI dataset and perform preprocessing;

[0079] The sMRI data set includes three-dimensional sMRI image data of several subjects;

[0080] S2: Establish a classification model for Alzheimer's disease;

[0081] The Alzheimer's disease classification model includes: a feature extraction backbone network, a Transformer encoder, and an MMSE score auxiliary classifier connected in sequence;

[0082] The feature extraction backbone network also includes: a feature rough extraction subnetwork, a plurality of multi-view feature fusion modules connected in sequence, and a cascade multi-scale feature fusion subnetwork; the output of each multi-view feature fusion module is connected to the input of the cascade multi-scale feature fusion subnetwork;

[0083] S3: input the preprocessed sMRI dataset into the Alzheimer's disease classification model for iterative training to obtain a trained Alzheimer's disease classification model;

[0084] S4: Obtain sMRI image data of the subject to be classified, input the sMRI image data of the subject to be classified into a trained Alzheimer's disease classification model for classification, and complete the classification of Alzheimer's disease.

[0085] In the specific implementation process, the original sMRI image dataset is first obtained and preprocessed, including skull stripping, spatial registration, smoothing denoising, blank background removal and resampling. Then, data enhancement and data amplification steps are performed. For example, before the data is loaded into the model, it is randomly rotated, flipped, occluded, 3D center cropped and contrast enhanced in sequence with a given probability to improve the diversity of the dataset and enhance the generalization ability of the model.

[0086] The Alzheimer's disease classification model was established. The model structure mainly consists of three parts: the first part is the three-dimensional convolution feature extraction backbone network (Backbone), which is responsible for aggregating the feature information from shallow to deep layers in the semantic and spatial dimensions of each local structure in sMRI; the second part is the stacked Transformer encoder network. The high-dimensional local features extracted by the backbone network are serialized and then enter the stacked encoder for context modeling to capture global features; the third part is the MMSE score auxiliary classifier network (MMSE auxiliary classifier), which is composed of a fully connected layer responsible for the prediction of the classification result and a multi-layer perceptron responsible for the prediction of the MMSE score, and by combining the MMSE prediction loss function and the classification loss function to construct a hybrid loss function, the MMSE score of each patient is introduced into the model as auxiliary supervision information to improve the classification performance; the following is a detailed description of each part of the model;

[0087] First is the feature extraction backbone network of the first part:

[0088] According to relevant studies, the brain tissue of patients with early Alzheimer's disease (AD) will experience progressive structural atrophy, involving damage to the cortex and gray matter tissue in different local brain regions such as the occipital lobe, parietal lobe, temporal lobe and frontal lobe. Since these structures are distributed in different areas of the brain, specific imaging modalities and models need to be considered to provide support in computer-aided diagnosis research. Among many medical scanning imaging technologies, structural magnetic resonance imaging (sMRI) can accurately reflect the subtle changes in the brain structure of AD patients due to its non-invasive and high-resolution characteristics. Therefore, the model of this method chooses to use sMRI as the main training data. In addition, in order to enable the model to better learn the spatial information of abnormal brain regions, this method adopts a full-image-level data input method, directly uses the preprocessed images as the input of the model, and uses three-dimensional convolution to extract features from the images.

[0089] This part is the feature extraction backbone network based on 3DCNN in the overall model. Its main task is to capture various local structural changes related to AD distributed in sMRI, and summarize this information into high-dimensional image feature maps for use in the next part of the model. The main structure of this part includes: a network backbone (Stem) responsible for coarse feature extraction and parameter simplification, a transition layer (Transition layer) responsible for feature resolution downsampling and dimension increase, a stacked multi-view feature fusion module (Multi-View Feature Fusion, MVFF) responsible for aggregating multi-view (sagittal, coronal, axial) features, a cascading multi-scale feature aggregation module (Cascading multi-scale fusion, CMSF) responsible for model multi-scale feature aggregation, and a convolution block (2x2x2 Conv) responsible for high-dimensional feature map scale adjustment and dimensional mapping. In general, this method introduces a multi-view feature fusion module and a cascaded multi-scale feature fusion strategy on the basis of 3DCNN to improve the efficiency and richness of feature extraction of the backbone network, thereby further enhancing the model's ability to identify local atrophy areas in AD patients.

[0090] In the three-dimensional convolutional feature extraction backbone network of the present method, when the scale of the sMRI data set is not large, the convergence and performance of the diagnostic modeling using the Transformer encoder alone will be greatly constrained; therefore, the inductive bias of the convolutional backbone network is introduced in the pre-model stage to compensate for the modeling difficulties of the Transformer and improve the generalization ability; in order to improve the local feature induction efficiency of the convolutional backbone network, the present embodiment constructs it into a four-layer feature pyramid hierarchical structure, and uses the Convnext block as a feature aggregation unit, thereby strengthening the local feature representation ability of the backbone network;

[0091] Design of multi-view feature fusion module in the 3D convolutional feature extraction backbone network: Different viewpoints of 3D sMRI feature maps usually have complementary and relevant information. Therefore, this method designs a hybrid attention module with channel and spatial weighting in the backbone network to extract and fuse feature maps from different viewpoints, so that the model can capture more comprehensive and richer high-dimensional feature representations, and improve model performance and robustness.

[0092] Design of cascaded multi-scale feature fusion sub-network in the 3D convolutional feature extraction backbone network: Shallow feature maps in deep learning models can describe more detailed brain spatial information, but they are not good at generalizing the semantic information of brain structure, while deep feature maps have the opposite characteristics. Specifically, in order to improve the reuse efficiency of information, this method fuses different scale features in pairs in a cascade manner according to the order of feature levels from deep to shallow, maximizes the use of useful information in sMRI, and improves diagnostic performance.

[0093] This is followed by the Transformer encoder in the second part:

[0094] In the study of the pathological progression pattern of Alzheimer's disease, by analyzing the changes of neurodegenerative atrophy in longitudinal sMRI, it was found that in the process of AD patients from early stage to late stage, the involvement patterns of neurodegenerative atrophy in different brain structures showed diversity, and had a temporal sequence and spatial diffusion; for example, under normal circumstances, the left hemisphere of the brain usually atrophies and degenerates earlier than the right hemisphere; the atrophy and degeneration of the nucleus accumbens will further extend to the amygdala and other areas (Amygdala); therefore, according to these research reports, there is a correlation between different local brain tissue structures that undergo atrophy changes in the brain;

[0095] In order to capture this local correlation and improve the diagnostic performance of AD, inspired by the success of the VIT model in computer vision (CV) classification applications, this part uses stacked Transformer encoders to construct the network, and confirms the optimal number of stacked Transformer encoders and the dimension of the local embedding vector through ablation studies; Transformer encoders are good at analyzing sequence relationships, so before the data stream enters this part, the high-dimensional feature map obtained in the first part is converted into a local representation vector matrix through a dimensional transformation and flattening operation, and each local representation vector corresponds to an element of the target sequence. Through the self-attention mechanism of the Transformer encoder, the model can weight the local representation vectors of the input sMRI one by one and learn the dependencies between each other. After repeated processes of multiple encoders, the correlation representation between sequences is mapped to the classification vector and MMSE score prediction vector for the final category prediction, realizing contextual modeling of all atrophy structural features of the whole brain;

[0096] Construction of Transformer encoder: In the local information feature map output by the feature extraction backbone network, each voxel in the original sMRI image corresponds to a large local receptive field, and studies have shown that there is certain correlation information between different parts of the brain; therefore, it is input into the encoder composed of a stack of Transformer blocks to learn the long-range dependencies between different voxels in the feature map and capture the internal correlation between features to generate a global feature representation of the brain sMRI, and the global feature representation vector is used as the input of the model classifier;

[0097] Finally, the MMSE score auxiliary classifier in the third part:

[0098] Mini-Mental State Examination (MMSE) score is an indicator used to measure the cognitive function of patients with Alzheimer's disease (AD). Similar to other clinical assessment scores, it has a certain correlation with the changes in the patient's brain structure. In early AD diagnosis research based on deep learning, MMSE scores can be used as a feature of numerical modality as a basis for disease classification, and can also be used as additional supervisory information to be integrated into the decision-making process of the model in a supervised learning manner;

[0099] This method constructs a classifier in which the MMSE score is introduced as additional auxiliary information; this classifier uses a parallel multi-layer perceptron (MLP) to take the category label and the true MMSE score as supervision information respectively; this method also uses two loss functions and combines them into a hybrid loss function to achieve gradient back propagation; in this way, the fusion of two modal information of clinical diagnostic numerical features and imaging features is achieved, thereby enhancing the model's ability to learn the identification features related to AD;

[0100] Design of MMSE score auxiliary classifier and hybrid loss function: In order to improve the model performance, a dual-branch auxiliary classifier network is designed. One branch performs category prediction normally, while the other branch is used to predict the MMSE score value. The real MMSE score is used as additional auxiliary information to guide model learning. The designed hybrid loss function is then used to integrate the two types of supervision information. By minimizing the hybrid loss function, the learnable parameters of the overall model are jointly optimized.

[0101] After the model training is completed, the sMRI image data of the subject to be classified is obtained, and the sMRI image data of the subject to be classified is input into the trained Alzheimer's disease classification model for classification, thereby completing the classification of Alzheimer's disease;

[0102] The model established by this method can fuse multi-perspective information, perform multi-scale information fusion on the output multi-perspective feature map, and then perform long-range dependency modeling to obtain higher-dimensional global features, so as to enable the model to characterize the correlation characteristics between different pathological areas of AD patients; in addition, the MMSE clinical examination score (Mini mental status examination) is introduced as auxiliary supervision information, which can effectively improve the reliability of computer-aided diagnosis and classification of Alzheimer's disease.

[0103] Example 2

[0104] This embodiment provides an Alzheimer's disease classification method based on a multi-view hybrid attention network, comprising the following steps:

[0105] S1: Acquire sMRI dataset and perform preprocessing;

[0106] The sMRI data set includes three-dimensional sMRI image data of several subjects;

[0107] S2: Establish a classification model for Alzheimer's disease;

[0108] The Alzheimer's disease classification model includes: a feature extraction backbone network, a Transformer encoder, and an MMSE score auxiliary classifier connected in sequence;

[0109] The feature extraction backbone network also includes: a feature rough extraction subnetwork, a plurality of multi-view feature fusion modules connected in sequence, and a cascade multi-scale feature fusion subnetwork; the output of each multi-view feature fusion module is connected to the input of the cascade multi-scale feature fusion subnetwork;

[0110] S3: input the preprocessed sMRI dataset into the Alzheimer's disease classification model for iterative training to obtain a trained Alzheimer's disease classification model;

[0111] S4: Obtain sMRI image data of the subject to be classified, input the sMRI image data of the subject to be classified into a trained Alzheimer's disease classification model for classification, and complete the classification of Alzheimer's disease.

[0112] In the specific implementation process, first obtain the data set and preprocess it:

[0113] The datasets used in this method are all from the Alzheimer's Disease Neuroimaging Initiative (ADNI). The downloaded data involve 4796 T1-weighted sMRI images of 1570 subjects in three different research phases of ADNI (ADNI-1 / GO, ADNI-2, and ADNI3). According to the clinical diagnostic criteria of AD, such as the Mini-Mental State Examination (MMSE) score and the Clinical Dementia Rating (CDR), ADNI divided these subjects into three groups: AD group (patients with Alzheimer's disease), NC group (healthy control group), and MCI group (patients with mild cognitive impairment). In the first screening, the subjects diagnosed with MCI were divided into pMCI group (developmental mild cognitive impairment) and sMCI group (stable mild cognitive impairment) according to whether they were converted to AD during the follow-up within the next 36 months. The specific demographic information is shown in Table 1:

[0114] Table 1 Demographic information

[0115]

[0116] The original structural MRIs downloaded from ADNI are usually not directly usable for research and analysis. First, these images are acquired by scanners from different manufacturers (such as Siemens, Philips, etc.), and their image quality and specifications vary. Second, the size, shape, and position of each subject’s head vary greatly, resulting in each subject’s image not being in the same spatial coordinate system. Finally, the non-brain tissue and background contained in the original sMRI will cause unnecessary waste of computing resources; these interference factors outside the brain structure will affect the feature learning and efficiency of the model. Therefore, it is necessary to preprocess the original sMRI. The main preprocessing steps are as follows: Figure 2 As shown, the first three preprocessing steps are completed in the batch processing flow of cat12 and spm toolbox on the matlab platform. The steps are as follows:

[0117] 1) Skull stripping: Use the "APRG (adaptive probability region-growing)" algorithm to perform separation of brain tissue and skull structures;

[0118] 2) Affine registration: MNI152 was selected as the standard brain space template to achieve spatial normalization. After registration, the image with a voxel spacing of 1.5mm×1.5mm×1.5mm and a size of 121px×145px×121px was obtained;

[0119] 3) Image smoothing: A 4×4×4 Gaussian kernel function was used to perform spatial filtering on sMRI to remove high-frequency noise in the image and improve the signal-to-noise ratio;

[0120] 4) Background space removal: by calculating the minimum distance of the current sMRI from the image edge in the X, Y, and Z directions, a mask of size 101px×125px×103px is generated to remove irrelevant background information;

[0121] 5) Resampling: To facilitate model analysis, each sMRI image size was resampled to 112px×112px×112px by trilinear interpolation;

[0122] 6) Format conversion: convert sMRI data in NIfTI format into NumPy format to facilitate model loading and processing;

[0123] After that, data enhancement is performed. The Alzheimer's disease diagnosis method used in this method is based on the full-image level of sMRI. A single image corresponds to a single sample. Therefore, in order to make the model fully converge, reduce overfitting, and improve robustness, data enhancement steps are used to increase sample diversity during model training. The following are the specific data enhancement steps used:

[0124] 1) The image was randomly rotated by ±15 degrees along the coronal, sagittal, and axial planes, with a probability of 0.5;

[0125] 2) The image is randomly flipped vertically with a probability of 0.5;

[0126] 3) Randomly select a specified number and size of local areas on the image for masking, with a size of 10px×10px×10px, a number of 3, and a probability of 0.5;

[0127] 4) The image is randomly cropped at the 3D center with a crop size of 80px×80px×80px, and then resampled to the original size by bilinear interpolation with a probability of 0.5;

[0128] 5) Random contrast enhancement of images, using a random number that satisfies uniform distribution as the contrast scaling factor, with a size in the range of (0.8, 1.3), and limiting the voxel intensity in the range of (-1, 1);

[0129] After the data set preprocessing was completed, the data set was randomly divided into three subsets: training set, validation set, and test set in a ratio of 4:1:1. Since each subject contains scan images at one or more time points, in order to avoid data leakage, the division process ensured that all time point scans of each subject only existed in the same subset, and a five-fold cross-validation evaluation method was used.

[0130] Establishing Alzheimer's disease classification model: The model overview of this method is as follows Figure 3 As shown, the following will introduce the specific implementation of the feature extraction backbone network (Backbone), Transformer encoder and MMSE score auxiliary classifier (MMSE auxiliaryclassifier) ​​in turn; the feature extraction backbone network contains three submodules: feature rough extraction subnetwork (Stem), multi-view feature fusion module (MVFF), transition layer (Transition layer) and cascade multi-scale feature fusion subnetwork (CMSF subnetwork); in addition, since the Convnext block combines effective model optimization ideas such as inverse bottleneck structure and deep separable convolution in CNN, it shows strong feature aggregation ability, so in this method, it is applied to the feature extraction backbone network of the model and used as a feature extraction subunit; the implementation of each part is introduced in turn below;

[0131] Part 1, feature extraction backbone network:

[0132] 1) Feature rough extraction sub-network (stem):

[0133] like Figure 4As shown, after the sMRI data with a size of 1×112×112×112 (C×H×W×D) is input into the model, shallow feature aggregation is completed in the stem through two steps;

[0134] Step 1: First, a shallow feature coarse extraction is performed through a 3D convolution layer with a convolution kernel size of 4×4×4, a step size of 2, and an output channel of 32, a BatchNorm (BN) layer, and a ReLU activation function to obtain the initial level feature map of the model;

[0135] Step 2: The output feature map of the previous step is passed through two stacked Convnext blocks for shallow feature aggregation; the Convnext block is as follows: Figure 5 As shown in the figure, the main path of the Convnext block includes: a 3D convolution layer with a convolution kernel size of 7×7×7 and an unchanged output channel, a LayerNorm (LN) layer; a 1×1×1 convolution layer with 4C output channels (where C is the input channel of the block structure) and a GeRU activation function; a 1×1×1 convolution layer with C output channels; in addition, the input and output are connected by a bypass residual connection with element-by-element addition; the Convnext block feature aggregation process is expressed by the following formula:

[0136] Conv next (x) = Conv s (Conv e (LN(Conv g (x))))

[0137] Among them, x is the input feature map, Conv g 、Conv e and Conv s The corresponding convolutional layers respectively;

[0138] 2) Multi-view feature fusion module (MVFF):

[0139] Different perspectives of 3D sMRI feature maps usually have complementary and relevant information. Fusion of feature maps from different perspectives can enable the model to capture more comprehensive and richer high-dimensional feature representations and enhance model robustness. To this end, this method designs an MVFF module to fuse the multi-perspective features of the input feature map to jointly locate the lesion area with multiple perspectives. The MVFF structure is as follows: Figure 6 As shown;

[0140] Step 3: Assume that the input sagittal plane view feature map of this level is After the dimension transformation, the remaining two view feature maps are obtained, that is, Get the coronal view feature map f (C) , same reason Get the axial view feature map f(A) , as follows:

[0141] f (C) =T C (f (S) ),f (A) =T A (f (S) )

[0142] Where, T A and T C They are respectively expressed as the dimension transformation process of the coronal plane view and the axial plane view;

[0143] Step 4: Design three parallel branches to aggregate the feature maps of each viewing angle, where each branch consists of two parts:

[0144] Part 1 is the convolutional stacking block responsible for fine-grained feature extraction of each view (corresponding to AV Conv, SV Conv and CVConv). They are all composed of 3×3×3 and 1×1×1 convolutional layers connected in sequence, where the 3×3×3 convolution is followed by a BN layer and ReLU nonlinear activation;

[0145] Part 2 is a simple and effective attention module that generates feature representations with spatial and channel attention perception to focus feature maps from different perspectives on the key lesion area. The attention module consists of a spatial domain attention branch and a channel domain attention branch. The specific structure is as follows: Figure 7 As shown in Figure 2, the design of the spatial domain attention branch is inspired by the inverse bottleneck structure of MobileNet V2 and adopts a similar structure. The attention map output by the spatial domain attention branch of any view feature map is defined as

[0146] SA (*) =BN(Conv g (Conv e2 (Conv e1 (Conv a (f (*) )))))

[0147] Where (*)∈[S,A,C] is used to identify different perspectives, Conv a Indicates that the 1×1×1 convolutional layer is responsible for feature integration and dimensionality reduction mapping. Here, a channel scaling ratio r is introduced to make Conv a The output channel is reduced to C′ (C′=C / r), and in this embodiment r is set to 4; Conv e1 and Conv e2 Represents the 3×3×3 convolutional layer responsible for extracting spatial context information, Conv gRepresents a 1×1×1 convolutional layer, which is responsible for spatial encoding to obtain the attention score, and adds a BN layer for batch normalization; a lightweight channel attention module ECA is used to generate the attention map on the channel domain attention branch, which only considers each channel and a certain number of adjacent channels for attention encoding, and has efficient cross-channel interaction capabilities. Any view feature map is output by this branch as a channel attention map

[0148] CA (*) =Conv 1D (Avgpool(f (*) ))

[0149] Where Avgpool represents the average pooling along the channel axis, which perceives the importance of the feature map channel; Conv 1D It represents the adaptive one-dimensional convolution for channel interaction. Its convolution kernel k changes adaptively with the input channel, thus focusing on more useful channel information, which can be described as the formula:

[0150]

[0151] Where c represents the input channel; β and γ represent adjustable parameters, which are set to 2 and 1 respectively in this embodiment; |t| odd It means that the result takes the odd number closest to t as the convolution kernel size;

[0152] After obtaining the attention scores of the spatial domain and channel domain, they are integrated by adding them element by element, and then normalized by the Sigmoid activation function to obtain the attention weighted score of the mixed domain. The weighted score is directly multiplied with the input view feature map to obtain a feature map with spatial and channel perception.

[0153] f HA(*) =f (*) *(CA (*) +SA (*) )

[0154] Step 5: After the feature extraction of the three perspectives is completed, the complementary perspective information is spliced ​​in the channel domain to obtain a more powerful feature representation; it is worth noting that in order to align the feature space of the complementary perspective feature map, the coronal perspective feature map f is calibrated before splicing. (C) And the axial view feature map f (A) Perform inverse dimensional transformation; finally, the multi-view fusion feature output of the MVFF module It can be expressed as:

[0155]

[0156] In the formula, Conv is a 1×1×1 convolutional layer responsible for channel downsampling, with 3C input channels and C output channels; Concat represents the channel concatenation operation; and They are the dimensionality recovery inverse transformation processes corresponding to the two view feature maps;

[0157] 3) Transition layer:

[0158] In the feature extraction backbone network of this embodiment, three MVFF modules are stacked to fuse the different perspective information of the feature map at this level. However, the increase in network depth will lead to a sharp increase in the number of network parameters, which may cause overfitting problems when training with a limited amount of training data. To address this, before the feature map enters the MVFF module, adding a transition layer for downsampling can achieve parameter simplification. The transition layer is constructed as follows: Figure 8 As shown, assuming that the feature map of the previous level is First, it is raised to 2C after passing through the convolution layer, then downsampled twice by the maximum pooling, and finally a Convnext block is used to aggregate the features to obtain the feature map of this level. Defined as the following formula:

[0159] f i =Conv next (Maxpool(Conv e (f i-1 )))

[0160] In the formula, Maxpool represents the maximum pooling downsampling (MAP), Conv e represents a 1×1×1 convolutional layer;

[0161] 4) Cascaded multi-scale feature fusion subnetwork (CMSF subnetwork):

[0162] In many classification tasks based on convolutional neural networks (CNNs), the strategy of fusing feature maps of information at different scales has been proven to effectively improve model performance. After extracting multiple view features from sMRI images through MVFF, the output feature maps of each MVFF module are saved in turn. Due to the downsampling of the transition layer, a set of feature maps of different scales can be obtained. i},in:

[0163]

[0164] Where s0 and c0 are the feature map size and number of channels of the first MVFF module output feature map F1, and it is known that s0 = 28, c0 = 64; {F i} contains feature maps with scales from large to small and feature levels from shallow to deep. It is worth noting that shallow feature maps can describe more detailed brain spatial information, but the generalization of semantic information is insufficient, while deep feature maps are the opposite. Therefore, the fusion {F i All scale features in} are conducive to improving the model representation ability;

[0165] The cascaded multi-scale fusion sub-network (CMSF) structure used to fuse these different scale feature maps is as follows: Fig. 9 As shown in Figure 1, in order to balance the representation ability of multi-scale fusion features and the amount of parameters introduced, feature maps of different scales are sequentially upsampled and combined, and then fused with the feature map of the next scale in a cascade information flow manner. The final multi-scale fusion feature is represented as

[0166] F MS =F3+P1

[0167] P1=Concat(M1,M2)

[0168] M1=y(Concat(F2,x(F3)))

[0169] M2=y(Concat(F1,x(Concat(F2,x(F3)))))

[0170] Where x represents the process of trilinear interpolation upsampling and 1×1×1 convolution channel alignment in sequence; y represents the process of convolution operation and average pooling downsampling in sequence. It is worth noting that due to the different granularity of brain structure features in F1 and F2, convolution layers with kernel sizes of 2×2×2 and 1×1×1 are selected for local feature induction before downsampling in y; the detailed steps are explained as follows:

[0171] Step 6: First, the number of channels of F3 is reduced to the same as that of F2 through a 1×1×1 convolutional layer, and then an interpolation operation is performed to align the spatial size of the features, and finally feature concatenation is performed. The operation process is as follows: Implementation; Finally, the intermediate output feature map M1 is obtained after the feature maps F3 and F2 are fused;

[0172] Step 7: Similar to step 6, the intermediate feature maps M1 and F1 are fused to obtain the intermediate output feature map M2;

[0173] Step 8: In order to make full use of the feature information at different scales, the intermediate feature maps M1 and M2 are aligned in feature space and channel dimension by δ, and then channel concatenation is performed to obtain P1. Finally, P1 is added element by element to the deep feature map F3 to obtain a fused feature representation F containing information at multiple scales. MS ;

[0174] Part 2, Transformer encoder:

[0175] After the feature extraction backbone network completes the local feature extraction of sMRI, an output feature with a resolution of 256×7×7×7 will be obtained. Figure X , where each voxel corresponds to a large local receptive field in the original sMRI image; X is input into an encoder consisting of a stack of Transformer blocks to learn the long-range dependencies between different voxels in the feature map and capture the internal correlations between features to produce a global feature representation of the brain sMRI; Fig.10 As shown in the figure, the Transformer encoder consists of a cascade of consecutive Transformer blocks. Each Transformer block includes a multi-head self-attention layer (MSA) and a multi-layer perceptron (MLP), and the modules all apply layer normalization (LN) and skip connections. Here, the number of Transformer blocks and heads is 5 and 12 respectively.

[0176] Step 9: Before entering the Transformer encoder, the channel dimension is increased to the Transformer embedding dimension 576 by a 2×2×2 convolution layer. After feature map flattening and dimension transformation, in addition to the classification projection vector c, the MMSE score prediction projection vector m is additionally introduced, where Finally, after position encoding, we get the sequence matrix consisting of token embedding vectors (Token) Where N = 7 × 7 × 7 + 2, each token embedding vector can reflect the local features of the input sMRI image;

[0177] Step 10: Input Z to the Transformer encoder, and the token vectors learn the long-range dependencies between each other through layer-by-layer self-attention weighting; the output Z of the lth Transformer block is l It is expressed as:

[0178] Z l =MLP(LN(Z′ l ))+Z′ l

[0179] Z′ l =MSA(LN(Z l-1 ))+Z l-1

[0180] Among them, each head in MSA performs self-attention weighted calculation (Self-Attention, SA) separately to learn richer representation capabilities in different representation spaces. The calculation of SA is expressed as:

[0181]

[0182] Where Q, K, and V are the query matrix, key matrix, and value matrix respectively; d is the vector dimension of the key matrix; here Q, K, and V are all input from the lth Transformer block token sequence matrix Z l-1 After linear transformation, it is expressed as:

[0183] Q l =Z l-1 W Q ,K l =Z l-1 W K ,V l =Z l-1 W V

[0184] Where the linear transformation is implemented by the linear layer, W Q , W K , W V Corresponding to its trainable parameter matrix;

[0185] For MSA, by slicing Q, K, V into h groups (called h heads), and performing self-attention weighting in the h heads respectively, different subspace representation capabilities can be obtained; the output of the i-th head of the l-th Transformer block can be expressed as:

[0186] head l i =Attention(Q l i ,K l i ,V l i )

[0187] Here i∈[1,...,12], after getting the output of each head, they are merged and passed through the linear layer W o Perform projection mapping to obtain the output Multihead of MSA l :

[0188] Multihead l =Concat(head l 1 ,...,head l 12 )W o

[0189] Step 11: The Token sequence matrix after global encoding is obtained through the Transformer encoder output

[0190] Part III, MMSE auxiliary classifier:

[0191] In order to obtain supervisory information sources other than sMRI images, the clinical diagnosis MMSE score is introduced as additional auxiliary information to guide model learning and improve model performance. Specifically, this embodiment designs an auxiliary classifier network with a dual-branch structure, in which one branch performs model classification and the other branch is used to predict the MMSE score value, thereby obtaining additional supervisory information. The auxiliary classifier is as follows: Fig.11 As shown;

[0192] Step 12: First, the output sequence vector matrix of the Transformer encoder (where N = 7 × 7 × 7 + 2) to perform matrix slicing to obtain the classification projection vector c after global information mapping and the additionally introduced MMSE score prediction projection vector

[0193] Step 13: Input the vector m into the MLP branch composed of the cascade of linear layers FC1 and FC2 for nonlinear mapping to obtain the predicted value of MMSE, while the vector c is directly inferred by the linear layer FC3 to obtain the category score;

[0194] In step 14, a hybrid loss function is designed to integrate the two types of supervision information. By minimizing the hybrid loss function, the joint optimization of the learnable parameters of the overall model is achieved. Specifically, the mean square error (MSE) is used as the loss function to supervise the prediction of the MMSE score, and the two-category label smoothing cross-entropy (LSCE) is used as the classification loss function to enhance the robustness of the model. Then, the two loss functions are added together to obtain the hybrid loss function L Hybird :

[0195] L Hybird =L Class +αL MMSE

[0196]

[0197]

[0198]

[0199] Where N is the total number of batch samples; s i ,p i are the MMSE prediction score and category prediction probability of the i-th sample, respectively,i ,y i Corresponding to the MMSE true score and true category label of the i-th sample, represents the true category label of the i-th sample after smoothing; ε represents the label smoothing parameter; the hyperparameter α is used as the multiplier coefficient of the mean square loss function to adjust the weight of the auxiliary supervision information; in this embodiment, ε and α are set to 0.1 and 0.5 respectively;

[0200] Afterwards, the model was implemented based on the Pytorch 1.8.1 framework, and the experiments were all performed on a GPU configured with NVIDIA RTX. TM The model was tested on the OASIS dataset to further evaluate the generalization performance of the proposed model on heterogeneous data. In general, the validation experiment of this embodiment studied two different Alzheimer's disease diagnosis tasks, namely, AD classification task (AD vs. NC) and MCI conversion prediction task (sMCI vs. pMCI).

[0201] The training strategy includes setting the number of iterations to 100, the batch size to 8, selecting the Adam optimizer with a weight decay of 1e-4, setting the initial learning rate lr to 1e-4 at the beginning of training, using a learning rate decay strategy with a fixed step size of 15 and a decay coefficient of 0.9, and limiting the minimum learning rate to 1e-5; for model parameters, on the feature extraction backbone network of the model, set the channel dimension change to [32, 64, 128, 256], set the number of stacked Transformer blocks on the Transformer global encoder to 5, and set the dimension of the embedding vector to 576;

[0202] In the model classification performance evaluation, the evaluation indicators used include accuracy (ACC), sensitivity (SEN), specificity (SPE) and area under the receiver operating characteristic curve (AUC). These indicators are defined as:

[0203]

[0204]

[0205]

[0206] In the formula, TP, TN, FP and FN represent true positive value, true negative value, false positive value and false negative value respectively;

[0207] In order to demonstrate the significant performance of the model, this embodiment compares the performance of the proposed method with eleven recently proposed methods. The compared methods are divided into 3DCNN, Transformer and hybrid methods combining CNN and visual transformer according to the basic model used, and divided into ROI, 2D slice, 3D image block and 3D full image according to the form of data features. The selected comparison methods are briefly described below:

[21] and

[22] The whole brain image is pre-segmented into multiple anatomical regions of interest according to the brain atlas, and then ROI features such as local image blocks of the hippocampus, cortical volume and thickness are extracted, and then 3DCNN or SVM is used as a classifier to obtain the classification results;

[23] LSTM is used to locate the local area of ​​significant atrophy in the sMRI two-dimensional slice, and the local features of each local area are learned through a convolutional subnetwork, and then the global slice features are integrated for AD classification;

[24] The slice sequence of each subject is input into multiple convolutional subnetworks to capture relevant information between slices;

[25]

[26] used statistical methods such as the Student t-test to obtain the discriminative locations of AD and NC brains, and then obtained fixed-size 3D image blocks based on these locations, and applied the multiple instance learning (MIL) method to conduct AD diagnosis research;

[27] learned the channel attention scores of multi-view 2D slices by aggregating the channel attention scores of each 2DCNN branch, and then performed channel weighting on the output feature map of the 3DCNN branch to improve accuracy;

[28] designed a 3DCNN-based global subnetwork and a multi-instance learning subnetwork parallel backbone, and then improved the generalization ability through a multi-level feature adversarial learning module;

[29] Based on the DAT model, adaptive positioning of the atrophic brain area was completed more accurately; The work of

[30] and

[31] is similar to this method, and the local feature extraction and global feature modeling from sMRI are completed by using 3DCNN and Transformer encoder in combination, thereby achieving more powerful feature representation;

[0208] To ensure the fairness of the comparison, only those methods that used the ADNI dataset for model training and performance testing were compared; the compared classification tasks included AD diagnosis and MCI conversion prediction, where indicators such as ACC, SEN, SPE and AUC were the average values ​​of five-fold cross validation; as shown in Table 2, this method achieved the highest AUC, ACC and SPE in the AD diagnosis task, and the highest AUC in the MCI conversion prediction task; first, compared with ROI-based and 2D slice-based methods, this method takes the entire image as input and uses 3DCNN to capture richer spatial and semantic information, so that more feature information can be involved in model decision-making; secondly, compared with the model based on 3D image blocks , since the method of multi-instance learning for each local image block will ignore the correlation between local areas, this method combines 3DCNN with Transformer encoder to model the dependency relationship of the lesion area in the brain space; finally, compared with the method based on the whole image, this method still achieves competitive results, thanks to the introduction of multi-view feature fusion module and multi-scale feature fusion module on the 3DCNN backbone, which enables the model to capture complementary information from multiple feature map perspectives and multi-fine-scale feature representation, and can more effectively identify the lesion area; in addition, the introduction of MMSE clinical score enables the model to obtain supervisory information outside the sMRI image, which can provide effective guidance for model learning;

[0209] Table 2 Performance comparison of the proposed model with other models on the ADNI data test set

[0210]

[0211] In order to verify the effectiveness of the model, this embodiment also conducts a series of ablation experiments on the entire model; as shown in Table 3, for a more concise representation, the multi-view feature fusion module, cascade multi-scale feature fusion module, Transformer encoder and MMSE score auxiliary classifier components in the model are respectively recorded as "MVFF", "CMSF", "VITE" and "MAC", and the feature extraction backbone network after removing "MVFF" and "CMSF" is recorded as "CN"; this embodiment conducts ablation experiments in two aspects:

[0212] 1) Taking "CN+VITE" as the baseline model of the ablation process, the core components "MVFF", "CMSF" and "MAC" are added one by one to evaluate their respective contributions to the model diagnosis; as shown in Table 3, compared with the baseline "CN+VITE", the combined models "MVFF+CN+VITE" and "CMSF+CN+VITE" have significant performance improvements. It is worth noting that the addition of the multi-view feature fusion module has achieved an improvement of 6.72% and 8.5% in the AUC and ACC indicators of the AD classification task, respectively, and an improvement of 10.43% and 7.62% in the MCI conversion prediction task, which fully demonstrates that the multi-view information of the fused feature map can greatly improve the performance of the model. Greatly improved model performance; "MVFF+CMSF+CN+VITE" added the CMSF module on the basis of MVFF, and the AUC and ACC were improved by 0.92%, 1.65%, 1.15% and 2.34% in the two tasks, respectively, which shows the effectiveness of the CMSF module in fusing different levels of feature information in sMRI images; In addition, the MMSE clinical diagnosis score as additional supervision information to provide guidance for model learning was verified in the results of the model "MVFF+CMSF+CN+VITE+MAC" (Proposal). By introducing the MMSE score prediction loss, the model learning can more easily approach the optimal point, and the optimal performance of the final model was obtained;

[0213] 2) In order to illustrate that the model constructed by using 3DCNN and Transformer encoder is better than the model constructed by only one of them, firstly, the standard model (3DVIT) that best matches the depth and scale of this model is trained on the same dataset (i.e., it contains 12 attention encoding layers and 768 embedding dimensions), denoted as "3DVIT"; secondly, a model consisting of two fully connected layers of MLP replacing "VITE" and "MAC" is also trained, and this model is denoted as "MVFF+CMSF+CN+MLP"; then the two are compared with the combined model "MVFF+CMSF+CN+VITE", and the results show that the performance of both is far lower than the performance of the combination of the two, which explains the fact that "3DVIT" has limited convergence in small-scale datasets, insufficient local feature induction ability, and "MVFF+CMSF+CN+MLP" lacks effective context modeling ability;

[0214] Table 3 Details of ablation experiment results

[0215]

[0216] This method proposes an early Alzheimer's disease auxiliary diagnosis model based on a multi-view hybrid attention convolutional neural network assisted by a Transformer; in this model, a feature extraction backbone network based on three-dimensional convolution is constructed, which contains an adaptive and efficient spatial and channel dimension weighted hybrid attention module to fuse the feature information of the sagittal, coronal and axial perspectives of sMRI to more accurately locate the subtle brain atrophy areas of AD patients; at the same time, a cascaded multi-scale feature fusion strategy is adopted to reuse feature maps of different granularity levels to further enhance the feature representation of the backbone network; secondly, a Transformer encoder network is introduced into the model to perform context modeling on the output feature map of the backbone network to learn the dependency between the whole brain pathological areas and achieve more generalized feature modeling; in addition, the MMSE clinical diagnosis score is introduced as additional supervision information, and a hybrid loss function is designed to guide the model to learn more efficiently and accurately; in summary, the model proposed by this method is an end-to-end model based on whole-brain imaging features. The evaluation results on the ADNI dataset show that compared with existing studies, the model performs well in both AD diagnosis and MCI conversion prediction tasks.

[0217] The same or similar reference numerals correspond to the same or similar components;

[0218] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0219] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for Alzheimer's disease classification based on multi-view hybrid attention network, characterized in that: The following steps are involved: S1: Acquire sMRI dataset and perform preprocessing; The sMRI data set includes three-dimensional sMRI image data of several subjects; S2: Establish a classification model for Alzheimer's disease; The Alzheimer's disease classification model includes: a feature extraction backbone network, a Transformer encoder, and an MMSE score auxiliary classifier connected in sequence; The feature extraction backbone network also includes: a feature rough extraction subnetwork, a plurality of multi-view feature fusion modules connected in sequence, and a cascade multi-scale feature fusion subnetwork; the output of each multi-view feature fusion module is connected to the input of the cascade multi-scale feature fusion subnetwork; S3: input the preprocessed sMRI dataset into the Alzheimer's disease classification model for iterative training to obtain a trained Alzheimer's disease classification model; S4: Obtain sMRI image data of the subject to be classified, input the sMRI image data of the subject to be classified into a trained Alzheimer's disease classification model for classification, and complete the classification of Alzheimer's disease.

2. According to claim 1, a method for classifying Alzheimer's disease based on a multi-view hybrid attention network is characterized in that: The preprocessing in step S1 includes: The three-dimensional sMRI image data of each subject is preprocessed by performing skull stripping, spatial registration, smoothing and denoising, blank background clipping, resampling, format conversion, data enhancement and data partitioning operations.

3. The Alzheimer's disease classification method based on multi-view hybrid attention network according to claim 1 is characterized in that: The feature rough extraction subnetwork in step S2 includes: a 3D convolution block, a Convnext feature extraction block 1 and a Convnext feature extraction block 2 connected in sequence; The structure of the 3D convolution block includes: a 4×4×4 convolution layer, a batch normalization BN layer, and a ReLU activation layer connected in sequence; The structure of each of the Convnext feature extraction blocks includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext feature extraction block also form a residual sum connection through a bypass.

4. The Alzheimer's disease classification method based on multi-view hybrid attention network according to claim 1, characterized in that: The multi-view feature fusion module in step S2 includes branches 1, 2 and 3 arranged in parallel, wherein branches 1, 2 and 3 correspond to the sagittal plane view, the coronal plane view and the axial plane view, respectively; The structure of the branch 1 includes: a convolution block 1 and a mixed attention block 1 connected in sequence; the structure of the branch 2 includes: a convolution block 2 and a mixed attention block 2 connected in sequence; the structure of the branch 3 includes: a convolution block 3 and a mixed attention block 3 connected in sequence; Each of the convolution blocks has the same structure, and includes: a 4×4×4 convolution layer, a batch normalization BN layer, a ReLU activation layer, and a 1×1×1 convolution layer connected in sequence; Each of the hybrid attention blocks has the same structure, and includes: a spatial domain attention branch and a channel domain attention branch arranged in parallel; the spatial domain attention branch includes: a 1×1×1 convolutional layer, a 3×3×3 convolutional layer, a 3×3×3 convolutional layer, a 1×1×1 convolutional layer, and a batch normalization BN layer connected in sequence; The channel domain attention branch includes: an adaptive pooling layer and a one-dimensional convolution layer connected in sequence; The outputs of the spatial domain attention branch and the channel domain attention branch are added and normalized by a Sigmoid activation function to obtain a weighted attention score of the mixed domain, the weighted attention score of the mixed domain is weighted multiplied by the input of the mixed attention block, and the weighted multiplication result is used as the output of the mixed attention block; The outputs of branches 1, 2 and 3 are spliced ​​and fused as the output of the multi-view feature fusion module.

5. The Alzheimer's disease classification method based on multi-view hybrid attention network according to claim 1, characterized in that: In the step S2, a transition layer is also provided before each of the multi-view feature fusion modules, and the transition layer includes: a 1×1×1 convolution layer, a batch normalization BN layer, a ReLU activation layer, a maximum pooling downsampling layer and a Convnext block connected in sequence; The Convnext block includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext block also form a residual sum connection through a bypass.

6. The Alzheimer's disease classification method based on multi-view hybrid attention network according to claim 1, characterized in that: In step S2, the outputs of each multi-view feature fusion module are recorded as: F1, F2, F3, ..., F n , n is a positive integer, indicating the number of multi-view feature fusion modules; In the cascaded multi-scale feature fusion sub-network, F n After 1×1×1 convolutional layer and upsampling layer, n-1 Perform feature splicing and fusion to obtain the intermediate fusion feature map M1; After the intermediate fusion feature map M1 passes through a 1×1×1 convolution layer and an upsampling layer, it is combined with F n-2 Perform feature splicing and fusion to obtain the intermediate fusion feature map M2; Repeat the above fusion steps recursively to obtain the intermediate fusion feature maps M1, M2, ..., M n-1 ; The intermediate fusion feature map M n-1 After convolution operation and average pooling downsampling operation, it is fused with the intermediate feature map M n-2 Perform secondary fusion to obtain the intermediate fusion feature map P1; After the convolution operation and average pooling downsampling operation, the intermediate fusion feature map P1 is combined with the intermediate fusion feature map M n-3 Perform secondary fusion to obtain the intermediate fusion feature map P2; Repeat the above secondary fusion steps recursively to obtain the intermediate fusion feature map P of the last secondary fusion n-2 , the intermediate fusion feature map P n-2 With F n An addition operation is performed, and the addition result is used as the output of the cascaded multi-scale feature fusion sub-network.

7. The Alzheimer's disease classification method based on multi-view hybrid attention network according to claim 1, characterized in that: The Transformer encoder in step S2 includes: a flat linear layer and a plurality of Transformer blocks connected in sequence; Each of the Transformer blocks has the same structure, and includes: a layer-normalized LN layer 1, a multi-head self-attention layer, a layer-normalized LN layer 2, and a multi-layer perceptron layer connected in sequence; the input of the layer-normalized LN layer 1 also forms a residual sum connection with the output of the multi-head self-attention layer; the input of the layer-normalized LN layer 2 also forms a residual sum connection with the output of the multi-layer perceptron layer.

8. The Alzheimer's disease classification method based on multi-view hybrid attention network according to claim 1, characterized in that: The MMSE score auxiliary classifier in step S2 includes: an MMSE score prediction branch and an Alzheimer's disease classification branch arranged in parallel; The MMSE score prediction branch includes a linear layer FC1 and a linear layer FC2 connected in sequence; the Alzheimer's disease classification branch includes a linear layer FC3; The output of the Transformer encoder is subjected to matrix slicing processing to obtain a classification projection vector c after global information mapping and a prediction projection vector m that additionally introduces an MMSE score; The predicted projection vector m is input into the MMSE score prediction branch for nonlinear mapping to obtain the predicted value of the MMSE score; the classified projection vector c is input into the Alzheimer's disease classification branch to obtain the predicted value of the class probability of Alzheimer's disease.

9. The Alzheimer's disease classification method based on a multi-view hybrid attention network according to any one of claims 1 to 8, characterized in that: The Alzheimer's disease classification model is also provided with a mixed loss function L Hybird , the mixed loss function L Hybird Including MMSE score loss function L MMSE And the classification loss function L Class , specifically: L Hybird =L Class +αL MMSE Where N is the total number of batch samples; s i ,p i are the MMSE prediction score and category prediction probability of the i-th sample, respectively, i ,y i are the MMSE true score and true category label of the i-th sample, represents the true category label of the i-th sample after smoothing; ε represents the label smoothing parameter; α is a hyperparameter.

10. An Alzheimer's disease classification system based on a multi-view hybrid attention network, applying the Alzheimer's disease classification method based on a multi-view hybrid attention network as described in any one of claims 1 to 9, characterized in that: include: Data acquisition and preprocessing unit: used to acquire sMRI data sets and perform preprocessing; The sMRI data set includes three-dimensional sMRI image data of several subjects; Model building unit: used to build Alzheimer's disease classification model; The Alzheimer's disease classification model includes: a feature extraction backbone network, a Transformer encoder, and an MMSE score auxiliary classifier connected in sequence; The feature extraction backbone network also includes: a feature rough extraction subnetwork, a plurality of multi-view feature fusion modules connected in sequence, and a cascade multi-scale feature fusion subnetwork; the output of each multi-view feature fusion module is connected to the input of the cascade multi-scale feature fusion subnetwork; Model training unit: used to input the preprocessed sMRI data set into the Alzheimer's disease classification model for iterative training to obtain a trained Alzheimer's disease classification model; Alzheimer's disease classification unit: used to obtain sMRI image data of subjects to be classified, input the sMRI image data of subjects to be classified into the trained Alzheimer's disease classification model for classification, and complete the classification of Alzheimer's disease.

Images