Alzheimer's disease classification method and system based on multi-modal hypergraph attention network

Abstract

The application provides an Alzheimer's disease classification method and system based on a multi-modal supergraph attention network, and the method comprises the following steps: acquiring sMRI image data of the brain of a plurality of Alzheimer's disease patients and performing preprocessing; performing feature extraction on the preprocessed sMRI image data, and constructing a plurality of cross-modal supergraphs according to the image features and morphological features of the brain regions of the patients; establishing a supergraph attention neural network model, training the cross-modal supergraphs, finally acquiring sMRI image data of the brain of a patient to be diagnosed, constructing a corresponding supergraph, inputting the trained supergraph attention neural network model for classification, and obtaining an Alzheimer's disease classification result and the attention weight corresponding to each supergraph; the application can effectively improve the accuracy of the Alzheimer's disease classification task, and can also find out which brain regions and morphological supergraphs have a significant contribution degree in the model, which is helpful for accurate diagnosis by doctors.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and neuroimaging processing, and more particularly, to an Alzheimer's disease classification method and system based on a multi-modal hypergraph attention network. BACKGROUND

[0002] Alzheimer's disease (AD) is a typical neurodegenerative disease, and clinically manifests as memory loss, loss of language ability, and loss of self-care ability, etc. Mild cognitive impairment (MCI) is a state between AD and healthy state HC (Healthy Controls, HC), and can be subdivided into MCI patients who will convert to AD (MCIc) and MCI patients who will not convert to AD (MCInc). Accurate screening of MCI patients helps early diagnosis of AD.

[0003] In recent years, machine learning techniques have been widely applied to analyze neuroimaging data. However, most of the existing solutions focus on extracting features from a single modality. However, the diagnosis of AD is essentially multi-modal, and doctors need to comprehensively analyze the physiological or behavioral symptoms, medical history and related medical images of patients to diagnose them. Different modalities can reveal different AD pathological changes, so the use of multi-modal data will help to more accurately diagnose AD.

[0004] Hypergraph is a generalized graph, which is the most general discrete structure in a finite set, and has a wide range of applications in the fields of information science, life science, etc. One edge of the hypergraph can connect any number of vertices. The hypergraph structure can better record the multi-element high-order correlation between the ROIs in the image data.

[0005] The prior art discloses a multi-modal Alzheimer's disease medical image classification method and system, which comprises the following steps: inputting a structural magnetic resonance imaging image into a first 3DCNN network to obtain a first feature of the structural magnetic resonance imaging image; inputting a positron emission tomography imaging image into a second 3DCNN network to obtain a second feature of the positron emission tomography imaging image; inputting the first feature and the second feature into an encoder module in a Transformer model to obtain a fusion feature of a brain to be classified; and inputting the fusion feature into a multi-layer perception network to obtain an Alzheimer's disease degree image classification result of the structural magnetic resonance imaging image and the positron emission tomography imaging image of the brain to be classified.

[0006] The prior art also discloses an abnormal brain connection prediction system, method, device and readable storage medium, which automatically extracts high-order correlation features within different modalities and high-order complementary features between different modalities by a deep learning method, and realizes analysis of multi-modal brain network abnormal connection and prediction of different cognitive diseases by an adversarial training method; the method uses prior knowledge to guide model learning of interpretable representation, and through a pair of collaborative discriminators, the consistency of representation distribution of different modalities is constrained, then a reverse generator and a decoder are used to reconstruct brain data from feature coding, finally a hypergraph perception fusion module is used to extract high-order correlation features between modalities and within modalities, and an adversarial loss, a reconstruction loss and a classification loss function are set to guide model learning, so as to achieve the purpose of mining abnormal brain connections of Alzheimer's disease; although the method in the prior art classifies and predicts Alzheimer's disease by constructing multi-modal hypergraph data, it is only a general brain connection abnormal region diagnosis method, and since important brain regions of AD have different manifestations at different times, especially for MCIc and MCInc classification with high complexity, the method in the prior art often cannot accurately predict. SUMMARY

[0007] To overcome the defects that the prior art uses multi-modal data for early diagnosis of AD and has low accuracy and cannot judge the importance of different brain regions, the application provides an Alzheimer's disease classification method and system based on a multi-modal hypergraph attention network, which not only improves the accuracy of the classification task, but also finds out which hypergraph contributes more to the classification result, thereby helping doctors pay more attention to the corresponding brain region when diagnosing different patients.

[0008] To solve the above technical problems, the technical scheme of the application is as follows:

[0009] The application provides an Alzheimer's disease classification method based on a multi-modal supergraph attention network, comprising the following steps:

[0010] S1: acquiring sMRI image data of brains of a plurality of Alzheimer's disease patients and performing preprocessing;

[0011] S2: performing feature extraction on the preprocessed sMRI image data to obtain image features and morphological features of brain regions of the patients;

[0012] S3: constructing a plurality of cross-modal supergraphs according to the image features and morphological features of the brain regions of the patients;

[0013] S4: establishing a supergraph attention neural network model, training the cross-modal supergraphs, and obtaining a trained supergraph attention neural network model;

[0014] S5: acquiring sMRI image data of a brain of a patient to be diagnosed and a plurality of cross-modal supergraphs of the patient to be diagnosed; inputting the plurality of cross-modal supergraphs of the patient to be diagnosed into the trained supergraph attention neural network model for classification, obtaining an Alzheimer's disease classification result of the patient to be diagnosed, and obtaining attention weights corresponding to the plurality of cross-modal supergraphs of the patient to be diagnosed.

[0015] Preferably, in the step S1, the specific method for acquiring sMRI image data of brains of a plurality of Alzheimer's disease patients and performing preprocessing is as follows:

[0016] acquiring sMRI image data of brains of a plurality of Alzheimer's disease patients and performing image preprocessing and morphological preprocessing on the sMRI image data respectively;

[0017] The specific method for image preprocessing is as follows:

[0018] sequentially performing spatial segmentation, skull removal, registration to a standard Montreal Neurological Institute space, and image smoothing processing on the sMRI image data to obtain smoothed sMRI image data;

[0019] The specific method for morphological preprocessing is as follows:

[0020] sequentially performing skull removal, intensity standardization, label volume, white matter segmentation, smoothing flattening, cortical division, statistics and mapping processing on the sMRI image data to obtain morphological indexes of 210 brain regions;

[0021] saving the smoothed sMRI image data and the morphological indexes of all brain regions together as preprocessed sMRI image data.

[0022] Preferably, the morphological indicators of the brain region include: average thickness, thickness standard deviation, gray matter volume, area, folding index, curvature, average curvature, and Gaussian curvature of the brain region.

[0023] Preferably, in the step S2, the specific method for extracting the image features and morphological features of the brain region of the patient from the preprocessed sMRI image data is:

[0024] The smoothed sMRI image data is aligned with the preset Brainneome template, and four hippocampal region regions of interest are extracted, specifically: left rostral hippocampus brain region, right rostral hippocampus brain region, left caudal hippocampus brain region, and right caudal hippocampus brain region.

[0025] The trained three-dimensional convolutional neural network is used to extract deep features of the brain region of all hippocampal regions of interest, and all extracted deep features of the brain region are saved as image features of the brain region of the patient.

[0026] All morphological indicators of the 210 brain regions are collectively saved as morphological features to be selected, and the morphological features to be selected are sequentially normalized and selected to obtain the morphological features of the brain region of the patient.

[0027] Preferably, the specific method for feature selection is: using the chi-square detection method for feature selection: for each brain region, the score corresponding to the normalized morphological feature to be selected is calculated, and the K morphological indicators with the highest score are selected from all morphological features to be selected as the morphological features of the brain region of the patient.

[0028] Preferably, in the step S3, the specific method for constructing a plurality of cross-modal hypergraphs according to the image features and morphological features of the brain region of the patient is:

[0029] The image features and morphological features of the brain region of the patient are combined to obtain cross-modal features of the four hippocampal regions of interest.

[0030] For each hippocampal region of interest, a cross-modal hypergraph is constructed using the corresponding cross-modal features, and the specific method is:

[0031] For the cross-modal features of each hippocampal region of interest, a cross-modal hypergraph is constructed using the K-nearest neighbor method, and the specific method is:

[0032] Select a patient as a center vertex, and select other patients as other vertices. The Euclidean distance is used to calculate the cross-modal feature difference between the center vertex and the other vertices, and a hyperedge centered on the center vertex is constructed, which is used to connect k other vertices with the smallest cross-modal feature difference.

[0033] If there are n patients, n central vertices are constructed, and the above method is repeated to obtain a cross-modal hypergraph containing n hyperedges;

[0034] The above steps are repeated to obtain four cross-modal hypergraphs containing n hyperedges.

[0035] Preferably, before calculating the cross-modal feature difference between the central vertex and other vertices using the Euclidean distance, the method further comprises converting the hyperedge weight between the central vertex and other vertices into a value less than 1, specifically:

[0036] The hyperedge weight W between the i-th central vertex and the j-th other vertex is calculated according to the following formula i,j :

[0037]

[0038] Where D i,j is the cross-modal feature distance between the i-th central vertex and the j-th other vertex, and Δ is the average cross-modal feature distance between the central vertex and the other vertices.

[0039] Preferably, the hypergraph attention neural network model established in step S4 comprises a plurality of hypergraph convolution layers, a first attention layer, a dynamic hypergraph construction layer, a second attention layer and a decision layer connected in sequence; the output end of the first attention layer is also connected to the input end of the second attention layer.

[0040] Preferably, in the dynamic hypergraph construction layer, the cross-modal hypergraph features fused by the first attention layer are dynamically updated using a k-NN algorithm and a k-means clustering algorithm, and new cross-modal hypergraph features are generated.

[0041] The application also provides an Alzheimer's disease classification system based on a multi-modal hypergraph attention network, which applies the above-mentioned Alzheimer's disease classification method based on a multi-modal hypergraph attention network, comprising:

[0042] A preprocessing unit is used to obtain sMRI image data of a plurality of Alzheimer's patients and perform preprocessing;

[0043] A feature extraction unit is used to extract features from the preprocessed sMRI image data, and obtain image features and morphological features of the patient's brain region;

[0044] A hypergraph construction unit is used to construct a plurality of cross-modal hypergraphs according to the image features and morphological features of the patient's brain region;

[0045] A model training unit is used to establish a hypergraph attention neural network model, train the model using the cross-modal hypergraphs, and obtain a trained hypergraph attention neural network model;

[0046] The classification prediction unit is used for obtaining sMRI image data of a brain of a patient to be diagnosed, and obtaining a plurality of cross-modal hypergraphs of the patient to be diagnosed; inputting the plurality of cross-modal hypergraphs of the patient to be diagnosed into the trained hypergraph attention neural network model for classification, obtaining an Alzheimer's disease classification result of the patient to be diagnosed, and attention weights corresponding to each cross-modal hypergraph of the patient to be diagnosed.

[0047] Compared with the prior art, the beneficial effects of the technical scheme of the present application are:

[0048] The present application provides an Alzheimer's disease classification method and system based on a multi-modal hypergraph attention network, first obtaining sMRI image data of a plurality of Alzheimer's disease patient brains and preprocessing; performing feature extraction on the preprocessed sMRI image data to obtain image features and morphological features of the patient brain regions; constructing a plurality of cross-modal hypergraphs according to the image features and morphological features of the patient brain regions; establishing a hypergraph attention neural network model, training using the cross-modal hypergraphs, and obtaining a trained hypergraph attention neural network model; finally obtaining sMRI image data of a brain of a patient to be diagnosed, and obtaining a plurality of cross-modal hypergraphs of the patient to be diagnosed; inputting the plurality of cross-modal hypergraphs of the patient to be diagnosed into the trained hypergraph attention neural network model for classification, obtaining an Alzheimer's disease classification result of the patient to be diagnosed, and attention weights corresponding to each cross-modal hypergraph of the patient to be diagnosed;

[0049] The present application constructs a cross-modal hypergraph by MRI and morphological features to represent the high-order structural relationship between patients, which can effectively improve the accuracy of the Alzheimer's disease classification task; in addition, the hypergraph attention neural network model established by the present application can output different contributions of different hypergraphs to the classification result by comparing and learning new hypergraph features and old hypergraph features, thereby helping doctors to focus on the corresponding brain regions when diagnosing different patients, and improving the accuracy of early diagnosis of Alzheimer's disease and the generalization ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A flowchart of an Alzheimer's disease classification method based on a multi-modal hypergraph attention network provided for embodiment 1.

[0051] Figure 2 A feature selection flowchart provided for embodiment 2.

[0052] Figure 3 A hypergraph attention neural network model structure diagram provided for embodiment 2.

[0053] Figure 4 A comparison diagram of attention weights of each cross-modal hypergraph provided for embodiment 2.

[0054] Figure 5 This is a structural diagram of an Alzheimer's disease classification system based on a multimodal hypergraph attention network provided in Example 3. Detailed Implementation

[0055] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0056] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0057] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0058] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] Example 1

[0060] like Figure 1 As shown, this invention provides an Alzheimer's disease classification method based on a multimodal hypergraph attention network, comprising the following steps:

[0061] S1: Acquire sMRI image data of the brains of several Alzheimer's patients and perform preprocessing;

[0062] S2: Feature extraction is performed on the preprocessed sMRI image data to obtain the image features and morphological features of the patient's brain regions;

[0063] S3: Construct several cross-modal hypermaps based on the image and morphological features of the patient's brain regions;

[0064] S4: Establish a hypergraph attention neural network model, train it using a cross-modal hypergraph, and obtain the trained hypergraph attention neural network model;

[0065] S5: Obtain sMRI image data of the brain of the patient to be diagnosed, and obtain several cross-modal hypermaps of the patient to be diagnosed; input the several cross-modal hypermaps of the patient to be diagnosed into the trained hypermap attention neural network model for classification, obtain the Alzheimer's disease classification results of the patient to be diagnosed, and the attention weights corresponding to each cross-modal hypermap of the patient to be diagnosed.

[0066] In the implementation process, first, sMRI image data of brains of a plurality of Alzheimer's disease patients is acquired and preprocessed; image features and morphological features of brain regions of the patients are acquired by performing feature extraction on the preprocessed sMRI image data; a plurality of cross-modal hypergraphs are constructed according to the image features and the morphological features of the brain regions of the patients; a hypergraph attention neural network model is established, the cross-modal hypergraphs are used for training, and a trained hypergraph attention neural network model is acquired; finally, sMRI image data of a brain of a patient to be diagnosed is acquired, and a plurality of cross-modal hypergraphs of the patient to be diagnosed are acquired; the plurality of cross-modal hypergraphs of the patient to be diagnosed are input into the trained hypergraph attention neural network model for classification, and an Alzheimer's disease classification result of the patient to be diagnosed and attention weights corresponding to the plurality of cross-modal hypergraphs of the patient to be diagnosed are acquired;

[0067] The brain region corresponding to the cross-modal hypergraph with the highest attention weight is taken as a brain region with the largest contribution to the Alzheimer's disease classification result, and classification and early diagnosis of Alzheimer's disease patients are completed;

[0068] The method can effectively improve the accuracy of the Alzheimer's disease classification task by constructing cross-modal hypergraphs from MRI and morphological features to represent high-order structural relationships between patients. In addition, the hypergraph attention neural network model established by the method can output different contributions of different hypergraphs to the classification result by comparing new hypergraph features with old hypergraph features, thereby helping doctors to focus on corresponding brain regions when diagnosing different patients and improving the accuracy of early diagnosis of Alzheimer's disease and the generalization ability of the model.

[0069] Embodiment 2

[0070] The embodiment provides an Alzheimer's disease classification method based on a multi-modal hypergraph attention network, including the following steps:

[0071] S1: Acquire sMRI image data of brains of a plurality of Alzheimer's disease patients and preprocess the sMRI image data;

[0072] S2: Acquire image features and morphological features of brain regions of the patients by performing feature extraction on the preprocessed sMRI image data;

[0073] S3: Construct a plurality of cross-modal hypergraphs according to the image features and the morphological features of the brain regions of the patients;

[0074] S4: Establish a hypergraph attention neural network model, use the cross-modal hypergraphs for training, and acquire a trained hypergraph attention neural network model;

[0075] S5: acquiring sMRI image data of a brain of a patient to be diagnosed and acquiring a plurality of cross-modality supergraphs of the patient to be diagnosed; inputting the plurality of cross-modality supergraphs of the patient to be diagnosed into the trained supergraph attention neural network model for classification to obtain an Alzheimer's disease classification result of the patient to be diagnosed and attention weights corresponding to each of the plurality of cross-modality supergraphs of the patient to be diagnosed;

[0076] In the step S1, the specific method of acquiring sMRI image data of a plurality of Alzheimer's disease patient brains and preprocessing the sMRI image data is as follows:

[0077] Acquire sMRI image data of a plurality of Alzheimer's disease patient brains and perform image preprocessing and morphological preprocessing on the sMRI image data respectively;

[0078] The specific method of the image preprocessing is as follows:

[0079] The sMRI image data is sequentially subjected to spatial segmentation, skull removal, registration to a standard Montreal Neurological Institute space, and image smoothing processing to obtain smoothed sMRI image data;

[0080] The specific method of the morphological preprocessing is as follows:

[0081] The sMRI image data is sequentially subjected to skull removal, intensity standardization, label volume, white matter segmentation, smoothing flattening, cortical division, statistics and mapping processing to obtain morphological indicators of 210 brain regions;

[0082] The smoothed sMRI image data and the morphological indicators of all brain regions are collectively saved as preprocessed sMRI image data;

[0083] The morphological indicators of the brain regions include average thickness, thickness standard deviation, gray matter volume, area, folding index, curvature, average curvature and Gaussian curvature of the brain regions;

[0084] In the step S2, the specific method of extracting image features and morphological features of the patient brain regions from the preprocessed sMRI image data is as follows:

[0085] Align the smoothed sMRI image data with a preset Brainneome template to extract 4 hippocampal region regions of interest, specifically, left rostral hippocampus brain region, right rostral hippocampus brain region, left caudal hippocampus brain region and right caudal hippocampus brain region;

[0086] Extract brain region deep features of all hippocampal region regions of interest using a trained three-dimensional convolutional neural network, and save the extracted brain region deep features as image features of the patient brain regions;

[0087] All morphological indexes of 210 brain regions are collectively saved as morphological features to be selected, and the morphological features to be selected are sequentially normalized and selected to obtain morphological features of brain regions of the patient;

[0088] The specific method of the feature selection is: using the chi-square test method to select features: for each brain region, the scores corresponding to the normalized morphological features to be selected are calculated, and the K morphological indexes with the highest scores are selected from all the morphological features to be selected as the morphological features of the brain region of the patient;

[0089] In the step S3, the specific method of constructing the cross-modal hypergraph according to the image features and the morphological features of the brain region of the patient is:

[0090] The image features and the morphological features of the brain region of the patient are combined to obtain cross-modal features of four hippocampal region of interest areas;

[0091] For each hippocampal region of interest area, a cross-modal hypergraph is constructed using the corresponding cross-modal features, and the specific method is:

[0092] For the cross-modal features of each hippocampal region of interest area, a cross-modal hypergraph is constructed using the K nearest neighbor method, and the specific method is:

[0093] Select a patient as a center vertex, and select other patients as other vertices, calculate the cross-modal feature difference between the center vertex and the other vertices using the Euclidean distance, and construct a hyperedge with the center vertex as the center, the hyperedge is used to connect k other vertices with the smallest cross-modal feature difference;

[0094] If there are n patients, n center vertices are constructed and the above method is repeated to obtain a cross-modal hypergraph containing n hyperedges;

[0095] Repeat the above steps to obtain four cross-modal hypergraphs containing n hyperedges;

[0096] Before the step of calculating the cross-modal feature difference between the center vertex and the other vertices using the Euclidean distance, the step of converting the hyperedge weight between the center vertex and the other vertices to a value less than 1 is further included, and the specific method is:

[0097] The hyperedge weight W between the i th center vertex and the j th other vertex is calculated according to the following formula i,j :

[0098]

[0099] Where D i,j is the cross-modal feature distance between the i th center vertex and the j th other vertex, and Δ is the average cross-modal feature distance between the center vertex and the other vertices;

[0100] The hypergraph attention neural network model established in the step S4 comprises a plurality of hypergraph convolution layers, a first attention layer, a dynamic hypergraph construction layer, a second attention layer and a decision layer connected in sequence; the output end of the first attention layer is also connected with the input end of the second attention layer;

[0101] In the dynamic hypergraph construction layer, the cross-modal hypergraph features fused by the first attention layer are dynamically updated by using a k-NN algorithm and a k-means clustering algorithm, and new cross-modal hypergraph features are generated.

[0102] In the specific implementation process, firstly, a plurality of sMRI image data of the brain of Alzheimer's disease patients are acquired and preprocessed. The data used in this embodiment is from the public Alzheimer's Disease Neuroimaging Initiative (ADNI) database. The main purpose of ADNI is to detect whether a series of magnetic resonance imaging (MRI), positron emission tomography (PET), other biomarkers and clinical and neuropsychological assessments can be jointly applied to measure the progression of mild cognitive impairment (MCI) and early Alzheimer's disease (AD).

[0103] The preprocessing of the method includes two directions, one is image preprocessing and the other is morphological preprocessing. The preprocessing steps of the image include spatial segmentation, skull removal, registration to the standard Montreal Neurological Institute (MNI) space and image smoothing. After preprocessing, the size of all images is 121x145x121 (XxYxZ), and the spatial resolution is 2x2x2mm 3 voxel;

[0104] The morphological preprocessing includes: removing the skull of the magnetic resonance imaging data of Alzheimer's disease; performing CA intensity standardization on the magnetic resonance imaging data after removing the skull; performing CA marker volume on the standardized magnetic resonance imaging data; performing white matter segmentation on the marked magnetic resonance imaging data; performing smoothing and inflation on the segmented magnetic resonance imaging data using Tessellation subdivision surface technology; performing spherical mapping and registration on the inflated magnetic resonance imaging data; performing cortical division, statistics and mapping on the spherical registered magnetic resonance imaging data; and finally obtaining 8 morphological index features of the brain region, including average thickness, thickness standard deviation, surface area, gray matter volume, integral correction mean curvature, integral correction Gaussian curvature, fold index and intrinsic curvature index. Among them, 210 brain region indexes on the cortex are extracted, so that 210x8=1680 brain region morphological features are extracted from each magnetic resonance imaging data of Alzheimer's disease.

[0105] After the pre-processed sMRI image data is feature extracted, the image features and morphological features of the patient's brain region are obtained;

[0106] With the passage of time, the reduction of hippocampus volume will lead to amnesia syndrome, which is the core feature of Alzheimer's disease; accordingly, the method extracts the hippocampus region of interest (ROI) in the image; when the size and coordinate space of the pre-processed image are consistent with the Brainneome template, the hippocampus brain region image is extracted as the input sample of the subsequent model according to the brain region division mask of the template; the hippocampus is divided into four ROIs in the Brainneome template, which are left rostral hippocampus (215), right rostral hippocampus (216), left caudal hippocampus (217) and right caudal hippocampus (218) brain regions, and the numbers in the brackets represent the label ID in the Brainneome template; therefore, each patient can obtain 4 hippocampus regions of interest; when the hippocampus region of interest is passed through the trained 3D CNN, the features in the full connection layer of the 3D CNN are taken as the deep features, and each ROI can obtain its corresponding deep features; all the extracted brain region deep features are saved as the image features of the patient's brain region;

[0107] Since not all brain region morphological features are effective information, it is necessary to use feature selection to screen 1680 brain region features and remove irrelevant and redundant features; the method saves all morphological indexes of 210 brain regions as morphological features to be selected, normalizes the morphological features to be selected first, so that the brain region feature values of all samples are between 0 and 1, and then performs feature selection to obtain the morphological features of the patient's brain region, which are combined with the image features of the patient's brain region;

[0108] As shown in Figure 2 , the method of chi-square test is used for feature selection: for each brain region, the score corresponding to the normalized morphological feature to be selected is calculated, and the K highest morphological indexes are selected from all morphological features to be selected as the morphological features of the patient's brain region, specifically:

[0109] Two operations are performed on the normalized morphological features to be selected: the first operation: the features of each column of brain region are accumulated according to the categories respectively, and observed (2x1680) is obtained; the second operation: the features of each column of brain region are accumulated, and fts (1x1680) is obtained, the frequencies of two labels are counted, and then the dot product operation is performed with fts, and expected (2x1680) is obtained; then the feature score score (i, j) = (observed (i, j)-expeted (i, j))**2 / expected (i, j) corresponding to the position is calculated, and then accumulated according to the column, and score (1x1680) is obtained, that is, the score of all morphological features to be selected, finally, the top K brain region features with the highest score are retained, and the feature selection is completed;

[0110] Then, four cross-modal hypergraphs are constructed according to the image features and morphological features of the brain regions of the patients, and the specific method is as follows:

[0111] The image features and morphological features of the brain regions of the patients are combined to obtain the cross-modal features of the four hippocampal region of interest regions;

[0112] For each hippocampal region of interest region, a cross-modal hypergraph is constructed using the corresponding cross-modal features, and the specific method is as follows:

[0113] For the cross-modal features of each hippocampal region of interest region, a cross-modal hypergraph is constructed using the K nearest neighbor method, and the specific method is as follows:

[0114] Select a patient as a center vertex, and use the Euclidean distance to calculate the cross-modal feature difference between the center vertex and other vertices, and construct a hyperedge with the center vertex as the center, which is used to connect k other vertices with the smallest cross-modal feature difference, and in this embodiment, k=16;

[0115] If there are n patients, n center vertices are constructed and the above method is repeated to obtain a cross-modal hypergraph containing n hyperedges;

[0116] The above steps are repeated to obtain four cross-modal hypergraphs containing n hyperedges; it is worth noting that each cross-modal hypergraph contains key information about ROI and morphology, which is used to standardize the high-order structural relationship between patients;

[0117] A hypergraph attention neural network model is established, and the cross-modal hypergraph is used for training to obtain a trained hypergraph attention neural network model;

[0118] As Figure 3As shown, the hypergraph attention neural network model established in the embodiment includes a plurality of hypergraph convolution layers, a first attention layer, a dynamic hypergraph construction layer, a second attention layer and a decision layer connected in sequence and arranged in parallel; the output end of the first attention layer is also connected with the input end of the second attention layer;

[0119] The hypergraph convolution layer performs hypergraph convolution on each cross-modal hypergraph, and then forms a set Gs and sends it to the first attention layer; the first attention layer can capture the interaction between hypergraphs and fuse them; then the fused hypergraph features are input into the dynamic hypergraph construction (DHG) layer to generate new hypergraph features; then the second attention layer is used to balance the new hypergraph features and the old hypergraph features; finally, the output is transmitted to the decision layer for classification;

[0120] In the dynamic hypergraph construction layer, the cross-modal hypergraph features fused by the first attention layer are dynamically updated by using the k-NN algorithm and the k-means clustering algorithm, and new cross-modal hypergraph features are generated;

[0121] The first attention layer and the second attention layer in the embodiment have similar structures, and for a sample u, the features x u ∈R 1×d of u in the hypergraph set Gs are taken in sequence, where d is the input dimension of the feature. The multi-layer perception (MLP) generates a weight value w u for the feature x u . Then w u is added to the weight set w. After obtaining all the weights, the softmax function is used to map the weights between 0 and 1. Then the features x u ∈R 1 ×d of u in the hypergraph set Gs are taken in sequence, and the result of multiplying x u with the corresponding weight value w i is accumulated to y u , and finally the fused feature y u of the sample u is output, and the specific algorithm is as follows:

[0122]

[0123] The traditional hypergraph method constructs a hypergraph for each modality, and then combines the hypergraphs horizontally into a large hypergraph; the traditional method is difficult to explain which hypergraph plays a more important role; however, the method in the embodiment can dynamically fuse the multi-modal hypergraphs in the network, compared with the traditional hypergraph, not only the structure of the original hypergraph is retained, but also which hypergraph is more important is explained, which is more conducive to the doctor to treat the disease;

[0124] Finally, the sMRI image data of the brain of the patient to be diagnosed is obtained, and a plurality of cross-modality supergraphs of the patient to be diagnosed are obtained; the plurality of cross-modality supergraphs of the patient to be diagnosed are input into the trained supergraph attention neural network model for classification, to obtain an Alzheimer's disease classification result of the patient to be diagnosed and attention weights corresponding to each cross-modality supergraph of the patient to be diagnosed;

[0125] The decision layer outputs the Alzheimer's disease classification result of the patient to be diagnosed, and the attention weights corresponding to each cross-modality supergraph are obtained at the second attention layer;

[0126] The brain region corresponding to the cross-modality supergraph with the highest attention weight is taken as the brain region with the greatest contribution to the Alzheimer's disease classification result, and early diagnosis of the Alzheimer's disease patient is completed;

[0127] In this embodiment, MRI images of 502 subjects were downloaded from the ADNI database, and the subjects were all between 55 and 90 years old, including 133 AD patients, 161 HC, 133 MCInc patients and 75 MCIc patients, and one sMRI image was taken for each sample; in order to verify the effectiveness of the method proposed in this embodiment, experiments were carried out in different scenarios, including AD vs. HC, MCIc vs. HC, MCIc vs. MCInc; a 5-fold cross-validation strategy was used to evaluate the classification performance; in addition, four common classification evaluation indicators were used to evaluate the performance of the model, namely Accuracy (ACC), Area Under Curve (AUC), F1-Score and Matthews Correlation Coefficient (MCC); the final comparison results are shown in Table 1:

[0128] Table 1 Comparison results of the method and other methods

[0129]

[0130] In all methods listed in Table 1, the test and training sets are identical. Graph methods include GCN, HGNN, DHGNN, and HGNN+. GCN is a classic graph convolutional network where each edge connects two nodes, while HGNN is a classic hypergraph convolutional network where each edge connects multiple nodes. DHGNN and HGNN+ are extensions of HGNN. MRI and morphological modalities are used in graph methods. CNN methods include CNN+EL and MADDi. CNN+EL combines convolutional neural networks and ensemble learning, and MRI is used in the CNN+EL method. MADDi is an attention-based multimodal deep learning framework for Alzheimer's disease diagnosis, and MRI and morphological modalities are used in the MADDi method. As shown in Table 1, our method outperforms existing methods in almost all cases.

[0131] The accuracy rates for AD and HC, MCIc and HC, and MCIc and MCInc are 88.00%, 87.21%, and 71.10%, respectively. Compared with other methods, this method improves the accuracy of AD and HC by almost 2.76%. Furthermore, compared with the previous state-of-the-art MADDi method, this method improves the accuracy of AD and HC by 9.24%. Compared with other methods, this method improves the accuracy of MCIc and HC by almost 1.25%. In particular, compared with the CNN+EL method, this method improves the accuracy of MCIc and HC by 8.5%. In addition, in the most difficult classifications, MCIc and MCInc, this method improves the accuracy by more than 4.37%. Moreover, this method also has significant advantages in other evaluation metrics.

[0132] Attention weights for each cross-modal hypergraph are as follows: Figure 4 As shown, by Figure 4 It can be observed that the cross-modal hypermap constructed from the left rostral hippocampus (215) and morphology maintains a stable contribution in all three classification tasks; the cross-modal hypermap constructed from the right caudal hippocampus (218) and morphology contributes the least in the early Alzheimer's disease classification task MCIc vs. MCInc, while it contributes the most in the late Alzheimer's disease classification task AD vs. HC. Therefore, the cross-modal hypermap constructed from morphological features and right caudal hippocampal features becomes increasingly important as the disease progresses, which also means that important areas of AD have different manifestations at different times. This will help doctors pay attention to how different areas of the brain change at different times.

[0133] The method can effectively improve the accuracy of the Alzheimer's disease classification task by constructing a cross-modal hypergraph through MRI and morphological features to represent the high-order structural relationship between patients; in addition, the hypergraph attention neural network model established by the method can output different contributions of different hypergraphs to the classification result by comparing and learning new hypergraph features and old hypergraph features, thereby helping doctors to focus on the corresponding brain regions when diagnosing different patients, and improving the accuracy of early diagnosis of Alzheimer's disease and the generalization ability of the model.

[0134] Embodiment 3

[0135] As shown in Figure 5 The embodiment provides an Alzheimer's disease classification system based on a multi-modal hypergraph attention network, which applies the Alzheimer's disease classification method based on the multi-modal hypergraph attention network in Embodiment 1 or 2, and includes:

[0136] A preprocessing unit 301 is configured to acquire sMRI image data of a brain of an Alzheimer's disease patient and perform preprocessing;

[0137] A feature extraction unit 302 is configured to perform feature extraction on the preprocessed sMRI image data, and acquire image features and morphological features of a brain region of the patient;

[0138] A hypergraph construction unit 303 is configured to construct a plurality of cross-modal hypergraphs according to the image features and morphological features of the brain region of the patient;

[0139] A model training unit 304 is configured to establish a hypergraph attention neural network model, train the model using the cross-modal hypergraphs, and acquire a trained hypergraph attention neural network model;

[0140] A classification and prediction unit 305 is configured to acquire sMRI image data of a brain of a patient to be diagnosed, and acquire a plurality of cross-modal hypergraphs of the patient to be diagnosed; input the plurality of cross-modal hypergraphs of the patient to be diagnosed into the trained hypergraph attention neural network model for classification, and acquire an Alzheimer's disease classification result of the patient to be diagnosed and attention weights corresponding to the plurality of cross-modal hypergraphs of the patient to be diagnosed.

[0141] In the implementation process, first, the preprocessing unit 301 acquires sMRI image data of several Alzheimer's disease patients and performs preprocessing; the feature extraction unit 302 performs feature extraction on the preprocessed sMRI image data, acquires image features and morphological features of the brain region of the patient; the hypergraph construction unit 303 constructs several cross-modal hypergraphs according to the image features and morphological features of the brain region of the patient; the model training unit 304 establishes a hypergraph attention neural network model, trains using the cross-modal hypergraph, and acquires the trained hypergraph attention neural network model; finally, the classification and prediction unit 305 acquires sMRI image data of the brain of the patient to be diagnosed, and acquires several cross-modal hypergraphs of the patient to be diagnosed; the several cross-modal hypergraphs of the patient to be diagnosed are input into the trained hypergraph attention neural network model for classification, and the Alzheimer's disease classification result of the patient to be diagnosed and the attention weight corresponding to each cross-modal hypergraph of the patient to be diagnosed are acquired;

[0142] The brain region corresponding to the cross-modal hypergraph with the highest attention weight is taken as the brain region with the largest contribution to the Alzheimer's disease classification result, and the classification and early diagnosis of the Alzheimer's disease patient are completed;

[0143] The system can effectively improve the accuracy of the Alzheimer's disease classification task by constructing cross-modal hypergraphs through MRI and morphological features to represent the high-order structural relationship between patients; in addition, the hypergraph attention neural network model established by the system can output different contributions of different hypergraphs to the classification result by comparing and learning new hypergraph features and old hypergraph features, thereby helping doctors to focus on the corresponding brain regions when diagnosing different patients, and improving the accuracy of early diagnosis of Alzheimer's disease and the generalization ability of the model.

[0144] The same or similar reference signs correspond to the same or similar components;

[0145] The positional relationship described in the drawings is only used for illustrative description, and cannot be understood as a limitation on the patent;

[0146] Obviously, the above embodiments of the application are only examples for clearly illustrating the application, and are not intended to limit the implementation modes of the application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all implementation modes need not and cannot be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the application should be included in the protection scope of the claims of the application.

Claims

1. A method for Alzheimer's disease classification based on multi-modal hypergraph attention network, characterized in that, The method comprises the following steps: S1: obtaining sMRI image data of the brains of a plurality of Alzheimer's disease patients and preprocessing the sMRI image data; S2: extracting features from the preprocessed sMRI image data to obtain image features and morphological features of brain regions of the patients; S3: constructing a plurality of cross-modality hypergraphs according to the image features and morphological features of the brain regions of the patients, specifically: combining the image features and morphological features of the brain regions of the patients to obtain cross-modality features of four hippocampal region of interest areas; for each hippocampal region of interest area, constructing a cross-modality hypergraph using the corresponding cross-modality features, specifically: for the cross-modality features of each hippocampal region of interest area, constructing a cross-modality hypergraph using a K-nearest neighbor method, specifically: selecting a patient as a center vertex and selecting other patients as other vertices, calculating the cross-modality feature difference between the center vertex and the other vertices using the Euclidean distance, and constructing a hyperedge centered on the center vertex, which is used to connect k other vertices with the smallest cross-modality feature difference; if there are n patients, then n center vertices are constructed and the above method is repeated to obtain a cross-modality hypergraph containing n hyperedges; repeating the above steps to obtain four cross-modality hypergraphs containing n hyperedges; S4: establishing a hypergraph attention neural network model, training the hypergraph attention neural network model using the cross-modality hypergraphs, and obtaining a trained hypergraph attention neural network model; the hypergraph attention neural network model comprises a plurality of hypergraph convolution layers, a first attention layer, a dynamic hypergraph construction layer, a second attention layer and a decision layer connected in sequence; the output end of the first attention layer is also connected to the input end of the second attention layer; S5: obtaining sMRI image data of the brain of a patient to be diagnosed, and obtaining a plurality of cross-modality hypergraphs of the patient to be diagnosed; inputting the plurality of cross-modality hypergraphs of the patient to be diagnosed into the trained hypergraph attention neural network model for classification to obtain an Alzheimer's disease classification result of the patient to be diagnosed and attention weights corresponding to the plurality of cross-modality hypergraphs of the patient to be diagnosed.

2. The Alzheimer's disease classification method based on the multi-modal hypergraph attention network according to claim 1, characterized in that, In the step S1, the specific method of obtaining sMRI image data of the brains of a plurality of Alzheimer's disease patients and preprocessing the sMRI image data is: obtaining sMRI image data of the brains of a plurality of Alzheimer's disease patients and preprocessing the sMRI image data respectively; the specific method of image preprocessing is: performing spatial segmentation, skull removal, registration to a standard Montreal Neurological Institute space and image smoothing processing on the sMRI image data in sequence to obtain smoothed sMRI image data; the specific method of morphological preprocessing is: performing skull removal, intensity standardization, label volume, white matter segmentation, smoothing flattening, cortical division, statistics and mapping processing on the sMRI image data in sequence to obtain morphological indicators of 210 brain regions; saving the smoothed sMRI image data and the morphological indicators of all brain regions together as preprocessed sMRI image data.

3. The Alzheimer's disease classification method based on the multi-modal hypergraph attention network of claim 2, characterized in that, The morphological indicators of the brain regions include: average thickness, thickness standard deviation, gray matter volume, area, folding index, curvature, average curvature and Gaussian curvature of the brain regions.

4. The Alzheimer's disease classification method based on the multi-modal hypergraph attention network of claim 3, characterized in that, In the step S2, the sMRI image data after preprocessing is subjected to feature extraction to obtain image features and morphological features of the brain region of the patient, and the specific method is: The sMRI image data after smoothing is aligned with a preset Brainneome template, and four hippocampal region regions of interest are extracted, specifically: left rostral hippocampus brain region, right rostral hippocampus brain region, left caudal hippocampus brain region and right caudal hippocampus brain region; The three-dimensional convolutional neural network trained is used to extract deep brain region features of all hippocampal region regions of interest, and all the extracted deep brain region features are saved as image features of the brain region of the patient. All morphological indexes of 210 brain regions are collectively saved as morphological features to be selected, and the morphological features to be selected are sequentially subjected to normalization processing and feature selection to obtain morphological features of the brain region of the patient.

5. The Alzheimer's disease classification method based on the multi-modal hypergraph attention network according to claim 4, characterized in that, The specific method of the feature selection is: the feature selection is performed by using the chi-square detection method: for each brain region, the score corresponding to the normalized morphological feature to be selected is calculated, and the K morphological indexes with the highest score are selected from all the morphological features to be selected as the morphological features of the brain region of the patient.

6. The Alzheimer's disease classification method based on the multi-modal hypergraph attention network according to claim 5, characterized in that, Before the Euclidean distance is used to calculate the cross-modal feature difference between the center vertex and other vertices, the hyperedge weight between the center vertex and other vertices is converted into a value less than 1, specifically: The hyperedge weight between the ith central vertex and the jth other vertex is calculated according to the following formula : wherein, is the cross-modal feature distance between the ith center vertex and the jth other vertex, is the average cross-modal feature distance between the center vertex and the other vertices.

7. The Alzheimer's disease classification method based on the multi-modal hypergraph attention network according to claim 6, characterized in that, In the dynamic hypergraph construction layer, the cross-modal hypergraph features fused by the first attention layer are dynamically updated by using the k-NN algorithm and the k-means clustering algorithm, and new cross-modal hypergraph features are generated.

8. An Alzheimer's disease classification system based on a multi-modal hypergraph attention network, applying the Alzheimer's disease classification method based on a multi-modal hypergraph attention network in any one of claims 1-7, characterized in that, It comprises: A preprocessing unit for obtaining sMRI image data of the brain of a plurality of Alzheimer's patients and preprocessing the sMRI image data; A feature extraction unit for extracting features from the sMRI image data after preprocessing to obtain image features and morphological features of the brain region of the patient; A hypergraph construction unit for constructing a plurality of cross-modal hypergraphs according to the image features and morphological features of the brain region of the patient; A model training unit for establishing a hypergraph attention neural network model, training the cross-modal hypergraph, and obtaining a trained hypergraph attention neural network model; A classification and prediction unit for obtaining sMRI image data of the brain of a patient to be diagnosed, and obtaining a plurality of cross-modal hypergraphs of the patient to be diagnosed; inputting the plurality of cross-modal hypergraphs of the patient to be diagnosed into the trained hypergraph attention neural network model for classification to obtain an Alzheimer's disease classification result of the patient to be diagnosed, and attention weights corresponding to the plurality of cross-modal hypergraphs of the patient to be diagnosed.

Images