A method and system for AD classification based on three-dimensional convolution and twin neural network

Abstract

The application provides an AD classification method and system based on three-dimensional convolution and a twin neural network, the method comprising: using a three-dimensional convolutional neural network to extract high-dimensional features in sMRI images of a subject at two time points; using a twin neural network to focus on detecting and identifying differences in sMRI images of a patient taken at different periods, extracting the development of the disease over a long time span and high-dimensional features thereof; extracting features from clinical data through a multilayer perceptron network, and fusing all the high-dimensional features to comprehensively classify Alzheimer's disease; the application combines a three-dimensional convolutional neural network model and a twin neural network model, and uses sMRI images of a subject at different periods and clinical data thereof to comprehensively classify, thereby effectively improving prediction performance and classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and structural magnetic resonance imaging, and more particularly, to an AD classification method and system based on three-dimensional convolution and a twin neural network. BACKGROUND

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease, and research shows that the disease can develop 10 years or even longer before the patient shows clinical symptoms. The most important risk factor for AD is the aging of the body, and changes such as low glucose metabolism, mitochondrial dysfunction, depression, and cognitive decline can occur during the aging process of the human body, and these changes can also occur in normal aging, which makes it difficult to distinguish early AD cases.

[0003] In recent years, brain age estimation driven by neuroimaging has been introduced as an effective biomarker for detecting different diseases and health conditions. Structural Magnetic Resonance Imaging (sMRI) provides a non-invasive and painless method to study the anatomy of the brain and changes in its structure through radio waves and strong magnetic fields, and is widely used for the analysis of progressive cognitive function. sMRI images are used to measure brain morphology, which can visually capture changes in various tissues in the brain of AD patients due to aging and disease, and have become an important means of studying early AD symptoms.

[0004] The prior art discloses a pre-diagnosis method for Alzheimer's disease based on a 3D convolutional neural network, including the following steps: selecting a data set: using the ADNI database, selecting MRI and DTI image data; data preprocessing: preprocessing the selected MRI and DTI image data; CNN-based DL method, the specific steps include: 3D convolution; using the batch normalization technique using a linear rectifier function as the activation function; 3D pooling, establishing a network architecture; implementation and performance evaluation; the scheme in the prior art only uses a 3DCNN model to pre-diagnose AD, but a pure 3DCNN network is difficult to identify subtle differences between images; in addition, the method in the prior art only uses a single image data for prediction tasks, i.e., only focuses on the current condition of the subject, but for chronic diseases, especially early AD, the patient's performance is easily confused with that of normal subjects, and only relying on the sMRI image of the subject currently taken lacks information about the changes in the brain of the subject after a period of time, making it difficult to accurately predict early AD. SUMMARY

[0005] The present application is to overcome the low accuracy of AD disease prediction and classification caused by the prior art of using three-dimensional convolutional neural network and single image for early diagnosis of AD, and provides an AD classification method and system based on three-dimensional convolution and twin neural network, which combines three-dimensional convolutional neural network and twin neural network model, and uses sMRI images of subjects at different times for disease classification, effectively improving the prediction performance and classification accuracy.

[0006] To solve the above technical problems, the technical scheme of the present application is as follows:

[0007] An AD classification method based on three-dimensional convolution and twin neural network, comprising the following steps:

[0008] S1: obtaining sMRI images of a subject at two different time points and pre-processing them respectively;

[0009] S2: obtaining sMRI image masks, and sequentially performing dot product operation and data enhancement operation on the sMRI image masks and the pre-processed sMRI images at the two different time points respectively to obtain first and second sMRI images;

[0010] Subtracting the first sMRI image from the second sMRI image to obtain an sMRI image difference value map;

[0011] S3: inputting the first sMRI image, the second sMRI image and the sMRI image difference value map into a trained three-dimensional convolutional neural network model for feature extraction to obtain a first high-dimensional feature vector;

[0012] S4: inputting the first sMRI image and the second sMRI image into a preset twin neural network model for feature extraction to obtain a second high-dimensional feature vector;

[0013] S5: obtaining clinical data of the subject and inputting it into a preset multi-layer perceptron network for feature extraction to obtain a third high-dimensional feature vector;

[0014] S6: performing feature fusion on the first, second and third high-dimensional feature vectors to obtain an Alzheimer's disease classification result of the subject.

[0015] Preferably, in step S1, the specific method for obtaining sMRI images of a subject at two different time points and pre-processing them respectively is as follows:

[0016] Obtain sMRI images of a subject at two different time points, and use SPM tool in MATLAB to perform registration operation respectively to complete pre-processing.

[0017] Preferably, in the step S2, the specific method for obtaining the sMRI image mask is:

[0018] obtaining a brain image template, extracting ROI regions in which the amygdala and hippocampus in the brain tissue are located in the brain image template, and collectively storing the ROI regions in which the amygdala and hippocampus are located and the peripheral adjacent ROI regions as an ROI region set;

[0019] constructing the sMRI image mask according to all the ROI regions in the ROI region set.

[0020] Preferably, in the step S2, the specific method for obtaining the first sMRI image and the second sMRI image is:

[0021] performing a dot product operation on the sMRI image mask and the two sMRI images at different time points after preprocessing respectively, performing a data enhancement operation on the result of the dot product operation after normalization, and cutting the result after data enhancement to a preset size to obtain the first sMRI image and the second sMRI image;

[0022] The data enhancement operation is specifically adding uniform random noise.

[0023] The preset size is a point set bounding box of the ROI region set.

[0024] Preferably, in the step S3, the structure of the three-dimensional convolutional neural network model is specifically:

[0025] The three-dimensional convolutional neural network model comprises, connected in sequence: an input layer, a VGG convolutional module, three dense modules, a linear module, and an output layer.

[0026] The VGG convolutional module comprises, connected in sequence: a first convolutional layer, a first batch normalization layer, a first activation layer, and a first 3D maximum pooling layer.

[0027] The three dense modules are the same in structure and connected in sequence, and each of the dense modules comprises, connected in sequence: a second convolutional layer, a second batch normalization layer, a second activation layer, and a second 3D maximum pooling layer.

[0028] The input of the second convolutional layer is also connected to the output of the second activation layer in a channel.

[0029] The linear module comprises, connected in sequence: a flattening layer, a linear layer, a third batch normalization layer, and a Dropout layer.

[0030] Preferably, the activation functions of the first activation layer and the second activation layer are both ReLU activation functions.

[0031] Preferably, in the step S4, the structure of the preset twin neural network model is specifically:

[0032] The preset twin neural network model comprises two trained three-dimensional convolutional neural network models which are arranged side by side and have the same structure, and the two trained three-dimensional convolutional neural network models share weights.

[0033] Preferably, in the step S4, the specific method for obtaining the second high-dimensional feature vector is as follows:

[0034] The first sMRI image and the second sMRI image are respectively input into the two trained three-dimensional convolutional neural network models sharing weights, feature extraction is respectively performed, and the output results of the two trained three-dimensional convolutional neural network models sharing weights are subtracted to obtain the second high-dimensional feature vector.

[0035] The application further provides an AD classification system based on three-dimensional convolution and twin neural network, which applies the AD classification method based on three-dimensional convolution and twin neural network.

[0036] The data acquisition unit is used for acquiring sMRI images of a subject at two different time points and performing preprocessing on the sMRI images respectively.

[0037] The data processing unit is used for acquiring an sMRI image mask, and performing dot product operation and data enhancement operation on the sMRI image mask and the sMRI images at the two different time points preprocessed respectively in sequence to obtain the first sMRI image and the second sMRI image.

[0038] The first sMRI image and the second sMRI image are subtracted to obtain an sMRI image difference value graph.

[0039] The first feature extraction unit is used for inputting the first sMRI image, the second sMRI image and the sMRI image difference value graph into a trained three-dimensional convolutional neural network model for feature extraction to obtain a first high-dimensional feature vector.

[0040] The second feature extraction unit is used for inputting the first sMRI image and the second sMRI image into a preset twin neural network model for feature extraction to obtain a second high-dimensional feature vector.

[0041] The third feature extraction unit is used for acquiring clinical data of the subject and inputting the clinical data into a preset multi-layer perceptron network for feature extraction to obtain a third high-dimensional feature vector.

[0042] The feature fusion and classification unit is used for performing feature fusion on the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector to obtain an Alzheimer's disease classification result of the subject.

[0043] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the method in any one of claims 1-8.

[0044] Compared with the prior art, the technical scheme of the application has the beneficial effects that:

[0045] The application provides an AD classification method and system based on three-dimensional convolution and a twin neural network, first, sMRI images of a subject at two different time points are acquired and preprocessed respectively; an sMRI image mask is acquired, and the sMRI image mask is sequentially subjected to dot product operation and data enhancement operation with the preprocessed sMRI images at the two different time points respectively to acquire a first sMRI image and a second sMRI image; the first sMRI image and the second sMRI image are subtracted to acquire an sMRI image difference value graph; the first sMRI image, the second sMRI image and the sMRI image difference value graph are jointly input into a trained three-dimensional convolution neural network model for feature extraction to acquire a first high-dimensional feature vector; the first sMRI image and the second sMRI image are respectively input into a preset twin neural network model for feature extraction to acquire a second high-dimensional feature vector; clinical data of the subject are acquired and input into a preset multilayer perceptron network for feature extraction to acquire a third high-dimensional feature vector; finally, the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector are subjected to feature fusion to acquire an Alzheimer's disease classification result of the subject.

[0046] By using the sMRI images of the subject at two different time periods, supplemented by some clinical information of the subject, the application focuses on the changes in the brain of the subject over a long period of time and comprehensively predicts the changes, so that the prediction performance and classification accuracy can be effectively improved; meanwhile, the application also combines the advantages of the twin neural network framework in image change detection, uses the twin neural network framework to extract high-dimensional features of the sMRI images of the subject at two different time points, and classifies AD by analyzing the differences between the two high-dimensional features, so that the performance is greatly improved compared with the scheme of identifying a single sMRI image by using a traditional 3DCNN network; in addition, the difference value graph of the two images of the subject is additionally added to the input data of the 3DCNN network, so that the knowledge of the image difference value is explicitly input to guide the 3DCNN network to pay more attention to the differences between the images, so that the 3DCNN network can extract more latent high-level abstract features in the images to assist in predicting the AD of the subject, and the prediction performance of the three-dimensional convolution neural network is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A flowchart of an AD classification method based on three-dimensional convolution and a twin neural network provided in Embodiment 1.

[0048] Figure 2 A three-dimensional convolutional neural network model structure diagram provided in Example 2.

[0049] Figure 3 A twin neural network model structure diagram provided in Example 2.

[0050] Figure 4 A three-dimensional convolutional and twin neural network-based AD classification method overall architecture diagram provided in Example 2.

[0051] Figure 5 A three-dimensional convolutional and twin neural network-based AD classification system structure diagram provided in Example 3. DETAILED DESCRIPTION

[0052] The accompanying drawings are only for illustrative purposes and should not be construed as limiting the patent;

[0053] In order to better illustrate the present embodiment, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0054] It is understandable that some well-known structures and their descriptions in the drawings may be omitted for those skilled in the art.

[0055] The technical solutions of the present application will be further described below in conjunction with the drawings and examples.

[0056] Example 1

[0057] As shown in Figure 1 The present embodiment provides a three-dimensional convolutional and twin neural network-based AD classification method, comprising the following steps:

[0058] S1: obtaining sMRI images of a subject at two different time points and pre-processing them respectively;

[0059] S2: obtaining sMRI image masks, and sequentially performing dot product operation and data enhancement operation on the sMRI image masks and the pre-processed sMRI images at the two different time points respectively to obtain first and second sMRI images;

[0060] Subtracting the first sMRI image from the second sMRI image to obtain an sMRI image difference value map;

[0061] S3: inputting the first sMRI image, the second sMRI image and the sMRI image difference value map into the trained three-dimensional convolutional neural network model for feature extraction to obtain a first high-dimensional feature vector;

[0062] S4: input the first sMRI image and the second sMRI image into a preset twin neural network model respectively for feature extraction, and obtain a second high-dimensional feature vector;

[0063] S5: obtain clinical data of the subject and input the clinical data into a preset multi-layer perceptron network for feature extraction, and obtain a third high-dimensional feature vector;

[0064] S6: perform feature fusion on the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector, and obtain an Alzheimer's disease classification result of the subject.

[0065] In the specific implementation process, the existing research is mainly aimed at the sMRI image (i.e. a single sMRI image) taken by the subject in real time. After screening the region of interest (ROI) of the disease in the brain from the image, a neural network model is used to calculate and predict whether the sMRI image shows AD symptoms. In the existing mainstream research, the model is mainly improved to improve the performance of the model in extracting high-dimensional features. However, due to the lack of public data sets for studying AD, the need to protect the privacy of the subject, and other situations, the sMRI image samples available for training the neural network model are few, which makes it difficult to support the training of large neural network models for identifying regular images, and it is prone to overfitting and poor generalization ability. Therefore, additional information needs to be integrated for training.

[0066] In the traditional method, less attention is paid to the sMRI images and clinical data obtained by the subject during regular follow-up visits in the future, i.e. insufficient research on AD over a long period of time. From the data in the public database ADNI and the diagnosis process in the hospital, it can be found that when the subject's early AD symptoms are not obvious (i.e. the subject only has mild cognitive impairment, which is very similar to the aging of normal middle-aged and elderly people), the doctor generally needs the patient to have regular follow-up visits within half a year or a year. The doctor needs to compare the data of the follow-up visits to make a more accurate classification of early AD. Therefore, using a 3DCNN network to extract high-dimensional features of sMRI images and integrating the subject's early sMRI images and follow-up sMRI images in the 3DCNN network can greatly improve the classification accuracy.

[0067] Firstly, the sMRI images of the subject at two different time points are acquired and preprocessed respectively; an sMRI image mask is acquired, and the sMRI image mask is sequentially subjected to dot product operation and data enhancement operation with the preprocessed sMRI images at the two different time points respectively to acquire a first sMRI image and a second sMRI image; the first sMRI image and the second sMRI image are subtracted to acquire an sMRI image difference value; the first sMRI image, the second sMRI image and the sMRI image difference value are input into the trained three-dimensional convolutional neural network model for feature extraction to acquire a first high-dimensional feature vector;

[0068] Since the three-dimensional convolutional neural network (3DCNN) model is mainly suitable for processing conventional image recognition and classification, it also has good performance in identifying subjects with AD disease, but the pure 3DCNN network is difficult to identify the subtle differences between images; the twin neural network model measures the similarity of two inputs through two sub-networks with the same architecture, similar parameters and weights, can distinguish the high-dimensional features of the two from the high-dimensional space of the input data, and has the characteristics of being insensitive to input data, but since the architecture of the twin neural network requires a large amount of data for effective training, the performance of using only the twin neural network architecture to train the prediction of early AD disease is also poor; for the above reasons, the method in the embodiment sets the twin neural network architecture while fusing the 3DCNN network to combine the advantages of the two modules as much as possible and improve the identification of early AD disease in subjects;

[0069] The first sMRI image and the second sMRI image are input into a preset twin neural network model for feature extraction to acquire a second high-dimensional feature vector;

[0070] The sMRI image of the subject mainly shows the structure of each part of the subject's brain; existing medical research shows that patients with early AD disease mainly show cognitive impairment, but the sMRI image of the patient cannot observe the microscopic changes, such as changes in the cell level of some tissues in the patient's brain; at the same time, according to existing research, the performance of a deep learning model using multi-modal data for a prediction task is better than that of using only a single type of input data for prediction; the above two network models mainly extract features from the sMRI image of the subject, and cannot obtain information that the sMRI image cannot provide; therefore, the method additionally sets a multi-layer perceptron (MLP) network, which uses some numerical clinical data obtained during the diagnosis of the subject to obtain high-dimensional features that cannot be obtained in the sMRI image;

[0071] Meanwhile, due to the scalability of the MLP network, the last few linear layers of the MLP network can appropriately fuse the high-dimensional features of the sMRI images obtained by the first two modules and the high-dimensional features obtained by the MLP network in the clinical information, and finally calculate the prediction result, which is specifically:

[0072] The clinical data of the subject is obtained and input into a preset multi-layer perceptron network for feature extraction to obtain a third high-dimensional feature vector. Finally, the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector are fused to obtain the Alzheimer's disease classification result of the subject.

[0073] The method focuses on the changes in the brain of the subject over a long period of time by using two sMRI images of the subject taken at different times and supplemented by some clinical information of the subject, and comprehensively predicts the changes in the brain of the subject over a long period of time. The prediction performance and classification accuracy can be effectively improved. Meanwhile, the method also combines the advantages of the twin neural network framework in image change detection, uses the twin neural network framework to extract high-dimensional features of the sMRI images of the subject taken at two different time points, and classifies AD by analyzing the differences between the two high-dimensional features. Compared with the traditional 3DCNN network scheme for recognizing a single sMRI image, the performance is greatly improved. In addition, the difference image of the two images of the subject is additionally added to the input data of the 3DCNN network, which explicitly inputs the knowledge of the image difference to guide the 3DCNN network to pay more attention to the differences between the images. The 3DCNN network can extract more latent high-level abstract features in the images to assist in predicting the AD disease of the subject, and significantly improve the prediction performance of the three-dimensional convolutional neural network.

[0074] Embodiment 2

[0075] The embodiment provides an AD classification method based on three-dimensional convolution and twin neural network, comprising the following steps:

[0076] S1: Obtain sMRI images of a subject at two different time points and pre-process them respectively;

[0077] S2: Obtain an sMRI image mask, and sequentially perform dot product operation and data enhancement operation on the sMRI image mask and the pre-processed sMRI images at the two different time points respectively to obtain a first sMRI image and a second sMRI image;

[0078] Subtract the first sMRI image and the second sMRI image to obtain an sMRI image difference image;

[0079] S3: Input the first sMRI image, the second sMRI image and the sMRI image difference image into a trained three-dimensional convolutional neural network model for feature extraction to obtain a first high-dimensional feature vector;

[0080] S4: input the first sMRI image and the second sMRI image into a preset twin neural network model respectively for feature extraction, and obtain a second high-dimensional feature vector;

[0081] S5: obtain clinical data of the subject and input the clinical data into a preset multi-layer perceptron network for feature extraction, and obtain a third high-dimensional feature vector;

[0082] S6: perform feature fusion on the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector, and obtain an Alzheimer's disease classification result of the subject;

[0083] In the step S1, the specific method of obtaining the sMRI images of the subject at two different time points and respectively performing preprocessing is as follows:

[0084] Obtain the sMRI images of the subject at two different time points, and perform registration operation by using SPM tool in MATLAB to complete preprocessing;

[0085] In the step S2, the specific method of obtaining the sMRI image mask is as follows:

[0086] Obtain a brain image template, extract ROI regions in which amygdaloid nucleus and hippocampus are located in the brain tissue in the brain image template, and save the ROI regions in which the amygdaloid nucleus and the hippocampus are located and the peripheral adjacent ROI regions as an ROI region set;

[0087] Construct the sMRI image mask according to all the ROI regions in the ROI region set;

[0088] In the step S2, the specific method of obtaining the first sMRI image and the second sMRI image is as follows:

[0089] Perform dot product operation on the sMRI image mask and the sMRI images at the two different time points after preprocessing respectively, normalize the results of the dot product operation, perform data enhancement operation, and cut the results after data enhancement to a preset size to obtain the first sMRI image and the second sMRI image;

[0090] The data enhancement operation is specifically adding uniform random noise;

[0091] The preset size is a point set bounding box of the ROI region set;

[0092] In the step S3, the structure of the three-dimensional convolutional neural network model is as follows:

[0093] As shown in Figure 2 , the three-dimensional convolutional neural network model comprises, which are connected in sequence: an input layer, a VGG convolutional module, three dense modules, a linear module and an output layer;

[0094] The VGG convolution module comprises, connected in sequence: a first convolution layer, a first batch normalization layer, a first activation layer and a first 3D maximum pooling layer;

[0095] The three dense modules are of the same structure and connected in sequence, and each of the dense modules comprises, connected in sequence: a second convolution layer, a second batch normalization layer, a second activation layer and a second 3D maximum pooling layer;

[0096] The input of the second convolution layer is also connected with the output of the second activation layer in a channel connection manner;

[0097] The linear module comprises, connected in sequence: a flattening layer, a linear layer, a third batch normalization layer and a Dropout layer;

[0098] The activation functions of the first activation layer and the second activation layer are both ReLU activation functions;

[0099] In the step S4, the structure of the preset twin neural network model is specifically:

[0100] As shown in Figure 3 The preset twin neural network model comprises: two training completed three-dimensional convolution neural network models which are of the same structure and arranged side by side, and the two training completed three-dimensional convolution neural network models share weights;

[0101] In the step S4, the specific method for obtaining the second high-dimensional feature vector is:

[0102] The first sMRI image and the second sMRI image are respectively input into the two training completed three-dimensional convolution neural network models sharing weights, and feature extraction is respectively performed, and the output results of the two training completed three-dimensional convolution neural network models sharing weights are subtracted to obtain the second high-dimensional feature vector.

[0103] In the specific implementation process, first, the sMRI images of the subject at two different time points are obtained and preprocessed respectively;

[0104] There are many manufacturers of global magnetic resonance equipment, each of which has different standards, so the data extracted from the machine cannot be directly used, and there are many pre-preprocessed data on ADNI, so it is necessary to uniformly select the pre-preprocessed data on ADNI to reduce the influence caused by different processing steps as much as possible;

[0105] The downloaded sample data contains image data and clinical data of AD patients, mild cognitive impairment patients (MCI), normal subjects (HC), etc.; although the ADNI preprocessing process has been unified, considering that there may be slight differences in brain tissue between subjects, such as shape, position deviation, etc., all sMRI images need to be registered and mapped to the same size three-dimensional space, and all tissues in the brain are mapped to the corresponding position according to the template file; only after registration, the data set can be actually used for training the model;

[0106] In this embodiment, the sMRI images of the subjects at two different time points are obtained, and the SPM tool in MATLAB is used for registration operation, the CAT12 segmentation function in the SPM tool is called, the sampling interval of the image is adjusted (the default is 1.5 mm), and the template file of the brain is set, and the batch processing script is generated; finally, the data set is batch processed, and the registered image data can be matched with the previously set brain template file, which facilitates the selection of part of the image data later;

[0107] According to the existing medical research, the development of AD mainly reflects the aging of amygdala (responsible for human emotional function) and hippocampus (responsible for memory function), which is consistent with the clinical symptoms of AD patients, such as memory impairment, behavior disorder, cognitive dysfunction, etc.; therefore, the morphology and change of these tissues can greatly affect or determine the diagnosis of AD, since these tissues only account for a small part of the brain, the tissues responsible for other functions in the brain have no direct connection with AD, if the whole brain image is input into the model, it will not be able to fit the label because there is too much irrelevant data; therefore, the tissues directly related to AD in the image can be extracted, that is, all ROIs involving amygdala and hippocampus are extracted as the input of the model, at the same time, the ROI directly adjacent to these ROIs in the brain has strong correlation in space, and should also be included in the input of the model;

[0108] The model of deep learning usually needs a large number of samples for training, so as to obtain higher prediction performance and generalization, therefore, data enhancement is needed to expand the data set; since the samples used in this embodiment are obtained after the preprocessing process mentioned above, the data in the sMRI image has corresponding spatial position information in space, therefore, the geometric transformation techniques such as rotation, translation and scaling in data enhancement are not suitable for this data set, therefore, the data set is expanded by adding uniform random noise, so that the data set can better simulate the real sample space, and the generalization and robustness of the model are improved;

[0109] The specific data processing and enhancement includes the following steps:

[0110] 1) Obtain the ROI template file of the brain, and query the ROI set S in which the amygdala and hippocampus of the human brain tissue are located;

[0111] 2) Traverse the ROI template file to find all the voxel points belonging to the set S, and make a mask matrix;

[0112] 3) Perform the inflation algorithm on the mask matrix in place, and the updated mask matrix will contain all the voxel points belonging to the set S and a layer of voxel points surrounding the set S. The ROI to which the layer of voxel points belongs is added to the set S, and finally the set S obtained contains all the required ROIs;

[0113] 4) Traverse the ROI template file, and include all the voxel points belonging to the set S into the mask matrix, and update the mask matrix;

[0114] 5) Perform dot product operation on each sample data and the mask matrix to obtain sample data containing only the ROIs in the set S;

[0115] 6) Normalize each sample data, and then perform data augmentation on the data set:

[0116] a) Expand each sample data in the data set to the same 5 parts;

[0117] b) Add uniform random noise to the voxel points belonging to the set S in each sample data, with an amplitude of [-0.08, 0.08];

[0118] 7) Calculate the bounding box of all effective voxel points according to the mask matrix, and then crop the input sample to reduce the input size of the sample;

[0119] The embodiment is based on DenseNet to construct a 3DCNN network, and a full connection layer is used after the 3DCNN network to calculate a prediction result. Unlike ResNet, the network constructed based on DenseNet can retain the feature maps before and after mapping in each Dense Block through the mode of densely connected feature maps, and clearly distinguish the information added to the next layer network and the information retained by the layer. The structure enables the final classifier of the network to make a decision according to all the feature mappings in the network, and maximally avoids the information loss problem caused by the mapping of the feature maps in multiple layers of the network. A great advantage of the network constructed based on DenseNet is to improve the flow of information and gradient in the entire network. Each layer can directly obtain the gradient from the loss function and the original input data, and the densely connected network structure has a regularization effect, which can reduce the risk of overfitting;

[0120] The 3DCNN network structure in the method is shown in the figure, wherein the input sample data contains two sMRI images of the same subject taken at different times; in addition, in order to enable the network to better extract the change features in the two sMRI images and find high-dimensional features that can be used as a judgment of AD, the study specially adds the change difference map of the two sMRI images of the subject in the input sample data, and the network model is trained by explicitly adding change knowledge; therefore, the two sMRI images of the subject and the change difference map are combined into a three-dimensional data of three channels as the input data of the 3DCNN for training; the first layer of the 3DCNN network uses the VGG convolution module to simply map the input data to a high-dimensional space, and then three dense modules are used in succession for feature extraction, and finally the network outputs a high-dimensional feature vector for the use of the following modules;

[0121] The specific steps of the three-dimensional convolutional neural network model for extracting the first high-dimensional feature vector are as follows:

[0122] 1) The input sample data contains two sMRI images X early and X late of the subject at different times; diff = X late -X early , the two images and the difference value map are connected to obtain X0 = [X early , X late , X diff ], at this time X0 is a three-dimensional matrix data containing three channels, and the size of the sample is 60x80x56;

[0123] 2) The VGG convolution module is calculated on X0, wherein the VGG convolution module contains a first convolution layer with a convolution kernel size of 3x3, a first batch normalization layer, a first activation layer, a ReLU activation function, and a first 3D maximum pooling layer, and the output feature map has 32 channels and a size of 30x40x28;

[0124] 3) The 3D dense module is calculated on the feature map output in the previous step, each dense module contains a second convolution layer with a convolution kernel size of 3x3, a second batch normalization layer, a second activation layer, a ReLU activation function, and a second 3D maximum pooling layer; the feature map output by the first layer of the dense module has 64 channels and a size of 15x20x14, the feature map output by the second layer of the dense module has 128 channels and a size of 7x10x7, and the feature map output by the third layer of the dense module has 256 channels and a size of 6x9x6;

[0125] 4) flatten the feature map outputted in the last step, then concatenate to a linear layer; the length of the feature vector outputted by the linear layer is 500, concatenate the third batch of normalization layers, and the Dropout layer with a probability of 0.5 to discard the features, and finally get the first high-dimensional feature vector Y0 with a length of 500;

[0126] Although the difference value map in the sMRI image is added in the input of the 3DCNN network, the 3DCNN network tends to find the pattern of the input data, which means that the high-dimensional feature data extracted by the 3DCNN network only contains part of the features of the image changes of the subjects, and is easily affected by the data distribution of the training set; in other words, the degree of attention of the pattern found in the 3DCNN network to the image changes is affected by the data distribution of the data set; the 3DCNN network has good performance in identifying whether the subject is an AD patient or an MCI patient, but the effect is poor in the identification task of early AD symptoms; this is mainly because the symptoms of early AD patients are similar to the aging process of normal people, that is, the difference between data samples is small, and the 3DCNN network is difficult to find the pattern of the data samples in the high-dimensional feature space;

[0127] Since the 3DCNN network is difficult to directly determine whether the subject is an early AD patient according to a single sMRI image of the subject, doctors often need the subsequent diagnosis of the patient in actual clinical diagnosis, so it is more necessary to pay attention to the changes of the sMRI images of the subject in a certain time span; in the technical field of detecting changes, the twin neural network framework focuses on the high-dimensional feature map of the input image, uses a feature extractor with shared weights to extract features from the image, and performs difference on the extracted high-dimensional feature map to obtain the difference between the images in the high-dimensional space; this approach can better process and calculate the similarity between images, can observe the changes between the two sMRI images of the subject in the high-dimensional space, can be used to detect the changes in the two sMRI images of the subject, and is more suitable for the prediction task of early AD symptoms;

[0128] The backbone network in the twin neural network framework in the method adopts a 3DCNN network with the same structure as described above. The accuracy of the 3DCNN network in the training set can be fitted to (0.95, 1.0), indicating that the parameter amount in the 3DCNN network is sufficient to extract and accommodate the pattern and features in the input image data, and therefore is suitable for being used as a backbone network in the twin neural network framework;

[0129] The specific steps for building the framework of the twin neural network model are as follows:

[0130] 1) Set the input channel number of the two trained three-dimensional convolutional neural network models to 1 channel;

[0131] 2) input two images Xearly and X late are respectively input into two trained three-dimensional convolutional neural network models to obtain Y early and Y late respectively.

[0132] 3) Let Y1=Y late -Y early , the difference Y1 of the two images in the high-dimensional space is the second high-dimensional feature vector.

[0133] In the process of diagnosing early AD symptoms in clinic, doctors often need to diagnose from multiple aspects of clinical information of the subjects. If only the sMRI image data of the subjects is relied on, the diagnosis result will be more one-sided. In the deep learning technology, more information can be integrated by using multi-modal data, so that the neural network can better extract features from different data sources, thereby improving the decision performance of the network. Therefore, in the present method, some clinical data of the subjects is added to the input to participate in the calculation of the network model.

[0134] Since the clinical data of the subjects is some index data about the human body, it cannot be directly fused with the sMRI image data, therefore, the MLP can be used to extract high-dimensional features from the clinical data of the subjects for fusion in the high-dimensional feature space. The clinical data of the subjects is acquired and input into the preset multi-layer perceptron network for feature extraction to obtain the third high-dimensional feature vector Y2.

[0135] The overall architecture of the present method is shown in Figure 4 Finally, the first high-dimensional feature vector Y0, the second high-dimensional feature vector Y1 and the third high-dimensional feature vector Y2 are fused to obtain the classification result of Alzheimer's disease of the subjects.

[0136] The accuracy of the present method is verified by a comparative experiment. In the experiment, the results of using only a single sMRI image of the subjects and using two sMRI images of the subjects at different times are compared in the case of a similar convolutional neural network structure. In the experiment, three different identification tasks are designed: identifying whether the subjects are AD patients (AD vs. HC), judging whether the subjects have mild cognitive impairment (MCIc vs. HC), and judging whether the mild cognitive impairment of the subjects will worsen (MCIc vs. MCInc). The accuracy performance of each experimental scheme in the three identification tasks is shown in Table 1.

[0137] Table 1 Comparison of accuracy of each experimental scheme in three identification tasks

[0138]

[0139] The three experimental tasks from easy to difficult can clearly show the diagnosis of AD disease, and more information of the subject needs to be included to improve the recognition performance of the neural network, especially in the MCIc vs. MCInc task. It can be seen that the recognition of early AD disease cannot rely only on the information provided by the single sMRI image of the subject, but the change information of the sMRI image of the subject over a period of time and the clinical information of the subject need to be included for comprehensive decision-making. In addition, it can also be seen from the ablation experiment that each module in the network model proposed in the present research has its specific function, and the removal of any module will cause performance loss to the network model.

[0140] The present method focuses on the changes in the brain of the subject over a long period of time by using the sMRI images of the subject taken at two different time periods and supplemented by some clinical information of the subject, and makes a comprehensive prediction to effectively improve the prediction performance and classification accuracy. At the same time, the present method also combines the advantages of the twin neural network framework in image change detection, uses the twin neural network framework to extract high-dimensional features of the sMRI images of the subject taken at two different time points, and analyzes the differences between the two high-dimensional features to classify AD. Compared with the traditional 3DCNN network recognition scheme for a single sMRI image, the performance is greatly improved. In addition, the difference image of the two images of the subject is additionally added to the input data of the 3DCNN network, which explicitly inputs the knowledge of the image difference to guide the 3DCNN network to pay more attention to the differences between the images, so that the 3DCNN network can extract more latent high-level abstract features in the images to assist in predicting the AD disease of the subject, and significantly improve the prediction performance of the three-dimensional convolutional neural network.

[0141] Embodiment 3

[0142] As shown in Figure 5 , the present embodiment provides an AD classification system based on three-dimensional convolution and twin neural network, which applies the AD classification method based on three-dimensional convolution and twin neural network described in embodiment 1 or 2, comprising:

[0143] The data acquisition unit 301 is used for acquiring the sMRI images of the subject at two different time points and performing preprocessing respectively;

[0144] The data processing unit 302 is used for acquiring the sMRI image mask, and sequentially performing dot product operation and data enhancement operation on the sMRI image mask and the preprocessed sMRI images at two different time points respectively to obtain the first sMRI image and the second sMRI image;

[0145] The first sMRI image and the second sMRI image are subtracted to obtain the sMRI image difference image;

[0146] The first feature extraction unit 303 is configured to input the first sMRI image, the second sMRI image and the sMRI image difference value map into a trained three-dimensional convolutional neural network model for feature extraction, and obtain a first high-dimensional feature vector.

[0147] The second feature extraction unit 304 is configured to input the first sMRI image and the second sMRI image into a preset twin neural network model for feature extraction, and obtain a second high-dimensional feature vector.

[0148] The third feature extraction unit 305 is configured to obtain clinical data of the subject and input the clinical data into a preset multi-layer perceptron network for feature extraction, and obtain a third high-dimensional feature vector.

[0149] The feature fusion and classification unit 306 is configured to fuse the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector, and obtain an Alzheimer's disease classification result of the subject.

[0150] In the specific implementation process, first, the data acquisition unit 301 obtains sMRI images at two different time points of the subject and respectively performs preprocessing; the data processing unit 302 obtains an sMRI image mask, and sequentially performs dot product operation and data enhancement operation on the sMRI image mask and the preprocessed sMRI images at the two different time points, to obtain a first sMRI image and a second sMRI image; the first sMRI image and the second sMRI image are subtracted to obtain an sMRI image difference value map.

[0151] The first feature extraction unit 303 inputs the first sMRI image, the second sMRI image and the sMRI image difference value map into a trained three-dimensional convolutional neural network model for feature extraction, and obtains a first high-dimensional feature vector.

[0152] The second feature extraction unit 304 inputs the first sMRI image and the second sMRI image into a preset twin neural network model for feature extraction, and obtains a second high-dimensional feature vector.

[0153] The third feature extraction unit 305 obtains clinical data of the subject and inputs the clinical data into a preset multi-layer perceptron network for feature extraction, and obtains a third high-dimensional feature vector.

[0154] Finally, the feature fusion and classification unit 306 fuses the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector, and obtains an Alzheimer's disease classification result of the subject.

[0155] The system can effectively improve the prediction performance and classification accuracy by using two sMRI images of a subject taken at different times, supplemented by some clinical information of the subject, focusing on the changes of the brain of the subject over a long period of time, and comprehensively predicting. Meanwhile, the system also combines the advantages of the twin neural network framework in image change detection, uses the twin neural network framework to extract high-dimensional features of the sMRI images of the subject taken at two different time points, and classifies AD by analyzing the differences between the two high-dimensional features. Compared with the traditional 3DCNN network scheme for recognizing a single sMRI image, the performance is greatly improved. In addition, the difference image of the two images of the subject is additionally added to the input data of the 3DCNN network, and the knowledge of the image difference is explicitly input to guide the 3DCNN network to pay more attention to the differences between the images, so that the 3DCNN network can extract more latent high-level abstract features in the images, assist in predicting the AD disease of the subject, and significantly improve the prediction performance of the three-dimensional convolutional neural network.

[0156] The same or similar reference numerals represent the same or similar components;

[0157] The terms describing the positional relationship in the drawings are only used for illustrative description, and cannot be understood as a limitation to the patent;

[0158] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not a limitation to the embodiments of the present application. For those skilled in the art, on the basis of the above description, other different forms of changes or variations can also be made. Here, all the embodiments need not and cannot be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for AD classification based on three-dimensional convolution and twin neural network, characterized in that, The method comprises the following steps: S1: obtaining sMRI images of a subject at two different time points and respectively pre-processing the sMRI images; S2: obtaining an sMRI image mask, and respectively performing dot product operation and data enhancement operation on the sMRI image mask and the pre-processed sMRI images at the two different time points to obtain a first sMRI image and a second sMRI image; S3: inputting the first sMRI image, the second sMRI image and the sMRI image difference value graph into a trained three-dimensional convolutional neural network model to extract features and obtain a first high-dimensional feature vector; S4: inputting the first sMRI image and the second sMRI image into a preset twin neural network model to extract features and obtain a second high-dimensional feature vector; S5: obtaining clinical data of the subject and inputting the clinical data into a preset multi-layer perceptron network to extract features and obtain a third high-dimensional feature vector; S6: fusing the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector to obtain an Alzheimer's disease classification result of the subject. In the step S1, the specific method of obtaining sMRI images of a subject at two different time points and respectively pre-processing the sMRI images is as follows: 2.The AD classification method based on three-dimensional convolution and twin neural network according to claim 1, wherein, Obtain sMRI images of a subject at two different time points, and use the SPM tool in MATLAB to perform registration operation respectively to complete pre-processing. In the step S2, the specific method of obtaining an sMRI image mask is as follows: 3.The AD classification method based on three-dimensional convolution and twin neural network according to claim 2, characterized in that, Obtain a brain image template, extract ROI regions where amygdalae and hippocampi in brain tissues are located in the brain image template, and save the ROI regions where the amygdalae and the hippocampi are located and the peripheral adjacent ROI regions as an ROI region set; Construct an sMRI image mask according to all the ROI regions in the ROI region set. In the step S2, the specific method of obtaining a first sMRI image and a second sMRI image is as follows:

4. The AD classification method based on three-dimensional convolution and twin neural network according to claim 3, characterized in that, Perform dot product operation on the sMRI image mask and the pre-processed sMRI images at the two different time points respectively, normalize the results of the dot product operation, perform data enhancement operation, and cut the results after data enhancement to a preset size to obtain the first sMRI image and the second sMRI image; The data enhancement operation is specifically adding uniform random noise; The preset size is a point set bounding box of the ROI region set. In the step S3, the structure of the three-dimensional convolutional neural network model is as follows:

5. The AD classification method based on three-dimensional convolution and twin neural network according to claim 4, characterized in that, The three-dimensional convolutional neural network model comprises an input layer, a VGG convolutional module, three dense modules, a linear module and an output layer connected in sequence; The VGG convolutional module comprises a first convolutional layer, a first batch normalization layer, a first activation layer and a first 3D maximum pooling layer connected in sequence; The three dense modules are the same in structure and connected in sequence, and each of the dense modules comprises a second convolutional layer, a second batch normalization layer, a second activation layer and a second 3D maximum pooling layer connected in sequence; The input of the second convolutional layer is also connected to the output of the second activation layer in a channel. ​ The linear module comprises, which are connected in sequence: a flattening layer, a linear layer, a third batch normalization layer and a Dropout layer.

6. The AD classification method based on three-dimensional convolution and twin neural network according to claim 5, characterized in that, The activation functions of the first activation layer and the second activation layer are both ReLU activation functions.

7. The AD classification method based on three-dimensional convolution and twin neural network according to claim 6, characterized in that, In the step S4, the structure of the preset twin neural network model is specifically: The preset twin neural network model comprises: two trained three-dimensional convolutional neural network models which are arranged in parallel and have the same structure, and the two trained three-dimensional convolutional neural network models share weights.

8. The AD classification method based on three-dimensional convolution and twin neural network according to claim 7, characterized in that, In the step S4, the specific method for obtaining the second high-dimensional feature vector is: The first sMRI image and the second sMRI image are respectively input into the two trained three-dimensional convolutional neural network models which share weights, and feature extraction is performed respectively, and the output results of the two trained three-dimensional convolutional neural network models which share weights are subtracted to obtain the second high-dimensional feature vector.

9. A system for AD classification based on 3D convolution and twin neural network, applying the method for AD classification based on 3D convolution and twin neural network in any one of claims 1-8, characterized in that, Comprise: a data acquisition unit configured to acquire sMRI images of a subject at two different time points and perform preprocessing on the sMRI images respectively; a data processing unit configured to acquire sMRI image masks, and perform dot product operation and data enhancement operation on the sMRI image masks and the preprocessed sMRI images at the two different time points respectively to obtain a first sMRI image and a second sMRI image; subtracting the first sMRI image and the second sMRI image to obtain an sMRI image difference value map; a first feature extraction unit configured to input the first sMRI image, the second sMRI image and the sMRI image difference value map into a trained three-dimensional convolutional neural network model for feature extraction to obtain a first high-dimensional feature vector; a second feature extraction unit configured to input the first sMRI image and the second sMRI image into a preset twin neural network model for feature extraction to obtain a second high-dimensional feature vector; a third feature extraction unit configured to acquire clinical data of the subject and input the clinical data into a preset multi-layer perceptron network for feature extraction to obtain a third high-dimensional feature vector; a feature fusion and classification unit configured to fuse the first high-dimensional feature vector, the second high-dimensional feature vector and the third high-dimensional feature vector to obtain an Alzheimer's disease classification result of the subject.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the steps in the method of any one of claims 1-8.

Images