Alzheimer's disease deep joint learning diagnosis method based on multi-modal feature fusion

Abstract

The invention discloses an Alzheimer's disease deep joint learning diagnosis method based on multi-modal feature fusion, and provides a new method for constructing a 3D fusion image through multi-modal feature fusion of an MRI-p value under the condition that a 3D image structure is not damaged. End-to-end deep joint learning diagnosis based on a residual neural network and an attention mechanism is planned to be constructed, and feature information and position information related to AD are automatically learned to obtain a classification basis of diagnosis. Experiments include classification of three levels, respectively, disease diagnosis based on AD and HC, AD and MCI, and MCI and HC. By introducing gene data, feature information of disease diagnosis can be expanded, and more comprehensive feature representation is formed, so that a diagnosis model is promoted to better capture complex features of diseases. And inputting the single-mode MRI into the deep joint learning diagnosis model to carry out an experiment so as to verify the effectiveness of the multi-mode fusion image information.

Description

Technical Field

[0001] The present invention relates to a disease diagnosis technology based on multimodal feature fusion, and in particular to a deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion. Background Art

[0002] In the past two decades, advances in neuroimaging technology have made great contributions to the early diagnosis of Alzheimer's disease (AD). Although magnetic resonance imaging (MRI) is an effective auxiliary detection basis for Alzheimer's disease and other brain diseases, the feature information obtained based on a single image data can only capture the obvious differences in brain structure and function, cannot learn subtle changes in brain images, and is difficult to identify subjects who do not show obvious symptoms in the brain area, which has certain limitations for the early diagnosis of the disease. More importantly, MRI-based AD diagnosis can only identify brain areas related to the disease, and it is difficult to deeply explore the key to the pathogenesis of the disease.

[0003] Genome-wide association studies (GWAS) have shown that genetic variation is the intrinsic cause of abnormal brain function and structure in AD subjects. In the past few decades, multiple GWAS studies have discovered hundreds of single nucleotide polymorphisms (SNPs) associated with AD. Through genetic linkage analysis, SNPs can help locate biomarkers and genetic risk factors associated with AD, and genetic data can provide new targets for the diagnosis of AD. Multimodal feature fusion based on imaging and genetic data mainly faces two problems. On the one hand, traditional fusion methods usually select features for each modality separately, and then connect the selected features for disease diagnosis, which ignores the connection between different modal data; on the other hand, not all subjects have complete multimodal data. In the past, samples with incomplete modalities were often discarded. The small number of available subjects may result in the diagnosis model being unable to be fully trained or overfitting. Therefore, it is still necessary to further explore new multimodal feature fusion methods.

[0004] Deep learning technology, especially convolutional neural network (CNN), has been successfully applied to the diagnosis of AD subjects using MRI images. Compared with traditional computer-aided diagnosis tools, CNN does not rely on predefined, handcrafted features and shows good results in medical imaging analysis tasks. Although CNN for diagnosis has achieved some success, CNN with too deep layers will actually reduce the classification accuracy to a certain extent. Therefore, obtaining the best performance diagnostic model is still an important goal of current research.

[0005] In the existing technology, (1) a single modality is difficult to explore the pathogenesis: Since AD ​​is an irreversible neurological disease, early diagnosis is of great significance for the treatment of AD and the development of potential drugs. MRI-based AD diagnosis can show brain atrophy and other anatomical changes, but these changes are often not obvious until the late stage of the disease, making early diagnosis difficult. Moreover, a single modality cannot deeply explore the genetic markers related to AD progression and pathogenesis and explore the key to AD pathogenesis. Therefore, prior knowledge of genetic data is introduced to form a more comprehensive feature representation.

[0006] (2) Insufficient training of small sample data and overfitting problems: Medical samples have problems such as small sample size and unbalanced sample size of each category, which make it impossible to fully train the diagnostic model. Therefore, by combining multimodal data, the diversity of samples is increased, which helps the model converge faster during training and reduces the risk of overfitting.

[0007] (3) Insufficient in-depth learning of local features: Sufficient learning of features provided by samples is the key issue to be addressed in building a diagnostic model. Therefore, by combining the attention mechanism with ResNet, a new deep joint learning diagnostic model is constructed to more comprehensively extract and learn the features of local pathological areas, thereby improving the accuracy and stability of diagnosis. Summary of the invention

[0008] In view of the problems existing in the prior art, due to the exploration of genetic variation information and the limitations of existing feature fusion methods, the present invention provides a method for mining common potential genetic markers related to AD progression and pathogenesis, and constructs a new feature fusion method to provide a comprehensive feature description, thereby improving the diagnosis of AD. At the same time, in order to promote higher accuracy in early AD diagnosis and identify significant biomarkers, a new deep joint learning diagnosis method is proposed to effectively capture the feature information of the image.

[0009] The purpose of the present invention is achieved through the following technical solutions.

[0010] A deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion, comprising the following steps:

[0011] 1) Imaging and genetic data were provided by the Alzheimer's Disease Neuroimaging Initiative Public Dataset, using samples with complete multimodal data;

[0012] 2) Data preprocessing: MRI scans were preprocessed using voxel-based morphology; the grayscale density images obtained after preprocessing were registered to the anatomical automatic labeling template to obtain brain region coordinate information, and finally the average volume of the left and right hippocampal regions was extracted as phenotypic data;

[0013] Genotyping of genetic data of target subjects using Illumina GWAS arrays and blood genomic DNA samples;

[0014] Obtain complete and valid genetic data: First, you need to screen SNPs and subjects, and perform quality control on genotype data;

[0015] A whole-genome association analysis experiment was conducted based on the target subject's genetic data and phenotypic data to explore genetic biomarkers that may affect brain volume; SNP data with p < 0.05 were screened, and the KGG software was used to perform univariate gene association analysis on the screened data using an effective chi-square test to obtain gene data related to AD; secondly, multiple tests and Bonferroni corrections were performed on the gene data to locate gene data significantly related to AD; finally, gene data with p < 0.001 after correction were selected for subsequent feature fusion operations;

[0016] 3) Experimental setup: The experiment was trained for five rounds, with the total epochs of each round set to 200 and the batch size to 4. Adam was used as the optimizer, and the learning rate was adjusted by learning rate warmup and cosine annealing;

[0017] 4) Pathological region feature information extraction: The proposed framework uses the residual block optimized by ResNet18 to construct this branch structure to obtain the feature information of the local pathological region of the whole brain image; the pathological region feature information extraction module is mainly composed of a convolutional layer, a maximum pooling layer and four residual blocks; they share learning parameters in the spatial dimension; the experiment continuously adjusts the convolution kernel and step size settings of each convolutional layer during the training process to learn the feature information of each pathological region and the distance between each region; the experiment fully considers the effective combination of the result information of the two branches, so the parameters of this branch are adjusted by controlling the receptive field size of the last layer feature map, and finally the feature information of the local pathological region of the whole brain image is obtained;

[0018] 5) The brain region position information capture branch is mainly composed of three building blocks, including the center position extraction module, the relative position information learning module and the discrimination probability generation module;

[0019] 6) Construction and classification of deep joint learning diagnosis model: After the training is completed, the two branches obtain corresponding result information. One branch uses ResNet training to obtain the feature information of the local pathological area of ​​the fusion image, which is recorded as X; the other branch uses attention convolution training to capture the location information of the brain area with different changes in different categories of brain, and finally obtains the position-based discrimination probability information recorded as G; inspired by the aggregation method, the discrimination probability information is used to aggregate the feature information of the local pathological area to define the diagnosis basis.

[0020] The described preprocessing operations of MRI scans using voxel-based morphology first convert the MRI scan format, remove non-brain tissue on the MRI image, normalize, and segment the MRI scan; secondly, the resulting image is aligned with the Montreal Neurological Institute space to eliminate the global linear difference of the image and make the image have the same spatial resolution; finally, the gray matter density image is extracted and smoothed using an 8 mm full-width half-maximum kernel; the obtained image is downsampled from 181×218×181 to 91×109×91 voxels to reduce data processing costs; and the final preprocessed 3D brain scan will be used as the image data input for multimodal feature fusion.

[0021] Six steps were used to control the quality of genotype data, including screening subjects based on heterozygosity, site loss rate, and individual independence, and screening site operations based on site information loss rate, Hardy-Weinberg equilibrium, and linkage disequilibrium.

[0022] Plink was used to perform principal component analysis on the SNP data after quality control, and the obtained eigenvector matrix obtained the first 5 principal components of population stratification analysis. Linear regression was used to perform association analysis on SNP data and phenotypic data, in which covariate information was used to correct for population stratification phenomena, including age, gender, education level, disease labels, and the first 5 principal components obtained from population stratification analysis. Finally, GWAS results containing 563,980 SNPs were obtained.

[0023] Specifically, the step 3) includes increasing the learning rate from 0 to 1e-2 within the first 10 epochs of each round, and then reducing the value to 0 through a cosine annealing learning rate scheduler; the network optimization uses cross entropy as a loss function to alleviate the imbalance problem between positive and negative samples and learn from samples that are difficult to classify; in order to reduce the difference between fused images, the experiment enhances the training data through operations such as image translation, random cropping, and randomly generated cropping starting points; the processed image size is (75, 93, 75); the experiment performs image translation and center cropping on the test data; the position data obtained from the three-dimensional Cartesian space is also converted in the same way as the fused image; the obtained cropped image position data is used for subsequent training.

[0024] The 3D fusion image is constructed by multimodal feature fusion of "MRI-p value". First, the MRI image tensor after preprocessing is recorded as P MRI , the image tensors of different categories are defined as P AD , P MCI , P HC ; Secondly, use the corrected gene p value of p < 0.001 to construct the gene tensor; take the -log of the p value 10 (p) value is recorded as L p, increase the annotation of significant features of gene data; in addition, use L P Same structure as P MRI Tensor G of uniform size p， A total of 35; finally, each G p Applied to P AD , P MCI , P HC , the fusion features S1, S2, and S3 are calculated; the definitions of S1, S2, and S3 composed of fusion features are shown in the formula:

[0025]

[0026] Where S1, S2 and S3 represent the fused feature tensors of the AD-HC group, AD-MCI group and MCI-HC group, respectively. After fusion, the image tensor of each subject is labeled with each gene tensor significantly associated with AD.

[0027] Construct an end-to-end deep joint learning diagnosis based on residual neural network and attention mechanism to automatically learn feature information and location information related to AD to obtain the classification basis for diagnosis. The experiment includes three levels of classification, namely disease diagnosis based on AD and HC, AD and MCI, and MCI and HC.

[0028] Compared with the prior art, the advantages of the present invention are: (1) The proposed "MRI-p value" feature fusion method introduces gene data to expand the feature information of disease diagnosis, successfully supplements the information that cannot be captured by single image information, forms a more comprehensive feature representation, and thus enables the diagnostic model to better capture the complex characteristics of the disease. Single-modality MRI is used as input into the deep joint learning diagnostic model for experiments to verify the effectiveness of multimodal fusion image information. (2) Through data fusion and model optimization, important genes related to Alzheimer's disease (AD) can be identified, including CSMD1, PTPRD, RBFOX1 and WWOX. At the same time, more new genes significantly related to AD are discovered, including NTM, MAML2, NAALADL2, FHIT, TMEM132D and PCSK5. These genes are potential therapeutic targets for neurodegenerative diseases and deserve further functional research in the future. (3) A new deep joint learning diagnostic model is constructed, which introduces the attention mechanism combined with superposition convolution to obtain position information related to disease significance. By fully learning the feature information and position information of the local pathological area, the basis for disease diagnosis is finally obtained through aggregation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the deep joint learning diagnostic information graph of the present invention.

[0030] In the figure: Conv With NKW: convolutional layer with non-parametric kernel weights;

[0031] CLE Block: center position extraction module;

[0032] RPILB Block: relative position information learning module;

[0033] DPG Block: Discriminant probability generation module. DETAILED DESCRIPTION

[0034] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0035] The present invention proposes a new method to construct 3D fusion images by multimodal feature fusion of "MRI-p value" without destroying the 3D image structure. First, the preprocessed MRI image tensor is recorded as P MRI , the image tensors of different categories are defined as P AD , P MCI , P HC . Next, we construct the gene tensor using the corrected p-values ​​of genes with p<0.001. We take the -log of the p-values 10 (p) value is recorded as L p , increase the annotation of significant features of gene data. In addition, L P Same structure as P MRI Tensor G of uniform size p， A total of 35. Finally, each G p Applied to P AD , P MCI , P HC , and the fusion features S1, S2, and S3 are calculated. The definitions of S1, S2, and S3 composed of fusion features are shown in formula (1):

[0036]

[0037] In the formula, S1, S2, and S3 represent the fused feature tensors of the AD-HC group, AD-MCI group, and MCI-HC group, respectively. After fusion, the image tensor of each subject is annotated with each gene tensor significantly associated with AD.

[0038] The present invention constructs an end-to-end deep joint learning diagnosis based on residual neural network (Residual Network, ResNet) and attention mechanism, automatically learning feature information and position information related to AD to obtain the classification basis of diagnosis. The experiment includes three levels of classification, namely disease diagnosis based on AD and HC, AD and MCI, and MCI and HC. It mainly includes two branches. One branch takes the whole brain image after multimodal fusion as input, and uses ResNet training to learn the feature information of the local pathological area of ​​the subject's brain image; the other branch takes the brain image position information obtained based on the three-dimensional Cartesian space as input, and uses attention convolution training to learn the brain area position information of different categories of subjects with different changes, and uses convolution and global average pooling to obtain position-based discrimination probability information. Finally, the local pathological area feature information and discrimination probability information obtained by the two branches are linearly interacted to obtain the diagnostic basis for classifying different categories of subjects.

[0039] 1. Research Materials

[0040] 1.1 Imaging and genetic data

[0041] The data used are provided by the Alzheimer's Disease Neuroimaging Initiative (ADNI) public dataset, which is publicly available on the website (http: / / adni.loni.usc.edu). A sample with complete multimodal data was used, with MRI data and genetic data of 502 subjects, including 180 healthy controls (HC), 198 patients with mild cognitive impairment (MCI) and 125 patients with Alzheimer's disease (AD). The detailed information of these subjects is shown in Table 1.

[0042] Table 1 Statistics of subjects’ information (HC = healthy control; MCI = early mild cognitive impairment; AD = Alzheimer’s disease; M / F = male / female; Edu = education; Sd = standard deviation; p = p value calculated by t-test.)

[0043]

[0044] 1.2 Data Preprocessing

[0045] Voxel-Based Morphometry (VBM) was used to preprocess the MRI scans. First, the MRI scans were converted to different formats, non-brain tissues (such as neck and skull) were removed, normalized, and segmented. Second, the resulting images were aligned with the Montreal Neurological Institute (MNI) space to eliminate the global linear differences in the images and make the images have the same spatial resolution (i.e., 1×1×1mm). 3 ); Finally, the gray matter density (GMD) image was extracted and smoothed using an 8 mm full width at half maximum (FWHM) kernel. In addition, the obtained image was downsampled from 181×218×181 to 91×109×91 voxels to reduce data processing costs. Therefore, the final preprocessed 3D brain scan will be used as the image data input for multimodal feature fusion.

[0046] On the other hand, the grayscale density image obtained after preprocessing was aligned to the anatomical automatic labeling template (AAL) to obtain the brain region coordinate information, and finally the average volume of the left and right hippocampus was extracted as phenotypic data.

[0047] The genetic data of the target subjects were genotyped using Illumina GWAS arrays (610-Quad v1.0, OmniExpress-24Kit or HumanOmni2.5-4v1) (Illumina, Inc., San Diego, CA, USA) and blood genomic DNA samples.

[0048] If you want to obtain complete and valid genetic data, you first need to screen SNPs and subjects. A total of 6 steps are used to control the quality of genotype data. These include screening subjects based on heterozygosity, site loss rate, and individual independence, and screening sites based on site information loss rate, Hardy-Weinberg equilibrium, and linkage disequilibrium. Finally, Plink v1.9 was used to extract SNPs that met the following conditions: (1) Extract SNPs on chromosomes 1-22; (2) The calling rate of each SNP is ≥95%; (3) The estimated value of individual kinship is ≤0.2; (4) The calling rate of each participant is ≥95%; (5) Hardy-Weinberg equilibrium test p≥1×10 -6 ; (6) The minor allele frequency of each SNP was ≥5%. A total of 563,980 SNPs passed the quality control.

[0049] A whole-genome association analysis experiment will be conducted based on the target subject's genetic data and phenotypic data to explore genetic biomarkers that may affect brain volume. Plink was used to perform principal component analysis on the quality-controlled SNP data, and the obtained eigenvector matrix obtained the top 5 principal components of population stratification analysis. Linear regression was used to perform association analysis on SNP data and phenotypic data, in which covariate information was used to correct population stratification, including age, gender, education level, disease labels, and the top 5 principal components obtained from population stratification analysis. Finally, GWAS results containing 563,980 SNPs were obtained.

[0050] SNP data with p < 0.05 were screened, and univariate gene association analysis was performed on the screened data using the effective chi-square test (ECS) using KGG software to obtain gene data related to AD; secondly, multiple tests and Bonferroni corrections were performed on the gene data to locate gene data significantly related to AD; finally, gene data with p < 0.001 after correction were selected for subsequent feature fusion operations.

[0051] (II) Experimental methods

[0052] 2.1 Experimental Setup

[0053] Figure 1 It is a deep joint learning diagnostic information graph, which constructs a deep joint learning diagnostic framework to fully learn the feature information provided by the image. One branch uses the fused image as input and uses the residual neural network to learn the feature information of the local pathological area; the other branch uses the position information of the fused image as input and uses attention convolution to learn the position information of the brain areas with different changes in the brains of subjects of different categories, and obtains the position-based discrimination probability information. Finally, the local pathological area feature information and discrimination probability information obtained by the two branches interact linearly to obtain the diagnostic basis for classifying subjects of different categories. Figure 1As shown in Figure 2, the optimal parameters of the proposed framework are obtained by the optimization function. Five training rounds were conducted, with the total epochs of each round set to 200 and the batch size of 4. Adam was used as the optimizer, and the learning rate was adjusted by learning rate warmup and cosine annealing. Specifically, the learning rate was increased from 0 to 1e-2 within the first 10 epochs of each round, and then the value was reduced to 0 by the cosine annealing learning rate scheduler. The optimization of the network uses cross entropy as the loss function to alleviate the imbalance problem between positive and negative samples and learn from samples that are difficult to classify. In order to reduce the difference between the fused images, the experiment enhances the training data through operations such as image translation, random cropping, and randomly generating cropping starting points. The processed image size is (75,93,75). The experiment performs image translation and center cropping on the test data. In addition, the position data obtained from the three-dimensional Cartesian space is also transformed in the same way as the fused image. The obtained cropped image position data is used for subsequent training.

[0054] 2.2 Pathological region feature information extraction

[0055] The proposed framework uses the residual block optimized by ResNet18 to construct this branch structure and obtain the characteristic information of the local pathological area of ​​the whole brain image. The pathological area feature information extraction module is mainly composed of a convolutional layer, a maximum pooling layer and four residual blocks. They share learning parameters in the spatial dimension. During the training process, the experiment continuously adjusts the settings of the convolution kernel and step size of each convolutional layer to learn the characteristic information of each pathological area and the distance between each area. The experiment fully considers the effective combination of the result information of the two branches, so the parameters of the branch are adjusted by controlling the receptive field size of the last layer feature map, and finally the characteristic information of the local pathological area of ​​the whole brain image is obtained.

[0056] Specifically, the feature information of the fused image is represented as: X∈R W×H×D×1 In the first part of the branch, the kernel and stride size of the convolution layer are set to (5, 5, 5) and (2, 2, 2) respectively, and the normalization layer and nonlinear activation function use instance normalization layer and rectified linear unit (ReLU) respectively. Secondly, the kernel and stride size of the maximum pooling layer are set to (3, 3, 3) and (2, 2, 2) respectively. The size of the feature information obtained through this part is (32, 19, 24, 19).

[0057] The second part uses four residual blocks to train hierarchical learning of richer feature information. The number of the four residual output feature maps is 32, 64, 128 and 256 respectively. The kernel and step size of the first convolution layer passed by each branch are [3, 3, 3, 3] and [1, 2, 1, 1] respectively. The normalization layer and activation function used in the residual block are normalization layer and ReLU respectively. The kernel and step size of other convolution layers are set to (1, 1, 1) and (1, 1, 1) respectively. After this training, the feature information size of the image is (256, 10, 12, 10). Under the premise of fixing the feature information size, the complexity of the feature information is increased to obtain more accurate feature information of the pathological area. After residual block training, f feature information of size (w, h, d) can be obtained. The f-dimensional vector is converted into a local feature based on image information through 1×1×1 convolution to obtain the feature information X∈R of the local pathological area. w×h×d×1 .

[0058] 2.3 Capturing brain region location information

[0059] The brain region position information capture branch is mainly composed of three building blocks, including the center position extraction module (CLEBlock), the relative position information learning module (RPILB Block) and the discrimination probability generation module (DPG Block).

[0060] In terms of the transformation of image coordinate information, the image coordinate information of different transformations is connected in the form of channels. Specifically, since all 3D images have been aligned with the same MNI space and the position information of the images has not been changed during the multimodal fusion construction process, the 3D fused images registered to the same template share a three-dimensional Cartesian space. Based on this space, the coordinate information of the fused image in the coronal, sagittal, and axial planes is obtained to obtain three tensors I∈R W×H×D×1 ; Secondly, use different channels to record the coordinates under different transformations to obtain a tensor I∈R W×H×D×3 , obtain the coordinate information that can fully provide the three-dimensional Cartesian space position of the fused image. This will serve as the input of the CLE module.

[0061] In the CLE module, a multi-layer convolution hierarchy is used to extract the center position of the pathological area. Each layer uses a deep convolution with a non-parametric kernel weight, where the kernel weight is set to 0 and the center position weight is set to 1. The size of the center position information finally obtained is (3, 10, 12, 10), which will be used as the input of the relative position information learning module.

[0062] In the RPILB module, the introduction of the attention module can learn the key areas through the weight matrix of the entire image. The attention mechanism can assign different attention weights to inputs at different positions, thereby accurately capturing the relevant information of each position. Although this avoids the interference caused by some unnecessary information, it is also easy to cause global information loss. Therefore, superimposed convolution layers are used to increase the number of feature maps and comprehensively detect the location information of different categories of brain differences. The combination of superimposed convolution and attention mechanism obtains location information related to disease significance from both comprehensive and local aspects, providing data support for subsequent training.

[0063] In the attention mechanism, a two-branch attention module is used to generate the attention map

[34] , capturing the location information of local changes in the brain. The information I∈R W×H×D×d As the input of the attention module, one branch generates an attention map W in the spatial domain. s (I)∈R W×H×D×1 , see formula (2):

[0064] W s (I)=BN(Conv3(Conv2(Conv1(I)))) (2)

[0065] Where Conv1 and Conv2 represent 1×1×1 convolutional layers and 3×3×3 convolutional layers, respectively, and the number of channels is C / r (r is the reduction rate); after that, the feature information in the spatial domain is obtained; Conv3 represents a 1×1×1 convolutional layer with 1 channel, which is used to generate attention maps in the spatial domain; BN represents a batch normalization (BN) layer to ensure that the result information of the two independent attention branches is at the same scale; the other branch is used to generate the attention map in the channel domain W c (I)∈R 1×1×1×C Generate an attention map in (3):

[0066] W c (I)=BN(FCLayer2(FCLayer1(GlobalAvgPool(I)))) (3)

[0067] In the formula, GlobalAvgPool represents the global average pooling (GAP) layer for spatial domain feature aggregation; FCLayer1 and FCLayer2 represent two FC layers for feature perception and attention map generation in the channel domain, and their channel numbers are C / r and C respectively. In the spatial domain and channel domain, the normalization layer and activation function used are batch normalization layer and ReLU respectively, and the reduction rate r is set to 4. In addition, the attention maps generated by the two branches are combined using the Softmax function on the last dimension to obtain the final attention map W(I)∈R W×H×D×C , see formula (4):

[0068] W(I)=Softmax(W s (I)+W c (I)) (4)

[0069] W(I) is used as the input of the discriminant probability generation module for further training. In the discriminant probability generation module, the number of output feature maps is first reduced by stacking convolutions, and the extracted location information that is obviously related to the disease is encoded; secondly, the selected result information is normalized using the sigmoid activation function after GAP; finally, a discriminant probability map based on location information is generated. where g i，j，k ∈[0, 1] represents the probability that the brain image (i, j, k) region is related to AD.

[0070] 2.4 Construction and classification of deep joint learning diagnosis model

[0071] After the training is completed, the two branches obtain the corresponding result information. One branch uses ResNet training to obtain the feature information of the local pathological area of ​​the fusion image, which is recorded as X; the other branch uses attention convolution training to capture the location information of the brain area with different changes in different categories of brain, and finally obtains the position-based discriminant probability information recorded as G. Inspired by the aggregation method, the discriminant probability information is used to aggregate the feature information of the local pathological area to define the diagnosis basis. Specifically, the feature information X∈R w×h×d×1 and the discriminant probability information G∈R w×h×d×1 The element-by-element multiplication of obtains the disease classification information E∈R w×h×d×1 Since the total amount of brain regions to be identified is unknown, it is necessary to perform a normalization operation based on the number and probability of identification information, as defined in formula (5):

[0072]

[0073] In the formula, e i,j,k =g i,j,k x i,j,k , the y obtained after the normalization operation is directly used to determine the classification of the disease.

[0074] Without destroying the 3D image structure, the present invention proposes to introduce genetic data by using the "MRI-p value" feature fusion method to expand the characteristic information of disease diagnosis, successfully supplement the information that cannot be captured by single image information, and form a more comprehensive feature representation, thereby prompting the diagnostic model to better capture the complex characteristics of the disease. Experiments were conducted using single-modality MRI input into the deep joint learning diagnostic model to verify the effectiveness of multimodal fusion image information. Table 2 shows the classification performance of three groups of diagnostic tasks. The results show that the diagnostic performance based on multimodal fusion features is superior to the diagnosis that only uses MRI as input in all aspects. Specifically, the accuracy of the AD-HC group, AD-MCI group, and MCI-HC group increased by 6.55%, 6.25%, and 5.33%, respectively. This finding shows that the proposed data fusion method fully considers the intrinsic relationship between image data and genetic information, and effectively enhances the differences between different data groups. In addition, the proposed data fusion method enriches the data set, providing a diverse data source and a more comprehensive feature representation for the training of the diagnostic model.

[0075] Table 2 Multimodal feature fusion comparison test results

[0076]

[0077] The present invention can identify important genes related to Alzheimer's disease (AD), including CSMD1, PTPRD, RBFOX1 and WWOX, through data fusion and model optimization. At the same time, more new genes significantly related to AD were found, including NTM, MAML2, NAALADL2, FHIT, TMEM132D and PCSK5. These genes are potential therapeutic targets for neurodegenerative diseases and deserve further functional studies in the future. There are 35 significant gene values ​​with p<0.001, and the significant gene information is listed in Table 3.

[0078] Table 3 Gene data significantly associated with AD

[0079]

[0080] The present invention constructs a new deep joint learning diagnosis model, which introduces the attention mechanism and combines it with superposition convolution to obtain the location information related to the significance of the disease. By fully learning the feature information and location information of the local pathological area, the basis for disease diagnosis is finally obtained by aggregation. We evaluated the method using subjects downloaded from the ADNI dataset. The accuracy (ACC) and area under the curve (AUC) of the proposed model in the diagnosis of healthy controls (HC) and Alzheimer's disease (AD) were 93.44% and 96.67%, respectively, the accuracy and AUC in the diagnosis of AD and mild cognitive impairment (MCI) were 89.06% and 92%, and the accuracy and AUC in the HC and MCI diagnosis tasks were 84% and 81.84%. Verify the effectiveness of the attention module in the attention convolution subnetwork for the deep joint diagnosis model. The attention module consists of two branches: spatial attention and channel attention. Therefore, the test is divided into four parts: model diagnosis without attention, model diagnosis with only channel attention, model diagnosis with only spatial attention, and model diagnosis with channel and spatial attention combined, which are recorded as Baseline, ACS-Channel, ACS-Spatial, and ACS-Combined, respectively. Table 4 shows the performance of the four groups of experiments on AD and HC, AD and MCI, and MCI and HC diagnosis. The results show that the diagnostic performance of the model with spatial attention and channel attention is further improved. Since spatial attention can aggregate more feature information, spatial attention has better classification performance than channel attention. However, channel-spatial mixed attention can combine channel and spatial attention to further improve the classification performance. Therefore, it can be obtained that the diagnosis results of the model without attention and the model with channel-spatial mixed attention are significantly different (8.19% for AD-HC group, 9.37% for AD-MCI group, and 6.67% for MCI-HC group).

[0081] Table 4 Attention module effectiveness verification results

[0082]

Claims

1. A deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion, characterized by the steps include: 1) Imaging and genetic data were provided by the Alzheimer's Disease Neuroimaging Initiative Public Dataset, using samples with complete multimodal data; 2) Data preprocessing: MRI scans are preprocessed using voxel-based morphology; The grayscale density image obtained after preprocessing was registered to the anatomical automatic labeling template to obtain the coordinate information of the brain region, and finally the average volume of the left and right hippocampus was extracted as the phenotypic data; The genetic data of the target subjects were genotyped using Illumina GWAS arrays and blood genomic DNA samples; complete and valid genetic data were obtained: first, SNPs and subjects were screened, and the genotype data were quality controlled; a whole-genome association analysis experiment was conducted based on the genetic data and phenotypic data of the target subjects to explore genetic biomarkers that may affect the volume of brain regions; SNP data with p < 0.05 were screened, and the univariate gene association analysis was performed on the screened data using the effective chi-square test through KGG software to obtain gene data related to AD; secondly, multiple tests and Bonferroni corrections were performed on the gene data to locate the gene data significantly related to AD; finally, the gene data with p < 0.001 after correction were selected for subsequent feature fusion operations; 3) Experimental setup: The experiment was trained for five rounds, with the total epochs of each round set to 200 and the batch size to 4. Adam was used as the optimizer, and the learning rate was adjusted by learning rate warmup and cosine annealing. 4) Pathological region feature information extraction: The proposed framework uses the residual block optimized by ResNet18 to construct this branch structure to obtain the feature information of the local pathological region of the whole brain image; the pathological region feature information extraction module is mainly composed of a convolutional layer, a maximum pooling layer and four residual blocks; they share learning parameters in the spatial dimension; the experiment continuously adjusts the convolution kernel and step size settings of each convolutional layer during the training process to learn the feature information of each pathological region and the distance between each region; the experiment fully considers the effective combination of the result information of the two branches, so the parameters of this branch are adjusted by controlling the receptive field size of the last layer feature map, and finally the feature information of the local pathological region of the whole brain image is obtained; 5) The brain region position information capture branch is mainly composed of three building blocks, including the center position extraction module, the relative position information learning module and the discrimination probability generation module; 6) Construction and classification of deep joint learning diagnosis model: After the training is completed, the two branches obtain corresponding result information respectively; one branch uses ResNet training to obtain the feature information of the local pathological area of ​​the fused image, which is denoted as X; the other branch uses attention convolution training to capture the location information of brain regions with different changes in different categories of brain, and finally obtains the position-based discrimination probability information, which is denoted as G; inspired by the aggregation method, the discrimination probability information is used to aggregate the feature information of the local pathological area to define the diagnosis basis.

2. According to claim 1, a deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion is characterized in that The described preprocessing operations of MRI scans using voxel-based morphology first convert the MRI scan format, remove non-brain tissue on the MRI image, normalize, and segment the MRI scan; secondly, the resulting image is aligned with the Montreal Neurological Institute space to eliminate the global linear difference of the image and make the image have the same spatial resolution; finally, the gray matter density image is extracted and smoothed using an 8 mm full-width half-maximum kernel; the obtained image is downsampled from 181×218×181 to 91×109×91 voxels to reduce data processing costs; and the final preprocessed 3D brain scan will be used as the image data input for multimodal feature fusion.

3. According to claim 1, a deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion is characterized in that Six steps were used to control the quality of genotype data, including screening subjects based on heterozygosity, site loss rate, and individual independence, and screening site operations based on site information loss rate, Hardy-Weinberg equilibrium, and linkage disequilibrium.

4. The deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion according to claim 1 is characterized in that Plink was used to perform principal component analysis on the SNP data after quality control, and the obtained eigenvector matrix obtained the first 5 principal components of population stratification analysis. Linear regression was used to perform association analysis on SNP data and phenotypic data, in which covariate information was used to correct for population stratification phenomena, including age, gender, education level, disease labels, and the first 5 principal components obtained from population stratification analysis. Finally, GWAS results containing 563,980 SNPs were obtained.

5. The deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion according to claim 1 is characterized in that Specifically, the learning rate of step 3) is increased from 0 to 1e-2 within the first 10 epochs of each round, and then reduced to 0 through the cosine annealing learning rate scheduler; the network optimization uses cross entropy as the loss function to alleviate the imbalance problem between positive and negative samples and learn from samples that are difficult to classify; in order to reduce the difference between fused images, the experiment enhances the training data through operations such as image translation, random cropping, and randomly generated cropping starting points; the processed image size is (75,93,75); the experiment performs image translation and center cropping on the test data; The position data obtained from the 3D Cartesian space were also transformed in the same way as the fused image; the obtained cropped image position data were used for subsequent training.

6. The deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion according to claim 1 is characterized in that The 3D fusion image is constructed by multimodal feature fusion of "MRI-p value". First, the MRI image tensor after preprocessing is recorded as P MRI , the image tensors of different categories are defined as P AD , P MCI , P HC ; Secondly, use the corrected gene p value of p < 0.001 to construct the gene tensor; take the -log of the p value 10 (p) value is recorded as L p , increase the annotation of significant features of gene data; in addition, use L P Same structure as P MRI Tensor G of uniform size p， A total of 35; finally, each G p Applied to P AD , P MCI , P HC , the fusion features S1, S2, and S3 are calculated; the definitions of S1, S2, and S3 composed of fusion features are shown in the formula: Where S1, S2, and S3 represent the fused feature tensors of the AD-HC group, AD-MCI group, and MCI-HC, respectively; After fusion, the image tensor of each subject was annotated with each gene tensor significantly associated with AD.

7. The deep joint learning diagnosis method for Alzheimer's disease based on multimodal feature fusion according to claim 1 is characterized in that Construct an end-to-end deep joint learning diagnosis based on residual neural network and attention mechanism to automatically learn feature information and location information related to AD to obtain the classification basis for diagnosis; the experiment includes three levels of classification, namely disease diagnosis based on AD and HC, AD and MCI, and MCI and HC.

Images