A multimodal Alzheimer's disease pathological image classification method

Abstract

The present invention discloses a multimodal Alzheimer's disease pathology image classification method, system, and computer-readable storage medium, comprising the following steps: S1: selecting samples with both MRI and PET images, and training a three-dimensional cycle-consistent generative adversarial network based on an adversarial generative training strategy; S2: selecting samples with only MRI images but missing PET images, and generating the missing PET images using the three-dimensional cycle-consistent generative adversarial network; S3: using the generated PET images to complete the missing PET images in the dataset, thereby obtaining a complete MRI and PET dataset; S4: inputting the complete dataset into a deformable self-attention mechanism module to extract multimodal image features under different modalities; S5: inputting the multimodal image features into a multimodal feature fusion network to extract and generate a fused feature map; S6: inputting the fused feature map into a classifier to output the classification result. The present invention can accurately locate the pathological area in the pathological image, thereby improving the classification accuracy of the pathological image.

Description

Technical Field

[0001] The present invention relates to the field of bioinformatics, and more specifically, to a multimodal Alzheimer's disease pathology image classification method, system, and computer-readable storage medium. Background Art

[0002] Alzheimer's disease (AD) is one of the most severe neurological disorders in the elderly, characterized by progressive memory impairment, such as memory loss and cognitive deficits. The primary reason AD is incurable and irreversible is that no pharmacological treatment has been proven sufficiently effective in reversing its progression. Therefore, early detection of mild cognitive impairment (MCI), the prodromal stage of AD, is of great clinical significance for implementing interventions and treatments to slow the progression to AD. Because AD-related brain structural changes appear earlier and more prominently than the visible symptoms of forgetfulness, MRI and PET are effective biomarkers for predicting AD progression and the transition to mild cognitive impairment. Structural magnetic resonance imaging (sMRI), an imaging technique that measures brain tissue structure, can quantify morphological abnormalities, such as gray and white matter. Positron emission tomography (PET) can measure metabolic processes in the brain. These different medical imaging techniques measure the same brain region from different perspectives and may carry complementary information. Therefore, integrating medical images from different modalities can help models make more accurate judgments.

[0003] In recent years, many studies based on multimodal diagnosis have made significant progress. Existing studies usually segment the entire image into multiple regions of different scales. Based on the scale division, existing research can be roughly divided into three research directions: voxel-based methods, ROI-based methods, and image block-based methods. However, due to problems such as time cost and economic cost in clinical practice, it is often difficult or impossible to obtain all modalities of medical images of the patient. In this case, cross-modal medical image reconstruction is a novel and effective solution. In addition, due to the black box characteristics of neural networks, few deep learning methods have specific outputs for pathological locations, which ignores the issue of interpretability in medical practice.

[0004] The prior art discloses a method for detecting Alzheimer's lesion areas based on a convolutional neural network, which divides the lesion detection task into three steps: feature extraction, lesion localization, and lesion classification. The method involves training an improved AlexNet neural network model to extract features from original brain MRI images and generate a feature map; then training a region generation network model to generate a proposal for locating the lesion area in the brain MRI image; and then feeding the feature map and proposal into a subsequent classification network for classification training to classify the stage of the disease. This solution does not combine MRI and PET imaging and cannot accurately locate the pathological area in the pathological image. Summary of the Invention

[0005] The present invention provides a multimodal Alzheimer's disease pathology image classification method, system and computer-readable storage medium, which can accurately locate the pathological area of the pathological image and improve the classification accuracy of the pathological image.

[0006] The primary purpose of the present invention is to solve the above technical problems, and the technical solutions of the present invention are as follows:

[0007] A first aspect of the present invention provides a multimodal Alzheimer's disease pathology image classification method, characterized by comprising the following steps:

[0008] S1: Select samples with both MRI and PET images and train a 3D cycle-consistent generative adversarial network based on the adversarial generative training strategy;

[0009] S2: Select samples that only have MRI but lack PET images, and generate the missing PET images through the 3D cyclic consistent generative adversarial network;

[0010] S3: Use the generated PET images to complete the missing PET images in the dataset to obtain a complete MRI and PET dataset;

[0011] S4: The complete MRI and PET datasets are fed into the deformable self-attention mechanism module to extract multimodal imaging features under different modalities.

[0012] S5: Input the multimodal image features into the multimodal feature fusion network to extract the potential complementary features between MRI and PET and generate a fusion feature map;

[0013] S6: Input the fused feature map into the classifier to output the classification result.

[0014] Furthermore, a deformable self-attention mechanism is used to adaptively adjust the scale and position of each image patch in the selected dataset and locate pathological areas with discriminative features.

[0015] Furthermore, in step S6, the fused feature map is compressed into a one-dimensional vector through a global average pooling layer in the classifier, and then a full convolutional layer with a Sigmund function is used to perform convolution operation and activation on the feature map to output the Alzheimer's disease pathology image classification result.

[0016] Furthermore, the three-dimensional cycle-consistent generative adversarial network includes: two generator networks and two discriminator networks, respectively denoted as: G1, G2, D1, D2. Each generator includes an encoder and a decoder. The generator consists of 9 residual blocks, 5 3x3x3 convolutional layers, 4 downsampling layers and 4 upsampling layers. Each residual block includes three directly connected convolutional layers. The convolution kernel size of each convolutional layer is 3×3×3, and ReLU is selected as the activation function. In the downsampling layer, the size of the output feature map is halved and the number of channels is doubled. In the upsampling layer, the size of the output feature map is expanded through the deconvolution layer, and the feature map of the sub-network is connected to the feature map through skip connections.

[0017] The discriminator contains 5 Conv layers with 16, 32, 64, 128 and 1 channels respectively. The discriminator inputs a pair of real and synthetic images and outputs a binary indicator.

[0018] Furthermore, a 3D cycle-consistent generative adversarial network uses cycle-consistent loss to learn a bidirectional mapping between MRI and PET, aiming to ensure the interactive relationship between the two modalities.

[0019] Furthermore, the 3D cycle-consistent generative adversarial network contains two complementary loss functions, including the adversarial loss function L g , cycle-consistent loss function L c , as follows:

[0020] L g (X MRI ,X PET ; G1, G2, D1, D2) = log (D2 (X PET ))+log(1-D2(G1(X MRI )))+log(D1(X MRI ))+log(1-D1(G2(X PET )))

[0021] L c (X MRI ,X PET ;G1,G2)=||G2(G1(X MRI ))-X PET ||1+||G1(G2(X PET ))-X PET ||1

[0022] Among them, X MRI Indicates MRI images, X PET Represents PET image, anti-loss function L g To ensure that the generated images are in principle indistinguishable from real images, the cycle-consistent loss function L c The interactive relationship between MRI images and PET images is guaranteed.

[0023] Furthermore, the processing process of the deformable self-attention mechanism module is as follows:

[0024] Set the input to be a single MRI modality image X MRI ∈R H×W×D×C , where H represents the length of the image, W represents the width of the image, D represents the depth of the image, C represents the characteristic dimension of the image, and X MRI First, N patches of the same size and position are divided into a series of image blocks through the patch embedding module of the linear layer, and the sizes of H, W, and D are set to be equal. The size of the embedded image block is s×s×s, where The entire image block sequence is in X MRI The above is considered as a uniform grid of size s×s×s, and the series of image blocks is represented as {X i} (0≤i≤N-1) , for each image block X i , and its left front coordinate and right back coordinate are expressed as and in:

[0025]

[0026] Afterwards, each image block X i The offset prediction network predicts a position offset and a size scaling amount. The position offset and size scaling can be represented by parameters Δposition=(Δx, Δy, Δz) and Δscale=(Δh, Δw, Δd). For each deformed image block The updated left front coordinate and right back coordinate are expressed as and in:

[0027]

[0028]

[0029]

[0030] Since the size of each deformed image block is different, the sampling and interpolation method is used to extract features, and a uniform grid of size k×k×k is sampled in each image block. Since the coordinates of the grid are usually fractional, p j The value of is calculated by trilinear interpolation of 8 adjacent points C j The weighted average of is obtained, the specific formula is as follows:

[0031] p j =H Trilinear (C j ,(x j ,y j ,z j ))(j∈(0,k 3 -1))

[0032]

[0033] Afterwards, the deformed image block Projected into a feature vector through a d-dimensional linear layer and embedded into a flattened feature map Then, through a standard multi-head self-attention module As input, it generates three sets of learnable feature vectors, q∈R N×d , k∈R N×d 、v∈R N×d ;W q ∈R N×d 、W k ∈R N×d 、W v ∈R N×d are three learnable linear embedding matrices, and the specific formulas are as follows:

[0034]

[0035]

[0036] Y=Concat{Atten1v1,...,Atten M v M}W o

[0037] in d m =d / M, Atten m Represents the similarity matrix between different image blocks, which is expressed by q m and k m Multiply and get v m The weighted weights are obtained by adding the M attention matrices Atten m The final feature Y∈R is obtained by concatenating the transformation N×d .

[0038] Furthermore, the processing process of the multimodal feature fusion network is:

[0039] Set the input of the multimodal feature fusion network to: MRI ∈R N×d 、Y PET ∈R N×d , where d is the feature dimension and N is the number of image blocks, where N = k × k × k;

[0040] The feature maps of the two modalities are concatenated in the d dimension and the dimension is adjusted to Y∈R 2d×k×k×k ;

[0041] Set Y = {Y1, Y2, ..., Y 2d}, where Y i ∈R k×k×k , the maximum pooling along the channel axis is expressed as:

[0042] Y max =ChannelMaxPooling(Y)

[0043] in The average pooling along the channel axis is expressed as:

[0044] Y mean =ChannelAvreagePooling(Y)

[0045] in These two feature maps are then concatenated to compute a spatial attention map:

[0046] A spatial =σ(W([Y max ; Y mean ]))

[0047] Where σ is the sigmoid activation function, W is the convolutional layer weight, [;] is the concatenation operation, and the final multimodal attention feature map F is expressed as:

[0048]

[0049] in, Represents element-wise multiplication.

[0050] A second aspect of the present invention provides a multimodal Alzheimer's disease pathology image classification system, comprising: a memory and a processor, wherein the memory includes a multimodal Alzheimer's disease pathology image classification method program, and when the multimodal Alzheimer's disease pathology image classification method program is executed by the processor, the following steps are implemented:

[0051] S1: Select samples with both MRI and PET images and train a 3D cycle-consistent generative adversarial network based on the adversarial generative training strategy;

[0052] S2: Select samples that only have MRI but lack PET images, and generate the missing PET images through the 3D cyclic consistent generative adversarial network;

[0053] S3: Use the generated PET images to complete the missing PET images in the dataset to obtain a complete MRI and PET dataset;

[0054] S4: The complete MRI and PET datasets are fed into the deformable self-attention mechanism module to extract multimodal imaging features under different modalities.

[0055] S5: Input the multimodal image features into the multimodal feature fusion network to extract the potential complementary features between MRI and PET and generate a fusion feature map;

[0056] S6: Input the fused feature map into the classifier to output the classification result.

[0057] The third aspect of the present invention provides a computer-readable storage medium, which includes a multimodal Alzheimer's disease pathology image classification method program. When the multimodal Alzheimer's disease pathology image classification method program is executed by a processor, it implements the steps of a multimodal Alzheimer's disease pathology image classification method described in the item.

[0058] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0059] The present invention effectively utilizes image context and latent vectors through a three-dimensional cyclic consistent generative adversarial network to generate missing PET images from MRI images; utilizes a deformable self-attention mechanism module to adaptively adjust the size and position of each selected image block and locate pathological areas with identifying features; and utilizes a multimodal feature fusion network to adaptively extract the potential complementary features of MRI and PET, thereby improving the classification accuracy of pathological images. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flow chart of a multimodal Alzheimer's disease pathology image classification method of the present invention.

[0061] Figure 2 A schematic diagram of a three-dimensional cycle-consistent generative adversarial network according to an embodiment of the present invention.

[0062] Figure 3 Schematic diagram of the generator of a three-dimensional cycle-consistent generative adversarial network according to an embodiment of the present invention.

[0063] Figure 4 Schematic diagram of the discriminator of the three-dimensional cycle-consistent generative adversarial network according to an embodiment of the present invention.

[0064] Figure 5 This is a network diagram of the deformable self-attention mechanism module according to an embodiment of the present invention.

[0065] Figure 6 Schematic diagram of a multimodal feature fusion network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0067] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0068] Example 1

[0069] like Figure 1 As shown, the first aspect of the present invention provides a multimodal Alzheimer's disease pathological image classification method, characterized by comprising the following steps:

[0070] S1: Select samples with both MRI and PET images and train a 3D cycle-consistent generative adversarial network based on the adversarial generative training strategy;

[0071] It should be noted that if Figure 2 As shown, in order to bridge the gap between the generated PET images and the real PET images, the present invention proposes an end-to-end three-dimensional generative adversarial network with a cycle-consistent mechanism. This method not only focuses on synthesizing realistic PET images from MR images, but also focuses on reflecting the different brain properties of different subjects. The three-dimensional cycle-consistent generative adversarial network includes: two generator networks and two discriminator networks, respectively denoted as: G1, G2, D1, D2. In a specific embodiment, a 3DU-Net network structure can be selected and a residual module can be introduced as a generator network, named 3D-ResUNet generator. In order to more intuitively demonstrate the operation process of the three-dimensional cycle-consistent generative adversarial network, MRI images and PET images are respectively denoted as X MRI and X PET Based on the potential correlation between MRI and PET, consider learning the generator G1:X MRI →X PET , generate missing PET images for MRI scanned subjects (ie samples). At the same time, it is required that G is a one-to-one mapping, that is, there exists a generator G2:X PET →XMRI , to maintain the consistency of the mapping. A cycle-consistent loss is used to learn the bidirectional mapping between MRI and PET, aiming to ensure the interactive relationship between the two modalities.

[0072] More specifically, Figure 3 As shown, each generator includes an encoder and a decoder. The generator consists of 9 residual blocks, 5 3x3x3 convolutional layers, 4 downsampling layers, and 4 upsampling layers. Each residual block includes three directly connected convolutional layers. The convolution kernel size of each convolutional layer is 3×3×3, and ReLU is selected as the activation function. In the downsampling layer, the size of the output feature map is halved and the number of channels is doubled. In the upsampling layer, the size of the output feature map is expanded through the deconvolution layer, and the feature map of the sub-network is connected to the feature map through skip connections.

[0073] like Figure 4 As shown in Figure 2, the discriminator consists of five Conv layers with 16, 32, 64, 128, and 1 channels, respectively. The discriminator takes as input a pair of real and synthetic images and outputs a binary indicator that provides feedback on whether the real sample and its corresponding synthetic image are distinguishable.

[0074] The 3D cycle-consistent generative adversarial network contains two complementary loss functions, including the adversarial loss function L g , cycle-consistent loss function L c , as follows:

[0075] L g (X MRI ,X PET ; G1, G2, D1, D2) = log (D2 (X PET ))+log(1-D2(G1(X MRI )))+log(D1(X MRI ))+log(1-D1(G2(X PET )))

[0076] L c (X MRI ,X PET ;G1,G2)=||G2(G1(X MRI ))-X PET ||1+||G1(G2(X PET ))-X PET ||1

[0077] Among them, X MRI Indicates MRI images, X PET Represents PET image, anti-loss function L g To ensure that the generated images are in principle indistinguishable from real images, the cycle-consistent loss function Lc The interactive relationship between MRI images and PET images is guaranteed.

[0078] S2: Select samples that only have MRI but lack PET images, and generate the missing PET images through the 3D cyclic consistent generative adversarial network;

[0079] S3: Use the generated PET images to complete the missing PET images in the dataset to obtain a complete MRI and PET dataset;

[0080] S4: The complete MRI and PET datasets are fed into the deformable self-attention mechanism module to extract multimodal imaging features under different modalities.

[0081] It should be noted that the deformable self-attention mechanism module is used to adaptively adjust the scale and position of each image block in the selected dataset and locate the pathological area with discriminative features.

[0082] like Figure 5 The figure shows a network diagram of the deformable self-attention mechanism module. The processing process of the deformable self-attention mechanism module is as follows:

[0083] Set the input to be a single MRI modality image X MRI ∈R H×W×D×C , where H represents the length of the image, W represents the width of the image, D represents the depth of the image, C represents the characteristic dimension of the image, and X MRI First, N patches of the same size and position are divided into a series of image blocks through the patch embedding module of the linear layer, and the sizes of H, W, and D are set to be equal. The size of the embedded image block is s×s×s, where The entire image block sequence is in X MRI The above is considered as a uniform grid of size s×s×s, and the series of image blocks is represented as {X i} (0≤i≤N-1) , for each image block X i , and its left front coordinate and right back coordinate are expressed as and in:

[0084]

[0085] Afterwards, each image block X i The offset prediction network predicts a position offset and a size scaling. The position offset and size scaling can be represented by parameters Δposition=(Δx, Δy, Δz) and Δscale=(Δh, Δw, Δd). For each deformed image block The updated left front coordinate and right back coordinate are expressed as and in:

[0086]

[0087]

[0088]

[0089] Since the size of each deformed image block is different, the sampling and interpolation method is used to extract features, and a uniform grid of size k×k×k is sampled in each image block. Since the coordinates of the grid are usually fractional, p j The value of C is calculated by trilinear interpolation at 8 adjacent points j The weighted average of is obtained, the specific formula is as follows:

[0090] p j =H Trilinear (C j ,(x j ,y j ,z j ))(j∈(0,k 3 -1))

[0091]

[0092] Afterwards, the deformed image block Projected into a feature vector through a d-dimensional linear layer and embedded into a flattened feature map Then, through a standard multi-head self-attention module As input, it generates three sets of learnable feature vectors, q∈R N×d , k∈R N×d 、v∈R N×d ;W q ∈R N×d 、W k ∈R N×d 、W v ∈R N×d are three learnable linear embedding matrices, and the specific formulas are as follows:

[0093]

[0094]

[0095] Y=Concat{Atten1v1,...,Atten M v M}W o

[0096] in dm =d / M, Atten m Represents the similarity matrix between different image blocks, which is expressed by q m and k m Multiply and get v m The weighted weights are obtained by adding the M attention matrices Atten m The final feature Y∈R is obtained by concatenating the transformation N×d .

[0097] Example 2

[0098] Based on the above process, this embodiment describes in detail the process of extracting the fusion feature map.

[0099] S5: Input the multimodal image features into the multimodal feature fusion network to extract the potential complementary features between MRI and PET and generate a fusion feature map;

[0100] like Figure 6 The figure shows a schematic diagram of a multimodal feature fusion network. The processing process of the multimodal feature fusion network is as follows:

[0101] Set the input of the multimodal feature fusion network to: MRI ∈R N×d 、Y PET ∈R N×d , where d is the feature dimension and N is the number of image blocks, where N = k × k × k;

[0102] The feature maps of the two modalities are concatenated in the d dimension and the dimension is adjusted to Y∈R 2d×k×k×k ;

[0103] Set Y = {Y1, Y2, ..., Y 2d}, where Y i ∈R k×k×k , the maximum pooling along the channel axis is expressed as:

[0104] Y max =ChannelMaxPooling(Y)

[0105] in The average pooling along the channel axis is expressed as:

[0106] Y mean =ChannelAvreagePooling(Y)

[0107] in These two feature maps are then concatenated to compute a spatial attention map:

[0108] A spatial =σ(W([Ymax ; Y mean ]))

[0109] Where σ is the sigmoid activation function, W is the convolutional layer weight, [;] is the concatenation operation, and the final multimodal attention feature map F is expressed as:

[0110]

[0111] in, Represents element-wise multiplication.

[0112] S6: Input the fused feature map into the classifier to output the classification result.

[0113] The fused feature map is compressed into a one-dimensional vector in the classifier through a global average pooling layer. Then, a full convolutional layer with a Sigmund function is used to perform convolution operations and activation on the feature map to output the Alzheimer's disease pathology image classification results.

[0114] Example 3

[0115] A second aspect of the present invention provides a multimodal Alzheimer's disease pathology image classification system, comprising: a memory and a processor, wherein the memory includes a multimodal Alzheimer's disease pathology image classification method program, and when the multimodal Alzheimer's disease pathology image classification method program is executed by the processor, the following steps are implemented:

[0116] S1: Select samples with both MRI and PET images and train a 3D cycle-consistent generative adversarial network based on the adversarial generative training strategy;

[0117] S2: Select samples that only have MRI but lack PET images, and generate the missing PET images through the 3D cyclic consistent generative adversarial network;

[0118] S3: Use the generated PET images to complete the missing PET images in the dataset to obtain a complete MRI and PET dataset;

[0119] S4: The complete MRI and PET datasets are fed into the deformable self-attention mechanism module to extract multimodal imaging features under different modalities.

[0120] S5: Input the multimodal image features into the multimodal feature fusion network to extract the potential complementary features between MRI and PET and generate a fusion feature map;

[0121] S6: Input the fused feature map into the classifier to output the classification result.

[0122] The third aspect of the present invention provides a computer-readable storage medium, which includes a multimodal Alzheimer's disease pathology image classification method program. When the multimodal Alzheimer's disease pathology image classification method program is executed by a processor, it implements the steps of a multimodal Alzheimer's disease pathology image classification method described in the item.

[0123] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multimodal Alzheimer's disease pathological image classification method, characterized in that: The following steps are involved: S1: Select samples with both MRI and PET images and train a 3D cycle-consistent generative adversarial network based on the adversarial generative training strategy; S2: Select samples that only have MRI but lack PET images, and generate the missing PET images through the 3D cyclic consistent generative adversarial network; S3: Use the generated PET images to complete the missing PET images in the dataset to obtain a complete MRI and PET dataset; S4: The complete MRI and PET datasets are fed into the deformable self-attention mechanism module to extract multimodal imaging features under different modalities. S5: Input the multimodal image features into the multimodal feature fusion network to extract the potential complementary features between MRI and PET and generate a fusion feature map; S6: Input the fused feature map into the classifier to output the classification result; Among them, the processing process of the deformable self-attention mechanism module is: Set the input to be a single MRI modality image X MRI ∈R H×W×D×C , where H represents the length of the image, W represents the width of the image, D represents the depth of the image, C represents the characteristic dimension of the image, and X MRI First, N patches of the same size and position are divided into a series of image blocks through the patch embedding module of the linear layer, and the sizes of H, W, and D are set to be equal. The size of the embedded image block is s×s×s, where The entire image block sequence is in X MRI The above is considered as a uniform grid of size s×s×s, and the series of image blocks is represented as {X i }, where 0≤i≤N-1, for each image block X i , and its left front coordinate and right back coordinate are expressed as and in: Afterwards, each image block X i The offset prediction network predicts a position offset and a size scaling amount, which can be represented by parameters Δposition=(Δx, Δy, Δz) and Δscale=(Δh, Δw, Δd). For each deformed image block The updated left front coordinate and right back coordinate are expressed as and in: Since the size of each deformed image block is different, sampling and interpolation methods are used to extract features. A uniform grid {p j }, where 0≤j≤k 3 -1, since the coordinates of the grid are usually fractional, therefore, p j The value of C is calculated by trilinear interpolation at 8 adjacent points j The weighted average of is obtained, the specific formula is as follows: p j =Η Trilinear (C j ,(x j ,y j ,z j )) Afterwards, the deformed image block Projected into a feature vector through a d-dimensional linear layer and embedded into a flattened feature map Then, through a standard multi-head self-attention module As input, it generates three sets of learnable feature vectors, q∈R N×d , k∈R N×d 、v∈R N×d ;W q ∈R N×d 、W k ∈R N×d 、W v ∈R N×d are three learnable linear embedding matrices, and the specific formulas are as follows: Y=Concat{Atten1v1,...,Atten M v M }W o Among them, 1≤m≤M, d m =d / M, Atten m Represents the similarity matrix between different image blocks, which is expressed by q m and k m Multiply and get v m The weighted weights are obtained by adding the M attention matrices Atten m The final feature Y∈R is obtained by concatenating the transformation N×d .

2. A multimodal Alzheimer's disease pathology image classification method according to claim 1, characterized in that: A deformable self-attention mechanism is used to adaptively adjust the scale and position of each image patch in the selected dataset and locate pathological regions with discriminative features.

3. A multimodal Alzheimer's disease pathology image classification method according to claim 1, characterized in that: In step S6, the fused feature map is compressed into a one-dimensional vector through a global average pooling layer in the classifier, and then a full convolutional layer with a Sigmoid function is used to perform convolution operation and activation on the feature map to output the Alzheimer's disease pathology image classification result.

4. A multimodal Alzheimer's disease pathology image classification method according to claim 1, characterized in that: The three-dimensional cycle-consistent generative adversarial network includes: two generator networks and two discriminator networks, respectively denoted as: G1, G2, D1, D2. Each generator includes an encoder and a decoder. The generator consists of 9 residual blocks, 5 3x3x3 convolutional layers, 4 downsampling layers and 4 upsampling layers. Each residual block includes three directly connected convolutional layers. The convolution kernel size of each convolutional layer is 3×3×3, and ReLU is selected as the activation function. In the downsampling layer, the size of the output feature map is halved and the number of channels is doubled. In the upsampling layer, the size of the output feature map is expanded by the deconvolution layer. The feature map of the sub-network is connected to the feature map through jump connection. The discriminator contains 5 Conv layers with 16, 32, 64, 128 and 1 channels respectively. The discriminator inputs a pair of real and synthetic images and outputs a binary indicator.

5. A multimodal Alzheimer's disease pathology image classification method according to claim 4, characterized in that: A 3D cycle-consistent generative adversarial network uses a cycle-consistent loss to learn a bidirectional mapping between MRI and PET.

6. A multimodal Alzheimer's disease pathology image classification system, characterized by: The system includes: a memory and a processor. The memory includes a multimodal Alzheimer's disease pathology image classification method program. When the multimodal Alzheimer's disease pathology image classification method program is executed by the processor, the following steps are implemented: S1: Select samples with both MRI and PET images and train a 3D cycle-consistent generative adversarial network based on the adversarial generative training strategy; S2: Select samples that only have MRI but lack PET images, and generate the missing PET images through the 3D cyclic consistent generative adversarial network; S3: Use the generated PET images to complete the missing PET images in the dataset to obtain a complete MRI and PET dataset; S4: The complete MRI and PET datasets are fed into the deformable self-attention mechanism module to extract multimodal imaging features under different modalities. S5: Input the multimodal image features into the multimodal feature fusion network to extract the potential complementary features between MRI and PET and generate a fusion feature map; S6: Input the fused feature map into the classifier to output the classification result; Among them, the processing process of the deformable self-attention mechanism module is: Set the input to be a single MRI modality image X MRI ∈R H×W×D×C , where H represents the length of the image, W represents the width of the image, D represents the depth of the image, C represents the characteristic dimension of the image, and X MRI First, N patches of the same size and position are divided into a series of image blocks through the patch embedding module of the linear layer, and the sizes of H, W, and D are set to be equal. The size of the embedded image block is s×s×s, where The entire image block sequence is in X MRI The above is considered as a uniform grid of size s×s×s, and the series of image blocks is represented as {X i }, where 0≤i≤N-1, for each image block X i , and its left front coordinate and right back coordinate are expressed as and in: Afterwards, each image block X i The offset prediction network predicts a position offset and a size scaling amount, which can be represented by parameters Δposition=(Δx, Δy, Δz) and Δscale=(Δh, Δw, Δd). For each deformed image block The updated left front coordinate and right back coordinate are expressed as and in: Since the size of each deformed image block is different, sampling and interpolation methods are used to extract features. A uniform grid {p j }, where 0≤j≤k 3 -1, since the coordinates of the grid are usually fractional, therefore, p j The value of C is calculated by trilinear interpolation at 8 adjacent points j The weighted average of is obtained, the specific formula is as follows: p j =Η Trilinear (C j ,(x j ,y j ,z j )) Afterwards, the deformed image block Projected into a feature vector through a d-dimensional linear layer and embedded into a flattened feature map Then, through a standard multi-head self-attention module As input, it generates three sets of learnable feature vectors, q∈R N×d , k∈R N×d 、v∈R N×d ;W q ∈R N×d 、W k ∈R N×d 、W v ∈R N×d are three learnable linear embedding matrices, and the specific formulas are as follows: Y=Concat{Atten1v1,...,Atten M v M }W o Among them, 1≤m≤M, d m =d / M, Atten m Represents the similarity matrix between different image blocks, which is expressed by q m and k m Multiply and get v m The weighted weights are obtained by adding the M attention matrices Atten m The final feature Y∈R is obtained by concatenating the transformation N×d .

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a multimodal Alzheimer's disease pathology image classification method program. When the multimodal Alzheimer's disease pathology image classification method program is executed by a processor, the steps of a multimodal Alzheimer's disease pathology image classification method as described in any one of claims 1 to 5 are implemented.

Images