A method for predicting SUVR in multimodal PET images of the brain

Abstract

The invention discloses a method for predicting SUVR of multimodal PET images of the brain, comprising step 1: collecting T1-weighted images and PET images of subjects; step 2: for the T1-weighted images, segmenting the tissues, and for the PET images, calculating the standard uptake ratio SUVR; step 3: segmenting the T1-weighted images aligned to the standard space into a number of 3D patches of equal size; step 4: constructing a model, including a regression module and a classification module, wherein the backbone of the network is constructed by a Mixer block; step 5: separately training the regression module to predict the SUVR of multimodal PET images, and then separately pre-training the classification module for AD and normal classification; merging the embedded features of the regression module with the embedded features of the classification module for final classification training; step 6: inputting a new T1-weighted image patch to obtain the predicted multimodal PET SUVR and AD state.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for predicting SUVR of multimodal PET images of the brain. Background Art

[0002] Alzheimer's disease (AD) is the leading cause of dementia, with an estimated prevalence of 10-30% in people over 65 years of age. AD is characterized by atrophy of the cerebral cortex and hippocampus, accumulation of amyloid-β in the brain, abnormal phosphorylation of tau in the brain, and decreased glucose metabolism in the brain. Advances in imaging techniques, such as structural magnetic resonance imaging (sMRI) and positron emission tomography (PET), have enabled us to diagnose AD more accurately.

[0003] For example, using sMRI, we can accurately detect brain areas with reduced volume. By using PET, such as fluorodeoxyglucose (FDG)-PET or florbetapir (AV45)-PET, we can understand the glucose metabolism or β-amyloid load in the brain. Accordingly, with the accumulation of imaging data, some machine learning or deep learning-based methods have been developed to predict disease states, such as diagnosing AD or predicting the prognosis of its prodromal mild cognitive impairment (MCI). Methods using PET are superior because PET can directly detect pathological biomarkers of AD, such as β-amyloid, and show higher diagnostic accuracy compared to sMRI, especially for the early detection of AD. In addition, PET can also be combined with sMRI to further improve AD diagnostic performance, because PET and sMRI can provide complementary information.

[0004] Although existing work has achieved some success in synthesizing PET images, there is still much room for improvement. First, when there is limited training data for matching sMRI and PET data, it may be difficult and unstable to synthesize the entire PET image using GAN;

[0005] Second, existing methods usually only synthesize single-modality FDG-PET images from sMRI, while PET of different modalities can provide complementary information for AD diagnosis;

[0006] Furthermore, most previous AD diagnosis or PET image synthesis methods have adopted CNN models. Since CNN uses local operators such as convolution and pooling, CNN-based models need to stack more convolution layers to capture more distant relationships in the input, which leads to an increase in the number of model parameters and increased training difficulty. However, given the limited number of sMRIs, large models are not a suitable choice. Summary of the invention

[0007] The purpose of the present invention is to provide a method for predicting the SUVR of brain multimodal PET images, adopt a multi-layer perceptron mixer (MLP-Mixer) architecture, establish a deep learning model, namely a three-dimensional multi-task multi-layer perceptron mixer (3D-Mixer), which is composed of a regression module and a classification module, and simultaneously predicts the standardized uptake value ratio of FDG-PET and AV45-PET through the regression module.

[0008] The purpose of the present invention can be achieved through the following technical solutions:

[0009] A method for predicting brain multimodal PET images SUVR comprises the following steps:

[0010] Step 1: Data collection: collecting brain images of the subjects, including T1-weighted images and PET images;

[0011] Step 2: Data processing: align the brain images in step 1 to the standard space, segment the tissues for T1-weighted images, and calculate the standard uptake ratio SUVR for PET images;

[0012] Step 3: Segment the T1-weighted image aligned to the standard space into several 3D patches of equal size, so that the position of each patch corresponds one to one, calculate the effectiveness of each patch area for diagnosing AD, and extract several areas with the most significant discrimination as landmarks;

[0013] Step 4: Build the model, including regression module and classification module. The backbone of the network is built by Mixer block.

[0014] Step 5: Train the model. Train the regression module separately to predict SUVR of multimodal PET images, and then pre-train the classification module separately for AD and normal classification. Combine the embedded features of the regression module with the embedded features of the classification module for final classification training.

[0015] Step 6: Prediction. After the training model is completed, the new T1-weighted image patch is input to obtain the predicted multimodal PET SUVR and AD status.

[0016] As a further solution of the present invention: in step 1, the T1 weighted image is obtained by scanning the subject's head with a common magnetic resonance imaging device.

[0017] According to a further embodiment of the present invention, in step 1, the PET images are obtained by using a positron emission tomography device to obtain the same subject using a standard dose of fluorodeoxyglucose FDG and a florbetapir AV45 tracer.

[0018] It is a further solution of the present invention: in step 2, for the processing of T1 weighted images: using the SPM12 tool, aligning the acquired T1 images to the MNI152 template, removing irrelevant tissues, and segmenting them into gray matter and white matter;

[0019] For PET images, all frames were averaged, and the averaged images were aligned to MNI space and the standard uptake ratio was calculated using the pons and cerebellar vermis as reference regions defined using the AAL template.

[0020] It is a further solution of the present invention: in step 4, the Mixer block is composed of two feature mixing layers implemented by multi-layer perceptrons, and the two multi-layer perceptrons act on the columns and rows of the input matrix in turn to perform feature mixing;

[0021] Specifically: Given an input matrix X, the output Y = Mixer (X) of the Mixer module can be expressed as follows:

[0022]

[0023]

[0024] in X *,i =[x 1i x 2i …x ki ] T and Y j,* =[y j1 y j2 …y jc ] represent the i column of X matrix and the j row of Y matrix respectively, and is the parameter that needs to be trained and learned, k represents the number of input patches, c is the number of features of each patch, r is a constant, r is set to 4, σ(·) is the activation function GELU, and LayerNorm represents LayerNormalization.

[0025] A further solution of the present invention: In step 4, the construction process of the regression module is:

[0026] The patch of T1-weighted image is used as input, and the corresponding SUVR of FDG-PET and AV45-PET is predicted through two output branches;

[0027] make Representing a flattened patch, the input series of patches are first embedded through a linear layer:

[0028]

[0029] in and is the learnable weight, k r Represents the number of patches input to the regression module;

[0030] Transform the input into After that, N-1 weight-sharing Mixer blocks and a task-specific Mixer block further transform it in turn:

[0031]

[0032]

[0033]

[0034] in, represents the nth shared weight Mixer block in the regression module, and They represent the Mixer blocks of the FDG-PET output branch and the AV45-PET output branch respectively. Finally, the predicted FDG / AV45-PET SUVR are expressed as:

[0035]

[0036]

[0037] in for

[0038] A further solution of the present invention: In step 4, the construction process of the classification module is:

[0039] The T1-weighted image patch is used as input, and the input patch only contains the landmark area;

[0040] The input of the classification module is transformed by the linear embedding layer and N Mixer blocks to obtain And v after average pooling clf , output o clf It is expressed as:

[0041]

[0042] A further solution of the present invention is to adopt the mean square error loss function based on the construction of the regression module:

[0043]

[0044] Among them, y is the real PET SUVR,

[0045] A further solution of the present invention is to use a cross entropy loss function based on the classification module:

[0046]

[0047] where y clf is a binary classification label.

[0048] It is a further solution of the present invention: in step 5, the training model adopts the stochastic gradient descent method and uses the optimizer Adam.

[0049] Beneficial effects of the present invention: The present invention predicts quantitative table description indicators of multiple modal PET images based on sMRI at the same time, has strong practicality, and uses Mixer blocks to construct the main skeleton of the model instead of the common CNN model, which can obtain a larger receptive field and better effect with a smaller number of model layers, a smaller scale, and faster calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described below in conjunction with the accompanying drawings.

[0051] Figure 1 It is a flow chart of the present invention;

[0052] Figure 2 is a schematic diagram of the multimodal PET SUVR and AD status prediction process of the present invention;

[0053] Figure 3 It is a schematic diagram of the SUVR prediction results of the multimodal PET image of the present invention;

[0054] Figure 4 It is a schematic diagram of the AD status prediction results of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Based on existing technology, PET is expensive and patients may have concerns about the use of radioactive tracers, which greatly limits the application of PET. For example, in many public imaging databases, the amount of available PET data is much less than that of sMRI. Many previous intelligent algorithms and studies have used sMRI and PET data to jointly diagnose AD, but subjects who lack PET data are usually deleted during the training process, which greatly reduces the number of subjects used for training.

[0057] To alleviate this problem, some computational methods have been developed to synthesize PET images from matching sMRI to increase the amount of available samples. For example, the generative adversarial network (GAN) based on convolutional neural network has also been used for FDG-PET image synthesis. The generative adversarial network model includes a classifier and a generator, where the generator is used to synthesize FDG-PET images corresponding to sMRI.

[0058] The specific steps of this method can be summarized as follows:

[0059] 1. Collect sMRI images and FDG-PET images of the same subject.

[0060] 2. Align sMRI and PET images to the MNI standard space and remove irrelevant tissues such as the skull.

[0061] 3. Build generator and classifier models.

[0062] 4. Train the generator to make it possible for the PET images generated based on sMRI and the real PET images to be incorrectly classified by the classifier.

[0063] 5. Train the classifier to correctly distinguish between synthetic PET images and real PET images as much as possible.

[0064] 6. Repeat 4-5 until convergence conditions are reached.

[0065] Based on the above technology, the 3D-Mixer proposed in the present invention simultaneously predicts the standardized uptake value ratio SUVR of FDG-PET and AV45-PET through a regression module. It can capture long-range relationships in the input with fewer layers and avoid the inherent shortcomings of CNN and GAN. The evaluation of independent datasets shows that 3D-Mixer can effectively predict FDG / AV45-PET SUVR, and the estimated SUVR also shows strong discrimination ability between cognitively normal (CN) and AD subjects.

[0066] In addition, based on the sMRI image and PET embedded features (i.e., the output of the penultimate layer of the regression module), the classification module can accurately predict the AD status, i.e., the diagnosis of AD or the prediction of whether MCI will transform into AD. Benchmark tests on three independent datasets show that the proposed 3D-Mixer model outperforms other CNN and non-CNN methods.

[0067] For details, see Figure 1 , a method for predicting SUVR of multimodal PET images of the brain, comprising:

[0068] Step 1: Data Collection

[0069] Collect three public datasets, including ADNI, OASIS, and MIRIAD, containing data from 1821, 733, and 69 subjects, respectively. All subjects acquired brain T1-weighted images, and most of the subjects from ADNI acquired PET images of more than one modality. The ADNI data were divided into training, validation, and test sets in a ratio of 6:2:2. OASIS and MIRIAD were not used for training, but only for evaluating the performance of the method.

[0070] Step 2: Data processing

[0071] All acquired images were preprocessed using a unified process. First, the SPM12 tool was used to align the acquired T1 images to the MNI152 template, and after removing irrelevant tissue, they were further segmented into gray matter and white matter.

[0072] For PET images, all frames were first averaged, and then the averaged images were aligned to the MNI space, and the standard uptake ratio was calculated by using the pons and cerebellar vermis defined by the AAL template as reference regions;

[0073] Step 3: Patch extraction and landmark calculation

[0074] The dimensions of the preprocessed T1 images are uniformly 121×145×121, and the middle 100×125×100 area is evenly divided into 80 patches of size 25×25×25;

[0075] For landmark, extract the same number of T1 images of AD patients and normal controls, and extract patches according to the above steps (the patches of each subject can correspond one to one in spatial position). Further, calculate the voxel mean of each patch of each subject, and then perform a t-test on the AD group and the normal control group based on the voxel mean of the sample for each patch area, and define the 50 patch areas with the smallest p-value as landmark areas;

[0076] Step 4: Build the model

[0077] like Figure 2 As shown, it contains a regression module and a classification module, and the backbone of the network is built by the Mixer block;

[0078] Mixer Block

[0079] This embodiment uses the latest Mixer block to build the backbone of the network instead of the widely used convolutional layer. Specifically, the Mixer module consists of two feature mixing layers implemented by multi-layer perceptrons (MLPs). The two MLPs act on the columns and rows of the input matrix in turn to mix features. Compared with the CNN architecture, this architecture can produce a larger receptive field. Given an input matrix X, the output Y=Mixer(X) of the Mixer module can be expressed as follows:

[0080]

[0081]

[0082] in X *,i =[x 1i x 2i …x ki ] T and Y j,* =[y j1 y j2 …y jc ] represent the i column of X matrix and the j row of Y matrix respectively, and are parameters that need to be trained and learned; k represents the number of input patches, c is the number of features of each patch, r is a constant, r is set to 4, σ(·) is the activation function GELU; LayerNorm represents layer normalization;

[0083] Regression Module

[0084] The regression module takes a series of sMRI patches as input and predicts the corresponding SUVR of FDG-PET and AV45-PET through two output branches. Representing a flattened patch, the input series of patches are first embedded through a linear layer:

[0085]

[0086] in and is a learnable weight. kr Represents the number of patches of the input of the regression module. The input is transformed into After that, N-1 weight-sharing Mixer blocks and a task-specific Mixer block further transform it in turn:

[0087]

[0088]

[0089]

[0090] in, represents the nth shared weight Mixer block in the regression module, and They represent the Mixer blocks of the FDG-PET output branch and the AV45-PET output branch respectively. Finally, the predicted FDG / AV45-PET SUVR are expressed as:

[0091]

[0092]

[0093] in for

[0094] Classification Module

[0095] The structure of the classification module is similar to that of the regression module, but its input patch only contains the landmark area, and the multimodal PET embedding features of the regression module are introduced internally. Specifically, the input of the classification module is transformed through the linear embedding layer and N Mixer blocks to obtain And v after average pooling clf , output o clf It can be expressed as:

[0096]

[0097] Loss function;

[0098] For the regression module, we use the mean squared error loss function:

[0099]

[0100] Among them, y is the real PET SUVR,

[0101] The classification module uses the cross entropy loss function:

[0102]

[0103] where y clf It is a binary classification label, for example, in AD and normal classification, 0 represents normal and 1 represents AD, while in MCI conversion prediction, 0 represents stable MCI (sMCI) and 1 represents progressive MCI (pMCI);

[0104] Step 5: Training

[0105] like Figure 2 As shown, first, the regression module is trained separately for multimodal PET image SUVR prediction. Then the classification module is pre-trained for AD and normal classification separately, and finally the embedded features of the regression module (the output of the second to last layer) are merged with the embedded features of the classification module for the final classification training (AD vs. normal classification, sMCI vs. pMCI classification);

[0106] To train the model, the stochastic gradient descent method is used, and the optimizer Adam is used, with the initial step size set to 1×10 -4 ;

[0107] The training data includes a training set and a validation set. The training set is used to calculate the gradient, and the validation set is used to verify the performance. Once the loss function on the validation set does not decrease within 10 epochs, the step size is multiplied by 0.5, and finally only the model with the smallest loss function on the validation set is used.

[0108] Step 6: SUVR prediction of multimodal PET images

[0109] After the model training is completed, the sMRI images of the independent ADNI test set are used as input to predict the SUVR of the multimodal PET images and compare them with the actual SUVR, such as Figure 3 As shown;

[0110] Step 7: AD status prediction

[0111] Using the sMRI of three datasets as input, we predicted whether the patient was ill or not, and whether it was progressive mild cognitive impairment. Based on the real labels, we evaluated the proposed method, such as Figure 4 shown.

[0112] The core of the present invention is to adopt a multi-layer perceptron mixer (MLP-Mixer) architecture to establish a deep learning model, namely a three-dimensional multi-task multi-layer perceptron mixer (3D-Mixer), which is composed of a regression module and a classification module. The regression module is used to simultaneously predict the standardized uptake value ratio of FDG-PET and AV45-PET, and the AD state is predicted by merging the features of the two modules.

[0113] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for predicting SUVR of multimodal PET images of the brain, characterized in that: The following steps are involved: Step 1: Data collection: collecting brain images of the subjects, including T1-weighted images and PET images; Step 2: Data processing: align the brain images in step 1 to the standard space, segment the tissues for T1-weighted images, and calculate the standard uptake ratio SUVR for PET images; Step 3: Segment the T1-weighted image aligned to the standard space into several 3D patches of equal size, so that the position of each patch corresponds one to one, calculate the effectiveness of each patch area for diagnosing AD, and extract several areas with the most significant discrimination as landmarks; Step 4: Build the model, including regression module and classification module. The backbone of the network is built by Mixer block. Step 5: Train the model. Train the regression module separately to predict SUVR of multimodal PET images, and then pre-train the classification module separately for AD and normal classification. Combine the embedded features of the regression module with the embedded features of the classification module for final classification training. Step 6: Prediction: After the training model is completed, the new T1-weighted image patch is input to obtain the predicted multimodal PET, SUV, R and AD status; In step 4, the Mixer block consists of two feature mixing layers implemented by multi-layer perceptrons, which act on the columns and rows of the input matrix in turn to perform feature mixing; Specifically: Given an input matrix X, the output Y = Mixer (X) of the Mixer module can be expressed as follows: in X *,i =[x 1i x 2i …x ki ] T and Y j,* =[y j1 y j2 …y jc ] represent the i column of X matrix and the j row of Y matrix respectively, and are parameters that need to be trained and learned; k represents the number of input patches, c is the number of features of each patch, r is a constant, r is set to 4, σ(·) is the activation function GELU, and LayerNorm represents layer normalization; In step 4, the construction process of the regression module is: The patch of T1-weighted image is used as input, and the corresponding SUVR of FDG-PET and AV45-PET is predicted through two output branches; make Represents a flattened patch, m = 15625, and the input series of patches are first embedded through a linear layer: in and is the learnable weight, k r Represents the number of patches input to the regression module; Transform the input into After that, N-1 weight-sharing Mixer blocks and a task-specific Mixer block further transform it in turn: in, represents the nth shared weight Mixer block in the regression module, and They represent the Mixer blocks of the FDG-PET output branch and the AV45-PET output branch respectively. Finally, the predicted FDG / AV45-PET SUVR are expressed as: in for In step 4, the construction process of the classification module is: The T1-weighted image patch is used as input, and the input patch only contains the landmark area; The input of the classification module is transformed by the linear embedding layer and N Mixer blocks to obtain And v after average pooling clf , output o clf It is expressed as: The construction based on the regression module adopts the mean square error loss function: Among them, y is the real PET SUVR, Based on the classification module, the cross entropy loss function is used: where y clf is a binary classification label.

2. The method for predicting SUVR of a brain multimodal PET image according to claim 1, characterized in that: In step 1, the T1-weighted image is obtained by scanning the subject's head using a common magnetic resonance imaging device.

3. The method for predicting SUVR of a brain multimodal PET image according to claim 1, characterized in that: In step 1, PET images were acquired in the same subjects using a positron emission tomography scanner using standard doses of fluorodeoxyglucose (FDG) and florbetapir AV45 tracers.

4. The method for predicting SUVR of a brain multimodal PET image according to claim 1, characterized in that: In step 2, for the processing of T1-weighted images: use the SPM12 tool to align the acquired T1 images to the MNI152 template, remove irrelevant tissues, and then segment them into gray matter and white matter; For PET images, all frames were averaged, and the averaged images were aligned to MN I space and the standard uptake ratio was calculated using the pons and cerebellar vermis as reference regions defined using the AAL template.

5. The method for predicting SUVR of multimodal PET images of the brain according to claim 1, characterized in that: In step 5, the training model adopts the stochastic gradient descent method and uses the optimizer Adam.

Images