Unsupervised reconstruction method of dynamic PET image based on implicit neural expression

Abstract

The invention discloses an implicit neural expression-based unsupervised reconstruction method for a dynamic PET image. An implicit neural network is innovatively used to replace a convolutional neural network to solve the reconstruction problem of the dynamic PET image. The limitation of reconstructing the dynamic PET image frame by frame is broken through, and the reconstruction quality of the PET image is improved by fully utilizing the time information contained in the data. And no complex regularization item or prior information is needed. The reconstruction quality of the image is improved by depending on the intrinsic priori of the implicit neural network and the intrinsic nature represented by the continuous function instead of an additional priori image or a regularization item. The calculation cost and time are remarkably reduced, the reconstruction quality of the PET image is greatly improved, and generalization is good. Compared with a reconstruction method based on the convolutional neural network, the method has the advantages that the reconstruction result is more excellent, and the best result is obtained. In addition, PET image reconstruction experiments of the brain and the thoracic cavity show that the method has excellent generalization performance.

Description

Technical Field

[0001] The present invention belongs to a medical image reconstruction method based on deep learning, specifically an unsupervised reconstruction method of dynamic PET images based on implicit neural representation for reconstructing dynamic positron emission tomography (PET) images without labels using implicit neural representation. Background Art

[0002] Dynamic PET is an advanced nuclear medicine imaging technology widely used in clinical diagnosis and biomedical research. It can reflect the distribution of radioactive tracers in various organs of the organism over time, thereby enabling quantitative analysis of specific physiological processes of the organism, especially playing an important role in the detection of cancer and tumors. However, in order to more accurately reflect the change in the distribution of the tracer over time, the scanning time for each frame of the dynamic PET image is relatively short, resulting in a small total number of photons in the measurement data and containing a large amount of noise, thus causing great difficulties in image reconstruction.

[0003] With the development of deep learning and artificial intelligence technology, deep learning networks are increasingly being used to solve the inverse problem of image reconstruction. PET image reconstruction methods based on deep learning can be divided into supervised and unsupervised methods. Supervised reconstruction methods were first used for PET image reconstruction and achieved image quality far higher than traditional reconstruction methods. This reconstruction method is generally designed based on convolutional neural networks (CNNs), such as encoder-decoder structures based on convolutional neural networks and U-net convolutional networks. However, this supervised learning method requires a large amount of high-quality labeled data sets for training, which is difficult to do for PET images. Therefore, reconstruction methods based on unsupervised learning have been gradually developed and used for PET image reconstruction. At present, the method widely used in unsupervised reconstruction of PET images is deep image prior (DIP). The DIP method uses the U-net convolutional network to reconstruct images. DIP believes that the network itself has certain image statistical prior information, so that the network first learns the natural information of the image, and then learns the noise in the image. If the iteration is stopped in advance before the network starts to learn the noise, a clean natural image can be obtained. The DIP method can also be combined with the kernel method to further improve the quality of unsupervised reconstruction of PET images. However, these methods reconstruct dynamic PET images frame by frame and cannot effectively utilize the temporal information in the data. In addition, the combination of the DIP method and the non-negative matrix separation (NMF) method solves the problem of frame-by-frame reconstruction, but introduces multiple convolutional neural networks and iterative steps, which increases the computational cost. At the same time, since these methods are designed based on convolutional networks, they naturally have fixed requirements on the input size of the network, which further limits their scope of application.

[0004] In recent years, implicit neural representation has developed rapidly and has gradually been used to solve image reconstruction problems. Unlike convolutional neural networks, implicit neural networks can continuously represent an image and are therefore not limited by resolution. They have played an important role in the fields of X-ray tomography (CT) and magnetic resonance imaging (MRI), but no one has yet extended them to the field of PET image reconstruction. Summary of the invention

[0005] Based on the above situation, the present invention discloses an unsupervised reconstruction method of dynamic PET images based on implicit neural expression. It is the first to use implicit neural expression to achieve fast reconstruction of high-quality dynamic PET images. The present invention does not require the use of a large number of high-quality labeled data sets for training, nor does it require any prior images, and performs well in generalization across different organs.

[0006] The present invention is achieved through the following technical solutions:

[0007] The present invention discloses an unsupervised reconstruction method for dynamic PET images based on implicit neural representation, including:

[0008] Determine the resolution of the reconstructed target image and calculate the spatial coordinates of the network input;

[0009] Perform Fourier encoding on the spatial coordinates to obtain the Fourier features corresponding to the spatial coordinates;

[0010] Input the Fourier features into an implicit neural network to obtain the pixel value corresponding to each spatial coordinate point, thereby obtaining a complete PET prediction image;

[0011] According to the system matrix of the PET instrument, calculate the sinogram predicted by the network corresponding to the PET prediction image;

[0012] Calculate the error between the sinogram predicted by the network and the sinogram obtained from actual measurement. The error calculation uses the logarithmic likelihood function;

[0013] Set the loss function Loss of the neural network to the negative of the logarithmic likelihood function; and perform gradient clipping on the loss function Loss to ensure the stability of network optimization;

[0014] Use the Adam optimizer to optimize the parameters of the neural network according to the loss function Loss.

[0015] As a further improvement, the Fourier encoding of the spatial coordinates to obtain the Fourier features corresponding to the spatial coordinates in the present invention is specifically:

[0016] In implicit neural representation, the input of the network is the spatial coordinates (x, y), and the output is the image pixel value at this point. A single implicit neural network tends to learn the low-frequency information of the image and cannot reconstruct the high-frequency information of the image; before inputting the coordinates (x, y) into the network, perform Fourier encoding on the spatial coordinates. The encoding step is represented by the following formula:

[0017] γ(c) = [cos(2πcB), sin(2πcB)] T ;

[0018] Among them, c represents the spatial coordinates (x, y), γ(c) represents the Fourier features corresponding to the spatial coordinates, B represents the encoding matrix, and the encoding matrix is a random number matrix that satisfies the normal Gaussian distribution. The encoding dimension is 2×256.

[0019] As a further improvement, the implicit neural network in the present invention is specifically:

[0020] The implicit neural network is a multi-layer perceptron (MLP) with five hidden layers. The number of neurons in the hidden layers is 256, and the activation function is the ReLU function. To better optimize the network, a Sigmoid activation function is added after the output layer. The number of output channels of the network is equal to the number of time frames of the dynamic PET, and multiple frames of dynamic PET images are reconstructed to utilize the time information in the data and avoid increasing the computational cost of the network. The output of the network is the reconstructed PET image, which is represented as:

[0021] x = f θ (γ(c)) = f θ ([cos(2πcB), sin(2πcB)] T );

[0022] where x is the dynamic PET image represented by the implicit neural network, f represents the implicit neural network, and θ represents the parameters in the network.

[0023] As a further improvement, the sinogram predicted by the network corresponding to the PET predicted image calculated in the present invention has the following specific calculation steps:

[0024]

[0025] where P is the system matrix of the PET instrument, x is the PET image output by the network, and r is the correction for the corresponding uniform random noise, which is the predicted sinogram required. The result obtained by the formula calculation needs to be corrected for the number of photons.

[0026] As a further improvement, the error between the sinogram predicted by the network and the sinogram obtained from the real measurement in the present invention is calculated using the log-likelihood function. The specific calculation formula is:

[0027]

[0028] where y i represents the sinogram obtained from the real measurement, represents the predicted sinogram, and M represents the dimension of the sinogram.

[0029] The beneficial effects of the present invention are as follows:

[0030] Innovatively use an implicit neural network to replace the convolutional neural network to solve the problem of reconstructing dynamic PET images. Previously, unsupervised PET reconstruction methods based on convolutional neural networks often required input images of a specific size, so the input images needed to be cropped; however, the implicit neural network can solve the drawback of the discrete representation of convolutional neural networks. It uses a fully connected network to represent a continuous natural function, and this continuous nature enables the implicit neural network to represent images of any resolution, greatly expanding the application scope of the present invention.

[0031] Break through the limitation of reconstructing dynamic PET images frame by frame, and make full use of the time information inherent in the data itself to improve the reconstruction quality of PET images. Previously, unsupervised reconstruction based on convolutional neural networks generally adopted a frame-by-frame reconstruction method, ignoring the time-varying relationship of the data itself; while the implicit neural network used in the present invention can output all frames of dynamic PET images simultaneously, and learn the time information inherent in the data itself by optimizing the parameters of the implicit neural network to enhance the reconstruction quality of the images.

[0032] Do not require complex regularization terms or prior information. Currently, some unsupervised reconstruction methods often require prior images, such as MRI images as the input of the network, or add regularization constraints such as smoothing terms; while the present invention does not require additional prior images or regularization terms, but relies on the inherent prior of the implicit neural network and the inherent nature of the continuous function representation to improve the reconstruction quality of the images.

[0033] Significantly reduce the computational cost and time. Since the present invention only uses an implicit neural network to simultaneously reconstruct full-frame dynamic PET images, compared with the frame-by-frame reconstruction based on convolutional networks, it can greatly save computational time and cost. In the experimental demonstration, compared with the reconstruction method DIPNMF based on convolutional networks, the computational time of the present invention is reduced by 98.5%.

[0034] Greatly improve the reconstruction quality of PET images and have good generalization performance. Whether in the PET image reconstruction experiments of the brain or the chest, the present invention has achieved better reconstruction results than the reconstruction methods based on convolutional neural networks. Whether in terms of the evaluation indexes of PSNR and SSIM of the overall image or the error analysis for a certain region of interest (ROI), the present invention has achieved the best results. In addition, the PET image reconstruction experiments of the brain and the chest also show that the present invention has excellent generalization performance. Brief Description of the Drawings

[0035] Figure 1 Is the flowchart of the present invention; Detailed Embodiment

[0036] The present invention discloses an unsupervised reconstruction method for dynamic PET images based on implicit neural representation, including:

[0037] Determine the resolution of the reconstructed target image and calculate the spatial coordinates of the network input;

[0038] Perform Fourier encoding on the spatial coordinates to obtain the Fourier features corresponding to the spatial coordinates; in the implicit neural representation, the input of the network is the spatial coordinates (x, y), and the output is the image pixel value at this point. A single implicit neural network tends to learn the low-frequency information of the image and cannot reconstruct the high-frequency information of the image; before inputting the coordinates (x, y) into the network, perform Fourier encoding on the spatial coordinates, and the encoding step is represented by the following formula:

[0039] γ(c) = [cos(2πcB), sin(2πcB)] T ;

[0040] where c represents the spatial coordinates (x, y), γ(c) represents the Fourier features corresponding to the spatial coordinates, B represents the encoding matrix, and the encoding matrix is a random number matrix that satisfies the normal Gaussian distribution, and the encoding dimension is 2×256.

[0041] Input the Fourier features into the implicit neural network to obtain the pixel value corresponding to each spatial coordinate point, thereby obtaining a complete PET prediction image; the implicit neural network is a multi-layer perceptron MLP, with 5 middle layers, the number of neurons in the middle layer is 256, and its activation function is the ReLU function. In order to better optimize the network, a Sigmoid activation function is added after the output layer. The number of output channels of the network is equal to the number of time frames of the dynamic PET, and multiple frames of dynamic PET images are reconstructed at the same time to utilize the time information in the data and avoid increasing the computational cost of the network; the output of the network is the reconstructed PET image, which is represented as:

[0042] x = f θ (γ(c)) = f θ ([cos(2πcB), sin(2πcB)] T ) ;

[0043] where x is the dynamic PET image represented by the implicit neural network, f represents the implicit neural network, and θ represents the parameters in the network;

[0044] According to the system matrix of the PET instrument, calculate the sinogram predicted by the network corresponding to the PET prediction image;

[0045] The specific calculation steps are as follows:

[0046] where P is the system matrix of the PET instrument, x is the PET image output by the network, and r is the corresponding uniform random noise correction, This is the desired predicted sinogram obtained through formula calculation. Photon number correction is required.

[0047] Calculate the error between the sinogram predicted by the network and the sinogram obtained from actual measurement. The error calculation uses the log-likelihood function; the error calculation uses the log-likelihood function, and the specific calculation formula is:

[0048]

[0049] Among them, y i represents the sinogram obtained from actual measurement, represents the predicted sinogram, and M represents the dimension of the sinogram;

[0050] The loss function Loss of the neural network is set to the negative of the log-likelihood function; and gradient clipping is performed on the loss function Loss to ensure the stability of network optimization;

[0051] Use the Adam optimizer to optimize the parameters of the neural network according to the loss function Loss.

[0052] To better illustrate the reconstruction process of the present invention, the following combines the attached drawings and elaborates in detail on the technical solution of the present invention through specific implementation cases, taking the brain simulation experiment as an example.

[0053] 1. Generation of brain simulation data.

[0054] First, generate dynamic PET images according to the brain digital model (phantom). The brain models used here are 20 patient brain models provided on the public database BrainWeb. For the convenience of calculation, after cropping, the pixel size of the model used in the experiment is 128×128 (denoted as N), and the actual size corresponding to each pixel is 2mm×2mm. Among these 20 brain models, 3 physiological regions of interest (ROIs) are selected: gray matter, white matter, and tumors to generate dynamic PET images. Here, a two-compartment model is used to generate PET, and its kinetic formula is:

[0055]

[0056] Among them, i represents a certain pixel, is the radioactivity concentration of this pixel point, k1, k2, k3, k4 are the corresponding kinetic parameters of the two-compartment model, C P is the plasma input function, V b is the volume fraction.

[0057] After obtaining the generated dynamic PET images, the time-activity curves are normalized so that their numerical range is between 0 and 1, which is convenient for the subsequent training of the network.

[0058] After that, the Michigan Image Reconstruction Toolbox (IRT) is used to generate the system matrix of the PET instrument, and the sinogram is generated according to the following formula:

[0059] y = Px + r

[0060] where P ∈ R M×N is the system matrix, x ∈ R N×T is the generated PET image, y ∈ R M×T is the sinogram, r ∈ R M×T is the added uniform random noise, and the noise ratio is 20%. After that, the photon number of the sinogram is corrected, and the corrected photon number is 2×10 7 and Poisson noise is added after that.

[0061] 2. Reconstruction process of PET images

[0062] In this case, the resolution of the reconstructed target image is 128×128. Based on this, the input spatial coordinates (x, y) are calculated, that is, the x-axis and y-axis are equally divided into 128 parts within the range of 0 - 1. Since a single implicit neural network tends to learn the low-frequency information of the image and cannot reconstruct the high-frequency information of the image. Therefore, before inputting the coordinates (x, y) into the network, the spatial coordinates are Fourier encoded, and the encoding step can be expressed by the following formula:

[0063] γ(c) = [cos(2πcB), sin(2πcB)] T

[0064] where c ∈ R N×2 represents the spatial coordinates (x, y), γ(c) represents the Fourier features corresponding to the spatial coordinates, and B ∈ R 2×G represents the encoding matrix. The encoding matrix used in the present invention is a random number matrix that satisfies the normal Gaussian distribution, and the encoding dimension G = 256.

[0065] After that, the Fourier features of the spatial coordinates are input into the implicit neural network. The implicit neural network used here is a multi-layer perceptron (MLP), with 5 middle layers, 256 neurons in the middle layer, and its activation function is the ReLU function. At the same time, to better optimize the network, a Sigmoid activation function is added after the output layer. The number of output channels of the network is equal to the number of time frames of the dynamic PET, which can reconstruct multiple frames of dynamic PET images simultaneously, utilize the time information in the data, and avoid increasing the computational cost of the network.

[0066] The output of the network is the reconstructed PET image, which can be expressed as:

[0067] x = f θ (γ(c)) = f θ ([cos(2πcB), sin(2πcB)] T

[0068] where x ∈ R N×T is the dynamic PET image represented by the implicit neural network, f represents the implicit neural network, and θ represents the parameters in the network.

[0069] According to the image output by the network and combined with the system matrix of the PET instrument, the predicted sonogram can be calculated, and the calculation formula is as follows:

[0070]

[0071] where P ∈ R M×N is the system matrix, x ∈ R N×T is the PET image output by the network, r ∈ R M×T is the corresponding uniform random noise correction, is the predicted sinogram required. Note that the obtained by this formula calculation needs to be corrected for the number of photons.

[0072] In the unsupervised reconstruction process, it is necessary to calculate the error between the predicted sinogram and the real sinogram. The error calculation formula uses the log-likelihood function, as follows:

[0073]

[0074] where y i represents the real sinogram, represents the sinogram predicted by the present invention. At the same time, since the last term log(y i )! in the above formula is a constant term and has no effect on the error calculation, this term is generally ignored in the calculation process.

[0075] During the network optimization process, the loss function is the negative of the above-mentioned logarithmic likelihood function, that is

[0076] Loss = -L(y|x)

[0077] At the same time, in order to ensure the stability of network optimization, it is necessary to perform gradient clipping on the loss function Loss. The maximum norm for clipping is 1.0. Then, Loss is passed into the Adam optimizer to optimize the network. The initial learning rate is 0.001, and it is set that the learning rate decays to 98% of the original every 100 iterations. The total number of iterations is 3000 times.

[0078] After the network completes iteration and optimization, an optimal solution can be obtained, that is:

[0079]

[0080]

[0081] Here, is the finally obtained reconstruction result.

[0082] 3. Reconstruction Results

[0083] Finally, the reconstruction results of 20 brains are obtained. The evaluation metrics used here are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The comparison methods include the traditional algorithms ML-EM and Kernel Method, and the unsupervised reconstruction method DIPNMF based on convolutional network. The comparison results are as follows:

[0084] ML-EM: PSNR is 22.97 and SSIM is 0.81;

[0085] Kernel Method: PSNR is 25.18 and SSIM is 0.88;

[0086] DIPNMF: PSNR is 26.96 and SSIM is 0.93;

[0087] This invention: PSNR is 29.46 and SSIM is 0.95.

[0088] The above results are the average calculation results of 20 brains, and this invention has achieved the optimal reconstruction quality.

[0089] Meanwhile, in terms of the average reconstruction time of 20 brains, the non-network traditional algorithm took the shortest time, with the ML-EM taking 6.9 s and the Kernel Method taking 7.8 s. The DIPNMF based on convolutional neural network took the longest time, with a total reconstruction time of 1.7 hours. However, the present invention uses an implicit neural network instead of a convolutional neural network, taking only 95 seconds, which is far lower than that of the convolutional network.

[0090] The above description of the embodiments is to enable those of ordinary skill in the art to understand and apply the present invention. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative efforts. Therefore, the present invention is not limited to the above embodiments, and all improvements and modifications made by those skilled in the art based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. An unsupervised reconstruction method for dynamic PET images based on implicit neural representations, characterized in that, Including: Determine the resolution of the reconstructed target image and calculate the spatial coordinates of the network input; Perform Fourier encoding on the spatial coordinates to obtain the Fourier features corresponding to the spatial coordinates; Input the Fourier features into an implicit neural network to obtain the pixel values corresponding to each spatial coordinate point, thereby obtaining a complete PET prediction image; According to the system matrix of the PET instrument, calculate the sinogram predicted by the network corresponding to the PET prediction image; Calculate the error between the sinogram predicted by the network and the sinogram obtained from the real measurement. The error calculation uses the logarithmic likelihood function; Set the loss function Loss of the neural network to the negative of the logarithmic likelihood function; and perform gradient clipping on the loss function Loss to ensure the stability of network optimization; Use the Adam optimizer to optimize the parameters of the neural network according to the loss function Loss.

2. The unsupervised reconstruction method of dynamic PET images based on implicit neural representation according to claim 1, characterized in that, The performing Fourier encoding on the spatial coordinates to obtain the Fourier features corresponding to the spatial coordinates is specifically as follows: In the implicit neural representation, the input of the network is the spatial coordinates (x, y), and the output is the image pixel value at this point. A single implicit neural network tends to learn the low-frequency information of the image and cannot reconstruct the high-frequency information of the image; before inputting the coordinates (x, y) into the network, perform Fourier encoding on the spatial coordinates. The encoding steps are represented by the following formula: γ(c) = [cos(2πcB), sin(2πcB)] T ; Among them, c represents the spatial coordinates (x, y), γ(c) represents the Fourier features corresponding to the spatial coordinates, B represents the encoding matrix, and the encoding matrix is a random number matrix that satisfies the normal Gaussian distribution. The encoding dimension is 2×256.

3. The unsupervised reconstruction method of dynamic PET images based on implicit neural representation according to claim 1, wherein The implicit neural network is specifically as follows: The implicit neural network is a multi-layer perceptron MLP. The middle layer has 5 layers, the number of neurons in the middle layer is 256, and its activation function is the ReLU function. To better optimize the network, a Sigmoid activation function is added after the output layer. The number of output channels of the network is equal to the number of time frames of the dynamic PET. Reconstruct multiple frames of dynamic PET images at the same time, utilize the time information in the data, and avoid increasing the computational cost of the network; the output of the network is the reconstructed PET image, which is represented as: x = f θ γ(c) = f θ [cos(2πcB), sin(2πcB)] T ) Among them, x is the dynamic PET image represented by the implicit neural network, f represents the implicit neural network, and θ represents the parameters in the network.

4. The unsupervised reconstruction method for dynamic PET images based on implicit neural representation according to claim 1 or 2 or 3, characterized in that, The calculating the sinogram predicted by the network corresponding to the PET prediction image, and its specific calculation steps are as follows: where P is the system matrix of the PET instrument, x is the PET image output by the network, and r is the corresponding uniform random noise correction, which is the required predicted sinogram, and the result obtained by the above formula calculation needs to be corrected for the number of photons.

5. The unsupervised reconstruction method for dynamic PET images based on implicit neural representation according to claim 4, characterized in that, The calculating the error between the sinogram predicted by the network and the sinogram obtained from the real measurement. The error calculation uses the logarithmic likelihood function, and the specific calculation formula is: Among them, y i represents the sinogram obtained by actual measurement, represents the predicted sinogram, and M represents the dimension of the sinogram.

Images