Medical image processing device and program

A DL-CNN trained on varied noise levels and multimodal imaging improves PET image quality by reducing noise and enhancing spatial resolution, addressing the inconsistency and degradation issues in existing noise removal methods.

JP7586999B2Active Publication Date: 2024-11-19CANON MEDICAL SYST CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023188300
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-01-25
Filing Date
2023-11-02
Publication Date
2024-11-19
Estimated Expiration
2039-05-29

AI Technical Summary

Technical Problem

PET imaging is prone to noise, which affects image quality and makes it difficult to detect small or low-contrast lesions, and existing noise removal methods often degrade image resolution or require user-adjusted parameters, leading to inconsistent image quality across different scans and patients.

Method used

A deep learning (DL)-based convolutional neural network (CNN) is trained on a wide range of noise levels to denoise PET images, using multimodality training with other medical imaging modalities like MRI or CT to improve spatial resolution and reduce noise while preserving lesion details, without requiring user-adjusted parameters.

Benefits of technology

The method produces consistently high-quality PET images with reduced noise and improved spatial resolution, enhancing clinical reliability and reproducibility by minimizing intra- and inter-patient variability and preserving small lesion details.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586999000006
    Figure 0007586999000006
  • Figure 0007586999000007
    Figure 0007586999000007
  • Figure 0007586999000008
    Figure 0007586999000008
Patent Text Reader

Abstract

To improve image quality.SOLUTION: A medical image processor includes an acquisition section and a processing section. The acquisition section acquires a reconstructed Positron Emission Tomography (PET) image, which is reconstructed from emission data representing coincidence counts of respective pairs of gamma rays arising from annihilation events, the coincidence counts being acquired on the basis of detection of a plurality of detection elements. The processing section outputs a PET image denoised by inputting a PET image reconstructed in a reconstruction section, with respect to a learned model that outputs a third image, which is a denoised PET image, by inputting a first image, which is a PET image, and a second image acquired by modality different from that of a PET device.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] An embodiment of the present invention relates to a medical image processing apparatus and a program. [Background technology]

[0002] In PET (Positron Emission Tomography) imaging, imaging is performed by detecting simultaneous pairs of gamma rays emitted by a tracer inside the patient's body using detectors placed around the patient. In PET imaging, image reconstruction is performed based on the data detected by the detectors, and an image showing, for example, the concentration of a tracer inside the patient's body can be obtained.

[0003] Here, in PET imaging, since images are easily affected by noise, noise removal processing may be performed, for example, by machine learning.

[0004] However, for example, if the noise level of an image used for learning differs from the actual noise level of an image to be subjected to noise removal processing, the image quality may not improve. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent No. 2016 / 0116603 [Patent Document 2] U.S. Pat. No. 1,007,889 [Patent Document 3] International Publication No. 2017 / 191643 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to improve image quality. [Means for solving the problem]

[0007] A medical image processing apparatus according to an embodiment includes an acquisition unit and a processing unit. The acquisition unit acquires a Positron Emission Tomography (PET) image reconstructed from emission data representing coincidence measurements of pairs of gamma rays resulting from pair annihilation events, acquired based on detection by a plurality of detection elements. The processing unit inputs a PET image reconstructed by the reconstruction unit to a trained model that outputs a third image, which is a denoised PET image, by inputting a first image that is a PET image and a second image acquired by a modality different from the PET device, and outputs the denoised PET image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a flowchart showing the flow of processing of a method 100 for training a deep learning (DL) convolutional neural network (CNN) included in a medical image processing apparatus according to an embodiment. [Diagram 2] FIG. 2 shows another flow chart of the methods 100 and 200 applied for denoising PET images according to an embodiment. [Figure 3A] FIG. 3A is a diagram illustrating a DL-CNN network provided in the medical image processing apparatus according to the embodiment. [Figure 3B] FIG. 3B is a diagram illustrating a DL-CNN network provided in the medical image processing apparatus according to the embodiment. [Figure 4] FIG. 4 is a flowchart illustrating the flow of processing of a process 400 in the medical image processing apparatus according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating the process of step 110' in the medical image processing apparatus according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating the process flow of a method 100' according to an embodiment. [Figure 7]FIG. 7 is a diagram illustrating the process according to the embodiment. [Figure 8A] FIG. 8A is a diagram illustrating the results of performing a process according to a comparative example. [Figure 8B] FIG. 8B is a diagram illustrating the result of processing using a uniform weight map in the embodiment. [Figure 8C] FIG. 8C is a diagram illustrating the result of processing using the weight map according to step 110' in the embodiment. [Figure 9] FIG. 9 is a diagram illustrating step 110'' and step 210'' in the embodiment. [Figure 10] FIG. 10 illustrates an example of a flowchart for training a DL-CNN network in an embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a flowchart for applying a general artificial neural network (ANN) in the embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a flowchart for applying a convolutional neural network (CNN) in an embodiment. [Figure 13A] FIG. 13A is a diagram illustrating an example of a feedforward ANN in an embodiment. [Figure 13B] FIG. 13B is a diagram illustrating an example of a CNN in the embodiment. [Figure 13C] FIG. 13C illustrates an example of implementing a convolutional layer (for one neuron node of the convolutional layer) in an embodiment. [Figure 14A] FIG. 14A illustrates an example perspective view of a Positron Emission Tomography (PET) scanner in accordance with an embodiment. [Figure 14B] FIG. 14B is a diagram showing an example of a schematic diagram of a PET scanner in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Here, the same components are denoted by common reference numerals, and duplicated descriptions will be omitted. In the following embodiment, the processing unit is realized, for example, by using the processing device 870 in FIG. 14B. Also, the acquisition unit and reconstruction unit are realized, for example, by using the data acquisition system 876 in FIG. 14B.

[0010] Due to various reasons, it is not possible to consistently obtain high-quality images from positron emission tomography (PET) imaging. For example, due to limitations in drug injection volume and scan time, the image quality of PET images is often degraded due to the effects of high noise and low spatial resolution (e.g., 5-7 mm).

[0011] Additionally, high noise levels and low spatial resolution can make it difficult to detect small or low-contrast lesions and improve the quantitative accuracy of PET. Image quality is often further degraded by degradation factors such as positron range, non-collinearity of photon pairs, intrinsic system resolution limits, finite reconstructed voxel size, and patient motion. In addition, many factors related to data acquisition can affect PET image quality, such as variability in scanning procedure, patient weight and physiology, drug injection volume, and the length of time between injection and the start of the scan. Thus, image quality within and between patients can vary significantly. Large variability in image quality can make treatment assessment difficult, as this variability makes it difficult to compare scans acquired at different times and between patients.

[0012] To address the above-mentioned challenges, the method described herein applies Deep Learning (DL) network and Convolutional Neural Network (CNN) methods to obtain consistently high-quality images from PET data with high variability (e.g., different noise levels). The DL-CNN network described herein can generate output images with uniform image quality even when the input images exhibit significant variations in their image quality and statistical properties (e.g., due to their varying noise levels).

[0013] Noise suppression methods are subject to various imperfections that can be corrected or mitigated by the methods described herein. For example, image quality can be improved by regularization or post-reconstruction filtering methods.

[0014] Regularization methods incorporate a user-defined penalty function into an objective / cost function that is optimized to iteratively reconstruct an image of the activity level (e.g., tracer concentration) within each voxel (i.e., volume pixel) of the reconstructed image. The user-defined penalty function can, for example, emphasize local smoothness, thereby suppressing noise. Some regularizers can be tuned or optimized with the goal of distinguishing between noise and lesions (e.g., with the goal of suppressing noise without attenuating the signal coming from features of interest such as lesions), i.e., background noise is suppressed but lesions are left intact.

[0015] Post-reconstruction filtering methods can apply nonlinear filters such as linear Gaussian filters or non-local means or median filters to suppress noise and thereby improve the signal to noise ratio (SNR). However, this noise suppression typically comes at the undesirable expense of resolution (e.g., fine details and sharp boundaries may be blurred and details may be lost).

[0016] Compared with the regularized reconstruction and post-reconstruction filtering methods, the DL-CNN method described herein has several advantages. First, the noise texture is more consistent in the DL-CNN method described herein (making the comparison between PET images easier and clearer), which improves clinical reliability. Second, the DL-CNN method described herein improves robustness and reproducibility in clinical environments, because the method does not require parameter selection or tuning to be optimized by the user.

[0017] The method described herein also offers several advantages over related DL-CNN methods, for example, in which the CNN is trained to enable faster convergence to the reconstructed image, but does not improve the image quality of the reconstructed image.

[0018] In the second related DL-CNN method, the CNN is trained on low-quality images with a uniform level of noise. Thus, the CNN is optimized to be effective for filtering images with the same level of noise as the training images, but may perform less well when filtering images with a different noise level than the training data. Furthermore, these CNNs may be susceptible to loss of low-contrast fine structure information, because the large contribution to the error function of many voxels in the background, mainly due to smoothing, can overwhelm the small contribution to the error function of a few voxels in the region of interest (e.g., a lesion region that shows fine details and requires high resolution). This is because the second related DL-CNN method described above lacks a mechanism to rebalance the loss function used to train the CNN to more strongly weight / emphasize the signal in the region of interest, which is a relatively large background region, in order to train the network to preserve small features and fine details in the image.

[0019] A third related CNN method uses CNN as a priori information in the reconstruction process, rather than as a filter applied after reconstruction. In this method, optimizing the cost function is difficult, and the optimization method does not guarantee convergence. Parameter selection is also a challenge (as in the regularization-based method mentioned above), making the method less reliable and robust.

[0020] In contrast to the related DL-CNN methods described above, the DL-CNN method described herein makes the network more robust to variations in noise levels of the input PET images. As described above, these variations in noise levels can occur due to variability in acquisition and reconstruction procedures. Furthermore, the method described herein produces consistent results with consistent image quality when the noise levels of the input PET images are in the range used in training the DL-CNN network. This homogeneous and consistent image quality of the output PET images reduces intra- and inter-patient image variations, leading to better diagnostics and clinical outcomes. The method described herein utilizes a computationally efficient 2.5D orthogonal training and denoising method, resulting in a reduced number or severity of artifacts in the coronal or sagittal views. Finally, the method described herein preserves good contrast of small lesions and provides improved image resolution. Such improved image resolution is achieved, for example, by using feature-oriented training methods and utilizing anatomical information and other types of medical images (e.g., Magnetic Resonance Imaging (MRI) and X-ray Computed Tomography (CT)) in training and applying DL-CNN networks.

[0021] The above-mentioned improvements are achieved in various ways by the function of the method described below. For example, the method described herein applies the DL-CNN method to improve PET images by suppressing noise while retaining lesion information. The method is adjusted to automatically adapt to different noise levels in the input images while producing output images of similar quality, without the need to fine-tune the denoising method with adjustable parameters (e.g., fine-tuning that must be done by a user or that requires statistical analysis of the input images) to adapt the denoising method to the variations in the input images. Rather, since the input images have a wide range of statistical characteristics (e.g., noise levels), after training the DL-CNN network according to the method described herein, the trained DL-CNN can be used without modification or adjustable parameters, i.e., the processing unit 870 as a processing unit can output a denoised high-quality PET image 253 without adjustment based on the statistical characteristics of the low-quality PET image 255 reconstructed by the data acquisition system 876 as a reconstruction unit, as described below in FIG. 1, thereby reducing the intra- and inter-patient variations in image quality. That is, the reconstructed images with multiple noise levels used as input images have statistical characteristics such as noise levels within a predetermined range, and data with various statistical characteristics can be used for learning, so the image quality of the denoised image is less affected by the statistical characteristics compared to a case where the statistical characteristics of the reconstructed images with multiple noise levels used for learning are assumed to be the same value. That is, the processing device 870 as a processing unit outputs a denoised high-quality PET image 253 with a constant image quality, regardless of the noise level of the reconstructed low-quality PET image 255.

[0022] The above improvements are achieved in various ways by using the following features: (i) the CNN is trained in a manner such that the image denoising process performed by the CNN is largely independent of the magnitude of the noise level in the input image, (ii) a CNN architecture that applies 2.5D orthogonal training and denoising methods (as opposed to purely 2D or 3D methods), (iii) a feature-focused training method that focuses on preserving small and low contrast signals, and (iv) the CNN is trained using multi-modality training and denoising for partial volume effect correction.

[0023] Reference will now be made to the drawings, in which like reference numbers refer to like or corresponding parts. Figure 1 shows a flowchart of a method 100 for training a DL-CNN network 162 and a method 200 for applying a low-quality (e.g., noisy) PET image 255 to the trained DL-CNN network 162 to generate a high-quality (e.g., denoised) PET image 235. Figure 2 shows a flowchart of the method 100 and the method 200 according to another embodiment.

[0024] In the method 100, the loss function is used to iteratively adjust the parameters of the DL-CNN network (e.g., weights and biases of the convolutional layer and the pooling layer) until a stopping criterion is met (e.g., convergence of the parameters to a predetermined threshold) to generate a trained DL-CNN network 162. That is, the processing unit as the processing device 870 performs learning by simultaneously minimizing the loss function for multiple reconstructed images of a first noise level, and performs learning while iteratively adjusting the adjustable parameters. The loss function compares the high-quality data 153 as a reconstructed image of a first noise level with the results of the current version of the DL-CNN network (to which low-quality data 155 as a reconstructed image of multiple noise levels having a noise level higher than that of the reconstructed image of the first noise level is applied). In the case of PET imaging, the high-quality data and the low-quality data are reconstructed PET images with high / good image quality and low / poor image quality, respectively. As shown in FIG. 1, the DL-CNN network 162 is trained using various samples of low-quality data with various noise levels, thereby making it robust to changes in noise level. In general, the signal-to-noise ratio (SNR) is smaller when a smaller data set is utilized (e.g., due to shorter scan times or other factors resulting in fewer coincidence counts) when reconstructing an image. Thus, first, a high quality image 153(1) can be generated using all of the coincidence counts from the PET scan of a first patient (i.e., Patient 1), which generates a PET image of the highest image quality possible. Lower quality images 155(1,1), 155(1,2), ..., 155(1,k) reconstructed from the scan of Patient 1 can then be generated using various subsets of the coincidence count data selected from the entire data set. Each of these lower quality images corresponds to a different number of counts, thus allowing images with a range of noise levels to be reconstructed.Similarly, various noise levels can be obtained from the subsets of the entire dataset generated from the PET scans of patient 2 and all other patients up to the last patient, with low quality images 155(2,1), 155(2,2), ..., 155(2,k) (i.e., for patient L, its low quality images are 155(L,1), 155(L,2), ..., 155(L,k) and the high quality image is 153(L)).

[0025] Although not required, the training method 100 may incorporate therein other medical images 158 generated from medical imaging scans using another medical imaging modality (e.g., X-ray computed tomography (CT) or magnetic resonance imaging (MRI)). The other medical imaging scans performed may be performed simultaneously in coordination with the PET scan or in close temporal proximity to the PET scan. The other medical imaging scans may also be utilized to advantage for obtaining attenuation models and for enhancing resolution.

[0026] For example, in a PET / MRI scanner, the other medical images 158 may be MRI images. In Figure 1, MRI image 158(1) is obtained for patient 1 and corresponds to high quality PET image 153(1). Similarly, MRI image 158(2) is obtained for patient 2 and corresponds to high quality PET image 153(2), and MRI image 158(L) is obtained for patient L and corresponds to high quality PET image 153(L).

[0027] In another example of a PET / CT scanner, the other medical image 158 may be a computed tomography (CT) image, i.e., the reconstructed images of the multiple first noise levels may be magnetic resonance images or X-ray CT images.

[0028] After generating the trained DL-CNN network 162, the high-quality PET image 253 is generated by applying the DL-CNN network 162 trained using method 200, regardless of the noise level of the low-quality PET image 255 reconstructed from the PET emission data 251. That is, the method of step 110 of training the DL-CNN network 162 using low-quality images with a wide range of noise levels may reduce the dependency of the image quality of the high-quality PET image 253 on the noise level of the low-quality PET image 255, as compared to other training methods in which the low-quality images all have similar noise levels.

[0029] In some embodiments, the PET emission data 251 can be pre-processed counts in step 202 (e.g., signal pre-conditioning, position correction, energy correction, etc.) and then in step 204, the pre-processed count data can be used to reconstruct an image of radioactivity level (tracer concentration) as a function of voxel position.

[0030] In method 200, PET emission data 251 is corrected in step 202, and then a low-quality PET image 255 is reconstructed from the corrected emission data using a PET image reconstruction process in step 204. That is, a data acquisition system 876 as a reconstruction unit reconstructs low-quality PET image 255 from PET emission data 251 representing coincidences of each pair of gamma rays resulting from an annihilation event, acquired based on detection by a plurality of detection elements, and the data acquisition system 876 as an acquisition unit acquires the reconstructed low-quality PET image 255.

[0031] In step 202, various calibration and geometric factors may be utilized to correct the emission data. For example, pre-processing may include detector corrections and gains, corrections for variations in quantum efficiency in the detector, etc. Furthermore, these corrections may be made based on calibration data, empirical parameters, and known parameters.

[0032] In step 204, image reconstruction can be performed using backprojection, filtered backprojection, Fourier transform based image reconstruction, sequential image reconstruction, matrix inversion image reconstruction, statistical image reconstruction, list mode, or other reconstruction methods understood by those skilled in the art.

[0033] In step 210, noise is removed from the reconstructed image using the DL-CNN network 162. As a result, a high-quality PET image 253 is obtained. In this manner, the high-noise PET image resulting from the PET reconstruction in step 204 can be processed by using a DL denoising algorithm that applies the network generated by the offline DL training method 100. That is, the processing device 870 as a processing unit inputs the low-quality PET image 255 acquired by the data acquisition 876 as an acquisition unit to the DL-CNN network 162 as a trained model to obtain a high-quality PET image 253 that is a denoised PET image. Since the DL-CNN network 162 is trained using the low-quality images 155 with various noise levels, the high-quality PET image 253 should be created with a uniformly high image quality regardless of the noise level of the image reconstructed from step 204. That is, the PET image noise of the reconstructed image obtained from step 204 can be affected by many factors such as the amount of drug injected, the size of the patient, the patient's physiological function, the waiting time from injection to the start of the scan, the reconstruction parameters, etc. In fact, any of these factors alone can be sufficient to affect the noise level of the resulting PET image. Furthermore, the noise distribution may also vary spatially within a given PET image. Nevertheless, thanks to the training method in step 110, high quality PET images 253 of uniform quality are generated, allowing improved clinical comparison between generated PET images using a combination of methods 100 and 200 described herein.

[0034] Related methods first require statistical analysis of low-quality PET images 255 to predetermine the noise level of the input image, and then adjust the reconstruction method to compensate for the given noise level. A better method compared to these related methods is to train the DL-CNN network 162 using reconstructed images with various noise levels, as applied in the method described in this application. This method makes the DL-CNN network robust to variations in the noise level of the input images. Otherwise, training the DL-CNN network with low-quality images with a fixed noise level may lead to poor denoising results or artifacts when applied to images with a noise level different from that used for training.

[0035] In some embodiments, to achieve noise-adaptive denoising (i.e., consistent denoising effect under various noise levels and statistical characteristics of reconstructed images resulting from patient and PET scan / procedure variability), multi-noise level training of DL-CNN network 162 is performed as described below. For each complete PET list-mode training data set (e.g., each complete PET list-mode training data set corresponds to each of the L patients shown in FIG. 1), the complete data set is re-allocated into smaller subsets of the complete data set, with each subset having a predefined percentage / amount of the complete data set. This re-allocation generates training data sets of various dimensions used to reconstruct low-quality PET images that are utilized to train DL-CNN network 162. For example, each complete PET list-mode training data set may be re-allocated into a wide range of count levels (e.g., noise levels 1, 2, ... K shown in FIG. 1, which correspond to 10%, 20%, ..., 90% of the complete data set). This reassignment can be accomplished, for example, by uniform / random sampling of the complete listmode data set to generate the subsets.

[0036] Each of these subsets can then be used to reconstruct a PET image. That is, the data acquisition system 876 as a reconstruction unit reconstructs the low-quality PET image 155 as each of the multiple noise level reconstructed images using different amounts or percentages of the subset of the data set used to reconstruct the high-quality PET image 153 as the first noise level image. In some embodiments, the reconstruction can be standardized by using the same reconstruction method and parameters to reconstruct each of the low-quality images corresponding to the various noise levels (i.e., the various noise levels do not result from variations in the reconstruction process but from differences in the sizes of the subsets from which they are reconstructed). The data acquisition system 876 as a reconstruction unit may also reconstruct the low-quality PET image 255 using the same reconstruction method used to reconstruct the high-quality PET image 153 as the first noise level reconstructed image and the low-quality PET image 155 as the multiple noise level reconstructed image. In some embodiments, the high-quality image can be reconstructed using the complete list mode data set and using the standardized reconstruction method and parameters used to reconstruct the low-quality image. In such cases, the discrepancy between high quality and low quality images of a given scan / patient can be attributed primarily to noise introduced by the reduced subset size.

[0037] After acquiring the training data as described above, the DL-CNN network is trained by adjusting various optimization parameters to minimize a loss function calculated between the training data pairs for each scan / patient. That is, the DL-CNN network 162 as a trained model is trained by optimizing the loss function to minimize the difference between a high-quality PET image 153 as a reconstructed image with a first noise level and a low-quality PET image 155 as a reconstructed image with multiple noise levels higher than the reconstructed image with the first noise level. Each of these pairs includes a high-quality image of a scan / patient and a corresponding low-quality image of the same scan / patient (i.e., for patient 1, these pairs can be {153(1), 155(1,1)}, {153(1), 155(1,2)}, ..., {153(1), 155(1,k)}). Because the DL-CNN network 162 is trained to take advantage of a wide range of noise levels in low quality images, the trained DL-CNN network 162 can be expected to consistently produce high quality PET images without the need for adjustable parameters, regardless of the noise level of the input image when using the trained DL-CNN network 162 in step 210. That is, all of the adjustments and tuning are done in step 110, and thus in step 210 the trained DL-CNN network 162 can be utilized without additional tuning or fine-tuning to match the noise level of the input image 255.

[0038] Here, a first non-limiting example of the embodiment of the methods 100 and 200 is presented. In this first example, a deep residual learning framework is applied for image denoising. Let x be a clear image and y be a noisy observation degraded by an additive noise n (i.e., y=x+n). The goal of the deep residual network is to estimate the noise n (i.e., F(y)≈n) from the noisy input image y, where F is the residual mapping process. Then, a denoised image x' can be obtained by subtracting the estimated noise F(y) from the reconstructed image y (i.e., x'≈yF(y). In some embodiments, the loss function can be formulated as follows:

number

[0039] Here, Θ denotes trainable weights, Ψ is an error function (e.g., Mean Square Error (MSE) is used in the first example), N is the number of training samples, y is the noisy image, and x is the target image. In the first example, stochastic gradient descent is used to minimize the loss function, and an eight-layer network is used to provide a good balance between performance and training efficiency. An example of the first deep residual network architecture is shown in Table 1. In Table 1, “Conv” denotes convolutional layer, “BN” denotes batch normalization, and “ReLU” denotes rectified linear function. In the first example, eight scans of the training dataset last for 14 minutes each. These complete datasets were used to generate high-quality images. Uniform reassignment of these complete datasets was also used to generate subsets corresponding to scans with lengths of 60, 120, 180, 240, 300, and 420 seconds, and these scans were used to generate low-quality images. These training datasets were then used to train the network and reconstruct images using Ordered-Subset Expectation Maximization (OS-EM). [Table 1]

[0040] To test / validate the DL-CNN network 162, it was then applied to OS-EM reconstructed images acquired from 2-, 3-, and 4-minute long PET scans, and the results were then compared to denoising using Gaussian filters with widths of 4, 6, and 8 mm at full-width half maximum (FWHM), respectively. Figures 3A and 3B show graphs of "contrast recovery" plotted against the "coefficient of variation" of two distinct regions in OS-EM reconstructed images from 2-, 3-, and 4-minute long PET scans after denoising (Figure 3A shows the lung region, and Figure 3B shows the liver region). Figure 3A shows a plot of contrast recovery (vertical axis) of the lung region versus liver coefficient of variation (horizontal axis) in denoised PET images (i) by using a DL-CNN network according to one embodiment, and (ii) by using a Gaussian filter. These PET images correspond to acquisition times of 2, 3, and 4 minutes. FIG. 3B also shows a plot of contrast recovery (vertical axis) versus liver coefficient of variation (horizontal axis) for liver regions in PET images denoised (i) by using a DL-CNN network according to an embodiment, and (ii) by using a Gaussian filter. These PET images correspond to acquisition times of 2, 3 and 4 minutes. Images denoised using the DL-CNN network 162 (shown using triangular symbols) tend to be more densely packed than images denoised using a Gaussian filter (GF) (shown using circular symbols), thus confirming that the method based on the DL-CNN network tends to be more robust to variations in the noise level of the reconstructed images. Images denoised using the method based on the DL-CNN network also tend to have better image quality than images denoised using a Gaussian filter.

[0041] FIG. 4 illustrates a 2.5D process 400 that can be utilized when performing DL-CNN network-based filtering / denoising (i.e., in steps 110 and 210). That is, FIG. 4 illustrates a flowchart of a process 400 for applying a 2.5-dimensional (2.5D) DL-CNN to a three-dimensional (3D) PET image according to an embodiment. In the process 400, the DL-CNN network 162 can include three networks that are respectively applied to two-dimensional (2D) slices of the reconstructed three-dimensional (3D) PET image, each of which corresponds to a different orientation. That is, a transaxial network 422 can be applied to a 2D slice parallel to the transaxial plane of the 3D PET image. Similarly, a coronal network 424 can be applied to a 2D slice parallel to the coronal plane of the 3D PET image, and a sagittal network 426 can be applied to a 2D slice parallel to the sagittal plane of the 3D PET image.

[0042] In step 410 of process 400, the low quality PET images are sliced ​​according to their respective planes / orientations.

[0043] In step 420 of process 400, each DL-CNN network 422, 424, and 426 is applied to the slice at its respective plane to generate a denoised image 430.

[0044] In step 440 of process 400, the denoised images are combined to form an aggregate image. For example, three denoised images 430 can be averaged to obtain a single 3D denoised image.

[0045] This 2.5D method offers several advantages over both 2D and 3D methods for denoising low-quality PET images. In contrast to the 2.5D methods, the 3D methods are computationally slower and more computationally intensive. Furthermore, pure 2D methods, which only handle 2D slices parallel to a single plane, do not reduce artifacts and lack the use of spatial information parallel to other planes.

[0046] While PET imaging provides 3D volumetric information of radiotracer uptake in the body, other DL-CNN methods deal with 2D images, which may result in streaking artifacts on the 3D volumetric database. For example, when DL-CNN deals with transaxial observations, it ignores voxel-voxel correlations in the coronal and sagittal directions, resulting in streaking artifacts in the coronal and sagittal observations. These artifacts can be addressed by employing a CNN network that utilizes full 3D convolutions, but such full 3D convolutions in DL-CNN are computationally expensive in terms of both training and denoising.

[0047] Thus, the process 400 utilizes a 2.5D solution that also utilizes 2D convolutions in the DL-CNN network 162, but the 2D slices in the 2.5D solution are not limited to being parallel to a single plane.

[0048] In some embodiments, the process 400 trains three orthogonal networks using training samples sliced ​​in each of the orthogonal observations (e.g., transaxial, coronal, and sagittal). That is, the processing unit 870 as a processing unit inputs a reconstructed image including three-dimensionally arranged voxels acquired by the data acquisition system 876 as an acquisition unit and outputs a denoised image to a trained model that inputs an image and outputs a denoised image. The learned model includes, as DL-CNN, a transaxial network 422 as a first network that inputs a slice image group (transaxial slice image group) in a first direction (transaxial direction) and outputs a corresponding first denoised image group (image corresponding to the transaxial plane), a coronal network 424 as a second network that inputs a slice image group (coronal slice image group) in a second direction (coronal slice direction) perpendicular to the first direction and outputs a corresponding second denoised image group (image corresponding to the coronal plane), and a sagittal network 426 as a third network that inputs a slice image group (sagittal slice image group) in a third direction (sagittal slice direction) perpendicular to the first direction and outputs a corresponding third denoised image group (image corresponding to the sagittal plane). Slice data in each direction can be processed by the corresponding network. These three denoised 3D volumes are then combined (e.g. averaged) to obtain the final result, i.e. high quality PET image 253. That is, the processing unit 870 as a processing part combines the first set of denoised images with the second set of denoised images with the third set of denoised images, e.g. by performing an averaging operation between the first set of denoised images with the second set of denoised images with the third set of denoised images, to output the high quality image 450 as an output image. When the process 400 is used, it is used in both the step of training data 110 and the step of denoising data 210.

[0049] However, the embodiment is not limited to the case where three denoised 3D volumes are combined to obtain a final result. These denoised 3D volumes have clinical value even before the combination process. Therefore, these denoised 3D volumes before the combination process may be displayed on the display as a final result.

[0050] The synthesis process is not limited to a simple average operation. Other operations such as a weighted average may also be used as the synthesis process. In addition, in performing the synthesis process, instead of performing the synthesis process using three denoised image groups, the synthesis process may be performed using, for example, two denoised image groups. For example, the processing device 870 as a processing unit may generate an output image by synthesizing a first denoised image group and a second denoised image group.

[0051] 5 and 6 show a feature-oriented method for improving the training of the DL-CNN network 162. Although the method 100 can be performed using unlabeled training data, denoising can be better achieved in the method 200 if the training data is segmented and labeled / weighted to indicate regions / features in the PET images that have higher clinical importance (e.g., regions of lesions). Thus, the trained network can improve image quality in segmented regions deemed to have higher clinical importance by weighting the loss function and generating denoised images that specifically match high quality images in the regions of interest (e.g., lesions).

[0052] The advantages of the supervision process 500 and the modified training step 110' (corresponding to a modified version of step 110) can be better understood by considering that denoising training does not generally require labeled data (e.g., segmenting and labeling the target object). This is because uniform weighting can be applied as a default value to all voxels used in training. Here, training is performed to minimize the discrepancy between the denoised low-quality image (i.e., the low-quality image after the low-quality image is processed using the DL-CNN network) and the high-quality image. This training is performed by adjusting various network parameters to minimize a loss function that applies equal weights to all voxel-to-voxel differences on an image patch. However, because signals of interest in PET images are often in spatially small regions compared to the background (e.g., lesion areas can be much smaller than background areas), a large number of background voxels can dominate the contribution of the loss function, leading the network toward a solution that overemphasizes smoothing to remove the noisy background and underemphasizes preserving small features and fine details in the smaller region of interest.

[0053] To counter this tendency, during training in step 110, a feature-focused denoising technique can be used to increase the weights applied to the lesion regions. As shown in FIG. 5, process 500 includes a segmentation step 510 to create a lesion mask and a weight calculation step 520 to create a weight map for each high quality image in the training data set. In some embodiments, the weight map can be created by manually segmenting the target lesions in the training data set. In some embodiments, the weight map can be automatically segmented (e.g., using threshold and region growing techniques) and then user input can be used to identify which segmented regions are background and which are regions of interest. That is, the processor 870 as a processing unit generates a region of interest mask based on the user input. The weight map can have higher values ​​in the lesion and lower values ​​in the background, and the weight map for patches that do not contain lesions can be uniform. The weight map is used to compensate for the difference in the number of voxels in the signal / lesion and background regions. This allows a balance between the competing demands of suppressing background noise while retaining the desired small features.

[0054] In step 510 of process 500, lesions are identified and manually segmented to obtain a lesion mask (also called a region of interest mask) corresponding to a given high-quality PET image, i.e., the processing unit 870 as a processing section generates a region of interest mask that distinguishes the region of interest from the background based on at least one of the low-quality PET image 155 as a low noise level image and the high-quality PET image 153 as a high noise level image.

[0055] In step 520 of process 500, these masks are used to generate a distance-dependent weight map (e.g., lesion voxels are weighted 1 and far away voxels are weighted 0.1). That is, the processor 870 as a processor generates the distance-dependent weight map based on the region of interest mask generated in step 510. In some embodiments, the weights can be gradually decreased to blur the boundary between lesion voxels and background voxels, thereby representing segmentation errors / uncertainties. That is, the processor 870 as a processor generates the weight map by applying a smoothing filter to the region of interest mask generated in step 510.

[0056] Next, in a modified training step 110', the weight map is used to compute a weighted loss function:

number

[0057] Here, w is a weight map, and the symbol "x" represents inter-pixel multiplication. That is, in the modified training step 110', the trained model obtains a DL-CNN neural network 162' including weight coefficients for connections of multiple layers between the input layer and the output layer. The DL-CNN network 162' receives as training data the low-quality PET image 155 as a low-noise level image, a weight map in which a voxel of the low-noise level image identified as being within a region of interest is assigned a larger weight value than the weight value assigned to the voxel of the low-noise level image, and a high-noise level image corresponding to the low-noise level image. The DL-CNN network 162' is trained by optimizing a loss function by multiplying the difference between the output obtained by inputting the low-quality PET image 155 as a low-noise level image and the high-quality PET image as a high-noise level image into the neural network by the value of the weight map generated in step 520.

[0058] A second non-limiting example now illustrates the advantages of using process 500 when training DL-CNN network 162'. DL-CNN network 162' can be the same as DL-CNN network 162, and the process of step 210 is applied in the same manner, except that DL-CNN network 162' is trained using a weight map rather than uniform weighting in the computation of the loss function.

[0059] The five-layer residual network shown in Table 2 was trained on simulations and phantoms and then applied to the noisy reconstruction of the NEMA body phantom (shown in the left side of Figure 7), the latter of which was reconstructed using the OS-EM reconstruction method. [Table 2]

[0060] Figure 7 shows the reconstructed image on the left and the residual image on the right for the second example. The evaluated residual image is shown on the right side of Figure 7.

[0061] Figure 7 shows that the smallest sphere (10 mm) was misclassified as noise and was largely removed / subtracted from the denoised image. This is due to the fact that this lesion contains only a few voxels and the feature is not visually evident. Thus, in the absence of sufficient training examples of images with a similarly small feature shown in the signal (as opposed to simply in the noise), the convolutional neural network works to map it as noise, as doing so reduces the loss function globally.

[0062] In a second example, the process 500 and method 110' utilize a feature-focused learning technique that assigns different weights to different voxels to compensate for the discrepancy between the number of voxels in the region of interest and the number in the background. In creating the weight map, the lesion / region of interest is first segmented (e.g., manually or by thresholding) in the target image to create a lesion mask as shown in FIG. 5. N b / N l Create a weight map from the lesion mask by assigning N times higher weights, where N b and N l are the total number of voxels in the background and the total number of voxels in the lesion, respectively. In other embodiments, other ratios and methods can be used to select the relative weight values ​​between the background and the region of interest, as will be appreciated by those skilled in the art, without departing from the spirit of the weight mask. Background voxels can be set to 1 in the weight map. The weight map is then convolved with a Gaussian kernel to adjust for segmentation error. The weight map can be uniform for patches that do not contain lesions. The weight map forces the network to learn to suppress background noise while preserving the desired small features.

[0063] To evaluate the improvement achieved by using the weight maps described above, the network was trained with a training data set including images of a computer-simulated phantom and images of two physical phantoms. The simulated phantom images were created from activity maps with spheres of different dimensions and contrasts, and different acquisition times (e.g., 1, 2, 3, and 4 minutes) were simulated to generate different noise levels. A 10-minute acquisition simulated high-quality target images. The training data set included a total of 12 data sets. Two networks were trained using the same architecture and training data set: (i) method 110, i.e., DL-CNN trained using uniform weighting (i.e., no weight map), and (ii) method 110', i.e., DL-CNN trained using the weights described above. For the DL-CNN trained using the weight maps, the lesion mask was created by convolution with a Gaussian kernel after thresholding was applied to the target image.

[0064] To evaluate the improvement from weight map applied training, the two trained networks described above were applied to an OS-EM reconstructed image of a standard NEMA phantom (2 mCi) scanned for 2 minutes (the NEMA phantom was not included in the training dataset). The OS-EM reconstructed image without denoising is shown in Figure 8A. Figures 8B and 8C show the same image after denoising by a DL-CNN network trained using uniform weighting in the loss function and a DL-CNN network trained using various weights provided by the weight map, respectively.

[0065] The denoised images were assessed by measuring the coefficient of variation of the contrast recovery of each sphere and the background. The results of this analysis are shown in Table 3. Table 3 shows the results of the coefficient of variation analysis of the denoised images obtained by denoising the OS-EM reconstructed images by applying a DL-CNN network that utilizes either uniform weighting or differential weighting determined by a weight map. [Table 3]

[0066] For large features, such as those corresponding to a 22 mm ball, the features are preserved in the denoised image regardless of whether the training data is weighted or uniform. That is, the coefficient of variation for denoising by a DL-CNN network is largely independent of whether the DL-CNN network was trained using uniform weights or different weights provided by a weight map. However, DL-CNN networks trained using different weights in a weight map do a better job of preserving fine features in the denoised image (e.g., the feature corresponding to a 10 mm ball). In contrast, DL-CNN networks trained using uniform weights tend to identify these fine features as noise and largely filter them out of the denoised image.

[0067] The correction training method 110'' will now be described with reference to FIG. 9. In some embodiments, the methods 100 and 200 can utilize other medical images. Medical imaging of other modalities can often achieve images with higher resolution than the corresponding PET images. These other medical images can be utilized to correct for partial volume effects, which occur when a voxel in a PET image is at the boundary between two different materials (e.g., straddling the boundary between a lesion (having a high level of activity) and a surrounding organ (having a lower level of activity)). The other medical images can be utilized to correct for partial volume effects, if they have a higher resolution than the PET image and can be utilized to identify two (or more) materials / organ parts that partially occupy a given voxel in the PET image. The type of medical image utilized can be any type of other medical image. By way of non-limiting example, the other medical images are assumed to be CT images in this application (unless otherwise stated).

[0068] FIG. 9 shows a flow chart of a modified embodiment of the methods 100 and 200, in which step 110'' trains a modified network (DL-CNN network) 162'', and step 210'' applies low-quality PET images 255 and low-quality CT images 258 to the modified network (DL-CNN network) 162''. That is, step 110'' and step 210'' utilize a combination of low-quality PET images and low-quality CT images (or other non-PET images) to generate high-quality PET images with PVC 253''. As mentioned above, in PET imaging, image quality may be degraded due to partial volume effect (PVE), e.g., due to the inherent resolution limit of the system. That is, coarse graining due to large voxels in the PET image may lead to PVE, which may cause activity spillover effects into neighboring regions, reducing image resolution and contrast. To mitigate PVEs, the correction methods 100 and 200 introduce high-resolution anatomical information acquired from CT (or MRI) imaging into the DL-CNN network 162'' to correct PVEs in the PET images. As shown in FIG. 8, a high-resolution CT (MRI) image can be acquired and reconstructed in tandem with the PET image. The reconstructed CT (MRI) image can then be projected forward using the PET system geometry, and a second CT (MRI) image can then be reconstructed from the forward projection image using the same PET reconstruction procedure. As a result, a CT image at PET resolution (i.e., a resolution-degraded CT image) is reconstructed and shown as low-resolution CT image 158 in FIG. 9. Alternatively, the resolution-degraded CT image 158 can be approximated using the attenuation map used in the PET reconstruction.

[0069] In step 110'', the low quality PET image 155 and the low resolution CT image 158 are dual channel inputs applied to the DL-CNN network 162'', and a loss function is calculated by comparing the low quality PET image 155 and the low resolution CT image 158 with the respective high quality PET image 153 and high resolution CT image 151. The objective is to train the network to remove noise from the low / high quality PET pair and to enhance resolution from the low / high resolution CT pair.

[0070] In step 210'', the trained DL-CNN network 162'' is applied to a combination of low-quality PET images and low-resolution CT (MRI) images acquired in a clinical setting to obtain high-quality PET images 253 with PVC''.

[0071] A more detailed description of step 110 is now provided. This description can be generalized to modified versions of step 110 (e.g., step 110' and step 110'') as understood by those skilled in the art. FIG. 10 shows a flow chart of one embodiment of the training step 110 performed in the method 100. In step 110 of the method 100, a DL-CNN network is obtained, which is the output from step 124, by utilizing low-quality (e.g., noisy) data 155 and high-quality (e.g., optimized) data 153 as training data to train the DL-CNN network. Here, the term "data" may refer to an image. More generally, the data 155 may also be referred to as defect-containing data. In this case, the "defect" may be any undesirable feature (e.g., noise or architecture) that can be affected by image processing. Similarly, the data 153 may also be referred to as defect-reduced data, defect-minimized data, or optimized data. In this case, the degree of "defect" is lower than the defect of the data 155. In an example utilizing reconstructed images of data 155 and 153, the offline DL training method 100 trains a DL-CNN network 162 using a number of noisy constructed images 155 that are paired with corresponding high-quality images 153. The goal is to train the DL-CNN network 162 to create images from the noisy reconstructed images that are similar to the high-quality images.

[0072] In step 110, a set of training data is obtained. The DL-CNN network 162 is then sequentially (iteratively) updated to reduce the error (e.g., a value calculated by a loss function) so that the noisy data 115 processed by the DL-CNN network closely matches the optimized data 153. In other words, the DL-CNN network estimates the mapping suggested by the training data, and then the cost function creates an error value for the discrepancy between the optimized data 153 and the denoised data created by applying the current state of the DL-CNN network 162 to the noisy data 115. For example, in some embodiments, the cost function may utilize a mean squared error to minimize the mean squared error. In the case of a Multilayer Perceptrons (MLP) neural network, an error backpropagation algorithm may be used to train the network by minimizing a cost function based on the mean squared error using (stochastic) gradient descent.

[0073] Training a neural network model essentially means selecting one model from a set of allowed models (or in a Bayesian framework, determining the distribution of the set of allowed models) that minimizes a cost criterion (i.e., an error value calculated using a cost function). In general, DL-CNN networks can be trained using any of a number of algorithms for training neural network models (e.g., by applying optimization theory and statistical evaluation).

[0074] For example, the optimization method used in training the DL-CNN network 162 may use some type of gradient descent method that includes backpropagation to calculate the actual gradient. This is done by differentiating the cost function with respect to the network parameters and then modifying these parameters in the gradient-related direction. The backpropagation training algorithm may be a steepest descent method (e.g., variable learning rate, variable learning rate and momentum, and elastic backpropagation), a quasi-Newton method (e.g., Broyden-Fletcher-Goldfarb-Shanno, one-step secant, and Levenberg-Marquardt), or a conjugate gradient method (e.g., Fletcher-Reeves updated, Polak-Ribiere updated, Powell-Beale restarted, and scaled conjugate gradient). Evolutionary methods such as gene expression programming, simulated annealing, expectation maximization, nonparametric methods, and particle swarm optimization may also be used for training the DL neural network 162.

[0075] 10 illustrates a non-limiting example of a flowchart for performing step 110 of the method 100 for training a network using training data. Data 115 in the training data can be noisy images or images exhibiting artifacts. For example, artifacts can result from a particular method of reconstruction or from the method used to acquire the emission data.

[0076] In step 112 of step 110, an initial guess is generated for the coefficients of the DL-CNN network 162. For example, the initial guess can be based on a priori knowledge of the region being imaged, one or more typical denoising methods, boundary detection methods, or motion detection methods, and the initial guess can be based on one of LeCun initialization, Xavier initialization, and Kaiming initialization.

[0077] Steps 114 to 124 show a non-limiting example of an optimization method for training a DL-CNN.

[0078] In step 114 of step 110, an error is calculated (e.g., using a loss or cost function) that represents the magnitude of difference (e.g., distance trajectory) between the optimized data 153 (i.e., ground truth data) and the noisy data 115 after applying the current version of the DL-CNN network 162. The error may be calculated using a known cost function, including the cost functions described above, or distance trajectory between the image data. Additionally, in some embodiments, the error / loss function may be calculated using one or more of hinge loss and cross entropy loss.

[0079] Also, by combining the loss function with a regularization method, it is possible to avoid overfitting the network to the specific cases represented in the training data. Regularization can help prevent overfitting in machine learning tasks. If training is too long and the model is presumed to be expressive enough, the network learns the noise specific to the dataset, a situation called overfitting. In the case of overfitting, DL-CNN generalizes poorly and has high variance as the noise varies across datasets. The total error is minimized when the sum of the bias and variance is minimized. Therefore, it is desirable to reach a local minimum that explains the data in the simplest way possible, maximizing the probability that the trained network represents a general solution rather than a solution specific to the noise in the training data. This goal can be achieved, for example, by early stopping, weight regularization, Lasso regularization, Ridge regularization or elastic net regularization.

[0080] In some embodiments, backpropagation is used to train the DL-CNN network 162. Backpropagation can be used to train neural networks and is used in conjunction with gradient descent optimization methods. During the forward pass, the algorithm calculates predictions of the network based on the current parameters Θ. These predictions are then input into a loss function, which compares them with the corresponding ground truth labels (i.e., high quality images 153). During the back pass, the model calculates the gradient of the loss function with respect to the current parameters, and then updates the parameters by taking a step of a given size in the direction of minimizing loss (e.g., in accelerated methods such as Nesterov inertia and various adaptive methods, the step size can be chosen to converge faster and optimize the loss function).

[0081] The optimization method that performs the backprojection may utilize one or more of gradient descent, batch gradient descent, stochastic gradient descent, and mini-batch stochastic gradient descent. The optimization method may also be accelerated using one or more inertial update techniques in the optimization method to provide a faster convergence rate for stochastic gradient descent in deep networks. These techniques include, for example, adaptive techniques such as the Nesterov inertial technique or the Adagrad subgradient method, the Adadelta or RMSProp parameter update variations of the Adagrad method, and the Adam adaptive optimization method. The optimization method may also apply a quadratic method by incorporating the Jacobian matrix into the update step.

[0082] Forward and backward passes may be performed sequentially (iteratively) through each layer of the network. In the forward pass, execution begins by feeding inputs through the first layer, which generates output activations for the next layer. This process is repeated until a loss function is reached at the last layer. During the backward pass, the last layer computes gradients with respect to its own learnable parameters (if any) and with respect to its own inputs, which serve as the upstream derivatives of the previous layer. This process is repeated until the input layer is reached.

[0083] Returning to the non-limiting example shown in FIG. 10, step 116 of step 110 determines the change in error since a function of the change in the network can be calculated (e.g., the error gradient). This change in error can then be used to select the direction and step size of the next change to the weights / coefficients of the DL-CNN network 162. Calculating the error gradient in this manner is consistent with some embodiments of gradient descent optimization. In some other embodiments, this step can be omitted or replaced by another step that follows a different optimization algorithm (e.g., a non-gradient descent optimization algorithm such as simulated annealing or a genetic algorithm), as will be appreciated by those skilled in the art.

[0084] In step 118 of step 110, a new set of coefficients for the DL-CNN network 162 is determined. For example, the weights / coefficients can be updated using the changes calculated in step 116, such as in a gradient descent optimization method or an overrelaxation accelerated method.

[0085] In step 122 of step 110, the updated weights / coefficients of the DL-CNN network 162 are utilized to calculate new error values.

[0086] In step 124 of step 110, a predetermined stopping criterion is utilized to determine whether training of the network is complete. For example, the predetermined stopping criterion may evaluate whether the new error or the total number of iterations performed exceeds a predetermined value. For example, the stopping criterion may be met if the new error falls below a predetermined threshold or if a maximum number of iterations has been reached. If the stopping criterion is not met, the training process performed in step 110 continues back to the beginning of the iteration loop by returning to step 116 and repeating step 116 using new weights and coefficients (the iteration loop includes steps 116, 118, 122 and 124). When the stopping criterion is met, the training process performed in step 110 is complete.

[0087] 11 and 12 show flow charts of an embodiment of step 210. FIG. 11 is general to any kind of layers of a feed-forward artificial neural network (ANN), including, for example, a fully connected layer, while FIG. 12 is specific to convolutional and pooling layers in a CNN. Since DL-CNN may include both fully connected and convolutional and pooling layers, the result is a flow chart that is a combination of FIG. 11 and FIG. 12, as will be understood by those skilled in the art. The embodiment of step 210 shown in FIG. 11 and FIG. 12 also corresponds to applying the DL-CNN network 162 to each image of the training dataset in step 110.

[0088] In step 212, weights / coefficients corresponding to the connections between neurons (ie, nodes) are applied to each input corresponding to a pixel of the reconstructed image.

[0089] The weighted inputs are summed in step 214. If the only non-zero weights / coefficients connecting to a given neuron in the next layer are regionally confined in the image represented in the previous layer, then the combination of steps 212 and 214 is essentially equivalent to performing a convolution operation.

[0090] In step 216, a respective threshold is applied to the weighted sum of each neuron.

[0091] In process 218, the weighting, summation, and thresholding steps are repeated for each subsequent layer.

[0092] 12 shows a flow chart of another embodiment of step 210. The embodiment of step 210 shown in FIG.

[0093] In step 222, the convolutional layer is computed as described above and in accordance with the understanding of convolutional layers by those skilled in the art.

[0094] In step 224, the output from the convolutional layer is the input to a pooling layer, in accordance with the description of pooling layers above and in accordance with the understanding of pooling layers by those skilled in the art.

[0095] In process 226, the convolutional layer step followed by the pooling step can be repeated for a predetermined number of layers. Following (or alternating with) the convolutional and pooling layers, the output from the pooling layer can be fed to a predetermined number of ANN layers according to the description of the ANN layers in Figure 11. The final output is a reconstructed image with the desired noise-free / artifact-free characteristics.

[0096] 13A, 13B, and 13C show various examples of mutual connections between layers in DL-CNN network 162. DL-CNN network 162 may include fully connected, convolutional, and pooling layers, all of which are described below. In some preferred embodiments of DL-CNN network 162, convolutional layers are placed near the input layer, whereas fully connected layers (which perform high-level reasoning) are placed much further down the architecture in the direction of the loss function. It can be seen that pooling layers can be inserted after convolution and are a reduction that reduces the spatial extent of the filters and thus the amount of learnable parameters. Activation functions are also incorporated into various layers to introduce nonlinearity and enable the network to learn complex predictive relationships. The activation functions can be saturated activation functions (e.g., sigmoid or hyperbolic tangent activation functions) or regularized activation functions (e.g., the rectified linear unit (ReLU) applied in the first and second examples above). The layers of DL-CNN162 may also include batch normalization as shown in the first and second examples above.

[0097] FIG. 13A shows an example of a general artificial neural network (ANN) with N inputs, K hidden layers, and three outputs. Each layer is composed of nodes (also called neurons), which perform a weighted sum of the inputs and compare the result of the weighted sum with a threshold to generate an output. ANNs create a set of functions, where the members of the set are obtained by varying the thresholds, the connection weights, or architectural details such as the number of nodes and their connectivity. The nodes in an ANN are sometimes called neurons (or neuron nodes), and these neurons may have inter-connections between the various layers of the ANN system. The simplest ANN has three layers and is called an autoencoder. DL-CNN networks 162 can have more than three neuron layers and have as many output neurons X as input neurons. Nwhere N is the number of pixels in the reconstructed image. Synapses (i.e., connections between neurons) accumulate values ​​called "weights" (also interchangeably called "coefficients" or "weighting coefficients") that handle the data in this calculation. The output of an ANN depends on three types of parameters: (i) the interconnection pattern between the different neuron layers, (ii) the learning process for updating the interconnection weights, and (iii) the activation function that transforms a neuron's weighted input into its output activation.

[0098] Mathematically, a function m(x) of a network of neurons is a function of other functions n i (x) and other functions n i (x) may further be defined as a composition of other functions. This may be conveniently represented as a network structure with arrows representing dependencies between variables as shown in Figure 13. For example, an ANN may utilize a non-linear weighted sum, where m(x)=K(Σ i w i n i (x)), and K (commonly called the activation function) is some predefined function, such as the hyperbolic tangent.

[0099] In Figure 13A (and similarly in Figure 13B), neurons (i.e., nodes) are depicted as circles around a threshold function. For the non-limiting example shown in Figure 13A, inputs are depicted as circles around a linear function, and arrows indicate directed connections between neurons. In some embodiments, DL-CNN network 162 is a feedforward network as illustrated in Figures 2A and 2B (e.g., it may be represented as a directed acyclic graph).

[0100] The DL-CNN network 162 operates to accomplish a particular task, such as denoising a CT image. It solves the particular task in some optimal sense (e.g., the stopping criterion used in step 124 of step 110 described above). *∈F. For example, in some embodiments, this is done by searching through a set of power functions F to learn using a set of observations to find the optimal solution m * For C(m * This may be achieved by defining a cost function C:F→R such that m(m)≦C(m), ∀m∈F (i.e., no solution has a cost less than the cost of this optimal solution). The cost function C is the degree to which a particular solution is away from the optimal solution for the problem to be solved (e.g., error). The learning algorithm iteratively searches through the solution space to find a function with the smallest possible cost. In some embodiments, the cost is minimized over sample data (i.e., training data).

[0101] FIG. 13B illustrates a non-limiting example in which DL-CNN network 162 is a convolutional neural network (CNN). CNNs are a type of ANN that have beneficial properties for image processing and are therefore particularly useful for denoising images. CNNs utilize feed-forward ANNs in which connectivity patterns between neurons can represent convolutions in image processing. For example, CNNs can be used for image processing optimization by utilizing multiple layers of small neuron populations (called receptive fields) that process portions of the input image. The outputs of these populations can then be tiled so that they overlap to better represent the original image. This processing pattern can be repeated for multiple layers with alternating convolutional and pooling layers.

[0102] Figure 13C shows an example of a 4x4 kernel applied to map values ​​from an input layer representing a two-dimensional image to the first hidden layer (which is a convolutional layer). This kernel maps each 4x4 pixel region to a corresponding neuron in the first hidden layer.

[0103] Following the convolutional layers, the CNN may include local and global pooling layers that combine the outputs of the neuron clusters in the convolutional layers. In some embodiments, the CNN may also include various combinations of convolutional and fully connected layers, where pointwise nonlinearity is applied at the end of or after each layer.

[0104] CNNs have several advantages in terms of image processing. In order to reduce the number of free parameters and to promote generalization, convolution operations on small regions of the input are introduced. One important advantage of some embodiments of CNNs is the use of shared weights in the convolutional layers, which means that the same filter (weight bank) is used as the coefficient for each pixel in the layer. This reduces the memory occupied and improves performance. Compared to other image processing methods, CNNs advantageously use a relatively small amount of pre-processing. This means that the network is responsible for learning the filters, which is done manually in traditional algorithms. The main advantage of CNNs is that they do not rely on prior knowledge and manual effort in designing functions.

[0105] 14A and 14B show an example, non-limiting embodiment, of a PET scanner 800 in which methods 100 and 200 can be performed. The PET scanner 800 includes a number of Gamma-Ray Detectors (GRDs) (e.g., GRD1, GRD2 through GRDN), each configured as a rectangular detector module. In one embodiment, the detector ring includes 40 GRDs. In another embodiment, 48 GRDs are used, and more GRDs are used to create a larger aperture for the PET scanner 800.

[0106] Each GRD may contain a two-dimensional array of individual detector crystals. These absorb gamma radiation and emit scintillation photons. The scintillation photons may be detected by a two-dimensional array of photomultiplier tubes (PMTs) also disposed in the GRD. A light guide plate may be disposed between the array of detector crystals and the PMTs.

[0107] Alternatively, the scintillation photons may be detected by an array of silicon photomultipliers (SiPMs), and each individual detector crystal may comprise a respective SiPM.

[0108] Each photodetector (e.g., PMT or SiPM) can generate an analog signal indicative of the time when a scintillation event occurred and the energy of the gamma ray that caused the detection event. Furthermore, photons emitted from one detection crystal can be detected by multiple photodetectors, and based on the analog signal generated at each photodetector, the detection crystal corresponding to the detection event can be determined using, for example, Anger logic and crystal decoding.

[0109] FIG. 14B shows a schematic diagram of a PET scanner system with gamma-ray photon counting detectors (GRDs) arranged to detect gamma rays emitted from an object OBJ. The GRDs can measure the timing, location, and energy corresponding to each gamma-ray detection. In one embodiment, the gamma-ray detectors are arranged in a ring as shown in FIG. 14A and FIG. 8B. The detection crystals can be scintillator crystals with individual scintillator elements arranged in a two-dimensional array, and the scintillator elements can be any known luminescent material. The PMTs can be arranged such that light from each scintillator element is detected by multiple PMTs to enable Anger calculation / crystal decoding of scintillation events.

[0110] FIG. 14B shows an example of the configuration of a PET scanner 800. In this example, an object OBJ to be imaged is placed on a stage 816, and GRD modules GRD1 to GRDN are arranged circumferentially around the object OBJ and the stage 816. The GRD can be fixedly coupled to a circular component 820 that is fixedly coupled to a gantry 840. The gantry 840 houses many parts of the PET imaging device. The gantry 840 of the PET imaging device also includes an opening through which the object OBJ and the stage 816 can pass, and gamma rays emitted in opposite directions from the object OBJ due to annihilation events can be detected by the RGD, and the coincidence of gamma ray pairs can be determined using timing and energy information.

[0111] Also shown in FIG. 14B are electronic circuits and hardware for acquiring, storing, processing, and distributing gamma ray detection data. These circuits and hardware include a processor 870, a network controller 874, a memory 878, and a data acquisition system (DAS) 876. The PET imaging device also includes a data channel that sends detection measurements from the GRD to the DAS 876, a processor 870, a memory 878, and a network controller 874. The data acquisition system 876 can control the acquisition, digitization, and transmission of detection data from the detector. In one embodiment, the DAS 876 controls the movement of the stage 816. The processor 870 performs functions including reconstruction of an image from the detection data, pre-reconstruction processing of the detection data, and post-reconstruction processing of image data, as described herein. The processor 870 is an example of a processor. The data acquisition system 876 is also an example of an acquisition and reconstruction unit.

[0112] The processing unit 870 may be configured to perform the various steps of the methods 100 and 200 described herein and variations thereof. The processing unit 870 may include a CPU that may be implemented as discrete logic gates, an Application Specific Integrated Circuit (ASIC), or a Complex Programmable Logic Device (CPLD) such as a Field Programmable Gate Array (FPGA). An FPGA or CPLD implementation may be coded in a hardware description language such as VHDL, Verilog, or the like, and the program may be stored in an electronic memory, either by direct input to the FPGA or CPLD or as a separate electronic memory. Furthermore, the memory may be a non-volatile memory such as a ROM, EPROM, EEPROM, or FLASH memory. The memory may also be a volatile memory such as a static or dynamic RAM, and a processing unit such as a microcontroller or microprocessor may be provided to manage the electronic memory and the interaction between the FPGA or CPLD and the memory.

[0113] Alternatively, the CPU in the processing unit 870 may execute a computer program that includes a series of computer readable instructions that perform the various steps of the method 100 or the method 200. The program may be stored in the non-transitory electronic memory, hard disk drive, CD, DVD, FLASH drive, or any other storage medium known in the art. Furthermore, the computer readable instructions may be provided as a utility application, background daemon, or operating system component, or combinations thereof, in which case the instructions operate in conjunction with a processing unit, such as Intel of America's Xenon® processor or AMD of America's Opteron® processor, and an operating system, such as Microsoft VISTA®, UNIX®, Solaris®, LINUX®, Apple®, MAC-OS®, and other operating systems known to those skilled in the art. Furthermore, the CPU may be implemented as multiple processing units working together in parallel to execute instructions.

[0114] The memory 878 may be a hard disk drive, a CD-ROM drive, a DVD drive, a FLASH drive, RAM, ROM or other electronic storage device known in the art.

[0115] A network controller 874, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, can interface with various portions of the PET imager. The network controller 874 can also interface with an external network. As will be appreciated, the external network can be a public network, such as the Internet, or a private network, such as a LAN or WAN network, or a combination thereof, and can also include PSTN or ISDN sub-networks. The external network can be wired, such as an Ethernet network, or wireless, such as a cellular network, including EDGE, 3G and 4G wireless cellular systems. The wireless network can be WiFi, Bluetooth, or other known wireless forms of communication.

[0116] In the embodiment, a case has been described in which a processing unit (e.g., the processing device 870) included in the medical image processing device reconstructs an image from data to generate a reconstructed image made up of voxels arranged three-dimensionally. However, the embodiment is not limited to this. For example, a processing unit or an acquisition unit included in the medical image processing device may acquire a reconstructed image made up of voxels arranged three-dimensionally that has already been reconstructed by a processing unit other than the processing unit that acquires the reconstructed image.

[0117] According to at least one of the medical image processing apparatus and the program of the embodiment described above, image quality can be improved.

[0118] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0119] 870 Processing Equipment 876 Data Collection System

Claims

1. The present invention includes an acquisition unit that acquires a positron emission tomography (PET) image reconstructed from emission data representing coincidence measurements of gamma rays of each pair resulting from an annihilation event, the emission data being acquired based on detection by a plurality of detection elements; and a processing unit that inputs a first image that is a PET image and a second image acquired by a modality different from the PET device, and outputs a denoised PET image to a trained model that outputs a third image that is a denoised PET image, by inputting the PET image acquired by the acquisition unit; The trained model is trained by optimizing a loss function so as to minimize a difference between a reconstructed image at a first noise level and reconstructed images at multiple noise levels higher than the first noise level.

2. The medical image processing apparatus according to claim 1 , wherein the second image is an X-ray CT image or an MRI image.

3. The medical image processing device according to claim 1 , wherein the trained model further outputs a fourth image related to the modality, the fourth image being an image with a higher resolution than the second image.

4. The medical image processing apparatus according to claim 3 , wherein the trained model performs training for noise removal using a pair of the first image and the third image, and performs training for resolution improvement using a pair of the second image and the fourth image.

5. A system comprising: an acquisition unit that acquires a positron emission tomography (PET) image reconstructed from emission data representing simultaneous measurements of each pair of gamma rays resulting from an annihilation event, the data being acquired based on detection by a plurality of detection elements; and a processing unit that inputs a first image that is a PET image and a second image acquired by a modality different from the PET device, and outputs a denoised PET image to a trained model that outputs a third image that is a denoised PET image, the processing unit inputting the PET image acquired by the acquisition unit; The processing unit compares the PET image acquired by the acquisition unit with the denoised PET image from which partial volume effect has been removed, and outputs the denoised PET image.

6. A program that causes a computer to execute the steps of: acquiring a positron emission tomography (PET) image reconstructed from emission data representing coincidence measurements of each pair of gamma rays resulting from an annihilation event, the emission data being acquired based on the detection of a plurality of detection elements; and outputting a denoised PET image by inputting the acquired PET image into a trained model that outputs a third image, which is a denoised PET image, by inputting a first image that is a PET image and a second image acquired by a modality different from the PET device, wherein the trained model is trained by optimizing a loss function so as to minimize a difference between a reconstructed image having a first noise level and reconstructed images having multiple noise levels higher than the first noise level.

7. A program that causes a computer to execute the steps of: acquiring a positron emission tomography (PET) image reconstructed from emission data representing the simultaneous measurement of each pair of gamma rays resulting from an annihilation event, the emission data being acquired based on the detection of multiple detection elements; and inputting the acquired PET image into a trained model that outputs a third image, which is a denoised PET image, by inputting a first image that is a PET image and a second image acquired by a modality other than the PET device, and outputting the denoised PET image from which partial volume effects have been removed by comparing it with the acquired PET image.

Citation Information

Patent Citations

  • Image data processing

    JP2014532504A

  • System and method for image calibration

    US10078889B2

  • Image data processing

    US20140270452A1

  • Method for pet attenuation correction

    US20160116603A1

  • Method and system for providing a locally-consistent enhancement of a low-quality image

    WO2017191643A1