Neural network learning method, program, medical image processing method, and medical device

Abstract

To improve the image quality of a medical image.SOLUTION: A learning method of a neural network includes the step of making a neural network perform learning by using a data set of an input image being low-quality data and a target image being high-quality data. The target image is generated by acquiring a plurality of different images by executing a plurality of different reconstructions to medical data collected from a subject, extracting different feature regions from the plurality of acquired images and combining the extracted different feature regions into one image.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] An embodiment of the present invention relates to a neural network learning method, a program, a medical image processing method, and a medical apparatus. [Background technology]

[0002] Medical imaging produces pictures of the inside of a living body. For example, Magnetic Resonance Imaging (MRI) uses radio waves, magnetic fields, and magnetic field gradients to produce images of the inside of a living body. Medical imaging modalities include, for example, X-ray, ultrasound, CT, and PET. Once the images are produced, doctors can use them to diagnose injuries or illnesses in patients.

[0003] In PET imaging, a tracer agent is administered to the patient to be imaged by injection, inhalation, or ingestion. After administration, the physical and biomolecular properties of the tracer agent cause it to concentrate at specific locations in the body. The actual spatial distribution of the tracer agent, the intensity of the tracer accumulation zone, and the kinetics of the process from administration to eventual excretion are all factors that can be of clinical importance.

[0004] During this process, the tracer attached to the tracer agent emits a positron. When the emitted positron collides with an electron, an annihilation event occurs in which the positron and the electron combine. In most cases, the annihilation event produces two gamma rays (511 keV) traveling in opposite directions, approximately 180 degrees apart.

[0005] To reconstruct the spatiotemporal distribution of the tracer using tomographic reconstruction, each detected event is characterized for its energy (i.e. the amount of light emitted), its location, and its timing. By detecting these two gamma rays and drawing a line between their locations, i.e. the line-of-response (LOR), the original location of the annihilation can be determined. This process only identifies a possible line of interaction, but by accumulating many such lines, the original distribution can be inferred using tomographic reconstruction.

[0006] As described above, the LORs of coincidence pairs and timing information are used to reconstruct radiological images and provide clinical information. However, this clinical information can often be obscured by noise. The noise can be reduced to some extent by performing statistical image reconstruction with regularization using an Iterative Reconstruction (IR) algorithm. However, there is a trade-off between reducing noise and introducing other adverse effects into the reconstructed image (e.g., high degrees of regularization can reduce resolution and obscure features in the reconstructed image).

[0007] Similarly, in X-ray CT, IR algorithms can be used to reconstruct internal images that represent the X-ray attenuation of various organs. Generally, lung and air-dominated regions show little attenuation, soft tissue regions show more attenuation, and bone regions show even more attenuation (i.e., radiographic shadowing). As with PET imaging, there is a trade-off between reducing noise and introducing other adverse effects, such as reduced resolution, into the reconstructed image. The optimal degree of regularization in the reconstructed image may vary from region to region. For example, for regions with larger signal amplitudes (e.g., lung and bone regions), less regularization may be better in order to maintain higher resolution, even though more noise is associated with less regularization. On the other hand, for regions with smaller signal amplitudes (e.g., soft matter regions with little contrast in attenuation between the background and clinically relevant lesions), the increased noise associated with less regularization may obscure the features the clinician is looking for. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] US Patent Application Publication No. 2016 / 0149439 [Patent Document 2] US Patent Application Publication No. 2015 / 0884089 [Patent Document 3] US Patent Application Publication No. 2016 / 0228512 Summary of the Invention [Problem to be solved by the invention]

[0009] The problem to be solved by the present invention is to improve the image quality of medical images. [Means for solving the problem]

[0010] A neural network training method of an embodiment is a training method that includes training a neural network using a data set of an input image that is low-quality data and a target image that is high-quality data, and the target image is generated by performing a plurality of different reconstructions on medical data collected from a subject to obtain a plurality of different images, extracting different feature regions from each of the obtained images, and combining the extracted different feature regions into a single image. [Brief description of the drawings]

[0011] [Figure 1A] FIG. 1A shows an example of a lung region in a reconstructed CT image generated using a small regularization parameter (i.e., less smoothing / noising) and displayed using lung settings (i.e., window level WL=-400 Hounsfield Units (HU) and window width WW=1500 HU) according to one embodiment. [Figure 1B] FIG. 1B shows an example of the same lung region using the same display settings as FIG. 1A, except that the CT image was generated using a large regularization parameter instead of the small smoothing / denoising parameter used in FIG. 1A, in accordance with one embodiment. [Figure 1C] FIG. 1C shows an example of a soft tissue region in a CT image generated using a small regularization parameter and displayed using soft tissue settings (i.e., WL=40HU, WW=400HU), according to one embodiment. [Figure 1D] FIG. 1D shows the same example soft tissue region using the same soft tissue display settings as FIG. 1C, except that the CT image was generated using a large regularization parameter, according to one embodiment. [Figure 1E] FIG. 1E shows a PET image using spatially constant regularization with low smoothing strength, according to one embodiment. [Figure 1F] FIG. 1F shows a PET image using spatially constant regularization with high smoothing strength, according to one embodiment. [Figure 2A] FIG. 2A illustrates a flow chart of a method for training a neural network and then using the neural network to denoise medical images reconstructed from data from medical imaging scans, according to one embodiment. [Figure 2B] FIG. 2B illustrates a flowchart of a neural network training method in which target data includes feature-aware denoising, according to one embodiment. [Figure 2C] FIG. 2C illustrates a flowchart of a method for generating feature recognition target data according to one embodiment. [Figure 2D] FIG. 2D illustrates an implementation flowchart of a method for generating feature-recognized target data, in which multiple reconstructions are performed with different reconstruction parameters, according to one embodiment. [Figure 3A] FIG. 3A illustrates a CT image generated with regularization parameters optimized for the abdominal region, according to one embodiment. [Figure 3B] FIG. 3B illustrates a CT image generated with regularization parameters optimized for the lung region, according to one embodiment. [Figure 3C] FIG. 3C illustrates a CT image generated with regularization parameters optimized for bone regions, according to one embodiment. [Figure 4] FIG. 4 illustrates a flow chart for training a DL-ANN network by iteratively adjusting the coefficients of the DL-ANN network to optimize a loss-error function, according to one embodiment. [Figure 5A] FIG. 5A illustrates an example of a feed-forward ANN, according to one embodiment. [Figure 5B] FIG. 5B illustrates an example of a type of ANN called a Convolutional Neural Network (CNN), according to one embodiment. [Figure 6]FIG. 6 illustrates an example of a medical imaging scanner (eg, the scanner may be an MRI scanner, a PET scanner, an X-ray CT scanner, or a combination thereof) according to one embodiment. [Figure 7] FIG. 7 illustrates an example of an X-ray CT scanner, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The embodiments relate to medical image reconstruction using deep learning networks based on feature recognition training to perform image denoising, and in particular to medical image reconstruction that can perform denoising and artifact reduction for medical images such as (i) X-ray Computed Tomography (CT) images, (ii) Positron Emission Tomography (PET) images, and (iii) Magnetic Resonance (MR) images. In particular, the embodiments relate to an improved method that allows for an appropriate amount of denoising to be performed per feature.

[0013] CT systems and methods are widely used, especially in medical imaging and diagnosis. A CT scan can be performed by placing a patient in the space between an x-ray source and an x-ray detector on a CT scanner, and then taking x-ray projection images through the patient at various angles while rotating the x-ray source and detector during the scan. The resulting projection data, called a CT sinogram, represents the attenuation through the body as a function of position along one or more axes, and as a function of projection angle along other axes. An image is reconstructed from the projection data represented in the sinogram by performing an inverse Radon transform, or any other image reconstruction method.

[0014] CT images can be reconstructed from the projection data using various methods, including the Filtered Back-Projection (FBP) algorithm and the statistical IR algorithm. Compared to the more traditional FBP reconstruction method, the IR method provides higher image quality at a lower radiation dose. There are various IR methods.

[0015] One common IR method involves performing an unconstrained (or constrained) optimization to find an argument p that minimizes the following equation (1):

[0016]

number

[0017]

number

[0018]

number

[0019] The function U(p) is a regularization term that imposes one or more constraints (e.g., a Total Variation (TV) minimization constraint) that often have the effect of smoothing or denoising the reconstructed image. The value β is a regularization parameter that weights the relative contributions of the data fidelity term and the regularization term.

[0020] Thus, the choice of the value of the regularization term β typically influences the tradeoff between noise and resolution. In general, a larger regularization term β reduces noise, but at the cost of reduced resolution. The ultimate usefulness of the regularization term β depends on several factors, primarily the application for which the reconstructed image is to be reconstructed. IR algorithms can be time-consuming and computationally intensive, making trial-and-error methods inefficient (e.g., for IR methods, different values ​​of the regularization term β are used until an optimal solution is found). Furthermore, because a single CT scan can be used for multiple clinical applications, it is desirable to be able to tune the reconstructed image for the noise-resolution tradeoff without iterating through computationally intensive IR algorithms. Therefore, improved methods for quickly generating and modifying reconstructed images to optimize the noise-resolution tradeoff are desirable.

[0021] Another commonly used reconstruction technique is Model-Based Iterative Reconstruction (MBIR). MBIR can obtain better image quality than FBP reconstruction method. Moreover, Deep Convolution Neural Network (DCNN) can improve and speed up various CT imaging applications / image processing tasks. The application of DCNN does not require accurate pre-designed models such as noise statistics, but instead requires the availability of a large amount of training dataset. Also, DCNN can capture the exact image features between the input and target of the training dataset.

[0022] Methods according to embodiments address the above-mentioned problem of optimizing the resolution vs. noise tradeoff based on the particular content / context of the displayed image using a neural network such as a DCNN. These methods address the problem by training the neural network to perform feature-aware denoising. In certain embodiments, the network is trained to perform feature-aware denoising using training data in which a target image is segmented into various regions corresponding to different features, and feature-dependent regularization / denoising is applied to the target image. Thus, by training the neural network using this specially constructed training data, the neural network learns to recognize specific features and apply the appropriate degree of denoising depending on the recognized features.

[0023] Reference is now made to the drawings, in which the same or corresponding parts in the several figures are given the same reference numbers. Figures 1A and 1B show two images of the same lung region, but with different degrees of denoising (interchangeably referred to in the embodiment as smoothing, and in certain contexts as regularisation). Similarly, Figures 1C and 1D show two images of the same soft tissue region, but with different degrees of denoising. Figures 1A and 1C represent a first degree of denoising, and Figures 1B and 1D represent a second degree of denoising, which provides more denoising / smoothing than the first degree of denoising shown in Figures 1A and 1C.

[0024] In comparing Figure 1C and Figure 1D, Figure 1D is usually considered better for clinical applications because the higher resolution in Figure 1C does not convey as much information and the increased noise in Figure 1C creates confusing textures and structures that may lead to inaccurate diagnostic or poor quality results in interventional imaging. Therefore, a higher degree of noise removal and smoothing may be advantageous for soft tissue images.

[0025] On the other hand, for lung images, a lower degree of noise removal and smoothing may be advantageous. Here, the term "smoothing" can be interpreted as a kind of noise removal, and the term "smoothing" can usually be replaced with the more general term "noise removal". In comparison of Fig. 1A and Fig. 1B, the higher resolution in Fig. 1A can prominently identify lung features (e.g., features indicated by arrows in Fig. 1A), and compared to the wider window width and corresponding higher contrast lung region signals in the lung setting, Fig. 1A is usually considered to be better for clinical applications because the increased noise is not as prominent in soft tissue regions. Therefore, the noise increase due to less smoothing is relatively less obscured in the lung regions, and the advantage of the improved resolution as seen in Fig. 1A outweighs the disadvantage of increased noise.

[0026] Thus, the optimal degree of noise reduction may be spatially dependent on the characteristics in various regions of the reconstructed image, i.e., different regions in the same image may benefit from different degrees of noise reduction.

[0027] So far, we have described the methodology in terms of X-ray CT. However, these methods can also be applied to other medical imaging modalities, e.g., PET. For example, feature-aware denoising can be used to obtain high image quality in PET imaging. For example, statistical IR algorithms can be used to reconstruct PET images of tracer / radiation concentrations in a subject. The IR algorithm iteratively searches for a reconstructed image that optimizes an objective function that includes a data fidelity term and a regularization term. The relative weighting between these two terms can affect the trade-off between higher SNR on the one hand and finer resolution on the other. This trade-off is often important because practical constraints (e.g., constraints on injected dose and duration of scan) result in fewer statistics, which in turn degrade the image quality of PET images due to high noise levels and relatively low spatial resolution (e.g., 5-7 mm resolution). These effects can result in misdiagnosis and incorrect treatment decisions. For this reason, methods to improve the quality of PET images are desirable.

[0028] Therefore, an improved method is desired for selecting the degree of regularization that produces an optimal reconstructed image. Moreover, a degree of smoothing that is optimal in one part of a PET image may be less optimal in another part of the PET image. Therefore, a method that can perform spatially varying smoothing to locally optimize PET image quality is also desired. In certain embodiments, this goal is achieved by the method according to the embodiment for generating target data for the training data set by using one or more radiation-dependent regularizations and / or radiation-dependent parameters in the regularization terms in the objective function of the IR algorithm.

[0029] For example, the quality of PET images can be improved by spatially varying regularization based on radiation level mapping representing different features in different regions. Here, training data can be used in which the target data is subdivided according to regions corresponding to different features, and spatially varying degrees of denoising / regularization are applied to each feature. The degree of variation in spatially varying regularization can be caused by the spatial dependency of the regularization parameter β, which provides a relative weighting between the regularization term and the data fidelity term, as described below. A neural network can then be trained using the training data from which the target data was generated using spatially varying regularization / denoising as described above. The neural network is then trained to perform feature-aware denoising of reconstructed PET images.

[0030] Regularization parameters, which can be adjusted to affect the degree and / or type of regularization, are best understood by considering the optimization approach and cost function (also called the objective function) when solving the reconstruction problem. In PET, the emission data can be organized as sinograms (similar to projection data in X-ray CT), so that methods similar to those applied in X-ray CT to reconstruct attenuation images can be applied to reconstruct radiation images that represent radiation / tracer concentration as a function of position / voxel index. These reconstruction methods include, for example, the FBP method and the IR method. Alternatively, instead of being organized into sinograms, the PET emission data can be kept as a list of individual detection events, and list-mode based reconstruction methods can be used to reconstruct images.

[0031]

number

[0032]

number

[0033]

number

[0034] In certain embodiments, the regularization function may be a patch-based edge-preserving penalty function, such as that illustrated in Equation (3) below, which calculates the penalty using nearby patches instead of individual pixels.

[0035]

number

[0036]

number

[0037] As in the case of X-ray CT described above, in PET image reconstruction, increasing the magnitude of the regularization parameter β suppresses noise in the reconstructed image and reduces resolution. Thus, regions of low radiation may benefit from a higher degree of smoothing since the counting statistics in these regions will reduce the SNR (i.e., relatively high noise levels), and fine resolution in these regions may not be needed. For example, higher tracer concentrations and radiation levels are typically found near the region of interest (e.g., due to higher metabolic rates of lesions). Thus, image reconstructions in which lower radiation corresponds to a larger regularization parameter β may be advantageous.

[0038]

number

[0039]

number

[0040] In addition, MRI reconstruction can be performed by minimizing an objective function that includes a regularization term. For example, compressed sensing (CS) image reconstruction can be performed by solving the optimization problem of the following equation (4).

[0041]

number

[0042]

number

[0043] Other examples of reconstruction processes used for MRI reconstruction include GRAPPA, SENSE, ARC, SPIRiT, LORAKS, ISTA, and FISTA. For CS data, the reconstruction process may be a nonlinear process that achieves both sparsity in the image representation in a certain (e.g., spatial, temporal, wavelet) domain and consistency of the reconstruction with the collected scan data.

[0044] As will be appreciated by those skilled in the art, the principles described above for X-ray CT imaging and PET imaging also apply to other medical imaging modalities, such as MRI, where images are denoised and / or reconstructed by minimising an objective function that includes a regularisation term together with a data fidelity term (e.g. MRI and Single-Photon Emission CT (SPECT)). Thus, the method described in the embodiments for training a neural network to perform feature-aware denoising generally applies to medical imaging modalities where a feature-dependent level of denoising / regularisation can improve the resulting image quality. For clarity, the method described in the embodiments is described using the non-limiting example of X-ray CT as a medical imaging modality. However, as will be appreciated by those skilled in the art, the method is not limited to X-ray CT.

[0045] Two different methods can be used to denoise an image using a neural network. In one method, the target data is the denoised image and the input data is the corresponding noisy image. The network is then trained to directly predict the denoised image from the noisy image. That is, when a noisy image is applied to the neural network, the network outputs the corresponding denoised image. Alternatively, the method according to the embodiment can denoise an image by treating the noise as an additive residue that can be directly removed from the dealiased image using a Residual Network (ResNet). In this case, the input data is again the noisy image, but the target data is the noise itself (e.g., the noisy image minus the denoised image). Thus, when a noisy image is applied to the neural network, the network outputs an image that represents the noise itself. The denoised image is then generated by subtracting the network output from the input.

[0046] 2A and 2B show a non-limiting example of a method 10 for training and using a DL network 170 to assist in performing SMS medical image reconstruction. As shown in FIG. 2A, the method 10 uses a DL network 170, which is a DL-ANN (Artificial Neural Network), to learn how to denoise a medical image reconstructed from raw data 105 collected by a medical image scan / acquisition modality, and generate a denoised reconstructed image 125. The method 10 has two parts: (i) an offline training process 150 and (ii) a medical imaging process 100. That is, the offline training process 150 trains the DL network 170, and the medical imaging process 100 uses the trained DL network 170 to denoise the medical image, applying a spatially varying degree of denoising depending on the features in the image, and finally generates a reconstructed image 125, which is a higher quality image with less noise.

[0047] The raw data 105 is an example of medical data collected from a subject. The raw data 105 may be radiation data collected using a modality such as CT, X-ray photography, PET, or SPECT, or may be medical data collected using another modality such as MRI or ultrasound diagnosis.

[0048] 2B shows a non-limiting example of medical imaging process 100 and offline training process 150. Similar steps are performed in training process 160 to train DL network 170. As described below, training DL network 170 may be an iterative process in which input data from a training data set is applied to DL network 170 (similar to how DL network 170 is used in the denoising 120 step) and the results are compared to a desired output (i.e., target data from the training data set) in an iterative process that adjusts weighting coefficients of DL network 170.

[0049] In a particular embodiment, the DL network 170 is a CNN in which a series of convolution (conv), batch normalization (BN), and rectified linear unit (ReLu) network layers are implemented. In the image reconstruction 110 step, a medical image is reconstructed from raw data 105. The reconstruction method in the image reconstruction 110 may be the same reconstruction method used to generate the input image 157 used in the offline training process 150 to train the DL network 170.

[0050] The DL network 170 is trained using a training process 160. In the training process 160, a loss function is used to iteratively adjust / optimize the parameters of the DL network 170 (e.g., the parameters of the DL network 170 may include weight coefficients for the network layers and activation functions / potentials of the nodes within the layers). The optimization of the network parameters continues until a stopping criterion is met (e.g., the stopping criterion may be whether the value of the loss function has converged to a predefined threshold) and a trained DL network 170 is generated.

[0051] The loss function compares target data, target images 153, to input data, input images 157, and the output collected using the updated DL network 170. For example, applying a noisy image from the input data to the updated DL network 170 produces a network output that is compared to a corresponding low-noise image from the target image 153. Training of the DL network 170 is deemed to be terminated when the difference between the output and the target data is minimized and sufficient to satisfy one or more predetermined stopping criteria of the training process 160. The trained DL network 170 can then be saved and used later in the medical imaging process 100.

[0052] For example, when the medical imaging modality is PET imaging, the method 10 includes training a DL network 170, which is a DL-CNN, and applying a low-quality (e.g., noisy) PET image reconstructed from raw data 105 to the trained DL network 170 to generate a reconstructed image 125, which is a high-quality (e.g., denoised) PET image.

[0053] In method 10, the parameters of the DL-CNN network (e.g., weights and biases of the convolutional and pooling layers) are iteratively adjusted using a loss function until a stopping criterion is met (e.g., the parameters converge to a predefined threshold), generating a trained DL network 170. The loss function compares the target image 153, which is high-quality data, with the results of an updated version of the DL-CNN network to which the input image 157, which is low-quality data, is applied. In the case of PET imaging, the high-quality data and the low-quality data are reconstructed PET images with high / good image quality and low / bad image quality, respectively. In general, the signal-to-noise ratio (SNR) is lower when a smaller data set is used in the reconstruction of an image (e.g., due to a short scan time or other factors that cause a reduction in coincidence counts). Thus, all of the coincidence counts from the patient's PET scan are used to generate the target image 153, which is a high-quality image, to generate a PET image with the highest possible image quality. Alternatively, a partial subset of coincidences selected from the total data set can be used to generate a lower quality image, input image 157, which will result in a noisier image than the high quality reconstructed image, target image 153. Additionally, a spatially varying regularization parameter can be used depending on the features present in the reconstructed image to generate a high quality image, target image 153. For example, less regularization can be used in regions of high radiation to allow for finer resolution in those regions.

[0054] After generating the trained DL network 170, the trained DL network 170 is applied using the medical imaging process 100 to generate a reconstructed image 125, which is a high quality PET image.

[0055] In certain embodiments, the raw data 105 collected by PET imaging may be counts that have been pre-processed (e.g., signal preconditioning, position correction, energy correction, etc.), and in the next step, image reconstruction 110, the pre-processed count data is used to reconstruct an image of radiation level (e.g., tracer concentration) as a function of voxel position.

[0056] In image reconstruction 110 of the medical imaging process 100, a PET image, ie, a reconstructed image 125, is reconstructed from raw data 105, ie, radiation data collected by PET imaging, using a PET image reconstruction process. As will be appreciated by those skilled in the art, image reconstruction may be performed using backprojection, filtered backprojection, Fourier transform-based image reconstruction, iterative image reconstruction, matrix inversion image reconstruction, statistical image reconstruction, list mode, or other reconstruction methods.

[0057] In denoising 120, the reconstructed image is denoised using DL network 170, which is a DL-CNN. The result is a high quality image, reconstructed image 125. Thus, the DL denoising algorithm that applies the network generated by the offline training process 150 can be used to process noisy PET images obtained from PET reconstruction in image reconstruction 110.

[0058]

number

[0059]

number

[0060] where Θ represents the trainable weights, ψ is the error function (e.g., Mean Square Error (MSE) is used in the first example), N represents the training samples, y represents the noisy image, and x represents the target image. In the first example, the loss function is minimized using stochastic gradient descent, and an 8-layer network is used to achieve a good balance between performance and training efficiency.

[0061] As described above, the target data has a spatially varying, feature-dependent degree of denoising / regularisation. This denoising variation can be generated by the feature-aware target image design 180 shown in FIG. 2B. Additionally, FIGS. 2C and 2D are flow charts illustrating a non-limiting example of the feature-aware target image design 180. These flow charts are illustrated for the medical imaging modality x-ray CT, but the embodiments are not so limited. For example, the spatially varying regularisation can be performed for PET imaging using the method described in US Pat. No. 6,399,363, which is incorporated by reference in its entirety. For x-ray CT imaging, the spatially varying degree of denoising can also be obtained using the method described in US Pat. No. 6,399,363 and US Pat. No. 6,399,375, both of which are incorporated by reference in their entirety. These methods include a spatial dependency for the regularisation parameters. Additionally, the spatially varying denoising can be performed independently of the image reconstruction process or using an additional denoising process to the same process. For example, a Gaussian kernel with a spatially dependent width (eg, narrower for features with high contrast signals, wider for features with low contrast signals, etc.) can be used to smooth the reconstructed image.

[0062] Representative noise removal methods include linear smoothing filters, anisotropic diffusion, nonlocal means, or nonlinear filters. Linear smoothing filters remove noise by convolving the original image with a mask that corresponds to a low-pass filter or smoothing operation. For example, a Gaussian mask is composed of elements calculated by a Gaussian function. This convolution makes each pixel's value better match the values ​​of its neighboring pixels. Anisotropic diffusion removes noise while preserving sharp edges by expanding the image with a smoothing partial differential equation similar to the heat equation. A median filter is an example of a nonlinear filter that, if properly designed, can also preserve edges and prevent blurring. A median filter is an example of a Rank-Conditioned Rank-Selection (RCRS) filter that can be applied to remove salt-and-pepper noise from an image without introducing significant blurring artifacts. Furthermore, if the imaging domain can be considered to be uniform over large areas bounded by sharp boundaries between uniform regions, a filter using a TV-minimization regularization term can be applied. TV filters are another example of nonlinear filters. Furthermore, nonlocal means filtering is a representative method that uses a weighted average over similar patches in an image to compute denoised pixels.

[0063] Returning to Figures 2C and 2D, target data can be generated by reconstructing a series of different images using steps 210(1), 210(2), 210(3), etc., where reconstruction is performed using reconstruction processing / regularization parameters optimized for each type of feature (e.g., abdominal features, lung features, bone features, etc.).

[0064] For example, images can be reconstructed with different reconstruction parameters for different clinical purposes, and differences in image reconstruction can include different regularization parameters as well as different reconstruction and other parameters and methods (e.g., edge enhancement).

[0065] In the example shown in Fig. 2D, the image reconstructed using the abdominal parameters has a lower resolution but a significantly reduced noise. Therefore, this image is optimally reconstructed for abdominal diagnosis. Furthermore, the image reconstructed using the bone parameters has a higher resolution but a higher noise.

[0066] A wide range of parameters may affect the image quality of a particular reconstructed image. Some or all of these parameters may be included in the reconstruction parameters used to tune the reconstruction method to be optimal for a particular clinical application and / or anatomical region. These various parameters that contribute to image quality may include, for example, clinical trial protocol parameters, image processing parameters, reconstruction kernels, and others. Additionally, the reconstruction parameters that may be optimized may include parameters for reconstruction artifact correction. Correction of metal artifacts, aliasing artifacts, etc. depends on the reconstruction method, which are related to the reconstruction parameters.

[0067] In steps 220(1), 220(2), 220(3), etc., respective features (e.g., abdominal feature 154(1), lung feature 154(2), bone feature 154(3), etc.) are extracted from the image generated in steps 210, including steps 210(1), 210(2), 210(3), etc. For example, threshold and region growing techniques can be used to segment the image. Additionally, Hounsfield maps can be used to associate HU ranges with specific features (e.g., abdominal, lung, bone, etc.). Additionally, image processing for specific textures, structures, or atlases of the human body can be used to segment and determine regions and features within the generated image. In certain embodiments, the generated image can be segmented into regions and tagged according to features as illustrated in the embodiments, via user input.

[0068] For example, the features extracted from each generated image may be those features for which a particular image was optimized, e.g., for an image to be reconstructed for optimal image quality of the abdomen, features corresponding to the abdomen may be extracted, similarly, for an image to be reconstructed for optimal image quality of bone, bone regions may be extracted, etc.

[0069] In step 240, the combined images are stitched together from each feature 154 to generate the target image 153. The target image 153 can then have optimal image quality in each region and for each feature type. In certain embodiments, the transitions between the segmented regions may be stitched (e.g., gradual transitions) so that the target image 153 appears uniform without abrupt boundaries between regions.

[0070] By training DL network 170 with target data having image quality and / or noise reduction according to the features, DL network 170 learns to recognize these features and output images with similar image quality and / or noise reduction when corresponding features are present in the input images applied to DL network 170.

[0071] 3A, 3B, and 3C show examples of reconstructed images produced using optimized reconstructions for the abdomen, lung, and bone regions, respectively. As shown in FIG. 3A, 3B, and 3C, MBIR can improve the tradeoff between noise texture and spatial resolution. This is often done by adjusting regularization parameters. However, in many clinical applications, simply adjusting the overall parameters may not be sufficient to create the desired or optimal noise texture and spatial resolution for all organs and features represented in the reconstructed images.

[0072] 3A, 3B, and 3C show three different images reconstructed using the same set of raw data 105, each with a different regularization parameter. In FIG. 3A, when the abdominal-optimized regularization parameter is used, the image shows good noise texture in the soft tissues, but with poor spatial resolution in the bone structures. In FIG. 3C, when the bone-optimized regularization parameter is used, the image shows good spatial resolution in the bone structures, but with poor noise texture in the soft tissues. In FIG. 3B, when the lung-optimized regularization parameter is used, the image shows a combination of intermediate spatial resolution and intermediate noise texture, which is a compromise optimized for the lung region.

[0073] The above can be better understood by considering that in many clinical cases, diagnosis is performed on different anatomical tissues / organs in one image. Therefore, optimizing three separate images for different anatomical tissues / organs is undesirable and inefficient. Good features reconstructed with different parameters should be integrated into one image. This could be achieved by reconstructing the image multiple times with different parameters for each image, as shown in Figures 3A, 3B, and 3C, and then synthesizing these features / regions together to form one feature-aware reconstruction, but this is inefficient for multiple image reconstruction. Therefore, the method according to the embodiment reconstructs one image and then applies the reconstructed image to a DL network 170 trained to perform feature-aware denoising in order to achieve a desired trade-off between the resolution and noise texture of each of the anatomical tissues / organs in the reconstructed image.

[0074] The training of the DL-ANN network will now be described in more detail (e.g., training process 160) and illustrated with a target image 153, which is feature-recognized denoised image target data, and an input image 157, which is a noisy image.

[0075] 4 shows a flow chart of one embodiment of the training process 160. In the training process 160, a DL-ANN network is trained using the input images 157, which are low quality data, and the target images 153, which are high quality data, as training data, resulting in a DL-ANN network output from step 319 of the training process 160. The training process 160 trains a DL network 170 using a number of input images 157 paired with corresponding target images 153, such that the DL network 170 is trained to generate images similar to the target images 153 from the input images 157.

[0076] In the training process 160, a set of training data is obtained and the DL network 170 is iteratively updated to reduce the error (e.g., the value generated by the loss function). The DL-ANN network infers the mapping implied by the training data, and the cost function generates an error value related to the deviation between the result generated by applying the latest update of the DL network 170 to the input image 157 and the target image 153. For example, in certain embodiments, the cost function may use the mean squared error to minimize the mean squared error. In the case of a Multilayer Perceptrons (MLP) neural network, a backpropagation algorithm may be used to train the network by minimizing the mean squared error-based cost function using (stochastic) gradient descent.

[0077] In step 316 of the training process 160, an initial guess is made for the coefficients of the DL network 170. For example, the initial guess may be based on a priori knowledge of the region being imaged, or one or more representative denoising, edge detection, and / or blob detection methods. Additionally, the initial guess may be based on any of LeCun initialization, Xavier initialization, and Kaiming initialization.

[0078] Steps 316-319 of the training process 160 illustrate a non-limiting example of an optimization method for training the DL network 170.

[0079] An error is calculated (e.g., using a loss or cost function) to indicate the degree of difference (e.g., a distance measure) between the target image 153 (i.e., the ground truth) and the input image 157 after applying the updated DL network 170. The error may be calculated using any known cost function or distance measure between image data, such as the cost functions described above. Furthermore, in certain embodiments, the error / loss function may be calculated using one or more of hinge loss and cross-entropy loss.

[0080] In a particular embodiment, the DL network 170 is trained using backpropagation. Backpropagation can be used to train neural networks and is used in combination with gradient descent optimization. During forward propagation, the algorithm calculates predictions of the network based on the latest parameters Θ. These predictions are then input into a loss function, which compares them with the corresponding ground truth signatures (i.e., the target images 153, which are high quality images). During backpropagation, the model calculates the gradient of the loss function with respect to the latest parameters, and then updates the parameters by taking a given step size in the direction of the minimized loss (e.g., in accelerated methods such as Nesterov momentum and various adaptive methods, the step size can be selected to converge faster and optimize the loss function).

[0081] The optimization method for performing backpropagation may use one or more of gradient descent, batch gradient descent, stochastic gradient descent, and mini-batch stochastic gradient descent. Forward and backpropagation can be performed incrementally at each layer of the network. In forward propagation, execution begins by feeding the inputs to the first layer, which generates output activations for the next layer. This process is repeated until the loss function of the last layer is reached. During backpropagation, the last layer calculates gradients with respect to its own learnable parameters (if any) and with respect to its inputs, which are the upstream derivatives of the previous layer. This process is repeated until the input layer is reached.

[0082] Returning to Figure 4, in step 317 of the training process 160, the change in error is calculated as a function of the change in the network (e.g., the error gradient), and this change in error is used to select the direction and step size for the next change to the weights / coefficients of the DL network 170. Such calculation of the error gradient is consistent with certain embodiments of gradient descent. In certain other embodiments, as will be appreciated by those skilled in the art, this step may be omitted and / or replaced with another step of a different optimization algorithm (e.g., a non-gradient descent optimization algorithm such as simulated annealing or a genetic algorithm).

[0083] In step 317 of the training process 160, a new set of coefficients is calculated for the DL network 170. For example, as in gradient descent optimization or over-relaxation accelerated methods, the weights / coefficients are updated with the changes calculated in step 317.

[0084] In step 318 of the training process 160, new error values ​​are calculated using the updated weights / coefficients of the DL network 170.

[0085] In step 319, a predetermined stopping criterion is used to determine whether the training of the network is complete. For example, the predetermined stopping criterion may evaluate whether the new error and / or the total number of iterations performed exceeds a predetermined value. For example, the stopping criterion may be met when either the new error falls below a predetermined threshold or a maximum number of iterations is reached. If the stopping criterion is not met, the training process performed in the training process 160 continues by returning to the start of an iterative loop, where step 317 is repeated with new weights and coefficients (the iterative loop includes steps 317, 318, and 319). If the stopping criterion is met, the training process in the training process 160 ends.

[0086] 5A and 5B show various examples of interconnections between layers in DL network 170. DL network 170 can include fully connected layers, convolutional layers, and pooling layers, all of which are described below. In a particular preferred embodiment of DL network 170, the convolutional layers are placed close to the input layers, while the fully connected layers, which perform advanced inference, are placed further down in the architecture towards the loss function. Pooling layers are inserted after the convolutions to allow for validation of shrinks that reduce the spatial extent of the filters and thus the amount of learnable parameters. Activation functions are also incorporated into various layers to introduce nonlinearity and allow the network to learn complex predictive relationships. The activation functions may be saturating activation functions (e.g., sigmoid or hyperbolic tangent activation functions) or regularized activation functions (e.g., Rectified Linear Unit (ReLU) as applied in the first and second examples above). The layers of DL network 170 may also be batch normalized, as shown in the first and second examples above.

[0087] FIG. 5A shows an example of a general ANN with N inputs, K hidden layers, and three outputs. Each layer is composed of nodes (also called neurons) where a weighted sum of the inputs is calculated and the result of the weighted sum is compared to a threshold to generate an output. ANNs constitute a class of functions whose members are obtained by varying architectural features such as thresholds, connection weights, or the number of nodes and / or node connectivity. The nodes in an ANN are called neurons (or neuron nodes), and neurons can interconnect between different layers of the ANN system. Synapses (i.e., connections between neurons) store "weights" (interchangeably called "coefficients" or "weighting coefficients") that manipulate data in computations. The output of an ANN depends on three types of parameters: (i) the interconnection pattern between different layers of neurons, (ii) a learning process for updating the interconnection weights, and (iii) an activation function that transforms a neuron's weighted input into its output activation.

[0088] Mathematically, a function m(x) of a network of neurons is a function of other functions n i (x) is defined as a composite function of the other functions n i (x) may be further defined as a composition of other functions. This can be conveniently represented as a network structure, as shown in Figure 5A, where arrows represent dependencies between variables. For example, if an ANN is fitted with a nonlinear weighted sum m(x)=K(Σ i w i n i (x)), where K (usually called the activation function) is some predefined function such as the hyperbolic tangent.

[0089] In Figure 5A (and similarly in Figure 5B), neurons (i.e., nodes) are depicted as circles around a threshold function. For the non-limiting example shown in Figure 5A, inputs are depicted as circles around linear functions, and arrows indicate directed connections between neurons. In certain embodiments, DL network 170 is a feedforward network.

[0090] FIG. 5B shows a non-limiting example in which DL network 170 is a CNN. A CNN is a type of ANN that has advantageous properties for image processing and is therefore particularly relevant for image denoising applications. CNNs use feed-forward ANNs, where the pattern of connections between neurons can represent convolutions in image processing. For example, CNNs can be used to optimize image processing using multiple layers of small neuron clusters that process input portions of an image, called receptive fields. The outputs of these clusters can then be arranged on top of each other to better represent the original image. This processing pattern can be repeated across multiple layers of alternating convolutional and pooling layers.

[0091] Following the convolutional layers, the CNN may include local and / or global pooling layers that combine the outputs of the neuron clusters in the convolutional layers. Additionally, in certain embodiments, the CNN may include various combinations of convolutional and fully connected layers, with pointwise nonlinearities applied at the end or after each layer.

[0092] 6 illustrates an example embodiment of a medical imaging system 40. The medical imaging system 40 includes at least one scanning device 430, one or more image generating devices 410, each of which is a specially configured computing device (e.g., a specially configured desktop computer, a specially configured laptop computer, a specially configured server), and a display device 420.

[0093] The scanning device 430 and the image generating device 410 are examples of medical devices. For example, the scanning device 430 can perform various steps and variations thereof in the above-mentioned method 10 on medical data acquired by performing a scan. Alternatively, various steps and variations thereof in the above-mentioned method 10 can be performed on medical data acquired from the image generating device 410 and the scanning device 430.

[0094] The scanning device 430 is configured to acquire scan data by scanning a region (e.g., a region, a volume, a slice) of a subject (e.g., a patient). The scanning modality may be, for example, MRI, CT, PET, X-ray, and ultrasound. The scanning device 430 may acquire a set of randomly undersampled scan data suitable for compressed sensing reconstruction or may acquire scan data that may be used in a compressed sensing process. Thus, the compressed sensing data includes scan data that may be used in a compressed sensing process or that is suitable for compressed sensing reconstruction.

[0095] The one or more image generating devices 410 receive scan data from the scanning device 430 and generate an image of a region of a subject based on the scan data. To generate an image, for example, if the scan data is compressed sensing data, the one or more image generating devices 410 may perform each step of the medical imaging process 100.

[0096] After the one or more image generating devices 410 generate an image, the one or more image generating devices 410 transmit the image to a display device 420 which displays the image.

[0097] 7 shows an embodiment in which the medical imaging system 40 includes a gantry 500 capable of performing a CT scan. As shown in Fig. 7, the gantry 500 is shown from a side view and further includes an X-ray tube 501, an annular frame 502, and a multi-row or two-dimensional array type X-ray detector 503. The X-ray tube 501 and the X-ray detector 503 are mounted on the annular frame 502 across a diameter of the object OBJ, and the annular frame 502 is supported rotatably about a rotation axis RA.

[0098] The multi-slice X-ray CT apparatus further includes a high-voltage generator 509, which generates a tube voltage applied to the X-ray tube 501 via a slip ring 508 so that the X-ray tube 501 generates X-rays. The X-rays are emitted toward the object OBJ, whose cross-sectional area is represented by a circle. For example, the X-ray tube 501 has an average X-ray energy during a first scan that is less than the average X-ray energy during a second scan. Thus, two or more scans corresponding to different X-ray energies can be acquired. The X-ray detector 503 is located on the opposite side of the object OBJ to the X-ray tube 501 to detect the irradiated X-rays transmitted through the object OBJ. The X-ray detector 503 further includes an independent detector element or detector unit.

[0099] The CT apparatus further includes other devices for processing detection signals from the X-ray detector 503. A data acquisition circuit or data acquisition system (DAS) 504 converts the signal output from the X-ray detector 503 for each channel into a voltage signal, amplifies the voltage signal, and further converts the voltage signal into a digital signal.

[0100] The above data is transmitted via a non-contact data transmitter 505 to a pre-processing circuit 506 housed in a console outside the gantry 500. The pre-processing circuit 506 performs certain corrections on the raw data, such as sensitivity correction. A memory 512 stores the resulting data (also called projection data) just before the reconstruction process. The memory 512 is connected to the processing circuit 510 via a data / control bus 511, together with a reconstruction circuit 514, an input interface 515, and a display 516. The processing circuit 510 controls a current regulation circuit 513 that limits the current to a level sufficient to drive the CT system.

[0101] The detectors are rotated and / or fixed relative to the patient in various generations of CT scanner systems. In one embodiment, the CT system described above may be an example of a system that combines third and fourth generation geometries. In a third generation system, the x-ray tube 501 and x-ray detector 503 are diametrically mounted on an annular frame 502 and rotate around the object OBJ when the annular frame 502 is rotated about a rotation axis RA. In a fourth generation geometry system, the detectors are fixedly positioned around the patient and the x-ray tube rotates around the patient. In an alternative embodiment, the gantry 500 has multiple detectors positioned on an annular frame 502 supported by a C-arm and a stand.

[0102] The memory 512 can store measurements representative of the irradiance of the X-rays at the X-ray detector 503. Furthermore, the memory 512 can store a dedicated program for executing the medical imaging process 100.

[0103] The reconstruction circuitry 514 can perform various steps of the medical imaging process 100. Additionally, the reconstruction circuitry 514 can perform pre-reconstruction and image processing, such as volume rendering and image differencing, if necessary.

[0104] Pre-reconstruction processing of the projection data performed by pre-processing circuitry 506 may include, for example, detector calibration, detector non-linearity, and correction for polar effects.

[0105] Post-reconstruction processing performed by the reconstruction circuitry 514 may include image filtering and smoothing, volume rendering processing, and image differencing processing, as necessary. The image reconstruction processing can implement various steps of the medical imaging process 100. The reconstruction circuitry 514 can use memory to store, for example, projection data, reconstructed images, calibration data and parameters, and computer programs.

[0106] The reconfiguration circuit 514 may include a CPU (processing circuitry) that may be realized as discrete logic gates, such as an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other Complex Programmable Logic Device (CPLD). The FPGA or CPLD implementation may be coded in VHDL, Verilog, or other hardware description language, and the code may be stored directly in the electronic memory in the FPGA or CPLD or as a separate electronic memory. Furthermore, the memory 512 may be non-volatile, such as ROM, EPROM, EEPROM, or FLASH memory. The memory 512 may also be volatile, such as static or dynamic RAM, and a processor, such as a microcontroller or microprocessor, may be provided to manage the electronic memory and the coordination between the FPGA or CPLD and the memory.

[0107] The CPU in the reconstruction circuit 514 may also execute a computer program including a set of computer-readable instructions to perform the functions described in the embodiment, the program being stored in any of the above non-transitory electronic memories and / or hard disk drives, CDs, DVDs, FLASH drives, or any other known storage media. Furthermore, the computer-readable instructions may be provided as a utility application, a background daemon, or a component of an operating system, or a combination thereof, and operate with a processor such as the Intel Xenon processor or the AMD Opteron processor, and with an operating system such as Microsoft VISTA, UNIX, Solaris, LINUX, Apple, MAC-OS, and other operating systems known to those skilled in the art. Furthermore, the CPU may be implemented as multiple processors operating in parallel and cooperatively to execute instructions.

[0108] In one embodiment, the reconstructed image may be displayed on a display 516. The display 516 may be an LCD display, a CRT display, a plasma display, an OLED, an LED, or any other display known in the art. The memory 512 may be a hard disk drive, a CD-ROM drive, a DVD drive, a FLASH drive, RAM, ROM, or any other electronic storage device known in the art.

[0109] According to at least one of the embodiments described above, the image quality of medical images can be improved.

[0110] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0111] 40 Medical Imaging Systems 500 Gantry 514 Reconfigurable circuit

Claims

1. 1. A method of training a neural network using a data set of an input image that is low quality data and a target image that is high quality data having less noise than the low quality data, comprising: The target image is A plurality of different images are obtained by performing a plurality of different reconstructions on medical data collected from a subject, Extracting different feature regions from each of the acquired images; A neural network learning method in which the extracted different feature regions are combined into one image.

2. The method of claim 1 , wherein the different feature regions represent different anatomical structures or organs within an image.

3. The method of claim 1 , wherein the different feature regions are determined based on image processing for specific textures, structures, or information from an atlas of the human body.

4. 2. The method of claim 1, wherein the neural network is trained by iteratively adjusting weighting coefficients to minimize a loss function that represents the difference between the target image and an output image generated by applying the corresponding input image to the neural network.

5. 2. The neural network training method of claim 1, wherein the input image or the target image is an image generated by one of X-ray computed tomography (CT) data, gamma ray positron emission tomography (PET), single-photon emission CT (SPECT) data, and magnetic resonance imaging (MRI) data.

6. A target image is high-quality data with less noise than low-quality data, and a plurality of different images are obtained by performing a plurality of different reconstructions on medical data collected from a subject, different feature regions are extracted from each of the plurality of obtained images, and the extracted different feature regions are combined into one image to generate a target image. A neural network is trained using a data set including the target image and an input image which is the low-quality data. A program that causes a computer to carry out processing.

7. Acquire a target image; A medical device including a circuit for applying a neural network trained on a plurality of pairs of a low-quality image and a high-quality image having less noise than the low-quality image to an acquired target image, thereby generating an output image whose quality is improved by the neural network, the high-quality image is an integrated medical image obtained by acquiring scan data obtained by scanning an area of ​​a subject, reconstructing a plurality of medical images under a plurality of different parameters using the scan data, extracting a plurality of features from different ones of the reconstructed medical images, each of the features corresponding to one of the plurality of different parameters, and integrating the extracted features; A medical apparatus, wherein each of the plurality of different parameters is a parameter for optimizing image quality of a corresponding feature.

8. 8. The medical device of claim 7, wherein the circuitry uses the scan data to obtain the plurality of medical images, including a first image reconstructed under first parameters corresponding to a first region of the subject, and a second image reconstructed under second parameters corresponding to a second region of the subject.

9. 9. The medical device of claim 8, wherein the circuitry extracts, as the plurality of different parameters, a first feature corresponding to the first region from the first image and a second feature corresponding to the second region from the second image.

10. The medical device of claim 8 , wherein the first parameter is a parameter for optimizing image quality of the first region, and the second parameter is a parameter for optimizing image quality of the second region.

11. The circuitry obtains the object image using a first reconstruction method; the reduced quality image is reconstructed using the first reconstruction method; The medical apparatus of claim 7 , wherein the plurality of medical images are reconstructed using an iterative reconstruction method.

12. The medical device of claim 7 , wherein the process of generating the high quality image is comprised in an offline process.

13. Acquire a target image; 1. A medical image processing method comprising: applying a neural network to an acquired target image, the neural network being trained based on a plurality of pairs of a low-quality image and a high-quality image having less noise than the low-quality image, to generate an output image having improved quality by the neural network, the high-quality image is an integrated medical image obtained by acquiring scan data obtained by scanning an area of ​​a subject, reconstructing a plurality of medical images under a plurality of different parameters using the scan data, extracting a plurality of features from different ones of the reconstructed medical images, each of the features corresponding to one of the plurality of different parameters, and integrating the extracted features; A medical image processing method, wherein each of the plurality of different parameters is a parameter for optimizing image quality of a corresponding feature.

Images