Image processing method, program, and image processing apparatus

Abstract

To improve image quality. [Solution] The image processing method according to the embodiment obtains a diffusion-based probabilistic model trained to perform denoising over a plurality of denoising steps using a target image and a conditional image, repeatedly performs the denoising steps based on the probabilistic model and a first seed image function to generate an intermediate image from the input image, and repeatedly performs the denoising steps based on the probabilistic model and a second seed image function to generate a denoised output image from the intermediate image.

Description

[Technical Field]

[0001] Embodiments disclosed herein and in the drawings are image processing methods, programs, and image processing apparatus. Regarding. [Background technology]

[0002] Deep learning-based image restoration models, such as denoising diffusion probabilistic models (DDPMs), typically use a series of steps to modify the input image to the desired degree.

[0003] However, the desirable properties of the reconstructed image may vary based on numerous controllable and uncontrollable factors. Furthermore, the image reconstruction process used by the deep learning model may influence the properties of the reconstructed image. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] U.S. Patent No. 2024 / 0135611 [Patent Document 2] U.S. Patent No. 2024 / 0161864 [Patent Document 3] U.S. Patent No. 2015 / 0093010 [Overview of the project] [Problems that the invention aims to solve]

[0005] One of the problems that the embodiments disclosed in this specification and drawings aim to solve is to improve image quality. However, the problems that the embodiments disclosed in this specification and drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0006] The image processing method according to the embodiment obtains a diffusion-based probabilistic model trained to perform denoising over a plurality of denoising steps using a target image and a conditional image, repeatedly performs the denoising steps based on the probabilistic model and a first seed image function to generate an intermediate image from the input image, and repeatedly performs the denoising steps based on the probabilistic model and a second seed image function to generate a denoised output image from the intermediate image. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows a series of noise-reduced images according to the embodiment. [Figure 2A] Figure 2A shows a noise-reduced image according to the embodiment. [Figure 2B] Figure 2B shows a denoised image using the scaling seed image function according to the embodiment. [Figure 2C] Figure 2C shows a denoised image using the low-pass filter seed image function according to the embodiment. [Figure 3A] Figure 3A shows a denoised image using the scaling seed image function according to the embodiment. [Figure 3B] Figure 3B shows a denoised image using the scaling seed image function according to the embodiment. [Figure 3C] Figure 3C shows a denoised image using the scaling seed image function according to the embodiment. [Figure 4A] Figure 4A shows a denoised image using the low-pass filter seed image function according to the embodiment. [Figure 4B] Figure 4B shows a denoised image using the low-pass filter seed image function according to the embodiment. [Figure 4C]FIG. 4C is a diagram showing a noise-removed image using a low-pass filter seed image function according to an embodiment. [Figure 5] FIG. 5 is a diagram showing a workflow of image noise removal according to an embodiment. [Figure 6] FIG. 6 is a diagram showing a workflow of image noise removal according to an embodiment. [Figure 7] FIG. 7 is a diagram showing a method of image noise removal according to an embodiment. [Figure 8] FIG. 8 is a schematic diagram of a hardware system for executing a method according to an embodiment. [Figure 9] FIG. 9 is a schematic diagram of an imaging system according to an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0008] Hereinafter, embodiments of an image processing method, a program, and an image processing apparatus will be described in detail with reference to the drawings.

[0009] Referring to the drawings, the same reference numerals indicate the same or corresponding parts throughout the plurality of drawings.

[0010] As used herein, the term "plurality" is defined as two or more. As used herein, the term "another" is defined as at least a second or more. As used herein, the terms "comprising" and / or "having" are defined as inclusive (i.e., an open language). Throughout this specification, references to "one embodiment", "a particular embodiment", "an embodiment", "an implementation", "an example", or similar terms mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases or various places throughout this specification are not necessarily all referring to the same embodiment. Further, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments without limitation.

[0011] In one embodiment, this disclosure relates to a system and method for image restoration using a deep learning-based model. Image restoration techniques may include, but are not limited to, denoising, deblurring, resolution enhancement (e.g., super-resolution imaging), and image / signal reconstruction (e.g., compressed sensing). Each of these techniques can be used independently or in combination to improve the visibility of features in an image. Image restoration has important applications in medical imaging modalities such as computed tomography (CT) scanning and magnetic resonance imaging (MRI), which are often affected by noise from physical interactions within the imaging system. It will be understood that the systems and methods described herein are not limited to medical imaging applications and can be used in a variety of imaging types and techniques. In particular, the methods of this disclosure may be useful for processing image data of any volume (e.g., a series of images or image slices) acquired over a spatial or temporal span.

[0012] Generative deep learning-based models can be used to reduce noise and similar artifacts in acquired images and generate restored images of higher quality than the acquired images. In one embodiment, a generative model can be used to denoise an image by transforming a first data distribution (noised image data) into a second data distribution (restored image data). In one embodiment, the generative model may be a denoising diffusion probabilistic model (DDPM). The DDPM is described herein as an exemplary example of a class of generative models, and it may be understood that other types of probabilistic models, particularly diffusion-based probabilistic models for image restoration, are also suitable for the methods of this disclosure.

[0013] DDPM can be used to denoise an image in a series of diffusion steps. DDPM can be trained to denoise an image in an iterative process, generating an increasingly denoised image with each diffusion step. DDPM can be trained to denoise an image by transforming a first probability distribution corresponding to the input image (e.g., a noisy image) into a predicted second probability distribution corresponding to the output image (e.g., a denoised image). In one example, the first probability distribution could be a normal distribution corresponding to the normal (Gaussian) noise present in the acquired image. DDPM can be trained to denoise by transforming the normal probability distribution into a predicted distribution corresponding to the denoised and restored image.

[0014] In one embodiment, DDPM can be trained using a set or sequence of training images. The set of training images may include a target image, which is a clean or denoised image, and a noisy image generated from the target image. In one embodiment, the noisy training image can be generated by applying modeled noise (e.g., Gaussian noise) to the target image in one or more steps. The set of training images may include images with increasing amounts of noise. The set of training images may further include pure noise images generated from the target image. In one embodiment, the modeled noise may be similar to, or based on, the type of noise expected in the acquired image used by DDPM for reconstruction. In one embodiment, the target image may be similar to, or based on, the type of image used by DDPM for reconstruction. Each training image can be input to DDPM. DDPM can be trained to denoise the input training images and output a reconstructed image at each step of a series of diffusion steps. The series of diffusion steps may correspond to one or more steps used to apply noise to the target image. In this way, DDPM can be trained to "reverse" the stepwise process of applying noise to an image in order to remove noise from the image.

[0015] In each diffusion step, the DDPM can be trained to minimize a loss function, which corresponds to the difference in noise between the predicted output image and the training image for a given diffusion step. Thus, the DDPM can be trained to accurately predict and model the difference in noise between each input and output image at each diffusion step. In one embodiment, training the DDPM may include setting one or more weights of the model. One or more weights of the model may change for each diffusion step in a set of diffusion steps, or for at least one diffusion step in that set of steps. In one embodiment, a conditional image may be input to the DDPM during the training process to guide the generation of the output image. In one embodiment, the conditional image may be a target image. The target image used to train the DDPM may be at least one target image, and may include multiple target images. For example, at least one target image may include a low-resolution medical image (e.g., a CT image) and an edge-detected medical image (e.g., a CT image) or a medical image processed in any other way. Similarly, the conditional image used to train the DDPM may be at least one conditional image, and may include multiple conditional images. In one example, at least one conditional image may include a low-resolution medical image (e.g., a CT image) and an edge-detected medical image (e.g., a CT image), or a medical image processed in another way. In one example, at least one conditional image or at least one target image may include three consecutive conditional images for a multi-dimensional (e.g., 2.5-dimensional) process.

[0016] In one embodiment, the input to a trained DDPM may be a noisy image, a conditional image, and a diffusion step (also called a time step or sampling step). The DDPM can predict a second probability distribution corresponding to the reconstructed image, using the conditional image as a known condition. In one embodiment, the DDPM can denoise a pure noisy image in a series of diffusion steps to generate the final reconstructed image. The pure noisy image may be the initial input image to the DDPM. The DDPM may output a denoised image (also called a reconstructed image herein) after each diffusion step. The denoised images output from each diffusion step in the series of diffusion steps may be input to the next diffusion step to iteratively denoise the pure noisy image. The DDPM may include one or more learned weights used to output the reconstructed image, the values ​​of which may depend on the diffusion step. Further details regarding the training and use of DDPM can be found in Ho, J. et al., (2020). “Denoising diffusion probabilistic models.” Advances in neural information processing systems, 33, 6840-6851, and in Xia, W. et al., (2022). “Low-Dose CT Using Denoising Diffusion Probabilistic Model for 20x times Speedup.” arXiv preprint arXiv:2209.15136, each of which is incorporated herein by reference in its entirety for all purposes.

[0017] Images reconstructed using deep learning models may vary in quality (e.g., noise, resolution, image texture, contrast enhancement) and other image characteristics. These variability can stem from differences in the acquisition protocol and quality of the initial noisy input images, pre- and post-reconstruction processing, and the denoising model itself (e.g., model architecture, model training). The relationship between input image characteristics and reconstructed image characteristics is complex and may not be easily modelable, making the characteristics of the reconstructed image unpredictable. Furthermore, depending on the context in which the reconstructed image is used, it may be desirable to give the reconstructed image specific image characteristics. For example, certain medical image analyses may require high-resolution images with reduced noise, while others may require high-fidelity images with minimal smoothing. Building and training a single DDPM to reconstruct images from diverse inputs and according to a variety of quality parameters and requirements can be challenging.

[0018] As a result, it is necessary to adjust the parameters of the DDPM during image restoration to generate a restored image with the desired image characteristics. A tunable DDPM may have broader functionality and can be used to restore input images of various quality levels with predictable results. In one embodiment, a tunable DDPM as described herein can be trained using a single training set in a similar manner to training a non-tunable DDPM. Thus, a tunable DDPM can be used to improve performance without increasing the complexity of training or model fitting.

[0019] Figure 1 shows a denoising process using a trained DDPM according to one embodiment of the present disclosure. Pure noise image x at t=T T This can be input to DDPM. Pure noise image x Tcan be generated using a probability distribution model such as a Gaussian distribution. Conditional images can also be input into the DDPM for noise removal processing. In one embodiment, the conditional image can be an acquired image (e.g., a medical image) to be denoised. The conditional image can include a plurality of conditional images such as an edge detection image or other processed images.

[0020] The DDPM can denoise a pure noise image x in a series of T diffusion steps (e.g., from t = T to t = 1) T and generate a series of restored images x T-1 , x T-2 etc. In each diffusion step, the restored image from the previous diffusion step can be input into the DDPM together with the conditional image and the time step (t = T - 1, t = T - 2, etc.). For example, the DDPM can denoise a pure noise image x T based on the conditional image Y and generate a restored image x T-1 at time step T - 1. The DDPM then denoises the image x T-1 based on the conditional image C and generates a further restored image x T-2 at time step T - 2. At time step t = 1, the DDPM can output an image x0 which is the denoised version of the conditional image Y.

[0021] In one embodiment, the DDPM can generate a restored image at each diffusion step using the following equation (1).

[0022]

Equation

[0023] Here, x t is the input image (e.g., a pure noise image at t = T), x t-1 is the denoised image output, and α t , α t (bar symbol), σ tis a scheduled (time-step dependent) parameter for noise, and ε θ represents a trained model that takes an input image and a time step as input, where Y is the conditional image, t is the time step, and z is a parameter that follows a normal distribution. For example, z can be a Gaussian noise image following a unit normal distribution N(0,1).

[0024] In one embodiment, DDPM is a noise term σ added to the denoised image. t This can be adjusted by adjusting z. In one embodiment, the noise may be an image generated using a piecewise function. More specifically, the noise image z can be input to one or more correction functions to generate a corrected noise image. In one embodiment, the noise image g(z) may be called a seed image or seed noise. The function used to generate the seed image may be called a seed image function or equation. The correction functions and the time intervals to which the correction functions are applied can be set and adjusted to adjust the DDPM. In one embodiment, the adjustable DDPM can generate a restored image at each diffusion step using the following equation (2).

[0025]

number

[0026] During the initial t' time step, the noise image z can be a Gaussian noise image following a unit normal distribution z~N(0,1), and the multiplicative noise parameter σ t This can depend on the time step t. After step t', the unit Gaussian noise image z ~ N(0,1) can be input to the correction function g to generate the corrected noise image g(z), to which the time step-dependent noise parameter σ can be applied. t A noise term can be generated by multiplying it by [this].

[0027] In one embodiment, the function g may be a linear function. In one embodiment, the function g may multiply the input (e.g., z) by a value to scale the input. For example, g(z) = fz, where f is a constant. Thus, the modified noise image g(z) may be a scaled Gaussian noise image if the input z is a unit Gaussian noise image. A linear function can scale the image values ​​(e.g., intensity values) of the noise image to create a wider range of values. In one embodiment, f may be a time-step dependent constant or variable. In one embodiment, the function g may be a nonlinear function.

[0028] In one embodiment, the function g can be a filter. The filter may include, but is not limited to, a low-pass filter, a high-pass filter, a band-pass filter, or a notch filter. In one embodiment, the filter may be an averaging filter. In one embodiment, the filter may be a kernel convolved with the image z. In one embodiment, the filter (e.g., a low-pass filter, a Gaussian filter) can be applied to the noisy image z to reduce noise in the image. The corrected noisy image has a time-step dependent noise parameter σ t It can be multiplied by .

[0029] In one embodiment, the function g can be selected according to the desired quality or characteristics of the reconstructed image. For example, the scaling function g(z)=fz can be used to generate a higher resolution, e.g., a more detailed reconstructed image. A scaled noise image can provide a wider range of noise values ​​for use in image reconstruction. High-resolution reconstructed images can be useful for evaluating details and small features of the reconstructed image. In one embodiment, a high-resolution reconstructed image can be used to evaluate changes in the imaging region over time. In another example, a low-pass filter can be used to smooth and reduce noise in the reconstructed image. A smoothed reconstructed image can be useful for evaluating larger features, such as anatomical structures, within the reconstructed image.

[0030] Figures 2A to 2C show examples of reconstructed images generated by DDPM using different functions g(z) of equation (2) according to one embodiment. Figure 2A is the reconstructed image when g(z) = z. Close-up details of the reconstructed image are also shown in Figure 2A. In this case, the noise image z is the time-step dependent noise parameter σ t This is a unit Gaussian noise image multiplied by [a specific factor]. In one embodiment, the DDPM can be trained using 2000 time steps and a reconstructed image can be generated in 200 time steps.

[0031] Figure 2B is the reconstructed image when g(z) = 1.1z. Close-up details of the reconstructed image are also shown in Figure 2B. The unit Gaussian noise image z can be scaled by a constant value of 1.1. Applying a constant multiplier to image z can yield a more detailed reconstructed image. The reconstructed image in Figure 2B can have a higher resolution than the reconstructed image in Figure 2A. In one embodiment, DDPM can be trained using 2000 time steps and can generate a reconstructed image in 200 time steps. In one embodiment, the correction function g(z) = 1.1z is applied when t < 17 and can generate Figure 2B.

[0032] Figure 2C is the reconstructed image when g(z) is a low-pass filter. Close-up details of the reconstructed image are also shown in Figure 2C. Applying a low-pass filter to image z can result in a smoother reconstructed image. The reconstructed image in Figure 2C may be smoother than the reconstructed images in Figure 2A and Figure 2B. Details in the reconstructed image Figure 2C can be smoothed. In one embodiment, DDPM can be trained using 2000 time steps and can generate a reconstructed image in 200 time steps. In one embodiment, the correction function g(z) is applied when t<16 and can generate Figure 2C.

[0033] Figures 3A to 3C are examples of reconstructed images generated by DDPM using the correction function g(z)=fz of equation (2) according to one embodiment. DDPM can be trained using 2000 time steps and can generate reconstructed images in 200 time steps. The value of the constant multiplier f can vary for each of Figures 3A to 3C. In one embodiment, the correction function g(z)=fz is applied when t<15 and can generate Figures 3A to 3C. The correction function used to generate the reconstructed image in Figure 3A can be g(z)=1.01z. The correction function used to generate the reconstructed image in Figure 3B can be g(z)=1.05z. The correction function used to generate the reconstructed image in Figure 3C can be g(z)=1.1z. Increasing the value of the multiplier f may make the details in the reconstructed image more visible, but it may also increase the noise in the image. The value of f can be set according to the desired image resolution and tolerance for noise.

[0034] In one embodiment, the time step t' to which the function g is applied to the noisy image z can be adjusted. By modifying the time step t', the effect of the function g on the denoising process can be adjusted. For example, a larger t' value may result in the function g being applied to the noisy image z earlier, while a smaller t' value may result in the function g being applied to the noisy image z later. Applying the function g earlier may make the effect of the function (improvement of resolution, smoothing, etc.) more pronounced in the denoised image. In one embodiment, the function g can be applied as an initial condition, for example, at t'=T.

[0035] By adjusting the correction function g(z) and the time interval at which the correction function is applied, different results can be obtained in the output of DDPM. Figures 4A to 4C show examples of DDPM output images when the correction function g(z) is a low-pass filter applied to various time intervals according to equation (2). The low-pass filter can be a Gaussian filter (kernel) with a standard deviation of 7. In one embodiment, DDPM can be trained using 2000 time steps and used to reconstruct an image using 200 time steps.

[0036] Figure 4A shows an example of a reconstructed image when t'=15. Figure 4B shows an example of a reconstructed image when t'=16. Figure 4C shows an example of a reconstructed image when t'=17. The earlier the correction function is applied (the larger the value of t'), the more pronounced the smoothing effect on the reconstructed image becomes. As shown in Figures 4A to 4C, changing the time interval by just one time step can produce a clear difference in the reconstructed image. The time t' at which the correction function is applied can be set based on the desired degree of image smoothing and detail.

[0037] In one embodiment, equation (2) can be further modified to include multiple modification functions (e.g., g1(z), g2(z), etc.). In one embodiment, the functions can have the same form. For example, g1(z) and g2(z) can both be linear functions such that g1(z)=f1z and g2(z)=f2z, where f1 and f2 are constants. In one embodiment, the functions can have different forms. For example, the first function g1(z) can be a low-pass filter, and the second function g2(z) can be a sampling function. In one embodiment, each function can be applied to a time interval. For example, the functions can be applied to equation (3) as follows: In other words, in the image processing method according to the embodiment, the processing circuit included in the device 601 in Figure 8 repeatedly performs a denoising step based on a diffusion-based probabilistic model and a first seed image function to generate a first intermediate image from an input image, repeatedly performs a denoising step based on the same probabilistic model and a second seed image function to generate a second intermediate image from the first intermediate image, and repeatedly performs a denoising step based on the same probabilistic model and a third seed image function to generate a denoised output image from the second intermediate image.

[0038]

number

[0039] In one embodiment, one or more correction functions can be applied to an intermediate time interval. For example, DDPM can generate a reconstructed image according to equation (4). In other words, in the image processing method according to the embodiment, the processing circuit included in the device 601 in Figure 8 repeatedly performs a denoising step based on a diffusion-based probabilistic model and a first seed image function to generate a first intermediate image from an input image, repeatedly performs a denoising step based on the same probabilistic model and a second seed image function to generate a second intermediate image from the first intermediate image, and repeatedly performs a denoising step based on the same probabilistic model and a first seed image function to generate a denoised output image from the second intermediate image.

[0040]

number

[0041] It will be understood that piecewise functions allow operations on any number and combination to be performed at time intervals of any number and length.

[0042] The DDPM is adjusted by modifying a piecewise function that applies functions g(z) and g(z) to generate noise terms. Thus, one DDPM can be used to generate different reconstructed images from an input image. As a result of the function g(z) used to generate the noise image, the reconstructed images may have different characteristics. In one embodiment, a DDPM as described herein can be used in an iterative reconstruction process. For example, a DDPM can be used to generate a first reconstructed image based on an input image in a first iteration. The piecewise function can be modified, and a second reconstructed image can be used to generate a second reconstructed image based on an input image in a second iteration.

[0043] Figure 5 shows an iterative workflow of image restoration according to one embodiment of the present disclosure. DDPM can generate a restored image according to equation (2). In the first iteration, the noise image z used for restoration can be a Gaussian noise image following a unit normal distribution N(0,1). In other words, the correction function can be defined as g(z)=z. Thus, the noise image z is generated using the same function and distribution from t=T to t=t' and from t=t' to t=0. DDPM can generate a first restored image.

[0044] The first restored image can have a certain amount of noise (e.g., Signal-to-Noise Ratio: SNR) and a certain image resolution. The degree of noise and the image resolution may be suitable for a specific purpose (e.g., a specific image analysis process). For other purposes, different degrees of noise and / or image resolutions may be desired. The DDPM can be adjusted to denoise the same input image in a second iteration and generate a second restored image. For example, the modification function g(z) can be modified to be g(z)=fz, where f>1. The scaling function can increase the image resolution so that the second restored image contains more detailed information than the first restored image. In one example, the modification function g(z) can be adjusted so that g(z) becomes a low-pass filter. The low-pass filter can smooth the image so that the third restored image has less noise than the first restored image. The parameters of the modification function g(z) and the time interval at which the modification function is applied can be further modified for further iterations.

[0045] In one embodiment, the modification function g(z) is applied when t<t´, as shown in Equation (2). In this case, even if the function g(z) is modified, it only affects the diffusion step when t<t´. The diffusion step when t≧t´ is the same regardless of the modification function. Therefore, when t´ is constant, it is possible to generate different restored images using different modification functions g(z) by simply repeating the diffusion step when t<t´. For example, the input image has a noise term σ tThe image can be denoised using z in steps Tt' to generate an intermediate denoised image, where T is the total number of steps. At t=t', the first correction function g1(z) can be applied to the remaining time steps t' to generate a reconstructed image. If a different reconstructed image is desired, the same intermediate denoised image generated from step Tt' of the first denoising process can be input to the DDPM at time step t' using a different correction function (e.g., g2(z)). The DDPM can then generate a second (or third, etc.) reconstructed image at step t' instead of step T. In general, the intermediate denoised image output after a first time interval corresponding to the first denoising function can be used as the input image for a second denoising function. This approach can reduce the time and processing power required to generate each reconstructed image.

[0046] Figure 6 illustrates a workflow for adjustable restoration according to one embodiment of the present disclosure. Sample input images to the DDPM (e.g., medical images) may differ in sample features, image quality, acquisition protocol, etc. In one embodiment, the DDPM can be adjusted to suit the type of image restoration based on the characteristics of the input image. For example, enhancing the contrast of a low-quality input image may result in a noisy output image unsuitable for analysis. The parameters of the DDPM (e.g., time intervals defined by the correction functions g(z) and t') can be adjusted according to the input image. These parameters may include, but are not limited to, the level or degree of denoising, resolution, and the level or degree of contrast enhancement. In one embodiment, less denoising and contrast enhancement may be referred to as a “low-risk” restoration process because it is likely to result in less inference and feature loss in the output image. Stronger denoising and contrast enhancement may be referred to as a “high-risk” restoration process because feature loss may occur as a result of smoothing and / or inference (e.g., by a low-pass filter) applied to generate the output image.

[0047] In one embodiment, the characteristics of the output image can be associated with tuning parameters of the DDPM. In one embodiment, the system described herein may include a user interface configured to receive input. For example, a computer (such as a server) that performs a restoration process using the DDPM may receive input data indicating desired characteristics of the output image or the restoration process. These characteristics may include, but are not limited to, image quality, noise level or degree, contrast level, range of image values ​​(e.g., pixel values), resolution, minimum feature size, and the difference between the input and output images. In one example, the user interface may receive an image quality selection. Furthermore, or alternatively, as described with reference to Figure 6, the user interface may receive input data indicating characteristics of the input image. The characteristics of the input image, output image, and / or restoration process can be used to determine a time interval defined by tunable parameters of the image restoration process, such as a correction function g(z) or t'. In one example, the correction function may be defined as g(z) = fz based on the input data received by the computer. The value of the constant f can be set based on the input data received by the computer, such as the desired degree of contrast enhancement of the output image. For example, a correction function g(z) can be defined as a low-pass filter based on the input data received by the computer. The cutoff frequency of the low-pass filter can be set based on the input data received by the computer, such as the desired image resolution or minimum feature size. The combination of the correction function and the corresponding time interval can also be determined based on the input data.

[0048] In one embodiment, DDPM can be used to selectively restore regions of an input image. For example, a first region of the input image may contain small features, and a second region of the input image may contain large structures. DDPM can denoise the first region of the input image using a first correction function g1(z) that enhances the contrast and detail of the first region. DDPM can denoise the second region of the input image using a second correction function g2(z) that smooths the second region. In one embodiment, neither the first nor the second correction function g2(z) is applied to the entire input image. Rather, the first correction function g1(z) can be used only to denoise the first region, and the second correction function g2(z) can be used only to denoise the second region. In one embodiment, regions can be defined by pixels or coordinate positions in the input image. For example, the location of a region and the corresponding image restoration characteristics can be input to DDPM to set the correction function and / or the corresponding time interval.

[0049] FIG. 7 is a method for removing noise from an image using DDPM according to an embodiment of the present disclosure. In step 7100, DDPM can be trained to remove noise from an image in a series of iterative diffusion-based steps. In step 7200, DDPM can be used to remove noise from an image in a first series of T diffusion steps. That is, in the image processing method of the embodiment, for example, the processing circuit included in the device 601 of FIG. 8 obtains a diffusion-based probability model trained to perform noise removal over a plurality of noise removal steps using a target image and a conditional image, and repeatedly executes the noise removal steps based on the probability model and the first seed image function to generate an intermediate image from the input image. The input image (for example, a medical image) can be used as the conditional image in the first series of T diffusion steps. In one embodiment, the noise removal step can utilize Equation (2), and the seed image function can be g(z)=z when t<t'. In other words, the seed noise image used in the first series of diffusion steps can be z for the entire series of T diffusion steps. The noise image z can be a unit Gaussian noise image. DDPM can generate an intermediate noise removal image after each diffusion step. DDPM can output a first output image after the first series of T diffusion steps.

[0050] In one embodiment, the first output image can have specific image characteristics, such as resolution, image resolution. The restored image may desirably have different image characteristics from the first output image. In this case, the seed image function of DDPM can be adjusted to generate a second output image having the desired image characteristics. The second seed image function can be, for example, g(z)=fz. In step 7300, DDPM can be used to denoise an image using the second seed image function in a second series of diffusion steps to generate a second output image. That is, in the image processing method of the embodiment, for example, the processing circuit included in the device 601 of FIG. 8 repeatedly executes a denoising step based on a probabilistic model and the second seed image function to generate a denoised output image from the intermediate image generated in step 7200. Here, the second seed image function is based on, for example, the image characteristics (image resolution, image noise level) of the output image. The second series of diffusion steps can include t' steps instead of T steps, where t<T. In one embodiment, the conditional image used for the second series of diffusion steps can be the intermediate denoised image generated after the T-t' steps of the first series of diffusion steps. In this way, DDPM is only used to repeat t' steps instead of the total T steps of step 7200. The second output image generated after t' steps using the second seed image function can have desired image characteristics different from those of the first output image. The second seed image function may include an application of a scalar multiplier to a unit Gaussian noise image, or may include an application of a low-pass filter to a unit Gaussian noise image. Also, the number of repetitions (T-t') of the denoising step executed when generating the intermediate image from the input image or the number of repetitions (t') of the denoising step executed when generating the output image from the intermediate image may be set based on the image characteristics of the output image.

[0051] In one embodiment, the method disclosed herein can be used in a bulk diffusion method of image denoising that utilizes the similarity between adjacent images within a series of images to achieve faster denoising of a series of images using a diffusion-based probabilistic model. The method described herein can reduce the number of diffusion steps required to denoise a series of images while maintaining inference accuracy. Reducing the number of diffusion steps can result in not only faster denoising but also a reduction in computational power usage.

[0052] In one embodiment, DDPM can be used to denoise a series of images. A series of images can be acquired sequentially over time or spatial dimensions. For example, a series of images can be acquired over time by a kidney scan to evaluate kidney function. In the denoising process (also called the inference process or sampling process), DDPM can denoise in an initial denoising (diffusion) step to identify larger or more generalized features. These larger features are usually consistent across a series of images. For example, the general shape and location of the kidney, as well as the structures within it, can be identified first in the scan images and are unlikely to change within a single kidney scan. Then, in a later diffusion step, DDPM can denoise to identify smaller features or details. In the kidney scan example, DDPM can identify finer details of the shape and size of the kidney structure, as well as the location of the contrast agent within the kidney. These details may change across a series of images as the renal system processes the body's fluids. Changes across a series of images are likely to be gradual and continuous over time. Therefore, adjacent images within a series may be similar to each other.

[0053] In another example, a series of images can be obtained by scanning one or more cross-sections of the body along one or more directions. Similarly, DDPM first denoises the image so that larger features, such as the general shape of the cross-sections of the body and the organs within them, can be identified. DDPM then denoises the image again so that smaller features and / or finer details of the organs can be identified. Adjacent images in a series also depict parts of the body that are close to each other. Therefore, as the scan progresses along the body, adjacent images in a series are likely to be similar to each other and share large features.

[0054] In one embodiment, the method may include grouping images into a series of images and performing initial batch denoising on the image groups using a trained DDPM. An image group may be a subset of adjacent images in a series of images collected over a period of time or across space. A series of images may be divided into one or more groups, each group may contain one or more images. Each of the one or more image groups may have the same or different number of images. For example, a first image group G0 may contain the first n images in a series of images, and a second image group G1 may contain the subsequent n+1 to n+m images in a series of images, and so on. The number of images in a group may be referred to herein as the group thickness. In one embodiment, the group thickness may be adjusted based on the type of image acquisition or the subject being imaged in the series of images. For example, the number of images in a group may be set such that one or more features or types of features (e.g., features of a certain size) are constant in each image of the group. In one embodiment, initial batch denoising may utilize a first correction function g(z) to denoise the image group in a series of diffusion steps. In subsequent denoising steps, the same or different correction functions can be used to refine the restored image.

[0055] Advantageously, the denoising method described herein can be used to gradually alter the restored image quality of medical imaging by adjusting the denoising process in real time (e.g., while the image is being denoised), rather than during the training process or as post-processing. To adjust image quality, a modification function can be applied at any time interval during denoising. This method does not require model retraining. Furthermore, this method does not require a dedicated model architecture or training targeting specific image quality or characteristics. A single DDPM, such as the one described herein, can be easily modified to adjust the denoising process and restore the image according to the desired parameters.

[0056] In one embodiment, the denoising method can be combined with other adjustable machine learning-based image restoration methods. For example, preprocessing can be applied to the input image before denoising. Preprocessing may include, but is not limited to, adjusting the noise level of the acquired image or applying enhancement to the acquired image used as a conditional image. In one example, partial information from the acquired image can be added to the restored output image as postprocessing. In one embodiment, piecewise functions with different noise terms can be integrated into the DDPM regardless of the training process. For example, different models can be trained using different loss functions and / or training sets. In one embodiment, the DDPM can be a parameter-encoded model that can take parameters as input to guide the diffusion process. In one example, the parameters may correspond to image characteristics.

[0057] The systems and methods described herein are compatible with other methods for reducing the processing time of diffusion-based image denoising models. Such methods include, but are not limited to, using denoising diffusion implicit models, reducing the number of diffusion steps T, implementing early termination of the diffusion process, using fast ordinary differential equation solvers, presegmentation diffusion sampling, and using high-frequency spatial diffusion models.

[0058] Next, the hardware of device 601 according to an exemplary embodiment will be described with reference to Figure 8. In Figure 8, device 601 includes a processing circuit. Device 601 can be used to perform any of the methods described herein relating to DDPM acquisition, DDPM training, reception of acquired images, and / or image denoising using DDPM. In one embodiment, device 601 may be a server, a computer, etc. In one embodiment, device 601 may be able to communicate with or be built into an image acquisition device, such as a CT device illustrated in Figure 8. In one embodiment, the methods described herein may be distributed across one or more devices, one or more devices including at least some of the elements of device 601. The processing circuit includes one or more of the elements described below with reference to Figure 8. Process data and instructions may be stored in memory 602. These processes and instructions may also be stored on a storage medium disk 604, such as a hard drive (HDD) or portable storage medium, or may be stored remotely. Furthermore, the claimed advances are not limited to the form of computer-readable medium in which the instructions of the present invention process are stored. For example, the instructions may be stored on a CD, DVD, flash memory (registered trademark), RAM, ROM, PROM (registered trademark), EPROM (registered trademark), EEPROM (registered trademark), hard disk, or other information processing device with which device 601 communicates, such as a server or computer.

[0059] Furthermore, the claimed advances may be provided as utility applications, background daemons, or components of an operating system, or a combination thereof, and will run in cooperation with the CPU600 and an operating system such as Microsoft® Windows®, UNIX®, Solaris®, LINUX®, Apple® MAC-OS®, and other systems known to those skilled in the art.

[0060] The hardware elements for realizing device 601 can be implemented by various circuit elements known to those skilled in the art. For example, CPU 600 may be an Intel® Xenon® or Core® processor, or an AMD® Opteron® processor, or any other processor type recognized to those skilled in the art. Alternatively, CPU 600 may be implemented using an FPGA, ASIC, PLD, or discrete logic circuit, as recognized to those skilled in the art. Furthermore, CPU 600 may be implemented as multiple processors working in parallel to execute the instructions of the process described above.

[0061] Device 601 in Figure 8 also includes a network controller 606, such as an Intel® Ethernet® PRO network interface card from Intel Corporation, for interface with network 650 and for communicating with other devices. As can be understood, network 650 can be a public network such as the Internet, a private network such as a LAN or WAN network, or any combination thereof, and may also include a PSTN or ISDN subnetwork. Network 650 can be wired, such as an Ethernet® network, or wireless, such as a cellular network, including EDGE, 3G, 4G, and 5G wireless cellular systems. The wireless network may be WiFi®, Bluetooth®, or any other known wireless communication format.

[0062] Device 601 further includes a display controller 608, such as an NVIDIA® GeForce GTX® or Quadro graphics adapter® from NVIDIA, Inc., USA, for interface with a display 610, such as an LCD monitor. A general-purpose I / O interface 612 interfaces with a keyboard / mouse 614, as well as a touchscreen 616, either on or separate from the display 610. The general-purpose I / O interface also connects to various peripherals 618, including printers and scanners.

[0063] A sound controller 620 is also provided in device 601 and provides sound and / or music by interface with speaker / microphone 622.

[0064] The general-purpose storage controller 624 connects the storage medium disk 604 to a communication bus 626, which may be ISA, EISA, VESA, PCI, or similar, in order to interconnect all the components of device 601. A general description of the features and functions of the display 610, keyboard / mouse 614, as well as the display controller 608, general-purpose storage controller 624, network controller 606, sound controller 620, and general-purpose I / O interface 612 is omitted herein for brevity, as these features are known.

[0065] In one embodiment, the image processed using the bulk diffusion method described herein may be a CT image acquired by a CT scanner or CT apparatus. Figure 9 shows an implementation of a radiation gantry included in a CT scanner or CT apparatus. As shown in Figure 9, the radiation gantry 9900, shown in a side view, further includes an X-ray tube 9901, an annular frame 9902, and a multi-row or two-dimensional array type X-ray detector 9903. The X-ray tube 9901 and X-ray detector 9903 are radially mounted on the annular frame 9902, which is rotatably supported around a rotation axis RA, with a subject, such as a patient, in between. A rotation unit 9907 rotates the annular frame 9902 at a high speed, such as 0.4 seconds / revolution, while the subject moves along axis RA in the back or front direction of the illustrated page.

[0066] Embodiments of the X-ray computed tomography (CT) apparatus described herein will be described below with reference to the accompanying drawings. Note that X-ray CT apparatuses include various types of devices, such as rotary / rotating devices in which the X-ray tube and X-ray detector rotate together around the subject being examined, and fixed / rotating devices in which many detector elements are arranged in annular or planar manner, and only the X-ray tube rotates around the subject being examined. This disclosure can be applied to either type. Here, the currently dominant rotary / rotating type is given as an example.

[0067] The multislice X-ray CT apparatus further includes a high-voltage generator 9909, which generates a tube voltage applied to the X-ray tube 9901 through a slip ring 9908 so that the X-ray tube 9901 generates X-rays. An X-ray detector 9903 is located on the opposite side of the subject from the X-ray tube 9901 to detect the irradiation X-rays that have propagated through the subject. The X-ray detector 9903 is, for example, a photon-counting type detector. The X-ray detector, or photon-counting type X-ray detector 9903, further includes individual detector elements or units, such as processing circuits.

[0068] The CT scanner further includes other devices for processing detection signals from the X-ray detector 9903. The data acquisition circuit or data acquisition system (DAS) 9904 converts the signals output from each channel's X-ray detector 9903 into voltage signals, amplifies those signals, and then converts those signals into digital signals. The X-ray detector 9903 and the data acquisition system 9904 are configured to manage a predetermined total number of projections per rotation (TPPR).

[0069] The data described above is sent via a non-contact data transmitter 9905 to a preprocessing device 9906 housed in a console outside the radiation gantry 9900. The preprocessing device 9906 performs specific corrections. Memory 9912 stores the resulting data, also called projection data, immediately before the reconstruction process. Memory 9912, along with the reconstruction device 9914, input device 9915, and display 9916, is connected to the system controller 9910 via a data / control bus 9911. The system controller 9910 controls a current regulator 9913 that limits the current to a level sufficient to drive the CT system.

[0070] In various generations of CT scanner systems, detectors are rotated and / or fixed relative to the subject being scanned, such as a patient. In one embodiment, the CT system described above may be an example of a combination of a third-generation geometry system and a fourth-generation geometry system. In the third-generation system, the X-ray tube 9901 and X-ray detector 9903 are radially mounted on an annular frame 9902, and the annular frame 9902 rotates around the rotation axis RA, causing them to rotate around the subject. In the fourth-generation geometry system, the detectors are fixedly positioned around the patient, and the X-ray tube 9901 rotates around the patient. In an alternative embodiment, the X-ray imaging gantry 9900 has a number of detectors arranged on an annular frame 9902 supported by a C-arm and a stand.

[0071] The post-reconstruction processing performed by the reconstruction device 9914 may include, as necessary, image filtering and smoothing, volume rendering, and image subtraction. The image reconstruction process can implement various CT image reconstruction methods. The reconstruction device 9914 can use memory to store, for example, projection data, reconstructed images, calibration data and parameters, and computer programs.

[0072] This specification includes numerous specific implementation details, which should not be construed as limitations on the scope of claims, but rather as descriptions of features that may be specific to particular embodiments.

[0073] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, although features are described above as acting in a particular combination, and may even be initially claimed to act as such, in some instances one or more features of the claimed combination may be removed from that combination. Alternatively, the claimed combination may be directed towards a subcombination or a variation of a subcombination.

[0074] Similarly, while operations are shown in a specific order in the drawings, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and that the described program components and systems may generally be combined into a single software product or packaged into multiple software products.

[0075] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the operations described in the claims can achieve the desired results even if they are performed in a different order. As an example, the process depicted in the accompanying drawings does not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0076] According to at least one embodiment described above, image quality can be improved.

[0077] Furthermore, embodiments of this disclosure may be described in the following appendix.

[0078] (1) A method for denoising an input image, comprising: acquiring a diffusion-based probabilistic model trained to perform denoising over a number of denoising steps using at least one target image and at least one condition image; performing a first sequence of T1 denoising sampling steps based on the input image using the acquired model and a first seed image to generate a first output image which is a first reconstructed image corresponding to the input image, wherein each denoising sampling step in the first sequence of T1 denoising sampling steps comprises generating a first seed image according to a first seed image function; and performing a second sequence of T2 denoising sampling steps based on an intermediate image generated as a result of the first sequence of T1 denoising sampling steps using the acquired model and a second seed image to generate a second output image which is a second reconstructed image corresponding to the input image, wherein each denoising sampling step in the second sequence of T2 denoising sampling steps comprises generating a second seed image according to a second seed image function.

[0079] (2) The method according to (1), wherein the second seed image function is based on the desired image characteristics of the second output image.

[0080] (3) The method according to (1) or (2), wherein the desired image characteristics are the desired image resolution.

[0081] (4) The method according to (1) to (3), wherein the desired image characteristic is the desired image noise level.

[0082] (5) A second seed image function is the method according to (1) to (4), which includes applying a scalar multiplier to a unit Gaussian noise image.

[0083] (6) A second seed image function is the method according to (1) to (5), which includes applying a low-pass filter to a unit Gaussian noise image.

[0084] (7) The method according to (1) to (6), wherein the value of T2 is set based on the desired image characteristics of the second output image.

[0085] (8) A non-temporary computer-readable storage medium for storing computer-readable instructions that cause a computer to perform a method, the method comprising: acquiring a diffusion-based probabilistic model trained to perform denoising over a number of denoising steps using at least one target image and at least one condition image; and performing a first sequence of T1 denoising sampling steps based on the input image using the acquired model and a first seed image to generate a first output image which is a first reconstructed image corresponding to the input image, wherein each noise in the first sequence of T1 denoising sampling steps A non-temporary computer-readable storage medium comprising: performing a denoising sampling step which includes generating a first seed image according to a first seed image function; and performing a second sequence of T2 denoising sampling steps based on an intermediate image generated as a result of a first sequence of T1 denoising sampling steps, using the acquired model and the second seed image to generate a second output image which is a second restored image corresponding to the input image, wherein each denoising sampling step in the second sequence of T2 denoising sampling steps includes generating a second seed image according to a second seed image function.

[0086] (9) The second seed image function is a non-temporary computer-readable storage medium as described in (8), based on the desired image characteristics of the second output image.

[0087] (10) A non-temporary computer-readable storage medium as described in (8) or (9), wherein the desired image characteristics are the desired image resolution.

[0088] (11) A non-temporary computer-readable storage medium according to (8) to (10), wherein the desired image characteristics are the desired image noise level.

[0089] (12) A non-temporary computer-readable storage medium as described in (8) to (11), wherein the second seed image function includes applying a scalar multiplier to a unit Gaussian noise image.

[0090] (13) A non-temporary computer-readable storage medium as described in (8) to (12), wherein the second seed image function includes applying a low-pass filter to a unit Gaussian noise image.

[0091] (14) The value of T2 is set based on desired image characteristics of the second output image, in the non-temporary computer-readable storage medium as described in (8) to (13).

[0092] (15) An apparatus comprising a processing circuit configured to perform: (15) acquiring a diffusion-based probabilistic model trained to perform denoising over a number of denoising steps using at least one target image and at least one condition image; (15) performing a first sequence of T1 denoising sampling steps based on an input image using the acquired model and a first seed image to generate a first output image which is a first reconstructed image corresponding to an input image, wherein each denoising sampling step in the first sequence of T1 denoising sampling steps includes generating a first seed image according to a first seed image function; and (15) performing a second sequence of T2 denoising sampling steps based on an intermediate image generated as a result of the first sequence of T1 denoising sampling steps using the acquired model and a second seed image to generate a second output image which is a second reconstructed image corresponding to an input image, wherein each denoising sampling step in the second sequence of T2 denoising sampling steps includes generating a second seed image according to a second seed image function.

[0093] (16) The apparatus according to (15), wherein the second seed image function is based on the desired image characteristics of the second output image.

[0094] (17) The apparatus according to (15) or (16), wherein the desired image characteristic is the desired image resolution.

[0095] (18) The apparatus according to (15) to (17), wherein the desired image characteristic is the desired image noise level.

[0096] (19) The apparatus described in (15) to (18), wherein the second seed image function includes applying a multiplier to a unit Gaussian noise image.

[0097] (20) The apparatus according to (15) to (19), wherein the second seed image function includes applying a low-pass filter to a unit Gaussian noise image.

[0098] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments are possible without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0099] 600 CPU 602 memory 604 Storage media disk 610 displays 612 General-Purpose I / O Interfaces 614 Keyboard / Mouse 616 Touchscreen 618 Peripherals 622 Speakers

Claims

1. Using the target image and conditional image, we obtain a diffusion-based stochastic model trained to perform denoising across multiple denoising steps. Based on the aforementioned probabilistic model and the first seed image function, the denoising step is repeatedly performed to generate an intermediate image from the input image. An image processing method that repeatedly performs a denoising step based on the aforementioned probability model and a second seed image function to generate a denoised output image from the intermediate image.

2. The image processing method according to claim 1, wherein the second seed image function is based on the image characteristics of the output image.

3. The image processing method according to claim 2, wherein the aforementioned image characteristic is image resolution.

4. The image processing method according to claim 2, wherein the aforementioned image characteristic is the image noise level.

5. The image processing method according to claim 1, wherein the second seed image function includes a unit Gaussian noise image to which a scalar multiplier has been applied.

6. The image processing method according to claim 1, wherein the second seed image function includes a unit Gaussian noise image to which a low-pass filter has been applied.

7. The image processing method according to claim 1, wherein the number of repetitions of the noise reduction step performed when generating the intermediate image from the input image, or the number of repetitions of the noise reduction step performed when generating the output image from the intermediate image, is set based on the image characteristics of the output image.

8. On the computer, Using the target image and conditional image, we obtain a diffusion-based stochastic model trained to perform denoising across multiple denoising steps. Based on the aforementioned probabilistic model and the first seed image function, the denoising step is repeatedly performed to generate an intermediate image from the input image. The process involves repeatedly executing a denoising step based on the aforementioned probability model and second seed image function to generate a denoised output image from the intermediate image. program.

9. The program according to claim 8, wherein the second seed image function is based on the image characteristics of the output image.

10. The program according to claim 9, wherein the aforementioned image characteristic is image resolution.

11. The program according to claim 9, wherein the aforementioned image characteristic is the image noise level.

12. The program according to claim 8, wherein the second seed image function includes applying a scalar multiplier to a unit Gaussian noise image.

13. The program according to claim 8, wherein the second seed image function includes a unit Gaussian noise image to which a low-pass filter has been applied.

14. The program according to claim 8, wherein the number of repetitions of the noise reduction step performed when generating the intermediate image from the input image, or the number of repetitions of the noise reduction step performed when generating the output image from the intermediate image, is set based on the image characteristics of the output image.

15. Using the target image and conditional image, we obtain a diffusion-based stochastic model trained to perform denoising across multiple denoising steps. Based on the aforementioned probabilistic model and the first seed image function, the denoising step is repeatedly performed to generate an intermediate image from the input image. An image processing apparatus comprising a processing circuit that repeatedly performs a denoising step based on the aforementioned probability model and a second seed image function to generate a denoised output image from the intermediate image.

16. The image processing apparatus according to claim 15, wherein the second seed image function is based on the image characteristics of the output image.

17. The image processing apparatus according to claim 16, wherein the aforementioned image characteristic is image resolution.

18. The image processing apparatus according to claim 16, wherein the aforementioned image characteristic is the image noise level.

19. The image processing apparatus according to claim 15, wherein the second seed image function includes a unit Gaussian noise image to which a multiplier has been applied.

20. The image processing apparatus according to claim 15, wherein the second seed image function includes a unit Gaussian noise image to which a low-pass filter has been applied.

Images