Image processing method, image processing apparatus, image processing system, and program

Abstract

To provide an image processing method with which it is possible to obtain a result of image processing with reproducibility by using a diffusion model.SOLUTION: An image processing method includes: a first step of acquiring input to a diffusion model (S201); a second step of initializing output from the diffusion model (S202); and a third step of iteratively updating the output by using the diffusion model on the basis of the input and output (S203-S205). In the third step, the output is updated by using an estimation result obtained in every iteration by using the diffusion model on the basis of the input and output, and a random number fixed for every iteration.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, an image processing system, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for sharpening an image using a diffusion model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 1,1769,228 Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Patent Document 1 cannot obtain reproducible image processing results using a diffusion model. This is because, as will be described later, conventional image processing using a diffusion model involves randomness in the process.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing method that can obtain reproducible image processing results using a diffusion model. [Means for solving the problem]

[0006] An image processing method according to one aspect of the present invention includes a first step of acquiring an input to a diffusion model, a second step of initializing an output from the diffusion model, and a third step of iteratively updating the output by using the diffusion model based on the input and the output, wherein in the third step, the output is updated by using an estimation result obtained for each iteration by using the diffusion model based on the input and the output, and a random number fixed for each iteration.

[0007] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing method that can obtain reproducible image processing results using a diffusion model. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing a flow of learning of a neural network in the first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 4 is a flowchart relating to learning of a neural network in the first embodiment. [Figure 5] 10 is a flowchart relating to generation of an output image using a neural network in the first and second embodiments. [Figure 6] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 7] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 8] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 9] 11 is a flowchart relating to generation of an output image using a neural network in the third embodiment. [Figure 10] FIG. 1 is an explanatory diagram of the gist of each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] First, before providing a detailed description of each embodiment, we will explain the diffusion model and the gist of each embodiment. In each embodiment, a diffusion model is used to perform reproducible image processing. A diffusion model is a type of generative AI (artificial intelligence) primarily used for image generation. By sampling from a target image distribution conditioned by image input or text input, it is possible to generate a high-quality target image corresponding to the input image or text. For example, Stable Diffusion uses a diffusion model to generate realistic images that look like they were drawn by a human in response to text entered by the user.

[0012] Diffusion models consist of two main processes: a forward diffusion process and a reverse diffusion process. The diffusion process incrementally adds Gaussian noise to a target image, slowly removing details from the target image until it becomes a pure noise image. The reverse diffusion process starts with a pure noise image, and reverses the diffusion process by conditioning it with image and text inputs and recovering it iteratively. The reverse diffusion process can be trained using a neural network (machine learning model) that takes a noisy target image and a conditioning image (or signal such as text) as inputs to remove noise. Details will be provided later.

[0013] The diffusion process is given by the following equation (1):

[0014]

number

[0015] where y t-1 and y t represents the step (iteration number) t-1 of the diffusion process q given by the Markov chain and the intermediate image of the next step t (t=1~T), respectively. N is a standard normal distribution (Gaussian distribution), α t (0<αt<1) is y at step t t-1 respectively represent the variance of the Gaussian noise to be added to

[0016] By transforming (marginalizing) equation (1), the intermediate image y given the target image y0 is t The distribution of is given by the following equation (2):

[0017]

number

[0018] where γ t is added to the target image y0 and t In other words, equation (2) expresses the variance of the Gaussian noise to obtain the intermediate image y t-1 Gaussian noise is added to the intermediate image y t By adding Gaussian noise to the target image y0, the intermediate image y t This indicates that it is possible to obtain

[0019] Further modifying equations (1) and (2) gives the target image y0 and the intermediate image y t Given the intermediate image y t-1 The posterior distribution of is given by the following equation (3):

[0020]

number

[0021] where μ and σ 2 is the target image y0 and the intermediate image y t The mean and variance of the standard normal distribution, which is the posterior distribution of the intermediate image yt-1 when t and the mean μ obtained from the target image y0, and the variance σ 2 This means that the intermediate image yt-1 at the t-1th step can be estimated by adding Gaussian noise with

[0022] The despreading process is given by the following equation (4).

[0023]

number

[0024] Here, the inverse diffusion process p(t=T~1) is defined as an inverse Markov chain that proceeds in the reverse direction of the diffusion process q. Note that, following the diffusion process q, the pure noise image y T We start with Gaussian noise sampled from a standard normal distribution (the second equation in equation (4)). In addition, each step of the de-diffusion process (the third equation in equation (4)) can be defined as a trainable isotropic Gaussian conditional distribution if the variance of the Gaussian noise added at each step of the diffusion process is set small.

[0025] The mean μ of the Gaussian distribution at each step of the dediffusion process θ is calculated from the posterior distribution of Equation (3), but the target image y0 is required. However, the pure noise image y T In the despreading process starting from , the target image y0 is generally unknown.

[0026] Therefore, in the diffusion model, a neural network (machine learning model) is trained to remove noise from a noise image in which Gaussian noise with various variances is added to the target image y0, and then the intermediate image y t The noise-removed target image y0^ is estimated from

[0027] The target image y0^ after noise removal in the de-diffusion process is given by the following equation (5).

[0028]

number

[0029] Here, f is a noise removal neural network with θ as a parameter, and x is a condition used for noise removal. As a condition used for noise removal, for example, a vector with embedded text is used in Stable Diffusion, and an interpolated image of a low-resolution image is used in Patent Document 1. Also, in Patent Document 1, an intermediate image y t The interpolated image is connected in the channel direction and input to the neural network, and noise is removed by conditioning the interpolated image. In addition, the noise removal neural network is also fed with the noise level (noise strength) γ t is introduced and noise is removed according to the noise level. Note that in each embodiment, the variance of the Gaussian noise added in the diffusion process is used as the noise level, but this is not limiting. A method for inputting the noise level into the noise removal neural network will be described later.

[0030] By substituting the estimated target image y0^ into equation (3), the mean of the posterior distribution is given parametrically by the following equation (6).

[0031]

number

[0032] The variance of the Gaussian noise is the variance of the Gaussian noise added in the diffusion process, α t Using this, each step of the despreading process is given by the following equation (7).

[0033]

number

[0034] By repeating this procedure from t=T to 1, we obtain a pure noise image y T The target image y0 can be generated from (more precisely, sampled from the target image distribution conditioned on x). The diffusion model is also called the Denoised Diffusion Probabilistic Model (DDPM).

[0035] In the prior art such as Patent Document 1, the intermediate image y t The estimated target image y0^ is then converted to the intermediate image y t-1 However, this does not allow the standard normal random number ε t Therefore, even if the same image is processed with a diffusion model (even if conditioned on the same image or text), the results will be randomly different each time.

[0036] Therefore, in each embodiment, instead of equation (7), the intermediate image y is calculated according to the following equation (8): t The estimated target image y0^ is then converted to the intermediate image y t-1 Ask for.

[0037]

number

[0038] where rand is a pseudorandom number generator, s t is the random number seed used in the tth iteration. Note that a pseudo-random number is a pseudo-random number calculated by a deterministic calculation, and a random number seed is the initial state (initial value of the random number) when generating the pseudo-random number. For example, the linear congruential method, one type of pseudo-random number, generates a sequence of random numbers according to a certain recurrence formula, and the random number seed is used as the initial value of the recurrence formula. Therefore, with pseudo-random numbers, the same random number can be generated if the random number seed is the same, and it is used when performing reproducible simulations using computers that can only perform deterministic calculations.

[0039] In each embodiment, a value based on the number of iterations is used as the random number seed used to generate pseudo-random numbers. That is, the pseudo-random numbers are generated by determining the random number seed based on the number of iterations. Therefore, although different Gaussian noise (pseudo-random numbers) are used between iterations of the dediffusion process, fixed Gaussian noise (fixed random numbers) are used for each iteration. This makes it possible to maintain high image quality of images generated by the diffusion model while achieving reproducibility of the process. Note that in each embodiment, the random numbers are not limited to pseudo-random numbers, and other random numbers such as uniform random numbers, normal random numbers, or standard normal random numbers may also be used.

[0040] Next, an overview (gist) of each embodiment will be described using upscaling (super-resolution) using a diffusion model as an example, with reference to Fig. 10. Fig. 10 is an explanatory diagram of the gist of each embodiment. Here, upscaling refers to image processing that generates a high-resolution image by enlarging the image by estimating high-frequency components that cannot be expressed in a low-resolution image.

[0041] In each embodiment, first, a low-resolution image is acquired and a bicubic interpolated image (input, corresponding to x in the formula) is generated. Next, the current intermediate image (output, corresponding to y in the formula) is generated. T ) is initialized with Gaussian noise. Note that initialization uses a noise image, which is Gaussian noise generated by pseudo-random numbers with a fixed random number seed. The size of the noise image is the same as that of the interpolated image. Next, the interpolated image is input to the neural network together with the current intermediate image to remove noise. Note that the variance of the noise image, which is the noise level (γ T Information about the noise image (corresponding to the interpolated image) is also input to the neural network. The interpolated image is connected to the current intermediate image in the channel direction and input to the neural network, where it is used as a condition for noise removal. The variance of the noise image is a predetermined value for each iteration of the de-diffusion process. For example, the maximum and minimum values ​​of the variance of the noise image may be determined, and a value determined linearly from the number of iterations may be used. The neural network is also trained in advance to be able to perform noise removal given by equation (5).

[0042] Next, the mean of the posterior distribution given by equation (5) is calculated from the noise-removed image (estimated result, corresponding to y^ in the formula). Finally, the current intermediate image (output, y^ in the formula) is calculated according to equation (8). t-1 (corresponding to ). Note that the random number seed s of the Gaussian noise in Eq. (8) is updated. t The number of iterations of the de-diffusion process is set as t. By repeating the above procedure a predetermined number of times, it is possible to estimate high-frequency components that cannot be expressed in a low-resolution image, and generate an enlarged high-resolution image with high reproducibility.

[0043] Diffusion models are currently used in the field of image generation (text-to-image generation), which creates various images in response to text input, such as Stable Diffusion and Midjourney. In other words, emphasis is placed on random image generation, and reproducibility of image processing is not given much attention. However, when applying diffusion models to conventional image processing such as blur removal, noise removal, upscaling, and debayering, the processing results change each time the image is processed, which poses a problem from the perspective of reproducibility. In such image processing fields, each embodiment can provide reproducible image processing results using a diffusion model.

[0044] The image processing method described above is an example, and the embodiments are not limited to this. Details of other image processing methods will be described later in the following embodiments.

[0045] [Example 1] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, image processing for generating a high-resolution image by upscaling a low-resolution image using a diffusion model is learned and executed.

[0046] Fig. 2 is a block diagram of an image processing system 100 in this embodiment. Fig. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an input device 106, an output device 107, and a network 108.

[0047] The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and a learning unit 101d.

[0048] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives an optical image of the subject formed via the optical system 102a to acquire a captured image (low-resolution image). The imaging element 102b is a charge-coupled device (CCD) sensor, a complementary metal-oxide semiconductor (CMOS) sensor, or the like. Information regarding the shooting conditions of the captured image (such as the pixel pitch of the imaging element 102b, the type of optical low-pass filter, and ISO sensitivity) can be acquired along with the image. Development conditions for the captured image (such as noise reduction strength, sharpness strength, and image compression rate) can also be acquired along with the image. This information acquired along with the image can also be transmitted to an acquisition unit 103b of the image estimation device 103 (described later) along with the image. A storage unit for storing the acquired image, a display unit for displaying it, a transmission unit for transmitting it to an external device, an output unit for storing it in an external storage medium, and the like are not shown. A control unit that controls each unit of the imaging device 102 is also not shown.

[0049] The image estimation device 103 includes a storage unit 103a, an acquisition unit 103b, an initialization unit 103c, a noise removal unit 103d, a random number generation unit 103e, and an update unit 103f. The acquisition unit 103b acquires a low-resolution image (captured image, input). The initialization unit 103c initializes the current intermediate image (output) with Gaussian noise. The noise removal unit 103d estimates (acquires) a noise-removed image (estimated result for each iteration) obtained by removing noise from the current intermediate image using a neural network conditioned on the low-resolution image. The random number generation unit 103e generates Gaussian noise (random numbers). The update unit 103f performs image processing to repeatedly update the current intermediate image using the Gaussian noise and the estimated result, ultimately generating an upscaled high-resolution image (output).

[0050] The method of initializing the current intermediate image by the initialization unit 103c and the method of removing noise from the current intermediate image by the noise removal unit 103d using a neural network will be described in detail later. Also, the method of generating Gaussian noise (random numbers) by the random number generation unit 103e and the method of updating the current intermediate image by the update unit 103f will be described in detail later.

[0051] The low-resolution image may be an image captured by the image capture device 102 or may be an image stored in the recording medium 105 .

[0052] Weight information of the neural network used for noise removal is read from the storage unit 103a. The weights have been learned in advance by the learning device 101. The image estimation device 103 reads the weight information from the storage unit 101a via the network 108 in advance and stores the weight information in the storage unit 103a. The stored weight information may be the weight values ​​themselves or may be in an encoded format. Details of the neural network weight learning method and the image processing method using a diffusion model that uses a neural network will be described later.

[0053] The upscaled high-resolution image is output to at least one of a display device 104, a recording medium 105, and an output device 107. The display device 104 is, for example, a liquid crystal display or a projector. A user can check the image being processed via the display device 104 and perform image editing work via the input device 106. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The input device 106 is, for example, a keyboard or a mouse, etc. The output device 107 is, for example, a printer, etc.

[0054] Next, a method for learning weights of a neural network (machine learning model) (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Fig. 1 and Fig. 4. Fig. 1 is a diagram showing the flow of learning weights of a neural network. Fig. 4 is a flowchart related to weight learning. Each step in Fig. 4 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the learning unit 101d.

[0055] First, in step S101, the acquisition unit 101b acquires a low-resolution patch (degraded image) 201 and a corresponding high-resolution patch (correct image) 200, as shown in FIG. 1. In this embodiment, a patch is a small image having a predetermined number of pixels. For example, a low-resolution patch may be 128 x 128 x 3 pixels (corresponding to the number of pixels in the vertical x horizontal x channel directions), and the corresponding high-resolution patch may be 256 x 256 x 3 pixels. In this case, since the vertical and horizontal sizes are doubled, the upscaling factor is doubled (the number of pixels is enlarged by four times).

[0056] The upscaling factor is not limited to 2x, and may be any factor as long as a low-resolution patch and a corresponding high-resolution patch can be acquired. A low-resolution patch and a corresponding high-resolution patch may be acquired by capturing images of the same subject using optical systems with different focal lengths and cropping corresponding portions from the two images. Alternatively, a corresponding low-resolution patch may be generated by downsampling a high-resolution patch. Alternatively, a corresponding low-resolution patch acquired by the image capture device 102 and a corresponding high-resolution patch with less influence (aberration and diffraction) from the optical system 102a may be generated by numerical calculation. In this embodiment, a low-resolution patch and a corresponding high-resolution patch are generated by numerical calculation, but the present invention is not limited to this.

[0057] Next, in step S102, the generation unit 101c generates an interpolated patch 202 by interpolating and enlarging the low-resolution patch 201 to have the same number of pixels as the corresponding high-resolution patch 200. In this embodiment, bicubic interpolation is used as the interpolation method, but this is not limiting, and other interpolation methods such as nearest neighbor interpolation or bilinear interpolation may also be used.

[0058] Next, in step S103, the generating unit 101c generates a noise patch (noise image) 203 by adding noise to the high-resolution patch 200.

[0059] Next, in step S104, the learning unit 101d uses a neural network to generate a noise-removed patch 204 by removing noise from the noise patch 203. The noise-removed patch 204 is generated using the interpolated patch 202, the noise patch 203, and information on the strength of the noise (noise level) added to the noise patch 203. Note that the noise-removed patch 204 is an estimate of the high-resolution patch 200, and ideally the two will match.

[0060] In this embodiment, the variance of the added Gaussian noise is used as the noise level, but this is not limited to this. Also, in this embodiment, noise removal is performed according to the strength of the noise added to the noise patch 203, but a method for inputting the noise level (variance of the Gaussian noise) to the neural network will be described later. Also, in this embodiment, the interpolation patch 202 and the noise patch 203 are concatenated in the channel direction and input to the neural network, but this is not limited to this.

[0061] Next, in step S105, the learning unit 101d updates the weights of the neural network based on the error between the high-resolution patch 200 and its estimated denoising patch 204. Here, the weights include the filter components and biases of each layer. In this embodiment, back propagation is used to update the weights, but this is not limitative. In mini-batch learning, the error between the high-resolution patch 200 and its corresponding denoising patch 204 is calculated, and the weights are updated. For example, the L2 norm or the L1 norm may be used as the loss function. The weight update method (learning method) is not limited to mini-batch learning, and batch learning or online learning may also be used.

[0062] Next, in step S106, the learning unit 101d determines whether weight learning is complete. Completion can be determined by, for example, whether the number of iterations of learning (weight update) has reached a specified value, or whether the amount of change in weight during update is smaller than a specified value. If it is determined that learning is incomplete, the process returns to step S101, and multiple new low-resolution patches 201 and corresponding high-resolution patches 200 are obtained. On the other hand, if it is determined that learning is complete, the learning device 101 ends learning and saves weight information in the storage unit 101a.

[0063] In this embodiment, a method for learning the weights of a neural network in the image domain has been described, but the method is not limited to this. For example, learning may be performed in the feature map (latent) domain using a noise patch 203 obtained by adding noise to a feature map obtained using a separately prepared neural network. In this case, learning can be performed using a low-resolution patch 201 or an interpolated patch 202.

[0064] In this embodiment, the neural network configuration shown in FIG. 1 is used, but the present invention is not limited to this.

[0065] CN in Figure 1 represents a convolutional layer. CN calculates the sum of the input, filter convolution, and bias, and then performs nonlinear transformation on the result using an activation function. The initial values ​​of each filter component and bias are arbitrary and are determined by random numbers in this embodiment. The activation function can be, for example, a rectified linear unit (ReLU) or a sigmoid function. The multidimensional array output from each layer except the final layer is a feature map. A feature map is generally a four-dimensional array with dimensions of batch, length, width, and channel. The skip connection 205 combines feature maps output from discontinuous layers. Feature maps can be combined by taking element-by-element sums or by concatenating in the channel direction. In this embodiment, feature maps are combined using element-by-element sums.

[0066] The elements (blocks or modules) within the dotted line frame in FIG. 1 represent residual blocks. A network with multiple layers of residual blocks is called a residual network, and is widely used in image processing using machine learning models. However, this embodiment is not limited to this, and a network may be configured with multiple layers of other elements. For example, an inception module may be used, which juxtaposes convolution layers with different convolution filter sizes and integrates the resulting multiple feature maps to form a final feature map. Alternatively, a dense block with dense skip connections may be used.

[0067] The vertical and horizontal sizes of the feature map are also expanded or reduced within the network. Here, pooling, stride convolution, etc. can be used to reduce the feature map. Furthermore, deconvolution (or transposed convolution), pixel shuffle, interpolation, etc. can be used to expand the feature map. Although not shown in the figures in this example, stride convolution is used to reduce the feature map, and nearest neighbor interpolation is used to expand it.

[0068] Furthermore, a noise level (γ in the figure) is input to the residual block, and noise is removed from the noise patch according to the noise level. In this embodiment, a vector with noise level information embedded is generated using a separately prepared multilayer perceptron (MLP in the figure) based on the noise level. Then, based on this, the feature map obtained between the two convolution layers of the residual block is scaled (an affine transformation using a gain to be multiplied and a bias to be added to the feature map). However, this embodiment is not limited to this. A multilayer perceptron is a type of neural network. Furthermore, since the noise level of the noise image varies with the number of iterations in the dediffusion process, a different noise level γt is input to the residual block for each iteration.

[0069] Next, generation of an upscaled image using a diffusion model executed by the image estimation device 103 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart related to generation of an upscaled image using a diffusion model (generation of an output image using a neural network). Each step in Fig. 5 is mainly executed by the acquisition unit 103b, initialization unit 103c, noise removal unit 103d, random number generation unit 103e, or update unit 103f of the image estimation device 103.

[0070] First, in step S201, the acquisition unit 103b acquires a captured image (input). The captured image is a low-resolution image, as in learning. In this embodiment, the captured image is transmitted from the imaging device 102, but this is not limiting. Image information may also be acquired along with the captured image and used in steps described later. Also, as in learning, an interpolated image is generated by performing bicubic interpolation on the captured image in an interpolation unit (not shown), but the interpolation may also be performed by the acquisition unit 103b. Also, the captured image may be interpolated based on the image size acquired together with the captured image. Also, the interpolated image corresponds to a conditioned image x in a mathematical formula.

[0071] Next, in step S202, the initialization unit 103c generates (initializes) a noise image (output from the diffusion model) of the same size as the interpolated image. The noise image is generated using pseudo-random numbers (fixed random numbers, second random numbers) with a fixed random number seed. In this embodiment, the value of the random number seed for initialization is 0 (zero) or T (maximum number of iterations of the diffusion process), which corresponds to the number of iterations of the process, but is not limited to this. For example, a recommended random number seed may be used in the metadata of the image information acquired together with the captured image. The noise image is generated by using a pure noise image y T is equivalent to

[0072] Next, in step S203, the noise removal unit 103d concatenates the interpolated image and the noise image in the channel direction and inputs the concatenated image to the neural network to generate a noise-removed image (estimated result). A neural network with the same configuration as that shown in FIG. 1 is used to generate the noise-removed image. The weight information is pre-trained information transmitted from the learning device 101 and stored in the storage unit 103a. In this embodiment, a predetermined value (fixed value) according to the number of iterations is used as the noise level used to generate the noise-removed image, but this is not limited to this. For example, a schedule of noise levels determined for each iteration, acquired together with the captured image, may also be used. The noise-removed image corresponds to the target image y^ in the mathematical formula.

[0073] Next, in step S204, the update unit 103f generates an updated upscaled image (output) from the noise-removed image based on equation (8). The noise image generated by the random number generation unit 103e is used as the Gaussian noise used in this process. The noise image is generated using pseudo-random numbers with a fixed random number seed. In this embodiment, a value corresponding to the number of iterations of the process is used as the random number seed value, but this is not limiting. As in step S202, a recommended random number seed may be used in the metadata of the image information acquired together with the captured image. In this embodiment, the output may be updated using at least a portion of the random numbers fixed for each iteration based on the input to the diffusion model. In this embodiment, the updated upscaled image is generated by using the intermediate image y t-1 , the noise image is Gaussian noise ε t correspond to the following:

[0074] Next, in step S205, the update unit 103f determines whether the processing is complete. If the processing is complete, the updated upscaled image is output as an output image to, for example, at least one of the display device 104, the recording medium 105, and the output device 107. If the processing is not complete, the upscaled image is used as a new output and the processing is repeated from step S203. In this embodiment, the number of iterations of the processing is used as a criterion for determining whether the processing is complete, but the present invention is not limited to this.

[0075] In this embodiment, the output is updated iteratively using Gaussian noise (Gaussian noise generated based on a fixed random number for each iteration) with a mean and known variance that is an estimation result using a diffusion model based on the input and output, although the present embodiment is not limited to this.

[0076] In this embodiment, the input includes at least one of an image, an image size, image metadata, information about coordinates for extracting patches from the image, information about random numbers used in processing the image, and information about the noise level of the image, although the embodiment is not limited thereto.

[0077] In this embodiment, the method for generating an upscaled image using a diffusion model has been described in the image domain, but the present invention is not limited to this. It may also be performed in the feature map (latent) domain using a feature map obtained from a captured image or an interpolated image using a separately prepared neural network. In this case, the initialization unit 103c generates a noise image of the same size as the feature map obtained from the interpolated image, and the noise removal unit 103d can use a neural network trained in the latent domain.

[0078] In this embodiment, a method of estimating the upscaled image itself has been described as a method of generating an upscaled image using a diffusion model, but the present invention is not limited to this. For example, it is also possible to estimate only the residual component between the upscaled image and the interpolated image.

[0079] In addition, in this embodiment, the captured image to be used for generating an upscaled image using the diffusion model is assumed to be a color image, but this is not a limitation. For example, an upscaled image may be generated using a monochrome luminance image extracted from a captured color image.

[0080] With the above configuration, according to this embodiment, it is possible to obtain reproducible upscale image processing results using a diffusion model.

[0081] [Example 2] Next, an image processing system according to a second embodiment of the present invention will be described. In this embodiment, image processing is learned and executed to generate a deblurred image from a blurred image using a diffusion model. The image processing system of this embodiment differs from that of the first embodiment in that an imaging device acquires a captured image (blurred image) and performs deblurring image processing.

[0082] Fig. 6 is a block diagram of an image processing system 300 in this embodiment. Fig. 7 is an external view of the image processing system 300. The image processing system 300 includes a learning device 301 and an imaging device 302 connected via a network 303. Note that the learning device 301 and the imaging device 302 do not need to be constantly connected via the network 303.

[0083] The learning device 301 includes a storage unit 311, an acquisition unit 312, a generation unit 313, and a learning unit 314. These units are used to learn the weights of a neural network for performing image processing to generate a deblurred image from a blurred image using a diffusion model.

[0084] The imaging device 302 captures an image of a subject space, acquires a captured image (blurred image), and generates a blurred image from the captured image. Details of the image processing executed by the imaging device 302 will be described later. The imaging device 302 has an optical system 321 and an image sensor 322. The image estimation unit (image processing device) 323 has an acquisition unit 323a, an initialization unit 323b, a noise removal unit 323c, a random number generation unit 323d, and an update unit 323e.

[0085] The learning of the weights of the neural network executed by the learning device 301 is substantially the same as in Example 1, except that blurred patches are used instead of low-resolution patches as degraded images, and sharp patches with no (little) blur are used instead of high-resolution patches as corresponding correct images. Therefore, in this example, a description of the learning executed by the learning device 301 will be omitted, and only details regarding the image processing executed by the imaging device 302 will be described later.

[0086] The weight information of the neural network is learned in advance by the learning device 301 and stored in the memory unit 311. The imaging device 302 reads the weight information from the memory unit 311 via the network 303 and stores it in the memory unit 324. The image estimation unit 323 generates a deblurred image from the captured image using the trained neural network weight information stored in the memory unit 324 and the captured image and its image information acquired by the acquisition unit 323a. The generated deblurred image is stored in the recording medium 325a. When a user issues an instruction regarding display of the deblurred image via the input unit 326, the stored image is read and displayed on the display unit 325b. Note that the captured image and its image information stored in the recording medium 325a may be read to generate the deblurred image in the image estimation unit 323. The above series of controls are performed by the system controller 327.

[0087] Next, the generation of a deblurred image executed by the image estimation unit 323 in this embodiment will be described. Since the image processing procedure of this embodiment is substantially the same as that of the first embodiment shown in Fig. 5, the image processing of this embodiment will be described with reference to Fig. 5. Each step of the image processing is mainly executed by the acquisition unit 323a, initialization unit 323b, noise removal unit 323c, random number generation unit 323d, or update unit 323e of the image estimation unit 323.

[0088] First, in step S201, the acquisition unit 323a acquires a captured image (blurred image, input). In this embodiment, the captured image is acquired by the imaging device 302 and stored in the storage unit 324, but this is not limiting. Image information may also be acquired together with the captured image and used in steps described below. The captured image corresponds to a conditioned image x in a mathematical formula.

[0089] Next, in step S202, the initialization unit 323b generates a noise image (output) of the same size as the captured image. Note that the noise image is generated using a pseudo-random number with a fixed random number seed, as in the first embodiment. The random number seed (initialization value) is a value determined based on the number of iterations from a predetermined numerical sequence. In this embodiment, a value corresponding to the number of iterations of 0 (zero) or T (maximum number of iterations of the diffusion process) is extracted and used as the random number seed, but this is not limiting. For example, as in the first embodiment, a recommended random number seed may be used in the metadata of the image information acquired together with the captured image. Also, the noise image is generated by using a noise image y T In this embodiment, the random numbers are generated by determining a random number seed based on the number of iterations and determining the size of the random number based on the input to the diffusion model.

[0090] Next, in step S203, the noise removal unit 323c concatenates the captured image and the noise image in the channel direction and inputs them into the neural network to generate a noise-removed image (estimated result). To generate the noise-removed image, a neural network similar to the configuration shown in FIG. 1 is used, as in the first embodiment. In this embodiment, as in the first embodiment, a predetermined value corresponding to the number of iterations is used as the noise level used to generate the noise-removed image, but this is not limited to this. For example, a schedule of noise levels determined for each iteration acquired together with the captured image may also be used. Furthermore, the noise-removed image corresponds to the estimate y^ of the target image in the mathematical formula.

[0091] Next, in step S204, the update unit 323e generates an updated deblurred image (output) from the noise-removed image based on equation (8). As the Gaussian noise used at this time, a noise image generated by the random number generation unit 323d is used, as in the first embodiment. The noise image is generated using pseudo-random numbers with a fixed random number seed. In this embodiment, a value corresponding to the number of iterations of the process is extracted from a predetermined sequence and used as the random number seed value, but this is not limited to this. As in the first embodiment, if a recommended random number seed is present in the metadata of the image information acquired together with the captured image, this may be used. The updated deblurred image is generated by extracting the intermediate image y t-1 , the noise image is Gaussian noise ε t is equivalent to

[0092] Next, in step S205, the update unit 103f determines whether the processing is complete. If the processing is complete, the updated deblurred image is saved as an output image in the recording medium 325a. If the processing is not complete, the updated deblurred image is used as a new output and the processing is repeated from step S202. In this embodiment, the number of iterations of the processing is used as a criterion for determining whether the processing is complete, as in the first embodiment, but the present invention is not limited to this.

[0093] With the above configuration, according to this embodiment, it is possible to obtain reproducible blur-removed image processing results using a diffusion model.

[0094] [Example 3] Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment differs from the first and second embodiments in that it includes a processing device (computer) that transmits a captured image (Bayer image) to be processed by an image estimation device and receives an output image (deBayer image) processed using a diffusion model from the image estimation device. A Bayer image is raw image data acquired by an image sensor with a Bayer array in which the color filter has a checkered color pattern. In a Bayer image, each pixel has information on only one color (RGB), so to obtain a full-color image in which each pixel has information on the three colors (RGB), interpolation processing must be performed using adjacent pixels. This image processing is called deBayer (or demosaic).

[0095] 8 is a block diagram of an image processing system 400 in this embodiment. The image processing system 400 includes a learning device 401, an imaging device 402, an image estimation device (image processing device) 403, and a computer (control device) 404. The learning device 401 and the image estimation device 403 are, for example, servers. The computer 404 is, for example, a user terminal (a personal computer or a smartphone). The computer 404 is connected to the image estimation device 403 via a network 405. The image estimation device 403 is connected to the learning device 401 via a network 406. That is, the computer 404 and the image estimation device 403 are configured to be able to communicate with each other, and the image estimation device 403 and the learning device 401 are configured to be able to communicate with each other.

[0096] The configuration of the learning device 401 is similar to that of the learning device 101 in the first embodiment, and therefore a description thereof will be omitted. The learning of the neural network executed by the learning device 401 is substantially similar to that in the first embodiment, except that Bayer patches are used instead of low-resolution patches as degraded images, and DeBayer patches are used instead of high-resolution patches as corresponding correct images, and therefore a description thereof will be omitted. The configuration of the imaging device 402 is similar to that of the imaging device 102 in the first embodiment, and therefore a description thereof will be omitted.

[0097] The image estimation device 403 includes a storage unit 403a, an acquisition unit 403b, an initialization unit 403c, a noise removal unit 403d, a random number generation unit 403e, an update unit 403f, and a communication unit 403g. The storage unit 403a, the acquisition unit 403b, the initialization unit 403c, the noise removal unit 403d, the random number generation unit 403e, and the update unit 403f function as an image processing unit of the image estimation device 403. The components of the image processing unit are similar to the storage unit 103a, the acquisition unit 103b, the initialization unit 103c, the noise removal unit 103d, the random number generation unit 103e, and the update unit 103f of the image estimation device 103 of the first embodiment. The communication unit 403g functions as a receiving unit that receives a request transmitted from the computer 404 and as a transmitting unit that transmits an output image (deBayer image) generated by the image estimation device 403 to the computer 404.

[0098] The computer 404 has a communication unit 404a, a display unit 404b, an input unit 404c, a processing unit 404d, and a storage unit 404e. The communication unit 404a functions as a transmitter that transmits a request to the image estimation device 403 to cause the image estimation device 403 to execute processing on a captured image (Bayer image). The communication unit 404a also functions as a receiver that receives an output image (deBayer image) processed by the image estimation device 403.

[0099] The display unit 404b has a function of displaying various information. The information displayed by the display unit 404b includes, for example, a captured image (Bayer image) to be transmitted to the image estimation device 403 and an output image (debayer image) received from the image estimation device 403. The input unit 404c receives instructions from the user to start image processing, etc. The processing unit 404d has a function of performing image processing including white balance adjustment and gamma correction on the output image (debayer image) received from the image estimation device 403. The storage unit 404e stores the captured image acquired from the imaging device 402, the output image received from the image estimation device 403, etc.

[0100] Next, image processing in this embodiment will be described with reference to Fig. 9. The image processing in this embodiment is equivalent to the image processing described in embodiment 1 with reference to Fig. 5. Fig. 9 is a flowchart related to generation of an output image using a neural network in this embodiment. The image processing shown in Fig. 9 is started when an instruction to start image processing is given by a user via computer 404. First, the operation of computer 404 will be described.

[0101] First, in step S401, the computer 404 transmits a request for processing a captured image (Bayer image) to the image estimation device 403. Note that the method for transmitting the captured image to be processed to the image estimation device 403 is not important. For example, the captured image may be uploaded to the image estimation device 403 simultaneously with step S401, or may be uploaded to the image estimation device 403 before step S401. Furthermore, the captured image may be an image stored on a server different from the image estimation device 403. Furthermore, in step S401, the computer 404 may transmit an ID for authenticating a user, image information, and the like together with the request for processing the captured image.

[0102] Next, in step S 402 , the computer 404 receives the output image (debayered image) generated in the image estimation device 403 .

[0103] Next, the operation of the image estimation device 403 will be described.

[0104] First, in step S501, the image estimation device 403 receives a request for processing a captured image (Bayer image) transmitted from the computer 404. The image estimation device 403 determines that processing for the captured image has been instructed, and executes the processing from step S502 onwards.

[0105] Next, in step S502, the acquisition unit 403b acquires a captured image (Bayer image, input). As in the upscaling in the first embodiment, an interpolated image is generated by simply de-Bayering the captured image using bilinear interpolation in an interpolation unit (not shown), but the interpolation may also be performed by the acquisition unit 403b. Also, image information may be acquired together with the captured image and used in the steps described below. The interpolated image corresponds to a conditioned image x in a mathematical formula.

[0106] Next, in step S503, the initialization unit 403c generates a noise image (output) of the same size as the interpolated image. The noise image is generated using a pseudo-random number with a fixed random number seed. In this embodiment, the value of the random number seed for initialization is 0 (zero) or T (the maximum number of iterations of the diffusion process), which corresponds to the number of iterations of the process, but is not limited to this. For example, a recommended random number seed may be used in the metadata of the image information acquired together with the captured image. In this embodiment, a noise image larger than the interpolated image is first generated and then a portion of it is cropped to generate a noise image of the same size as the interpolated image, but is not limited to this. For example, a noise image smaller than the interpolated image may be first generated and then concatenated with a predetermined noise image to generate a noise image of the same size as the interpolated image. In addition, when a patch is extracted from the captured image and processed using a diffusion model, the predetermined noise image may be determined based on coordinate information for extracting the patch acquired together with the captured image. The noise image is generated based on a pure noise image y T is equivalent to

[0107] Next, in step S504, the noise removal unit 403d concatenates the interpolated image and the noise image in the channel direction and inputs the concatenated image to the neural network to generate a noise-removed image (estimated result). The noise-removed image is generated using a neural network with the same configuration as that shown in FIG. 1. The weight information is pre-trained information transmitted from the learning device 401 and stored in the storage unit 403a. In this embodiment, as in the first embodiment, a predetermined value according to the number of iterations is used as the noise level used to generate the noise-removed image, but this is not limited to this. For example, a schedule of noise levels determined for each iteration and acquired together with the captured image may also be used. The noise-removed image corresponds to the target image y^ in the mathematical formula.

[0108] Next, in step S505, the updating unit 403f generates an updated debayered image (output) from the noise-removed image based on equation (8). The noise image generated by the random number generating unit 403e is used as Gaussian noise. The noise image is generated using pseudo-random numbers with a fixed random number seed.

[0109] In this embodiment, a value corresponding to the number of iterations of the process is used as the random number seed value, but this is not limiting. As in step S202 of the first embodiment, if a recommended random number seed is available in the metadata of the image information acquired together with the captured image, this may be used. In this embodiment, a noise image larger than the updated debayered image is first generated, and a portion of this image is then trimmed to generate a noise image of the same size as the updated debayered image, but this is not limiting. For example, a noise image smaller than the updated debayered image may be first generated, and then concatenated with a predetermined noise image to generate a noise image of the same size as the updated debayered image. In addition, when a patch is extracted from the captured image and processed using a diffusion model, the predetermined noise image may be determined based on coordinate information for extracting the patch acquired together with the captured image. In addition, the updated debayered image is calculated based on the intermediate image y t-1 , the noise image is Gaussian noise ε tis equivalent to

[0110] Next, in step S506, the update unit 403f determines whether the processing is complete. If the processing is not complete, the updated debayered image is used as the new output, and the processing is repeated from step S503. In this embodiment, the number of iterations of the processing is used as a criterion for determining whether the processing is complete, but this is not limiting. If the processing is complete, the processing proceeds to step S507.

[0111] Next, in step S507, the image estimation device 403 transmits the processed output image (updated debayered image) to the computer 404.

[0112] With the above configuration, according to this embodiment, it is possible to obtain reproducible debayer image processing results using a diffusion model.

[0113] [Other Examples] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0114] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that are capable of obtaining reproducible image processing results using a diffusion model. The image processing device may be any device that has the image processing function of each embodiment, and may be realized in the form of an imaging device, a personal computer, or the like.

[0115] The disclosure of each embodiment includes the following methods and compositions: (Method 1) The first step is to obtain the inputs to the diffusion model; a second step of initializing an output from the diffusion model; a third step of iteratively updating the output using the diffusion model based on the input and the output; an image processing method characterized in that, in the third step, the output is updated using an estimation result obtained for each iteration using the diffusion model based on the input and the output and a random number fixed for each iteration. (Method 2) 2. The image processing method of claim 1, wherein the random number is generated using a random number seed determined based on the number of iterations. (Method 3) 3. The image processing method according to method 2, wherein the random number seed is the number of iterations. (Method 4) The image processing method according to Method 2, wherein the random number seed is a value determined from a predetermined sequence of numbers based on the number of iterations. (Method 5) determining a random number seed based on the number of iterations and determining a size of the random number based on the input to the diffusion model; The image processing method of method 1, wherein the random number is generated based on the random number seed and the size of the random number. (Method 6) The image processing method described in Method 1, characterized in that in the third step, the output is updated using at least a portion of the random numbers fixed for each iteration based on the input to the diffusion model. (Method 7) 7. The image processing method according to any one of Methods 1 to 6, wherein the random numbers are at least one of uniform random numbers, Gaussian random numbers, standard Gaussian random numbers, and pseudorandom numbers. (Method 8) In the third step, the output is iteratively updated using the estimation result and Gaussian noise with known variance; 8. The image processing method according to any one of methods 1 to 7, wherein the Gaussian noise is generated based on the random number that is fixed for each iteration. (Method 9) 9. An image processing method according to any one of methods 1 to 8, characterized in that the input includes at least one of the image, image size, image metadata, information about coordinates for extracting patches from the image, information about random numbers used in processing the image, and information about the noise level of the image. (Method 10) 9. The image processing method according to any one of methods 1 to 8, wherein in the second step, the output is initialized using a fixed second random number. (Configuration 1) A program causing a computer to execute the image processing method according to any one of Methods 1 to 9. (Configuration 2) an acquisition unit for acquiring inputs to the diffusion model; an initialization unit that initializes an output from the diffusion model; an update unit that iteratively updates the output using the diffusion model based on the input and the output, the update unit updates the output using an estimation result obtained for each iteration using the diffusion model based on the input and the output and a random number fixed for each iteration. (Configuration 3) 3. The image processing device according to configuration 2, further comprising a noise removal unit that acquires the estimation result for each iteration. (Configuration 4) 4. The image processing device according to configuration 2 or 3, further comprising a random number generation unit that generates the random numbers. (Configuration 5) An image processing system including the image processing device according to any one of configurations 2 to 4 and a control device capable of communicating with the image processing device, The control device a transmission unit that transmits a request to cause the image processing device to execute processing in response to the input; the image processing device has a receiving unit and an image processing unit, the receiving unit receives the request transmitted by the transmitting unit; The image processing system is characterized in that the image processing unit executes processing for the input in response to the request.

[0116] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0117] 103 Image estimation device (image processing device) 103b Acquisition Department 103c Initialization section 103f Update section

Claims

1. a first step of obtaining inputs to the diffusion model; a second step of initializing the output from the diffusion model; a third step of iteratively updating the output using the diffusion model based on the input and the output; In the third step, the output is updated by using an estimation result obtained for each iteration by using the diffusion model based on the input and the output, and a random number fixed for each iteration.

2. 2. The image processing method of claim 1, wherein the random number is generated using a random number seed determined based on the number of iterations.

3. 3. The image processing method of claim 2, wherein the random number seed is the number of iterations.

4. 3. The image processing method according to claim 2, wherein the random number seed is a value determined from a predetermined sequence of numbers based on the number of iterations.

5. determining a random number seed based on the number of iterations and determining a size of the random number based on the input to the diffusion model; The image processing method according to claim 1 , wherein the random number is generated based on the random number seed and a size of the random number.

6. 2. The image processing method according to claim 1, wherein in the third step, the output is updated by using at least a portion of the random numbers fixed for each iteration based on the input to the diffusion model.

7. 2. The image processing method according to claim 1, wherein the random numbers are at least one of uniform random numbers, normal random numbers, standard normal random numbers, and pseudo-random numbers.

8. In the third step, the output is iteratively updated using the estimation result and Gaussian noise with known variance; 2. The image processing method according to claim 1, wherein the Gaussian noise is generated based on the random number that is fixed for each iteration.

9. 2. The image processing method according to claim 1, wherein the input includes at least one of an image, an image size, image metadata, information about coordinates for extracting patches from the image, information about random numbers used in processing the image, and information about a noise level of the image.

10. 2. The image processing method according to claim 1, wherein in the second step, the output is initialized by using a fixed second random number.

11. A program causing a computer to execute the image processing method according to any one of claims 1 to 10.

12. an acquisition unit for acquiring inputs to the diffusion model; an initialization unit that initializes an output from the diffusion model; an update unit that iteratively updates the output by using the diffusion model based on the input and the output, the update unit updates the output by using an estimation result obtained for each iteration by using the diffusion model based on the input and the output, and a random number fixed for each iteration.

13. The image processing device according to claim 12 , further comprising a noise removal unit that obtains the estimation result for each iteration.

14. 13. The image processing device according to claim 12, further comprising a random number generating unit that generates the random numbers.

15. An image processing system including the image processing device according to any one of claims 12 to 14 and a control device capable of communicating with the image processing device, The control device a transmission unit that transmits a request to cause the image processing device to execute processing in response to the input; the image processing device has a receiving unit and an image processing unit, the receiving unit receives the request transmitted by the transmitting unit; The image processing system is characterized in that the image processing unit executes processing for the input in response to the request.

Images