Image processing method, image processing apparatus, image processing system, and program

Abstract

[Problem] To provide an image processing method capable of obtaining a reproducible image processing result by using a diffusion model. [Solution] The image processing method includes: a first step (S201) for acquiring an input to the diffusion model; a second step (S202) for initializing an output obtained by using the diffusion model based on the input; and a third step (S203 to S205) for iteratively updating the output by using an estimation result obtained by the diffusion model based on the input and the output for each iteration and a first random number fixed for each iteration.

Description

Technical Field

[0001] This invention relates to image processing methods, image processing apparatus, image processing systems, and programs. Background Technology

[0002] Patent document 1 discloses a method for sharpening images using a diffusion model.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: US Patent No. 11769228 Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] The method disclosed in Patent Document 1 cannot obtain reproducible image processing results using a diffusion model. As will be described later, this is because conventional image processing using a diffusion model contains randomness in its processing.

[0008] Accordingly, the present invention aims to provide an image processing method configured to use a diffusion model to obtain reproducible image processing results.

[0009] Problem-solving methods

[0010] One aspect of the present invention provides an image processing method comprising: a first step of obtaining input to a diffusion model; a second step of initializing an output obtained using the diffusion model based on the input; and a third step of iteratively updating the output using an estimation result obtained by the diffusion model for each iteration based on the input and the output, and a first random number fixed for each iteration.

[0011] Further objects and features of the present invention will become clear from the following description of embodiments.

[0012] The effects of the invention

[0013] The present invention can provide an image processing method configured to use a diffusion model to obtain reproducible image processing results. Attached Figure Description

[0014] Figure 1 This is a diagram illustrating the process of training a neural network according to the first embodiment.

[0015] Figure 2 This is a block diagram of an image processing system according to the first embodiment.

[0016] Figure 3This is an external view of the image processing system according to the first embodiment.

[0017] Figure 4 This is a flowchart relating to the training of a neural network according to the first embodiment.

[0018] Figure 5 This is a flowchart relating to the generation of an output image using a neural network according to the first and second embodiments.

[0019] Figure 6 This is a block diagram of an image processing system according to the second embodiment.

[0020] Figure 7 This is an external view of the image processing system according to the second embodiment.

[0021] Figure 8 This is a block diagram of an image processing system according to a third embodiment.

[0022] Figure 9 This is a flowchart relating to the generation of an output image using a neural network according to a third embodiment.

[0023] Figure 10 These are explanatory diagrams illustrating the gist of each embodiment. Detailed Implementation

[0024] A detailed description of embodiments of the present invention will now be given with reference to the accompanying drawings.

[0025] Before describing each embodiment in detail, the diffusion model and the essence of each embodiment will be described. In each embodiment, a diffusion model is used to perform reproducible image processing. The diffusion model is a type of generative artificial intelligence (AI) primarily used for image generation. By sampling from a distribution of target images conditioned on image or text input, a target image corresponding to the input image or text can be generated with high image quality. For example, Stable Diffusion uses a diffusion model to generate realistic images based on user-input text, just like those drawn by a human.

[0026] Broadly speaking, the diffusion model consists of two processes: forward diffusion and backward diffusion. In forward diffusion, Gaussian noise is progressively added to the target image, gradually removing details until a purely noisy image is obtained. In backward diffusion, the diffusion process is reversed by starting with a purely noisy image, conditioned on an image input or text input, and iteratively reconstructing the image. Backward diffusion can be trained using a neural network (a machine learning model) that removes noise by taking the noisy target image and the conditioned image (or a signal such as text) as input. Details will be described later.

[0027] The diffusion process is represented by the following equation (1): [Equation 1]

[0028] Here, y t-1 and y t Let represent the intermediate images at step t-1 and the next step t (t = 1 to T) of the diffusion process q represented by the Markov chain. N represents the standard normal distribution (Gaussian distribution), and... (0 < <1) indicates that y is added at step t. t-1 The variance of Gaussian noise.

[0029] By transforming (edge-shaping) equation (1), the intermediate image y is obtained given the target image y0. t The distribution of is represented by the following equation (2): [Equation 2]

[0030] here, This indicates adding to the target image y0 to obtain y t The variance of the Gaussian noise. In other words, equation (2) indicates that because at each step of the diffusion process, by feeding the intermediate image y t-1 Add Gaussian noise to obtain the intermediate image y at the next step. t Therefore, the intermediate image y at the t-th step can be obtained by adding Gaussian noise to the target image y0. t .

[0031] By further transforming equations (1) and (2), when given the target image y0 and the intermediate image y0, t The intermediate image y t-1 The posterior distribution is represented by the following equation (3): [Equation 3]

[0032] here, and These represent the target image y0 and the intermediate image y0, respectively. t In the case of the intermediate image y t-1 The mean and variance of the standard normal distribution of the posterior distribution. That is, equation (3) indicates that the intermediate image y from the t-th step can be obtained. t The mean obtained from the target image y0 Adding variance Gaussian noise is used to estimate the intermediate image y at the (t-1)th step.t-1 In other words, the diffusion process can be reversed. The reverse diffusion process described below uses the relationship of equation (3).

[0033] The reverse diffusion process is represented by the following equation (4): [Equation 4]

[0034] Here, the backdiffusion process p (t = T to 1) is defined as a reverse Markov chain traveling in the opposite direction to the diffusion process q. Based on the diffusion process q, the backdiffusion process extracts data from the purely noisy image y... t (Gaussian noise sampled from a standard normal distribution; the second expression in equation (4)) begins. With the variance of the Gaussian noise added at each step of the diffusion process set to small, each step of the backdiffusion process (the third expression in equation (4)) can be defined as a trainable isotropic Gaussian conditional distribution.

[0035] The mean of the Gaussian distribution at each step of the back diffusion process It is calculated based on the posterior distribution of equation (3), which requires the target image y0. However, in the case of a purely noisy image y t In the initial backdiffusion process, the target image y0 is generally unknown.

[0036] Accordingly, in the diffusion model, a neural network (machine learning model) is trained to remove noise from a noisy image obtained by adding Gaussian noise with various variances to the target image y0, and the trained neural network is used to extract noise from the intermediate image y0. t Estimating the target image for denoising .

[0037] Target image for denoising in back diffusion processing This is represented by the following equation (5): [Equation 5]

[0038] Here, f represents a parameter. A denoising neural network is used, and x represents the condition used for denoising. As an example of the condition used for denoising, Stable Diffusion uses text embedding vectors, and Patent Document 1 uses an interpolated image of a low-resolution image. In Patent Document 1, the intermediate image y is used... t The interpolated image is concatenated along the channel direction and then fed into a neural network to perform denoising conditioned on the interpolated image. The noise level (noise intensity) is also considered. The noise level is introduced into the denoising neural network to perform denoising corresponding to the noise level. In each embodiment, the variance of the Gaussian noise added during the diffusion process is used as the noise level, but this disclosure is not limited to this example. The method for inputting the noise level into the denoising neural network will be described later.

[0039] In the estimated target image Substituting into equation (3), the mean of the posterior distribution is given parametrically by the following equation (6): [Equation 6]

[0040] The variance of the Gaussian noise is determined by the variance of the Gaussian noise added during the diffusion process. Given. Using this, each step of the backdiffusion process is represented by the following equation (7): [Equation 7]

[0041] By repeating this process from t = T to 1, the image y can be transformed from pure noise. t Generate the target image y0 (more precisely, sample from the target image distribution conditioned on x). The diffusion model is also known as the denoising diffusion probability model (DDPM).

[0042] In conventional techniques such as Patent Document 1, the intermediate image y is obtained according to equation (7). t and estimated target image Calculate the intermediate image y t-1 However, standard normal random variables It is included in the backdiffusion process, and therefore the results change randomly for each processing even when the same image is processed by the diffusion model (or conditioned on the same image or text).

[0043] Accordingly, in each embodiment, instead of equation (7), the intermediate image y is obtained according to the following equation (8). t and estimated target image Obtain intermediate image y t-1 : [Equation 8]

[0044] Here, rand represents a pseudo-random number generator, and s tThis represents the random seed used in the t-th iteration. Pseudo-random numbers are pseudo-random values ​​obtained through deterministic computation, and the random seed is the initial state (initial random value) used to generate pseudo-random numbers. For example, in the linear congruential method, a type of pseudo-random number generation method, a sequence of random numbers is generated according to a recursive formula, and the random seed is used as the initial value for the recursive formula. Therefore, the same random numbers can be generated when the same random seed is used, and pseudo-random numbers are used for reproducible simulations on computers that can only perform deterministic computations.

[0045] In each embodiment, the value based on the number of iterations is used as a random seed for generating pseudo-random numbers. That is, pseudo-random numbers are generated by determining the random seed based on the number of iterations. Therefore, although different Gaussian noise (pseudo-random numbers) are used between iterations of the backdiffusion process, a fixed Gaussian noise (fixed random numbers) is used for each iteration. Thus, while maintaining the high image quality of the images generated by the diffusion model, reproducibility of the processing can be achieved. In each embodiment, the random numbers are not limited to pseudo-random numbers, and other random numbers such as uniform random numbers, normal random numbers, or standard normal random numbers can be used.

[0046] Next, refer to Figure 10 The overview (essentials) of each embodiment will be described using diffusion-based upsampling (super-resolution) as an example. Figure 10 These are explanatory diagrams illustrating the gist of each embodiment. Here, upsampling refers to image processing that generates a high-resolution image by magnifying the image while estimating high-frequency components that cannot be represented by a low-resolution image.

[0047] In each embodiment, firstly, a low-resolution image is acquired and a bicubic interpolated image is generated (the input corresponding to x in each equation). Next, the current intermediate image is initialized with Gaussian noise (the output corresponding to yT in each equation). This initialization uses a noise image as Gaussian noise generated from pseudo-random numbers with a fixed random seed. The size of the noise image is the same as the size of the interpolated image. Next, the interpolated image, along with the current intermediate image, is input into a neural network, and denoising is performed. The variance of the noise image, which is the noise level (corresponding to x in each equation), is... The corresponding information is also input into the neural network. The interpolated image is concatenated with the current intermediate image in the channel direction and is input into the neural network as conditional information for denoising. The variance of the noisy image can use a predetermined value for each iteration of the backdiffusion process. For example, the maximum and minimum values ​​of the variance of the noisy image can be determined and can use values ​​that are linearly determined from the number of iterations. The neural network is pre-trained to perform denoising as expressed by equation (5).

[0048] Next, from the denoised image (and in each equation) The mean of the posterior distribution, expressed by equation (5), is calculated based on the corresponding estimation results. Finally, the current intermediate image is updated according to equation (8) (with the y in each equation). t-1 (Corresponding output). Here, the iteration number t of the back diffusion process is used as the random seed s of the Gaussian noise in equation (8). t By repeating the above process a predetermined number of times, a high-resolution image can be generated with high reproducibility by simultaneously magnifying the image while estimating high-frequency components that cannot be represented by a low-resolution image.

[0049] Diffusion models are currently used in the field of image generation (text-to-image generation) to produce various images from text input, such as Stable Diffusion and Midjourney. That is, the focus has been on random image generation, and the reproducibility of image processing has been neglected. However, when applying diffusion models to conventional image processing such as deblurring, denoising, upsampling, and deBayer (de-mosaicing), the fact that the processing results change with each execution of image processing is problematic from a reproducibility perspective. In this type of image processing, each implementation can use a diffusion model to provide reproducible image processing results.

[0050] The image processing methods described above are merely examples, and each embodiment is not limited to this example. Details of other image processing methods will be described later in the embodiments.

[0051] [First Embodiment]

[0052] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, a neural network for performing image processing to generate a high-resolution image sampled from a low-resolution image using a diffusion model is trained, and image processing is performed.

[0053] Figure 2 This is a block diagram of the image processing system 100 according to this embodiment. Figure 3 This is an external view of the image processing system 100. The image processing system 100 includes a training device 101, an image picking device 102, an image estimation device (image processing device) 103, a display device 104, a storage medium 105, an input device 106, an output device 107, and a network 108.

[0054] The training device 101 includes a memory 101a, an acquisition unit 101b, a generator 101c, and a training unit 101d.

[0055] Image acquisition device 102 includes an optical system 102a and an image sensor 102b. Optical system 102a collects light entering image acquisition device 102 from the object space. Image sensor 102b receives an optical image of the object formed by optical system 102a and acquires a captured image (low-resolution image). Image sensor 102b is, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor. Information about imaging conditions of the captured image, such as pixel pitch of image sensor 102b, type of optical low-pass filter, and ISO sensitivity, can be acquired along with the image. Development conditions of the captured image, such as noise reduction intensity, sharpening intensity, and image compression rate, can also be acquired along with the image. Such information acquired along with the image can be sent, along with the image, to acquisition unit 103b of image estimation device 103, described later. A memory for storing the acquired image, a display unit for displaying the acquired image, a transmitter for sending the acquired image externally, and an output unit for storing the acquired image on an external storage medium are not shown. The control units that control each unit of the image acquisition device 102 are not shown.

[0056] The image estimation device 103 includes a memory 103a, an acquisition unit 103b, an initialization unit 103c, a denoising unit 103d, a random number generator 103e, and an updater 103f. The acquisition unit 103b acquires a low-resolution image (capture image, input). The initialization unit 103c initializes the current intermediate image with Gaussian noise (output). The denoising unit 103d estimates (acquires) a denoised image obtained by denoising the current intermediate image using a neural network conditioned on the low-resolution image (estimation result for each iteration). The random number generator 103e generates Gaussian noise (random numbers). The updater 103f performs image processing to generate a final upsampled high-resolution image (output) by repeatedly updating the current intermediate image using Gaussian noise and the estimation result.

[0057] The details of the initialization unit 103c's method for initializing the current intermediate image and the denoising unit 103d's method for denoising the current intermediate image using a neural network will be described later. The details of the random number generator 103e's method for generating Gaussian noise (random numbers) and the updater 103f's method for updating the current intermediate image will also be described later.

[0058] The low-resolution image can be an image captured by the image pickup device 102 or an image stored in the storage medium 105.

[0059] The weight information of the neural network used for denoising is read from memory 103a. The weights are pre-trained by training device 101. Image estimation device 103 reads the weight information from memory 101a via network 108 and stores the weight information in memory 103a. The stored weight information can be the weight values ​​themselves or it can be in an encoded format. Details of the method for training the weights of the neural network and the method for performing image processing using a diffusion model employing a neural network will be described later.

[0060] The upsampled high-resolution image is output to at least one of the display device 104, storage medium 105, and output device 107. The display device 104 is, for example, a liquid crystal display or a projector. The user can view the image being processed via the display device 104 and perform image editing operations via the input device 106. The storage medium 105 is, for example, a semiconductor memory, a hard disk, or a server on a network. The input device 106 is, for example, a keyboard or a mouse. The output device 107 is, for example, a printer.

[0061] Next, we will refer to Figure 1 and Figure 4 This describes a method for training the weights of a neural network (machine learning model) performed by the training device 101 according to this embodiment (a method for generating a trained model). Figure 1 This is a diagram illustrating the process of training the weights of a neural network. Figure 4 This is a flowchart of the training process related to the weights. Figure 4 The various steps are mainly executed by the acquisition unit 101b, the generator 101c, or the training unit 101d.

[0062] First, in step S101, as Figure 1 As shown, the acquisition unit 101b obtains a low-resolution patch (degraded image) 201 and a corresponding high-resolution patch (ground truth image) 200. In this embodiment, a patch is a small image with a predetermined number of pixels. For example, a low-resolution patch may have 128×128×3 pixels (corresponding to the number of pixels in the vertical, horizontal, and channel directions), and the corresponding high-resolution patch may have 256×256×3 pixels. In this case, since the vertical and horizontal dimensions are doubled, the upsampling factor is doubled (2×) (corresponding to a fourfold increase in the number of pixels).

[0063] The upsampling factor is not limited to two times (2×), and any factor can be used as long as a low-resolution patch and a corresponding high-resolution patch can be obtained. The low-resolution patch and the corresponding high-resolution patch can be obtained by capturing the same object using optical systems with different focal lengths and extracting corresponding portions from the two acquired images. Alternatively, a corresponding low-resolution patch can be generated by downsampling the high-resolution patch. Alternatively, the corresponding low-resolution patch acquired by the image picking device 102 and the corresponding high-resolution patch less affected by the optical system 102a (aberrations and diffraction) can be generated through numerical calculation. This embodiment generates the low-resolution patch and the corresponding high-resolution patch through numerical calculation, but this disclosure is not limited to this example.

[0064] Next, in step S102, generator 101c generates interpolated patch 202 by interpolating and enlarging low-resolution patch 201, so that it has the same number of pixels as the corresponding high-resolution patch 200. This embodiment uses bicubic interpolation as the interpolation method, but the interpolation method is not limited to this example, and other interpolation methods such as nearest neighbor interpolation or bilinear interpolation can be used.

[0065] Next, in step S103, generator 101c generates a noisy patch (noisy image) 203 by adding noise to the high-resolution patch 200.

[0066] Next, in step S104, training unit 101d generates denoised patch 204 by denoising noisy patch 203 using a neural network. Denoising patch 204 is generated using interpolated patch 202, noisy patch 203, and information about the intensity (noise level) of noise added to noisy patch 203. Denoising patch 204 is an estimate of high-resolution patch 200, and ideally, they are consistent with each other.

[0067] In this embodiment, the variance of the added Gaussian noise is used as the noise level, but this disclosure is not limited to this example. In this embodiment, denoising is performed corresponding to the intensity of the noise added to noise patch 203. The method of inputting the noise level (variance of Gaussian noise) into the neural network will be described later. In this embodiment, interpolation patch 202 and noise patch 203 are cascaded in the channel direction and input into the neural network, but this disclosure is not limited to this example.

[0068] Next, in step S105, training unit 101d updates the weights of the neural network based on the error between the high-resolution patch 200 and the denoised patch 204, which serves as an estimate of it. Here, the weights include the biases and filter coefficients of each layer. In this embodiment, backpropagation is used to update the weights, but this disclosure is not limited to this example. In mini-batch training, the error between the high-resolution patch 200 and the corresponding denoised patch 204 is calculated, and the weights are updated. The loss function can be, for example, an L2 norm or an L1 norm. The weight update method (training method) is not limited to mini-batch training, but can instead be batch training or online training.

[0069] Next, in step S106, the training unit 101d determines whether the training of the weights has been completed. Completion can be determined based on whether the number of training iterations (weight updates) has reached a predetermined value or whether the change in weights during the update has become less than a predetermined value. If training is not completed, the process returns to step S101, and new low-resolution patches 201 and corresponding high-resolution patches 200 are acquired. On the other hand, if training is completed, the training device 101 terminates the training and stores the weight information in the memory 101a.

[0070] This embodiment has discussed methods for training the weights of a neural network in the image domain, but this disclosure is not limited to this example. For example, training can be performed in the feature map (latent) domain using a noise patch 203 obtained by adding noise to a feature map generated by a separately prepared neural network. In this case, training can be performed using a low-resolution patch 201 or an interpolated patch 202.

[0071] This embodiment uses Figure 1 The neural network configuration shown is not limited to this example.

[0072] exist Figure 1 In this diagram, CN represents a convolutional layer. CN computes the convolution between the input and the filter, adds a bias to it, and performs a non-linear transformation on the result using an activation function. The initial values ​​for each component of the filter and the bias can be arbitrary, and in this embodiment, they are determined using random numbers. Examples of activation functions include the Rectified Linear Unit (ReLU) function and the sigmoid function. The multidimensional array output from each layer except the last layer is called a feature map. Generally, a feature map is a four-dimensional array with batch, vertical-horizontal, and channel dimensions. Skip connections 205 combine the feature maps output from discontinuous layers. Feature maps can be combined by element-wise addition or by cascading along the channel direction. This embodiment combines feature maps by element-wise addition.

[0073] exist Figure 1The elements (blocks or modules) within the dashed boxes represent residual blocks. Networks with multiple layers of residual blocks are called residual networks and are widely used in image processing employing machine learning models. However, this disclosure is not limited to this example, and networks can be constructed by stacking other types of elements. For example, inception modules can be used, where convolutional layers with mutually different filter sizes are arranged in parallel and the resulting multiple feature maps are integrated into a final feature map. Alternatively, dense blocks with skip connections and dense connections can be used.

[0074] The vertical and horizontal dimensions of the feature map are scaled up and down within the network. For scaling down the feature map, pooling or strided convolution can be used. For scaling up the feature map, deconvolution (or transposed convolution), pixel rearrangement, or interpolation can be used. Although not illustrated in this embodiment, strided convolution is used to scale down the feature map, and nearest neighbor interpolation is used to scale it up.

[0075] Noise level (in the figure) A noise level is input to each residual block, causing denoising to be performed on the noisy patch corresponding to that noise level. In this embodiment, a vector embedded with information about the noise level is generated using a separately prepared multilayer perceptron (MLP in the figure) based on the noise level. Based on the generated vector, the feature map obtained between two convolutional layers of the residual block is scaled (by using an affine transformation multiplied by the gain of the feature map and the bias added to it). However, this disclosure is not limited to this example. A multilayer perceptron is a type of neural network. Since the noise level of the noisy image is different for each iteration (number) in the backdiffusion process, a different noise level is applied in each iteration. Input into the residual block.

[0076] Next, we will refer to Figure 5 The image is described as being generated using a diffusion model by the image estimation apparatus 103 according to this embodiment. Figure 5 This is a flowchart about using a diffusion model to generate upsampled images (using a neural network to generate the output image). Figure 5 Each step in the process is mainly performed by the acquisition unit 103b, initialization unit 103c, denoising unit 103d, random number generator 103e, or updater 103f in the image estimation device 103.

[0077] First, in step S201, the acquisition unit 103b acquires the captured image (input). As in training, the captured image is a low-resolution image. In this embodiment, the captured image is sent from the image picking device 102, but this disclosure is not limited to this example. Image information can also be acquired together with the captured image and used in the steps described later. As in training, an interpolated image obtained by bicubic interpolation of the captured image is generated by the interpolation unit (not shown), but alternatively, the interpolation can be performed by the acquisition unit 103b. The captured image can also be interpolated based on image size information acquired together with the captured image. The interpolated image corresponds to the conditional image x in the equation.

[0078] Next, in step S202, initialization unit 103c generates (initializes) a noise image (from the output of the diffusion model) having the same size as the interpolated image. This noise image is generated using pseudo-random numbers (fixed random numbers, second random numbers) with a fixed random seed. In this embodiment, the value of 0 (zero) corresponding to the number of processing iterations or T (the maximum number of iterations for the diffusion process) is used as the initial random seed value, but this disclosure is not limited to this example. For example, a recommended random seed included in the metadata of the image information acquired along with the captured image can be used. The noise image and the pure noise image y in the equation... t correspond.

[0079] Next, in step S203, the denoising unit 103d cascades the interpolated image and the noisy image in the channel direction, inputs them into the neural network, and generates a denoised image (estimation result). This has the characteristics of... Figure 1 A neural network with the same configuration as shown is used to generate a denoised image. Weight information is trained weight information sent from training device 101 and stored in memory 103a. In this embodiment, a predetermined value (fixed value) corresponding to the number of iterations is used as the noise level when generating a denoised image, but this disclosure is not limited to this example. For example, a schedule of noise levels defined for each iteration and acquired together with the captured images can be used. The denoised image and the target image in the equation... correspond.

[0080] Next, in step S204, updater 103f generates an updated upsampled image (output) from the denoised image based on equation (8). The noisy image generated by random number generator 103e is used as Gaussian noise at this time. The noisy image is generated using pseudo-random numbers with a fixed random seed. In this embodiment, a value corresponding to the number of processing iterations is used as the random seed value, but this disclosure is not limited to this example. As in step S202, a recommended random seed included in the metadata of the image information acquired together with the captured image can be used. In this embodiment, the output can be updated based on the input to the diffusion model using at least a portion of random numbers fixed for each iteration. In this embodiment, the updated upsampled image and the intermediate image y in the equation are compared. t-1 Correspondingly, and the noisy image is associated with Gaussian noise. correspond.

[0081] Next, in step S205, the updater 103f determines whether the processing has been completed. If the processing has been completed, the updated upsampled image is output as an output image to at least one of the display device 104, storage medium 105, and output device 107. If the processing has not been completed, the updated upsampled image is used as the new output, and the processing starting from step S203 is repeated. In this embodiment, the number of processing iterations is used as a criterion for determining the completion of processing, but this disclosure is not limited to this example.

[0082] In this embodiment, the update of the output is performed iteratively using the mean obtained from the estimation results of the input and output obtained from the diffusion model, and Gaussian noise with known variance (based on Gaussian noise generated for a fixed number of random numbers for each iteration). However, this disclosure is not limited to this example.

[0083] In this embodiment, the input includes at least one of the following: an image, image size, image metadata, information about the coordinates used when extracting tiles from the image, information about the random numbers used when processing the image, and information about the noise level of the image. However, this disclosure is not limited to this example.

[0084] This embodiment has discussed methods in the image domain as a way to generate upsampled images using a diffusion model, but this disclosure is not limited to this example. Alternatively, processing can be performed in the feature map (latent) domain using feature maps obtained from the captured or interpolated images by a separately prepared neural network. In this case, initialization unit 103c generates a noisy image with the same size as the feature map obtained from the interpolated image, and denoising unit 103d can use a neural network trained in the latent domain.

[0085] This embodiment has discussed methods for estimating the upsampled image itself as a method for generating an upsampled image using a diffusion model, but this disclosure is not limited to this example. For example, only the residual components between the upsampled image and the interpolated image can be estimated.

[0086] This embodiment assumes that the captured image undergoing the generation of an upsampled image using a diffusion model is a color image, but this disclosure is not limited to this example. For example, an upsampled image can be generated as a target for extracting a monochromatic brightness image from a captured color image.

[0087] Due to the above configuration, this embodiment can use a diffusion model to obtain reproducible upsampled image processing results.

[0088] [Second Embodiment]

[0089] Next, an image processing system according to a second embodiment of the present invention will be described. In this embodiment, a neural network for performing image processing to generate a deblurred image from a blurred image using a diffusion model is trained, and image processing is performed. The image processing system according to this embodiment differs from that of the first embodiment in that the image picking device acquires a captured image (blurred image) and performs deblurred image processing.

[0090] Figure 6 This is a block diagram of the image processing system 300 according to this embodiment. Figure 7 This is an external view of the image processing system 300. The image processing system 300 includes a training device 301 and an image acquisition device 302 connected via a network 303. The training device 301 and the image acquisition device 302 can be continuously connected without being connected via the network 303.

[0091] The training device 301 includes a memory 311, an acquisition unit 312, a generator 313, and a training unit 314. Using these components, training is performed on the weights of a neural network used to perform image processing to generate deblurred images from blurred images using a diffusion model.

[0092] Image acquisition device 302 captures the object space to obtain a captured image (blurred image) and generates a deblurred image from the captured image. Details of the image processing performed by image acquisition device 302 will be described later. Image acquisition device 302 includes optical system 321 and image sensor 322. Image estimator (image processing device) 323 includes acquisition unit 323a, initialization unit 323b, denoising unit 323c, random number generator 323d, and updater 323e.

[0093] Except for using blurred patches as degraded images instead of low-resolution patches, and using clear patches with no blur or very little blur instead of high-resolution patches as corresponding ground truth images, the training of the neural network weights performed by the training device 301 is largely similar to the training in the first embodiment. Therefore, this embodiment will omit the description of the training performed by the training device 301 and will only discuss the details of the image processing performed by the image picking device 302.

[0094] Information regarding the weights of the neural network is pre-trained by the training device 301 and stored in the memory 311. The image acquisition device 302 reads the weight information from the memory 311 via the network 303 and stores it in the memory 324. The image estimator 323 uses the trained weight information of the neural network stored in the memory 324, along with the captured image and its image information acquired by the acquisition unit 323a, to generate a deblurred image from the captured image. The generated deblurred image is stored in the storage medium 325a. When the user issues a command regarding the display of the deblurred image via the input unit 326, the stored image is read out and displayed on the display unit 325b. The captured image and its image information stored in the storage medium 325a can be read out, and the image estimator 323 can generate a deblurred image. This control sequence is executed by the system controller 327.

[0095] Next, the generation of the deblurred image performed by the image estimator 323 according to this embodiment will be described. Since the image processing procedure according to this embodiment differs from that of the first embodiment... Figure 5 The image processing process is roughly similar, therefore, reference will be used. Figure 5 The image processing according to this embodiment is described. Each step in the image processing is mainly performed by the acquisition unit 323a, initialization unit 323b, denoising unit 323c, random number generator 323d, or updater 323e of the image estimator 323.

[0096] First, in step S201, the acquisition unit 323a acquires a captured image (blurred image, input). In this embodiment, the captured image is acquired by the image picking device 302 and stored in the memory 324, but this disclosure is not limited to this example. Image information can also be acquired together with the captured image and used in the steps described later. The captured image corresponds to the conditional image x in the equation.

[0097] Next, in step S202, initialization unit 323b generates a noise image (output) with the same size as the captured image. As in the first embodiment, the noise image is generated using pseudo-random numbers with a fixed random seed. The random seed (initialization value) is a value determined from a predetermined numerical sequence based on the number of iterations. In this embodiment, a value corresponding to an iteration number of 0 (zero) or T (the maximum number of iterations for diffusion processing) is extracted and used as the random seed, but this disclosure is not limited to this example. For example, as in the first embodiment, a recommended random seed included in the metadata of the image information acquired along with the captured image can be used. The noise image and the noise image y in the equation... t Correspondingly, in this embodiment, random numbers are generated by determining the random seed based on the number of iterations and determining the size of the random number based on the input to the diffusion model. For example, when the captured image size is 256×256×3 pixels, 0 (zero) can be used as the random seed to generate a noisy image (random number) with a size of 256×256×3 pixels.

[0098] Next, in step S203, the denoising unit 323c cascades the captured image and the noisy image in the channel direction, inputs the cascaded image into the neural network, and generates a denoised image (estimation result). As in the first embodiment, it has the same characteristics as... Figure 1 A neural network with a configuration similar to that shown is used to generate a denoised image. Furthermore, in this embodiment, as in the first embodiment, a predetermined value based on the number of iterations is used as the noise level when generating the denoised image; however, this disclosure is not limited to this example. For example, a schedule of noise levels determined for each iteration and acquired along with the captured image can be used. The denoised image is estimated in relation to the target image in the equation. correspond.

[0099] Next, in step S204, updater 323e generates an updated deblurred image (output) from the denoised image based on equation (8). As in the first embodiment, the noise image generated by random number generator 323d is used as Gaussian noise at this time. The noise image is generated using pseudo-random numbers with a fixed random seed. In this embodiment, a value corresponding to the number of processing iterations is extracted from a predetermined numerical sequence and used as the random seed value, but this disclosure is not limited to this example. As in the first embodiment, if the metadata of the image information acquired together with the captured image includes a recommended random seed, the recommended random seed can be used. The updated deblurred image and the intermediate image y in the equation t-1 Correspondingly, and the noisy image is associated with Gaussian noise. correspond.

[0100] Next, in step S205, the updater 103f determines whether the processing has been completed. When the processing is complete, the updated deblurred image is stored as an output image in the storage medium 325a. If the processing is not yet complete, the updated deblurred image is used as the new output, and the processing starting from step S202 is repeated. In this embodiment, as in the first embodiment, the number of processing iterations is used as the criterion for determining whether the processing has been completed, but this disclosure is not limited to this example.

[0101] Due to the above configuration, this embodiment can use a diffusion model to obtain reproducible deblurred image processing results.

[0102] [Third Embodiment]

[0103] Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system according to this embodiment differs from the image processing systems of the first and second embodiments in that it includes a processing device (computer) configured to send a captured image (Bayer image) as the target of image processing to an image estimation device and receive an output image (de-Bayer image) processed by a diffusion model from the image estimation device. The Bayer image is raw image data acquired by an image sensor having a Bayer arrangement in which color filters are arranged in a checkerboard pattern. Since each pixel of the Bayer image contains information about only one of the R, G, and B colors, interpolation processing using neighboring pixels is required to obtain a full-color image where each pixel contains information about all three RGB colors. This image processing is called de-Bayer (or de-mosaicing).

[0104] Figure 8 This is a block diagram of an image processing system 400 according to this embodiment. The image processing system 400 includes a training device 401, an image acquisition device 402, an image estimation device (image processing device) 403, and a computer (control device) 404. The training device 401 and the image estimation device 403 are, for example, servers. The computer 404 is, for example, a user terminal such as a personal computer or a smartphone. The computer 404 is connected to the image estimation device 403 via a network 405. The image estimation device 403 is connected to the training device 401 via a network 406. That is, the computer 404 and the image estimation device 403 are configured to communicate with each other, and the image estimation device 403 and the training device 401 are configured to communicate with each other.

[0105] Since the configuration of training device 401 is similar to that of the training device according to the first embodiment, its description will be omitted. The training of the neural network performed by training device 401 is substantially similar to that of the first embodiment, except that Bayer patches are used instead of low-resolution patches as the degraded image and de-Bayer patches are used instead of high-resolution patches as the corresponding ground truth image. Therefore, its description will be omitted. Since the configuration of image picking device 402 is similar to that of image picking device 102 according to the first embodiment, its description will also be omitted.

[0106] The image estimation device 403 includes a memory 403a, an acquisition unit 403b, an initialization unit 403c, a denoising unit 403d, a random number generator 403e, an updater 403f, and a communication unit 403g. The memory 403a, acquisition unit 403b, initialization unit 403c, denoising unit 403d, random number generator 403e, and updater 403f serve as the image processing unit of the image estimation device 403. These components of the image processing unit are similar to the memory 103a, acquisition unit 103b, initialization unit 103c, denoising unit 103d, random number generator 103e, and updater 103f of the image estimation device 103 of the first embodiment. The communication unit 403g functions as a receiver for receiving requests sent from the computer 404 and as a transmitter for sending the output image (de-Bayer image) generated by the image estimation device 403 to the computer 404.

[0107] Computer 404 includes a communication unit 404a, a display unit 404b, an input unit 404c, a processing unit 404d, and a memory 404e. Communication unit 404a functions as a transmitter to the image estimation device 403, sending a request to perform processing on the captured image (Bayer image). Communication unit 404a also functions as a receiver to receive the output image (de-Bayer image) processed by the image estimation device 403.

[0108] Display unit 404b has the function of displaying various types of information. The information displayed by display unit 404b includes, for example, the captured image (Bayer image) to be sent to image estimation device 403 and the output image (de-Bayer image) received from image estimation device 403. Input unit 404c receives user commands such as commands to start image processing. Processing unit 404d has the function of performing image processing, including white balance adjustment and gamma correction, on the output image (de-Bayer image) received from image estimation device 403. Memory 404e stores the captured image acquired from image pickup device 402, the output image received from image estimation device 403, etc.

[0109] Next, we will refer to Figure 9Image processing according to this embodiment is described. The image processing according to this embodiment is equivalent to that described in the first embodiment. Figure 5 The image processing described. Figure 9 This is a flowchart relating to the generation of an output image using a neural network according to this embodiment. In response to a user instruction to start image processing via computer 404, the process begins... Figure 9 The image processing shown is described below. First, the operation of computer 404 will be described.

[0110] In step S401, computer 404 sends a request to image estimation device 403 to process the captured image (Bayer image). The captured image to be processed can be sent to image estimation device 403 using any method. For example, the captured image can be uploaded to image estimation device 403 simultaneously with step S401, or it can be uploaded before step S401. The captured image can also be stored on a different server than image estimation device 403. In step S401, computer 404 may also send user authentication ID, image information, etc., along with the processing request.

[0111] Next, in step S402, computer 404 receives the output image (de-Bayer image) generated by image estimation device 403.

[0112] Next, the operation of the image estimation device 403 will be described.

[0113] In step S501, the image estimation device 403 receives a request to process a captured image (Bayer image) sent from the computer 404. The image estimation device 403 determines that processing of the captured image has been instructed and performs the processing in step S502 and subsequent steps.

[0114] Next, in step S502, the acquisition unit 403b acquires the captured image (Bayer image, input). As in the upsampling process of the first embodiment, an interpolated image obtained by simply removing the Bayer signal from the captured image using bilinear interpolation is generated by the interpolation unit (not shown), but alternatively, the interpolation can be performed by the acquisition unit 403b. Image information can also be acquired together with the captured image and used in the steps described later. The interpolated image corresponds to the conditional image x in the equation.

[0115] Next, in step S503, the initialization unit 403c generates a noise image (output) with the same size as the interpolated image. The noise image is generated using pseudo-random numbers with a fixed random seed. In this embodiment, a value of 0 (zero) or T (the maximum number of iterations for the diffusion process) corresponding to the number of processing iterations is used as the initial random seed, but this disclosure is not limited to this example. For example, a recommended random seed included in the metadata of the image information acquired along with the captured image can be used. In this embodiment, a noise image larger than the interpolated image is first generated, and a portion of it is cropped to generate a noise image with the same size as the interpolated image. For example, if the interpolated image has a size of 256×256×3 pixels, a noise image with a size of 1024×1024×3 pixels can be generated first using a random seed of 0 (zero), and then a portion of it can be cropped to generate a noise image with a size of 256×256×3 pixels. This invention is not limited to this example. A noise image smaller than the interpolated image can be generated first and then concatenated with a predetermined noise image to generate a noise image of the same size as the interpolated image. For example, if the interpolated image is 256×256×3 pixels in size, a noise image of 256×128×3 pixels can be generated first using a random seed of 0 (zero), and then concatenated horizontally with a predetermined noise image of 256×128×3 pixels to generate a noise image of 256×256×3 pixels. When extracting tiles from a captured image and processing the tiles using a diffusion model, the predetermined noise image can be determined based on coordinate information used to extract the tiles, which is acquired along with the captured image. For example, when extracting a 256×256×3 pixel patch from the coordinates (x, y) of a captured image of size 4000×6000×3 pixels and processing the patch using a diffusion model, a noisy image of size 4000×6000×3 pixels can be generated first using a fixed random seed. A patch of size 256×128×3 pixels can then be extracted from the coordinates (x, y) of the noisy image and used as the predetermined noisy image. The noisy image is compared to the pure noise image y in the equation. t correspond.

[0116] Next, in step S504, the denoising unit 403d cascades the interpolated image and the noise image in the channel direction, inputs the cascaded image into the neural network, and generates a denoised image (estimation result). This has the characteristics of... Figure 1A neural network with a configuration similar to that shown is used to generate a denoised image. Weight information has been trained, sent by training device 401, and stored in memory 403a. As in the first embodiment, a predetermined value corresponding to the number of iterations is used as the noise level employed when generating the denoised image; however, this disclosure is not limited to this example. For example, a schedule of noise levels determined for each iteration and acquired along with the captured image can be used. The denoised image is compared with the target image estimation in the equation. correspond.

[0117] Next, in step S505, updater 403f generates an updated de-Bayer image (output) from the denoised image based on equation (8). The noisy image generated by random number generator 403e is used as Gaussian noise for this purpose. The noisy image is generated using pseudo-random numbers with a fixed random seed.

[0118] In this embodiment, a value corresponding to the number of processing iterations is used as a random seed, but this disclosure is not limited to this example. As in step S202 of the first embodiment, it can be used as long as it is a recommended random seed included in the metadata of the image information acquired together with the captured image. In this embodiment, a noise image larger than the updated de-Bayer image is first generated and a portion of it is cropped to generate a noise image with the same size as the updated de-Bayer image, but this disclosure is not limited to this example. For example, as described in step S503, a noise image smaller than the updated de-Bayer image can be first generated and concatenated with a predetermined noise image to generate a noise image with the same size as the updated de-Bayer image. In the case of extracting a tile from the captured image and processing the tile by the diffusion model, the predetermined noise image can also be determined based on the coordinate information for extracting the tile acquired together with the captured image. The updated de-Bayer image and the intermediate image y in the equation t-1 Correspondingly, and the noisy image is associated with Gaussian noise. correspond.

[0119] Next, in step S506, updater 403f determines whether the processing is complete. If the processing is not yet complete, the updated de-Bayer image is used as the new output, and the process starting from step S503 is repeated. In this embodiment, the number of processing iterations is used as the criterion for determining the completion of processing, but this disclosure is not limited to this example. If the processing is complete, the process proceeds to step S507.

[0120] Next, in step S507, the image estimation device 403 sends the completed output image (updated debayer image) to the computer 404.

[0121] Due to the above configuration, this embodiment can use a diffusion model to obtain reproducible deBayer image processing results.

[0122] (Other embodiments)

[0123] The present invention can also be implemented by a process in which a program implementing one or more functions of the above embodiments is supplied to a system or device via a network or storage medium, and this process can be implemented by one or more processors in a computer of the system or device configured to read and execute the program. The present invention can also be implemented by circuitry (e.g., an ASIC) implementing one or more functions.

[0124] Each embodiment can provide an image processing method, an image processing apparatus, an image processing system, and a program, wherein each can use a diffusion model to obtain reproducible image processing results. The image processing apparatus can be any apparatus having the image processing functions of each embodiment, and can be implemented in the form of an image picking device, a personal computer, etc.

[0125] While preferred embodiments of the invention have been described above, the invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the spirit of the invention.

Claims

1. An image processing method, comprising: The first step in obtaining the input to the diffusion model; The second step is to initialize the output obtained using the diffusion model based on the input; as well as The third step involves iteratively updating the output using an estimate obtained by the diffusion model for each iteration based on the input and the output, and a first random number fixed for each iteration.

2. The image processing method according to claim 1, wherein the first random number is generated using a random seed determined based on the number of iterations.

3. The image processing method according to claim 2, wherein the random seed is the number of iterations.

4. The image processing method according to claim 2, wherein the random seed is a value determined from a predetermined numerical sequence based on the number of iterations.

5. The image processing method according to any one of claims 1 to 4, further comprising: The random seed is determined based on the number of iterations. as well as The size of the first random number is determined based on the input. The first random number is generated based on the random seed and the size of the first random number.

6. The image processing method according to any one of claims 1 to 5, wherein the third step updates the output based on the input using at least a portion of the first random number fixed for each iteration.

7. The image processing method according to any one of claims 1 to 6, wherein the first random number is at least one of a uniform random number, a normal random number, a standard normal random number, or a pseudo-random number.

8. The image processing method according to any one of claims 1 to 7, wherein the third step iteratively updates the output using the estimation result and Gaussian noise with known variance, and The Gaussian noise is generated based on the first random number fixed for each iteration.

9. The image processing method according to any one of claims 1 to 8, wherein the input includes at least one of an image, the size of the image, metadata of the image, information about the coordinates of tiles for extracting from the image, information about random numbers for processing the image, or information about the noise level of the image.

10. The image processing method according to any one of claims 1 to 9, wherein the second step initializes the output using a second random number.

11. The image processing method according to claim 10, wherein the second random number is a fixed random number.

12. A non-transitory computer-readable storage medium storing a program that causes a computer to perform the image processing method according to any one of claims 1 to 11.

13. An image processing apparatus, comprising: An acquisition unit, configured to acquire input to the diffusion model; An initialization unit is configured to initialize the output obtained using the diffusion model based on the input; as well as An updater configured to iteratively update the output using an estimate obtained by the diffusion model for each iteration based on the input and the output, and a first random number fixed for each iteration.

14. The image processing apparatus of claim 13, further comprising a denoising unit configured to obtain the estimation result for each iteration.

15. The image processing apparatus according to claim 12 or 14, further comprising a random number generator configured to generate the first random number.

16. An image processing system, comprising: The image processing apparatus according to any one of claims 13 to 15; as well as A control device, communicatively connected to the image processing device. The control device includes a transmitter configured to send a request to cause the image processing device to perform processing on the input. The image processing device includes a receiver and an image processing unit. The receiver is configured to receive the request sent by the transmitter, and The image processing unit is configured to perform processing on the input according to the request.

17. An image processing method, comprising the following steps: Obtain the input image; as well as An estimated image is generated by feeding the input image into a trained diffusion model. The trained diffusion model is obtained through training including the following steps: Obtain the input to the diffusion model; The output obtained using the diffusion model is initialized based on the input tiles; as well as The output is iteratively updated using an estimate obtained by the diffusion model for each iteration based on the input and the output, and a first random number fixed for each iteration.

Images