Image processing method, image processing device, image processing system, and program

Abstract

[Problem] To provide an image processing method capable of appropriately performing patch segmentation processing using a diffusion model. [Solution] This image processing method comprising: a first step (S201) for acquiring an input patch to a diffusion model; a second step (S202) for initializing an output patch from the diffusion model; a third step (S203) for acquiring an adjacent output patch that is adjacent to the input patch and has already been processed by the diffusion model; and a fourth step (S204 to S206) for iteratively updating the output patch using the diffusion model on the basis of the input patch, the output patch, and the adjacent output patch.

Description

Image processing method, image processing device, image processing system, and program

[0001] The present invention relates to an image processing method, an image processing device, an image processing system, and a program.

[0002] Patent Literature 1 discloses a method for sharpening an image using a diffusion model. Non-Patent Literature 1 discloses a method for reconstructing an image acquired by a thin camera using a metalens using a diffusion model.

[0003] U.S. Pat. No. 1,176,928

[0004] arXiv:2308.02797v1 (Thin On-Sensor Nanophotonic Array Cameras)

[0005] In Patent Document 1 and Non-Patent Document 1, patch division processing cannot be performed appropriately using a diffusion model. This is because, as will be described later, conventional image processing using a diffusion model involves randomness in the process. Therefore, when patch division processing is performed, texture is generated randomly within the resulting patch, resulting in a lack of texture continuity between patches, and the seams between patches are visible when the patches are combined.

[0006] Patent Document 1 does not disclose any solution to the above-mentioned problems. In Non-Patent Document 1, the results of dividing an image into patches (sub-images) are further processed by a separately prepared neural network to synthesize the patches. However, this method does not allow for patch division processing without problems using only a diffusion model, and a separate neural network is required.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing method that is capable of appropriately performing patch division processing using a diffusion model.

[0008] An image processing method according to one aspect of the present invention includes a first step of acquiring an input patch to a diffusion model, a second step of initializing an output patch from the diffusion model, a third step of acquiring an adjacent output patch that is adjacent to the input patch and has already been processed by the diffusion model, and a fourth step of iteratively updating the output patch using the diffusion model based on the input patch, the output patch, and the adjacent output patch.

[0009] Other objects and features of the present invention are illustrated in the following examples.

[0010] According to the present invention, it is possible to provide an image processing method that is capable of appropriately performing patch division processing using a diffusion model.

[0011] FIG. 1 is a diagram showing a learning flow of a neural network in Example 1. FIG. 2 is a block diagram of an image processing system in Example 1. FIG. 3 is an external view of the image processing system in Example 1. FIG. 4 is a flowchart relating to learning of a neural network in Example 1. FIG. 5 is a flowchart relating to generation of an output image using a neural network in Example 1. FIG. 6 is a block diagram of an image processing system in Example 2. FIG. 7 is an external view of an image processing system in Example 2. FIG. 8 is a block diagram of an image processing system in Example 3. FIG. 9 is a flowchart relating to generation of an output image using a neural network in Example 3. FIG. 10 is an explanatory diagram of the gist of each example.

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0013] First, before providing a detailed description of each embodiment, a diffusion model and an outline of each embodiment will be described. In each embodiment, patch division processing is appropriately performed without any adverse effects in image processing using a diffusion model. A diffusion model is a type of generative AI (artificial intelligence) primarily used for image generation. By sampling from a target image distribution conditioned by image input or text input, a high-quality target image corresponding to the input image or text can be generated. For example, Stable Diffusion uses a diffusion model to generate a realistic image that looks like it was drawn by a human in response to text entered by a user. Note that in each embodiment, patch division processing refers to dividing a large captured image into multiple small images (patches) and sequentially processing the patches.

[0014] Diffusion models consist of two main processes: a forward diffusion process and a reverse diffusion process. The diffusion process incrementally adds Gaussian noise to a target image, slowly removing details from the target image until it becomes a pure noise image. The reverse diffusion process starts with a pure noise image, and reverses the diffusion process by conditioning it with image or text inputs and iteratively recovering it. The reverse diffusion process can be trained using a neural network (machine learning model) that takes a noisy target image and a conditioning image (or signal such as text) as inputs to remove noise. Details will be provided later.

[0015] The diffusion process is given by the following equation (1):

[0016]

[0017] where y t-1 and y t represents the step (iteration number) t-1 of the diffusion process q given by the Markov chain and the intermediate image of the next step t (t=1~T), respectively. N is a standard normal distribution (Gaussian distribution), α t (0<αt<1) is y at step t t-1 respectively represent the variance of the Gaussian noise to be added to

[0018] By transforming (marginalizing) equation (1), the intermediate image y given the target image y0 is t The distribution of is given by the following equation (2).

[0019]

[0020] where γ t is added to the target image y0 and t In other words, equation (2) expresses the variance of the Gaussian noise to obtain the intermediate image y t-1 Gaussian noise is added to the intermediate image y t By adding Gaussian noise to the target image y0, the intermediate image y t This indicates that it is possible to obtain

[0021] Further modifying equations (1) and (2) gives the target image y0 and the intermediate image y t Given an intermediate image y t-1 The posterior distribution of is given by the following equation (3).

[0022]

[0023] where μ and σ 2 is the target image y0 and the intermediate image y t The equation (3) represents the mean and variance of the standard normal distribution, which is the posterior distribution of the intermediate image yt-1 when t and the mean μ obtained from the target image y0, and the variance σ 2 This means that the intermediate image yt-1 at the t-1th step can be estimated by adding Gaussian noise with

[0024] The despreading process is given by the following equation (4).

[0025]

[0026] Here, the inverse diffusion process p (t=T~1) is defined as an inverse Markov chain that proceeds in the reverse direction of the diffusion process q. Note that, following the diffusion process q, the pure noise image y T We start with Gaussian noise sampled from a standard normal distribution (the second equation in equation (4)). In addition, each step of the de-diffusion process (the third equation in equation (4)) can be defined as a trainable isotropic Gaussian conditional distribution if the variance of the Gaussian noise added at each step of the diffusion process is set small.

[0027] The mean μ of the Gaussian distribution at each step of the dediffusion process θ is calculated from the posterior distribution of Equation (3), but the target image y0 is required. However, the pure noise image y T In the despreading process starting from , the target image y0 is generally unknown.

[0028] Therefore, in the diffusion model, a neural network (machine learning model) is trained to remove noise from a noise image in which Gaussian noise with various variances is added to the target image y0, and the intermediate image y ｔ The noise-removed target image y0^ is estimated from

[0029] The target image y0^ after noise removal in the de-diffusion process is given by the following equation (5).

[0030]

[0031] Here, f is a noise removal neural network with θ as a parameter, and x is a condition used for noise removal. As a condition used for noise removal, for example, a vector in which text is embedded is used in Stable Diffusion, and an interpolated image of a low-resolution image is used in Patent Document 1. Also, in Patent Document 1, an intermediate image y t The interpolated image is connected in the channel direction and input to the neural network, and noise is removed by conditioning the interpolated image. In addition, the noise removal neural network is also provided with a noise level (noise strength) γ tis introduced and noise is removed according to the noise level. Note that in each embodiment, the variance of the Gaussian noise added in the diffusion process is used as the noise level, but this is not limiting. A method for inputting the noise level into the noise removal neural network will be described later.

[0032] By substituting the estimated target image y0^ into equation (3), the mean of the posterior distribution is given parametrically by the following equation (6).

[0033]

[0034] The variance of the Gaussian noise is the variance of the Gaussian noise added in the diffusion process, α t Using this, each step of the despreading process is given by the following equation (7).

[0035]

[0036] By repeating this procedure from t=T to 1, we obtain a pure noise image y T The target image y0 can be generated from (more precisely, sampled from the target image distribution conditioned on x). The diffusion model is also known as the Denoised Diffusion Probabilistic Model (DDPM).

[0037] In the prior art such as Patent Document 1, the intermediate image y t The estimated target image y0^ is then converted to the intermediate image y t-1 However, this does not allow the Gaussian noise (standard normal random number) ε t Therefore, when dividing the patch into parts using a conventional diffusion model, the texture is generated randomly for each patch, and there is no texture continuity between patches. As a result, when the patches are combined, the seams between the patches become visible.

[0038] Therefore, in each embodiment, instead of formula (7), the intermediate patch y t The estimated target patch y0^ is then t-1It should be noted that equation (8) assumes that a large original image I is divided into multiple small patches x, which are then used as conditioning patches x and sequentially processed using a diffusion model (patch division processing). Furthermore, equation (8) assumes that when dividing the original image I into patches, there is some overlap between adjacent patches (corresponding parts exist).

[0039]

[0040] Here, z is the processed neighboring patch (neighboring output patch) obtained by processing the conditioned patch x and the patch extracted from the neighboring location with the diffusion model. m is the neighboring patch z and the intermediate patch y t ∇ is a mask that extracts the corresponding part of the adjacent patch z and the intermediate patch y t In other words, in addition to approaching the target patch distribution conditioned by patch x at each step of the dediffusion process given by Equation (7), the gradient is calculated from the corresponding part of the processed neighbor patch z and the intermediate patch y t Constraints are added to each step of the de-diffusion process so that the corresponding parts of

[0041] Depending on the location of the patch x extracted from the original image I, there may be multiple adjacent patches z that have been processed using the diffusion model. In that case, the intermediate image y is generated using multiple masks m from the multiple adjacent patches z. t and execute equation (8). The gradient may be calculated from the adjacent patch z and the target patch y^, or from the adjacent patch z and the mean μθ of the posterior distribution given by equation (6). The calculated gradient may also be weighted. Depending on the location and order in which patch x is extracted from the original image I, there may be no adjacent patch z. In that case, equation (7) can be used as usual.

[0042] In each example, the intermediate patch y tThe corresponding portions of adjacent patches z processed using the diffusion model are gradually brought closer together. Therefore, in the final patch y0 obtained after repeated iterations of the de-diffusion process (t = T → 1), the corresponding portions of adjacent patches processed using the diffusion model ideally match. This means that even when patches resulting from processing in each embodiment are combined, no seams are created between the adjacent patches that have already been processed, and patch division processing using the diffusion model can be performed without any problems.

[0043] Next, an example of upscaling (super-resolution) using patch division processing with a diffusion model will be described with reference to Fig. 10. Here, upscaling refers to image processing that generates an enlarged high-resolution image by estimating high-frequency components that cannot be expressed in a low-resolution image.

[0044] In each embodiment, a low-resolution captured image is first acquired, and input patches (corresponding to x in the formula) are extracted from the bicubic-interpolated image (corresponding to I in the formula), followed by sequential processing (patch division processing) using a diffusion model. When extracting input patches from the bicubic-interpolated image, each patch is acquired so that it partially overlaps with adjacent patches (having overlapping areas (common areas)). In FIG. 10 , the bicubic-interpolated image of the captured image is divided into four input patches so that it partially overlaps with adjacent patches. Therefore, for example, the input patch at the bottom right has corresponding portions to the input patches at the top left, top right, and bottom left. Also, FIG. 10 illustrates an example of sequential processing, starting with the input patch at the top left and processing the input patch at the bottom right last. The dotted arrows in the figure indicate the order of sequential processing.

[0045] Next, the input patch is processed according to equation (8). Note that Fig. 10 shows an example in which the bottom right input patch is processed. Therefore, it is assumed that the other input patches (top left, top right, bottom left) have already been processed using the diffusion model.

[0046] In equation (8), first, the current intermediate patch (output patch, y T (corresponding to ) is initialized with Gaussian noise. Note that the size of the noise patch is the same as that of the interpolated patch.

[0047] Next, the input patch is input to the neural network along with the current intermediate patch for noise removal. Information about the variance of the noise patch (corresponding to γT in the formula), which is the noise level, is also input to the neural network. The input patch is also connected to the current intermediate patch in the channel direction and input to the neural network, where it is used as a condition for noise removal. The variance of the noise patch uses a value determined in advance for each iteration of the dediffusion process. For example, the maximum and minimum values ​​of the variance of the Gaussian noise of the noise patch may be determined, and a value determined linearly from the number of iterations may be used. The neural network is also trained in advance to perform noise removal given by Equation (5).

[0048] Next, the mean of the posterior distribution given by equation (5) (corresponding to μθ in the equation) is calculated from the noise-removed patch (corresponding to y0^ in the equation).

[0049] Next, the gradient given by Equation (8) is calculated from the noise removal patch and the adjacent patch (corresponding to z in the formula). The adjacent patches are output patches resulting from processing the upper left, upper right, and lower left input patches that have already been processed using the diffusion model. In FIG. 10 , a mask (corresponding to m in the formula) (not shown) is used to extract and combine portions of the three adjacent output patches that correspond to the currently processed input patch. Technically, this is not an adjacent patch but a combined patch, as described above, but for ease of understanding, it is referred to as such in FIG. 10 . Furthermore, as described with reference to Equation (8), the gradient may be calculated using the current intermediate patch or the average of the posterior distribution instead of the noise removal patch.

[0050] Next, the intermediate patch is updated using the calculated gradient using equation (8) (the output patch, y t-1 equivalent to).

[0051] The procedure according to the above formula (8) is repeated a predetermined number of times for each input patch, sequentially applying it to multiple input patches extracted from the entire surface of the bicubic interpolated image. The resulting multiple upscaled output patches are then synthesized. Therefore, according to each embodiment, an upscaled image that has been subjected to patch division processing using a diffusion model can be generated without any drawbacks associated with the patch division processing.

[0052] In conventional diffusion models, including those disclosed in Patent Document 1, when patch division processing is performed, seams occur between adjacent processed patches when the patches are combined, for the reasons described above. Therefore, in order to process large images without any problems using conventional diffusion models, it was necessary to process them all at once. However, processing large images all at once requires expensive graphics processing units (GPUs) with large memory, which places a heavy financial burden on users. In contrast, each embodiment can perform patch division processing using a diffusion model without any problems, allowing large images to be processed even with an inexpensive GPU, thereby reducing the financial burden on users.

[0053] In Non-Patent Document 1, patches processed by a diffusion model are processed by a separately prepared neural network to perform patch synthesis. On the other hand, in each embodiment, the diffusion model expressed by Equation (8) is devised, so that a separate neural network is not required, thereby reducing processing time.

[0054] The image processing method described above is an example, and the embodiments are not limited to this. Details of other image processing methods will be described in the following embodiments.

[0055] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, a low-resolution captured image is divided into patches, and the resulting low-resolution patches are upscaled using a diffusion model to generate high-resolution patches, which are then trained and executed.

[0056] Fig. 2 is a block diagram of an image processing system 100 according to this embodiment. Fig. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device 103, a display device 104, a storage medium 105, an input device 106, an output device 107, and a network 108.

[0057] The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and a learning unit 101d.

[0058] The imaging device 102 includes an optical system 102a and an imaging element 102b. The optical system 102a focuses light incident on the imaging device 102 from the subject space. The imaging element 102b receives an optical image of the subject formed via the optical system 102a to acquire a captured image (low-resolution image). The imaging element 102b is, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. Information regarding the shooting conditions of the captured image (such as the pixel pitch of the imaging element 102b, the type of optical low-pass filter, and ISO sensitivity) can be acquired along with the image. Development conditions for the captured image (such as noise reduction strength, sharpness strength, and image compression rate) can also be acquired along with the image. This information acquired along with the image can also be transmitted to an input acquisition unit 103b of the image estimation device 103 (described later) along with the image. A storage unit for storing the acquired image, a display unit for displaying the image, a transmission unit for transmitting the image to an external device, and an output unit for storing the image in an external storage medium are not shown. A control unit that controls each unit of the imaging device 102 is also not shown.

[0059] The image estimation device 103 includes a storage unit 103a, an input acquisition unit 103b, an initialization unit 103c, an output acquisition unit 103d, a noise reduction unit 103e, and an update unit 103f. The input acquisition unit 103b acquires a low-resolution patch (input patch) from the captured image (more precisely, a bicubic interpolated image of the captured image). The initialization unit 103c then initializes the current intermediate patch (output patch) with Gaussian noise. The noise reduction unit 103e estimates a noise-removed patch from the current intermediate patch using a neural network conditioned on the low-resolution patch. The output acquisition unit 103d acquires a patch (adjacent patch) extracted from a location adjacent to the low-resolution patch in the captured image and already processed using a diffusion model. The update unit 103f updates the current intermediate patch from the noise-removed patch and adjacent patch. This procedure is repeated to perform image processing that ultimately generates an upscaled high-resolution patch (output patch). Details of the method for initializing the current intermediate patch in the initialization unit, the method for removing noise from the current intermediate patch using a neural network in the noise removal unit, and the method for updating the current intermediate patch from the noise removal patch and adjacent patches in the update unit will be described later.

[0060] The low-resolution patches may be patches extracted from an image captured by the image capturing device 102, or may be extracted from an image stored in the storage medium 105. Alternatively, patches that have already been divided from an image and stored in the storage medium 105 may be used.

[0061] The weight information of the neural network used for noise removal is read from the storage unit 103a. The weights are learned by the learning device 101, and the image estimation device 103 reads the weight information from the storage unit 101a via the network 108 in advance and stores it in the storage unit 103a. The stored weight information may be the weight values ​​themselves or in an encoded format. Details of the neural network weight learning method and the image processing method using a diffusion model that uses a neural network will be described later.

[0062] The upscaled high-resolution patches are synthesized and then output to at least one of a display device 104, a storage medium 105, and an output device 107. The display device 104 is, for example, a liquid crystal display or a projector. A user can check the patches being processed via the display device 104 and perform image editing work via the input device 106. The storage medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The input device 106 is, for example, a keyboard or a mouse, etc. The output device 107 is, for example, a printer, etc.

[0063] Next, a weight learning method (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Figs. 1 and 4. Fig. 1 is a diagram showing the flow of learning weights for a neural network (machine learning model). Fig. 4 is a flowchart related to weight learning. Each step in Fig. 4 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the learning unit 101d.

[0064] First, in step S101, the acquisition unit 101b acquires a low-resolution patch (degraded patch) 201 and a corresponding high-resolution patch (correct patch) 200, as shown in FIG. 1 . In this embodiment, a patch is a small image having a predetermined number of pixels. For example, the low-resolution patch may be 128 x 128 x 3 pixels (corresponding to the number of pixels in the vertical, horizontal, and channel directions), and the corresponding high-resolution patch may be 256 x 256 x 3 pixels. In this case, since the vertical and horizontal sizes are doubled, the upscaling factor is 2 (enlarged by 4 times in terms of the number of pixels). Note that the upscaling factor is not limited to 2, and may be any factor as long as a high-resolution patch corresponding to the low-resolution patch can be acquired.

[0065] Furthermore, a low-resolution patch and a corresponding high-resolution patch may be obtained by capturing images of the same subject using optical systems with different focal lengths and then cutting out corresponding portions of the two images. Alternatively, a high-resolution patch may be downsampled to generate a corresponding low-resolution patch. Alternatively, a corresponding high-resolution patch that is less affected by (aberration and diffraction from) the optical system 102a and an equivalent low-resolution patch captured by the imaging device 102 may be generated by numerical calculation. In this embodiment, a low-resolution patch and a corresponding high-resolution patch are generated by numerical calculation, but the present invention is not limited to this.

[0066] Next, in step S102, the generation unit 101c generates an interpolated patch 202 by interpolating and enlarging the low-resolution patch 201 to have the same number of pixels as the corresponding high-resolution patch 200. In this embodiment, bicubic interpolation is used as the interpolation method, but the present invention is not limited to this, and nearest neighbor interpolation or bilinear interpolation may also be used.

[0067] Next, in step S103, the generating unit 101c generates a noise patch 203 by adding noise to the high-resolution patch 200.

[0068] Next, in step S104, the learning unit 101d uses a neural network to generate a noise removal patch 204 by removing noise from the noise patch 203. The noise removal patch 204 is generated based on the interpolation patch 202, the noise patch 203, and information about the intensity (noise level) of the noise added to the noise patch 203. Note that the noise removal patch 204 is an estimate of the high-resolution patch 200, and ideally the two will match. In this embodiment, the variance of the added Gaussian noise is used as the noise level, but this is not limited thereto. Furthermore, noise removal is performed according to the intensity of the noise added to the noise patch 203, but a method for inputting the noise level (variance of Gaussian noise) to the neural network will be described later. In this embodiment, the interpolation patch 202 and the noise patch 203 are concatenated in the channel direction and input to the neural network, but this is not limited thereto.

[0069] Next, in step S105, the learning unit 101d updates the weights of the neural network based on the error between the high-resolution patch 200 and its estimated denoising patch 204 (performs neural network learning). Here, the weights include the filter components and biases of each layer. In this embodiment, back propagation is used to update the weights, but this is not limitative. In mini-batch learning, the error between the high-resolution patch 200 and its corresponding denoising patch 204 is calculated, and the weights are updated. For example, the L2 norm or the L1 norm may be used as the loss function. The weight update method (learning method) is not limited to mini-batch learning, and batch learning or online learning may also be used.

[0070] Next, in step S106, the learning unit 101d determines whether learning (weight update) is complete. Completion can be determined by, for example, whether the number of iterations of learning (weight update) has reached a specified value, or whether the amount of change in weight during update is smaller than a specified value. If it is determined that learning is incomplete, the process returns to step S101, and multiple new low-resolution patches 201 and corresponding high-resolution patches 200 are obtained. On the other hand, if it is determined that learning is complete, the learning device 101 ends learning and saves weight information in the storage unit 101a.

[0071] In this embodiment, a method for learning the weights of a neural network in the image domain has been described, but the method is not limited to this. Learning may also be performed in the feature map (latent) domain from the low-resolution patches 201 and the interpolated patches 202. In this case, learning can be performed using a feature map obtained using a separately prepared neural network and a noise patch 203 obtained by adding noise to a feature map obtained from the high-resolution patches 200 using a separately prepared neural network.

[0072] In this embodiment, the neural network configuration shown in FIG. 1 is used, but the present invention is not limited to this.

[0073] CN in Figure 1 represents a convolutional layer. CN calculates the sum of the input, filter convolution, and bias, and then performs nonlinear transformation on the result using an activation function. The initial values ​​of each filter component and bias are arbitrary and are determined by random numbers in this embodiment. The activation function can be, for example, a rectified linear unit (ReLU) or a sigmoid function. The multidimensional array output from each layer except the final layer is a feature map. A feature map is generally a four-dimensional array with dimensions of batch, length, width, and channel. The skip connection 205 combines feature maps output from discontinuous layers. Feature maps can be combined by summing elements or by concatenating in the channel direction. In this embodiment, element-by-element sums are used.

[0074] The elements (blocks or modules) within the dotted line frame in FIG. 1 represent residual blocks. A network with multiple layers of residual blocks is called a residual network and is widely used in image processing using machine learning models. However, this embodiment is not limited to this, and other elements may be layered to form a network. For example, an inception module may be used, which juxtaposes convolutional layers with different convolutional filter sizes and integrates the resulting multiple feature maps to form a final feature map. Alternatively, a dense block with dense skip connections may be used.

[0075] The vertical and horizontal sizes of the feature map are also expanded and contracted within the network. Here, pooling, stride convolution, etc. can be used to reduce the feature map. Furthermore, deconvolution (or transposed convolution), pixel shuffle, interpolation, etc. can be used to expand the feature map. Although not shown in the figures in this embodiment, stride convolution is used to reduce the feature map, and nearest neighbor interpolation is used to expand it.

[0076] Additionally, a noise level (γ in the figure) is input to the residual block, and noise is removed from the noise patch according to the noise level. In this embodiment, a vector with noise level information embedded is generated using a separately prepared multilayer perceptron (MLP in the figure) based on the noise level. Then, based on this, the feature map obtained between the two convolutional layers of the residual block is scaled (an affine transformation using a gain to be multiplied and a bias to be added to the feature map). However, this embodiment is not limited to this. A multilayer perceptron is a type of neural network. Furthermore, since the noise level of the noise image varies with each iteration in the dediffusion process, a different noise level γt is input to the residual block for each iteration.

[0077] Next, generation of an upscaled image using a diffusion model executed by the image estimation device 103 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart related to generation of an upscaled image using a diffusion model. Each step in Fig. 5 is mainly executed by the input acquisition unit 103b, the initialization unit 103c, the output acquisition unit 103d, the noise removal unit 103e, or the update unit 103f of the image estimation device 103.

[0078] First, in step S201, the input acquisition unit 103b acquires input patches for the diffusion model from a bicubic-interpolated image of the captured image. The captured image is a low-resolution image, as in the learning process. In this embodiment, the captured image is transmitted from the imaging device 102, but this is not limiting. Note that image information may also be acquired along with the captured image and used in the steps described below. Also, as in the learning process, the input patches are acquired after an interpolated image (not shown) is generated by an interpolation unit (not shown) that performs bicubic interpolation on the captured image. Alternatively, the input patches may be bicubic-interpolated based on the image size acquired along with the captured image. Furthermore, multiple input patches are acquired from the captured image and sequentially processed (patch division process) using the diffusion model in the steps described below. During patch division, adjacent patches have overlapping areas (corresponding portions). That is, each input patch has an overlapping area (common area, corresponding portion) that overlaps (is common to) a portion of another input patch. Furthermore, information (location information) and patch size information acquired along with the captured image from which the input patch is extracted may be used to acquire the input patches.

[0079] In this embodiment, acquisition of an input patch starts from the upper left corner of the bicubic interpolated image of the captured image. Then, shifting to the right, the next input patch is acquired. When the upper right corner is reached, shifting downward from the starting point, the next input patch is acquired, and then shifting to the right, the next input patch is acquired. This step is repeated to acquire multiple input patches from across the entire captured image in the order in which horizontally written text is read. However, this embodiment is not limited to this. Depending on the location where the input patch is acquired, it may extend beyond the captured image. In that case, the shift amount or the overlap amount with adjacent patches may be adjusted. The input patch corresponds to the conditioning patch x of the formula, and the captured image (more precisely, its bicubic interpolated image) corresponds to the original image I of the formula.

[0080] Next, in step S202, the initialization unit 103c generates a noise patch (output patch) of the same size as the input patch from Gaussian noise (initializes the output patch). The noise patch may be generated using a pseudo-random number with a fixed random number seed. In this case, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, this may be used. The noise patch may also be generated by using a pure noise image y T is equivalent to

[0081] Next, in step S203, the output acquisition unit 103d acquires a patch (adjacent patch) that has been acquired from a location adjacent to the input patch in the captured image and has already been processed using the diffusion model. Note that the acquisition of the adjacent patch may use information (position information) about the location where the input patch is extracted from the captured image, which information was acquired together with the captured image and the input patch. The output acquisition unit 103d also simultaneously acquires a mask that extracts the corresponding portion between the adjacent patch and the input patch. The adjacent patch corresponds to z in the formula, and the mask corresponds to m in the formula.

[0082] Next, in step S204, the noise removal unit 103e connects the input patch and the noise patch (output patch) in the channel direction and inputs them into the neural network to generate a noise removal patch. Note that a neural network with the same configuration as that shown in FIG. 1 is used to generate the noise removal patch. The weight information is pre-trained information transmitted from the learning device 101 and stored in the memory unit 103a. In this embodiment, a default value corresponding to the number of iterations is used as the noise level used to generate the noise-removed image, but this is not limited to this. For example, a noise level schedule determined for each iteration and acquired together with the captured image may also be used. The noise removal patch corresponds to the estimated target patch y^ in the mathematical formula.

[0083] Next, in step S205, the update unit 103f generates an updated upscaled patch (output patch) using the noise removal patch, the adjacent patch, and the mask based on equation (8). In this embodiment, the noise patch used for updating is generated from Gaussian noise, but it may also be generated using a pseudo-random number with a fixed random number seed. In this case, a value corresponding to the number of iterations of the process or a value corresponding to the number of iterations extracted from a separately prepared sequence may be used as the random number seed value. Also, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, it may be used. The updated upscaled patch is generated by using the intermediate patch y t-1 , the noise patch is Gaussian noise ε t is equivalent to

[0084] Next, in step S206, the update unit 103f determines whether the processing is complete. If the processing is complete, the updated upscaled patch is output as an output patch to the storage unit 103a, the storage medium 105, or the like. If the processing is not complete, the upscaled patch is used as a new output and the processing is repeated from step S204. In this embodiment, the number of iterations of the processing is used as a criterion for determining whether the processing is complete, but this is not limited to this. Note that information (position information) about the location where the input patch was extracted from the captured image may also be output together with the output patch and used in the synthesis of the output patch, which will be described later. Furthermore, the upper limit of the number of iterations of the processing may be, for example, a predetermined value acquired together with the captured image or the input patch.

[0085] Next, in step S207, the update unit 103f determines whether processing is complete for all input patches acquired from the bicubic interpolated image of the captured image. If processing is complete, the processed output patches are composited by a composite unit (not shown) and output as an upscaled image to at least one of the display device 104, the storage medium 105, and the output device 107. If processing is not complete, a new input patch is acquired from the bicubic interpolated image of the captured image, and the processing is repeated from step S201.

[0086] In this way, the update unit 103f iteratively updates the output patch by using a diffusion model based on the input patch, the output patch, and the adjacent patch. Preferably, the update unit 103f iteratively updates the output patch based on an intermediate patch obtained by using the diffusion model based on the input patch and the output patch, and a corresponding portion between the intermediate patch and the adjacent patch. More preferably, the update unit 103f iteratively updates the output patch based on the gradient of the corresponding portion.

[0087] Preferably, the input acquisition unit 103b acquires a plurality of input patches divided from the image. The plurality of input patches includes a first input patch and a second input patch, and the second input patch has an area that overlaps with a portion of the first input patch. More preferably, a synthesis unit (not shown) provided in the image estimation device 103 performs synthesis processing based on a plurality of output patches including a first output patch corresponding to the first input patch and a second output patch corresponding to the second input patch. Note that the synthesis unit (not shown) may synthesize a processed output patch by cutting (trimming) the processed output patch so that there is no overlap between the processed output patch and an adjacent processed output patch. Alternatively, the synthesis unit (not shown) may synthesize the processed output patch by weighted averaging the overlap between the processed output patch and an adjacent processed output patch. In this embodiment, patch synthesis is performed by trimming, but this is not limited to this. Furthermore, information (position information) about the location where the input patch was extracted from the captured image is used to synthesize the output patch, but this is not limited to this. Note that the update unit 103f may also function as a synthesis unit.

[0088] In this embodiment, a method for generating an upscaled image using a diffusion model has been described in the image domain, but the present invention is not limited to this. The method may be performed in the feature map (latent) domain using a feature map obtained using a separately prepared neural network for an input patch extracted from a bicubic interpolated image of a captured image. In this case, the initialization unit 103c generates a noise patch (output patch) of the same size as the feature map obtained from the input patch, and the output acquisition unit 103d generates a feature map for the adjacent patch using a separately prepared neural network. The noise removal unit 103e removes noise in the latent domain from the feature map and noise patch obtained from the input patch and the adjacent patch using a neural network trained in the latent domain.

[0089] In this embodiment, a method for estimating the upscaled image itself has been described as a method for generating an upscaled image using a diffusion model, but the present invention is not limited to this. For example, it is also possible to estimate only the residual component of the upscaled image and the bicubic interpolated image of the captured image.

[0090] In addition, in this embodiment, it is assumed that the captured image to be used for generating an upscaled image using the diffusion model is a color image, but the present invention is not limited to this. For example, an upscaled luminance image may be generated using a monochrome luminance image extracted from a captured color image.

[0091] In this embodiment, the input patch is a patch extracted from an image and not yet processed by the diffusion model. The input patch may include at least one of image metadata, patch size and coordinate information when extracting the patch from the image, information about a random number used when processing the patch extracted from the image, and information about the noise level of the patch extracted from the image.

[0092] In this embodiment, the adjacent patch is a patch that overlaps with a part of the input patch, is extracted from the image, and is processed by the diffusion model.

[0093] With the above configuration, according to this embodiment, patch division processing can be performed without any problems in upscale image processing using a diffusion model.

[0094] [Embodiment 2] Next, an image processing system according to embodiment 2 of the present invention will be described. In this embodiment, a blurred captured image is divided into patches, and the obtained blurred patches are deblurred using a diffusion model. The image processing system of this embodiment differs from embodiment 1 in that an imaging device acquires a captured image (blurred image) and performs deblurring image processing.

[0095] Fig. 6 is a block diagram of an image processing system 300 in this embodiment. Fig. 7 is an external view of the image processing system 300. The image processing system 300 includes a learning device 301 and an imaging device 302 connected via a network 303. Note that the learning device 301 and the imaging device 302 do not need to be constantly connected via the network 303.

[0096] The learning device 301 includes a storage unit 311, an acquisition unit 312, a generation unit 313, and a learning unit 314. These units are used to learn a neural network (update weights) for image processing that generates deblurred patches from blurred patches using a diffusion model.

[0097] The imaging device 302 captures an image of a subject space, acquires a captured image (blurred image), and generates a blurred image from the captured image. Details of the image processing executed by the imaging device 302 will be described later. The imaging device 302 has an optical system 321 and an image sensor 322. The image estimation unit 323 has an input acquisition unit 323a, an initialization unit 323b, an output acquisition unit 323c, a noise removal unit 323d, and an update unit 323e.

[0098] Regarding the neural network learning (weight update) executed by the learning device 301, only the differences from the first embodiment will be described with reference to FIG.

[0099] First, in step S101, the acquisition unit 312 acquires a sharp patch (correct patch) with little blur corresponding to a blurred patch (degraded patch).

[0100] Next, in step S102, the generation unit 313 generates a noise patch by adding noise to the sharp patch.

[0101] Next, in step S103, the generation unit 313 generates an adjacent patch by cutting out a portion of the sharp patch. Here, the adjacent patch corresponds to z in the formula (the adjacent patch obtained by processing a patch extracted from a location adjacent to the conditioned patch x using a diffusion model). That is, this embodiment is characterized in that the adjacent patch is taken into consideration from the time of learning the weights of the neural network and is used to remove noise from the noise patch.

[0102] Next, in step S104, the learning unit 314 uses a neural network to generate a patch obtained by removing noise from the noise patch, using the blur patch, the noise patch, the adjacent patch, and information about the strength of noise added to the noise patch (noise level). Note that the denoised patch is an estimate of the sharp patch, and ideally the two will match.

[0103] Furthermore, by removing noise from a noise patch while taking into account adjacent patches, the corresponding portions of the adjacent patch and the denoised patch ideally coincide. As a result, when a patch that has undergone deblurring image processing and the adjacent patch are combined using a diffusion model (described later), no seams are created between the patches, allowing for patch division processing without any problems. Note that in this embodiment, the blurred patch, the noise patch, and the adjacent patch are concatenated in the channel direction and input to the neural network, but this is not limited to this.

[0104] The subsequent steps S105 and S106 are the same as those in the first embodiment, and therefore their description will be omitted.

[0105] Next, details of the image processing executed by the imaging device 302 will be described. Neural network weight information is learned in advance by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads the weight information from the storage unit 311 via the network 303 and stores it in the storage unit 324. The image estimation unit 323 generates a deblurred patch from the blurred patch obtained by dividing the blurred captured image into patches by the image estimation unit 323. The deblurred patch can be generated using the trained neural network weight information stored in the storage unit 324 and the blurred captured image and its image information acquired by the input acquisition unit 323a. The generated deblurred patch is synthesized into a deblurred image and then stored in the storage medium 325a. When a user issues an instruction to display the deblurred image via the input unit 326, the stored image is read and displayed on the display unit 325b. The captured image and its image information stored in the storage medium 325a may be read out, and the blur-removed image may be generated by the image estimation unit 323. The above series of controls are performed by the system controller 327.

[0106] Next, generation of a deblurred image using a diffusion model executed by the image estimation unit 323 in this embodiment will be described. The image processing procedure of this embodiment is substantially the same as that of the first embodiment shown in Fig. 5, and therefore the image processing of this embodiment will be described with reference to Fig. 5. Each step of the image processing is mainly executed by the input acquisition unit 323a, initialization unit 323b, output acquisition unit 323c, noise removal unit 323d, or update unit 323e of the image estimation unit 323.

[0107] First, in step S201, the input acquisition unit 323a acquires a captured image (blurred image). The captured image is a blurred image, similar to the image used in learning. In this embodiment, the captured image is acquired by the imaging device 302 and stored in the storage unit 324, but this is not limited thereto. Note that image information may also be acquired along with the captured image and used in the steps described below. The input acquisition unit 323a also acquires multiple input patches from the captured image and sequentially processes them (patch division processing) using a diffusion model in the steps described below. When dividing the patches, overlapping portions (corresponding portions) are provided between adjacent patches. Furthermore, information (position information) about the location where the input patch is extracted from the captured image and the patch size, which are acquired together with the captured image, may also be used to acquire the input patches. In this embodiment, multiple input patches are acquired randomly so as to cover the entire captured image, but this is not limited thereto. The input patches correspond to the conditioning patches x of the mathematical formula, and the captured image corresponds to the original image I of the mathematical formula.

[0108] Next, in step S202, the initialization unit 323b generates a noise patch (output patch) of the same size as the input patch from Gaussian noise. Note that the noise patch may be generated using pseudo-random numbers with a fixed random number seed, as in the first embodiment. In this case, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, this may also be used, as in the first embodiment. Also, the noise patch corresponds to a pure noise image yT in the mathematical formula.

[0109] Next, in step S203, the output acquisition unit 323c acquires a patch (adjacent patch) that has been acquired from a location adjacent to the input patch in the captured image and has already been processed using the diffusion model. Note that the acquisition of the adjacent patch may use information (position information) about the location where the input patch is extracted from the captured image, which information was acquired together with the captured image and the input patch. The output acquisition unit 323c also simultaneously acquires a mask that extracts the corresponding portion of the adjacent patch and the input patch. The adjacent patch corresponds to z in the formula, and the mask corresponds to m in the formula.

[0110] Next, in step S204, the noise removal unit 323d connects the input patch, the noise patch (output patch), and the adjacent patch in the channel direction and inputs them into the neural network to generate a noise removal patch. To generate the noise removal patch, a neural network with a configuration substantially similar to that shown in FIG. 1 is used. The difference from FIG. 1 is the number of images input to the neural network; whereas in FIG. 1, two images (202 and 203) are input, in this embodiment, three images are input, including adjacent patches. The weight information is learned information transmitted from the learning device 301 and stored in the memory unit 324.

[0111] In this embodiment, as in the first embodiment, a predetermined value corresponding to the number of iterations is used as the noise level used when generating the noise removal patch, but this is not limited to this. For example, a schedule of noise levels determined for each iteration acquired together with the captured image may be used. Furthermore, the adjacent patch is used by extracting the portion corresponding to the input patch using a mask. Furthermore, the noise removal patch corresponds to the estimated y^ of the target patch in the mathematical formula.

[0112] Next, in step S205, the update unit 323e generates an updated deblurring patch (output patch) from the noise removal patch based on Equation (7). In this embodiment, the noise patch used for updating is generated from Gaussian noise, but it may also be generated using a pseudo-random number with a fixed random number seed. In this case, the random number seed value may be a value corresponding to the number of iterations of the process, or a value corresponding to the number of iterations extracted from a separately prepared sequence. Furthermore, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, it may also be used. Furthermore, since Equation (7) represents the dediffusion process in a conventional diffusion model, the input of the neural network fθ does not include the neighboring patch z. However, in this embodiment, the input includes the neighboring patch z. The neural network learning and the method of inputting the neighboring patch z are as described above. Furthermore, the updated deblurring patch is the intermediate patch yt-1 of the formula, and the noise patch is the Gaussian noise ε t is equivalent to

[0113] Next, in step S206, the update unit 323e determines whether the processing is complete. If the processing is complete, the updated blur removal patch is output to the storage medium 325a as an output patch. If the processing is not complete, the processing is repeated from step S204 using the blur removal patch as a new output patch. In this embodiment, the number of processing iterations is used as a criterion for determining whether the processing is complete, but the present invention is not limited to this. Note that information (position information) about the location where the input patch was extracted from the captured image may also be output together with the output patch, and may be used in the synthesis of the output patch, which will be described later. Furthermore, a default value acquired together with the captured image or the input patch may be used as the upper limit of the number of processing iterations.

[0114] Next, in step S207, the update unit 323e determines whether processing has been completed for all input patches acquired from the captured image. If processing has been completed, the processed output patches are combined in a combining unit (not shown) and output as a blur-removed image to at least one of the storage medium 325a or the display unit 325b, for example. If processing has not been completed, a new input patch is acquired from the captured image, and the processing is repeated from step S201. Note that patch combination in the combining unit (not shown) is the same as in the first embodiment.

[0115] In this manner, in this embodiment, the update unit 323e iteratively updates the output patch based on the corresponding portions among the input patch, the output patch, and the adjacent patch. Preferably, the update unit 323e iteratively updates the output patch by linking the corresponding portions among the input patch, the output patch, and the adjacent patch and inputting the linked portions into the diffusion model.

[0116] With the above configuration, according to this embodiment, patch division processing can be performed without any problems in blur removal image processing using a diffusion model.

[0117] Third Embodiment Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment transmits a captured image (Bayer image) to be processed to an image estimation device. The image processing system of this embodiment also includes a processing device (computer) that processes Bayer patches divided into patches using a diffusion model and receives an image (deBayer image) obtained by combining the resulting deBayer patches from the image estimation device. This embodiment differs from the first and second embodiments in these respects. Note that a Bayer image is raw image data acquired by an image sensor with a Bayer array in which the color filter color pattern is a checkerboard pattern. In a Bayer image, each pixel contains information for only one of the RGB colors. Therefore, to obtain a full-color image in which each pixel contains information for all three RGB colors, interpolation processing from adjacent pixels is required. This image processing is called deBayering (or demosaicing).

[0118] 8 is a block diagram of an image processing system 400 in this embodiment. The image processing system 400 includes a learning device 401, an imaging device 402, an image estimation device 403, and a computer (control device) 404. The learning device 401 and the image estimation device 403 are, for example, servers. The computer 404 is, for example, a user terminal (a personal computer or a smartphone). The computer 404 is connected to the image estimation device 403 via a network 405. The image estimation device 403 is connected to the learning device 401 via a network 406. In other words, the computer 404 and the image estimation device 403 are configured to be able to communicate with each other, and the image estimation device 403 and the learning device 401 are configured to be able to communicate with each other.

[0119] The configuration of the learning device 401 is similar to that of the learning device 101 in Example 1, and therefore a description thereof will be omitted. The learning of the neural network executed by the learning device 401 is almost similar to that in Example 1, except that Bayer patches are used instead of low-resolution patches as degraded patches, and DeBayer patches are used instead of high-resolution patches as corresponding correct patches, and therefore a description thereof will be omitted. The configuration of the imaging device 402 is similar to that of the imaging device 102 in Example 1, and therefore a description thereof will be omitted.

[0120] The image estimation device 403 includes a storage unit 403a, an input acquisition unit 403b, an initialization unit 403c, an output acquisition unit 403d, a noise removal unit 403e, an update unit 403f, and a communication unit 403g. The storage unit 403a, the input acquisition unit 403b, the initialization unit 403c, the output acquisition unit 403d, the noise removal unit 403e, and the update unit 403f function as an image processing unit of the image estimation device 403. The components of the image processing unit are similar to the storage unit 103a, the input acquisition unit 103b, the initialization unit 103c, the output acquisition unit 103d, the noise removal unit 103e, and the update unit 103f of the image estimation device 103 of Example 1. The communication unit 403g functions as a receiving unit that receives a request transmitted from the computer 404 and as a transmitting unit that transmits an output image (deBayer image) generated by the image estimation device 403 to the computer 404.

[0121] The computer 404 has a communication unit 404a, a display unit 404b, an input unit 404c, a processing unit 404d, and a storage unit 404e. The communication unit 404a functions as a transmitter that transmits a request to the image estimation device 403 to cause the image estimation device 403 to execute processing on the captured image (Bayer image). The communication unit 404a also functions as a receiver that receives an output image (debayer image) processed by the image estimation device 403. The display unit 404b has a function to display various information. The information displayed by the display unit 404b includes, for example, the captured image (Bayer image) to be transmitted to the image estimation device 403 and the output image (debayer image) received from the image estimation device 403. The input unit 404c receives instructions from the user to start image processing, etc. The processing unit 404d has a function to perform image processing, including white balance adjustment and gamma correction, on the output image (debayer image) received from the image estimation device 403. The storage unit 404e stores the captured image acquired from the imaging device 402, the output image received from the image estimation device 403, and the like.

[0122] Next, image processing in this embodiment will be described with reference to Fig. 9. The image processing in this embodiment is equivalent to the image processing described in embodiment 1 with reference to Fig. 5. Fig. 9 is a flowchart related to generation of an output image using a neural network in this embodiment. The image processing shown in Fig. 9 is started when an instruction to start image processing is given by a user via computer 404. First, the operation of computer 404 will be described.

[0123] First, in step S501, the computer 404 transmits a request for processing a captured image (Bayer image) (processing an input patch related to the captured image) to the image estimation device 403. Note that the method for transmitting the captured image to be processed to the image estimation device 403 is not important. For example, the captured image may be uploaded to the image estimation device 403 simultaneously with step S501, or may be uploaded to the image estimation device 403 before step S501. Furthermore, the captured image may be an image stored on a server different from the image estimation device 403. Furthermore, in step S501, the computer 404 may transmit an ID for authenticating a user, image information, and the like together with the request for processing the captured image.

[0124] Next, in step S 502 , the computer 404 receives the output image (debayered image) generated in the image estimation device 403 .

[0125] Next, the operation of the image estimation device 403 will be described.

[0126] First, in step S601, the image estimation device 403 receives a request for processing a captured image (Bayer image) transmitted from the computer 404. The image estimation device 403 determines that processing for the captured image has been instructed, and executes the processing from step S602 onwards.

[0127] Next, in step S602, the input acquisition unit 403b acquires a captured image (Bayer image, input). Similar to the upscaling in the first embodiment, an interpolated image is generated by simply de-Bayering the captured image using bilinear interpolation in an interpolation unit (not shown), but the interpolation may also be performed by the input acquisition unit 403b. In this embodiment, the captured image is transmitted from the imaging device 402, but this is not limiting. Furthermore, multiple input patches are acquired from a bilinearly interpolated image of the captured image, and sequential processing (patch division processing) is performed using a diffusion model in a step described below. During patch division, overlapping portions (corresponding portions) are provided between adjacent patches. Furthermore, information (position information) and patch size information about the location where the input patch is extracted from the captured image, acquired together with the captured image, may also be used to acquire the input patch.

[0128] In this embodiment, acquisition of an input patch starts from the top right corner of the bilinearly interpolated image of the captured image. Then, a downward shift is made to acquire the next input patch. When the bottom right corner is reached, a shift is made to the left of the starting point to acquire an input patch, and a downward shift is made to acquire the next input patch. This process is repeated to acquire multiple input patches from across the entire captured image in the order in which vertically written text is read. However, this embodiment is not limited to this. Furthermore, depending on the location where an input patch is acquired, it may extend beyond the captured image. In such cases, the shift amount or the amount of overlap with adjacent patches may be adjusted. Furthermore, the input patch corresponds to a conditioning patch x of the formula, and the captured image (more precisely, its bilinearly interpolated image) corresponds to the original image I of the formula.

[0129] Next, in step S603, the initialization unit 403c generates a noise patch (output patch) of the same size as the input patch from Gaussian noise. The noise patch may be generated using a pseudo-random number with a fixed random number seed. In this case, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, this may be used. The noise patch is generated by using a pure noise image y T is equivalent to

[0130] Next, in step S604, the output acquisition unit 403d acquires a patch (adjacent patch) that has been acquired from a location adjacent to the input patch in the bilinear interpolated image of the captured image and has already been processed using the diffusion model. Note that the acquisition of the adjacent patch may use information (position information) about the location where the input patch is extracted from the captured image, acquired together with the captured image and the input patch. A mask that extracts the corresponding portion of the adjacent patch and the input patch is also acquired at the same time. The adjacent patch corresponds to z in the formula, and the mask corresponds to m in the formula.

[0131] Next, in step S605, the noise removal unit 403e connects the input patch and the noise patch (output patch) in the channel direction and inputs them into the neural network to generate a noise removal patch. Note that a neural network with the same configuration as that shown in FIG. 1 is used to generate the noise removal patch. The weight information is pre-trained information transmitted from the learning device 401 and stored in the storage unit 403a. In this embodiment, a default value corresponding to the number of iterations is used as the noise level used to generate the noise-removed image, but this is not limited to this. For example, a noise level schedule determined for each iteration and acquired together with the captured image may also be used. The noise removal patch corresponds to the estimated target patch y^ in the mathematical formula.

[0132] Next, in step S606, the update unit 403f generates an updated debayer patch (output patch) from the noise removal patch, adjacent patch, and mask based on equation (8). In this embodiment, the noise patch used for updating is generated from Gaussian noise, but it may also be generated using a pseudo-random number with a fixed random number seed. In this case, a value corresponding to the number of iterations of the process or a value corresponding to the number of iterations extracted from a separately prepared sequence may be used as the random number seed value. Also, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, it may be used. The updated debayer patch is the intermediate patch yt-1 of the formula, and the noise patch is the Gaussian noise ε t is equivalent to

[0133] Next, in step S607, the update unit 403f determines whether the processing is complete. If the processing is complete, the updated de-bayer patch is output to the storage unit 403a as an output patch. If the processing is not complete, the Bayer patch is used as a new output patch and the processing is repeated from step S605. In this embodiment, the number of iterations of the processing is used as a criterion for determining whether the processing is complete, but this is not limited to this. Note that information (position information) about the location where the input patch was extracted from the captured image may also be output together with the output patch, and may be used in the synthesis of the output patch, which will be described later. In addition, a default value acquired together with the captured image or the input patch is used as the upper limit of the number of iterations of the processing.

[0134] Next, in step S608, the update unit 403f determines whether processing has been completed for all input patches acquired from the bilinearly interpolated image of the captured image. If processing has been completed, the processed output patches are combined in a combining unit (not shown) to generate a de-Bayered image. If processing has not been completed, a new input patch is acquired from the bilinearly interpolated image of the captured image, and the processing is repeated from step S602. Note that patch combination in the combining unit (not shown) is the same as in the first embodiment.

[0135] Next, in step S609, the image estimation device 403 transmits the synthesized debayered image to the computer 404.

[0136] With the above configuration, according to this embodiment, patch division processing can be performed without any problems in debayer image processing using a diffusion model.

[0137] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0138] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that are capable of appropriately performing patch division processing using a diffusion model. The image processing device may be any device that has the image processing function of each embodiment, and may be realized in the form of an imaging device, a personal computer, or the like.

[0139] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

Claims

1. An image processing method comprising: a first step of obtaining an input patch to a diffusion model; a second step of initializing an output patch obtained using the diffusion model based on the input patch; a third step of generating an adjacent output patch that was output before the output patch by the diffusion model based on an adjacent input patch adjacent to the input patch; and a fourth step of iteratively updating the output patch using the diffusion model based on the input patch, the output patch, and the adjacent output patch.

2. The image processing method of claim 1, wherein in the fourth step, the output patch is iteratively updated based on an intermediate patch obtained by using the diffusion model based on the input patch and the output patch, and the corresponding portion between the intermediate patch and the adjacent output patch.

3. The image processing method according to claim 2, wherein in the fourth step, the output patch is iteratively updated based on the gradient of the corresponding portion.

4. An image processing method according to any one of claims 1 to 3, characterized in that in the fourth step, the output patch is iteratively updated based on the corresponding portions of the input patch, the output patch and the adjacent output patch.

5. The image processing method according to claim 4, wherein in the fourth step, the output patch is iteratively updated by concatenating the corresponding portions and inputting them into the diffusion model.

6. An image processing method according to any one of claims 1 to 5, characterized in that the input patch is a patch extracted from an input image and includes at least one of the following: patch size, metadata of the input image, information about coordinates when extracting the patch from the input image, information about random numbers used when processing the patch extracted from the input image, and information about the noise level of the patch extracted from the input image.

7. An image processing method according to any one of claims 1 to 6, characterized in that the input patch and the adjacent input patch are patches extracted from the input image by overlapping a portion of the input image.

8. An image processing method according to any one of claims 1 to 7, characterized in that in the first step, a plurality of input patches divided from the image are obtained, the plurality of input patches include a first input patch and a second input patch, and the second input patch has an area that overlaps with a portion of the first input patch.

9. The image processing method according to claim 8, further comprising a fifth step of performing synthesis processing based on a plurality of output patches including a first output patch corresponding to the first input patch and a second output patch corresponding to the second input patch.

10. The image processing method according to any one of claims 1 to 9, wherein in the second step, the output is initialized by using a first random number.

11. The image processing method according to any one of claims 10 to 15, wherein the first random number is a fixed random number.

12. A program for causing a computer to execute the image processing method according to any one of claims 1 to 9.

13. An image processing device comprising: an input acquisition unit that acquires an input patch to a diffusion model; an initialization unit that initializes an output patch obtained using the diffusion model based on the input patch; an output acquisition unit that generates an adjacent output patch that was output before the output patch by the diffusion model based on an adjacent input patch adjacent to the input patch; and an update unit that iteratively updates the output patch by using the diffusion model based on the input patch, the output patch, and the adjacent output patch.

14. An image processing system comprising the image processing device of claim 11 and a control device capable of communicating with the image processing device, wherein the control device has a transmitting unit that transmits a request to the image processing device to execute processing on an image related to the input patch, the image processing device has a receiving unit and an image processing unit, the receiving unit receives the request transmitted by the transmitting unit, and the image processing unit executes the processing in response to the request.

15. An image processing method comprising the steps of: acquiring an input image; and generating an estimated image by inputting the input image into a trained diffusion model, wherein the trained diffusion model is obtained by training comprising the steps of acquiring an input patch to the diffusion model; initializing an output patch obtained using the diffusion model based on the input patch; generating an adjacent output patch that was output before the output patch by the diffusion model based on an adjacent input patch adjacent to the input patch; and iteratively updating the output patch using the diffusion model based on the input patch, the output patch, and the adjacent output patch.

Images