Image processing method, image processing apparatus, image processing system, and program

Abstract

To provide an image processing method with which it is possible to appropriately performing patch division processing by using a diffusion model.SOLUTION: An image processing method includes: a first step of acquiring an input patch to a diffusion model (S201); a second step of initializing an output patch from the diffusion model (S202); a third step of acquiring an adjacent patch that is adjacent to the input patch and has already been processed with the diffusion model (S203); and a fourth step of iteratively updating the output patch by using the diffusion model on the basis of the output patch and adjacent patch (S204-S206).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, an image processing system, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for sharpening an image using a diffusion model. Non-Patent Document 1 discloses a method for reconstructing an image acquired by a thin camera using a metalens using a diffusion model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 1,1769,228 [Non-patent literature]

[0004] [Non-Patent Document 1] arXiv:2308.02797v1(Thin On-Sensor Nanophotonic Array Cameras) Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1 and Non-Patent Document 1, patch division processing cannot be performed appropriately using a diffusion model. This is because, as will be described later, image processing using conventional diffusion models involves randomness in the process. Therefore, when patch division processing is performed, texture is generated randomly within the resulting patch, resulting in a lack of texture continuity between patches, and the seams between patches become visible when the patches are combined.

[0006] Patent Document 1 does not disclose any solution to the above-mentioned problems. In Non-Patent Document 1, the results of dividing an image into patches (sub-images) are further processed by a separately prepared neural network to synthesize the patches. However, this method does not allow for patch division processing without problems using only a diffusion model, and a separate neural network is required.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing method that is capable of appropriately performing patch division processing using a diffusion model. [Means for solving the problem]

[0008] An image processing method according to one aspect of the present invention includes a first step of acquiring an input patch to a diffusion model, a second step of initializing an output patch from the diffusion model, a third step of acquiring an adjacent patch adjacent to the input patch that has already been processed by the diffusion model, and a fourth step of iteratively updating the output patch using the diffusion model based on the input patch, the output patch, and the adjacent patch.

[0009] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image processing method that is capable of appropriately performing patch division processing using a diffusion model. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram showing a flow of learning of a neural network in the first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 4 is a flowchart relating to learning of a neural network in the first embodiment. [Figure 5] 1 is a flowchart relating to generation of an output image using a neural network in the first embodiment. [Figure 6] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 7] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 8] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 9] 11 is a flowchart relating to generation of an output image using a neural network in the third embodiment. [Figure 10] FIG. 1 is an explanatory diagram of the gist of each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0013] First, before providing a detailed description of each embodiment, a diffusion model and an outline of each embodiment will be described. In each embodiment, patch division processing is performed appropriately without any adverse effects in image processing using a diffusion model. A diffusion model is a type of generative AI (artificial intelligence) primarily used for image generation. By sampling from a target image distribution conditioned by image input or text input, a high-quality target image can be generated that corresponds to the input image or text. For example, Stable Diffusion uses a diffusion model to generate a realistic image that looks like it was drawn by a human in response to text entered by a user. Note that in each embodiment, patch division processing refers to dividing a large captured image into multiple small images (patches) and processing the patches sequentially.

[0014] Diffusion models consist of two main processes: a forward diffusion process and a reverse diffusion process. The diffusion process incrementally adds Gaussian noise to a target image, slowly removing details from the target image until it becomes a pure noise image. The reverse diffusion process starts with a pure noise image, and reverses the diffusion process by conditioning it with image and text inputs and recovering it iteratively. The reverse diffusion process can be trained using a neural network (machine learning model) that takes a noisy target image and a conditioning image (or signal such as text) as inputs to remove noise. Details will be provided later.

[0015] The diffusion process is given by the following equation (1):

[0016]

number

[0017] where y t-1 and y t represents the step (iteration number) t-1 of the diffusion process q given by the Markov chain and the intermediate image of the next step t (t=1~T), respectively. N is a standard normal distribution (Gaussian distribution), α t (0<αt<1) is y at step t t-1 respectively represent the variance of the Gaussian noise to be added to

[0018] By transforming (marginalizing) equation (1), the intermediate image y given the target image y0 is t The distribution of is given by the following equation (2):

[0019]

number

[0020] where γ t is added to the target image y0 and tIn other words, equation (2) expresses the variance of the Gaussian noise to obtain the intermediate image y t-1 Gaussian noise is added to the intermediate image y t By adding Gaussian noise to the target image y0, the intermediate image y t This indicates that it is possible to obtain

[0021] Further modifying equations (1) and (2) gives the target image y0 and the intermediate image y t Given an intermediate image y t-1 The posterior distribution of is given by the following equation (3):

[0022]

number

[0023] where μ and σ 2 is the target image y0 and the intermediate image y t The mean and variance of the standard normal distribution, which is the posterior distribution of the intermediate image yt-1 when t and the mean μ obtained from the target image y0, and the variance σ 2 This means that the intermediate image yt-1 at the t-1th step can be estimated by adding Gaussian noise with

[0024] The despreading process is given by the following equation (4).

[0025]

number

[0026] Here, the inverse diffusion process p(t=T~1) is defined as an inverse Markov chain that proceeds in the reverse direction of the diffusion process q. Note that, following the diffusion process q, the pure noise image y TWe start with Gaussian noise sampled from a standard normal distribution (the second equation in equation (4)). In addition, each step of the de-diffusion process (the third equation in equation (4)) can be defined as a trainable isotropic Gaussian conditional distribution if the variance of the Gaussian noise added at each step of the diffusion process is set small.

[0027] The mean μ of the Gaussian distribution at each step of the dediffusion process θ is calculated from the posterior distribution of Equation (3), but the target image y0 is required. However, the pure noise image y T In the despreading process starting from , the target image y0 is generally unknown.

[0028] Therefore, in the diffusion model, a neural network (machine learning model) is trained to remove noise from a noise image in which Gaussian noise with various variances is added to the target image y0, and then the intermediate image y t The noise-removed target image y0^ is estimated from

[0029] The target image y0^ after noise removal in the de-diffusion process is given by the following equation (5).

[0030]

number

[0031] Here, f is a noise removal neural network with θ as a parameter, and x is a condition used for noise removal. As a condition used for noise removal, for example, a vector with embedded text is used in Stable Diffusion, and an interpolated image of a low-resolution image is used in Patent Document 1. Also, in Patent Document 1, an intermediate image y t The interpolated image is connected in the channel direction and input to the neural network, and noise is removed by conditioning the interpolated image. In addition, the noise removal neural network is also fed with the noise level (noise strength) γ tis introduced and noise is removed according to the noise level. Note that in each embodiment, the variance of the Gaussian noise added in the diffusion process is used as the noise level, but this is not limiting. A method for inputting the noise level into the noise removal neural network will be described later.

[0032] By substituting the estimated target image y0^ into equation (3), the mean of the posterior distribution is given parametrically by the following equation (6).

[0033]

number

[0034] The variance of the Gaussian noise is the variance of the Gaussian noise added in the diffusion process, α t Using this, each step of the despreading process is given by the following equation (7).

[0035]

number

[0036] By repeating this procedure from t=T to 1, we obtain a pure noise image y T The target image y0 can be generated from (more precisely, sampled from the target image distribution conditioned on x). The diffusion model is also called the Denoised Diffusion Probabilistic Model (DDPM).

[0037] In the prior art such as Patent Document 1, the intermediate image y t The estimated target image y0^ is then converted to the intermediate image y t-1 However, this does not allow the Gaussian noise (standard normal random number) ε t Therefore, when dividing the patch into parts using a conventional diffusion model, the texture is generated randomly for each patch, and there is no texture continuity between patches. As a result, when the patches are combined, the seams between the patches become visible.

[0038] Therefore, in each embodiment, instead of formula (7), the intermediate patch y t The estimated target patch y0^ is then t-1 It should be noted that equation (8) assumes that a large original image I is divided into multiple small patches x, which are then used as conditioning patches x and sequentially processed using a diffusion model (patch division processing). Furthermore, equation (8) assumes that when dividing the original image I into patches, there is some overlap between adjacent patches (corresponding parts exist).

[0039]

number

[0040] where z is the processed neighboring patch obtained by processing the conditioned patch x and the patch extracted from the neighboring location with the diffusion model. m is the neighboring patch z and the intermediate patch y t ∇ is a mask that extracts the corresponding parts of the adjacent patch z and the intermediate patch y t In other words, in addition to approaching the target patch distribution conditioned by patch x at each step of the dediffusion process given by Equation (7), the processed neighbor patch z and the intermediate patch y t Constraints are added to each step of the de-diffusion process so that the corresponding parts of

[0041] Depending on the location of the patch x extracted from the original image I, there may be multiple adjacent patches z that have been processed using the diffusion model. In that case, the intermediate image y is created using multiple masks m from the multiple adjacent patches z. tand execute equation (8). The gradient may be calculated from the adjacent patch z and the target patch y0^, or from the adjacent patch z and the mean μθ of the posterior distribution given by equation (6). The calculated gradient may also be weighted. Depending on the location and order in which patch x is extracted from the original image I, there may be no adjacent patch z. In that case, equation (7) can be used as usual.

[0042] In each example, the intermediate patch y t and the corresponding parts of adjacent patches z processed by the diffusion model gradually approach each other. Therefore, in the final patch y0 obtained by repeating the inverse diffusion process (t=T→1), the corresponding parts of the adjacent patches processed by the diffusion model ideally match. As a result, even when the patches resulting from processing in each embodiment are combined, no seams are created between the adjacent patches that have already been processed, and patch division processing can be performed using the diffusion model without any problems.

[0043] Next, an example of upscaling (super-resolution) using patch division processing with a diffusion model will be described with reference to Fig. 10. Here, upscaling refers to image processing that generates an enlarged high-resolution image by estimating high-frequency components that cannot be expressed in a low-resolution image.

[0044] In each embodiment, a low-resolution captured image is first acquired, and input patches (corresponding to x in the formula) are extracted from the bicubic-interpolated image (corresponding to I in the formula), and sequential processing (patch division processing) is performed using a diffusion model. When extracting input patches from the bicubic-interpolated image, each patch is acquired so that it partially overlaps with adjacent patches (having overlapping areas (common areas)). In FIG. 10, the bicubic-interpolated image of the captured image is divided into four input patches so that it partially overlaps with adjacent patches. Therefore, for example, the input patch at the bottom right has parts corresponding to the input patches at the top left, top right, and bottom left. Also, FIG. 10 shows an example of sequential processing, starting with the input patch at the top left and processing the input patch at the bottom right last. The dotted arrows in the figure indicate the order of sequential processing.

[0045] Next, the input patch is processed according to equation (8). Note that Fig. 10 shows an example in which the bottom right input patch is processed. Therefore, it is assumed that the other input patches (top left, top right, bottom left) have already been processed by the diffusion model.

[0046] In equation (8), we first calculate the current intermediate patch (output patch, y T (corresponding to ) is initialized with Gaussian noise. Note that the size of the noise patch is the same as that of the interpolated patch.

[0047] Next, the input patch is input to the neural network along with the current intermediate patch for noise removal. Information about the variance of the noise patch (corresponding to γT in the formula), which is the noise level, is also input to the neural network. The input patch is also connected to the current intermediate patch in the channel direction and input to the neural network, where it is used as a condition for noise removal. The variance of the noise patch uses a value determined in advance for each iteration of the dediffusion process. For example, the maximum and minimum values ​​of the variance of the Gaussian noise of the noise patch can be determined, and a value determined linearly from the number of iterations can be used. The neural network is also trained in advance to be able to perform noise removal as given by equation (5).

[0048] Next, the mean of the posterior distribution (corresponding to μθ in the formula) given by formula (5) is calculated from the noise-removed patch (corresponding to y0^ in the formula).

[0049] Next, the gradient given by Equation (8) is calculated from the noise removal patch and the adjacent patch (corresponding to z in the formula). The adjacent patches are output patches that are the processing results of the upper left, upper right, and lower left input patches that have already been processed by the diffusion model. In FIG. 10, a mask (corresponding to m in the formula) (not shown) is used to extract and combine portions of the three adjacent output patches that correspond to the currently processed input patch, and this is the adjacent patch. Technically, this is not an adjacent patch but a combined patch, as mentioned above, but for ease of understanding, it is referred to as such in FIG. 10. Furthermore, as explained with reference to Equation (8), the gradient may be calculated using the current intermediate patch or the average of the posterior distribution instead of the noise removal patch.

[0050] Next, the intermediate patch is updated using the calculated gradient using equation (8) (the output patch, y t-1 equivalent to).

[0051] The procedure according to the above formula (8) is repeated a predetermined number of times for each input patch, and is sequentially performed on multiple input patches extracted from the entire surface of the bicubic interpolated image. The resulting multiple upscaled output patches are then synthesized. Therefore, according to each embodiment, an upscaled image that has been subjected to patch division processing using a diffusion model can be generated without any drawbacks associated with the patch division processing.

[0052] In conventional diffusion models, including Patent Document 1, when patch division processing is performed, seams occur between adjacent processed patches when the patches are combined, for the reasons described above. Therefore, in order to process large images without any problems using conventional diffusion models, it was necessary to process them all at once. However, processing large images all at once requires expensive GPUs (Graphics Processing Units) with large memory, which places a heavy financial burden on users. In contrast, each embodiment can perform patch division processing using a diffusion model without any problems, allowing large images to be processed even with an inexpensive GPU, thereby reducing the financial burden on users.

[0053] In Non-Patent Document 1, patches processed by a diffusion model are processed by a separately prepared neural network to perform patch synthesis. On the other hand, in each embodiment, by devising the diffusion model expressed by equation (8), it is not necessary to prepare a separate neural network, and therefore the processing time can be shortened.

[0054] The image processing method described above is an example, and the embodiments are not limited to this. Details of other image processing methods will be described in the following embodiments.

[0055] [Example 1] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, a low-resolution captured image is divided into patches, and the resulting low-resolution patches are upscaled using a diffusion model to generate high-resolution patches, which are then trained and executed.

[0056] Fig. 2 is a block diagram of an image processing system 100 in this embodiment. Fig. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device 103, a display device 104, a recording medium 105, an input device 106, an output device 107, and a network 108.

[0057] The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and a learning unit 101d.

[0058] The imaging device 102 includes an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives an optical image of the subject formed via the optical system 102a to acquire a captured image (low-resolution image). The imaging element 102b is, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. Information regarding the shooting conditions of the captured image (such as the pixel pitch of the imaging element 102b, the type of optical low-pass filter, and ISO sensitivity) can be acquired along with the image. Development conditions for the captured image (such as noise reduction strength, sharpness strength, and image compression rate) can also be acquired along with the image. This information acquired along with the image can also be transmitted to an input acquisition unit 103b of the image estimation device 103 (described later) along with the image. A storage unit for storing the acquired image, a display unit for displaying it, a transmission unit for transmitting it to an external device, an output unit for storing it in an external storage medium, and the like are not shown. A control unit that controls each unit of the imaging device 102 is also not shown.

[0059] The image estimation device 103 includes a storage unit 103a, an input acquisition unit 103b, an initialization unit 103c, an output acquisition unit 103d, a noise reduction unit 103e, and an update unit 103f. The input acquisition unit 103b acquires a low-resolution patch (input patch) from the captured image (more precisely, a bicubic interpolated image of the captured image). Then, the initialization unit 103c initializes the current intermediate patch (output patch) with Gaussian noise. The noise reduction unit 103e estimates a noise-removed patch from the current intermediate patch using a neural network conditioned on the low-resolution patch. The output acquisition unit 103d acquires a patch (adjacent patch) extracted from a location adjacent to the low-resolution patch in the captured image and already processed using a diffusion model. The update unit 103f updates the current intermediate patch from the noise-removed patch and adjacent patch. This procedure is repeated to perform image processing that ultimately generates an upscaled high-resolution patch (output patch). Details of the method for initializing the current intermediate patch in the initialization unit, the method for removing noise from the current intermediate patch using a neural network in the noise removal unit, and the method for updating the current intermediate patch from the noise removal patch and adjacent patches in the update unit will be described later.

[0060] The low-resolution patches may be patches extracted from an image captured by the image capturing device 102, or may be extracted from an image stored in the recording medium 105. Alternatively, patches that have already been divided from an image and stored in the recording medium 105 may be used.

[0061] The weight information of the neural network used for noise removal is read from the storage unit 103a. The weights are learned by the learning device 101, and the image estimation device 103 reads the weight information from the storage unit 101a via the network 108 in advance and stores it in the storage unit 103a. The stored weight information may be the weight's numerical value itself or may be in an encoded format. Details of the neural network weight learning method and the image processing method using a diffusion model that uses a neural network will be described later.

[0062] The upscaled high-resolution patches are synthesized and then output to at least one of a display device 104, a recording medium 105, and an output device 107. The display device 104 is, for example, an LCD display or a projector. A user can check the patches being processed via the display device 104 and perform image editing work via the input device 106. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The input device 106 is, for example, a keyboard or a mouse, etc. The output device 107 is, for example, a printer, etc.

[0063] Next, a weight learning method (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Fig. 1 and Fig. 4. Fig. 1 is a diagram showing the flow of learning weights of a neural network (machine learning model). Fig. 4 is a flowchart related to weight learning. Each step in Fig. 4 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the learning unit 101d.

[0064] First, in step S101, the acquisition unit 101b acquires a low-resolution patch (degraded patch) 201 and a corresponding high-resolution patch (correct patch) 200, as shown in FIG. 1. In this embodiment, a patch is a small image having a predetermined number of pixels. For example, a low-resolution patch may be 128 x 128 x 3 pixels (corresponding to the number of pixels in the vertical x horizontal x channel directions), and the corresponding high-resolution patch may be 256 x 256 x 3 pixels. In this case, since the vertical and horizontal sizes are doubled, the upscaling factor is 2 (enlarged by 4 times in terms of the number of pixels). Note that the upscaling factor is not limited to 2, and may be any factor as long as a high-resolution patch corresponding to the low-resolution patch can be acquired.

[0065] Furthermore, a low-resolution patch and a corresponding high-resolution patch may be obtained by photographing the same subject with optical systems having different focal lengths and cutting out corresponding portions of the two images obtained. Alternatively, a high-resolution patch may be downsampled to generate a corresponding low-resolution patch. Alternatively, a corresponding high-resolution patch that is less affected by (aberration, diffraction) an equivalent low-resolution patch captured by the image capture device 102 and an optical system 102a may be generated by numerical calculation. In this embodiment, a low-resolution patch and a corresponding high-resolution patch are generated by numerical calculation, but the present invention is not limited to this.

[0066] Next, in step S102, the generation unit 101c generates an interpolated patch 202 by interpolating and enlarging the low-resolution patch 201 to have the same number of pixels as the corresponding high-resolution patch 200. In this embodiment, bicubic interpolation is used as the interpolation method, but the present invention is not limited to this, and nearest neighbor interpolation or bilinear interpolation may also be used.

[0067] Next, in step S103, the generating unit 101c generates a noise patch 203 by adding noise to the high-resolution patch 200.

[0068] Next, in step S104, the learning unit 101d uses a neural network to generate a noise-removed patch 204 by removing noise from the noise patch 203. The noise-removed patch 204 is generated based on the interpolated patch 202, the noise patch 203, and information about the intensity (noise level) of the noise added to the noise patch 203. Note that the noise-removed patch 204 is an estimate of the high-resolution patch 200, and ideally, the two will match. In this embodiment, the variance of the added Gaussian noise is used as the noise level, but this is not limited to this. Furthermore, noise removal is performed according to the intensity of the noise added to the noise patch 203, but a method for inputting the noise level (variance of Gaussian noise) to the neural network will be described later. In this embodiment, the interpolated patch 202 and the noise patch 203 are concatenated in the channel direction and input to the neural network, but this is not limited to this.

[0069] Next, in step S105, the learning unit 101d updates the weights of the neural network based on the error between the high-resolution patch 200 and its estimated denoising patch 204 (performs neural network learning). Here, the weights include the filter components and biases of each layer. In this embodiment, back propagation is used to update the weights, but this is not limitative. In mini-batch learning, the error between the high-resolution patch 200 and its corresponding denoising patch 204 is calculated, and the weights are updated. For example, the L2 norm or the L1 norm may be used as the loss function. The weight update method (learning method) is not limited to mini-batch learning, and batch learning or online learning may also be used.

[0070] Next, in step S106, the learning unit 101d determines whether learning (weight update) is complete. Completion can be determined by, for example, whether the number of iterations of learning (weight update) has reached a specified value, or whether the amount of change in weight during update is smaller than a specified value. If it is determined that learning is incomplete, the process returns to step S101, and multiple new low-resolution patches 201 and corresponding high-resolution patches 200 are obtained. On the other hand, if it is determined that learning is complete, the learning device 101 ends learning and saves weight information in the storage unit 101a.

[0071] In this embodiment, a method in the image domain has been described as a method for learning the weights of the neural network, but the present invention is not limited to this. Learning may also be performed in the feature map (latent) domain from the low-resolution patch 201 or the interpolated patch 202. In this case, learning can be performed using a feature map obtained using a separately prepared neural network and a noise patch 203 obtained by adding noise to a feature map obtained from the high-resolution patch 200 using a separately prepared neural network.

[0072] In this embodiment, the neural network configuration shown in FIG. 1 is used, but the present invention is not limited to this.

[0073] CN in Figure 1 represents a convolutional layer. CN calculates the sum of the input, filter convolution, and bias, and then performs nonlinear transformation on the result using an activation function. The initial values ​​of each filter component and bias are arbitrary and are determined by random numbers in this embodiment. The activation function can be, for example, a ReLU (Rectified Linear Unit) or a sigmoid function. The multidimensional array output from each layer except the final layer is a feature map. A feature map is generally a four-dimensional array with dimensions of batch, length, width, and channel. The skip connection 205 combines feature maps output from discontinuous layers. Feature maps can be combined by taking the sum of each element, or by concatenating in the channel direction. In this embodiment, the sum of each element is used.

[0074] The elements (blocks or modules) within the dotted line frame in FIG. 1 represent residual blocks. A network with multiple layers of residual blocks is called a residual network and is widely used in image processing using machine learning models. However, this embodiment is not limited to this, and a network may be configured with multiple layers of other elements. For example, an inception module may be used, which juxtaposes convolution layers with different convolution filter sizes and integrates the resulting multiple feature maps to form a final feature map. Alternatively, a dense block with dense skip connections may be used.

[0075] The vertical and horizontal sizes of the feature map are also enlarged or reduced within the network. Here, pooling, stride convolution, etc. can be used to reduce the feature map. Furthermore, deconvolution (or transposed convolution), pixel shuffle, interpolation, etc. can be used to expand the feature map. Although not shown in the figure in this embodiment, stride convolution is used to reduce the feature map, and nearest neighbor interpolation is used to expand it.

[0076] Furthermore, a noise level (γ in the figure) is input to the residual block, and noise is removed from the noise patch according to the noise level. In this embodiment, a vector with noise level information embedded is generated using a separately prepared multilayer perceptron (MLP in the figure) based on the noise level. Then, based on this, the feature map obtained between the two convolution layers of the residual block is scaled (an affine transformation using a gain to be multiplied and a bias to be added to the feature map). However, this embodiment is not limited to this. A multilayer perceptron is a type of neural network. Furthermore, since the noise level of the noise image differs with each iteration in the dediffusion process, a different noise level γt is input to the residual block for each iteration.

[0077] Next, generation of an upscaled image using a diffusion model executed by the image estimation device 103 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart related to generation of an upscaled image using a diffusion model. Each step in Fig. 5 is mainly executed by the input acquisition unit 103b, initialization unit 103c, output acquisition unit 103d, noise removal unit 103e, or update unit 103f of the image estimation device 103.

[0078] First, in step S201, the input acquisition unit 103b acquires input patches for the diffusion model from a bicubic-interpolated image of the captured image. The captured image is a low-resolution image, as in the learning process. In this embodiment, the captured image is transmitted from the imaging device 102, but this is not limiting. Note that image information may also be acquired along with the captured image and used in the steps described below. Also, as in the learning process, the input patches are acquired after an interpolated image (not shown) is generated by an interpolation unit (not shown) that performs bicubic interpolation on the captured image. Alternatively, the input patches may be bicubic-interpolated based on the image size acquired along with the captured image. Furthermore, multiple input patches are acquired from the captured image, and sequentially processed (patch division process) using the diffusion model in the steps described below. During patch division, adjacent patches have overlapping areas (corresponding portions). That is, each input patch has an overlapping area (common area, corresponding portion) that overlaps (is common to) a portion of another input patch. Furthermore, information (position information) about the location where the input patch is extracted from the captured image and the patch size, acquired along with the captured image, may be used to acquire the input patches.

[0079] In this embodiment, acquisition of an input patch starts from the upper left corner of the bicubic interpolated image of the captured image. Then, a shift is made to the right to acquire the next input patch. When the upper right corner is reached, a shift is made downward from the starting point to acquire an input patch, and another shift is made to the right to acquire the next input patch. This step is repeated to acquire multiple input patches from the entire captured image in the order in which horizontally written text is read. However, this embodiment is not limited to this. Also, depending on the location where the input patch is acquired, it may extend beyond the captured image. In that case, the shift amount or the overlap amount with adjacent patches may be adjusted. Furthermore, the input patch corresponds to the conditioning patch x of the formula, and the captured image (more precisely, its bicubic interpolated image) corresponds to the original image I of the formula.

[0080] Next, in step S202, the initialization unit 103c generates a noise patch (output patch) of the same size as the input patch from Gaussian noise (initializes the output patch). The noise patch may be generated using a pseudo-random number with a fixed random number seed. In this case, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, this may be used. The noise patch may also be generated by generating a pure noise image y T is equivalent to

[0081] Next, in step S203, the output acquisition unit 103d acquires a patch (adjacent patch) that has been acquired from a location adjacent to the input patch in the captured image and has already been processed using the diffusion model. Note that the acquisition of the adjacent patch may use information (position information) about the location where the input patch is extracted from the captured image, which information was acquired together with the captured image and the input patch. The output acquisition unit 103d also simultaneously acquires a mask that extracts the corresponding portion between the adjacent patch and the input patch. The adjacent patch corresponds to z in the formula, and the mask corresponds to m in the formula.

[0082] Next, in step S204, the noise removal unit 103e connects the input patch and the noise patch (output patch) in the channel direction and inputs them into the neural network to generate a noise removal patch. A neural network with the same configuration as that shown in FIG. 1 is used to generate the noise removal patch. The weight information is pre-trained information transmitted from the learning device 101 and stored in the memory unit 103a. In this embodiment, a predetermined value according to the number of iterations is used as the noise level used to generate the noise-removed image, but this is not limited to this. For example, a schedule of noise levels determined for each iteration and acquired together with the captured image may also be used. The noise removal patch corresponds to the estimated target patch y^ in the mathematical formula.

[0083] Next, in step S205, the update unit 103f generates an updated upscaled patch (output patch) using the noise removal patch, the adjacent patch, and the mask based on equation (8). In this embodiment, the noise patch used for updating is generated from Gaussian noise, but it may also be generated using a pseudo-random number with a fixed random number seed. In this case, a value corresponding to the number of iterations of the process or a value corresponding to the number of iterations extracted from a separately prepared sequence may be used as the value of the random number seed. Also, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, it may be used. The updated upscaled patch is generated by using the intermediate patch y t-1 , the noise patch is Gaussian noise ε t is equivalent to

[0084] Next, in step S206, the update unit 103f determines whether the processing is complete. If the processing is complete, the updated upscaled patch is output as an output patch to the storage unit 103a, the recording medium 105, or the like. If the processing is not complete, the upscaled patch is used as a new output and the processing is repeated from step S204. In this embodiment, the number of iterations of the processing is used as a criterion for determining whether the processing is complete, but this is not limiting. Note that information (position information) about the location where the input patch was extracted from the captured image may also be output together with the output patch, and may be used in the synthesis of the output patch, which will be described later. Furthermore, the upper limit of the number of iterations of the processing may be, for example, a predetermined value acquired together with the captured image or the input patch.

[0085] Next, in step S207, the update unit 103f determines whether processing is complete for all input patches acquired from the bicubic interpolated image of the captured image. If processing is complete, the processed output patches are composited by a composite unit (not shown) and output as an upscaled image to at least one of the display device 104, the recording medium 105, and the output device 107. If processing is not complete, a new input patch is acquired from the bicubic interpolated image of the captured image, and the processing is repeated from step S201.

[0086] In this way, the update unit 103f iteratively updates the output patch by using a diffusion model based on the input patch, the output patch, and the adjacent patch. Preferably, the update unit 103f iteratively updates the output patch based on an intermediate patch obtained by using a diffusion model based on the input patch and the output patch, and a corresponding portion between the intermediate patch and the adjacent patch. More preferably, the update unit 103f iteratively updates the output patch based on the gradient of the corresponding portion.

[0087] Preferably, the input acquisition unit 103b acquires a plurality of input patches divided from the image. The plurality of input patches includes a first input patch and a second input patch, and the second input patch has an area that overlaps with a portion of the first input patch. More preferably, a synthesis unit (not shown) provided in the image estimation device 103 performs synthesis processing based on a plurality of output patches including a first output patch corresponding to the first input patch and a second output patch corresponding to the second input patch. Note that the synthesis unit (not shown) may synthesize the processed output patch by cutting (trimming) the processed output patch so that there is no overlap between the processed output patch and an adjacent processed output patch. Alternatively, the synthesis unit (not shown) may synthesize the processed output patch by weighted averaging the overlap between the processed output patch and an adjacent processed output patch. In this embodiment, patch synthesis is performed by trimming, but this is not limited to this. Furthermore, information (position information) of the location where the input patch was extracted from the captured image is used to synthesize the output patch, but this is not limited to this. Note that the update unit 103f may also function as the synthesis unit.

[0088] In this embodiment, a method for generating an upscaled image using a diffusion model has been described in the image domain, but the present invention is not limited to this. The method may be performed in the feature map (latent) domain using a feature map obtained using a separately prepared neural network for an input patch extracted from a bicubic interpolated image of a captured image. In this case, the initialization unit 103c generates a noise patch (output patch) of the same size as the feature map obtained from the input patch, and the output acquisition unit 103d generates a feature map for the adjacent patch using a separately prepared neural network. The noise removal unit 103e removes noise in the latent domain from the feature map and noise patch obtained from the input patch and the adjacent patch using a neural network trained in the latent domain.

[0089] In this embodiment, a method for estimating the upscaled image itself has been described as a method for generating an upscaled image using a diffusion model, but the present invention is not limited to this. For example, it is also possible to estimate only the residual component of the upscaled image and the bicubic interpolated image of the captured image.

[0090] In addition, in this embodiment, it is assumed that the captured image to be used for generating an upscaled image using the diffusion model is a color image, but the present invention is not limited to this. For example, an upscaled luminance image may be generated using a monochrome luminance image extracted from a captured color image.

[0091] In this embodiment, the input patch is a patch extracted from an image that has not been processed by the diffusion model, and may include at least one of information about the patch size or metadata of the image, information about the coordinates used to extract the patch from the image, information about a random number used to process the patch extracted from the image, and information about the noise level of the patch extracted from the image.

[0092] In this embodiment, the adjacent patch is a patch that overlaps with a part of the input patch, is extracted from the image, and is processed by the diffusion model.

[0093] With the above configuration, according to this embodiment, patch division processing can be performed without any problems in upscale image processing using a diffusion model.

[0094] [Example 2] Next, an image processing system according to a second embodiment of the present invention will be described. In this embodiment, a blurred captured image is divided into patches, and the obtained blurred patches are deblurred using a diffusion model. The image processing system of this embodiment differs from the first embodiment in that an imaging device acquires a captured image (blurred image) and performs deblurring image processing.

[0095] Fig. 6 is a block diagram of an image processing system 300 in this embodiment. Fig. 7 is an external view of the image processing system 300. The image processing system 300 includes a learning device 301 and an imaging device 302 connected via a network 303. Note that the learning device 301 and the imaging device 302 do not need to be constantly connected via the network 303.

[0096] The learning device 301 includes a storage unit 311, an acquisition unit 312, a generation unit 313, and a learning unit 314. These units are used to learn a neural network (update weights) for image processing that generates deblurred patches from blurred patches using a diffusion model.

[0097] The imaging device 302 captures an image of a subject space to obtain a captured image (blurred image), and generates a blurred image from the captured image. Details of the image processing executed by the imaging device 302 will be described later. The imaging device 302 has an optical system 321 and an image sensor 322. The image estimation unit 323 has an input acquisition unit 323a, an initialization unit 323b, an output acquisition unit 323c, a noise removal unit 323d, and an update unit 323e.

[0098] Regarding the neural network learning (weight update) executed by the learning device 301, only the differences from the first embodiment will be described with reference to FIG.

[0099] First, in step S101, the acquisition unit 312 acquires a sharp patch (correct patch) with little blur corresponding to a blurred patch (degraded patch).

[0100] Next, in step S102, the generation unit 313 generates a noise patch by adding noise to the sharp patch.

[0101] Next, in step S103, the generation unit 313 generates an adjacent patch by cutting out a part of the sharp patch. Here, the adjacent patch corresponds to z in the formula (the processed adjacent patch obtained by processing the patch extracted from the location adjacent to the conditioned patch x with the diffusion model). That is, this embodiment is characterized in that the adjacent patch is taken into consideration from the time of learning the weights of the neural network and is used to remove noise from the noise patch.

[0102] Next, in step S104, the learning unit 314 uses a neural network to generate a patch obtained by removing noise from the blurred patch, the noise patch, the neighboring patch, and information about the strength of the noise (noise level) added to the noise patch. Note that the denoised patch is an estimate of the sharp patch, and ideally the two will match.

[0103] Furthermore, by removing noise from a noise patch while taking into account adjacent patches, the corresponding portions of the adjacent patch and the denoised patch ideally coincide. As a result, when a patch that has undergone deblurring image processing and the adjacent patch are combined using a diffusion model (described later), no seams are generated between the patches, and patch division processing can be performed without any problems. Note that in this embodiment, the blurred patch, the noise patch, and the adjacent patch are concatenated in the channel direction and input to the neural network, but this is not limited to this.

[0104] The subsequent steps S105 and S106 are the same as those in the first embodiment, and therefore their explanation will be omitted.

[0105] Next, details of the image processing executed by the imaging device 302 will be described. Neural network weight information is learned in advance by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads the weight information from the storage unit 311 via the network 303 and stores it in the storage unit 324. The image estimation unit 323 generates deblurred patches from blurred patches obtained by dividing the blurred captured image into patches in the image estimation unit 323. The deblurred patches can be generated using trained neural network weight information stored in the storage unit 324 and the blurred captured image and its image information acquired by the input acquisition unit 323a. The generated deblurred patches are combined into a deblurred image, which is then stored in the recording medium 325a. When a user issues an instruction to display the deblurred image via the input unit 326, the stored image is read out and displayed on the display unit 325b. The captured image and its image information stored in the recording medium 325a may be read out, and the blur-removed image may be generated by the image estimation unit 323. The above series of controls are performed by the system controller 327.

[0106] Next, generation of a deblurred image using a diffusion model executed by the image estimation unit 323 in this embodiment will be described. The image processing procedure of this embodiment is substantially the same as that of the first embodiment shown in Fig. 5, and therefore the image processing of this embodiment will be described with reference to Fig. 5. Each step of the image processing is mainly executed by the input acquisition unit 323a, initialization unit 323b, output acquisition unit 323c, noise removal unit 323d, or update unit 323e of the image estimation unit 323.

[0107] First, in step S201, the input acquisition unit 323a acquires a captured image (blurred image). The captured image is a blurred image, similar to that used in learning. In this embodiment, the captured image is acquired by the imaging device 302 and stored in the storage unit 324, but this is not limited to this. Note that image information may also be acquired along with the captured image and used in the steps described below. The input acquisition unit 323a also acquires multiple input patches from the captured image and performs sequential processing (patch division processing) using a diffusion model in the steps described below. When dividing the patches, overlapping portions (corresponding portions) are provided between adjacent patches. Furthermore, information (position information) about the location where the input patch is extracted from the captured image and the patch size, which are acquired together with the captured image, may be used to acquire the input patches. In this embodiment, multiple input patches are acquired randomly so as to cover the entire surface of the captured image, but this is not limited to this. The input patch corresponds to a conditioning patch x of the formula, and the captured image corresponds to the original image I of the formula.

[0108] Next, in step S202, the initialization unit 323b generates a noise patch (output patch) of the same size as the input patch from Gaussian noise. Note that the noise patch may be generated using pseudo-random numbers with a fixed random number seed, as in the first embodiment. In this case, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, this may be used, as in the first embodiment. Also, the noise patch corresponds to a pure noise image yT in the formula.

[0109] Next, in step S203, the output acquisition unit 323c acquires a patch (adjacent patch) that has been acquired from a location adjacent to the input patch in the captured image and has already been processed using the diffusion model. Note that the acquisition of the adjacent patch may use information (position information) about the location where the input patch is extracted from the captured image, which information was acquired together with the captured image and the input patch. The output acquisition unit 323c also simultaneously acquires a mask that extracts the corresponding portion of the adjacent patch and the input patch. The adjacent patch corresponds to z in the formula, and the mask corresponds to m in the formula.

[0110] Next, in step S204, the noise removal unit 323d connects the input patch, the noise patch (output patch), and the adjacent patch in the channel direction and inputs them to the neural network to generate a noise removal patch. To generate the noise removal patch, a neural network with a configuration substantially similar to that shown in FIG. 1 is used. The difference from FIG. 1 is the number of images input to the neural network; whereas in FIG. 1, there are two images (202 and 203), in this embodiment, three images are input by adding adjacent patches. Furthermore, the weight information is learned information transmitted from the learning device 301 and stored in the memory unit 324.

[0111] In this embodiment, as in the first embodiment, a predetermined value corresponding to the number of iterations is used as the noise level used when generating the noise removal patch, but this is not limited to this. For example, a schedule of noise levels determined for each iteration acquired together with the captured image may be used. Furthermore, the adjacent patch is used by extracting the portion corresponding to the input patch using a mask. Furthermore, the noise removal patch corresponds to the estimated y^ of the target patch in the mathematical formula.

[0112] Next, in step S205, the update unit 323e generates an updated deblurring patch (output patch) from the noise removal patch based on equation (7). In this embodiment, the noise patch used for updating is generated from Gaussian noise, but it may also be generated using a pseudo-random number with a fixed random number seed. In this case, a value corresponding to the number of iterations of the process or a value corresponding to the number of iterations extracted from a separately prepared sequence may be used as the random number seed value. If a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, this may also be used. Furthermore, since equation (7) represents the dediffusion process in a conventional diffusion model, the input of the neural network fθ does not include the adjacent patch z. However, in this embodiment, the input includes the adjacent patch z. The neural network learning and the method of inputting the adjacent patch z are as described above. The updated deblurring patch is the intermediate patch yt-1 of the formula, and the noise patch is the Gaussian noise ε t is equivalent to

[0113] Next, in step S206, the update unit 323e determines whether the processing is complete. If the processing is complete, the updated deblurring patch is output to the recording medium 325a as an output patch. If the processing is not complete, the processing is repeated from step S204 using the deblurring patch as a new output patch. In this embodiment, the number of iterations of the processing is used as a criterion for determining whether the processing is complete, but the present invention is not limited to this. Note that information (position information) about the location where the input patch was extracted from the captured image may also be output together with the output patch, and may be used in the synthesis of the output patch, which will be described later. Furthermore, a default value acquired together with the captured image or the input patch may be used as the upper limit of the number of iterations of the processing.

[0114] Next, in step S207, the update unit 323e determines whether processing has been completed for all input patches acquired from the captured image. If processing has been completed, the processed output patches are composited by a composite unit (not shown) and output as a deblurred image to at least one of the recording medium 325a or the display unit 325b, for example. If processing has not been completed, a new input patch is acquired from the captured image, and the processing is repeated from step S201. Note that patch composition by the composite unit (not shown) is the same as in the first embodiment.

[0115] As described above, in this embodiment, the update unit 323e iteratively updates the output patch based on the corresponding portions among the input patch, the output patch, and the adjacent patch. Preferably, the update unit 323e iteratively updates the output patch by linking the corresponding portions among the input patch, the output patch, and the adjacent patch and inputting the result into the diffusion model.

[0116] With the above configuration, according to this embodiment, patch division processing can be performed without any problems in blur removal image processing using a diffusion model.

[0117] [Example 3] Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment transmits a captured image (Bayer image) to be processed to an image estimation device. The image processing system of this embodiment also includes a processing device (computer) that processes the Bayer patches, which have been divided into patches, using a diffusion model and receives an image (deBayer image) obtained by combining the resulting deBayer patches from the image estimation device. This embodiment differs from the first and second embodiments in these respects. Note that a Bayer image is raw image data acquired by an image sensor with a Bayer array in which the color filter has a checkerboard color pattern. In a Bayer image, each pixel has information on only one of the RGB colors. Therefore, to obtain a full-color image in which each pixel has information on all three RGB colors, interpolation processing must be performed using adjacent pixels. This image processing is called deBayering (or demosaicing).

[0118] 8 is a block diagram of an image processing system 400 in this embodiment. The image processing system 400 includes a learning device 401, an imaging device 402, an image estimation device 403, and a computer (control device) 404. The learning device 401 and the image estimation device 403 are, for example, servers. The computer 404 is, for example, a user terminal (a personal computer or a smartphone). The computer 404 is connected to the image estimation device 403 via a network 405. The image estimation device 403 is connected to the learning device 401 via a network 406. That is, the computer 404 and the image estimation device 403 are configured to be able to communicate with each other, and the image estimation device 403 and the learning device 401 are configured to be able to communicate with each other.

[0119] The configuration of the learning device 401 is similar to that of the learning device 101 in the first embodiment, and therefore a description thereof will be omitted. The learning of the neural network executed by the learning device 401 is almost similar to that in the first embodiment, except that Bayer patches are used instead of low-resolution patches as degraded patches, and DeBayer patches are used instead of high-resolution patches as corresponding correct patches, and therefore a description thereof will be omitted. The configuration of the imaging device 402 is similar to that of the imaging device 102 in the first embodiment, and therefore a description thereof will be omitted.

[0120] The image estimation device 403 includes a storage unit 403a, an input acquisition unit 403b, an initialization unit 403c, an output acquisition unit 403d, a noise removal unit 403e, an update unit 403f, and a communication unit 403g. The storage unit 403a, the input acquisition unit 403b, the initialization unit 403c, the output acquisition unit 403d, the noise removal unit 403e, and the update unit 403f function as an image processing unit of the image estimation device 403. The components of the image processing unit are similar to the storage unit 103a, the input acquisition unit 103b, the initialization unit 103c, the output acquisition unit 103d, the noise removal unit 103e, and the update unit 103f of the image estimation device 103 of the first embodiment. The communication unit 403g functions as a receiving unit that receives a request transmitted from the computer 404 and as a transmitting unit that transmits an output image (deBayer image) generated by the image estimation device 403 to the computer 404.

[0121] The computer 404 has a communication unit 404a, a display unit 404b, an input unit 404c, a processing unit 404d, and a storage unit 404e. The communication unit 404a functions as a transmitter that transmits a request to the image estimation device 403 to cause the image estimation device 403 to execute processing on a captured image (Bayer image). The communication unit 404a also functions as a receiver that receives an output image (debayer image) processed by the image estimation device 403. The display unit 404b has a function to display various information. The information displayed by the display unit 404b includes, for example, the captured image (Bayer image) to be transmitted to the image estimation device 403 and the output image (debayer image) received from the image estimation device 403. The input unit 404c receives instructions from a user to start image processing, etc. The processing unit 404d has a function to perform image processing, including white balance adjustment and gamma correction, on the output image (debayer image) received from the image estimation device 403. The storage unit 404e stores the captured image acquired from the imaging device 402, the output image received from the image estimation device 403, and the like.

[0122] Next, image processing in this embodiment will be described with reference to Fig. 9. The image processing in this embodiment is equivalent to the image processing described in embodiment 1 with reference to Fig. 5. Fig. 9 is a flowchart related to generation of an output image using a neural network in this embodiment. The image processing shown in Fig. 9 is started when an instruction to start image processing is given by a user via computer 404. First, the operation of computer 404 will be described.

[0123] First, in step S501, the computer 404 transmits a request for processing a captured image (Bayer image) (processing an input patch related to the captured image) to the image estimation device 403. Note that the method for transmitting the captured image to be processed to the image estimation device 403 does not matter. For example, the captured image may be uploaded to the image estimation device 403 simultaneously with step S501, or may be uploaded to the image estimation device 403 before step S501. Furthermore, the captured image may be an image stored on a server different from the image estimation device 403. Furthermore, in step S501, the computer 404 may transmit an ID for authenticating a user, image information, and the like together with the request for processing the captured image.

[0124] Next, in step S 502 , the computer 404 receives the output image (debayered image) generated in the image estimation device 403 .

[0125] Next, the operation of the image estimation device 403 will be described.

[0126] First, in step S601, the image estimation device 403 receives a request for processing a captured image (Bayer image) transmitted from the computer 404. The image estimation device 403 determines that processing for the captured image has been instructed, and executes the processing from step S602 onwards.

[0127] Next, in step S602, the input acquisition unit 403b acquires a captured image (Bayer image, input). Similar to the upscaling in the first embodiment, an interpolated image is generated by simply de-Bayering the captured image using bilinear interpolation in an interpolation unit (not shown), but the interpolation may also be performed by the input acquisition unit 403b. In this embodiment, the captured image is transmitted from the imaging device 402, but this is not limiting. A plurality of input patches are acquired from a bilinearly interpolated image of the captured image, and sequential processing (patch division processing) is performed using a diffusion model in a step described later. During patch division, overlapping portions (corresponding portions) are provided between adjacent patches. Information (position information) about the location where the input patch is to be extracted from the captured image and the patch size, which are acquired together with the captured image, may also be used to acquire the input patch.

[0128] In this embodiment, acquisition of an input patch starts from the top right corner of the bilinearly interpolated image of the captured image. Then, a shift is made downward to acquire the next input patch. When the bottom right corner is reached, a shift is made to the left of the starting point to acquire an input patch, and a shift is made downward to acquire the next input patch. This process is repeated to acquire multiple input patches from across the entire captured image in the order in which vertically written text is read. However, this embodiment is not limited to this. Also, depending on the location where the input patch is acquired, it may extend beyond the captured image. In such cases, the shift amount or the amount of overlap with adjacent patches may be adjusted. Furthermore, the input patch corresponds to a conditioning patch x of the formula, and the captured image (more precisely, its bilinearly interpolated image) corresponds to the original image I of the formula.

[0129] Next, in step S603, the initialization unit 403c generates a noise patch (output patch) of the same size as the input patch from Gaussian noise. The noise patch may be generated using a pseudo-random number with a fixed random number seed. In this case, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, this may be used. The noise patch is generated by using a pure noise image y T is equivalent to

[0130] Next, in step S604, the output acquisition unit 403d acquires a patch (adjacent patch) that has been acquired from a location adjacent to the input patch in the bilinear interpolated image of the captured image and has already been processed using the diffusion model. Note that the acquisition of the adjacent patch may use information (position information) about the location where the input patch is extracted from the captured image, acquired together with the captured image and the input patch. Also, a mask that extracts the corresponding portion of the adjacent patch and the input patch is simultaneously acquired. Also, the adjacent patch corresponds to z in the formula, and the mask corresponds to m in the formula.

[0131] Next, in step S605, the noise removal unit 403e connects the input patch and the noise patch (output patch) in the channel direction and inputs them into the neural network to generate a noise removal patch. A neural network with the same configuration as that shown in FIG. 1 is used to generate the noise removal patch. The weight information is pre-trained information transmitted from the learning device 401 and stored in the storage unit 403a. In this embodiment, a predetermined value according to the number of iterations is used as the noise level used to generate the noise-removed image, but this is not limited to this. For example, a schedule of noise levels determined for each iteration and acquired together with the captured image may also be used. The noise removal patch corresponds to the estimated target patch y^ in the mathematical formula.

[0132] Next, in step S606, the update unit 403f generates an updated debayer patch (output patch) from the noise removal patch, adjacent patch, and mask based on equation (8). In this embodiment, the noise patch used for updating is generated from Gaussian noise, but it may also be generated using a pseudo-random number with a fixed random number seed. In this case, a value corresponding to the number of iterations of the process or a value corresponding to the number of iterations extracted from a separately prepared sequence may be used as the random number seed value. Also, if a recommended random number seed is included in the metadata of the image information acquired together with the captured image or input patch, it may be used. The updated debayer patch is generated from the intermediate patch yt-1 of the formula, and the noise patch is generated from the Gaussian noise ε t is equivalent to

[0133] Next, in step S607, the update unit 403f determines whether the processing is complete. If the processing is complete, the updated de-bayer patch is output to the storage unit 403a as an output patch. If the processing is not complete, the Bayer patch is used as a new output patch and the processing is repeated from step S605. In this embodiment, the number of iterations of the processing is used as a criterion for determining whether the processing is complete, but this is not limited to this. Note that information (position information) about the location where the input patch was extracted from the captured image may also be output together with the output patch, and may be used in the synthesis of the output patch, which will be described later. In addition, a default value acquired together with the captured image or the input patch is used as the upper limit of the number of iterations of the processing.

[0134] Next, in step S608, the update unit 403f determines whether processing has been completed for all input patches acquired from the bilinearly interpolated image of the captured image. If processing has been completed, the processed output patches are combined in a combining unit (not shown) to generate a de-bayered image. If processing has not been completed, a new input patch is acquired from the bilinearly interpolated image of the captured image, and the processing is repeated from step S602. Note that patch combination in the combining unit (not shown) is the same as in the first embodiment.

[0135] Next, in step S609, the image estimation device 403 sends the synthesized debayered image to the computer 404.

[0136] With the above configuration, according to this embodiment, patch division processing can be performed without any problems in debayer image processing using a diffusion model.

[0137] [Other Examples] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0138] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that are capable of appropriately performing patch division processing using a diffusion model. The image processing device may be any device that has the image processing function of each embodiment, and may be realized in the form of an imaging device, a personal computer, or the like.

[0139] The disclosure of each embodiment includes the following methods and compositions: (Method 1) The first step is to obtain input patches for the diffusion model; a second step of initializing output patches from the diffusion model; a third step of obtaining neighboring patches adjacent to the input patch and which have already been processed by the diffusion model; and a fourth step of iteratively updating the output patch based on the input patch, the output patch, and the adjacent patches by using the diffusion model. (Method 2) The image processing method described in Method 1, characterized in that in the fourth step, the output patch is iteratively updated based on an intermediate patch obtained by using the diffusion model based on the input patch and the output patch, and on a correspondence between the intermediate patch and the adjacent patch. (Method 3) 3. The image processing method according to method 2, wherein in the fourth step, the output patch is iteratively updated based on the gradient of the corresponding portion. (Method 4) The image processing method according to method 1, characterized in that in the fourth step, the output patch is iteratively updated based on the correspondence between the input patch, the output patch, and the adjacent patches. (Method 5) The image processing method according to Method 4, characterized in that in the fourth step, the output patch is iteratively updated by concatenating the corresponding parts and inputting them into the diffusion model. (Method 6) 6. An image processing method according to any one of methods 1 to 5, wherein the input patch is a patch extracted from an image and not processed by the diffusion model, and includes at least one of the following: patch size, metadata of the image, information about coordinates when extracting the patch from the image, information about a random number used when processing the patch extracted from the image, and information about the noise level of the patch extracted from the image. (Method 7) 7. An image processing method according to any one of methods 1 to 6, wherein the adjacent patch is a patch extracted from an image that overlaps with a portion of the input patch and is processed using the diffusion model. (Method 8) In the first step, a plurality of input patches are obtained by dividing an image; the plurality of input patches include a first input patch and a second input patch; 8. The image processing method according to any one of Methods 1 to 7, wherein the second input patch has an area common to a part of the first input patch. (Method 9) The image processing method described in Method 8, further comprising a fifth step of performing a synthesis process based on a plurality of output patches including a first output patch corresponding to the first input patch and a second output patch corresponding to the second input patch. (Configuration 1) A program causing a computer to execute the image processing method according to any one of Methods 1 to 7. (Configuration 2) an input acquisition unit that acquires input patches for the diffusion model; an initialization unit for initializing output patches from the diffusion model; an output acquisition unit that acquires adjacent patches that are adjacent to the input patch and have already been processed by the diffusion model; an update unit that iteratively updates the output patch by using the diffusion model based on the input patch, the output patch, and the adjacent patch. (Configuration 3) An image processing system including the image processing device according to configuration 2 and a control device capable of communicating with the image processing device, The control device a transmission unit that transmits a request to the image processing device to execute processing on an image related to the input patch; the image processing device has a receiving unit and an image processing unit, the receiving unit receives the request transmitted by the transmitting unit; The image processing system is characterized in that the image processing unit executes the processing in response to the request.

[0140] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0141] 103 Image estimation device (image processing device) 103b Input acquisition unit 103c Initialization section 103d Output acquisition section 103f Update section

Claims

1. A first step of obtaining input patches for the diffusion model; a second step of initializing output patches from the diffusion model; a third step of obtaining adjacent patches adjacent to the input patch and which have already been processed by the diffusion model; and a fourth step of iteratively updating the output patch based on the input patch, the output patch and the adjacent patches by using the diffusion model.

2. 2. The image processing method according to claim 1, wherein in the fourth step, the output patch is iteratively updated based on an intermediate patch obtained by using the diffusion model based on the input patch and the output patch, and a corresponding portion between the intermediate patch and the adjacent patch.

3. 3. The image processing method according to claim 2, wherein in the fourth step, the output patch is iteratively updated based on the gradient of the corresponding portion.

4. 2. The image processing method according to claim 1, wherein in the fourth step, the output patch is iteratively updated based on the corresponding portions of the input patch, the output patch, and the adjacent patches.

5. 5. The image processing method according to claim 4, wherein in the fourth step, the output patch is iteratively updated by connecting the corresponding portions and inputting them into the diffusion model.

6. 2. The image processing method according to claim 1, wherein the input patch is a patch extracted from an image and not processed by the diffusion model, and includes at least one of a patch size, metadata of the image, information about coordinates when extracting the patch from the image, information about a random number used when processing the patch extracted from the image, and information about a noise level of the patch extracted from the image.

7. 7. The image processing method according to claim 1, wherein the adjacent patch is a patch that overlaps with a part of the input patch, extracted from the image, and processed by the diffusion model.

8. In the first step, a plurality of input patches are obtained by dividing an image; the plurality of input patches include a first input patch and a second input patch; 7. The image processing method according to claim 1, wherein the second input patch has an area that overlaps with a part of the first input patch.

9. 9. The image processing method according to claim 8, further comprising a fifth step of performing a synthesis process based on a plurality of the output patches including a first output patch corresponding to the first input patch and a second output patch corresponding to the second input patch.

10. A program causing a computer to execute the image processing method according to any one of claims 1 to 6.

11. an input acquisition unit that acquires input patches for the diffusion model; an initialization unit for initializing output patches from the diffusion model; an output acquisition unit that acquires adjacent patches that are adjacent to the input patch and have already been processed by the diffusion model; an update unit that iteratively updates the output patch by using the diffusion model based on the input patch, the output patch, and the adjacent patch.

12. An image processing system including the image processing device according to claim 11 and a control device capable of communicating with the image processing device, The control device a transmission unit that transmits a request to the image processing device to execute processing on an image related to the input patch; the image processing device has a receiving unit and an image processing unit, the receiving unit receives the request transmitted by the transmitting unit; The image processing system is characterized in that the image processing unit executes the processing in response to the request.

Images