Image processing method, image processing apparatus, and program

Abstract

To provide an image processing method capable of generating high-quality images. [Solution] The image processing method is characterized by comprising: a first step of generating a first residual image as an added component based on an input image and an estimated noise image using a first machine learning model; a second step of generating a second residual image by reducing the absolute value of at least some of the pixel values ​​of the first residual image; and a third step of generating a first image based on the second residual image and the input image.

Description

[Technical Field]

[0001] This invention relates to an image processing method for estimating images using a machine learning model. [Background technology]

[0002] Methods for generating images using machine learning models such as generative and non-generative models are known (see Patent Document 1). Generative models can generate fine components in an image, but they can sometimes generate unnatural structures. On the other hand, non-generative models tend to produce solutions that are closer to the mean solution, so they rarely generate unnatural structures, but fine components in the image tend to disappear. Non-Patent Document 1 discloses a two-stage configuration in which an output image generated by a convolutional neural network, which is a non-generative model, and a residual image generated by a diffusion model, which is a generative model, are added together. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Publication No. 11769228 [Non-patent literature]

[0004] [Non-Patent Document 1] “ResDiff: Combining CNN and Diffusion Model for Image Super-Resolution”, Shuyao Shang et al.,AAAI, vol. 38, no.8, pp.8975-8983, Mar.2024. [Overview of the project] [Problems that the invention aims to solve]

[0005] However, when an image is input to a generative model to generate a residual image, residuals having an absolute value smaller than that of a noise image tend to disappear during the generation process, making it difficult to generate high-quality residual images and output images. [Means for Solving the Problem]

[0006] An image processing method according to one aspect of the present invention comprises: a first step of generating a first residual image as an applied component based on an input image and a noise image for estimation using a first machine learning model; a second step of generating a second residual image by reducing the absolute value of at least part of pixel values of the first residual image; and a third step of generating a first image based on the second residual image and the input image. [Effect of the Invention]

[0007] According to the present invention, an image processing method capable of generating high-quality images can be provided. [Brief Description of the Drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a flow of weight learning of the first machine learning model of the first embodiment. [Figure 2] FIG. 2 is a block diagram of the image processing system of the first embodiment. [Figure 3] FIG. 3 is an external view of the image processing system of the first embodiment. [Figure 4] FIG. 4 is a flowchart related to learning of the first machine learning model of the first embodiment. [Figure 5] FIG. 5 is a diagram related to a processing method for a correct residual image of the first embodiment. [Figure 6] FIG. 6 is a flowchart related to estimation by the first machine learning model of the first embodiment. [Figure 7] FIG. 7 is a diagram related to a residual image generated by the first machine learning model of the first embodiment. [Figure 8] FIG. 8 is a diagram showing an overall image of image processing of a second embodiment. [Figure 9]This diagram shows the flow of the second machine learning model in Example 2. [Figure 10] This is a block diagram of the image processing system in Example 2. [Figure 11] This is an external view of the image processing system in Example 2. [Figure 12] This is a flowchart illustrating the training of the second machine learning model in Example 2. [Figure 13] This is a flowchart illustrating the training of the first machine learning model in Example 2. [Figure 14] This is a flowchart illustrating the generation of output images using the machine learning model of Example 2. [Figure 15] This is a block diagram of the image processing system in Example 3. [Figure 16] This is a flowchart illustrating the generation of output images using the machine learning model of Example 3. [Modes for carrying out the invention]

[0009] The embodiments of the present invention will be described in detail below with reference to the drawings. In each figure, the same reference numeral is used for identical components, and redundant explanations are omitted.

[0010] First, before describing the specific examples, the gist of the present invention will be explained. In this invention, a generative model is used for image estimation. Compared to non-generative models, generative models have the characteristic of generating fine structures in the estimated image. However, when a noisy image is input to a generative model to generate a residual image, residuals with small absolute values ​​relative to the noise tend to disappear during the generation process, making it difficult to generate high-quality residual images and output images. Therefore, in this embodiment, an image processing method is provided that uses a machine learning model to generate residual images with high accuracy and to generate high-quality images.

[0011] In this embodiment, a model that acquires a probability distribution for generating desired data through learning and outputs based on that distribution during estimation is called a generative model. Generative models are known to perform better than conventional CNN (Convolutional Neural Network) based machine learning models in regression tasks such as image blur correction, depth estimation, and upsampling.

[0012] The generative models in this embodiment include, for example, diffusion models, GANs (Generative Adversarial Networks), VAEs (Variable Auto Encoders), flow-based generative models, and autoregressive generative models. The above are just examples of generative models, and the present invention is not limited thereto. Diffusion models can generate high-quality target images corresponding to input images or text by sampling from a target image distribution conditioned by image or text input. For example, Stable Diffusion generates high-quality images using a diffusion model in response to text input by the user.

[0013] In this embodiment, a residual image, which is an added component to an image, is generated. A residual image is a component added to the input in order to obtain a desired output. For example, when performing upscaling, which is image enlargement, the residual image is the difference between the input image, which is a low-resolution image, and the output image, which is a high-resolution image, and is a correction component for increasing resolution. Also, when removing blur from an image, the residual image is the difference between the input image, which has blur, and the output image, which has less blur, and is a correction component for reducing blur.

[0014] Furthermore, in this embodiment, residual images with large absolute values ​​are intentionally generated, and residual images obtained by reducing the absolute value of these large residual images are used. That is, in order to generate residuals with large absolute values ​​in the generative model, residuals obtained by increasing the absolute value of the ground truth residual image are used as the target image for training. Through this training, the trained machine learning model can generate large residual images, reduce the absolute value of the generated residual images, and add them to the input to produce high-quality images. [Examples]

[0015] In this embodiment, an image processing model is trained and executed to generate a high-resolution image by upscaling a low-resolution image using a diffusion model, which is a first machine learning model.

[0016] In this process, a first machine learning model is trained using an interpolated image, which is a first training image obtained by interpolating and enlarging a low-resolution image, and a ground truth residual image, which is the difference between the first training image and a high-resolution image (the ground truth image). The trained first machine learning model is then input with an input image to estimate the residual image. Finally, an upscaled image is obtained by adding the residual image and the input image.

[0017] In this embodiment, a first machine learning model is trained based on a second ground truth residual image in which the absolute values ​​of at least some of the pixel values ​​of the first ground truth residual image have been increased. Then, the second residual image, which is generated using the trained first machine learning model in which the absolute values ​​of at least some of the pixel values ​​of the first residual image have been decreased, is added to the input image.

[0018] From here, we will describe the diffusion model as an example of a generative model used in this embodiment. The diffusion model is also called a denoised diffusion probabilistic model (DDPM). The diffusion model includes a forward diffusion process and a reverse diffusion process. In the diffusion process, Gaussian noise is added to the target image step by step, gradually reducing the details of the target image until it becomes a purely noisy image. In the reverse diffusion process, the diffusion process is reversed by starting with a purely noisy image and iteratively recovering it by conditioning with image input or text input. The reverse diffusion process can be learned using a neural network (machine learning model) that takes a noisy target image and a conditioned image (or signal such as text) as input to denoise it.

[0019] The diffusion process is given by the following equations (1) and (2).

[0020]

number

[0021]

number

[0022] Here, y t-1 and y t These represent the intermediate images (t=1 to T) between step (number of iterations) t-1 and the next step t of the diffusion process q given by the Markov chain. Also, N is the standard normal distribution (Gaussian distribution), and α t (0<α t <1) step t y t-1 This represents the variance of the Gaussian noise added to it.

[0023] By transforming equations (1) and (2) (marginalizing), we obtain the intermediate image y given the target image y0. tThe distribution is given by the following equations (3) and (4).

[0024]

Math

[0025]

Math

[0026] Here, γ t represents the variance of Gaussian noise added to the target image y0 to obtain y t . That is, by adding Gaussian noise to the intermediate image y t-1 at each step of the diffusion process to obtain the intermediate image y t of the next step, equations (3) and (4) express that the intermediate image y t at the t-th step can be obtained by adding Gaussian noise to the target image y0.

[0027] By further modifying equations (1) to (4), the posterior distribution of the intermediate image y t when the target image y0 and the intermediate image y t-1 are given is given by the following equations (5) to (7).

[0028]

Math

[0029]

Math

[0030]

Math

[0031] Here, μ and σ 2 are the intermediate image y when the target image y0 and the intermediate image y t are given, intermediate image y t-1These represent the mean and variance of the standard normal distribution, which is the posterior distribution of y. That is, equations (5) through (7) represent the intermediate image y at the tth step. t The mean μ obtained from the target image y0 is given by the variance σ 2 By adding Gaussian noise, the intermediate image y at the t-1 step is obtained. t-1 This indicates that we can estimate this. In other words, equations (5) through (7) show that the diffusion process can be reversed.

[0032] The backdiffusion process is given by the following equations (8) to (10).

[0033]

number

[0034]

number

[0035]

number

[0036] Here, the inverse diffusion process p(t=T~1) is defined as an inverse Markov chain that proceeds in the opposite direction to the diffusion process q. Furthermore, following equation (9), the pure noise image y follows the diffusion process q. T It starts with Gaussian noise sampled from a standard normal distribution. Furthermore, each step of the backdiffusion process can be defined as a learnable isotropic Gaussian conditional distribution if the variance of the Gaussian noise added at each step of the diffusion process is set to be small, according to equation (10).

[0037] The mean μ of the Gaussian distribution at each step of the backdiffusion process θ The posterior distribution is calculated from equations (5) to (7), using the target image y0. However, the pure noise image y T In a backdiffusion process starting from , the target image y0 is generally unknown.

[0038] Therefore, in the diffusion model, a neural network (machine learning model) is trained to denoise the noise image obtained by adding Gaussian noise with various variances to the target image y0, and then the intermediate image y is created using this model. t The target image is estimated after denoising from the original image. The target image is represented by number 11.

[0039]

number

[0040] The noise reduction in the backdiffusion process is given by the following equation (11).

[0041]

number

[0042] Here, f is a denoising neural network with θ as a parameter, and x is the condition used for denoising. For example, Stable Diffusion uses an embedded vector for text as the condition for denoising, while Patent Document 1 uses an interpolated image of a low-resolution image. Also, Patent Document 1 uses an intermediate image y t The interpolated images are concatenated in the channel direction and input into a neural network, thereby performing noise reduction conditioned by the interpolated images. Furthermore, the noise reduction neural network is conditioned on a noise level (noise intensity) γ. t This is introduced, and noise reduction is performed according to the noise level. In this embodiment, the variance of the Gaussian noise added in the diffusion process is used as the noise level, but the present invention is not limited to this. The method for inputting the noise level to the noise reduction neural network will be described later.

[0043] Substituting the estimated target image into equations (5) through (7), the mean of the posterior distribution is parametrically given by equation (12). The estimated target image is represented by equation 11.

[0044]

number

[0045] Note that the variance of Gaussian noise is the variance α of the Gaussian noise added during the diffusion process. t It is being given.

[0046] Using this, each step of the backdiffusion process is given by the following equations (13) and (14).

[0047]

number

[0048]

number

[0049] By repeating this procedure from t=T to 1, a pure noise image y T From this, the target image y0 can be generated (more precisely, sampled from a target image distribution conditioned on x).

[0050] Next, the generation of residual images in this embodiment will be described. Residual images are components added to the input image in order to obtain the desired output.

[0051] By adding a residual image to the input image, the signal values ​​of the images are added together to obtain a sum. For example, when performing upscaling, which is image enlargement, the residual image is the difference in pixel values ​​between the low-resolution input image and the high-resolution output image, and is a correction component for increasing resolution. Also, when removing blur from an image, the residual image is the difference between the blurred input image and the less blurred output image, and is a correction component for reducing blur. In Patent Document 1, the target image was generated directly using a diffusion model, but when generating a residual image using a diffusion model, the target image becomes the residual image.

[0052] Next, we will explain upscaling by generating residual images using a diffusion model as an example. Here, upscaling is an image processing technique that generates a high-resolution image by estimating high-frequency components that cannot be represented in a low-resolution image and expanding the image based on that image.

[0053] In this embodiment, first, the neural network (machine learning model) used in the diffusion model is trained. High-resolution and low-resolution images are acquired, and an interpolated image is generated by enlarging the low-resolution image through interpolation processing. Bicubic interpolation is used for the interpolation processing, but the interpolation method in this invention is not limited to this. The neural network uses a ground truth residual image (target image y0), which is the difference between the high-resolution image and the interpolated image, and a noise image y in the ground truth residual image during the diffusion process. T The model is trained to estimate the denoised image using equation (11) by inputting the image with the added noise and the interpolated image.

[0054] Next, a neural network in the trained diffusion model generates residual images, and then an upscaled image is produced. First, a low-resolution image is acquired, and an interpolated image is generated by interpolation.

[0055] Next, the current intermediate image (output, y in the equation) T The noise level (equivalent to γ ​​in the equation) is initialized with Gaussian noise. Next, the interpolated image is input to the neural network along with the current intermediate image to perform noise reduction. T It is also preferable to input information (corresponding to) into the neural network.

[0056] Furthermore, the interpolated image may be concatenated with the current intermediate image in the channel direction and input to the neural network, and used as a condition for noise reduction. Also, the variance of the noise image may be a predetermined value for each iteration of the dediffusion process. For example, the maximum and minimum values ​​of the noise image variance can be defined and a value linearly determined from the number of iterations can be used. The noise image and the interpolated image are of similar size, and the noise image is a pure noise image y following the diffusion process q described above. T That is the case.

[0057] The neural network is pre-trained to perform noise reduction as given by equation (11). Next, the mean of the posterior distribution given by equation (11) is calculated from the noise reduction result (estimated result). The noise reduction result can be expressed by equation 13.

[0058] Finally, according to equations (13) and (14), the current intermediate image (output, y in equation) t-1 The corresponding value is updated. By repeating the above procedure, a residual image of high-frequency components that cannot be represented in a low-resolution image can be estimated and added to the interpolated image to generate a high-quality, high-resolution image.

[0059] However, when generating residual images using a diffusion model, residuals with small absolute values ​​relative to the image tend to disappear during the denoising process. When residuals with small absolute values ​​disappear, the components that contribute to high resolution decrease, making it difficult to generate high-quality, high-resolution images.

[0060] Therefore, in this embodiment, a large residual image is intentionally generated, and a reduced residual image is used. That is, in order to generate a large residual in the diffusion model, the residual is trained using a large residual image of the ground truth residual image as the target image. To achieve this, the trained diffusion model generates a large residual image in response to the input of a noise image and an interpolated image, reduces the size of the generated residual image, and adds it to the interpolated image to obtain a high-resolution upscaled image.

[0061] Figure 2 is a block diagram of the image processing system 100A of this embodiment. Figure 3 is an external view of the image processing system 100A. The image processing system 100A includes a learning device 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an input device 106, an output device 107, and a network 108.

[0062] The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and a learning unit 101d.

[0063] The imaging device 102 includes an optical system 102a and an image sensor 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The image sensor 102b receives the optical image of the subject formed through the optical system 102a and acquires an image (low-resolution image). The image sensor 102b is a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor, etc. Information regarding the shooting conditions of the image (pixel pitch of the image sensor 102b, type of optical low-pass filter, ISO sensitivity, etc.) can be acquired along with the image. Furthermore, development conditions of the image (noise reduction intensity, sharpness intensity, image compression ratio, etc.) can also be acquired along with the image. In addition, this information acquired along with the image may be configured to be transmitted along with the image to the acquisition unit 103b of the image estimation device 103. A storage unit for saving the acquired image, a display unit for displaying it, a transmission unit for sending it externally, an output unit for saving it to an external storage medium, etc. are not shown. Furthermore, the control unit that controls each part of the imaging device 102 is not shown.

[0064] The image estimation device 103 includes a storage unit 103a, an acquisition unit 103b, a residual generation unit 103c, a residual adjustment unit 103d, and a residual addition unit 103e. The image estimation device 103 acquires an image captured by the acquisition unit 103b and acquires an interpolated image obtained by interpolating the image captured as the input image.

[0065] The residual generation unit 103c inputs the interpolated image and the noise image into a diffusion model (first machine learning model) to generate a first residual image. The residual adjustment unit 103d generates a second residual image by reducing the absolute values ​​of at least some of the pixel values ​​of the first residual image. The residual addition unit 103e adds the second residual image and the interpolated image to generate an upscaled high-resolution image (output). The captured image may be an image taken by the imaging device 102 or an image stored on the recording medium 105. Furthermore, the image estimation device 103 only needs to be a device having the image processing function of this embodiment, and can be implemented in the form of an imaging device or a personal computer.

[0066] The weight information for the first machine learning model is read from the memory unit 103a. The weights are learned by the learning device 101, and the image estimation device 103 has previously read the weight information from the memory unit 101a via the network 108 and stored it in the memory unit 103a. The weight information stored may be the numerical weights themselves or in an encoded format.

[0067] The upscaled high-resolution image is output to at least one of the display device 104, the recording medium 105, and the output device 107. The display device 104 is, for example, a liquid crystal display or a projector. The user can check the image in progress via the display device 104 and perform image editing work via the input device 106. The recording medium 105 is, for example, semiconductor memory, a hard disk, or a server on a network. The input device 106 is, for example, a keyboard or a mouse. The output device 107 is, for example, a printer.

[0068] The following describes the method for learning the weights of the first machine learning model (the method for generating a trained model) executed by the learning device 101 of this embodiment, with reference to Figures 1 and 4. Figure 1 is a diagram showing the flow of learning the weights of the first machine learning model. Figure 4 is a flowchart related to the training of the first machine learning model. Each step in Figure 4 is mainly performed by the acquisition unit 101b, the generation unit 101c, and the learning unit 101d.

[0069] In step S101, the acquisition unit 101b acquires a low-resolution image 21 as a training image and a high-resolution image 20 as a ground truth image corresponding to the low-resolution image 21. In this embodiment, the image is divided into patches, which are small images with a predetermined number of pixels. For example, the low-resolution image is a patch of 64 × 64 × 3 pixels (corresponding to the number of pixels in the vertical × horizontal × depth directions), and the high-resolution image corresponding to this low-resolution image is a patch of 128 × 128 × 3 pixels. In this case, since the vertical and horizontal sizes are doubled, the upscaling ratio is doubled (the number of pixels is quadrupled). Note that the upscaling ratio is not limited to double; any ratio is acceptable as long as a high-resolution image corresponding to the low-resolution image can be acquired.

[0070] Furthermore, a low-resolution image and a corresponding high-resolution image may be obtained by photographing the same subject with an optical system having different focal lengths and cropping the corresponding portion of the two resulting images. Alternatively, a high-resolution image may be downsampled to generate a corresponding low-resolution image. Alternatively, a corresponding high-resolution image with minimal influence (aberration and diffraction) from the optical system 102a may be generated numerically from a comparable low-resolution image acquired by the imaging device 102. In this embodiment, the low-resolution image and the corresponding high-resolution image are generated numerically, but the present invention is not limited thereto.

[0071] In step S102, the generation unit 101c obtains (generates) an interpolated image (first training image) 22 by interpolating and enlarging the low-resolution image 21 to the same number of pixels as the high-resolution image 20, which is the corresponding ground truth image. In this embodiment, Bicubic interpolation was used as the interpolation method, but the present invention is not limited thereto, and Nearest Neighbor interpolation or Bilinear interpolation may also be used.

[0072] In step S103, the generation unit 101c obtains a first ground truth residual image 23, which is the difference between the interpolated image 22 and the ground truth high-resolution image 20.

[0073] In step S104, the generation unit 101c obtains a second ground truth residual image 24 in which the absolute values ​​of at least some of the pixel values ​​of the first ground truth residual image 23 are increased. For example, the generation unit 101c obtains the second ground truth residual image 24 by multiplying the first ground truth residual image 23 by a constant greater than 1. Alternatively, the second ground truth residual image 24 may be obtained by adding a constant to the first ground truth residual image 23. However, since the objective is to increase the absolute values ​​of at least some of the pixel values ​​of the first ground truth residual image 23, it is preferable to multiply the first ground truth residual image 23 by a constant greater than 1.

[0074] Figure 5 illustrates a method for processing ground truth residual images, showing a first ground truth residual image 23 and a second ground truth residual image 24. In Figure 5, 11 represents the values ​​of one cross-section in the first ground truth residual image 23. 12 in Figure 5 represents a cross-section of the second ground truth residual image 24, obtained by multiplying the first ground truth residual image 23 by a constant, thereby increasing the absolute values ​​of at least some of the pixel values ​​in the first ground truth residual image 23. The constant multiplied by the first ground truth residual image 23 is preferably determined so that the residual falls within a representable numerical range. For example, if the numerical range representable by the machine learning model is ±1, and the range of the first ground truth residual image 23 is -0.2 to 0.2, the constant is set to 5. Alternatively, processing may be performed to increase the value of the residual as its absolute value decreases. For example, nonlinear transformations such as gamma correction may be performed.

[0075] The method for obtaining a second ground truth residual image 24 by increasing the absolute values ​​of at least some of the pixel values ​​of the first ground truth residual image 23 is not limited to these, and the constant may be determined according to the upscaling ratio. For example, the larger the upscaling ratio, the smaller the constant may be. The ratio may be vertical, horizontal, or the entire image. Also, when blur correction is performed, the constant may be determined according to the magnitude of the blur. For example, the larger the blur, the smaller the constant may be. In this embodiment, the magnitude of the blur is OTF (Optical Transfer Function) or PSF (Point Spread Function). By training the first machine learning model using the second ground truth residual image 24, which has increased the absolute values ​​of at least some of the pixel values ​​of the first ground truth residual image 23, the effect of reducing the tendency for residuals with small absolute values ​​to disappear during the noise reduction process during estimation, as described later, can be obtained.

[0076] In step S105, the generation unit 101c generates a noise-added ground truth residual image 25 by adding the first noise image (training noise image) to the second ground truth residual image 24.

[0077] In step S106, the learning unit 101d obtains a denoised residual image 26 by denoising the noise-added ground truth residual image 25. Denoising is performed by the first machine learning model using the interpolated image 22, the noise-added ground truth residual image 25, and information on the noise intensity (the noise level added to the second ground truth residual image 24). The denoised residual image 26 is an estimate of the second ground truth residual image 24, and ideally, the two will coincide. In this embodiment, the variance of the added Gaussian noise was used as the noise level, but the present invention is not limited to this. In this embodiment, the interpolated image 22 and the noise-added ground truth residual image 25 are concatenated in the channel direction and input to the first machine learning model, but the present invention is not limited to this.

[0078] In step S107, the learning unit 101d updates the weights of the first machine learning model based on the error between the second ground truth residual image 24 and its estimated denoised residual image 26. The weights include the filter components and biases of each layer.

[0079] In this embodiment, backpropagation is used to update the weights. However, the present invention is not limited to this. For example, mini-batch learning may be used. In mini-batch learning, the error between the second ground truth residual image 24 and the denoised residual image corresponding to the second ground truth residual image 24 is calculated, and the weights are updated. The loss function can be, for example, the L2 norm or the L1 norm. The weight update method (learning method) is not limited to mini-batch learning; the same effect can be achieved with batch learning or online learning. In addition, the error may be calculated between the estimated noise to be removed and the noise added to the noise-added ground truth residual image 25.

[0080] In step S108, the learning unit 101d determines whether learning is complete. Learning completion can be determined by whether the number of weight update iterations has reached a predetermined value, or whether the amount of change in weight during the update is less than a predetermined value. If it is determined that learning is incomplete, the process returns to step S101 and multiple new training images, which are low-resolution images 21, and corresponding high-resolution images 20 are the correct images. On the other hand, if it is determined that learning is complete, the learning device 101 terminates learning and stores the weight information in the storage unit 101a.

[0081] In this embodiment, a method for learning the weights of the first machine learning model in the image domain was described, but the present invention is not limited thereto. Learning may also be performed in the feature map (latent) domain using a feature map obtained from training images by a neural network and a noise-added ground residual image 25 obtained from the second ground residual image 24 by a neural network with added noise.

[0082] In Figure 1, CN represents a convolutional layer. CN calculates the sum of the input, the filter convolution, and the bias, and then non-linearly transforms the result using an activation function. The initial values ​​of each filter component and the bias are arbitrary and, in this embodiment, are determined by random numbers. The activation function can be, for example, ReLU (Rectified Linear Unit) or a sigmoid function. The multidimensional array output from each layer except the final layer is the feature map. Generally, a feature map is a four-dimensional array with batch, length / width, and channel dimensions. The skip connection (also called a shortcut connection) 27 synthesizes feature maps output from non-contiguous layers. The feature maps can be synthesized by element-wise summation or by concatenation in the channel direction. In this embodiment, element-wise summation is used.

[0083] The elements (blocks or modules) within the dotted lines in Figure 1 represent residual blocks. A network with multiple layers of residual blocks is called a residual network and is widely used in image processing using machine learning models. The present invention is not limited to this, and convolutional layers with different convolutional filter sizes may be placed side by side. For example, the network may be constructed by layering other elements such as an inception module that integrates multiple resulting feature maps into a final feature map, or a dense block with densely packed skip connections.

[0084] Furthermore, the feature map's width and height are scaled up or down within the network. For feature map reduction, techniques such as pooling and stride can be used. For feature map expansion, techniques such as deconvolution (or transposed convolution), pixel shuffle, and interpolation can be used. Although not shown in this example, stride convolution is used for feature map reduction, and Nearest Neighbor interpolation is used for expansion.

[0085] Furthermore, a noise level (γ in the figure) is input to the residual block, and noise reduction is performed from the noisy image according to the noise level. In this embodiment, a vector embedding noise level information is generated using a multilayer perceptron (MLP in the figure) based on the noise level. Then, based on the generated vector, the feature map obtained between the two convolutional layers of the residual block is scaled (affine transformation using gain to be integrated and bias to be added to the feature map). However, the present invention is not limited to this. A multilayer perceptron is a type of neural network. Also, in the despreading process, the noise level of the noisy image differs with each iteration, so a different noise level γt is input to the residual block for each iteration.

[0086] In this embodiment, the configuration of the first machine learning model shown in Figure 1 is used, but the present invention is not limited thereto.

[0087] Next, with reference to Figure 6, the generation of upscaled images by a diffusion model, which is a first machine learning model, performed in the image estimation device 103 of this embodiment will be described. Figure 6 is a flowchart relating to the estimation of the diffusion model (generation of upscaled images by the diffusion model). Each step in Figure 6 is mainly performed by the acquisition unit 103b, residual generation unit 103c, residual adjustment unit 103d, and residual addition unit 103e of the image estimation device 103.

[0088] In step S20, the acquisition unit 103b acquires the captured image. The captured image is a low-resolution image, similar to the training image. In this embodiment, the captured image is transmitted from the imaging device 102, but the present invention is not limited to this. Image information may also be acquired along with the captured image and used in the steps described later. In addition, similar to the training image, an interpolated image obtained by bicubic interpolation of the captured image is used as the input image and generated by an interpolation unit (not shown), but the acquisition unit 103b may also perform the interpolation. Furthermore, the captured image may be interpolated based on the image size acquired along with the captured image.

[0089] In step S202, the residual generation unit 103c acquires a second noise image (estimation noise image) of the same size as the interpolated image. In this embodiment, the second noise image generated has random values ​​following a standard normal distribution (mean 0, standard deviation 1), but the present invention is not limited to this.

[0090] In step S203 (the first step), the residual generation unit 103c inputs the interpolated image (input image) and the second noise image into the first machine learning model to generate the first residual image. In this embodiment, a diffusion model is used to generate the first residual image. First, the interpolated image and the second noise image are concatenated in the channel direction and input into the first machine learning model to generate a denoised image. The denoised image corresponds to the estimated target image in equation (11). Here, the estimated target image is represented by equation 11.

[0091] Furthermore, a first machine learning model similar to the configuration shown in Figure 1 is used to generate the denoised image. The weight information is the weight information of the trained first machine learning model, transmitted from the learning device 101 and stored in the storage unit 103a. In this embodiment, a predetermined value corresponding to the number of iterations was used as the noise level when generating the denoised image, but the present invention is not limited thereto. For example, a noise level schedule defined for each iteration acquired together with the captured image may be used.

[0092] Next, based on equation (11), an updated intermediate image is generated from the denoised image. The intermediate image is the intermediate image y from equation (13). t-1 The noise image is the Gaussian noise ε of equation (14) t This corresponds to the following. Next, it is determined whether the processing is complete. If the processing is complete, the updated intermediate image is output as the first residual image. If the processing is not complete, the intermediate image is used as a new input, and the processing from the generation of the denoised image is repeated. In this embodiment, the number of processing iterations was used as the criterion for determining whether the processing is complete, but the present invention is not limited thereto.

[0093] Through the above process, the residual generation unit 103c repeatedly performs noise reduction to update the denoised image, ultimately obtaining a first residual image. The first residual image has a value greater than that of the ideal residual image. In this embodiment, a specific example of generating a residual image using a diffusion model is described, but generative models such as GANs may also be used. When using a GAN, the interpolated image and the noise image are input to the first machine learning model neural network to generate the residual image.

[0094] In step S204 (second step), the residual adjustment unit 103d acquires the first residual image generated in step S203 and acquires a second residual image in which the absolute values ​​of at least some of the pixel values ​​of the first residual image are reduced. For example, the absolute values ​​are reduced by multiplying the first residual image by the reciprocal of a constant. The constant that constitutes the reciprocal of the constant multiplied by the first residual image may be the same constant used when acquiring the second correct residual image in step S104. Alternatively, the second residual image may be acquired by subtracting the constant from the first residual image. However, since the objective is to reduce the absolute values ​​of at least some of the pixel values ​​of the first residual image, it is preferable to multiply the first residual image by the reciprocal of a constant.

[0095] Figure 7 shows the residual image generated by the first machine learning model. 13 in Figure 7 represents the values ​​of one cross-section in the first residual image. 14 in Figure 7 represents a cross-section of the second residual image, obtained by multiplying the first residual image by a constant, thereby increasing the absolute values ​​of at least some of the pixel values ​​in the first residual image. If a nonlinear transformation such as gamma correction is performed on the first ground truth residual image during training, it is preferable to perform the reverse operation of the process that enlarged the first residual image during training. This process scales the residual image to match the scale of the first ground truth residual image. The second residual image also contains the small absolute values ​​that were present in the first residual image.

[0096] In step S205 (the third step), the residual summing unit 103e obtains an upscaled image (the first image) by adding the second residual image and the interpolated image (input image). By using the second residual image which has pixel value components with small absolute values, a highly accurate upscaled image can be generated. The generated upscaled image (the first image) is output as an output image to the display device 104, recording medium 105, and output device 107.

[0097] In this embodiment, a method for generating residual images using a diffusion model and then generating upscaled images was described in the image domain, but the present invention is not limited to this. The method may also be performed in the feature map (latent) domain using a feature map obtained from captured images or interpolated images using a neural network. In that case, the residual generation unit 103c will generate a noise image of the same size as the feature map obtained from the interpolated image and use the trained first machine learning model.

[0098] Furthermore, although this embodiment implicitly assumes that the target image for generating an upscaled image using the diffusion model is a color image, the present invention is not limited to this. For example, an upscaled image may be generated using a monochrome luminance image extracted from a captured color image.

[0099] With the above configuration, it is possible to generate residual images with high accuracy and produce high-quality images using a machine learning model. [Examples]

[0100] In this embodiment, image processing is performed on captured images to correct blur and generate blur-reduced images using a non-generative model and a generative model. In this embodiment, image processing is learned and executed using a convolutional neural network (CNN) as the second non-generative machine learning model and a diffusion model as the first machine learning model.

[0101] In this embodiment, a non-generative model such as ViT (Vision Transformer) may be used as the second machine learning model. Furthermore, although the first machine learning model in this embodiment is a generative diffusion model, the present invention is not limited to this. Similar to Example 1, for example, VAEs or GANs may be used.

[0102] Figure 8 shows the overall image processing in this embodiment. In Embodiment 1, only the first machine learning model performed image estimation in one stage, but in this embodiment, two models, the first machine learning model and the second machine learning model, are used to generate images in two stages.

[0103] In this embodiment, first, a second machine learning model is trained using a first training image (a blurred image) and a ground truth image (an image with less blur). Next, the first training image is input to the trained second machine learning model to obtain a second training image with blur correction. In Embodiment 1, the first machine learning model was trained using a first ground truth residual image, which is the difference between a low-resolution image and a high-resolution image. On the other hand, in this embodiment, the first machine learning model is trained using a fourth ground truth residual image, which is the difference between the second training image and the third ground truth residual image, which is the difference between the second training image and the ground truth image corresponding to the second training image, with at least some of the absolute values ​​of the pixels of the third ground truth residual image increased. Note that the second machine learning model estimates the entire image, not just the residual components.

[0104] A blur-reduced image is generated using the first and second machine learning models obtained through training. In Example 1, the first machine learning model obtained through training generates a first residual image for a low-resolution input image, and an upscaled first image is obtained by adding it to the input image.

[0105] In this embodiment, a second image generated using a second machine learning model is input to the first machine learning model. Furthermore, in this embodiment, a second residual image obtained by reducing the absolute values ​​of at least some of the pixel values ​​of the first residual image generated by the first machine learning model is added to the second image generated by the second machine learning model to obtain a blur-reduced image (first image).

[0106] This embodiment describes the generation of blur-reduced images, but the present invention is not limited thereto. For example, an upscaled image may be generated as in Example 1. In that case, a low-resolution image or an interpolated image is input to the second machine learning model instead of a blurred image.

[0107] Figure 10 is a block diagram of the image processing system 100B of this embodiment. Figure 11 is an external view of the image processing system 100B. The image processing system 100B includes an imaging device 200, an image estimation device 202, a display device 203, a storage medium 204, an input device 205, an output device 206, and a network 207. The imaging device 200 includes an optical system 200a, an image sensor 200b, and a learning device 201.

[0108] The learning device 201 includes a storage unit 201a, an acquisition unit 201b, a generation unit 201c, and a learning unit 201d. The learning device 201 learns the weights of a first machine learning model and a second machine learning model.

[0109] The image estimation device 202 includes a storage unit 202a, an acquisition unit 202b, a residual generation unit 202c, a residual adjustment unit 202d, a residual addition unit 202e, and a generation unit 202f. The image estimation device 202 acquires a second image using the acquisition unit 202b.

[0110] The generation unit 202f inputs the input image (a blurred image) into a second machine learning model and generates a second image with reduced blur.

[0111] The residual generation unit 202c inputs the second image and the noise image into the first machine learning model to generate the first residual image.

[0112] The residual adjustment unit 202d generates a second residual image in which the absolute values ​​of at least some of the pixel values ​​of the first residual image are increased.

[0113] The residual adjustment unit 202d adds the second residual image to the second image acquired by the acquisition unit 202b to generate a first image with further reduced blur.

[0114] Next, with reference to Figures 9, 10, and 12, the processing performed by the learning device 201 of this embodiment will be described. The learning device 201 learns the weights of the second machine learning model and the weights of the first machine learning model. This learning method corresponds to the method for generating the trained model of the second machine learning model and the trained model of the first machine learning model.

[0115] In this embodiment, the third ground truth residual image used to train the first machine learning model is the difference between the second training image, obtained by inputting the first training image into the second machine learning model, and the ground truth image corresponding to the second training image.

[0116] Figure 9 shows the flow of the second machine learning model. Figure 12 is a flowchart of the training of the second machine learning model. Each step in Figure 12 is mainly performed by the acquisition unit 201b, the generation unit 201c, and the training unit 201d. The second machine learning model in this embodiment is a CNN that performs image processing to reduce blur from the input image.

[0117] In step S301, the acquisition unit 201b acquires one or more pairs of first training images and the corresponding ground truth images 30 from the storage unit 201a. In this embodiment, the first training images are blurred images obtained from the ground truth images 30, which are images with less blur. The first training images and the ground truth images 30 contain the same subject. The data used for learning may include other images or maps in addition to blurred images. For example, it may include noise or maps representing the degree of blur. The pairs of the first training images (blurred images) and the ground truth images 30 (images with less blur) can be prepared using live-action footage, imaging simulations, and computer graphics (CG).

[0118] In step S302, the generation unit 201c inputs the first training image 31 into the second machine learning model to generate an estimated image 32 with reduced blur.

[0119] The second machine learning model in this embodiment is a CNN that uses the convolution of the input and the filter as a linear sum. The values ​​of each element of the filter in the CNN correspond to the weights. Note that the sum with the bias may also be included. In addition, each layer performs a nonlinear transformation using an activation function such as ReLU (Rectified Linear Unit) or a sigmoid function as needed. Furthermore, the second machine learning model may have residual blocks and skip connections 33 as needed. The elements (blocks or modules) within the dotted frame in Figure 9 represent residual blocks.

[0120] In step S303, the learning unit 201d updates the weights of the second machine learning model using an error function. In this embodiment, an error function is used that is based on the error (loss) between the output of the second machine learning model and the ground truth image. The Mean Squared Error (MSE) is used to calculate the error. However, the error function of the present invention is not limited to this. In addition, methods such as backpropagation can be used to update the weights using the error function. Alternatively, the error may be obtained for the difference component between the training image and the ground truth image. In that case, the error between the difference component between the estimated image 32 output by the second machine learning model and the first training image, and the error between the difference component between the ground truth image 30 and the first training image are used.

[0121] In step S304, the learning unit 201d determines whether the learning of the second machine learning model is complete. Completion can be determined by whether the number of iterations of weight updates has reached a predetermined number, or whether the amount of change in weights during updates is less than a predetermined value. If it is determined that learning is not complete, the process returns to step S301, and the acquisition unit 201b acquires one or more new sets of first training images and ground truth images. On the other hand, if it is determined that learning is complete, the learning unit 201d terminates training and stores the configuration and weight information of the second machine learning model in the storage unit 201a.

[0122] Next, the training method for the first machine learning model will be described. Figure 13 is a flowchart of the training of the first machine learning model. Each step in Figure 13 is mainly performed by the acquisition unit 201b, the generation unit 201c, and the training unit 201d. The first machine learning model in this embodiment is a diffusion model, similar to that in Embodiment 1.

[0123] In step S401, the acquisition unit 202b (or generation unit 202f) inputs the first training image 31 into a trained second machine learning model to acquire (generate) a second training image. The second training image is an image in which the blurring of the first training image 31 has been reduced.

[0124] In this embodiment, the third ground truth residual image used for training the first machine learning model is the difference between the second training image generated by the second machine learning model and the ground truth image corresponding to the second training image. The fourth ground truth residual image is obtained by increasing the absolute values ​​of at least some of the pixel values ​​of the third ground truth residual image. In this embodiment, the fourth ground truth residual image is used instead of the second ground truth residual image 24 of Example 1.

[0125] In step S402, the acquisition unit 201b acquires the second training image output from the trained second machine learning model and the ground truth image corresponding to the second training image. Then, the acquisition unit 201b acquires a third ground truth residual image from the difference between the second training image and the ground truth image.

[0126] In step S403, the acquisition unit 201b acquires a fourth ground truth residual image obtained by increasing the absolute values ​​of at least some of the pixel values ​​of the third ground truth residual image. The method for acquiring the fourth ground truth residual image may be the same as in step S104 of Example 1. In this example, when acquiring the fourth ground truth residual image by multiplying the third ground truth residual image by a constant, the constant may be determined according to the degree of blur in the training image, and the larger the blur, the smaller the constant may be.

[0127] In step S404, the generation unit 201c acquires a noise-added ground residual image by adding the second noise image to the fourth ground residual image, similar to step S105 of Example 1.

[0128] In step S405, the learning unit 201d acquires a denoised residual image using the first machine learning model, similar to step S106 of Example 1. The denoised residual image is generated by denoising the noise-added ground residual image using the fourth ground residual image, the noise-added ground residual image, and information on the strength of the noise (noise level) added to the noise-added ground residual image.

[0129] In step S406, the learning unit 201d updates the weights of the first machine learning model based on the error between the fourth ground truth residual image and the denoised residual image which is its estimated result. The method for updating the weights of the first machine learning model is the same as in Example 1.

[0130] In step S407, the learning unit 201d determines whether the learning of the weights of the first machine learning model is complete, similar to step S107 in Example 1. If it is determined that the learning is incomplete, the unit returns to step S405 and acquires multiple new denoising residual images and corresponding fourth ground truth residual images. On the other hand, if it is determined that the learning is complete, the learning device 201 terminates the learning process and stores the weight information in the storage unit 201a.

[0131] In this invention, the learning methods for the second machine learning model and the first machine learning model are not limited to those described herein. For example, the second machine learning model and the first machine learning model may be learned together, or each machine learning model may be learned alternately.

[0132] Next, with reference to Figure 14, the generation of blur-reduced images using the first and second machine learning models will be explained. Figure 14 is a flowchart of the generation of blur-reduced images using the diffusion model. Each step in Figure 14 is mainly performed by the acquisition unit 202b, residual generation unit 202c, residual adjustment unit 202d, residual addition unit 202e, and generation unit 202f of the image estimation device 202.

[0133] In step S501, the acquisition unit 202b acquires the captured image as the input image. The captured image is a blurred image.

[0134] In step S502, the generation unit 202f inputs the captured image (input image) into a second machine learning model that has already been trained, thereby obtaining a second image with reduced blur.

[0135] In step S503, the residual generation unit 202c acquires a second noise image of the same size as the second image, similar to step S202 in Example 1.

[0136] In step S504, the residual generation unit 202c inputs the second image and the second noise image to the first machine learning model, similar to step S203 of Example 1, to obtain a first residual image (de-noised image). The second image and the first noise image are concatenated in the channel direction and input to the first machine learning model to generate a first residual image. The first residual image corresponds to the estimated target image in equation (11). Here, the estimated target image is represented by equation 11.

[0137] Furthermore, the first machine learning model, with the same configuration as shown in Figure 2, is used to generate the second residual image. In Example 1, the second image, which is an interpolated image, was input to the first machine learning model during estimation to obtain the first residual image. However, in this example, the second image generated from the second machine learning model is input to the first machine learning model.

[0138] In step S505, the residual adjustment unit 202d acquires a second residual image by reducing the absolute values ​​of at least some of the pixel values ​​of the first residual image generated in step S504, similar to step S204 in Example 1.

[0139] In step S506, the residual addition unit 202e acquires the first image by adding the second residual image and the second image, similar to step S205 in Example 1. In this embodiment, the addition of the second residual image and the second image generates a first image with reduced blur. By using a second residual image having pixel value components with small absolute values, an image with corrected blur can be generated with high accuracy.

[0140] With the above configuration, high-quality images can be generated by using a machine learning model to produce residual images with high accuracy. [Examples]

[0141] The image processing system of this embodiment differs from embodiments 1 and 2 in that it has a processing unit (computer) that transmits the captured image to be processed to the image estimation device and receives the processed output image from the image estimation device.

[0142] Figure 15 is a block diagram of the image processing system 100C of this embodiment. The configuration of the learning device 301 in this embodiment is the same as that of the learning device 201 in Embodiment 2.

[0143] The image processing system 100C includes a learning device 301, an imaging device 302, an image estimation device 303, and a processing device 304. The learning device 301 and the image estimation device 303 may be servers. The processing device 304 is, for example, a user terminal such as a personal computer or a smartphone. The processing device 304 is connected to the image estimation device 303 via a network 305. The image estimation device 303 is connected to the learning device 301 via a network 306. In other words, the processing device 304 and the image estimation device 303, as well as the image estimation device 303 and the learning device 301, are configured to communicate with each other.

[0144] Since the learning device 301 and the imaging device 302 have the same configuration as the learning device 201 of Example 2 and the imaging device 102 of Example 1, respectively, their descriptions will be omitted.

[0145] The image estimation device 303 includes a storage unit 303a, an acquisition unit 303b, a residual generation unit 303c, a residual adjustment unit 303d, a residual addition unit 303e, and a generation unit 303f. The storage unit 303a and the acquisition unit 303b of the image estimation device 303 have the same configuration as the storage unit 202a and the acquisition unit 202b of Example 2, respectively, so their description is omitted. The residual generation unit 303c and the residual adjustment unit 303d have the same configuration as the residual generation unit 202c and the residual adjustment unit 202d of Example 2, respectively, so their description is omitted. The residual addition unit 303e and the generation unit 303f have the same configuration as the residual addition unit 202e and the generation unit 202f of Example 2, so their description is omitted.

[0146] The processing unit 304 includes a communication unit 304a, a display unit 304b, an input unit 304c, a processing unit 304d, and a recording unit 304e. The communication unit 304a transmits a request to the image estimation unit 303 to perform processing on the captured image, and receives the output image processed by the image estimation unit 303. The communication unit 304a may also communicate with the imaging unit 302.

[0147] The display unit 304b displays various information. The various information displayed by the display unit 304b includes, for example, the captured image transmitted to the image estimation device 303 and the estimated image received from the image estimation device 303.

[0148] The input unit 304c allows the user to input instructions such as starting image processing.

[0149] The processing unit 304d can perform image processing on the output image received from the image estimation device 303.

[0150] The recording unit 304e stores the captured images acquired from the imaging device 302 and the output images received from the image estimation device 303.

[0151] The method of transmitting the captured image to be processed to the image estimation device 303 is not limited; for example, the captured image may be uploaded to the image estimation device at the same time as step S601 described later, or it may be uploaded to the image estimation device 303 before step S601. Also, the captured image may be an image stored on a server different from the image estimation device 303.

[0152] Next, with reference to Figure 16, the generation of blur-reduced images by a first machine learning model executed in the image estimation device 303 of this embodiment will be described. Figure 16 is a flowchart of the generation of blur-reduced images by a diffusion model. Each step in Figure 16 is mainly executed by the acquisition unit 303b, residual generation unit 303c, residual adjustment unit 303d, residual addition unit 303e, and generation unit 303f of the image estimation device 303. The flow in Figure 16 is started by an instruction from the user to start image processing via the processing device 304.

[0153] First, let's explain the operation of the processing unit 304.

[0154] In step S601, the communication unit 304a transmits a request for processing of the captured image to the image estimation device 303. In step S601, the processing device 304 may also transmit, along with the request for processing of the captured image, an ID for user authentication, and shooting conditions corresponding to the captured image. The acquisition unit 303b acquires the captured image as an input image. The captured image is a blurred image.

[0155] In step S602, the processing unit 304 receives the output image (second image) generated in the image estimation device 303. The output image (second image) is an image in which blur correction has been performed on the captured image, similar to Example 2.

[0156] Next, the operation of the image estimation device 303 will be described.

[0157] In step S603, the image estimation device 303 receives a request for processing of the captured image transmitted from the processing device 304. The image estimation device 303 determines that processing of the captured image has been instructed and executes the processing from step S604 onward.

[0158] In step S604, the acquisition unit 303b acquires the second image as an input image and inputs it to the first machine learning model. In this embodiment, the second image is transmitted from the processing unit 304. At this time, the shooting conditions corresponding to the captured image may be acquired along with the second image. The residual generation unit 303c generates a second noise image of the same size as the second image, similar to step S503 of Embodiment 2.

[0159] The processing details in steps S605 to S608 are the same as those in steps S502 to S505 of Example 2, so their explanation will be omitted.

[0160] In step S609, the residual addition unit 303e obtains the first image by adding the second residual image and the second image, similar to step S506 in Example 2. In this embodiment, the addition of the second residual image and the second image generates a first image with reduced blur. By using a second residual image having pixel value components with small absolute values, a high-quality, blur-corrected image can be generated.

[0161] In step S610, the image estimation device 303 transmits the output image to the processing device 304.

[0162] In step S611, the processing unit 304 determines whether to terminate the process. If the processing unit 304 determines to terminate the process, it terminates this flow; otherwise, it executes the process in step S601.

[0163] In this embodiment, the blur correction process of Embodiment 2 is described, but it can also be applied similarly to the upscaling of Embodiment 1 (Figure 6). In that case, a low-resolution image is input to the second machine learning model instead of a blurred image.

[0164] With the above configuration, high-quality images can be generated by creating highly accurate residual images using a machine learning model. [Other examples] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0165] This embodiment includes the following methods and configurations. (Method 1) A first step involves using a first machine learning model to generate a first residual image as an added component based on the input image and the estimated noise image, A second step of generating a second residual image by reducing the absolute value of at least some of the pixel values ​​of the first residual image, An image processing method characterized by comprising a third step of generating a first image based on the second residual image and the input image. (Method 2) The image processing method according to Method 1, characterized in that the first machine learning model is trained on a training noise image, a first training image, and a second ground truth residual image generated by increasing the absolute values ​​of at least some of the pixel values ​​of the first ground truth residual image, which is the difference between the first training image and the ground truth image. (Method 3) The image processing method according to Method 2, characterized in that the second ground truth residual image is generated by multiplying the first ground truth residual image by a constant greater than 1. (Method 4) The image processing method according to Method 1, characterized in that the second residual image is generated by multiplying the first residual image by the reciprocal of a constant greater than 1. (Method 5) The image processing method according to Method 1, characterized in that the first step includes the step of generating a second image based on an input image using a second machine learning model, and the step of generating a first residual image based on the second image and the estimation noise image using the first machine learning model. (Method 6) The image processing method according to Method 5, characterized in that the first machine learning model is trained on a second training image generated based on a first training image using the second machine learning model, a training noise image, and a fourth ground truth residual image generated by increasing the absolute values ​​of at least some of the pixel values ​​of a third ground truth residual image, which is the difference between the second training image and the ground truth image. (Method 7) The image processing method according to method 6, characterized in that the fourth ground truth residual image is generated by multiplying the third ground truth residual image by a constant greater than 1. (Method 8) The image processing method according to method item 1 or 5, characterized in that the second residual image is generated by multiplying the first residual image by the reciprocal of a constant greater than 1.

[0166] The image processing method according to method 6, characterized in that the second ground truth residual image is generated by multiplying the first residual image by the reciprocal of a constant greater than 1. (Method 9) The first image is an upscaled version of the input image. The image processing method according to any one of methods 3, 4, 7, or 8, characterized in that the constant is determined according to the upscaling magnification. (Method 10) The image processing method according to method 9, characterized in that the constant is determined such that it becomes smaller as the magnification increases. (Method 11) The image processing method according to method 9, characterized in that the resolution of the first training image is lower than the resolution of the ground truth image. (Method 12) The first image is an image obtained by removing blur from the input image. The image processing method according to any one of methods 3, 4, 7, or 8, characterized in that the constant is determined according to the degree of blur in the input image. (Method 13) The image processing method according to method 12, characterized in that the constant is determined such that it becomes smaller as the blur of the input image increases. (Method 14) The image processing method according to method 12, characterized in that the blur of the first training image is greater than the blur of the ground truth image. (Method 15) The image processing method according to any one of methods 1 to 14, characterized in that the first machine learning model is a generative model. (Method 16) The image processing method according to any one of methods 1 to 14, characterized in that the first machine learning model is a diffusion model. (Composition 1) A first means for generating a first residual image as an added component based on an input image and an estimated noise image using a first machine learning model, A second means for generating a second residual image by reducing the absolute value of at least some of the pixel values ​​of the first residual image, An image processing apparatus characterized by having a second residual image and a third means for generating a first image based on the input image. (Configuration 2) The image processing apparatus according to configuration 1, characterized in that the first means includes means for generating a second image based on an input image using a second machine learning model, and means for generating a first residual image based on the second image and the estimation noise image using the first machine learning model. (Method 17) A program characterized by causing a computer to execute the image processing method described in any one of methods 1 to 16.

[0167] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of Symbols]

[0168] 103 Image Estimation Device

Claims

1. A first step involves generating a first residual image as an added component based on an input image and an estimated noise image using a first machine learning model, A second step of generating a second residual image by reducing the absolute value of at least some of the pixel values ​​of the first residual image, An image processing method characterized by comprising a third step of generating a first image based on the second residual image and the input image.

2. The image processing method according to claim 1, characterized in that the first machine learning model is trained on a training noise image, a first training image, and a second ground truth residual image generated by increasing the absolute values ​​of at least some of the pixel values ​​of the first ground truth residual image, which is the difference between the first training image and the ground truth image.

3. The image processing method according to claim 2, characterized in that the second ground truth residual image is generated by multiplying the first ground truth residual image by a constant greater than 1.

4. The image processing method according to claim 1, characterized in that the second residual image is generated by multiplying the first residual image by the reciprocal of a constant greater than 1.

5. The image processing method according to claim 1, characterized in that the first step includes the step of generating a second image based on an input image using a second machine learning model, and the step of generating a first residual image based on the second image and the estimation noise image using the first machine learning model.

6. The image processing method according to claim 5, characterized in that the first machine learning model is trained on a second training image generated based on a first training image using the second machine learning model, a training noise image, and a fourth ground truth residual image generated by increasing the absolute values ​​of at least some of the pixel values ​​of a third ground truth residual image, which is the difference between the second training image and the ground truth image.

7. The image processing method according to claim 6, characterized in that the fourth ground truth residual image is generated by multiplying the third ground truth residual image by a constant greater than 1.

8. The image processing method according to claim 1 or 5, characterized in that the second residual image is generated by multiplying the first residual image by the reciprocal of a constant greater than 1.

9. The first image is an upscaled version of the input image. The image processing method according to any one of claims 3, 4, 7, or 8, characterized in that the constant is determined according to the upscaling magnification.

10. The image processing method according to claim 9, characterized in that the constant is determined such that it becomes smaller as the magnification increases.

11. The image processing method according to claim 9, characterized in that the resolution of the first training image is lower than the resolution of the ground truth image.

12. The first image is an image obtained by removing blur from the input image. The image processing method according to any one of claims 3, 4, 7, or 8, characterized in that the constant is determined according to the degree of blur in the input image.

13. The image processing method according to claim 12, characterized in that the constant is determined such that it becomes smaller as the blur of the input image increases.

14. The image processing method according to claim 12, characterized in that the blur of the first training image is greater than the blur of the ground truth image.

15. The image processing method according to any one of claims 1 to 8, characterized in that the first machine learning model is a generative model.

16. The image processing method according to any one of claims 1 to 8, characterized in that the first machine learning model is a diffusion model.

17. A first means for generating a first residual image as an added component based on an input image and an estimated noise image using a first machine learning model, A second means for generating a second residual image by reducing the absolute value of at least some of the pixel values ​​of the first residual image, An image processing apparatus characterized by having a third means for generating a first image based on the second residual image and the input image.

18. The image processing apparatus according to claim 17, wherein the first means includes means for generating a second image based on an input image using a second machine learning model, and means for generating a first residual image based on the second image and the estimation noise image using the first machine learning model.

19. A program characterized by causing a computer to execute the image processing method described in any one of claims 1 to 8.

Images