Image processing method, program, and storage medium

Abstract

To provide a highly accurate image processing method which prevents appearance of false structures in image processing using machine learning models.SOLUTION: An image processing method disclosed herein comprises a step of generating an estimated image from an input image using multiple machine learning models including a generative model and a non-generative model, where the estimated image is generated by using different weighting between an output of the generative model and an output of the non-generative model for each of multiple areas of the input image on the basis of information on the input image.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image processing method using a machine learning model. [Background technology]

[0002] There are multiple machine learning models, each differing in network structure and learning method. A model that acquires a probability distribution for generating various desired data through learning and outputs an estimation based on that distribution is called a generative model. Generative models are known to perform better than conventional convolutional neural network (CNN)-based machine learning models in regression tasks such as image blur correction, depth estimation, and upsampling. Non-Patent Document 1 discloses a method for sharpening image blur using a diffusion model, which is one type of generative model. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, Peyman Milanfar, “Deblurring via Stochastic Refinement“, https: / / arxiv.org / abs / 2112.02475 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to provide a highly accurate image processing method that suppresses the occurrence of false structures in image processing using a machine learning model. [Means for solving the problem]

[0005] An image processing method as one aspect of the present invention includes a step of generating an estimated image from an input image using a plurality of machine learning models including generative models and non-generative models, and is characterized in that in the step, the estimated image is generated by varying the weights of the output of the generative model and the output of the non-generative model for each of a plurality of regions of the input image based on information about the input image. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide a highly accurate image processing method that suppresses the occurrence of false structures in image processing using a machine learning model. [Brief explanation of the drawings]

[0007] [Figure 1] 10A and 10B are diagrams illustrating point spread functions at defocus distances in the respective examples. [Figure 2] FIG. 1 is a block diagram of an image processing system according to a first embodiment and a fourth embodiment. [Figure 3] FIG. 1 is an external view of an image processing system according to a first embodiment and a fourth embodiment. [Figure 4] 1 is a flowchart of training a non-generative model according to Examples 1 to 3. [Figure 5] 1 is a flowchart of training a generative model according to Examples 1 to 3. [Figure 6] 1 is a flowchart illustrating generation of a model output in the first embodiment. [Figure 7] 4A and 4B are diagrams illustrating a captured image and a defocus map according to the first embodiment. [Figure 8] FIG. 2 is a diagram illustrating region division of a captured image in the first embodiment. [Figure 9] FIG. 2 is an explanatory diagram of a non-generative model according to the first embodiment. [Figure 10] FIG. 2 is an explanatory diagram of a generative model according to the first embodiment. [Figure 11] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 12] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 13]10 is a flowchart illustrating generation of a model output in the second embodiment. [Figure 14] 10A and 10B are explanatory diagrams of a captured image and focused object information in the second embodiment. [Figure 15] FIG. 10 is an explanatory diagram of a depth map according to the second embodiment. [Figure 16] FIG. 10 is an explanatory diagram of a machine learning model according to a second embodiment. [Figure 17] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 18] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 19] FIG. 10 is an explanatory diagram of a machine learning model according to a third embodiment. [Figure 20] 10 is a flowchart illustrating generation of a model output in the third embodiment. [Figure 21] 1 is a conceptual diagram of the first machine learning model. [Figure 22] 1 is a flowchart of the first machine learning model. [Figure 23] A conceptual diagram of the second machine learning model. [Figure 24] 10 is a flowchart of the second machine learning model. [Figure 25] FIG. 10 is a conceptual diagram of image processing according to a fourth embodiment. [Figure 26] 10 is a flowchart of image processing according to a fourth embodiment. [Figure 27] FIG. 10 is a block diagram of an image processing system according to a fifth embodiment. [Figure 28] FIG. 13 is a conceptual diagram of image processing in the sixth embodiment. [Figure 29] 13 is a flowchart of image processing according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same components, and redundant explanations will be omitted.

[0009] There are multiple machine learning models depending on the network structure and learning method. For example, a model that acquires a probability distribution for generating various desired data through learning and outputs based on that distribution during estimation is called a generative model in this embodiment. The generative model is learned (trained) so that the distribution of generated data matches the distribution of learning data. Typical generative models include VAE (Variational Auto Encoder), GAN (Generative Adversarial Networks), flow-based generative models, autoregressive generative models, and diffusion models. Each generative model will be described below.

[0010] A VAE consists of two main parts: an encoder and a decoder. The encoder outputs two variables, the mean μ and the variance σ, and samples the latent variable Z from these values. The decoder restores the latent variable Z to its original dimensions. A VAE explicitly measures and minimizes the distance between the distributions of the generated data and the training data. To train a model to minimize the distance between distributions, it is sufficient to maximize the log-likelihood of the training data. However, in complex neural networks, it is difficult to directly calculate the log-likelihood for the data. Therefore, a VAE acquires the desired data distribution by training a model to maximize the lower bound of the log-likelihood, which is easy to calculate. Using a model trained in this way, data that conforms to the desired data distribution can be generated.

[0011] GAN is a machine learning model that has a generator and a discriminator. The generator generates fake data x based on input data z. f = G(z). The discriminator receives real data x as input data (discrimination input data). r or fake data x f The discriminator outputs a discrimination output C(x j ) where j=r,f. Generally, the classification output C(x j ) is input to a transformation function h, such as a sigmoid function σ, to obtain the discriminant label D(x j )=h(C(xj In GAN training, the generator updates the weights (including biases) so that the classifier can distinguish fake data from real data, and the classifier acquires weights to correctly distinguish fake data from real data.

[0012] In this way, GANs allow the generator and discriminator to learn by adversarially improving each other's accuracy, making it possible to generate high-quality fake data (with properties similar to real data). Unlike other generative models, GANs do not explicitly train to bring the probability distributions of training data and the data output by the generator closer together, but instead implicitly do so through adversarial learning.

[0013] Like VAE, the diffusion model trains the model to maximize the lower bound of the log-likelihood. The diffusion model is based on a diffusion process, which gradually adds noise to an image with desired properties to ultimately obtain a completely noisy image, and a dediffusion process, which gradually removes noise from the noisy image to obtain an image with the desired properties.

[0014] Here, the noise is Gaussian noise. Each stage is represented by time t, with t = 0 representing the image state with the desired properties and t = T representing the completely noisy image state. Note that because the diffusion model is inspired by thermodynamics, the expression "time" is used for convenience, but time does not actually change, and t can also be represented as a calculation step or the intensity level of the noise.

[0015] The diffusion process is easy to implement because it adds noise to the image, but the dediffusion process does not produce an image with the desired properties even if an appropriate Gaussian noise is subtracted. Therefore, the diffusion model uses a neural network to remove noise at each time point in the dediffusion process.

[0016] In this case, to maximize the lower bound of the log-likelihood, we train a neural network to minimize the error of the estimated noise. By repeatedly removing noise using a neural network trained from a completely noisy image at t=T, we obtain a noisy image at t=0 (data that follows the desired data distribution).

[0017] The flow-based generative model is a method for generating data that conforms to the desired data distribution by gradually converting the distribution that generates the data and obtaining the distribution of the training data.

[0018] To train a model to minimize the distance between the distributions of the generated data and the training data, we need to maximize the log-likelihood in the training data. In flow-based generative models, a simple transformation f that can invert the complex transformation between distributions is used to directly calculate the log-likelihood. i This is expressed by repeating the above. By learning to maximize the log-likelihood that can now be calculated, the desired data distribution can be obtained. In addition, since the conversion between distributions is reversible, unlike GANs and VAEs, a single model can convert data to latent variables and generate data from latent variables. A typical flow-based generative model uses a simple conversion f i Examples include Glow, which applies 1x1 convolution and affine transformation.

[0019] In an autoregressive generative model, the likelihood of the data is calculated directly by accumulating one-dimensional likelihood calculations, which are easy to calculate, and then maximized through learning. For example, in an image, the likelihood of the data can be calculated by calculating and accumulating the conditional probability for each pixel in turn, starting from the top left. To implement this, PixelCNN, for example, performs convolution with a masked kernel so that each pixel only references the pixel before it.

[0020] Although a generative model may also use a CNN for internal processing, in this embodiment, a machine learning model that uses a CNN is referred to as a generative model if it acquires a probability distribution for generating various desired data through learning and outputs based on that distribution during estimation. On the other hand, even if the machine learning model uses a CNN, if it does not acquire a probability distribution for generating various desired data through learning and outputs an average solution that is expected to be correct for input data, it is referred to as a non-generative model in this embodiment.

[0021] In particular, diffusion models are known to outperform conventional CNNs in regression tasks such as deblurring, depth estimation, and upsampling. However, while generative models can generate high-resolution textures, they are known to have the potential to produce spurious structures that do not exist in the original subject.

[0022] Therefore, in this embodiment, a generative model is used to shape the defocus blur on the out-of-focus plane based on the defocus map, and a CNN is used to sharpen the blur on the in-focus plane. This makes it possible to sharpen the blur on the in-focus plane of the captured image and shape the defocus blur on the out-of-focus plane into various shapes while suppressing the occurrence of false structures. Note that the tasks processed using the machine learning model are not limited to this. Furthermore, the map used is not limited to the defocus map. For example, a depth map, a segmentation map (semantic region division map), information on saturated regions, and an optical performance map may also be used. The weights of the machine learning model are varied based on the information on the captured image described above.

[0023] A segmentation map is a map in which a captured image is divided into regions by classifying each pixel. For example, by identifying people and other regions (buildings, cars, plants, etc.), blur sharpening is performed on the person region using CNN and on other regions using a generative model. False structures are more likely to be a problem in the person region than in other regions. Alternatively, a person may be further divided into a skin region and other regions (eyes, mouth, etc.), and the skin region may be processed using a generative model while the other regions may be processed using CNN. There are no restrictions on how the machine learning model is determined, and it may be freely determined depending on the region.

[0024] The optical performance map is a map calculated from the optical characteristics of the optical system used to capture the captured image. In areas with low optical performance, blur sharpening is likely to cause problems. Therefore, blur sharpening is performed using CNN in areas with good performance, and using a generative model in areas with poor performance.

[0025] Information about saturated regions includes brightness saturation maps and saturation influence maps. A brightness saturation map is a map that shows brightness saturated regions in a captured image, and a saturation influence map is a map that shows the magnitude and range of signal values that have expanded due to degradation during capture for subjects in brightness saturated regions of the captured image. Brightness saturated regions can occur in an image depending on the dynamic range of the image sensor and the exposure during capture. When blur sharpening is performed using a machine learning model, it is difficult to obtain information about the structure of the subject's space in brightness saturated regions, which can easily lead to problems. Therefore, blur sharpening is performed using CNN for non-brightness saturated regions and a generative model for brightness saturated regions.

[0026] As described above, by determining a machine learning model based on information about the segmentation map, optical performance map, and saturated region, it becomes possible to perform processing that controls the blur sharpening effect and the occurrence of false structures.

[0027] In the following explanation, the stage in which the weights of the machine learning model are updated will be referred to as the learning (training) phase, and the stage in which the machine learning model uses the weights obtained in the learning phase to sharpen the blur will be referred to as the estimation phase. [Example 1] The tasks performed by the machine learning model in this embodiment are to sharpen the blur on in-focus planes in a captured image and to shape the defocus blur on out-of-focus planes. CNN is used to sharpen the blur on in-focus planes, and a diffusion model, which is a type of generative model, is used to shape the defocus blur on out-of-focus planes. The diffusion model will be described later. Furthermore, the blur to be sharpened targets aberrations and diffraction that occur in the optical system, as well as blur caused by an optical low-pass filter. However, the effects of the present invention can also be obtained in the case of sharpening blur caused by pixel aperture or blur.

[0028] Shaping of defocus blur includes, for example, shaping from two-line blur to Gaussian blur or spherical blur. Details of various defocus blur shapes will be described later. Other defocus blur that can be shaped include, for example, defocus blur missing due to vignetting and ring-shaped defocus blur caused by pupil obstruction by a catadioptric lens or the like. There are no restrictions on the shape of the defocus blur that can be shaped, and there are also no restrictions on the shape of the defocus blur after shaping. In this embodiment, a structure that does not exist in the actual subject is called a false structure, but shaping of defocus blur is different from a false structure because it is a process of shaping the aberration shape generated by the optical system.

[0029] Here, the difference from the addition of defocus blur performed in an imaging device having a relatively wide-angle lens and a small sensor size (for example, a smartphone, etc.) will be described. Images captured using a wide-angle lens and a small imaging device are less likely to blur. For this reason, defocus blur may be added to create the desired defocus blur for the user. On the other hand, the defocus blur shaping performed in this embodiment shapes the defocus blur of an object that is already defocus blurred into the desired defocus blur. In other words, it is necessary to apply defocus blur that satisfies the difference between the defocus blur that has already occurred and the desired defocus blur, which requires more advanced processing.

[0030] 2 and 3 are a block diagram and an external view, respectively, of an image processing system 100 according to this embodiment. The image processing system 100 includes a training device 101 and an image processing device 103, which are connected via a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 via wired or wireless connections. A captured image of a subject space captured using the imaging device 102 is input to the image processing device 103. The captured image is blurred due to aberration and diffraction caused by the optical system 102a in the imaging device 102 and an optical low-pass filter in the image sensor 102b, resulting in attenuation of subject information. The image processing device 103 uses a machine learning model to sharpen blur on in-focus planes and to reshape defocus blur on out-of-focus planes in the captured image. The machine learning model is trained by the training device 101, and the image processing device 103 acquires information about the machine learning model from the training device 101 in advance and stores it in a storage unit 103a. The image processing device 103 also has a function of calculating a weighted average of the outputs of the CNN and the generative model in the boundary region between the in-focus and out-of-focus planes based on a defocus map. Training and estimation of the machine learning model will be described later. The processed image is stored in the storage unit 103a or the recording medium 105, and is output to an output device 106 such as a printer as needed. The captured image may be grayscale or may have multiple color components. The captured image may also be an undeveloped RAW image or a developed image.

[0031] The image processing device may be any device having the image processing function of this embodiment, and may be realized in the form of an imaging device or a PC.

[0032] Hereinafter, with reference to FIG. 4, the training of the machine learning model that learns blur sharpening, which is executed by the training device 101, will be described. FIG. 4 is a flowchart of training of the machine learning model (CNN) in this embodiment. In this embodiment, CNN is used as the machine learning model that learns blur sharpening. The training device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, and any of these components executes each of the following steps.

[0033] In step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. Since the machine learning model is trained based on the original images, it is desirable that the original images have various frequency components (edges with different orientations and intensities, gradations, flat areas, etc.). The original images may be real images or CG (Computer Graphics).

[0034] In step S102, the calculation unit 101c applies blur to the original image to generate a blurred image. The blurred image is an image input to the machine learning model during training and corresponds to the captured image during estimation. The blur to be applied is the blur to be sharpened. In this embodiment, blur generated by aberration and diffraction of the optical system 102a and the optical low-pass filter of the image sensor 102b is applied. The shape of the blur caused by the aberration and diffraction of the optical system 102a varies depending on the image plane coordinates (image height and azimuth). It also varies depending on the magnification, aperture, and focus state of the optical system 102a. When simultaneously training a machine learning model that sharpens all of these blurs, it is preferable to generate multiple blurred images using multiple blurs generated by the optical system 102a. If necessary, noise generated by the image sensor 102b may be applied to the blurred image.

[0035] In step S103, the acquisition unit 101b acquires a correct model output. Since the task is blur sharpening, the correct model output is an image with less blur than the blurred image. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct model output. In this case, reduction is also performed when generating a blurred image in step S102. Furthermore, step S103 may be executed at any time after step S101 and before step S104.

[0036] In step S104, the calculation unit 101c generates a model output based on the blurred image using a machine learning model. In this embodiment, the machine learning model shown in Fig. 9 is used, but the present invention is not limited to this. The blurred image 201 is input to the machine learning model.

[0037] The machine learning model has multiple layers, and a linear sum of the layer's input and weights is calculated in each layer. The initial value of the weight may be determined using a random number or the like. In the blur sharpening of this embodiment, a CNN is used as the machine learning model, which uses the convolution of the input and filter as the linear sum (the value of each element of the filter corresponds to the weight, and may also include a sum with a bias). However, the present invention is not limited to this. In addition, in each layer, nonlinear transformation is performed using an activation function such as a ReLU (Rectified Linear Unit) or a sigmoid function as necessary. Furthermore, the machine learning model may have a residual block or a skip connection (also called a shortcut connection) as necessary. As a result of passing through multiple layers, a model output 202 is generated.

[0038] In step S105, the update unit 101d updates the weights of the machine learning model based on the error function. In the blur sharpening of this embodiment, the error function is the error between the model output 202 and the correct model output. MSE (Mean Squared Error) is used to calculate the error. However, the error function is not limited to this. Backpropagation or the like may be used to update the weights. The error may also be taken for the residual component. In the case of the residual component, the error between the difference component between the model output 202 and the blurred image 201 and the difference component between the correct model output and the blurred image 201 is used.

[0039] In step S106, the update unit 101d determines whether training of the machine learning model is complete. Completion can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If the update unit 101d determines that training is complete, it ends this flow, and if it determines that training is not complete, it executes the processing of step S101. After this flow ends, the configuration of the machine learning model and information on the weights are stored in the storage unit 101a.

[0040] Hereinafter, with reference to Fig. 5, the training of the machine learning model that learns to shape defocus blur, which is executed by the training device 101, will be described. Fig. 5 is a flowchart of training of the machine learning model (diffusion model) of this embodiment. In this embodiment, a diffusion model is used as the generative model that learns to shape defocus blur. The training device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, and any of these components executes each of the following steps.

[0041] In step S201, the acquisition unit 101b acquires one or more original images from the storage unit 101a.

[0042] In step S202, the calculation unit 101c generates training images and stores them in the storage unit 101a. The training images are images obtained by performing an imaging simulation by applying the defocus blur to be shaped to the original image. In order to accommodate various captured images, it is preferable to apply defocus blur corresponding to various defocus amounts. The application of defocus blur can be performed by convolving the original image with the PSF or by calculating the product of the frequency characteristics of the original image and the OTF (Optical Transfer Function).

[0043] In step S203, the calculation unit 101c generates a correct image corresponding to the training image and stores it in the storage unit 101a. The correct image is an image obtained by performing an imaging simulation by applying post-shaping defocus blur to the original image. The shape of the post-shaping defocus blur is, for example, a circular blur equivalent to F2.0 or a Gaussian blur equivalent to F1.0. The correct image and the training image may be undeveloped RAW images or developed images. The order in which the training images and the correct image are generated may be reversed. Double-line blur, circular blur, and Gaussian blur will be described with reference to FIG. 1. The upper diagram in FIG. 1 shows the PSF of double-line blur.

[0044] In the upper diagram of Figure 1, the horizontal axis represents spatial coordinates (position), and the vertical axis represents intensity. This also applies to the middle diagram and lower diagram of Figure 1, which will be described later. As shown in the upper diagram of Figure 1, double-line blur has a PSF with separated peaks. When the PSF at the defocus distance has a shape like the upper diagram of Figure 1, an object that is actually a single line appears to be doubly blurred when defocused. The middle diagram of Figure 1 shows the PSF of circular blur. Circle blur has a PSF with flat intensity. The lower diagram of Figure 1 shows the PSF of Gaussian blur. Gaussian blur has a PSF with a Gaussian distribution.

[0045] In step S204, the calculation unit 101c adds noise to the correct image.

[0046] Here, each step in which noise is gradually added is time t, and the final time is T. The noise added at each time t is Gaussian noise. If T=1000, Gaussian noise is added 1000 times. In other words, multiple noise images with different amounts of noise and time data are generated for one ground truth image. The amount of noise added at each time is determined by a noise scheduler. The noise scheduler is a parameter that controls the amount of noise added at each time step. There are linear schedulers and cosine schedulers as noise schedulers, but in this embodiment, a cosine scheduler is used. Note that the number of steps in which the noise is added and the noise scheduler are not limited to those described above.

[0047] In step S205, the calculation unit 101c generates a model output based on the training image using the machine learning model. In the shaping of defocus blur in this embodiment, the model output uses the configuration of the machine learning model shown in FIG. 10, but the present invention is not limited to this. The input data to the machine learning model are the training image 203, the noise image 204 at time t generated in step S204, and time data 205 representing time t. The shape of the time data is not limited, and it may be a scalar value or a two-dimensional map. Furthermore, there is no limit to the position at which the time data is input to the neural network; it may be input from the same position as the training image 203, or it may be input separately from an intermediate layer.

[0048] In step S206, the update unit 101d updates the weights of the machine learning model based on the error function. In this embodiment, the error function is the error between the model output 206 and the correct model output. Here, when a noise image at time t is input, the correct model output is the noise image at time t-1. MSE (Mean Squared Error) is used to calculate the error. However, the error function is not limited to this. Backpropagation or the like may be used to update the weights. The error may also be taken for the residual component. In the case of the residual component, the error between the difference component between the model output 206 and the training image 203 and the difference component between the correct model output and the training image 203 is used.

[0049] In step S207, the update unit 101d determines whether training of the machine learning model is complete. Completion can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If the update unit 101d determines that training is complete, it ends this flow, and if it determines that training is not complete, it executes the processing of step S201. After this flow ends, the configuration of the machine learning model and information on the weights are stored in the storage unit 101a.

[0050] 6, a description will be given of blur sharpening on an in-focus plane in a captured image and shaping of defocus blur on an out-of-focus plane using a trained machine learning model, which are executed by the image processing device 103. The image processing device 103 has a storage unit 103a, an acquisition unit 103b, and a processing unit 103c, and any of these components executes the following steps.

[0051] In step S301, the acquisition unit 103b acquires a captured image and a machine learning model. Information about the configuration and weight of the machine learning model is acquired from the storage unit 103a. In this embodiment, a CNN is used to sharpen the blur on the in-focus plane, and a diffusion model, which is a generative model, is used to shape the defocus blur on the out-of-focus plane, so two machine learning models are acquired. Note that there is no limit to the number of machine learning models to be acquired, and three or more machine learning models may be acquired.

[0052] In step S302, the acquisition unit 103b acquires information about the distance of the captured image. The information about the distance of the captured image is a depth map, a defocus map, etc. A depth map is a map that shows information about the distance to a subject in a captured image, and indicates the distance to the subject numerically. A depth map can be acquired by a distance measuring device such as a ToF sensor. For example, when pixel values range from 0 to 255, the distance to the subject can be expressed by setting a value closer to 255 as the distance to the subject increases, and a value closer to 0 as the distance to the subject decreases. Note that the value closer to 0 as the distance to the subject increases, and the value closer to 255 as the distance to the subject decreases, and the range of pixel values is not limited to 0 to 255.

[0053] When using a depth map, information on which subject is in focus may be required to identify the in-focus subject. This information on which subject is in focus can be obtained from focus information at the time of shooting, etc. A defocus map is a map that shows information on the defocus blur that acts on the subject in a captured image, and indicates the defocus amount of the subject as a numerical value. A defocus map can be obtained by capturing parallax images or using DFD (Depth from Defocus), etc. For example, the focus plane can be set to 0, and the direction away from the imaging device can be set to negative and the direction toward the imaging device can be set to positive. In this embodiment, a defocus map is used as information on the distance to the captured image. Note that maps related to the distance to the subject are not limited to depth maps or defocus maps. The upper diagram in FIG. 7 shows a captured image 111, and the lower diagram in FIG. 7 shows a defocus map 115. There are a focused subject 112, an out-of-focus subject 113 (small defocus amount), and an out-of-focus subject 114 (large defocus amount), and the numerical values of the defocus map vary depending on the defocus amount.

[0054] In step S303, the processing unit 103c divides the captured image into multiple regions. The upper diagram of FIG. 8 shows an example of division of the captured image 111. In the upper diagram of FIG. 8, the captured image is divided into 10 vertical and 12 horizontal patches, for a total of 120 patches, but there is no set number of divisions. When inputting each patch to a machine learning model, it is preferable to include a wider range in the patch than the area originally desired to be processed, because the surrounding pixels of the patch are affected by the convolutional layer of the machine learning model. After excluding the surrounding pixels affected by the convolutional layer from the output of the machine learning model, adjacent patches can be combined, allowing the entire captured image to be processed without being affected by the convolutional layer.

[0055] In step S304, the processing unit 103c inputs the captured image divided into multiple regions into different machine learning models for each of the in-focus and out-of-focus regions. An explanatory diagram of the input method is shown in the middle of Figure 8. In the captured image divided into multiple regions, the in-focus region 116 is input to the CNN, the out-of-focus region 117 is input to the diffusion model, and the boundary region 118 between the in-focus and out-of-focus regions is input to both the CNN and the diffusion model. The defocus map acquired in step S302 is used to determine the in-focus and out-of-focus regions. In this step, if the final time T is set to 1000 and noise addition is repeated 1000 times, the forward propagation of the diffusion model is repeated 1000 times to gradually remove noise. Note that the number of times noise is added and the number of forward propagations do not necessarily need to match, and certain times may be skipped. This reduces processing time.

[0056] In step S305, the processing unit 103c combines the model outputs based on the defocus map. The model outputs of the CNN are used for the in-focus region, and the model outputs of the diffusion model are used for the out-of-focus region. The model outputs of the CNN and the diffusion model are combined in the boundary region between the in-focus region and the out-of-focus region based on the defocus map. The defocus map acquired in step S301 indicates pixel values of in-focus regions in the image as 1 and pixel values of out-of-focus regions in the image as 0. In this embodiment, the model outputs of the CNN and the diffusion model can be combined by weighted addition based on the defocus map. Note that the combination method is not limited, and other methods may be used. For example, a defocus map that continuously changes from 0 to 1 depending on the defocus amount may be used, and the model outputs of the CNN and the diffusion model may be weighted and added at a ratio of 50% in the region where the defocus amount is 0.5.

[0057] As described above, according to the configuration of this embodiment, it is possible to provide a highly accurate image processing method while suppressing the occurrence of false structures in image processing using a machine learning model. [Example 2] 11 and 12 are a block diagram and an external view, respectively, of an image processing system 300 according to this embodiment. The image processing system 300 includes a training device 301, an image capture device 302, and an image processing device 303. The training device 301 and the image processing device 303, and the image processing device 303 and the image capture device 302, are connected via a wired or wireless network. The image capture device 302 includes an optical system 321, an image sensor 322, a storage unit 323, a communication unit 324, and a display unit 325. The captured image is transmitted to the image processing device 303 via the communication unit 324. The image processing device 303 receives the captured image via the communication unit 332 and, using information on the configuration and weights of the machine learning model stored in the storage unit 331, sharpens the blur of the in-focus plane in the captured image and shapes the defocus blur of the out-of-focus plane. The configuration of the machine learning model and information on the weights are trained by the training device 301, and are acquired in advance from the training device 301 and stored in the storage unit 331. The image in which the blur sharpening of the in-focus plane in the captured image and the shaping of the defocus blur of the out-of-focus plane have been performed is transmitted to the imaging device 302, stored in the storage unit 323, and displayed on the display unit 325.

[0058] The generation of learning data and the update of weights (learning phase) performed by the training device 301 are the same as those in the first embodiment, and therefore will not be described here.

[0059] The image processing device may be any device having the image processing function of this embodiment, and may be realized in the form of an imaging device or a PC.

[0060] Hereinafter, with reference to FIG. 13, the sharpening of blur on an in-focus plane in a captured image and the shaping of defocus blur on an out-of-focus plane will be described, which are executed by the image processing device 303 using a trained machine learning model.

[0061] In step S311, the acquisition unit 333 acquires a captured image and a machine learning model. Information about the configuration and weight of the machine learning model is acquired from the storage unit 331. In this embodiment, two machine learning models are acquired because a CNN is used to sharpen the blur on the in-focus plane, and a diffusion model, which is one of the generative models, is used to shape the defocus blur on the out-of-focus plane.

[0062] In step S312, the acquisition unit 333 acquires information about the distance of the captured image. In this embodiment, a depth map is used as information about the distance of the captured image. In this embodiment, pixel values of the depth map range from 0 to 1, with the pixel values being 0 the farther the distance to the subject, and 1 the closer the distance to the subject. Note that the method for setting pixel values is not limited to this, and the pixel values may be 0 the farther the distance to the subject, and 255 the closer the distance to the subject.

[0063] In step S313, the acquisition unit 333 acquires focused subject information. The focused subject information is information indicating which subject is in focus in the captured image. In this embodiment, autofocus information recorded at the time of shooting is used as the focused subject information. The upper diagram of FIG. 14 shows a captured image, and the lower diagram of FIG. 14 shows an example of autofocus information. A captured image 211 includes a focused subject 212, an out-of-focus subject 213 (back blur), and an out-of-focus subject 214 (foreground blur). A dotted line 215 represents the focusable area, and a solid line 216 represents the focused area. From this information, it can be seen that the focused subject 212 is in focus in the captured image 211. Note that the focusable area shown in the lower diagram of FIG. 14 is an example and is not limited to this.

[0064] In step S314, the sharpening unit 334 changes the pixel values of the depth map based on the focused object information. The depth map acquired in step S312 is shown in the upper diagram of Fig. 15, and the depth map after the change is shown in the lower diagram of Fig. 15. In this embodiment, the pixel values of the depth map are changed so that the pixel values of the focused object regions are 1 and the pixel values of the out-of-focus object regions are 0.

[0065] In step S315, the sharpening unit 334 inputs the captured image to the machine learning model. In this embodiment, the entire captured image is input to both the CNN and the diffusion model, and the outputs are combined based on the depth map. When inputting the captured image to the machine learning model, the captured image may be divided into multiple patches before input. FIG. 16 is an explanatory diagram of the machine learning model of this embodiment. As in the first embodiment, in this embodiment, the input data 217 to the CNN is the captured image. Furthermore, the input data 218 to the diffusion model is the captured image, a noise image, and time data. There is no limitation on the input data, and additional data may be input.

[0066] In step S316, the sharpening unit 334 generates a composite image 220 based on the model output and the depth map 219. The composite image 220 is generated by weighting and adding the outputs of the CNN and the diffusion model using the depth map acquired in step S314 so that the in-focus surface becomes the output 221 and the out-of-focus surface becomes the output 222, respectively.

[0067] As described above, according to the configuration of this embodiment, it is possible to provide a highly accurate image processing method while suppressing the occurrence of false structures in image processing using a machine learning model. [Example 3] 17 and 18 are a block diagram and an external view, respectively, of an image processing system 400 of this embodiment. The image processing system 400 includes a learning device 401, a lens device 402, an imaging device 403, a control device (first device) 404, an image processing device (second device) 405, and networks 406 and 407.

[0068] The learning device 401 and image processing device 405 are, for example, servers. The control device 404 is a device operated by a user, such as a personal computer or a mobile terminal. The learning device 401 has a storage unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d. The learning device 401 updates the weights of a machine learning model that sharpens blur on an in-focus plane and shapes defocus blur on an out-of-focus plane in a captured image captured using a lens device 402 and an imaging device 403. The imaging device 403 has an imaging element 403a, which photoelectrically converts an optical image formed by the lens device 402 to acquire the captured image. The lens device 402 and the imaging device 403 are detachable and can be combined with each other in multiple types.

[0069] The control device 404 has a communication unit 404a, a display unit 404b, a storage unit 404c, and an acquisition unit 404d, and controls processing to be performed on a captured image acquired from the imaging device 403 connected via a wired or wireless connection in accordance with a user's operation. Alternatively, the captured image captured by the imaging device 403 may be stored in advance in the storage unit 404c and the captured image may be read out. The image processing device 405 has a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a processing unit 405d. The image processing device 405 sharpens the blur of the in-focus plane in the captured image and reshapes the defocus blur of the out-of-focus plane in response to a request from the control device 404 connected via a network 406. The image processing device 405 acquires weight information obtained by learning from the learning device 401 connected via the network 406 during or in advance of estimating the blur sharpening of the in-focus plane and the shaping of the defocus blur of the out-of-focus plane, and uses the information to estimate the captured image. The estimated image after the blur sharpening on the in-focus plane and the shaping of the defocus blur on the out-of-focus plane are transmitted to the control device 404, stored in the storage unit 404c, and displayed on the display unit 404b.

[0070] The image processing device may be any device having the image processing function of this embodiment, and may be realized in the form of an imaging device or a PC.

[0071] The weight update (learning phase) will be described below with reference to Fig. 19. Fig. 19 is an explanatory diagram of the machine learning model of this embodiment. In this embodiment, CNN is used for sharpening the blur on the in-focus plane, and a diffusion model, which is one of the generative models, is used for shaping the defocus blur on the out-of-focus plane.

[0072] In this embodiment, a captured image 411 and a defocus map 412 are input to a CNN, which is a first machine learning model, to generate a first estimated image 413 in which only the blur on the in-focus plane is sharpened. Next, the first estimated image 413, the defocus map 412, a noise image 414, and time data 415 are input to a diffusion model, which is a second machine learning model, to generate a second estimated image 416 in which only the defocus blur on the out-of-focus plane is shaped. By inputting the defocus map to the machine learning model, it is possible to distinguish between the object in focus and the defocus blur with high accuracy. Without a defocus map, it is impossible to distinguish between the object in focus and the defocus blur with high-frequency components, and the blur on the in-focus plane is sharpened by the CNN, while it is difficult to shape the defocus blur on the out-of-focus plane using the diffusion model.

[0073] For this reason, in this embodiment, the learning data for learning how to sharpen the blur on the in-focus plane also includes images of the defocus blur on the out-of-focus plane, and the learning data for learning how to shape the defocus blur on the out-of-focus plane also includes the blur on the in-focus plane.Then, in the CNN that learns how to sharpen the blur on the in-focus plane based on the defocus map, learning is performed to sharpen only the blur on the in-focus plane and output the defocus blur as is without doing anything to it.On the other hand, in the diffusion model that learns how to shape the defocus blur on the out-of-focus plane based on the defocus map, learning is performed to shape only the defocus blur and output the blur on the in-focus plane as is.The other learning methods are the same as in embodiments 1 and 2, so they will not be described here.

[0074] In this embodiment, an estimated image generated using CNN is input to the diffusion model, but the order may be reversed. That is, an estimated image generated using either CNN or the diffusion model may be input to the other.

[0075] The following describes the blur sharpening on the in-focus plane and the shaping of the defocus blur on the out-of-focus plane, which are executed by the control device 404 and the image processing device 405, with reference to Fig. 20. Fig. 20 is a flowchart showing the generation of the model output in this embodiment.

[0076] In step S401, the acquisition unit 404d acquires a captured image.

[0077] In step S402, the communication unit 404a transmits to the image processing device 405 a captured image and a request for execution of blur sharpening estimation processing.

[0078] In step S403, the communication unit 405a receives and acquires the captured image and processing request that have been sent.

[0079] In step S404, the acquisition unit 405b acquires information on learned weights corresponding to the captured image from the storage unit 405c. The weight information has been read out from the storage unit 401a in advance and stored in the storage unit 405c.

[0080] In step S405, the acquisition unit 405b acquires input data, which includes a defocus map, a noise image, and time data.

[0081] In step S406, the processing unit 405d uses the machine learning model to generate an estimated image (model output) from the captured image in which blur sharpening on the in-focus plane and shaping of defocus blur on the out-of-focus plane have been performed.

[0082] In step S407, the communication unit 405a transmits the estimated image to the control device 404.

[0083] In step S408, the communication unit 404a acquires the transmitted estimated image.

[0084] As described above, according to the configuration of this embodiment, it is possible to provide a highly accurate image processing method while suppressing the occurrence of false structures in image processing using a machine learning model.

[0085] In the image processing method in each embodiment described below, a first machine learning model performs image estimation processing, such as resolution enhancement, blur correction, and noise removal, on an input image to generate a first image. In training a machine learning model that performs image estimation, the difference between the output image of the machine learning model and a desired correct image is expressed using a loss function such as MSE (Mean Square Error). Minimizing the value of the loss function determines the model parameters of the machine learning model (such as the weights and biases of each layer). In general, in image estimation tasks, a solution is not uniquely determined for the input data, so even if the loss function is minimized, the value does not become zero, and a finite error remains. A second machine learning model generates this error (hereinafter, a residual component), weights the residual component, and adds it to a first image to generate an estimated image. The residual component added to the first image during estimation is referred to as an imparted component.

[0086] Here, we consider the case where a generative model is applied to the second machine learning model. Diffusion models, in particular, are known to outperform non-generative models in regression tasks such as blur correction, depth estimation, and pixelization. However, while generative models can generate high-resolution textures, they are also known to potentially introduce unnatural structures that do not exist in the original subject.

[0087] Therefore, in the image processing method in each embodiment, the second machine learning model generates only the residual component, thereby suppressing the generation of unnatural structures, and by weighting the residual component (first additive component) and adding it to the first image (as the second additive component), it becomes possible to obtain a high-quality image while making it possible to adjust the weight for the output by the generative model.

[0088] If the authenticity of the estimated image is important, it is preferable to use the second machine learning model as a generative model. Estimating only the residual components using a generative model can suppress the occurrence of unnatural structures and reduce degradation of image quality compared to when the generative model generates the entire image.

[0089] Here, the characteristics of the diffusion model will be described. The diffusion model is a model that generates a desired image by gradually removing (reducing) noise from a noisy image, and has a diffusion process and a dediffusion process.

[0090] The diffusion process in the diffusion model is the process of gradually adding noise to the correct image to make it a complete noise image. t , the noise image at time t is ε t , the noise intensity at time t determined by the noise scheduler is β t , the image at time t-1 is x t-1 Then, noise is added at each time point according to the following equation (1).

[0091]

number

[0092] The de-diffusion process in the diffusion model is a process of gradually removing noise from a completely noisy image to finally generate a noise-free image. For example, a neural network can be used for the de-diffusion process in the diffusion model.

[0093] Here, each step in which noise is gradually added is time t, and the final time is T. The noise added at each time t is Gaussian noise. If T=1000, Gaussian noise will be added 1000 times. In other words, for one correct image, multiple noise images with different amounts of noise and time data are generated.

[0094] The amount of noise to be added at each time is determined by a noise scheduler. The noise scheduler is a parameter that controls the amount of noise to be added at each time step. Examples of noise schedulers include linear schedulers and cosine schedulers. Note that the number of steps to add noise and the noise scheduler described above are merely examples, and the configurations in each embodiment are not limited to those described above. [Example 4] An image processing system 100 according to the fourth embodiment will be described.

[0095] The first machine learning model in this embodiment generates a model output by removing (reducing) noise in an input image. Furthermore, a CNN, which is one of the non-generative models, is used as the first machine learning model, but the structure of each machine learning model is not limited to this. For example, a ViT (Vision Transformer) may be used as the non-generative model. The weight for the added component is adjusted according to the strength of noise removal or ISO sensitivity. Note that this embodiment is not limited to this, and similar effects can be obtained for tasks other than noise removal (reduction). Examples of other tasks include blur and aberration correction, and high resolution (upsampling).

[0096] In this embodiment, a diffusion model, which is a type of generative model, is used as the second machine learning model, but the structure of each machine learning model is not limited to this. For example, a VAE or GAN may be used as a generative model. Furthermore, the first machine learning model may be a generative model, and the second machine learning model may be a non-generative model.

[0097] Fig. 1 shows the configuration of an image processing system 100 in this embodiment. Fig. 2 shows the appearance of the image processing system 100. The configuration of the image processing system 100 in this embodiment is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0098] Next, a method for training a first machine learning model (a method for generating a trained model) executed by the training device 101 in this embodiment will be described with reference to Fig. 21 and Fig. 22. Fig. 21 is a conceptual diagram showing the learning (training) of a machine learning model. Fig. 22 is a flowchart related to the learning (training) of a first machine learning model. Note that the first machine learning model in this embodiment is a CNN that performs image processing to remove (reduce) noise in an input image.

[0099] In step S411, the acquisition unit 101b acquires one or more pairs of first training input data and a correct answer image from the storage unit 101a.

[0100] In this embodiment, the first training input data is a high-noise image containing noise, and the ground truth image is a low-noise image containing the same subject as the high-noise image, with noise removed from the training input data. The first training input data may also contain other images or maps in addition to the noisy image, such as a map representing the intensity of noise. Pairs of low-noise and high-noise images can be prepared using live photography, imaging simulation, CG (Computer Graphics), or the like.

[0101] In step S412, the calculation unit 101c inputs the first training input data to the first machine learning model to generate the first model output 12. In this embodiment, the first model output 12 is a low-noise image obtained by reducing noise from a high-noise image.

[0102] The first machine learning model in this embodiment is a CNN that uses a convolution of an input and a filter as a linear sum. The value of each element of a filter in a CNN corresponds to a weight. Note that a sum with a bias may also be included. Furthermore, in each layer, nonlinear transformation is performed using an activation function such as a ReLU (Rectified Linear Unit) or a sigmoid function as needed. Furthermore, the first machine learning model may have a residual block or a skip connection (also called a shortcut connection) as needed.

[0103] In step S413, the update unit 101d updates the weights of the first machine learning model using an error function. In this embodiment, an error function based on the error (Loss) between the first model output 12 and the correct image is used. Note that MSE (Mean Squared Error) is used to calculate the error. However, the error function is not limited to this. Also, backpropagation or the like can be used to update the weights using the error function. Also, the error may be taken for the difference component with the high-noise image. In this case, the error between the difference component between the first model output 12 and the high-noise image 11 and the difference component between the correct image and the high-noise image 11 is used.

[0104] In step S414, the update unit 101d determines whether training of the first machine learning model is complete. Completion can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined in step S4104 that training is not complete, the process returns to step S4101, and the acquisition unit 101b acquires one or more pairs of new first training input data and a ground truth image. On the other hand, if it is determined that training is complete, the update unit 101d ends training and stores the configuration of the first machine learning model and weight information in the storage unit 101a.

[0105] Next, a method for training a second machine learning model (a method for generating a trained model) executed by the training device 101 in this embodiment will be described with reference to Fig. 23 and Fig. 24. Fig. 23 is a conceptual diagram showing the training of a machine learning model. Fig. 24 is a flowchart related to the training of a second machine learning model. Note that the second machine learning model in this embodiment is a diffusion model that performs image processing to generate residual components.

[0106] In step S421, the acquisition unit 101b acquires one or more pairs of second training input data and correct residual components from the storage unit 101a. In this embodiment, the first training input data includes high-noise images, and the output images of the first machine learning model are low-noise images. This configuration is preferable because it makes it easier to suppress adverse effects associated with noise. It is also preferable that the second training input data includes output images of the trained first machine learning model. This configuration can improve the accuracy of image processing in the estimation process.

[0107] Alternatively, an output image of the first machine learning model that is currently being trained may be used as the second training input data, and it is more preferable to use an output image of the first machine learning model that has completed training. This configuration is preferable because the output image of the first machine learning model that has completed training has smaller components generated by the second machine learning model (the difference between the input image of the second machine learning model and the correct image), enabling the generation of an estimated image of higher quality.

[0108] Note that the method for training the first and second machine learning models is not limited to this. For example, the first and second machine learning models may be jointly trained, or each machine learning model may be trained alternately. The correct residual component is a component (image) obtained by taking the difference between the output image of the first machine learning model and the corresponding correct image.

[0109] In step S422, the calculation unit 101c adds noise to the correct residual component. In this embodiment, a sign scheduler is used to determine the amount of noise to be added to the correct residual component at each time.

[0110] In step S423, the calculation unit 101c generates a second model output based on the second training input data using the second machine learning model. The input data to the second machine learning model in this embodiment are the second training input data 13, the noise image 14 at time t generated in step S422, and time data 15 representing time t. The time data may be in the form of a scalar value or a two-dimensional map. There are no restrictions on the position at which the time data is input to the neural network; the time data may be input from the same layer as the second training input data 13, or may be input to an intermediate layer.

[0111] In step S424, the update unit 101d updates the weights of the second machine learning model based on the error function. In this embodiment, an error function based on the error (Loss) between the second model output 16 and the correct model output is used. Here, when a noise image at time t is input, the correct model output is the noise image at time t-1. Note that MSE (Mean Squared Error) is used to calculate the error. However, the error function is not limited to this. Also, error backpropagation or the like can be used to update the weights using the error function.

[0112] In step S425, the update unit 101d determines whether training of the second machine learning model is complete. Completion can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined in step S425 that training is not complete, the process returns to step S421, and the acquisition unit 101b acquires one or more new sets of second training input data and correct residual component images. On the other hand, if it is determined that training is complete, the update unit 101d ends training and stores the configuration of the second machine learning model and weight information in the storage unit 101a.

[0113] Next, generation of an estimated image using the trained first and second machine learning models executed by the image processing device 103 in this embodiment will be described with reference to Fig. 25 and Fig. 26. Fig. 25 is a conceptual diagram showing image processing using the first and second machine learning models. Fig. 26 is a flowchart related to image processing using the first and second machine learning models.

[0114] In step S431, the acquisition unit 103b acquires a captured image, a first machine learning model, and a second machine learning model. Information on the configuration and weight of each machine learning model is acquired from the storage unit 103a. Here, the captured image is an image obtained by capturing an image using the imaging device 102 having an imaging optical system (optical system 102a). The captured image may be expressed in grayscale (an image containing only luminance components) or may have channel components corresponding to multiple colors. In the first embodiment, a CNN is used to remove noise from the captured image, and a diffusion model, which is one of the generative models, is used to generate the first imparting component, so two machine learning models are acquired. Note that there is no limit to the number of machine learning models to be acquired, and three or more machine learning models may be acquired and the following processing may be performed using the three or more machine learning models.

[0115] In step S432, the acquisition unit 103b acquires adjustment parameters (weights). In this embodiment, information related to the ISO sensitivity associated with the captured image is acquired as information for determining the adjustment parameters, and the adjustment parameters are acquired based on the information related to the ISO sensitivity. The information related to the ISO sensitivity represents the ISO sensitivity set when the imaging device 102 captured the captured image, and may be the ISO sensitivity value itself or a value converted from the ISO sensitivity value. Note that the information for determining the adjustment parameters is not limited to this. For example, the information related to the strength of noise reduction associated with the first machine learning model may also be used. The information related to the strength of noise reduction represents the strength of noise reduction performed by the first machine learning model, and may be expressed as a scalar value or may be text information indicating the level, such as "weak," "medium," or "strong."

[0116] In step S433, the processing unit 103c generates an estimated image 25 based on the captured image 21 using the first machine learning model, the second machine learning model, and the adjustment parameters.

[0117] First, the first machine learning model generates a low-noise image (first image) 22 based on a captured image 21 (input image).

[0118] Next, the second machine learning model generates a first imparting component 23 based on the first image 22. The imparting component in the estimation process corresponds to the residual component in the training process. In step S422, if the final time is set to T=1000 (noise impartation is repeated 1000 times), the dediffusion process of the second machine learning model is repeated 1000 times to gradually remove noise, thereby generating the first imparting component 23. The number of times noise is imparted and the number of times the dediffusion process is performed do not necessarily need to match, and certain times may be skipped to shorten the processing time.

[0119] Next, a weight for the first addition component 23 is determined based on information related to the adjustment parameters. In this embodiment, the weight value ranges from 0 to 1. For example, when adjusting based on ISO sensitivity, the higher the ISO sensitivity, the more likely high-frequency components are to disappear in the first image 22 after noise removal, so the weight for the addition component is set closer to 1. When adjusting based on noise removal strength, similarly, the higher the value of noise removal strength, the more likely high-frequency components are to disappear, so the weight for the addition component is set closer to 1. Here, when the information indicating noise removal strength is text information, the stronger the condition for noise removal strength, the closer the weight for the addition component should be to 1.

[0120] Finally, the weighted contribution component (second contribution component 24) is summed with the first image 22 to generate an estimated image 25.

[0121] In this embodiment, the estimated image 25 is generated by adding the weighted second imparting component 24 to the first image 22, but the method of generating the estimated image is not limited to this. For example, the first imparting component 23 may be generated by performing the de-diffusion process a number of times according to information related to the adjustment parameter and added to the first image 22. Alternatively, multiple first imparting components 23 may be generated based on information related to the adjustment parameter, their average value may be calculated, and the average value of the multiple first imparting components 23 may be added to the first image 22 as the second imparting component 24. As a modified example, the first image 22 may be weighted by the adjustment parameter to generate a second image, and the second image and the first imparting component 23 may be added to generate an estimated image.

[0122] With the above configuration, it is possible to provide an image processing method etc. that can generate high-quality images using multiple machine learning models. [Example 5] An image processing system 500 in the fifth embodiment will be described with reference to Fig. 27. Fig. 27 is a block diagram of the image processing system 500 in the fifth embodiment. The image processing system 500 includes a training device 501 and an imaging device 502. The configuration of the training device 501 in the present embodiment is the same as that of the training device in the fourth embodiment.

[0123] The image capturing device 502 in this embodiment performs blur correction on an input image using a machine learning model. Note that in this embodiment, the weight of the first additive component is adjusted according to at least one of the image capturing conditions, the correction conditions of the first machine learning model 20a, the optical characteristics, and the image characteristics.

[0124] The imaging device 502 has an optical system 521, an imaging element 522, an image estimation unit 523, a memory unit 5224, a recording medium 525, a display unit 528, an input unit 526, and a system controller 527. The imaging device 502 captures an image of the subject space to obtain a captured image and generates an estimated image. The optical system 5221 and the imaging element 522 in the imaging device 5202 are the same as those in Example 4, and therefore their description will be omitted. The imaging device 502 also reads out weight information for the trained first machine learning model 20a and second machine learning model 20b from the training device 501 via the network 503 and stores the information in the memory unit 5224.

[0125] The image estimation unit 523 has an acquisition unit 523a and a processing unit 523b. The acquisition unit 523a acquires a captured image, etc. The processing unit 523b is similar to the processing unit 103c in the fourth embodiment. The captured image acquired by the acquisition unit 5223a is subjected to image processing based on the weight information stored in the storage unit 524, and an estimated image is generated.

[0126] The recording medium 525 stores the estimated image. When a user issues an instruction to display the estimated image via the input unit 526, the stored output image is read out and displayed on the display unit 528. The image estimation unit 523 may read out the captured image stored in the recording medium 525 and perform processing to generate an estimated image. The system controller 527 controls processing performed by the imaging device 502.

[0127] The image estimation unit 523 performs blur correction processing on the input image using the first machine learning model 20a to generate a first image 22, and uses the second machine learning model 20b to generate a first imparted component 23. The image estimation unit 523 in this embodiment determines weights based on the intensity of blur correction or the imaging conditions (information related to adjustment parameters), and generates an estimated image 25 by taking the sum of the second imparted component 24 generated by the weighting and the first image 22.

[0128] The image capturing conditions include information indicating the ISO sensitivity, the focal length of the optical system 521, the aperture value, or the object distance that was set when the image capturing device 502 captured an image.

[0129] The optical characteristics are characteristics based on the optical system 521, and are obtained based on a point spread function (PSF) and an optical transfer function (OTF). Note that the optical characteristics in this embodiment may be any characteristics that indicate image blur due to aberration or diffraction of the optical system. For example, the optical characteristics may be a modulation transfer function (MTF), which is the amplitude component of the OTF, or a phase transfer function (PTF), which is the phase component of the OTF.

[0130] The correction condition is information indicating the strength of blur correction performed by the first machine learning model 20a. Note that, when the process performed by the first machine learning model 20a is resolution enhancement, the correction condition preferably indicates the resolution enhancement magnification or the strength of the resolution enhancement.

[0131] The image characteristics are determined based on the magnitude of blur or shake contained in the captured image. The magnitude of blur can be determined by estimating the PSF. The magnitude of shake can be determined by an acceleration sensor (not shown) or the like included in the image capture device 502.

[0132] As in the fourth embodiment, a training method for the first machine learning model 20a executed by the training device 501 in this embodiment will be described with reference to Fig. 22. The first machine learning model 20a in this embodiment is a CNN that corrects image blur. The first training input data includes blurred images that have been blurred due to the optical characteristics of the optical system 102a, and the target image is an image that is less blurred than the blurred image.

[0133] The first machine learning model 20a is not limited to a CNN that corrects image blurring, but may be a model that increases the resolution of an image. In this case, the first training input data includes low-resolution images, and the target image is a high-resolution image that contains the same subject as the low-resolution image.

[0134] The processing by the training device 501 (steps S411 to S414) is the same as the processing using the training device 101 in the fourth embodiment, and therefore will not be described.

[0135] As in Example 4, the training method for the second machine learning model 20b executed by the training device 5201 in this example will be described with reference to Fig. 24. The second machine learning model 20b in this example is a diffusion model that learns to generate a first impartial component.

[0136] The second training input data in this embodiment is the same as the first training input data. Furthermore, in step S421, the acquisition unit 101b acquires one or more pairs of second training input data and a correct residual component image from the storage unit 101a. With this configuration, even if adverse effects occur in the processing of the first machine learning model 20a due to erroneous estimation or the like, it becomes possible to train the second machine learning model 20b regardless of the processing of the first machine learning model 20a. Note that the correct residual component in this embodiment is an image obtained by taking the difference between the output image of the first machine learning model 20a and the corresponding correct image.

[0137] The processing by the training device 501 (steps S422 to S425) is the same as the processing using the training device 101 in the fourth embodiment, and therefore will not be described.

[0138] The image processing in this embodiment uses a first machine learning model 20a, which is a non-generative model, and a second machine learning model 20b, which is a generative model. The first machine learning model 20a generates a first image 22 from an input image 21, and the second machine learning model 20b generates a first impartial component from the first image 22 (or the input image 21). The estimated image 25 is then generated based on the first image 22, the first impartial component, and the adjustment parameters. By generating only the first impartial component (residual component) using the second machine learning model 20b, which is a generative model, it is possible to suppress the generation of unnatural structures by the generative model. As a result, it is possible to provide an image processing method, etc., capable of generating high-quality images. [Example 6] Next, with reference to Figures 17, 28, and 29, generation of an estimated image using the trained first machine learning model 30a and second machine learning model 30b executed by the image processing device 405 in this embodiment will be described. Figure 28 is a flowchart related to image processing using the first machine learning model 30a and the second machine learning model 30b. Figure 29 is a flowchart related to image processing using the first machine learning model 30a and the second machine learning model 30b. Figure 17 is the same as in Example 3, so detailed description will be omitted.

[0139] In this embodiment, the first machine learning model 30a performs processing to correct blur in the captured image, and the second machine learning model 30b generates a first impartial component 33. The estimated image is generated based on adjustment parameters (weights). Here, the first impartial component 33 corresponds to a residual component during training.

[0140] In the fourth and fifth embodiments, examples were shown in which the weight for the first imparting component 33 was determined based on each condition, but in this embodiment, the adjustment parameter (weight) is determined based on a value (information on the adjustment parameter) designated by the user. In this case, the user can change the designated value while checking the result of the estimated image generated on the display device 104.

[0141] First, a description will be given of the operation of the control device 404. In this embodiment, image processing is started by a user issuing an instruction to start image processing via the control device 404.

[0142] In step S601 (first transmission step), the communication unit 405a transmits a request for processing the captured image to the image processing device 303. Note that in step S601, the control device 404 may transmit an ID for authenticating a user, shooting conditions corresponding to the captured image, and the like, together with the request for processing the captured image.

[0143] In step S602 (first receiving step), the communication unit 405a receives the output image generated by the image processing device 405.

[0144] In step S603, the control device 404 confirms receipt of the estimated image and determines whether to end the process. At this time, the process may be ended by a user operation as a trigger. For example, if the user inputs information regarding a different value for the adjustment parameter after step S604 described below, the process returns to step S603, and the acquisition unit 404d acquires a new value specified by the user. If the user does not re-specify a value after step S604, the process may proceed to step S603 and end the process of generating an estimated image.

[0145] Next, a description will be given of the operation of the image processing device 405. The communication unit 303f receives a request for processing the captured image 31 transmitted from the communication unit 405a. Upon receiving the instruction to process the captured image, the image processing device 405 executes the processing from step S601 onwards.

[0146] In step S611, the acquisition unit 405b acquires (weight information of) the first machine learning model 30a, (weight information of) the second machine learning model 30b, and the captured image 31. In this embodiment, the captured image 31 is transmitted from the control device 404. At this time, the imaging conditions corresponding to the captured image 31 may be acquired together with the captured image 31.

[0147] In step S612, the processing unit 405d generates a first image 32 using the first machine learning model 30a, and generates a first imparting component 33 using the second machine learning model 30b. Note that the dediffusion process of the diffusion model in generating the first imparting component 33 is the same as in the fourth embodiment.

[0148] In step S613, the acquisition unit 303b acquires information about the adjustment parameter. In this embodiment, the value acquired is a value designated by the user. The user can designate the value, for example, via an input unit (not shown) in the control device 404. The adjustment parameter may be the value designated by the user itself, or a converted value. For example, the value designated by the user may be converted into a value between 0 and 1, and the converted value may be applied as a weight to the additional component.

[0149] In step S614, the processing unit 405d generates a second assignment component 34 based on the first assignment component 33 and the weight.

[0150] In step S615, an estimated image 35 is generated based on the first image 32 and the second impartial component 34. In this embodiment, the estimated image 35 is generated by taking the sum of the first image 32 and the second impartial component 34.

[0151] In step S616, the processing unit 405d determines whether to terminate the process of generating the estimated image 35. The processing unit 405d determines whether to terminate the process by parameterizing, for example, the blur or the amount of artifacts in the estimated image 35 and comparing the parameterized amount with a threshold. The processing unit 303c may also display the generated estimated image 35 to the user and determine to terminate the process if the user does not re-input adjustment parameters.

[0152] When the first machine learning model 30a performs blur correction, the adjustment parameters may be determined based on information indicating the focal length, aperture, and object distance of the optical system 102a set when the captured image was captured, and the optical characteristics of the optical system in the lens device 402. In this case, the weight for the first additive component should be closer to 1 as the diameter of the PSF determined from the imaging conditions and optical characteristics becomes larger. Similarly, when adjustment is made based on image characteristics, the weight for the additive component should be closer to 1 as the PSF of the estimated blur becomes larger.

[0153] With the above configuration, it is possible to provide an image processing method and the like that is capable of generating high-quality images using two machine learning models.

[0154] [Other Examples] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0155] The disclosure of this embodiment includes the following methods and configurations. (Method 1) generating an estimated image from an input image using a plurality of machine learning models, including generative and non-generative models; An image processing method characterized in that in the step, the estimated image is generated by varying the weights of the output of the generative model and the output of the non-generative model for each of multiple regions of the input image based on information about the input image. (Method 2) The image processing method according to Method 1, characterized in that the information about the input image includes at least one of information about the distance of the input image, a segmentation map, information about saturated regions, and an optical performance map. (Method 3) The image processing method according to Method 2, wherein the information about the distance of the input image is a defocus map or a depth map. (Method 4) the information about the input image includes information about a distance of the input image; The estimated image is a weight of the output of the non-generative model is made greater than a weight of the output of the generative model for an area determined to be a focused area among the plurality of areas based on information about the distance of the input image; An image processing method described in any one of methods 1 to 3, characterized in that for areas among the multiple areas determined to be out-of-focus areas based on information regarding the distance of the input image, the weight of the output of the generative model is generated to be greater than the weight of the output of the non-generative model. (Method 5) the information about the input image includes a segmentation map; The estimated image is a weight of the output of the non-generative model is increased relative to a weight of the output of the generative model for a region determined to include a person among the plurality of regions based on the segmentation map; An image processing method described in any one of methods 1 to 3, characterized in that for areas of the plurality of areas determined to not contain a person based on the segmentation map, the weight of the output of the generative model is increased compared to the weight of the output of the non-generative model. (Method 6) The estimated image is a region determined to be a first region among the plurality of regions based on information about the input image is generated using an output of the non-generative model; An image processing method described in any one of methods 1 to 5, characterized in that a region determined to be a second region among the plurality of regions based on information about the input image is generated using the output of the generative model. (Method 7) An image processing method described in any one of methods 1 to 5, characterized in that in the above step, the estimated image is generated by taking a weighted average of a first image generated by inputting the input image into the generative model and a second image generated by inputting the input image into the non-generative model based on information about the input image. (Method 8) In the above step, generating a first image by inputting the input image and information about the input image into one of the generative model and the non-generative model; An image processing method described in any one of configurations 1 to 5, characterized in that the estimated image is generated by inputting the first image and information about the input image into the other of the generative model and the non-generative model. (Method 9) 6. The image processing method according to any one of the methods of configurations 1 to 5, wherein the estimated image is an image having a shape of defocus blur different from a shape of defocus blur of the input image. (Method 10) generating a first image based on the input image using a first machine learning model; generating a first contribution component based on the input image or the first image using a second machine learning model; obtaining an adjustment parameter for the first additive component; generating a second imparting component based on the first imparting component and the adjustment parameter; generating an estimated image based on the first image and the second additive component. (Method 11) generating a first image based on the input image using a first machine learning model; generating a first contribution component based on the input image or the first image using a second machine learning model; obtaining adjustment parameters for the first image; generating a second image based on the first image and the adjustment parameters; generating an estimated image based on the second image and the first additive component. (Method 12) 11. The image processing method of claim 10, wherein the estimated image is generated by summing the second additive component and the first image. (Method 13) the second machine learning model generates two or more first contribution components based on the adjustment parameters; 11. The image processing method according to claim 10, wherein the second imparting component is an average value of the two or more first imparting components. (Method 14) An image processing method described in any one of methods 10 to 13, characterized in that the adjustment parameters are values determined based on at least one of shooting conditions, correction conditions of the first machine learning model, optical characteristics, and image characteristics. (Method 15) 15. The image processing method according to method 14, wherein the shooting condition is ISO sensitivity. (Method 16) 15. The image processing method according to method 14, wherein the shooting conditions are at least one of focal length, aperture value, and object distance. (Method 17) 15. The image processing method according to method 14, wherein the correction condition is a resolution increase magnification, a resolution increase strength, a blur correction strength, or a noise removal strength. (Method 18) 15. The image processing method of claim 14, wherein the optical characteristics are characteristics of an optical system used to acquire the input image. (Method 19) 15. The image processing method of claim 14, wherein the image characteristics are determined based on the amount of blur present in the input image. (Method 20) An image processing method described in any one of methods 10 to 19, characterized in that the second machine learning model generates the first attribute component by performing a number of dediffusion processes according to the adjustment parameters. (Method 21) 20. An image processing method according to any one of methods 10 to 19, wherein the second machine learning model is trained using the output of the first machine learning model. (Method 22) generating a first image based on an input image using a first machine learning model, the first machine learning model being a non-generative model; generating a first contribution component based on the input image or the first image using a second machine learning model that is a generative model; obtaining adjustment parameters; an image processing method comprising the step of generating an estimated image based on the first image, the first additive component, and the adjustment parameters; (Configuration 1) A program that causes a computer to execute the image processing method according to any one of Methods 1 to 22. (Configuration 2) A computer-readable storage medium storing the program according to configuration 1.

[0156] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0157] 103 Estimation device (image processing device) 103b Acquisition Department 103c Processing section

Claims

1. generating an estimated image from an input image using a plurality of machine learning models, including generative and non-generative models; An image processing method characterized in that in the step, the estimated image is generated by varying the weights of the output of the generative model and the output of the non-generative model for each of multiple regions of the input image based on information about the input image.

2. 2. The image processing method according to claim 1, wherein the information about the input image includes at least one of information about the distance of the input image, a segmentation map, information about a saturated region, and an optical performance map.

3. 3. The image processing method according to claim 2, wherein the information about the distance of the input image is a defocus map or a depth map.

4. the information about the input image includes information about a distance of the input image; The estimated image is a weight of the output of the non-generative model is made greater than a weight of the output of the generative model for an area determined to be a focused area among the plurality of areas based on information about the distance of the input image; The image processing method described in claim 1, characterized in that for areas among the multiple areas determined to be out-of-focus areas based on information about the distance of the input image, the weight of the output of the generative model is generated greater than the weight of the output of the non-generative model.

5. the information about the input image includes a segmentation map; The estimated image is a weight of the output of the non-generative model is increased relative to a weight of the output of the generative model for a region determined to include a person among the plurality of regions based on the segmentation map; The image processing method described in claim 1, characterized in that for areas of the plurality of areas determined to not contain a person based on the segmentation map, the weight of the output of the generative model is increased compared to the weight of the output of the non-generative model.

6. The estimated image is a region determined to be a first region among the plurality of regions based on information about the input image is generated using an output of the non-generative model; An image processing method according to any one of claims 1 to 5, characterized in that a region determined to be a second region among the plurality of regions based on information about the input image is generated using the output of the generative model.

7. An image processing method according to any one of claims 1 to 5, characterized in that in the step, the estimated image is generated by taking a weighted average of a first image generated by inputting the input image into the generative model and a second image generated by inputting the input image into the non-generative model based on information about the input image.

8. In the above step, generating a first image by inputting the input image and information about the input image into one of the generative model and the non-generative model; An image processing method according to any one of claims 1 to 5, characterized in that the estimated image is generated by inputting the first image and information about the input image into the other of the generative model and the non-generative model.

9. 6. The image processing method according to claim 1, wherein the estimated image is an image having a shape of defocus blur different from a shape of defocus blur of the input image.

10. generating a first image based on the input image using a first machine learning model; generating a first contribution component based on the input image or the first image using a second machine learning model; obtaining an adjustment parameter for the first added component; generating a second imparting component based on the first imparting component and the adjustment parameter; generating an estimated image based on the first image and the second additive component.

11. generating a first image based on the input image using a first machine learning model; generating a first contribution component based on the input image or the first image using a second machine learning model; obtaining adjustment parameters for the first image; generating a second image based on the first image and the adjustment parameters; generating an estimated image based on the second image and the first additive component.

12. 11. The image processing method according to claim 10, wherein the estimated image is generated by taking the sum of the second additive component and the first image.

13. the second machine learning model generates two or more first contribution components based on the adjustment parameters; 11. The image processing method according to claim 10, wherein the second imparted component is an average value of the two or more first imparted components.

14. 14. The image processing method according to claim 10, wherein the adjustment parameter is a value determined based on at least one of shooting conditions, correction conditions of the first machine learning model, optical characteristics, and image characteristics.

15. 15. The image processing method according to claim 14, wherein the photographing condition is ISO sensitivity.

16. 15. The image processing method according to claim 14, wherein the photographing condition is at least one of a focal length, an aperture value, and an object distance.

17. 15. The image processing method according to claim 14, wherein the correction condition is a resolution increasing magnification, a resolution increasing strength, a blur correction strength, or a noise removal strength.

18. 15. The image processing method according to claim 14, wherein the optical characteristics are characteristics of an optical system used to acquire the input image.

19. 15. The image processing method according to claim 14, wherein the image characteristics are determined based on the magnitude of blur contained in the input image.

20. The image processing method according to any one of claims 10 to 13, wherein the second machine learning model generates the first additive component by performing a dediffusion process a number of times according to the adjustment parameter.

21. 14. The image processing method according to claim 10, wherein the second machine learning model is trained using an output of the first machine learning model.

22. generating a first image based on an input image using a first machine learning model, the first machine learning model being a non-generative model; generating a first contribution component based on the input image or the first image using a second machine learning model that is a generative model; obtaining adjustment parameters; generating an estimated image based on the first image, the first additive component, and the adjustment parameters.

23. 23. A program for causing a computer to execute the image processing method according to any one of claims 1 to 5, 10 to 13, and 22.

24. A computer-readable storage medium storing the program according to claim 23.

Images