Image processing method, learned model generation method, image processing apparatus, and storage medium

Abstract

To provide an image processing method capable of acquiring a machine learning model whose estimation accuracy is hardly affected by the magnitude of a signal of a developed image.SOLUTION: The method includes acquiring a training image and a correct image, inputting the training image to a machine learning model to generate an output image, acquiring a weighting coefficient, calculating an error using the correct image, the output image, and the weighting coefficient, and updating a parameter of the machine learning model based on the error, wherein the weighting coefficient changes according to a signal value of at least one of the training image, the correct image, or the output image when the output image and the correct image are not residuals from the training image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing method, a trained model generation method, an image processing device, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a network configuration that can be generally applied to various regression problems. Non-Patent Document 1 also discloses that the network can be used to perform upsampling of input images, JPEG deblocking (removal of compression noise), denoising, non-blind deblurring, or inpainting. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] X. Mao, C. Shen, Y. Yang, “Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections”, https: / / arxiv.org / abs / 1606.08921. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method disclosed in Non-Patent Document 1 cannot perform appropriate estimation when the input image is a RAW image. Incidentally, when a RAW image is input into a machine learning model to estimate a desired output image, the machine learning model must also be trained using the RAW image. In training, the machine learning model is optimized by inputting a training RAW image into the machine learning model and minimizing the error between the output obtained and the correct RAW image. Therefore, the error during training is minimized in the RAW image state (where the light intensity and signal value are approximately linear). However, when a user actually views the developed output image, gamma correction is performed, and the error changes depending on the magnitude of the signal value. Specifically, the error is magnified in dark areas of the estimated image, reducing the estimation accuracy.

[0005] Therefore, an object of the present invention is to provide an image processing method capable of obtaining a machine learning model whose estimation accuracy is less affected by the magnitude of the signal of the developed image. [Means for solving the problem]

[0006] An image processing method as one aspect of the present invention includes the steps of acquiring a training image and a reference image, inputting the training image into a machine learning model to generate an output image, acquiring a weighting coefficient, calculating an error using the reference image, the output image, and the weighting coefficient, and updating parameters of the machine learning model based on the error, wherein if the output image and the reference image are not residuals from the training image, the weighting coefficient changes depending on at least one signal value of the training image, the reference image, or the output image.

[0007] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing method capable of acquiring a machine learning model whose estimation accuracy is less affected by the magnitude of the signal of the developed image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a configuration diagram of a machine learning model in a first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 1 is a flowchart of a learning process of a machine learning model in Example 1. [Figure 5] 3A and 3B are explanatory diagrams of gamma correction in the first and second embodiments. [Figure 6] FIG. 10 is a diagram showing the relationship between signal values ​​and weighting coefficients in the first and second embodiments. [Figure 7] 10 is an explanatory diagram of signal values ​​and weighting coefficients in the first and second embodiments. FIG. [Figure 8] 1 is a flowchart of an estimation process of a machine learning model in Examples 1 and 2. [Figure 9] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 10] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 11] 10 is a flowchart of a learning process of a machine learning model in Example 2. [Figure 12] FIG. 10 is a diagram showing the relationship between the difference between a correct image and an output image and a weighting coefficient in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0011] In the following, image processing using a machine learning model involves two processes: a processing process for updating parameters (weights and biases) and a processing process for making inferences about unknown inputs using the updated parameters. Hereinafter, the former will be referred to as the learning process, and the latter will be referred to as the estimation process. Each embodiment has its own characteristics in the learning process. Furthermore, the value used as an index when updating the parameters of the machine learning model will be referred to as the error, and the value obtained by simply subtracting images will be referred to as the difference. In the learning process, the machine learning model updates its parameters to minimize the error.

[0012] Next, we will define the names of the images used in the learning and estimation processes. The image input to the machine learning model is called the input image, and the input image used in the learning process, for which the correct answer image is known, is called the training image. The image output from the network is called the output image, and the output image in the estimation process is called the estimated image. The input image, output image, and correct answer image are RAW images, and are images before gamma correction.

[0013] Here, a RAW image is undeveloped image data output from an image sensor, and the light intensity and signal value of each pixel have a roughly linear relationship. The RAW image is developed before the user views the image, during which gamma correction is performed. Gamma correction is a process of raising the input signal value to a power, such as 1 / 2.2.

[0014] Before describing each embodiment in detail, the gist of each embodiment will be described. Each embodiment considers the influence of gamma correction in the learning process of a machine learning model that uses a RAW image as input. This allows the estimated image after development to maintain a roughly consistent estimation accuracy regardless of the magnitude of the signal value. In particular, undershoot and ringing, which tend to occur with high resolution (upsampling and deblurring), are suppressed. Examples of machine learning models include a convolutional neural network (CNN), a generative adversarial network (GAN), and a diffusion model. As a method for considering gamma correction during learning, each embodiment determines a weighting coefficient based on the signal value of at least one of a target image, an output image, and a training image, and calculates an error using the weighting coefficient and the difference between the target image and the output image to perform learning. Specifically, the weight in the dark region is increased. This makes it possible to train a machine learning model whose estimation accuracy is less affected by the signal magnitude of the developed image. [Example]

[0015] First, an image processing system 100 according to a first embodiment of the present invention will be described. In this embodiment, the task performed by the machine learning model is to sharpen (deblur) blur in a captured image. The blur to be sharpened is aberration, diffraction, and blur caused by an optical low-pass filter that occur in the optical system. However, the effect of this embodiment can be similarly obtained when sharpening blur caused by pixel aperture, defocus, or shaking. Furthermore, this embodiment can be similarly implemented to obtain the effect for tasks other than blur sharpening.

[0016] FIG. 2 is a block diagram of the image processing system 100. FIG. 3 is an external view of the image processing system 100. The image processing system 100 includes a training device 101 and an image processing device 103, which are connected via a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 via wired or wireless connections. A captured image of a subject space captured using the imaging device 102 is input to the image processing device 103. Blurring occurs in the captured image due to aberration and diffraction caused by the optical system 102a in the imaging device 102 and the optical low-pass filter of the image sensor 102b, resulting in attenuation of subject information. The image processing device 103 performs blur sharpening on the captured image using a machine learning model to generate a blur-sharpened image. The machine learning model is trained by the training device 101, and the image processing device 103 acquires information about the machine learning model from the training device 101 in advance and stores it in a storage unit 103a. The blur-sharpened image is stored in the storage unit 103a or the recording medium 105, and is output to an output device 106 such as a printer as needed.

[0017] Next, the learning process of the machine learning model (a method for generating a trained model) executed by the training device (image processing device) 101 will be described with reference to Figure 4. Figure 4 is a flowchart of the learning process of the machine learning model. The training device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit (generation unit) 101c, and an update unit 101d, and any of these components executes each of the following steps.

[0018] First, in step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. Since the machine learning model is trained based on the original images, it is desirable that the original images have various frequency components (edges with different orientations and intensities, gradations, flat areas, etc.). The original images may be real images or CG (Computer Graphics).

[0019] Next, in step S102, the calculation unit (generation unit) 101c applies blur to the original image to generate a blurred image. The blurred image is a training image input to the machine learning model during training and corresponds to the captured image during estimation. The blur to be applied is the blur to be sharpened. In the first embodiment, blur generated by aberration and diffraction of the optical system 102a and the optical low-pass filter of the image sensor 102b is applied. The shape of the blur caused by the aberration and diffraction of the optical system 102a varies depending on the image plane coordinates (image height and azimuth). It also varies depending on the focal length, aperture, and focus state of the optical system 102a. If it is desired to simultaneously train a machine learning model that sharpens all of these blurs, it is preferable to generate multiple blurred images using multiple blurs generated by the optical system 102a. If necessary, noise generated by the image sensor 102b may be applied to the blurred image.

[0020] Next, in step S103, the acquisition unit 101b acquires a correct image. In the first embodiment, since the task is blur sharpening, the correct image is an image with less blur than the blurred image. In the first embodiment, the original image is the correct image. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct image. In this case, reduction is also performed when generating training images in step S102. Furthermore, step S103 may be executed any time after step S101 and before step S104.

[0021] For processes other than blur correction, the learning process can be performed by similarly preparing pairs of training images and reference images through simulation. Regarding denoising, training images can be generated by adding expected noise to low-noise reference images. Regarding upsampling, training images are low-resolution images, and reference images are high-resolution images. Training images can be prepared by downsampling the reference images. The training images and reference images may or may not be matched in size. If training images are generated by downsampling the reference images to 1 / n, the training images can be matched in size by stretching them n times using interpolation such as bicubic. Regarding compression noise removal, training images can be generated by compressing reference images with no compression or a low compression ratio. It is not necessary to prepare pairs of training images and reference images in simulation; they can also be prepared using real-life images.

[0022] Next, in step S104, the calculation unit 101c generates an output image based on the training image using the machine learning model. Note that the signal value of the output image may exceed the range that the image signal can take. For this reason, in step S104, it is preferable to clip the signal value of the output image. If normalized, the signal value is clipped to 0 to 1, and if not normalized, the signal value is clipped to the optical black signal value and the luminance saturation value. In the first embodiment, the machine learning model shown in FIG. 1 is used, but this embodiment is not limited to this. A training image 201 is input to the machine learning model. The machine learning model has multiple layers, and in each layer, a linear sum of the layer input and weight is calculated. The initial value of the weight may be determined using a random number or the like. In the first embodiment, the machine learning model is a CNN that uses a convolution of the input and a filter as a linear sum (the value of each element of the filter corresponds to the weight, and may also include a sum with a bias), but this embodiment is not limited to this. In addition, nonlinear transformations using activation functions such as ReLU (Rectified Linear Unit) and sigmoid functions are performed in each layer as needed. Furthermore, the machine learning model may have residual blocks and skip connections (also called shortcut connections) as needed. As a result of passing through multiple layers, an output image 202 is generated.

[0023] Next, in step S105, the update unit 101d calculates the difference between the output image 202 and the correct image for each pixel. That is, at this stage, the difference becomes a two-dimensional map. In this embodiment, the update unit 101d calculates the difference S using the following equation (1):

[0024]

number

[0025] In equation (1), t is the signal value of the correct image, y is the signal value of the output image 202, and j is the pixel number. In equation (1), the signal value of the output image is subtracted from the signal value of the correct image, but the signal value of the correct image may also be subtracted from the signal value of the output image. The difference may be taken of the residual component. In the case of the residual component, the difference between the residual component between the output image 202 and the training image 201 and the residual component between the correct image and the training image 201 is used.

[0026] Next, in step S106, the update unit 101d calculates (determines) a weighting coefficient based on the signal value of the reference image after normalization, and weights the difference acquired in step S105. In this embodiment, the weighting coefficient is calculated (acquired) based on the signal value of the reference image after normalization, but this is not limiting. The weighting coefficient may be, for example, the signal value of the input image or the signal value of the output image. Alternatively, the weighting coefficient may be the signal value before normalization. Note that, when the output image is a residual component from the training image and the reference image is a residual component from the training image, the weighting coefficient is calculated (acquired) based on the signal value of the first image obtained by adding the output image to the training image, or the signal value of the second image obtained by adding the reference image to the training image.

[0027] In this embodiment, if the output image and the reference image are not residuals from the training images, the weighting coefficients vary depending on at least one signal value of the training image, the reference image, or the output image. Also, in this embodiment, if the output image and the reference image are residuals from the training images, the weighting coefficients vary depending on at least one signal value of the training image, the first image obtained by adding the output image to the training image, or the second image obtained by adding the reference image to the training image.

[0028] In this embodiment, the weighting coefficients may be determined based on the signal values ​​after clipping the output image at a predetermined value (upper limit value). Furthermore, in this embodiment, it is preferable that the weighting coefficients vary depending on the signal values ​​in the non-saturated region of each image. However, this embodiment is not limited to this, and the weighting coefficients may vary between the saturated and non-saturated regions of the image.

[0029] When inputting RAW images into a machine learning model to estimate a desired output image, the machine learning model must also be trained using RAW images. During training, the machine learning model is optimized by inputting training RAW images into the model and minimizing the error between the output obtained and the correct RAW image. Therefore, errors during training are minimized in the RAW image state (where light intensity and signal value are approximately linear). However, when a user actually views the developed output image, gamma correction is performed, causing errors to change depending on the magnitude of the signal value. Specifically, errors are magnified in dark areas of the estimated image, reducing estimation accuracy.

[0030] As shown in Fig. 5, gamma correction is a process in which a curve showing the relationship between signal values ​​before and after correction (a gamma curve shown by a solid line in Fig. 5) is positioned above a straight line with a slope of 1 (a dashed line in Fig. 5). Fig. 5 is an explanatory diagram of gamma correction. In Fig. 5, the horizontal axis represents signal values ​​before gamma correction, and the vertical axis represents signal values ​​after gamma correction.

[0031] A specific example of the function g representing the gamma correction is given by the following equation (2).

[0032]

number

[0033] In equation (2), s is a normalized signal value before gamma correction, and γ (>1) is a constant, but gamma correction is not limited to the form of equation (2).

[0034] To suppress errors in dark areas when training the machine learning model, weighting coefficients are calculated based on the signal values ​​of the correct image, and the error is calculated from the weighting coefficients and the difference between the correct image and the output image. This enables the machine learning model to train in a way that further suppresses errors in areas with larger weighting coefficients.

[0035] Fig. 6 is a diagram showing the relationship (relational formula) between the normalized signal value of the correct image and the weighting coefficient assigned to the difference. In Fig. 6, the horizontal axis represents the normalized signal value, and the vertical axis represents the weighting coefficient. As shown by the solid line 211, the smaller the normalized signal value of the correct image, the larger the weighting coefficient assigned to the difference. The solid line 211 reflects the characteristic of gamma correction that magnifies signal values ​​on the dark side, so the weighting coefficient becomes larger on the dark side.

[0036] Note that the solid line 211 is not the only example, and any relational expression may be set as long as the weighting coefficient is larger on the dark side than on the bright side, such as the dashed-dotted line 212. In this case, if the normalized signal value is 0, the weighting coefficient is 2, and if the normalized signal value is 1, the weighting coefficient is 0.5, and the difference calculated in step S105 is multiplied by the weighting coefficient.

[0037] Furthermore, instead of calculating weights corresponding to signal values ​​using a function, weights corresponding to signal values ​​may be stored as a table. For signal values ​​not listed in the table, weights can be created by interpolation using weighting coefficients corresponding to the stored preceding and following signal values. Note that when any signal value conversion process, not limited to gamma correction, is performed on an estimated image of a machine learning model, the weighting coefficients may be determined according to the characteristics of the signal value conversion process.

[0038] Fig. 7(A) shows the signal values ​​after normalization of the correct image, and Fig. 7(B) shows an example of weighting coefficients applied to the difference. The normalized signal values ​​of subjects 213 and 214 are different, with subject 213 having a smaller signal value. Therefore, the weighting coefficients applied to the difference are different, with a larger weighting coefficient applied to the difference in the area corresponding to subject 213. The above weighting coefficients are calculated for each pixel of the difference obtained in step S105, and the final error L1 can be expressed by the following equation (3).

[0039]

number

[0040] In equation (3), N represents the total number of pixels, and W1 represents a weighting coefficient. Although equation (3) uses MSE (Mean Squared Error) for the error, any other index may be used as long as it is a value that represents the error between the correct image and the output image. In the first embodiment, after calculating the difference between the correct image and the output image, the difference is multiplied by the weighting coefficient, but the order of processing is not limited to this. For example, the update unit 101d may calculate a weighting coefficient for each of the correct image and the output image, multiply the correct image and the output image by the weighting coefficient, and calculate the difference between the two differences to calculate the error.

[0041] 4, the update unit 101d updates the parameters of the machine learning model based on the error. Backpropagation or the like may be used to update the parameters.

[0042] Next, in step S108, the update unit 101d determines whether training of the machine learning model is complete. Completion can be determined by, for example, whether the number of iterations of parameter update has reached a predetermined number, or whether the amount of change in parameters during update is smaller than a predetermined value. If it is determined in step S108 that training is not complete, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, if it is determined that training is complete, the update unit 101d ends training and stores the configuration and parameter information of the machine learning model in the storage unit 101a.

[0043] Next, blur sharpening of a captured image using a trained machine learning model (trained model) (machine learning model estimation process) executed by the image processing device 103 will be described with reference to Fig. 8. Fig. 8 is a flowchart of the machine learning model estimation process. The image processing device 103 has a storage unit 103a, an acquisition unit 103b, and a sharpening unit 103c, any of which executes the following steps.

[0044] First, in step S201, the acquisition unit 103b acquires a captured image and a machine learning model. Information about the configuration and parameters of the machine learning model is acquired from the storage unit 103a.

[0045] Next, in step S202, the sharpening unit 103c uses the machine learning model to generate a blur-sharpened image (estimated image) from the captured image, in which the blur of the captured image has been sharpened. The machine learning model has the configuration shown in FIG. 1, as in the training period.

[0046] Next, preferred conditions for enhancing the effects of this embodiment will be described. First, it is preferable to provide a process for normalizing the signal values ​​of the input image and the correct image input to the machine learning model. The range of possible signal values ​​for a captured image differs depending on the configuration of the imaging device. If the range of signal values ​​in the input image differs between the learning process and the estimation process, an accurate estimation result cannot be obtained. For this reason, it is preferable to normalize the signal values. The range of possible signal values ​​is defined by a lower limit (optical black signal value) and an upper limit (brightness saturation value). Information regarding the range of signal values ​​can be obtained from the header or optical black area of ​​the captured image.

[0047] Furthermore, it is preferable that the weighting coefficients be based on the signal values ​​of at least one of the training image, the correct image or second image, and the output image or first image after white balance adjustment. When a user views a developed output image, not only gamma correction but also white balance processing has been performed, so if a RAW image is learned without taking white balance processing into consideration, the color balance may differ from that at the time of development.

[0048] With the above configuration, it is possible to provide an image processing method that can obtain a machine learning model whose estimation accuracy is less affected by the signal magnitude of the developed image. [Example]

[0049] Next, an image processing system 300 according to a second embodiment of the present invention will be described. In this embodiment, a weighting coefficient is calculated based on the magnitude relationship between the signal values ​​of the correct image and the output image in addition to the signal value of the correct image.

[0050] FIG. 9 is a block diagram of the image processing system 300. FIG. 10 is an external view of the image processing system 300. The image processing system 300 includes a training device 301, an imaging device 302, and an image processing device 303. The training device 301 and the image processing device 303, and the image processing device 303 and the imaging device 302 are connected via a wired or wireless network. The imaging device 302 includes an optical system 321, an image sensor 322, a memory unit 323, a communication unit 324, and a display unit 325. The captured image is transmitted to the image processing device 303 via the communication unit 324. The image processing device 303 receives the captured image via the communication unit 332 and sharpens the blur using the sharpening unit 334, using information about the configuration and parameters of the machine learning model stored in the memory unit 331. The information about the configuration and parameters of the machine learning model is trained by the training device 301, and is acquired from the training device 301 in advance and stored in the memory unit 331.

[0051] Next, the learning process of the machine learning model (a method for generating a trained model) executed by the training device 301 will be described with reference to Fig. 11. Fig. 11 is a flowchart of the learning process of the machine learning model in this embodiment. Note that steps S301 to S306, S308, and S309 in Fig. 11 are similar to steps S101 to S108 in Fig. 4 described in the first embodiment, respectively, and therefore their details will be omitted.

[0052] In step S301, the acquisition unit 312 acquires one or more original images from the storage unit 311. Subsequently, in step S302, the calculation unit 313 blurs the original images to generate blurred images. Subsequently, in step S303, the acquisition unit 312 acquires a reference image. Subsequently, in step S304, the calculation unit 313 uses a machine learning model to generate an output image based on the training images. Subsequently, in step S305, the update unit 314 calculates the difference between the output image and the reference image for each pixel. Subsequently, in step S306, the update unit 314 calculates a weighting coefficient based on the normalized signal value of the reference image.

[0053] Next, in step S307, the update unit 314 calculates weighting coefficients based on the magnitude relationship between the signal values ​​of the reference image and the output image. In this embodiment, in addition to the weighting coefficients based on the signal values ​​of the reference image, weighting coefficients are also calculated based on the magnitude relationship between the signal values ​​of the reference image and the output image. The effect of this will be explained using specific numerical values.

[0054] For example, suppose there are pixels A and B whose normalized signal values ​​in the correct image are both 0.1. In this case, if the signal values ​​of pixels A and B in the output image of the machine learning model are 0.05 and 0.15, respectively, the difference between the two images and the correct image is the same: 0.05. However, pixel A, which is located on the darker side, will have its difference from the correct image magnified by gamma correction during development. Specifically, if 1 / 2.2 is used as the exponent for gamma correction, the signal value of the correct image after gamma correction will be 0.35, and the pixel values ​​of pixels A and B after gamma correction will be 0.26 and 0.42, respectively, resulting in a difference of 0.09 and 0.07. Therefore, in this step, the signal values ​​of the correct image and the output image are compared, and the weighting coefficient is increased in areas where the signal value of the correct image is higher. In other words, the weighting coefficient in areas where the signal value of the correct image is higher than the output image is larger than the weighting coefficient in areas where the signal value of the correct image is lower than the output image.

[0055] FIG. 12 is a diagram showing the relationship (relational equation) between the difference between the correct image and the output image and the weighting coefficient. In FIG. 12, the horizontal axis represents the difference between the correct image and the output image, and the vertical axis represents the weighting coefficient. As shown by the solid line 411, the weighting coefficient is increased in areas where the signal value of the correct image is larger than that of the output image. Note that the relational equation is not limited to the solid line 411 and may be freely set. As long as the relational equation increases the weighting coefficient in areas where the signal value of the correct image is larger than that of the output image, it may be as shown by the dashed-dotted line 412. Furthermore, weighting coefficients corresponding to the difference between the correct image and the output image may be stored as a table. Note that while FIG. 12 shows the value obtained by subtracting the signal value of the output image from the signal value of the correct image on the horizontal axis, the value obtained by subtracting the signal value of the output image from the signal value of the correct image on the horizontal axis may also be used. In this case, the solid line 411 and the dashed-dotted line 412 are mirror-inverted. Using the weighting coefficients calculated in steps S306 and S307, the final error L2 can be expressed by the following equation (4):

[0056]

number

[0057] In equation (4), t is the signal value of the correct image, y is the signal value of the output image 202, j is the pixel number, N is the total number of pixels, W1 is the weighting coefficient calculated in step S306, and W2 is the weighting coefficient calculated in step S307.

[0058] 11, the update unit 314 updates the parameters of the machine learning model based on the error. Then, in step S309, the update unit 314 determines whether training of the machine learning model is complete.

[0059] Note that the blur sharpening of a captured image using a trained machine learning model, which is executed by the image processing device 303, is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0060] With the above configuration, it is possible to provide an image processing method that can obtain a machine learning model whose estimation accuracy is less affected by the signal magnitude of the developed image.

[0061] In addition, in each embodiment, the training image, output image, and ground truth image are described as images that have not undergone gamma correction as described with reference to Fig. 5, but this is not limiting. In each embodiment, the error may be calculated using the image after gamma correction and a weighting coefficient.

[0062] At this time, the calculation unit 101c performs gamma correction on each of the ground truth image and the output image. The acquisition unit 101b acquires information regarding the gamma correction used in the learning process. By combining the gamma correction used in the learning process with the gamma correction used to develop the estimated image, more stable estimation accuracy is possible regardless of the magnitude of the signal value after correction. In each embodiment, in order to support various gamma corrections, the training device 101 executes a learning process for each of multiple gamma corrections and stores the network parameters optimized for each gamma correction in the storage unit 101a. Information regarding gamma correction can be acquired by selecting from information regarding multiple gamma corrections stored in the storage unit 101a, or the user may input a gamma correction formula, lookup table, or the like.

[0063] The update unit 101d then calculates the error between the gamma-corrected correct image and the gamma-corrected output image using the gamma-corrected correct image, the gamma-corrected output image, and the weighting coefficient, thereby further improving the estimation accuracy. (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. The image processing device can be any device that has the image processing functions of each embodiment, and can be realized in the form of an imaging device or a PC.

[0064] According to each embodiment, the above configuration makes it possible to provide an image processing method, a method for generating a trained model, an image processing device, and a program that can obtain a machine learning model whose estimation accuracy is less affected by the signal magnitude of the developed image.

[0065] The disclosure of each embodiment includes the following methods and compositions: (Configuration 1) obtaining training images and ground truth images; inputting the training images into a machine learning model to generate output images; obtaining weighting coefficients; calculating an error using the ground truth image, the output image, and the weighting coefficient; and updating parameters of the machine learning model based on the error. An image processing method characterized in that, when the output image and the correct image are not residuals from the training image, the weighting coefficients vary depending on signal values ​​of at least one of the training image, the correct image, or the output image. (Method 2) obtaining training images and ground truth images; inputting the training images into a machine learning model to generate output images; obtaining weighting coefficients; calculating an error using the ground truth image, the output image, and the weighting coefficient; and updating parameters of the machine learning model based on the error. An image processing method characterized in that, when the output image and the correct image are residuals from the training image, the weighting coefficient changes depending on at least one signal value of the training image, a first image obtained by adding the output image to the training image, or a second image obtained by adding the correct image to the training image. (Method 3) 3. The image processing method according to method 1 or 2, wherein the weighting coefficients vary depending on the signal values ​​in the non-saturated regions of each image. (Method 4) 4. The image processing method according to any one of methods 1 to 3, wherein in the step of calculating the error, a difference between the correct image and the output image is calculated, and the error is calculated using the difference and the weighting coefficient. (Method 5) 5. The image processing method according to any one of methods 1 to 4, wherein the training images, the output images, and the ground truth images are not gamma-corrected images. (Method 6) 6. The image processing method according to any one of methods 1 to 5, wherein the weighting coefficient changes depending on the magnitude relationship between the signal value of the correct image and the signal value of the output image. (Method 7) The image processing method described in Method 6, characterized in that the weighting coefficient in an area where the signal value of the correct image is larger than the signal value of the output image is larger than the weighting coefficient in an area where the signal value of the correct image is smaller than the signal value of the output image. (Method 8) 8. The image processing method according to any one of methods 1 to 7, wherein the weighting coefficient is larger as the signal value is smaller. (Method 9) The image processing method according to Method 1, wherein the weighting coefficient is larger as the signal value of the correct image is smaller. (Method 10) The image processing method according to Method 2, wherein the weighting coefficient is larger as the signal value of the second image is smaller. (Method 11) 11. The image processing method according to any one of methods 1 to 10, wherein the weighting coefficients are determined based on signal values ​​of the output image after the signal values ​​have been clipped at a predetermined value. (Method 12) 12. The image processing method according to any one of Methods 1 to 11, wherein the weighting coefficients are determined using signal values ​​obtained by normalizing the signal values. (Method 13) 13. The image processing method according to any one of Methods 1 to 12, wherein the weighting coefficients are determined using signal values ​​obtained by performing white balance adjustment on the signal values. (Method 14) The image processing method according to Method 1, wherein the output image is a high-resolution image of the training image. (Method 15) The image processing method according to Method 2, wherein the output image is a residual between the training image and an image obtained by increasing the resolution of the training image. (Method 16) further comprising the step of performing gamma correction on each of the ground truth image and the output image; 16. An image processing method according to any one of methods 1 to 15, wherein in the step of calculating the error, the error is calculated using the gamma-corrected target image, the gamma-corrected output image, and the weighting coefficient. (Method 17) obtaining training images and ground truth images; inputting the training images into a machine learning model to generate output images; obtaining weighting coefficients; calculating an error using the ground truth image, the output image, and the weighting coefficient; and updating parameters of the machine learning model based on the error. A method for generating a trained model, characterized in that when the output image and the correct image are not residuals from the training image, the weighting coefficient varies depending on at least one signal value of the training image, the correct image, or the output image. (Method 18) obtaining training images and ground truth images; inputting the training images into a machine learning model to generate output images; obtaining weighting coefficients; calculating an error using the ground truth image, the output image, and the weighting coefficient; and updating parameters of the machine learning model based on the error. A method for generating a trained model, characterized in that when the output image and the correct image are residuals from the training image, the weighting coefficient changes depending on at least one signal value of the training image, a first image obtained by adding the output image to the training image, or a second image obtained by adding the correct image to the training image. (Configuration 1) an acquisition unit that acquires training images, ground truth images, and weighting coefficients; a generation unit that inputs the training images into a machine learning model to generate an output image; an update unit that calculates an error using the correct image, the output image, and the weighting coefficient, and updates parameters of the machine learning model based on the error; An image processing device characterized in that, when the output image and the correct image are not residuals from the training image, the weighting coefficient varies depending on at least one signal value of the training image, the correct image, or the output image. (Configuration 2) an acquisition unit that acquires training images, ground truth images, and weighting coefficients; a generation unit that inputs the training images into a machine learning model to generate an output image; an update unit that calculates an error using the correct image, the output image, and the weighting coefficient, and updates parameters of the machine learning model based on the error; An image processing device characterized in that, when the output image and the correct image are residuals from the training image, the weighting coefficient changes depending on at least one signal value of the training image, a first image obtained by adding the output image to the training image, or a second image obtained by adding the correct image to the training image. (Configuration 3) A program causing a computer to execute the image processing method according to any one of Methods 1 to 16.

[0066] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0067] 101 Training equipment (image processing equipment) 101b Acquisition Department 101c Arithmetic section 101d Update Department

Claims

1. obtaining training images and ground truth images; inputting the training images into a machine learning model to generate output images; obtaining weighting coefficients; calculating an error using the ground truth image, the output image, and the weighting coefficient; and updating parameters of the machine learning model based on the error. An image processing method characterized in that, when the output image and the correct image are not residuals from the training image, the weighting coefficients vary depending on signal values ​​of at least one of the training image, the correct image, or the output image.

2. obtaining training images and ground truth images; inputting the training images into a machine learning model to generate output images; obtaining weighting coefficients; calculating an error using the ground truth image, the output image, and the weighting coefficient; and updating parameters of the machine learning model based on the error. An image processing method characterized in that, when the output image and the correct image are residuals from the training image, the weighting coefficient changes depending on at least one signal value of the training image, a first image obtained by adding the output image to the training image, or a second image obtained by adding the correct image to the training image.

3. 3. The image processing method according to claim 1, wherein the weighting coefficients vary depending on the signal values ​​in a non-saturated region of each image.

4. 3. The image processing method according to claim 1, wherein in the error calculation step, a difference between the correct image and the output image is calculated, and the error is calculated using the difference and the weighting coefficient.

5. 3. The image processing method according to claim 1, wherein the training images, the output images, and the ground truth images are not gamma-corrected images.

6. 3. The image processing method according to claim 1, wherein the weighting coefficient changes depending on the magnitude relationship between the signal value of the correct image and the signal value of the output image.

7. The image processing method according to claim 6, wherein the weighting coefficient in an area where the signal value of the correct image is greater than the signal value of the output image is greater than the weighting coefficient in an area where the signal value of the correct image is smaller than the signal value of the output image.

8. 3. The image processing method according to claim 1, wherein the weighting coefficient is larger as the signal value is smaller.

9. 2. The image processing method according to claim 1, wherein the weighting coefficient is larger as the signal value of the correct image is smaller.

10. 3. The image processing method according to claim 2, wherein the weighting coefficient is larger as the signal value of the second image is smaller.

11. 3. The image processing method according to claim 1, wherein the weighting coefficient is determined based on a signal value of the output image after the signal value has been clipped at a predetermined value.

12. 3. The image processing method according to claim 1, wherein the weighting coefficients are determined using signal values ​​obtained by normalizing the signal values.

13. 3. The image processing method according to claim 1, wherein the weighting coefficients are determined using signal values ​​obtained by performing white balance adjustment on the signal values.

14. 2. The image processing method according to claim 1, wherein the output image is a high-resolution image of the training image.

15. 3. The image processing method according to claim 2, wherein the output image is a residual between the training image and an image obtained by increasing the resolution of the training image.

16. further comprising the step of performing gamma correction on each of the ground truth image and the output image; 3. The image processing method according to claim 1, wherein in the step of calculating the error, the error is calculated using the gamma-corrected target image, the gamma-corrected output image, and the weighting coefficient.

17. obtaining training images and ground truth images; inputting the training images into a machine learning model to generate output images; obtaining weighting coefficients; calculating an error using the ground truth image, the output image, and the weighting coefficient; and updating parameters of the machine learning model based on the error. A method for generating a trained model, characterized in that when the output image and the correct image are not residuals from the training image, the weighting coefficient varies depending on at least one signal value of the training image, the correct image, or the output image.

18. obtaining training images and ground truth images; inputting the training images into a machine learning model to generate output images; obtaining weighting coefficients; calculating an error using the ground truth image, the output image, and the weighting coefficient; and updating parameters of the machine learning model based on the error. A method for generating a trained model, characterized in that when the output image and the correct image are residuals from the training image, the weighting coefficient changes depending on at least one signal value of the training image, a first image obtained by adding the output image to the training image, or a second image obtained by adding the correct image to the training image.

19. an acquisition unit that acquires training images, ground truth images, and weighting coefficients; a generation unit that inputs the training images into a machine learning model to generate an output image; an update unit that calculates an error using the correct image, the output image, and the weighting coefficient, and updates parameters of the machine learning model based on the error; An image processing device characterized in that, when the output image and the correct image are not residuals from the training image, the weighting coefficient varies depending on at least one signal value of the training image, the correct image, or the output image.

20. an acquisition unit that acquires training images, ground truth images, and weighting coefficients; a generation unit that inputs the training images into a machine learning model to generate an output image; an update unit that calculates an error using the correct image, the output image, and the weighting coefficient, and updates parameters of the machine learning model based on the error; An image processing device characterized in that, when the output image and the correct image are residuals from the training image, the weighting coefficient changes depending on at least one signal value of the training image, a first image obtained by adding the output image to the training image, or a second image obtained by adding the correct image to the training image.

21. 3. A program for causing a computer to execute the image processing method according to claim 1.

Images