Image processing method

Abstract

A machine learning model is used to generate high-quality images from images with varying contrast levels. [Solution] The image processing method includes a generation step of generating a first output image using a first training image 201 and a machine learning model, an acquisition step of acquiring an error based on a first ground truth image 203 containing the same object as the object included in the first training image and the first output image, and an update step of updating the parameters of the machine learning model based on the error. The acquisition step acquires the error based on contrast information 202 generated based on at least one of the first training image and the first ground truth image.

Description

Technical Field

[0001] The present invention relates to image processing using a machine learning model.

Background Art

[0002] As an example of image processing as described above, Patent Document 1 discloses image processing for generating a high-pixel image using a low-image image and a machine learning model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a need for image processing that can generate a good image from images having various contrasts using a machine learning model.

Means for Solving the Problems

[0005] An image processing method (image processing apparatus) as one aspect of the present invention includes a generation step (generation means) for generating a first output image using a first training image and a machine learning model, an acquisition step (acquisition means) for acquiring an error based on a first correct image including the same object as the object included in the first training image and the first output image, and an update step (update means) for updating the parameters of the machine learning model based on the error. In the acquisition step, the error is acquired based on information regarding the contrast generated based on at least one of the first training image and the first correct image.

[0006] Furthermore, another aspect of the present invention is an image processing method (image processing apparatus) comprising: a first generation step (first generation means) for generating a third image using a first image and a machine learning model; a second generation step (second generation means) for generating contrast information based on the first image; and a third generation step (third generation means) for generating a second image based on the first image, the third image, and the contrast information. The contrast information includes a plurality of pixels and the pixel value for each of those pixels. The first image includes pixels corresponding to each of the plurality of pixels in the contrast information. The second generation step is characterized by generating the pixel value of a specific pixel in the contrast information based on the amount of change in the pixel value of a sub-region including the pixel corresponding to a specific pixel in the first image.

[0007] Furthermore, a program that causes a computer to perform processing according to the above-described image processing method also constitutes another aspect of the present invention. [Effects of the Invention]

[0008] According to the present invention, image processing can be performed to generate good images from images with various contrasts using a machine learning model. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the flow of generating training data in Example 1. [Figure 2] This is a block diagram showing the configuration of the image processing system in Example 1. [Figure 3] This is an external view of the image processing system in Example 1. [Figure 4] This is a flowchart showing the training data generation process in Example 1. [Figure 5] This is a flowchart showing the weight learning process for the machine learning model in Example 1. [Figure 6] This is a flowchart showing the second image generation process in Example 1. [Figure 7]It is a block diagram showing the configuration of the image processing system in Example 2. [Figure 8] It is an external view of the image processing system in Example 2. [Figure 9] It is a flowchart showing the learning data generation process in Example 2. [Figure 10] It is a diagram showing the flow of generation of learning data in Example 2. [Figure 11] It is a flowchart showing the learning data generation process in Example 3. [Figure 12] It is a flowchart showing the second image generation process in Example 3.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0011] First, before explaining specific embodiments, an overview of the embodiments will be explained. In the image processing disclosed in Patent Document 1 described above, a high - pixel image is generated by correcting a low - pixel image using a machine learning model. However, if the distribution of the learning data of the machine learning model is biased towards low contrast, the machine learning model is learned so that the error in the low - contrast part of the low - pixel image is smaller than that in the high - contrast part. When such a machine learning model is used, there is a possibility that the high - contrast part of the low - pixel image may not be appropriately corrected. Therefore, in the image processing of the following embodiments, even when the distribution of the learning data of the machine learning model is biased towards low contrast, the image is corrected with high accuracy using the machine learning model regardless of its contrast.

[0012] In the following description, the stage of determining the weights of the machine learning model is called the learning phase. Also, the stage of correcting the first image using the machine learning model with the weights determined by learning to generate the second image is called the estimation phase.

[0013] First, the first process (first image processing method) executed in the learning phase has four steps from the first step to the fourth step, and the machine learning model is learned by repeating the four steps from the first step to the fourth step.

[0014] In the first step, a first training image and a first correct image including an object (image) identical to the object (image) included in the first training image are acquired.

[0015] In the second step (generation step), a first output image is generated using the first training image and the machine learning model.

[0016] In the third step (acquisition step), an error is calculated (acquired) based on the first correct image and the first output image.

[0017] In the fourth step (update step), the parameters of the machine learning model are updated based on the error calculated in the third step.

[0018] The error in the third step is calculated based on a contrast map (which will be described later) generated based on the first training image or the first correct image.

[0019] By using the machine learning model learned in this way, it is possible to generate a second image by accurately correcting a first image having various contrasts in the estimation phase.

[0020] The first process will be described in more detail. In the first step, a first training image and a first correct image in which an image of the same object as the first training image exists are acquired. The number of pixels of the first training image and the number of pixels of the first correct image do not have to be the same. Further details will be described in Example 1, but the first correct image is preferably an image sufficiently containing high-frequency components.

[0021] In the second step, the first output image is generated using the first training image and the machine learning model. For example, the first output image may be generated by inputting the first training image into the machine learning model. Alternatively, the first training image may be enlarged beforehand using interpolation or other methods, and then the enlarged first training image may be input into the machine learning model to generate the first output image. The number of pixels in the first training image and the number of pixels in the first output image do not have to be the same.

[0022] In the third step, the error is calculated based on the first ground truth image and the first output image. The error here is calculated based on a contrast map generated from the first training image or the first ground truth image. The contrast map will be explained in more detail later, but the contrast map in the first process is two-dimensional information about the contrast in the first training image or the first ground truth image. The two-dimensional information about contrast may be information that directly indicates contrast, or it may be information that can be converted to contrast.

[0023] Here, we will explain three specific examples (Examples 1 to 3) of how to calculate the error in the third step.

[0024] As a first example, the error may be calculated based on the difference between the second ground truth image and the first output image. The second ground truth image is an image generated based on at least the first ground truth image and a contrast map. The difference between the second ground truth image and the first output image in the first process indicates how accurately the machine learning model reproduced the ground truth image, with a smaller difference indicating a more accurate reproduction. Therefore, the difference in the first example indicates how accurately the machine learning model reproduced the second ground truth image in the first output image. The difference in the first example is, for example, the Euclidean norm of the difference between the pixel values ​​in the first output image and the pixel values ​​in the second ground truth image.

[0025] The second ground truth image may be generated based on the first ground truth image, the first training image, and a contrast map. For example, the second ground truth image may be generated by weighting the first training image and the first ground truth image using a contrast map. Alternatively, the second ground truth image may be generated based on the first ground truth image, the third ground truth image, and a contrast map. The third ground truth image is the first ground truth image to which at least one of the following processes has been applied: blurring, contrast reduction, and brightness reduction.

[0026] Let's explain the relationship between the first and second ground truth images in this case. First, let's explain the case where the contrast map is generated based on the first training image. Assume that the first pixel of the first training image corresponds to the third pixel of the first ground truth image, and the second pixel of the first training image corresponds to the fourth pixel of the first ground truth image. In this case, when the contrast of the first pixel is higher than the contrast of the second pixel, the second ground truth image is generated as an image in which the weight of the third pixel in the weighted average for the pixel corresponding to the third pixel is smaller than the weight of the fourth pixel in the pixel corresponding to the fourth pixel. In other words, the higher the contrast of a pixel in the first training image, the smaller the weight of the corresponding pixel in the first ground truth image is generated in the second ground truth image.

[0027] Next, we will explain the case where the contrast map is generated based on the first ground truth image. When the contrast of the third pixel in the first ground truth image is higher than the contrast of the fourth pixel in the first ground truth image, the second ground truth image is generated in the same way as above, by making the weight of the third pixel in the weighted average of the pixels corresponding to the third pixel smaller than the weight of the fourth pixel in the pixels corresponding to the fourth pixel. In other words, the second ground truth image is generated such that the weight of a pixel with high contrast in the first ground truth image becomes smaller.

[0028] As a second example, the error may be calculated based on the difference between the first ground truth image and the second output image. The first transformed image is generated by adjusting the pixel values ​​of each pixel in the first output image based on a contrast map. The second output image is generated by adding the pixel values ​​of the first transformed image and the pixel values ​​of the image based on the first training image, pixel by pixel. The image based on the first training image is, for example, the first training image or an image obtained by enlarging the first training image through interpolation or the like.

[0029] The relationship between the first output image and the first transformed image in this case will be explained. First, let's explain the case where the contrast map is generated based on the first training image. Assume that the first pixel of the first training image corresponds to the ninth pixel of the first output image, and the second pixel of the first training image corresponds to the tenth pixel of the first output image. In this case, when the contrast of the first pixel is higher than the contrast of the second pixel, the first transformed image is generated as an image in which the weight of the ninth pixel in the above transformation is smaller than the weight of the tenth pixel in the pixel corresponding to the tenth pixel. That is, the higher the contrast of a pixel in the first training image, the smaller the weight of the pixel in the first output image corresponding to that pixel becomes when the first transformed image is generated. Similarly, when the contrast map is generated based on the first ground truth image, the higher the contrast of a pixel in the first ground truth image, the smaller the weight of the pixel in the first output image corresponding to that pixel becomes when the first transformed image is generated.

[0030] As a third example, if the first difference is based on the first output image and the first ground truth image, and the second difference is based on the first output image and the first training image, the error may be calculated by weighting the first and second differences using a contrast map. Similarly, if the third difference is based on the first output image and the third ground truth image, the error may be calculated by weighting the first and third differences using a contrast map. In this third example, the first, second, and third differences are calculated for each pixel of the image.

[0031] The relationship between the first difference and the second or third difference in this case will be explained. First, let's explain the case where the contrast map is generated based on the first training image. Assume that the first pixel of the first training image corresponds to the first difference value of the first difference, and the second pixel of the first training image corresponds to the second difference value of the first difference. In this case, when the contrast of the first pixel is higher than the contrast of the second pixel, the error is calculated by making the weight of the first difference value in the weighted average of the difference values ​​corresponding to the first difference value smaller than the weight of the second difference value in the difference value corresponding to the second difference value. That is, the higher the contrast of a pixel in the first training image, the smaller the weight of the difference value of the first difference corresponding to that pixel will be when the error is calculated. Similarly, when the contrast map is generated based on the first ground truth image, the higher the contrast of a pixel in the first ground truth image, the smaller the weight of the difference value of the first difference corresponding to that pixel will be when the error is calculated.

[0032] Furthermore, the contrast map may be generated based on both the first training image and the first ground truth image. In this case, for example, the contrast may be calculated from both the first training image and the first ground truth image, and the contrast map may be generated by weighting the contrast of the first training image and the contrast of the first ground truth image. Then, the second ground truth image may be generated such that the weight of the pixel in the first ground truth image corresponding to a pixel in the first training image with high contrast becomes smaller.

[0033] Furthermore, in addition to the weighted average of the two pixels mentioned above, weighted addition may also be used. More precisely, any combination of two pixels, including these weighted averages and weighted additions, is acceptable.

[0034] In the fourth step, the parameters of the machine learning model are updated based on the error calculated in the third step. The parameters may be updated so that the machine learning model has at least one of the following functions: upscaling, deblurring, dehazing, debayering, and noise reduction. In the first process, the machine learning model is trained by repeating these first through fourth steps one or more times (a total of two or more times).

[0035] Next, we will explain the effects of the first processing described above. Note that the first training image and the first ground truth image each consist of multiple images. The first processing makes it possible to generate a second image that corrects the first image with high accuracy during the estimation phase. The first processing is particularly effective when the contrast distribution of the first training image and the first ground truth image is concentrated in the low contrast range. This is the case, for example, when the format of the first training image and the first ground truth image is HEIF (High Efficiency Image File Format).

[0036] The effect of the first processing step will be explained in comparison to the conventional method. In the conventional method, the machine learning model is trained by repeating steps 1 through 4. On the other hand, in the conventional method, the error is calculated in step 3 without relying on a contrast map. Specifically, the error is calculated based on the difference between the first output image and the first ground truth image, and the machine learning model is trained so that the image quality of the first output image approaches the image quality of the first ground truth image. When the first image is corrected using this machine learning model in the conventional estimation phase to generate the second image, the high-contrast areas of the first image are overcorrected. The reason for this is explained below.

[0037] In the conventional learning phase, based on the relationship between the first training image and the target first ground truth image, the amount of correction that the machine learning model should apply to the first training image is greater in the low-contrast areas of the first training image than in the high-contrast areas. On the other hand, if the contrast distribution of the first training image and the first ground truth image is concentrated in the low-contrast areas, the machine learning model is trained to make the error smaller in the low-contrast areas than in the high-contrast areas of the first training image. In other words, the machine learning model is trained to make the image quality of the first output image closer to that of the first ground truth image in the low-contrast areas than in the high-contrast areas. Therefore, the high-contrast areas of the first training image are affected by the amount of correction that should be applied in the low-contrast areas, and are trained to be overcorrected beyond the first ground truth image. From the above, it follows that in the conventional estimation phase, the high-contrast areas of the first image are overcorrected.

[0038] On the other hand, the first processing method solves the conventional problem by generating a second image in which the first image has been corrected with high accuracy during the estimation phase. In the first processing method, the error calculation in the third step is performed based on a contrast map. This allows the amount of correction that the machine learning model needs to correct for the high-contrast areas of the first training image in the first processing step of the learning phase to be set smaller compared to conventional processing.

[0039] Next, we will explain the second process (second image processing method) performed in the estimation phase. The second process consists of the fifth and sixth steps. In the fifth step (first generation step), the third image is generated using the first image and a machine learning model. In the sixth step (second and third generation steps), a contrast map is generated based on the first image. Furthermore, the second image is generated based on the first image, the third image, and the contrast map. The contrast map will be explained later, but in the second process, the contrast map is two-dimensional information about the contrast of the first image.

[0040] This second processing method, similar to the first processing method, makes it possible to generate a second image in which the first image, which has various contrasts in the estimation phase, is corrected with high accuracy.

[0041] In the second process, as with the conventional process described above, the error may be calculated in the third step of the learning phase without relying on a contrast map. Specifically, in the second process, the machine learning model may be trained by repeating the following steps 1 through 4.

[0042] In the first step of the second process, the first training image and the first ground truth image are acquired. In the second step, the first output image is generated using the first training image and the machine learning model. In the third step, the error is calculated based on the difference between the first output image and the first ground truth image. In the fourth step, the parameters of the machine learning model are updated based on the error calculated in the third step.

[0043] Furthermore, in the fifth step of the second processing, a third image is generated using the first image and a machine learning model. For example, the third image may be generated by inputting the first image into the machine learning model. Alternatively, the first image may be enlarged beforehand using interpolation or other methods, and then the enlarged first image may be input into the machine learning model to generate the third image. The number of pixels in the first image and the third image do not have to be the same. The machine learning model may have at least one function from among upscaling, deblurring, dehazing, debayering, and noise reduction.

[0044] In the sixth step, a second image is generated based on the first image, the third image, and the contrast map. For example, the second image may be generated by weighting the first image and the third image using the contrast map.

[0045] Let's explain the relationship between the third and second images in this case. Assume that the fifth pixel of the first image corresponds to the seventh pixel of the third image, and the sixth pixel of the first image corresponds to the eighth pixel of the third image. In this case, when the contrast of the fifth pixel is higher than the contrast of the sixth pixel, the second image is generated as an image in which the weight of the pixel corresponding to the seventh pixel in the weighted average is smaller than the weight of the eighth pixel in the pixel corresponding to the eighth pixel. In other words, the second image is generated such that the weight of the pixel in the third image corresponding to a pixel with high contrast in the first image becomes smaller.

[0046] Next, the effects of the second processing described above will be explained. Similar to the first processing, the second processing allows for the generation of a second image with high accuracy correction of the first image during the estimation phase. The second processing, like the first processing, is particularly effective when the contrast distribution between the first training image and the first ground truth image is concentrated in the low-contrast range. This is, for example, when the format of the first training image and the first ground truth image is HEIF.

[0047] In comparison with the conventional processing described above, the effects of the second processing will be explained in more detail. As mentioned earlier, when the first image is corrected and the second image is generated in the conventional estimation phase, the high-contrast areas of the first image are overcorrected. In the estimation phase of the second processing, in the fifth step, a third image is obtained in which the high-contrast areas of the first image are overcorrected. On the other hand, in the sixth step, the second image is generated in such a way that the contribution of the pixels of the third image to the high-contrast areas of the first image is reduced, that is, the contribution of the pixels of the first image is increased. As a result, a second image can be obtained in which the high-contrast areas of the first image are also corrected with high accuracy.

[0048] As described above, both the first and second processing methods enable the generation of a second image with high accuracy correction of the first image, even when the contrast distribution between the first training image and the first ground truth image is concentrated in low contrast. Furthermore, the first processing method allows the above effect to be obtained without requiring any additional processing by the user during the estimation phase.

[0049] Next, we will describe the contrast maps used in the first and second processes. The contrast map in the first process is generated based on the first training image or the first ground truth image. The contrast map in the second process is generated based on the first image. In the following description, the contrast value refers to the contrast of an image or pixel.

[0050] Michelson contrast, used to calculate contrast focused on visual stimuli, calculates a single contrast value for an image using the maximum and minimum pixel values ​​within the image. In other words, with Michelson contrast, the contrast value does not change for each pixel in the image. On the other hand, the contrast maps in the first and second processes may have multiple pixels, for example, the first training image, the first ground truth image, or the same number of pixels as the first image. Also, each pixel in the contrast map may have a different pixel value.

[0051] The following describes the case where the contrast map is generated based on the first training image. This description is also applicable when the contrast map is generated based on the first ground truth image or the first image, and a detailed explanation will be provided in each example.

[0052] The first training image (or first ground truth image or first image) has pixels corresponding to each of the multiple pixels in the contrast map. The pixel value of each pixel (specific pixel) in the contrast map may be calculated based on the change in the pixel value of a subregion in the first training image that includes the pixel corresponding to the specific pixel. This allows the first process to calculate the error in the third step based on the contrast of each pixel in the first training image. However, the number of pixels in the contrast map and the number of pixels in the first training image do not have to be the same.

[0053] Furthermore, the contrast map may be calculated based on the ratio of the difference between the pixel value of a corresponding pixel in the first training image and the pixel value of at least one adjacent pixel, and the sum of these pixel values. For example, it may be calculated based on the ratio of the absolute difference between the pixel value of a corresponding pixel in the first training image and the pixel values ​​of eight adjacent pixels, and the sum of these pixel values, as shown in equation (1) below. Equation (1) is an equation that shows the pixel value C(p,q) at position (p,q) in the contrast map. At the same time, C(p,q) also shows the contrast value of the pixel located at position (p,q) in the first training image. In equation (1), let I(p,q) be the pixel value of the pixel at position (p,q) in the first training image.

[0054]

number

[0055] Alternatively, as shown in equation (2), the calculation may be based on the ratio of the sum of the positive difference between the pixel value of a corresponding pixel in the first training image and the pixel values ​​of the eight adjacent pixels (adjacent pixels). In equation (2), the positive difference is calculated by subtracting the pixel value of the corresponding pixel from the pixel value of the adjacent pixels, but it may also be calculated by subtracting the pixel values ​​of the adjacent pixels from the pixel value of the corresponding pixel.

[0056]

number

[0057] Furthermore, thresholding may be performed during the contrast map generation process. For example, after calculating the pixel value of each pixel in the contrast map using equation (1), a process may be performed to replace pixel values ​​smaller than the threshold with zero.

[0058] Furthermore, in the first processing step, a contrast map may be used to extract low-contrast areas from the first training image, and a new first training image may be generated by blurring the extracted low-contrast areas. Alternatively, a contrast map may be used to extract low-contrast areas from the first ground truth image, and a new second ground truth image may be generated by sharpening the pixels of the second ground truth image corresponding to the extracted low-contrast areas. This makes it possible to train a machine learning model that can correct low-contrast areas with higher accuracy.

[0059] The machine learning models in the examples include neural networks, genetic programming, and Bayesian networks. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), RNNs (Recurrent Neural Networks), and diffusion models. [Examples]

[0060] In Example 1, a machine learning model is trained to generate a second image in which the first image is upscaled with high accuracy.

[0061] Figure 2 shows the configuration of the image processing system 100 in this embodiment. Figure 3 shows the external appearance of the image processing system 100. The image processing system 100 includes a learning device 101 as a first image processing device, an imaging device 102 as a second image processing device, and a network 103. The learning device 101 and the imaging device 102 are connected to each other by a wired or wireless network 103. The image processing system 100 may also be referred to as an image processing device.

[0062] The learning device 101 is composed of a computer and includes a storage unit 111, an acquisition unit 112, a generation unit (generation means and acquisition means) 113, and an update unit (update means) 114, and determines the weights of the machine learning model. The storage unit 111 stores the first training image and the first ground truth image in advance. The acquisition unit 112 acquires the first training image and the first ground truth image from the storage unit 111. The generation unit 113 generates a contrast map based on the first training image and generates a second ground truth image based on the second training image (which is an enlarged version of the first training image), the first ground truth image, and the contrast map. Furthermore, the generation unit 113 calculates the error as the difference between the first output image (which is output by inputting the first training image into the machine learning model) and the second ground truth image. The update unit 114 updates the parameters of the machine learning model based on the error calculated by the generation unit 113.

[0063] The imaging device 102 includes an optical system 121, an image sensor 122, an image estimation unit 123, a storage unit 124, a recording medium 125, a display unit 126, and a system controller 127. The optical system 121 collects light incident from the subject space to generate a subject image. The optical system 121 may have functions such as zoom, aperture, and autofocus. The image sensor 122 converts the subject image generated by the optical system 121 into an electrical signal to generate an image. The image sensor 122 is a photoelectric conversion element such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.

[0064] The image estimation unit 123, acting as an image processing device, is computer-based and generates a second image by upscaling the first image using a machine learning model whose weights have been predetermined by the learning device 101. The weights of the machine learning model are stored in the storage unit 124. In this embodiment, the first image is an image generated by the user taking an image using the optical system 121 and the image sensor 122. The recording medium 125 records the second image. The display unit 126 displays the second image when the user gives an instruction to output the second image. The above operations are controlled by the system controller 127.

[0065] The processing in this embodiment includes generating training data for a machine learning model, training the weights of the machine learning model (training phase), and estimation using the machine learning model with the trained weights (estimation phase).

[0066] The generation of training data will be explained with reference to Figures 1 and 4. Figure 1 shows the flow of training data generation. The flowchart in Figure 4 shows the process of generating training data. Training data is a pair of training patches and ground truth patches, and is used to train a machine learning model. In the training phase, output patches are obtained by inputting training patches into the machine learning model, and the weights of the machine learning model are determined to minimize the difference between the output patches and the ground truth patches. Training patches are generated from the first training image 201, and ground truth patches are generated from the second ground truth image 205.

[0067] The generation of the second ground truth image 205 will be explained with reference to Figure 1. The second ground truth image 205 is generated by weighting the first ground truth image 203 and the second training image 204, which is an enlarged version of the first training image 201, based on the contrast map 202 generated from the first training image 201. The contrast map 202 shown in Figure 1 is an example of a contrast map, showing that pixels with a gray color closer to white have a higher contrast in the corresponding pixels of the training image 201. The second ground truth image 205 shown in Figure 1 shows that pixels with a gray color closer to white have less of the first ground truth image 203 (and more of the second training image 204) mixed in. In other words, in the generation of the second ground truth image, pixels with a higher contrast in the training image 201 have less of the first ground truth image 203 (and more of the second training image 204) mixed in.

[0068] The acquisition unit 112 and the generation unit 113 in the learning device 101 execute the learning data generation process shown in Figure 4 according to the program. In this embodiment, this process is performed in the learning device 101, but it may be performed in other devices as well.

[0069] First, in step S101 of Figure 4, the acquisition unit 112 acquires the first ground truth image 203 from the storage unit 111. The first ground truth image 203 is a set of images, which may be captured images or computer graphics (CG) images. It is preferable that the first ground truth image 203 has a sufficient amount of high-frequency components. This is to determine the weights of the machine learning model so that it can estimate an image with high resolution that contains a sufficient amount of high-frequency components. For example, if the first ground truth image 203 is a captured image, it is preferable to use an image captured with an optical system that is more efficient than the optical system 121, or an image obtained by reducing the size of a captured image. Also, in order to improve the robustness of the machine learning model to subjects included in the first image, the first ground truth image 203 should be an image that includes various objects. For example, it should be an image that includes objects such as edges, textures, gradients, and flat areas with various intensities and directions.

[0070] Next, in step S102, the acquisition unit 112 acquires the first training image 201 from the storage unit 111. The first training image 201 is a set of images, which may be captured images or computer graphics images. The number of pixels in the first training image 201 is smaller than the number of pixels in the first ground truth image 203. The first training image 201 also contains the same objects as the first ground truth image 203 and has a larger sampling pitch than the first ground truth image 203. It is more preferable that the first training image 201 contains the same image quality degradation as the first image that is upscaled in the estimation phase. Image quality degradation includes jagged edges, spatial aliasing, compression artifacts, and noise in contours and edges. This improves the robustness of the machine learning model against the image quality degradation of the first image.

[0071] Alternatively, the first training image 201 may be generated using the first ground truth image 203. For example, the first training image 201 may be generated by downscaling the first ground truth image 203 to the same quality degradation as the first image. Furthermore, the first ground truth image 203 and the first training image 201 may be generated using images different from the first ground truth image 203 and the first training image 201, respectively.

[0072] In this embodiment, the first ground truth image 203 and the first training image 201 are acquired from the storage unit 111, respectively. However, the first ground truth image 203 and the first training image 201 may also be acquired using an image captured using the optical system 121 and the image sensor 122. For example, the image captured may be used as the first training image 201, and the first ground truth image 203 may be acquired by capturing the same object as the object included in the first training image 201 using an image sensor with a smaller sampling pitch than the image sensor 122. Furthermore, it is even more preferable that the first ground truth image 203 is acquired by imaging using an optical system with less aberration than the optical system 121. This results in a higher resolution first ground truth image 203, which in turn allows for the generation of a more accurate second image during the estimation phase.

[0073] Next, in step S103, the generation unit 113 generates a contrast map 202 as a second contrast map. In this embodiment, a first contrast map with the same number of pixels as the first training image 201 is generated based on the first training image 201, and then the first contrast map is enlarged to generate the second contrast map. The first training image 201 has pixels corresponding to each pixel of the first contrast map. The pixel value of each pixel of the first contrast map is calculated based on the ratio of the sum of the absolute difference between the pixel value of the corresponding pixel in the first training image 201 and the pixel values ​​of the eight adjacent pixels, as shown in equation (1) above.

[0074] The first contrast map indicates that the higher the pixel value, the higher the contrast of the corresponding pixels in the first training image 201. The second contrast map is generated by expanding the first contrast map 202 using interpolation. Known interpolation methods such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation are used for the interpolation process. The ratio of the number of pixels in the first contrast map to the second contrast map is equal to the ratio of the number of pixels in the first image to the second image during the estimation phase.

[0075] Next, in step S104, the generation unit 113 generates a second training image 204. In this embodiment, the second training image 204 is an image obtained by enlarging the first training image 201 using interpolation processing. The ratio of the number of pixels between the first training image 201 and the second training image 204 is equal to the ratio of the number of pixels between the first image and the second image in the estimation phase.

[0076] In addition to generating the contrast map (second contrast map) 202 using the first training image 201 as described above, it may also be generated using the second training image 204. For example, the pixel value of each pixel in the contrast map 202 may be calculated based on the ratio of the sum of the absolute difference between the pixel value of the corresponding pixel in the second training image 204 and the pixel values ​​of the eight adjacent pixels, as shown in equation (1) above.

[0077] Next, in step S105, the generation unit 113 generates the second ground truth image 205. In this embodiment, the second ground truth image 205 is generated by weighting the second training image 204 and the first ground truth image 203 using the contrast map 202 according to the following equation (3). Equation (3) is where I is the pixel value of position (p,q) in the second ground truth image 205. new_gt (p,q) is shown. Here, the pixel value at position (p,q) in the contrast map 202 is C(p,q). Also, the pixel value at position (p,q) in the second training image 204 is I. tr (p,q) is the pixel value I of position (p,q) in the first correct image 203. old_gt Let (p, q).

[0078]

number

[0079] As mentioned above, the contrast map indicates that the larger the pixel value, the higher the contrast of the corresponding pixel in the first training image 201 (and the second training image 204). Therefore, according to equation (3), the second ground truth image 205 is generated such that the weight of the pixel in the first ground truth image 203 corresponding to a pixel with high contrast in the first training image 201 becomes smaller.

[0080] Next, in step S106, the generation unit 113 generates training patches and ground truth patches. Each patch is an image with a predetermined number of pixels (e.g., 64 x 64 pixels), and in this embodiment, the number of pixels in the ground truth patches is greater than the number of pixels in the training patches. The ratio of the number of pixels in the training patches to the ground truth patches is equal to the ratio of the number of pixels in the first image to the second image during the estimation phase.

[0081] Images with a predetermined number of pixels are extracted from regions containing the same object in the first training image 201 and the second ground truth image 205, and these are designated as the training patch and ground truth patch, respectively. That is, the training patch contains the same object as the ground truth patch, and its sampling pitch is greater than that of the ground truth patch. The training patch and ground truth patch are extracted from multiple regions in the first training image 201 and the second ground truth image 205, respectively.

[0082] In this embodiment, training patches and ground truth patches are generated from the first training image 201 and the second ground truth image 205. However, if the number of pixels in the first training image 201 and the second ground truth image 205 is the same as the number of pixels in the required patch, the process of extracting the patch is unnecessary.

[0083] The flowchart in Figure 5 shows the process of learning the weights of the machine learning model executed by the learning device 101 during the learning phase. The acquisition unit 112, generation unit 113, and update unit 114 execute the weight learning process in Figure 5 according to the program.

[0084] First, in step S201, the acquisition unit 112 acquires one or more training patches and correct answer patches from the storage unit 111.

[0085] Next, in step S202, the generation unit 113 inputs the training patch into the machine learning model to generate an output patch. In this embodiment, the machine learning model is a CNN having multiple convolutional layers. In the first training, the weights of the convolutional layers (filter coefficients and biases) are generated by random numbers. However, the machine learning model is not limited to a CNN; it may also be other machine learning models such as GANs, RNNs, or diffusion models.

[0086] Next, in step S203, the update unit 114 updates the weights of the machine learning model based on the difference between the output patch and the ground truth patch. In this embodiment, the Euclidean norm of the difference in pixel values ​​between the output patch and the ground truth patch is used as the loss function. However, the loss function is not limited to this. If multiple sets of training patches are input in step S201, the value of the loss function is calculated for each set. The weights are updated from the calculated loss function values ​​using methods such as backpropagation.

[0087] Next, in step S204, the update unit 114 determines whether or not the learning of the machine learning model is complete. The completion of learning can be determined by whether the number of iterations of weight updates has reached a predetermined number, or whether the amount of change in the weights during the update is less than a predetermined value. If it is determined in step S204 that the learning of the weights is not complete, the acquisition unit 112 returns to step S201 and acquires one or more sets of new training patches and correct patches. If it is determined that the learning of the weights is complete, the update unit 114 terminates the learning and stores the weight information in the storage unit 111.

[0088] The flowchart in Figure 6 shows the estimation process of the second image using a machine learning model with trained weights, which is performed in the imaging device 102 during the estimation phase. In the estimation phase of this embodiment, a second image is generated by upscaling the first image using the machine learning model. The acquisition unit 123a and estimation unit 123b of the image estimation unit 123 of the imaging device 102 execute the estimation process shown in Figure 6 according to the program.

[0089] First, in step S301, the acquisition unit 123a acquires the first image and information on the weights of the machine learning model. The first image may be represented in grayscale or may have multiple channel components. The first image to be acquired may also be a part of the image captured by the optical system 121 and the image sensor 122. The weight information is read in advance from the storage unit 111 and stored in the storage unit 124.

[0090] Next, in step S302, the estimation unit 123b inputs the first image into the machine learning model to generate a second image. The second image is an image obtained by upscaling the first image with high accuracy.

[0091] According to the embodiment described above, a machine learning model can be trained to generate a second image in which the first image has been upscaled with high accuracy. This machine learning model is particularly effective when the contrast distribution of the first training image 201 and the first ground truth image 203 is concentrated in low contrast. [Examples]

[0092] In Example 2, a machine learning model is trained to generate a second image in which the first image has been de-blurred with high accuracy.

[0093] Figure 7 shows the configuration of the image processing system 300 in this embodiment. Figure 8 shows the external appearance of the image processing system 300. The image processing system 300 includes a learning device 301 as a first image processing device, an imaging device 302, an image estimation device 303 as a second image processing device, a display device 304, a storage medium 305, an output device 306, and a network 307.

[0094] The learning device 301 is composed of a computer and includes a storage unit 301a, an acquisition unit 301b, a generation unit (generation means and acquisition means) 301c, and an update unit (update means) 301d, and determines the weights of the machine learning model. The storage unit 301a stores the first training image and the first ground truth image in advance. The acquisition unit 301b acquires the first training image and the first ground truth image from the storage unit 301a. The generation unit 301c generates a contrast map based on the first training image. The generation unit 301c then generates a second ground truth image based on the first training image, a third ground truth image obtained by blurring the first ground truth image, and the contrast map. Furthermore, the generation unit 301c calculates the error as the difference between the first output image, which is output by inputting the first training image into the machine learning model, and the second ground truth image. The update unit 301d updates the parameters of the machine learning model based on the error calculated by the generation unit 301c.

[0095] The imaging device 302 includes an optical system 302a and an image sensor 302b. The optical system 302a collects light incident from the subject space to generate a subject image. The image sensor 302b converts the subject image generated by the optical system 302a into an electrical signal to generate an image.

[0096] The image estimation device 303, which functions as an image processing device, includes a storage unit 303a, an acquisition unit 303b, and an estimation unit 303c. The image estimation device 303 generates a second image by de-blurring the first image using a machine learning model whose weights have been predetermined by the learning device 301. The weights of the machine learning model are stored in the storage unit 303a. In this embodiment, the first image is an image acquired by the user using the imaging device 302.

[0097] The second image is output to at least one of the display device 304, storage medium 305, and output device 306. The display device 304 is a liquid crystal display, projector, etc. The user can perform editing work, etc., while checking the image in progress via the display device 304. The storage medium 305 is a semiconductor memory, hard disk, server on a network, etc., and stores the second image. The output device 306 is a printer, etc.

[0098] The processing in this embodiment is similar to the processing in Embodiment 1, and includes generating training data for the machine learning model, training the weights of the machine learning model (training phase), and estimation using the machine learning model with the trained weights (estimation phase).

[0099] First, the generation of training data will be explained with reference to Figures 9 and 10. The flowchart in Figure 9 shows the process of generating training data. Figure 10 shows the flow of training data generation. Similar to Example 1, the training data consists of a set of training patches and ground truth patches, and is used to train the machine learning model. The training patches are generated from the first training image 401, and the ground truth patches are generated from the second ground truth image 405. However, in Example 2, the second ground truth image 405 is generated by weighting the first ground truth image 403 and the third ground truth image 404, which is obtained by blurring the first ground truth image 403, based on the contrast map 402 generated from the first training image 401.

[0100] The contrast map 402 shown in Figure 10 is an example of a contrast map, indicating that pixels with a gray color closer to white have higher contrast in the corresponding pixels of the training image 401. In this case, for the second ground truth image 405 shown in Figure 10, pixels with a gray color closer to white are mixed with less of the first ground truth image 403 (and more of the third ground truth image 404). In other words, in the generation of the second ground truth image 405, pixels with a high contrast value in the training image 401 are mixed with less of the first ground truth image 403 (and more of the third ground truth image 404).

[0101] The acquisition unit 301b and generation unit 301c of the learning device 301 execute the learning data generation process shown in Figure 10 according to the program. In this embodiment, this process is performed by the learning device 301, but it may be performed by other devices.

[0102] First, in step S401 of Figure 10, the acquisition unit 301b acquires the first correct image 403 from the storage unit 301a. The first correct image 403 is the same as the first correct image 203 acquired in step S101 of Example 1.

[0103] Next, in step S402, the acquisition unit 301b acquires the first training image 401 from the storage unit 301a. Similar to Embodiment 1, the first training image 401 is a set of images, which may be captured images or computer graphics images. In this embodiment, the number of pixels in the first training image 401 is the same as the number of pixels in the first ground truth image 403. The first training image 401 also contains the same objects as the first ground truth image 403. It is also preferable that the first training image 401 contains the same image quality degradation as the first image that undergoes de-blurring in the estimation phase. The image quality degradation is the same as in Embodiment 1. This improves the robustness of the machine learning model against image quality degradation of the first image.

[0104] The first training image 401 may be generated using the first ground truth image 403. For example, the first training image 401 may be generated by blurring the first ground truth image 403 to impart the image quality degradation inherent in the first image. Also, similar to Example 1, the first ground truth image 403 and the first training image 401 may be generated using images different from the first ground truth image 403 and the first training image 401, respectively.

[0105] In this embodiment, the first ground truth image 403 and the first training image 401 were acquired from the storage unit 301a. However, the first ground truth image 403 and the first training image 401 may also be acquired using an image captured by the imaging device 302. For example, the image captured may be used as the first training image 401, and the first ground truth image 403 may be acquired by imaging the same object as the object included in the first training image 401 using an optical system with less aberration than the optical system 302a.

[0106] Next, in step S403, the generation unit 301c generates a contrast map 402. In this embodiment, a contrast map 402 with the same number of pixels as the first training image 401 is generated based on the first training image 401. Similar to Embodiment 1, the first training image 401 has pixels corresponding to each pixel in the contrast map 402. The pixel value of each pixel in the contrast map 402 is calculated based on the ratio of the sum of the absolute difference between the pixel value of the corresponding pixel in the first training image 401 and the pixel values ​​of the eight adjacent pixels, as shown in equation (1) above. In this embodiment, a larger pixel value in the contrast map 402 indicates a higher contrast of the corresponding pixel in the first training image 401.

[0107] Next, in step S404, the estimation unit 303c generates a third ground truth image 404. In this embodiment, the third ground truth image 404 is an image obtained by blurring the first ground truth image 403. Blurring is a process that blurs the details of an image, and in this embodiment, the third ground truth image 404 is generated by applying Gaussian blur to the first ground truth image 403. The number of pixels in the third ground truth image 404 is the same as the number of pixels in the first ground truth image 403.

[0108] Next, in step S405, the estimation unit 303c generates a second ground truth image 405. In this embodiment, the second ground truth image 405 is generated by weighting the first ground truth image 403 and the third ground truth image 404 using the contrast map 402, according to equation (3) described above. However, in this embodiment, I in equation (3) tr Let (p,q) be the pixel value at position (p,q) in the third correct image 404.

[0109] Next, in step S406, the estimation unit 303c generates training patches and ground truth patches. In this embodiment, the number of pixels in the ground truth patches is the same as the number of pixels in the training patches. The training patches also contain the same objects as those contained in the ground truth patches. Similar to Embodiment 1, images with a predetermined number of pixels are extracted from regions containing the same objects in the first training image 401 and the second ground truth image 405, respectively, to create the training patches and ground truth patches. Also similar to Embodiment 1, the training patches and ground truth patches are extracted from multiple regions of the first training image 401 and the second ground truth image 405, respectively.

[0110] In this embodiment, training patches and ground truth patches are generated from the first training image 401 and the second ground truth image 405. However, if the number of pixels in the first training image 401 and the second ground truth image 405 is the same as the number of pixels in the required patch, the process of extracting the patch is unnecessary.

[0111] In this embodiment, as in Embodiment 1, the weight learning process shown in the flowchart of Figure 5 is performed during the learning phase. In this embodiment, the processes performed by the acquisition unit 112, generation unit 113, and update unit 114 in the learning device 101 in Embodiment 1 are performed by the acquisition unit 301b, generation unit 301c, and update unit 301d in the learning device 301.

[0112] In this embodiment, the estimation phase involves performing estimation processing of the second image using a machine learning model with trained weights, as shown in the flowchart of Figure 6. In the estimation phase of this embodiment, the machine learning model is used to generate a second image in which the first image has been de-blurred. In this embodiment, the acquisition unit 303b and estimation unit 303c in the image estimation device 303 perform the processing that was performed by the acquisition unit 123a and estimation unit 123b in the image estimation unit 123 within the imaging device 102 in Embodiment 1.

[0113] According to the embodiment described above, it is possible to train a machine learning model that generates a second image in which the first image has been de-blurred with high accuracy. This machine learning model is particularly effective when the contrast distribution of the first training image 201 and the first ground truth image 203 is concentrated in low contrast. [Examples]

[0114] In Example 3, additional processing other than the machine learning model processing is performed during the estimation phase to generate a second image in which the first image has been upscaled with high accuracy.

[0115] The configuration of the image processing system in this embodiment is basically the same as that shown in Figures 2 and 3 in Embodiment 1.

[0116] The learning device 101 includes a storage unit 111, an acquisition unit 112, a generation unit 113, and an update unit 114, and determines the weights of the machine learning model. The storage unit 111, as in Embodiment 1, pre-stores the first training image and the first ground truth image. In this embodiment, the generation unit 113 calculates the error as the difference between the first output image, which is output when the first training image is input to the machine learning model, and the first ground truth image. The update unit 114, as in Embodiment 1, updates the parameters of the machine learning model based on the error calculated by the generation unit 113.

[0117] The imaging device 102 includes an optical system 121, an image sensor 122, an image estimation unit (first, second, and third generation means) 123, a storage unit 124, a recording medium 125, a display unit 126, and a system controller 127. All parts except the image estimation unit 123 are the same as in Embodiment 1.

[0118] In this embodiment, the image estimation unit 123 generates a third image by upscaling the first image using a machine learning model whose weights have been predetermined by the learning device 101. The image estimation unit 123 also generates a contrast map based on the first image. Furthermore, the image estimation unit 123 generates a second image based on the first image, the third image, and the contrast map. The weights of the machine learning model are stored in the storage unit 124. In this embodiment, the first image is an image acquired by the user through imaging using the optical system 121 and the image sensor 122.

[0119] The processing in this embodiment, like that in Embodiment 1, includes generating training data for the machine learning model, training the weights of the machine learning model (training phase), and estimation using the machine learning model with the trained weights (estimation phase).

[0120] First, the generation of training data will be explained with reference to Figure 11. The flowchart in Figure 11 shows the process of generating training data. Training data consists of a set of training patches and ground truth patches, and is used to train a machine learning model. In the training phase, output patches are obtained by inputting training patches into the machine learning model, and the weights of the machine learning model are determined to minimize the difference between the output patches and the ground truth patches. Training patches are generated from the first training image, and ground truth patches are generated from the first ground truth image. The acquisition unit 112 and the generation unit 113 in the learning device 101 execute the process shown in Figure 11 according to the program. In this embodiment, the learning device 101 performs the training data generation process, but other devices may also perform it.

[0121] First, in step S501 of Figure 11, the acquisition unit 112 acquires the first correct answer image from the storage unit 111. The first correct answer image is the same as the first correct answer image 203 acquired in step S101 of Example 1.

[0122] Next, in step S502, the acquisition unit 112 acquires the first training image from the storage unit 111. The first training image is the same as the first training image 201 acquired in step S102 of Embodiment 1.

[0123] Next, in step S503, the generation unit 113 generates training patches and ground truth patches, respectively. Each patch is an image with a predetermined number of pixels (e.g., 64 x 64 pixels), and in this embodiment, the number of pixels in the ground truth patch is greater than the number of pixels in the training patch. The ratio of the number of pixels in the training patch to the ground truth patch is equal to the ratio of the number of pixels in the first image to the second image in the estimation phase. Images with a predetermined number of pixels are extracted from regions containing the same object in the first training image and the first ground truth image, and these are used as the training patch and ground truth patch, respectively. That is, the training patch is a patch that contains the same object as the ground truth patch and has a sampling pitch greater than that of the ground truth patch. The training patch and ground truth patch are extracted from multiple regions of the first training image and the first ground truth image, respectively.

[0124] In this embodiment, training patches and ground truth patches are generated from the first training image and the first ground truth image. However, if the number of pixels in the first training image and the first ground truth image is the same as the number of pixels in the required patch, the process of extracting the patch is unnecessary.

[0125] In this embodiment, as in Embodiment 1, the acquisition unit 112, generation unit 113, and update unit 114 of the learning device 101 perform the weight learning process shown in the flowchart of Figure 5 during the learning phase.

[0126] In this embodiment, the process shown in the flowchart of Figure 12 is executed during the estimation phase. Figure 12 shows the estimation process of the second image using a machine learning model with trained weights, which is executed in the image estimation unit 123 of the imaging device 102. In the estimation phase of this embodiment, the image estimation unit 123 generates a third image by upscaling the first image using the machine learning model. The image estimation unit 123 also generates a contrast map based on the first image. Furthermore, the image estimation unit 123 generates a second image based on the first image, the third image, and the contrast map. The acquisition unit 123a and the estimation unit 123b in the image estimation unit 123 execute the process shown in Figure 12 according to the program.

[0127] First, in step S601 of Figure 12, the acquisition unit 123a acquires the first image and information on the weights of the machine learning model. The first image and weight information are the same as the first image and weight information acquired in step S301 of Example 1.

[0128] Next, in step S602, the estimation unit 123b inputs the first image into the machine learning model to generate a third image. The third image is an upscaled version of the first image.

[0129] Next, in step S603, the estimation unit 123b generates a contrast map (second contrast map). In this embodiment, first, a first contrast map with the same number of pixels as the first image is generated based on the first image. Next, the first contrast map is enlarged to generate a second contrast map with the same number of pixels as the third image.

[0130] In this embodiment, the first image has pixels corresponding to each pixel of the first contrast map. The pixel value of each pixel in the first contrast map is calculated based on the ratio of the sum of the absolute difference between the pixel value of the corresponding pixel in the first image and the pixel values ​​of the eight adjacent pixels, as shown in equation (1) above. In this embodiment, a larger pixel value in the first contrast map indicates a higher contrast of the corresponding pixel in the first image.

[0131] The second contrast map is generated by expanding the first contrast map using interpolation. Similar to Example 1, known interpolation methods such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation are used for the interpolation process.

[0132] Next, in step S604, the estimation unit 123b generates a fourth image. In this embodiment, the fourth image has the same number of pixels as the third image and is generated by enlarging the first image using interpolation processing.

[0133] In this embodiment, the contrast map (second contrast map) was generated using the first image, but it may also be generated using the fourth image. For example, the pixel value of each pixel in the contrast map may be calculated based on the ratio of the sum of the absolute difference between the pixel value of the corresponding pixel in the fourth image and the pixel values ​​of the eight adjacent pixels, as shown in equation (1) above.

[0134] Next, in step S605, the estimation unit 123b generates a second image. The second image is an image obtained by upscaling the first image with high accuracy. In this embodiment, the second image is generated by weighting the fourth image and the third image using a contrast map according to equation (3) described above. However, in this embodiment, I in equation (3) gt Let (p,q) be the pixel value at position (p,q) in the second image, and I old_gt Let (p,q) be the pixel value at position (p,q) in the third image, and I tr Let (p,q) be the pixel value at position (p,q) in the fourth image.

[0135] In this embodiment, a larger pixel value in the contrast map indicates a higher contrast in the corresponding pixels of the first image (and fourth image). Therefore, according to equation (3), the second image is generated such that the weight of the pixel in the third image corresponding to a pixel with high contrast in the first image becomes smaller.

[0136] According to the embodiment described above, a machine learning model can be trained to generate a second image in which the first image is upscaled with high accuracy during the estimation phase. This machine learning model is particularly effective when the contrast distribution of the first training image 201 and the first ground truth image 203 is concentrated in the low contrast range.

[0137] The above embodiments include the following methods.

[0138] (Composition 1) A generation process that generates a first output image using the first training image and a machine learning model, An acquisition step of obtaining an error based on a first ground truth image containing the same object as the object included in the first training image and the first output image, The process includes an update step of updating the parameters of the machine learning model based on the aforementioned error, An image processing method characterized in that, in the acquisition step, the error is acquired based on contrast information generated based on at least one of the first training image and the first ground truth image. (Configuration 2) In the acquisition step, the error is acquired based on the difference between the second correct image and the first output image. The image processing method according to configuration 1, characterized in that the second ground truth image is an image generated based on the first ground truth image and the contrast information. (Composition 3) The image processing method according to configuration 2, characterized in that the second ground truth image is an image generated based on the first ground truth image, the first training image, and the contrast information. (Composition 4) The aforementioned second ground truth image is an image generated by combining the second training image using the contrast information with the aforementioned first ground truth image. The image processing method according to configuration 3, characterized in that the second training image is an image generated based on the first training image. (Composition 5) The second training image is an image obtained by enlarging the first training image through interpolation. The image processing method according to configuration 4, characterized in that the first correct image and the second correct image are images with a larger number of pixels than the first training image. (Composition 6) The second ground truth image is an image generated based on the first ground truth image, the third ground truth image, and the contrast information. The image processing method according to configuration 2, characterized in that the third correct image is an image to which at least one of the following processes has been performed: blurring, contrast reduction, and brightness reduction, on the first correct image. (Composition 7) When the contrast information is generated based on the first training image, the first pixel of the first training image corresponds to the third pixel of the first ground truth image, the second pixel of the first training image corresponds to the fourth pixel of the first ground truth image, and the contrast corresponding to the first pixel is higher than the contrast corresponding to the second pixel, The image processing method according to configuration 4, characterized in that the weight for the third pixel in the synthesis is smaller than the weight for the fourth pixel in the synthesis. (Composition 8) When the contrast information is generated based on the first ground truth image, and the contrast corresponding to the third pixel of the first ground truth image is higher than the contrast corresponding to the fourth pixel of the first ground truth image, The image processing method according to configuration 4, characterized in that the weight for the third pixel in the synthesis is smaller than the weight for the fourth pixel in the synthesis. (Composition 9) In the acquisition step, the error is acquired based on the difference between the first correct image and the second output image. The image processing method according to any one of configurations 1 to 8, characterized in that the second output image is an image generated by adding, pixel by pixel, the pixel values ​​of an image obtained by transforming the first output image based on the contrast information and the pixel values ​​of an image based on the first training image. (Composition 10) When the first difference is based on the first output image and the first ground truth image, the second difference is based on the first output image and the first training image, and the third difference is based on the first output image and the third ground truth image, and the third ground truth image is an image on which at least one of the following processes has been applied to the first ground truth image: blurring, contrast reduction, and brightness reduction, The image processing method according to any one of configurations 1 to 8, characterized in that, in the acquisition step, the error is acquired by combining the first difference and the second difference or the third difference using the contrast information. (Composition 11) The process further includes generating information related to the contrast, The contrast information includes a plurality of pixels and the pixel value for each pixel, The first training image includes pixels corresponding to each of the plurality of pixels in the contrast information, The image processing method according to any one of configurations 1 to 10, characterized in that the pixel value of a specific pixel in the contrast information is calculated based on the amount of change in the pixel value of a subregion including the pixel corresponding to the specific pixel in the first training image. (Composition 12) The image processing method according to configuration 11, characterized in that the pixel value of the specific pixel in the contrast information is calculated based on the ratio of the difference between the pixel value of the pixel corresponding to the specific pixel in the first training image and the pixel value of the pixel adjacent to the corresponding pixel, and the sum of the pixel value of the corresponding pixel and the pixel values ​​of the adjacent pixels. (Composition 13) The image processing method according to any one of configurations 1 to 12, further comprising the step of generating a second image using the first image and the machine learning model with updated parameters. (Composition 14) The first generation step involves generating a third image using the first image and a machine learning model, A second generation step generates contrast information based on the first image, The process includes a third generation step of generating a second image based on the first image, the third image, and the contrast information, The contrast information includes a plurality of pixels and the pixel value for each pixel, The first image includes pixels corresponding to each of the plurality of pixels in the contrast information, An image processing method characterized in that, in the second generation step, the pixel value of the specific pixel in the contrast information is generated based on the amount of change in the pixel value of a subregion including the pixel corresponding to the specific pixel in the first image. (Composition 15) In the third generation step, the second image is generated by combining the third image and the fourth image using the contrast information. The image processing method according to configuration 14, characterized in that the fourth image is an image generated based on the first image. (Composition 16) The fourth image is an image obtained by upscaling the first image through interpolation. The image processing method according to configuration 15, characterized in that the second image and the third image have a greater number of pixels than the first image. (Composition 17) When the fifth pixel of the first image corresponds to the seventh pixel of the third image, the sixth pixel of the first image corresponds to the eighth pixel of the third image, and the contrast corresponding to the fifth pixel of the first image is higher than the contrast corresponding to the sixth pixel of the first image, The image processing method according to any one of configurations 14 to 16, characterized in that the second image is an image in which the weight of the seventh pixel corresponding to the seventh pixel is smaller than the weight of the eighth pixel corresponding to the eighth pixel. (Composition 18) The contrast information includes a plurality of pixels and the pixel value for each pixel. The first image has pixels corresponding to each of the plurality of pixels in the contrast information, The image processing method according to any one of configurations 14 to 17, characterized in that, in the second generation step, the pixel value of the specific pixel in the contrast information is calculated based on the ratio of the difference between the pixel value of the pixel corresponding to the specific pixel in the first image and the pixel value of the pixel adjacent to the corresponding pixel, and the sum of the pixel value of the corresponding pixel and the pixel value of the adjacent pixel.

[0139] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0140] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention. [Explanation of Symbols]

[0141] 101,301 Learning device 102,302 Imaging devices 123 Image Estimation Unit 303 Image Estimation Device

Claims

1. A generation process that generates a first output image using the first training image and a machine learning model, An acquisition step of obtaining an error based on a first ground truth image containing the same object as the object included in the first training image and the first output image, The process includes an update step of updating the parameters of the machine learning model based on the aforementioned error, An image processing method characterized in that, in the acquisition step, the error is acquired based on contrast information generated based on at least one of the first training image and the first correct image.

2. In the acquisition step, the error is acquired based on the difference between the second correct image and the first output image. The image processing method according to claim 1, characterized in that the second correct image is an image generated based on the first correct image and the contrast information.

3. The image processing method according to claim 2, characterized in that the second correct image is an image generated based on the first correct image, the first training image, and the contrast information.

4. The second ground truth image is an image generated by combining the second training image using the contrast information with the first ground truth image. The image processing method according to claim 3, characterized in that the second training image is an image generated based on the first training image.

5. The second training image is an image obtained by enlarging the first training image through interpolation. The image processing method according to claim 4, characterized in that the first correct image and the second correct image are images with a larger number of pixels than the first training image.

6. The second correct image is an image generated based on the first correct image, the third correct image, and the contrast information. The image processing method according to claim 2, characterized in that the third correct answer image is an image to which at least one of the following processes has been performed: blurring, contrast reduction, and brightness reduction, on the first correct answer image.

7. When the contrast information is generated based on the first training image, the first pixel of the first training image corresponds to the third pixel of the first ground truth image, the second pixel of the first training image corresponds to the fourth pixel of the first ground truth image, and the contrast corresponding to the first pixel is higher than the contrast corresponding to the second pixel, The image processing method according to claim 4, characterized in that the weight for the third pixel in the synthesis is smaller than the weight for the fourth pixel in the synthesis.

8. When the contrast information is generated based on the first ground truth image, and the contrast corresponding to the third pixel of the first ground truth image is higher than the contrast corresponding to the fourth pixel of the first ground truth image, The image processing method according to claim 4, characterized in that the weight for the third pixel in the synthesis is smaller than the weight for the fourth pixel in the synthesis.

9. In the acquisition step, the error is acquired based on the difference between the first correct image and the second output image. The image processing method according to claim 1, characterized in that the second output image is an image generated by adding, pixel by pixel, the pixel values ​​of an image obtained by converting the first output image based on the contrast information and the pixel values ​​of an image based on the first training image.

10. When the first difference is based on the first output image and the first ground truth image, the second difference is based on the first output image and the first training image, and the third difference is based on the first output image and the third ground truth image, and the third ground truth image is an image on which at least one of the following processes has been applied to the first ground truth image: blurring, contrast reduction, and brightness reduction, The image processing method according to claim 1, characterized in that, in the acquisition step, the error is acquired by combining the first difference and the second difference or the third difference using the contrast information.

11. The process further includes a step of generating information related to the contrast, The contrast information includes a plurality of pixels and the pixel value for each pixel, The first training image includes pixels corresponding to each of the plurality of pixels in the contrast information, The image processing method according to claim 1, characterized in that the pixel value of a specific pixel in the contrast information is calculated based on the amount of change in the pixel value of a subregion including the pixel corresponding to the specific pixel in the first training image.

12. The image processing method according to claim 11, characterized in that the pixel value of the specific pixel in the contrast information is calculated based on the ratio of the difference between the pixel value of the pixel corresponding to the specific pixel in the first training image and the pixel value of the pixel adjacent to the corresponding pixel, and the sum of the pixel value of the corresponding pixel and the pixel values ​​of the adjacent pixels.

13. The image processing method according to claim 1, further comprising the step of generating a second image using the first image and the machine learning model with updated parameters.

14. A first generation step involves generating a third image using the first image and a machine learning model, A second generation step generates information about contrast based on the first image, The process includes a third generation step of generating a second image based on the first image, the third image, and the contrast information, The contrast information includes a plurality of pixels and the pixel value for each pixel, The first image includes pixels corresponding to each of the plurality of pixels in the contrast information, An image processing method characterized in that, in the second generation step, the pixel value of the specific pixel in the contrast information is generated based on the amount of change in the pixel value of a subregion including the pixel corresponding to the specific pixel in the first image.

15. In the third generation step, the second image is generated by combining the third image and the fourth image using the contrast information, The image processing method according to claim 14, characterized in that the fourth image is an image generated based on the first image.

16. The fourth image is an image obtained by upscaling the first image through interpolation. The image processing method according to claim 15, characterized in that the second image and the third image have a greater number of pixels than the first image.

17. When the fifth pixel of the first image corresponds to the seventh pixel of the third image, the sixth pixel of the first image corresponds to the eighth pixel of the third image, and the contrast corresponding to the fifth pixel of the first image is higher than the contrast corresponding to the sixth pixel of the first image, The image processing method according to claim 14, characterized in that the second image is an image in which the weight of the seventh pixel corresponding to the seventh pixel is smaller than the weight of the eighth pixel corresponding to the eighth pixel.

18. The contrast information includes a plurality of pixels and the pixel value for each pixel. The first image has pixels corresponding to each of the plurality of pixels in the contrast information, The image processing method according to claim 14, characterized in that, in the second generation step, the pixel value of the specific pixel in the contrast information is calculated based on the ratio of the difference between the pixel value of the pixel corresponding to the specific pixel in the first image and the pixel value of the pixel adjacent to the corresponding pixel, and the sum of the pixel value of the corresponding pixel and the pixel values ​​of the adjacent pixels.

19. A program characterized by causing a computer to perform processing according to the image processing method described in any one of claims 1 to 18.

Images