Image processing method, image processing device, image processing system, and program

Abstract

To provide an image processing method capable of preventing inherent adverse effects occurring in image estimation using a machine learning model.SOLUTION: An image processing method includes the steps of: acquiring a first model output generated on the basis of a photographed image by a first machine learning model; acquiring a second model output generated on the basis of the photographed image by a second machine learning model different from the first machine learning model; and generating an estimation image by using the first model output and the second model output on the basis of a comparison based on the second model output and the photographed image or the first model output.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing method for suppressing inherent problems that occur when image estimation is performed based on a captured image using a machine learning model. [Background technology]

[0002] Image estimation using machine learning models can achieve high effectiveness compared to theory-based image estimation using assumptions and approximations. With theory-based image estimation, the effectiveness is reduced due to elements ignored by assumptions and approximations. However, with machine learning models, by learning using training data that includes these elements, image estimation based on training data without assumptions or approximations can be achieved, making it more effective.

[0003] For example, in a technique for correcting blur in a captured image, saturated pixels of the captured image are cited as the aforementioned element. In theory-based methods such as the Wiener filter, it is assumed that saturated pixels do not exist, and therefore blur is not properly corrected around saturated pixels, resulting in problems such as ringing. In response to this, Non-Patent Document 1 discloses a method for correcting blur in a captured image using a convolutional neural network (CNN), which is one of the machine learning models. By blurring images having signal values ​​equal to or greater than the saturation value of the captured image to generate a learning dataset, and training a CNN with the learning dataset, it is possible to perform blur correction while suppressing problems even around saturated pixels. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Li Xu,et al.,Deep Convolutional Neural Network for Image Deconvolution,Advances in Neural Information Processing Systems 27,NIPS2014 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the method disclosed in Non-Patent Document 1, artifacts (false structures) may occur in the subject at positions unrelated to the saturated pixels. Specifically, artifacts are local decreases or increases in signal values ​​that are different from the actual structure of the subject. Details regarding artifacts and the reasons for their occurrence will be described later, but artifacts are one of the inherent problems that occur in image estimation using machine learning models. In image estimation other than blur correction, adverse effects inherent to machine learning models may also occur.

[0006] Therefore, an object of the present invention is to provide an image processing method, an image processing device, an image processing system, and a program that are capable of suppressing the inherent disadvantages that arise in image estimation using a machine learning model. [Means for solving the problem]

[0007] An image processing method as one aspect of the present invention includes a step of acquiring a first model output generated based on a captured image by a first machine learning model, a step of acquiring a second model output generated based on the captured image by a second machine learning model different from the first machine learning model, and a step of generating an estimated image using the first model output and the second model output based on a comparison between the second model output and the captured image or the first model output.

[0008] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0009] According to the present invention, it is possible to provide an image processing method, an image processing device, an image processing system, and a program capable of suppressing the inherent drawbacks that arise in image estimation using a machine learning model. [Brief description of the drawings]

[0010] [Figure 1] FIG. 4 is a diagram showing a flow of generating an estimated image in the first embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] FIG. 1 is an explanatory diagram of an artifact in the first embodiment. [Diagram 5] FIG. 4 is an explanatory diagram of a cause of artifact occurrence in the first embodiment. [Figure 6] 1 is a flowchart of learning a machine learning model in each embodiment. [Figure 7] 4 is a flowchart of generation of an estimated image in the first embodiment. [Figure 8] FIG. 2 is an explanatory diagram regarding generation of a first map in the first embodiment. [Figure 9] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 10] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 11] FIG. 11 is a configuration diagram of a machine learning model in the second embodiment. [Figure 12] 13 is a flowchart of generation of an estimated image in the second embodiment. [Figure 13] FIG. 11 is a block diagram of an image processing system according to a third embodiment. [Figure 14] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 15] 13 is a flowchart of generation of an estimated image in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0012] Before describing the embodiments in detail, the inherent disadvantages of image estimation using a machine learning model, which is the subject of the present invention, will be described. When estimating an image, the machine learning model can execute processing with different effects depending on the features of the image, rather than processing with a uniform effect on the input image. In the above-mentioned example of blur correction, the machine learning model internally determines whether a blur image involving saturated pixels (saturated blur image) is a blur image involving saturated pixels or not (non-saturated blur image), and executes different correction processes to correct both blur images. However, the machine learning model may make a misjudgment, and when a correction process corresponding to a saturated blur image is executed on a non-saturated blur image, the above-mentioned artifacts occur. The same is true for image estimation other than blur correction including saturated pixels. When the machine learning model executes processing with different effects depending on the features of the input image, executing a process different from the process corresponding to the features will cause disadvantages. What kind of phenomenon the disadvantages will be depends on the content of the image estimation executed by the machine learning model and the learning method. Specific examples will be introduced in each embodiment.

[0013] Next, the gist of the present invention for solving this problem will be described. In the present invention, by using the estimation results (first model output and second model output) estimated from the captured image by each of two different machine learning models (first machine learning model and second machine learning model), adverse effects specific to the machine learning models are suppressed. For the purpose of image estimation, the second machine learning model is more effective than the first machine learning model, but there is a possibility that adverse effects occur in the second model output. On the other hand, the first machine learning model is less effective for the purpose of image estimation, but there is no (or little) adverse effect in the first model output. An estimated image corresponding to the purpose of image estimation is generated based on a comparison based on the captured image or the first model output and the second model output. The estimated image is an image in which adverse effects are suppressed, and the first model output and the second model output are used for generation. It is also preferable to generate a first map based on a comparison based on the captured image or the first model output and the second model output, and generate an estimated image based on the first map. The first map is a map for determining an area where the first model output or the second model output is used in generating an estimated image. That is, the first map shows an adverse area in the second model output. By using the first model output without any adverse effects in the adverse area, an estimated image can be obtained in which the desired effect of image estimation is large and the adverse effects are suppressed.

[0014] In the following description, the stage of determining the weights of the machine learning model based on the learning data set is called learning, and the stage of performing image estimation from a captured image with the machine learning model using the learned weights is called estimation. Machine learning models include, for example, neural networks, genetic programming, Bayesian networks, etc. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), RNNs (Recurrent Neural Networks), etc. EXAMPLES

[0015] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, the blur correction of a captured image including saturated pixels is the object of image estimation using a machine learning model. The blur to be corrected is aberration and diffraction occurring in the optical system, an optical low-pass filter, and blur caused by the pixel aperture of the image sensor. However, this embodiment is not limited to this, and the effect of this embodiment can be obtained similarly with respect to the correction of blur caused by defocus and shaking. The effect of this embodiment can be obtained similarly with respect to image estimation other than blur correction.

[0016] Fig. 2 is a block diagram of image processing system 100. Fig. 3 is an external view of image processing system 100. Image processing system 100 has a learning device 101 and an imaging device 102 connected to each other via a wired or wireless network. Learning device 101 has a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114.

[0017] The imaging device 102 includes an optical system 121, an imaging element 122, an image processing unit (image processing device) 123, a storage unit 124, a communication unit 125, a display unit 126, and a system controller 127. The optical system 121 collects light incident from a subject space to form a subject image. The optical system 121 has functions such as zooming, aperture adjustment, and autofocus as necessary. The imaging element 122 converts the subject image into an electric signal by photoelectric conversion to generate a captured image. The imaging element 122 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The captured image is affected by aberration and diffraction occurring in the optical system 121 and blurring due to the pixel aperture of the imaging element 122. In addition, if an optical low-pass filter is present, the blurring also acts.

[0018] The image processing unit 123 has an acquisition unit 123a, a blur correction unit 123b, and a calculation unit 123c, and generates an estimated image in which the blur of the captured image is corrected. The first machine learning model and the second machine learning model generate a first model output and a second model output that estimate the blur correction component acting on the captured image, respectively, and execute the image processing method of this embodiment to generate an estimated image in which adverse effects are suppressed. The weights learned by the learning device 101 are used for the first machine learning model and the second machine learning model. Information on the weights is read in advance from the learning device 101 via the communication unit 125 and stored in the storage unit 124. Details regarding the learning and estimation of the machine learning model and the suppression of adverse effects will be described later. The estimated image is stored in the storage unit 124 and displayed on the display unit 126. The above operations are controlled by the system controller 127. The captured image and the estimated image may be grayscale images or images having multiple color components. Moreover, it may be an undeveloped RAW image or a developed image.

[0019] Next, a detailed description will be given of the inherent problems that occur when blur correction of a captured image is performed using a machine learning model. The problem in this case is the occurrence of artifacts (false structures) caused by local decreases or increases in signal values ​​that are different from the actual structure of the subject. The artifacts will be described with reference to FIG. 4. FIG. 4 is an explanatory diagram of the artifacts. FIGS. 4(A) to 4(C) show spatial changes in the signal values ​​of an image, each of which corresponds to a color component of RGB (Red, Green, Blue). In FIGS. 4(A) to 4(C), the horizontal axis indicates the spatial coordinate, and the vertical axis indicates the signal value.

[0020] Here, since the image is an 8-bit image, the saturation value is 255. In Fig. 4 (A) to (C), the dashed line indicates the captured image (blurred image), and the thin solid line indicates the correct image without blur. Since none of the pixels reach the saturation value, there are no saturated pixels. The dotted line is the second model output (blur-corrected image) in which the blur of the captured image is corrected using the second machine learning model. The second machine learning model is a model that corrects both the blurred image involving saturated pixels (saturated blurred image) and the blurred image not involving saturated pixels (non-saturated blurred image). In the second model output, the blur of the edge is corrected, but a decrease in signal value occurs near the center that is not present in the correct image. This decrease occurs at a position away from the edge, not adjacent to it, and the occurrence area is wide, so it is understood to be a problem different from undershoot. This is an artifact that is a problem unique to the machine learning model. Also, as can be seen from the comparison of Fig. 4 (A) to (C), the degree of decrease in signal value differs depending on the color component. In Figure 4, the degree of reduction increases in the order of G, R, and B. Therefore, even though the correct image has flat areas, the second model output produces dark green areas as artifacts. Note that Figure 4 shows an example in which the signal is reduced from the correct image, but there are also cases in which the signal increases.

[0021] Next, the reason why artifacts occur will be explained. The reason is that the machine learning model cannot completely distinguish between a blurred image including saturated pixels (saturated blurred image) and a blurred image including only non-saturated pixels (non-saturated blurred image). To explain this, first, it will be shown that the machine learning model performs different blur correction depending on the presence or absence of saturation. Figures 5(A) to (D) show spatial changes in signal values ​​in a captured image and a genuine image without blur. In Figures 5(A) to (D), the horizontal axis indicates spatial coordinates, and the vertical axis indicates signal values. In Figures 5(A) to (D), the solid lines indicate the captured image, and the dashed lines indicate the genuine image.

[0022] 5(A) to (C) show images of the same object, each with a different exposure during imaging. In the exposure of FIG. 5(C), there are saturated pixels. The arrows in FIG. 5 indicate the signal difference between the captured image and the correct image at the pixel where the signal value of the object in the correct image is zero. This signal difference corresponds to the ideal correction residual for correcting blur. The more overexposed, the larger the correction residual. Therefore, in blur correction, the correction residual is determined by the brightness of the object, and the brighter it is, the larger the correction residual becomes. FIG. 5(D) shows the spatial change in signal value when an object of a different shape is imaged at the limit of exposure without saturation. The blurred images in FIG. 5(C) and FIG. 5(D) are almost the same, but the correction residual at the same signal value is larger in FIG. 5(C) than in FIG. 5(D). In other words, a machine learning model capable of correcting saturated blurred images makes different estimates for saturated blurred images and non-saturated blurred images even at the same brightness, and estimates a larger correction residual for saturated blurred images.

[0023] However, it is not easy to determine whether a pixel to be subjected to blur correction is a part of a saturated blur image or a non-saturated blur image. This determination becomes more difficult as the pixel to be subjected to blur correction is farther away from the saturated pixel. In particular, the brighter the subject is or the greater the blur, the more the saturated blur image spreads to positions farther away from the saturated pixel, making the determination more difficult. Therefore, when performing blur correction, the machine learning model may determine that the target pixel is a part of a saturated blur image even though it is a non-saturated blur image. As described above, when performing correction on a saturated blur image, the machine learning model estimates a correction residual that is larger than that of a non-saturated blur image. Therefore, an excessive correction residual is applied to the non-saturated blur image, and an artifact occurs in which the signal value decreases only in the area erroneously determined to be a saturated blur image.

[0024] Conversely, in a machine learning model (first machine learning model) that does not correct saturation blurred images, there is no judgment between non-saturation blurred images and saturation blurred images, so no artifacts occur. The dashed lines in Figures 4(A) to (C) are the first model outputs generated by the first machine learning model that does not correct saturation blurred images and corrects only non-saturation blurred images, and although the blur is corrected, no artifacts occur. However, in this case, the blur of the saturated blurred images is not corrected.

[0025] In addition, optical systems for visible light are generally designed to maximize the performance of G among RGB. In other words, because the blur (PSF: point spread function) of R and B is larger than that of G, the edges of a saturated blurred image of a highly luminous subject tend to be colored R or B (purple fringing corresponds to this). When correcting this saturated blurred image, the correction residual of R or B is larger than that of G. Therefore, if a pixel of a non-saturated blurred image is erroneously determined to be a pixel of a saturated blurred image, the reduction in the signal values ​​of R and B is greater than that of G, and artifacts occur as dark areas colored green, as shown in Figures 4(A) to (C).

[0026] In addition, while Fig. 5 shows a case where the captured image has a larger signal value than the correct image, in areas close to the high-brightness parts of the blurred image, the correct image has a larger signal value. Therefore, when a pixel of a non-saturated blurred image is erroneously determined to be a saturated blurred image, there is a possibility that an artifact that increases the signal value more than that of the correct image may occur.

[0027] Next, a method for suppressing artifacts will be described. The cause of artifacts is that the machine learning model cannot accurately determine whether the target pixel is included in a saturated blur image or a non-saturated blur image. Therefore, artifacts can be suppressed by teaching the machine learning model which image the target pixel is included in. However, this method is difficult to realize. For example, assume that there is a HDR (high dynamic range) correct answer image in which there is no blur (or the blur is sufficiently small) and the signal value is not clipped at the saturation value. If only the area of ​​the saturated pixels of the captured image is extracted from the HDR correct answer image and blur generated by imaging is applied, a map indicating the range of the saturated blur image can be generated. This map is input to the machine learning model together with the captured image and estimation is performed (a similar map is also input during learning), and artifacts are suppressed. However, since the purpose of blur correction is to estimate the correct answer image (or HDR correct answer image) in the first place, the HDR correct answer image is unknown and this method cannot be used. Therefore, in this embodiment, artifacts are suppressed by using two machine learning models (first and second machine learning models) that have different blur correction effects for high-luminance objects. A specific method for suppressing artifacts will be described later in the description of the learning of the learning device 101 and the processing executed by the image processing unit 123.

[0028] Next, learning of weights of machine learning models (first machine learning model and second machine learning model) executed by learning device 101 will be described with reference to FIG. 6. FIG. 6 is a flowchart of learning of machine learning models. Each step in FIG. 6 is executed by any of memory unit 111, acquisition unit 112, calculation unit 113, or update unit 114 of learning device 101. Learning is performed separately for each of the first machine learning model and the second machine learning model. Therefore, in the following description, parts that are the same between the first machine learning model and the second machine learning model will be simply referred to as machine learning models.

[0029] First, in step S101, the acquisition unit 112 acquires one or more pairs of correct images and input data from the storage unit 111. The input data is data to be input to the machine learning model, and includes a training image during learning. The training image and the correct image are a pair of images in which the same object exists and the strength of blurring acting thereon is different. The first machine learning model prepares a pair of training images and correct images as shown below in order to learn the second machine learning model so as to reduce the blur correction effect on a high-luminance object. In either method, a pair of training images and correct images is generated by performing an imaging simulation on an original image. Note that the original image is an image captured by a CG (Computer Graphics) or an optical system with higher performance than the optical system 121, and has higher frequency components than the captured image. Alternatively, an original image having higher frequency components than the captured image may be generated by reducing the image. The original image is one or more images having various objects (edges of different directions and strengths, gradations, flat parts, etc.).

[0030] The first example of pair generation is a method of changing the original image used in each of the first machine learning model and the second machine learning model. In the first machine learning model, an image to which blur generated by the optical system 121 and the image sensor 122 is added to an original image having only a signal value up to the saturation value of the captured image is set as a training image, and an image to which no blur is added (or a smaller blur is added) is set as a correct image. In this case, since the original image has only a signal value up to the saturation value, there is no pair in which subject information is lost due to saturation in both the training image and the correct image. Therefore, the first machine learning model trained with this training image and the correct image only learns correction for non-saturated blurred images, and there is no judgment between saturated blurred images and non-saturated blurred images, so no artifacts occur.

[0031] Next, in the second machine learning model, an image to which blurring generated by the optical system 121 and the image sensor 122 is added to an original image having a signal value exceeding the saturation value of the captured image is set as a training image, and an image to which no blurring is added (or with a small blur) is set as a correct image. Here, the signal values ​​of the training image and the correct image are clipped at the saturation value of the captured image. The second machine learning model learns how to correct both saturated and non-saturated blurred images, so it can also correct saturated blurred images, but artifacts may occur.

[0032] The second example of pair generation is a method of changing the method of generating a correct image in each of the first machine learning model and the second machine learning model. In this method, both the first machine learning model and the second machine learning model use an original image that also has a signal value that exceeds the saturation value of the captured image. The second machine learning model is the same as the first example. In the first machine learning model, a training image is generated in the same manner as in the first example. The correct image is generated as follows. First, an area having a signal value equal to or greater than the saturation value of the captured image is extracted from the original image or the training image. The original image, in which the signal value is set to zero except for the extracted area, is given a blur similar to that of the training image, and is added to the original image in which only the extracted area is set to zero, and clipped at the saturation value. As a result, a correct image is generated in which only high-brightness objects exceeding the saturation value are given the same blur as the training image. By using this pair of correct image and training image, the first machine learning model learns to correct the blur of only non-saturated blur images, and no artifacts are generated.

[0033] It should be noted that the format of the captured image to be blurred must match the format of the training image. For example, if blurring is to be performed on an undeveloped RAW image, the training image must also be equivalent to an undeveloped RAW image. If blurring is to be performed on captured images having multiple color components all at once, the training image must also have multiple color components. If only blurring is to be performed in the machine learning model, the format of the correct image is the same as that of the training image. If blurring is to be corrected and RAW image development processing is to be performed simultaneously in the machine learning model, the training image is an undeveloped RAW image, and the correct image is a developed image. In this way, the format of the correct image may be changed as necessary.

[0034] The input data may also include information other than the training image. For example, the information is information on the focal length, F-number, focal distance, image height, azimuth, separation method and cutoff frequency of the optical low-pass filter, pixel pitch of the image sensor 122, etc. of the optical system 121 corresponding to the blur added to the training image. This allows the machine learning model to identify the blur even when multiple different blurs are mixed, so that blur correction can be performed with high accuracy. In addition, when noise is added to the training image, information indicating the strength of the added noise may be included. When information other than the training image is added to the learning input, similar information is also input to the machine learning model during estimation.

[0035] Next, in step S102, the calculation unit 113 inputs the input data to the machine learning model to generate a model output. The model output is a training image that has been deblurred by the machine learning model in the middle of learning. In this embodiment, the machine learning model uses CNN, but other models can be similarly applied. The CNN has multiple convolution layers. In the first learning, the weights of the CNN (filter coefficients and biases) can be determined by random numbers. The first machine learning model and the second machine learning model are CNNs with the same network structure, but it is not necessary to use CNNs with the same structure.

[0036] Next, in step S103, the update unit 114 updates the weights of the machine learning model based on the difference (error, i.e., loss function) between the model output and the correct image. In this embodiment, the Euclidean norm of the difference in signal value between the model output and the correct image is used as the loss function. However, the loss function is not limited to this. If multiple sets of input data and correct images are acquired in step S101, the value of the loss function is calculated for each set. The weights are updated from the calculated value of the loss function by backpropagation or the like. The estimated model output may also be the difference (corrected residual) between the correct image and the training image. In this case, the weights are updated using the error between the model output and the corrected residual (the difference between the correct image and the training image).

[0037] Next, in step S104, the update unit 114 determines whether the learning of the machine learning model is completed. The completion of the learning can be determined by whether the number of iterations of the learning (updating of the weight) reaches a predetermined number, whether the amount of change in the weight at the time of updating is smaller than a predetermined value, or the like. If it is determined in step S104 that the learning of the weight is not completed, the process returns to step S101, and the acquisition unit 112 acquires one or more pairs of new input data and a correct image. On the other hand, if it is determined that the learning of the weight is completed, the update unit 114 ends the learning, and stores the weight information in the storage unit 111.

[0038] Next, the generation of an estimated image in which blur of a captured image is corrected, which is executed by the image processing unit 123, will be described with reference to Fig. 1 and Fig. 7. Fig. 1 is a diagram showing a flow of generation of an estimated image. Each step in Fig. 7 is executed by any one of the acquisition unit 123a, the blur correction unit 123b, or the calculation unit 123c of the image processing unit 123.

[0039] First, in step S201, the acquisition unit 123a acquires input data 201 and information on weights used in each of the first machine learning model and the second machine learning model. The input data 201 includes captured images, and when the input data during learning includes information other than training images, the input data also includes similar information. The weight information is acquired from the storage unit 124.

[0040] Next, in step S202, the blur correction unit (acquisition means) 123b generates (acquires) a first model output 202 and a second model output 203 from input data 201 including a captured image using the first machine learning model and the second machine learning model, respectively. The first model output 202 and the second model output 203 are captured images with blur corrected. The second model output 203 is an image in which all blur images (non-saturated blur images and saturated blur images) have been corrected, but there is a possibility that artifacts have occurred. The first model output 202 is an image in which only non-saturated blur images have been corrected, and no artifacts exist. Note that, when the model output is set as a correction residual during learning, the first model output 202 and the second model output 203 are also correction residuals.

[0041] Next, in step S203, the calculation unit 123c generates a second map 204 based on a comparison between the first model output 202 and the second model output 203. In this embodiment, the difference between the first model output 202 and the second model output 203 is used as the comparison, but is not limited to this. As another example, the ratio of the difference (correction residual) between each of the first model output 202 and the second model output 203 and the captured image may be used. Alternatively, the first model output 202 and the second model output 203 may be divided into a plurality of partial regions, and correlation may be taken between partial regions at the same position. Also, when the model output is a correction residual, the comparison can be made using any of the difference, the ratio, or the correlation.

[0042] The second map 204 is a map showing an area where the difference between the first model output 202 and the second model output 203 satisfies a predetermined condition, and has a second value in the area. The predetermined condition indicates that the difference between the first model output 202 and the second model output 203 is large (the absolute value is large in the case of difference or ratio, and small in the case of correlation). In this embodiment, the predetermined condition is that the absolute value of the difference exceeds a predetermined threshold, and the difference is binarized depending on whether it exceeds the predetermined threshold to generate the second map 204. The area that exceeds the predetermined threshold is set to 1 (second value), and the rest is set to 0 (first value). However, this embodiment is not limited to this, and the second map 204 may be generated by clipping the difference at a predetermined threshold (second value).

[0043] FIG. 8(A) is an example of the second map 204. Areas where the difference exceeds a predetermined threshold are represented by white (value 1), and other areas are represented by diagonal lines (value 0). Both the first model output 202 and the second model output 203 correct non-saturated blurred images, so the difference is small in areas corresponding to non-saturated blurred images. On the other hand, correction of saturated blurred images and artifacts occur only in the second model output 203, so the difference is large in those areas and appears in the second map 204 as white areas 211 and 212 in FIG. 8(A). Note that FIG. 8(A) shows an example of only one color component. When the captured image has multiple color components, the second map 204 also has multiple color components. When multiple color components are arranged in pixels at different positions like a Bayer image, it is preferable to separate the captured image, the first model output 202, and the second model output 203 into each color component and execute the processing from step S203 onwards.

[0044] 7, the calculation unit 123c generates a first map 205 based on the second map 204 and the positions of the saturated pixels of the captured image. The white areas 211 and 212 of the second map 204 indicate areas where the saturated blurred image has been corrected or areas where an artifact has occurred. The saturated blurred image includes saturated pixels of the captured image. In other words, if the areas surrounded by the white areas 211 and 212 do not include saturated pixels of the captured pixels, it is an artifact.

[0045] Fig. 8(B) is a map in which the captured image is binarized by the saturation value, where only saturated pixels are 1 (second value) and others are 0. From the positions of the saturated pixels shown in Fig. 8(B), it can be seen that the area surrounded by the white area 212 of the second map 204 does not contain saturated pixels. In other words, the white area 212 is an area where artifacts occur. In this way, it is possible to extract the area where artifacts occur from the second model output 203.

[0046] In this embodiment, specifically, the artifact area is extracted from the second map 204 as follows. First, the positions of saturated pixels in the captured image are obtained. For example, there are five points in FIG. 8B, and when the horizontal and vertical coordinates (x, y) of the upper left corner are (0, 0), they are (3, 8), (3, 9), (4, 7), (4, 8), and (4, 9), respectively.

[0047] Next, one of the positions of the saturated pixels is selected, and its surrounding positions are obtained. For example, if (3,8) is selected, the surrounding positions are four points: (2,8), (3,7), (4,8), and (3,9) (8 points including diagonal positions may be used). Each of these four points is set as a starting point, and the second map 204 is filled. Filling is a process of replacing the signal value of a closed space that includes the starting point and has the same signal value as the starting point with a predetermined value (first value).

[0048] In the example of FIG. 8(A), since the area with a large difference is represented by 1 (second value), it is filled with 0 (first value). For example, a scan line seed fill algorithm or the like is used for filling. Since four starting points are set for each of five saturated pixels, the filling process is repeated 20 times here. By this filling, the white area 211 in FIG. 8(A) is filled with 0, and only the white area 212 remains as the area where the artifact occurs. This becomes a first map 205 indicating the area where the adverse effect occurs in the second model output 203.

[0049] The position of the saturated pixel may be obtained from another image, so long as it is an image based on the captured image. For example, the first model output 202 or the second model output 203 may be used. The first map 205 may be generated by the following method without using binarization or clipping. That is, the second map 204 is set to the difference between the first model output 202 and the second model output 203, and a closed space that includes a starting point based on the position of the saturated pixel and whose difference value satisfies a predetermined condition (the difference is equal to or greater than a predetermined threshold) is removed. Then, a map in which the remaining difference value satisfies the predetermined condition is extracted is set to the first map 205.

[0050] Next, in step S205, the calculation unit (generation means) 123c generates an estimated image 206 based on the first model output 202, the second model output 203, and the first map 205. In the second model output 203, the entire blurred image (non-saturated blurred image and saturated blurred image) of the captured image is corrected, but there is a possibility that artifacts exist. In the first model output 202, only the non-saturated blurred image is corrected, but there are no artifacts. The first map 205 is a map indicating an area in the second model output 203 where artifacts occur (here, pixels having the second value 1 indicate artifacts). Therefore, by replacing the area indicated by the first map 205 in the second model output 203 with the first model output 202, it is possible to obtain an estimated image 206 in which the entire blurred image of the captured image is corrected and artifacts are suppressed.

[0051] Instead of the substitution, a weighted average may be used. In this case, the weight of the first model output 202 is greater than that of the second model output 203 in the artifact occurrence region shown in the first map 205. The generated estimated image 206 is stored in the storage unit 124. In addition, when the first model output 202 and the second model output 203 are correction residuals, the estimated image 206 can be obtained by adding them to the captured image after substitution. In addition, the first model output 202 and the second model output 203 do not need to be synthesized. For example, when it is determined from the first map 205 that there is no artifact, the second model output 203 may be used as the estimated image 206, and when it is determined that there is an artifact, the first model output 202 may be used as the estimated image 206. There is a possibility that the correction effect of the blurred image will be reduced, but it is possible to obtain an estimated image 206 in which adverse effects are suppressed.

[0052] Next, the desirable conditions of this embodiment will be described. The following conditions are also applicable to each embodiment other than this embodiment. First, when the captured image has a plurality of color components, it is desirable that the first map 205 is common to all color components. When steps S203 and S204 are executed for each color component, the area of ​​the artifact indicated by the first map 205 differs depending on the color component. For example, when the captured image is a RAW image, G is likely to be saturated, and there is a possibility that only G is saturated and the neighboring RB is not saturated. For this reason, the position of the saturated pixel differs depending on the color, and the first map 205 differs. If a different first map 205 is used for each color component, for example, the first model output 202 is used for RB and the second model output 203 is used only for G in the same area of ​​the image, and the estimated image appears discontinuous. For this reason, it is desirable that the first map 205 is common to all color components. Examples of such methods include a method of applying the first map 205 generated for a certain color component to all color components, and a method of determining that there is no artifact in all color components (setting a value other than the second value) if at least one color is not the second value in the same or nearby pixels in the first map 205. The method that most effectively suppresses artifacts is a method of determining that there is an artifact in all color components (setting a value other than the second value) if at least one color is the second value in the same or nearby pixels in the first map 205.

[0053] Next, when the captured image, the first model output 202, and the second model output 203 have multiple color components and are undeveloped RAW images, it is preferable that the first map 205 is generated based on a white balance coefficient. This is because the image viewed by the user is in a state in which white balance correction has been applied. The predetermined threshold value in step S203 may be set to a different value for each color component, and the ratio of the values ​​may be determined by the ratio of the white balance coefficients. Alternatively, the predetermined threshold value may be set to a constant value for each color component, and white balance correction may be performed on the captured image, the first model output 202, and the second model output 203 to be compared.

[0054] Similarly, when the captured image, the first model output 202, and the second model output 203 are undeveloped RAW images, it is preferable that the first map 205 is generated based on gamma correction. As described above, the image viewed by the user is in a state in which gamma correction has been applied. It is preferable to change the predetermined threshold in step S203 based on the magnitude of the signal value of the pixel of either the captured image, the first model output 202, or the second model output 203, and the gamma correction. Gamma correction increases the gradation of the dark area and decreases the gradation of the bright area. For this reason, the predetermined threshold is made smaller for pixels with larger signal values ​​and larger for pixels with smaller signal values. Alternatively, the predetermined threshold may be fixed and gamma correction may be performed on the captured image, the first model output 202, and the second model output 203 to be compared. When the model output is a correction residual, gamma correction is taken into consideration in a state in which it is added to the captured image.

[0055] Next, in the second map 204, it is preferable to replace the pixel at the position of the saturated pixel based on the captured image with the second value before performing the filling with the first value. In a comparison between the first model output 202 and the second model output 203, the difference in the position of the saturated pixel of the captured image is likely to be small, and in the second map 204, the position often does not have the second value. In FIG. 8(A), the pixel at the position of the saturated pixel of the captured image has the first value. Therefore, even if the filling process to the first value is performed starting from the position of the saturated pixel of the captured image, the white area 211 is not filled. In order to fill the white area 211, it is necessary to set the periphery of the position of the saturated pixel as the starting point. However, if, for example, four surrounding points are set as the starting points, filling is performed for each starting point, and four fillings are performed, which increases the processing load. However, if the pixel at the position of the saturated pixel is replaced with 1 (second value) in the second map 204, the white area 211 can be filled in only starting from the position of the saturated pixel, thereby reducing the processing load. In the case of Fig. 8(A), the number of times filling is repeated is reduced from 20 to 5.

[0056] It is also preferable to correct the positions of the saturated pixels based on the positions of the pixels replaced with the first value by the filling of the second map 204, and perform filling again with the pixel based on the corrected position of the saturated pixel as the starting point. For example, in FIG. 8(A), there are 20 starting points for filling, including 4 surrounding points for each of the 5 saturated pixels. However, if filling is performed with (2,8) as the starting point, for example, the white area 211 is completely filled, so there is no need to repeat filling thereafter. For this reason, the positions of the pixels replaced with the first value by filling are obtained, and the positions are excluded from the positions of the saturated pixels. Then, the next filling is performed with the position of the remaining saturated pixel as the starting point, thereby reducing the processing load. As a result, in the example of FIG. 8(A), the first map can be generated with one filling.

[0057] As described above, according to this embodiment, it is possible to provide an image processing system capable of suppressing the adverse effects inherent to image estimation using a machine learning model. EXAMPLES

[0058] Next, an image processing system according to a second embodiment of the present invention will be described. The purpose of this embodiment is to correct the blur of a captured image by suppressing overcorrection even when a deviation from the learned blur occurs due to manufacturing variations, etc. However, this embodiment can be similarly applied to other image estimations.

[0059] FIG. 9 is a block diagram of the image processing system 300. FIG. 10 is an external view of the image processing system 300. The image processing system 300 includes a learning device 301, an imaging device 302, and an image processing device 303, and each device is connected to each other via a network or the like. The imaging device 302 includes an optical system 321, an imaging element 322, a storage unit 323, a communication unit 324, and a display unit 325. The captured image acquired by the optical system 321 and the imaging element 322 is blurred due to aberrations generated in the optical system 321. The captured image is transmitted to the image processing device 303 via the communication unit 324. The image processing device 303 includes a storage unit 331, a communication unit 332, an acquisition unit 333, a blur correction unit 334, and a calculation unit 335. The image processing device 303 receives the captured image via the communication unit 332, performs blur correction and suppression of overcorrection using a machine learning model, and generates an estimated image. The blur correction uses weights learned by the learning device 301. Weight information is acquired in advance from the learning device 301 and stored in the storage unit 331. Note that although this embodiment uses CNN as the machine learning model, it is also applicable to other models.

[0060] Next, the machine learning model in this embodiment will be described with reference to FIG. 11. FIG. 11 is a configuration diagram of the machine learning model in this embodiment. In this embodiment, a first machine learning model and a second machine learning model shown in FIG. 11(A) are used. From input data 401, the second machine learning model generates a second model output 402, and the first machine learning model generates a first model output 403. In the configuration of FIG. 11(A), a part of the first machine learning model is the same as the second machine learning model. However, as shown in FIG. 11(B), a configuration in which each of the first machine learning model and the second machine learning model is the same may be used. In FIG. 11(B), the second machine learning model has only one layer after branching, but this is not limited to this. Details regarding the learning of the first machine learning model and the second machine learning model will be described later.

[0061] The second machine learning model performs blur correction based on the design value of the optical system 321, and generates a second model output 402. Therefore, when the blur of the optical system 321 changes due to the influence of manufacturing variations, etc., the accuracy of the blur correction decreases. When the blur is larger than the design value, the second model output 402 is undercorrected. Conversely, when the blur is smaller than the design value, the second model output 402 is overcorrected. Undercorrection results in a sense of resolution between the captured image and an image of an ideal subject without blur (the blur is smaller than the captured image), whereas overcorrection results in an image with a structure that does not actually exist, such as undershoot. Therefore, in this embodiment, overcorrection due to manufacturing variations, etc. is suppressed. The first machine learning model performs blur correction of the captured image and suppression of overcorrection, and generates a first model output 403. Suppression of overcorrection is performed in a subsequent layer that is not common to the second machine learning model. However, it is not possible to accurately determine where in the image will be overcorrected. For example, in the case of edges, excessive correction can cause problems such as undershooting and overshooting, so it is easy to determine whether the correction is excessive. However, excessive correction of textures is unlikely to cause problems such as undershooting, so it is difficult to distinguish whether the correction is excessive or the texture components are originally strong. For this reason, in the first model output 403, parts of the over-corrected texture that are suppressed and parts that are not suppressed are mixed together in a mottled manner, and the resolution of the texture becomes discontinuous.

[0062] In summary, the second model output 402 has the blur of the captured image corrected, but may be overcorrected due to the influence of manufacturing variations and the like. The first model output 403 has the blur of the captured image corrected and the overcorrection of edges suppressed, but may be discontinuous in the resolution of the texture. By combining the first and second model outputs, an estimated image is generated in which the blur of the captured image is corrected and the overcorrection of edges and the discontinuity of the resolution of the texture are suppressed. The details will be described later. Note that when overcorrection occurs, the second model output 402 has a spatial frequency intensity higher than that of the first model output 403. On the other hand, when overcorrection does not occur, the second model output 402 has a spatial frequency intensity equivalent to that of the first model output 403. Therefore, the second model output 402 has a spatial frequency intensity higher than or equal to that of the first model output 403 (i.e., equal to or higher than that of the first model output). The estimated image is transmitted to the imaging device 302 , stored in the storage unit 323 , and displayed on the display unit 324 .

[0063] The learning of the weights of the first machine learning model and the second machine learning model executed by the learning device 301 is similar to the flowchart of Fig. 6. However, in this embodiment, the first machine learning model is learned after the learning of the second machine learning model is completed. Each step of Fig. 6 is executed by any one of the memory unit 311, the acquisition unit 312, the calculation unit 313, or the update unit 314 of the learning device 301.

[0064] In step S101, the acquisition unit 312 acquires one or more pairs of input data and a correct image from the storage unit 311. A training image (blurred image) of the second machine learning model is generated by adding a blur obtained from the design value of the optical system 321 to the original image. Optical parameters such as the focal length, F-number, focusing distance, image height, and azimuth of the optical system 321 are determined, and a training image is generated with a blur corresponding to the optical parameters. The input data also includes information on the optical parameters. A similar procedure is repeated for multiple optical parameters to generate multiple input data corresponding to various blurs generated by the optical system 321. Since the input data includes information on the optical parameters, the machine learning model can distinguish the blur of the training image by the optical parameters, and highly accurate blur correction is possible. The correct image uses the original image as it is (or adds a blur smaller than the blur of the optical system 321).

[0065] In addition to the method of generating the training image of the second machine learning model, the training image of the first machine learning model also generates an image to which a blur different from the design value of the optical system 321 is added. This allows learning when a deviation from the design value of the optical system 321 occurs. Deviations occur due to manufacturing variations, defocusing when axial chromatic aberration or curvature of field occurs, and the like. For the blur different from the design value, it is preferable to use, for example, a blur obtained by enlarging or reducing the blur obtained from the design value of the optical system 321. In addition, deformation or rotation may be used. In addition, there are a method of using a blur of different optical parameters, a method of sharpening (or blurring) the blur, and the like. The correct image for the training image to which a blur approximately matching the design value is added is the same as that of the second machine learning model. Alternatively, the second model output 402 for the training image may be the correct image. For the correct image for the training image to which a blur larger than the design value is added, the second model output 402 for the training image is the correct image. That is, when the captured image is blurred more than the design value, the image is trained to remain undercorrected. The correct image for the training image to which a blur smaller than the design value is added is the same as the second machine learning model. Alternatively, the second model output 402 for an image to which a blur approximately equal to the design value is added to the same original image may be set as the correct image. A model that suppresses only overcorrection can be trained by using this pair of training image and correct image.

[0066] Next, in step S102, the calculation unit 313 generates a first model output 403 or a second model output 402 from the input data 401 using the first or second machine learning model.

[0067] Next, in step S103, the update unit 314 updates the weights of the first machine learning model or the second machine learning model based on the error between the first model output 403 or the second model output 402 and the correct image. When learning the first machine learning model, the weights of the same parts as those of the already learned second machine learning model are fixed and not updated. That is, the weights of the second machine learning model are used as they are for the first machine learning model.

[0068] Next, in step S104, the update unit 314 determines whether the learning is completed. Information on the learned weights is stored in the storage unit 311.

[0069] Next, blur correction of a captured image and suppression of excessive correction executed by the image processing device 303 will be described with reference to Fig. 12. Fig. 12 is a flowchart of generation of an estimated image in this embodiment. First, in step S301, the acquisition unit 333 acquires input data and information on the weights of the first machine learning model and the second machine learning model. The input data includes a captured image and information on the optical parameters of the optical system 321 when the captured image is captured. In the case of the configuration of Fig. 11(A), the second machine learning model is entirely included in the first machine learning model, so it is only necessary to acquire information on the weights of the first machine learning model.

[0070] Next, in step S302, the blur correction unit (acquisition means) 334 generates (acquires) a first model output 403 and a second model output 402 from the input data using the first machine learning model and the second machine learning model, respectively. In the case of the configuration in Fig. 11(A), since the second machine learning models are all included in the first machine learning model, only the first machine learning model is executed to generate the second model output 402, which is an intermediate output, and the first model output 403.

[0071] Next, in step S303, the calculation unit 335 generates a first map based on a comparison between the captured image and the second model output 402. In this embodiment, the first map is generated by taking the difference between the captured image and the second model output 402, and indicating an area exceeding a predetermined threshold. The first map indicates an area in the second model output 402 where the correction residual is large (having a second value), and includes an edge area where overcorrection has occurred. However, it also includes an area where the correction residual is large but is properly corrected.

[0072] Next, in step S304, the calculation unit (generation means) 335 generates an estimated image based on the first model output 403, the second model output 402, and the first map. In the second model output 402, an estimated image is generated by replacing the area (where the correction residual is large) shown in the first map with the first model output 403. By replacing only the area where the correction residual is large with the first model output 403, textures that cause discontinuous resolution are not used in the estimated image. The first model output 403 suppresses only the overcorrected edges to a proper resolution, and other edges have a resolution equivalent to that of the second model output 402, so that only overcorrection can be suppressed without reducing the edge correction effect.

[0073] As described above, according to this embodiment, it is possible to provide an image processing system capable of suppressing the adverse effects inherent to image estimation using a machine learning model. EXAMPLES

[0074] Next, an image processing system according to a third embodiment of the present invention will be described. In this embodiment, the purpose of image estimation using a machine learning model is to convert blur (defocus blur) including saturated pixels. The conversion of blur refers to a process of converting the distribution of defocus blur in a non-focused area of ​​a captured image to a different distribution. For example, double line blur caused by the separation of the peaks of the PSF is converted to a circular blur with a flat distribution or a Gaussian distribution function. This makes it possible to make a focused subject stand out more and change the impression of the captured image. However, the invention can be similarly implemented and advantageously obtained for image estimation other than the conversion of blur.

[0075] Fig. 13 is a block diagram of an image processing system 500 in this embodiment. Fig. 14 is an external view of the image processing system 500. The image processing system 500 has a learning device 501, an imaging device 502, a blur conversion device 503, and an image processing device 504, and each device is connected to each other via a wired or wireless network.

[0076] The imaging device 502 has an optical system 521 and an imaging element 522. The captured image has defocus blur such as double line blur generated by the optical system 521. The image processing device 504 has a storage unit 541, a communication unit 542, a calculation unit 543, and a display unit 544, and acquires a captured image from the imaging device 502 according to a user's input and transmits it to the blur conversion device 503. The blur conversion device 503 has a storage unit 531, a communication unit 532, an acquisition unit 533, and a blur conversion unit 534, and converts the blur of the captured image using a first machine learning model and a second machine learning model. The blur conversion device 503 acquires information on the weights of the first and second machine learning models from the learning device 501 and stores it in the storage unit 531. The learning device 501 has a storage unit 511, an acquisition unit 512, a calculation unit 513, and an update unit 514, and learns weights for performing blur conversion using a first machine learning model and a second machine learning model.

[0077] The image processing device 504 acquires the first model output and the second model output that have been subjected to blur conversion by the first machine learning model and the second machine learning model, respectively, and generates an estimated image in which adverse effects specific to the machine learning models are suppressed. The estimated image is displayed on the display unit 544, and the user can perform editing operations such as exposure correction while checking the displayed image. Note that this embodiment uses CNN as the machine learning, but it is also applicable to other models.

[0078] The drawback inherent to the machine learning model that occurs in this embodiment is artifacts due to local signal increases and decreases that do not exist in an actual subject, similar to the first embodiment. The cause of the artifacts is similar to the first embodiment. Since there is a lack of information due to saturation, the machine learning model needs to change the process of bokeh conversion between a defocus blur image (saturated bokeh) that includes saturated pixels and a defocus blur image (non-saturated bokeh) that includes only non-saturated pixels. However, since it is not possible to completely determine saturated bokeh and non-saturated bokeh, artifacts may occur.

[0079] The learning of the weights of the first machine learning model and the second machine learning model executed by the learning device 501 is the same as the flowchart in FIG. 6. This embodiment has a configuration similar to that shown in FIG. 11(A). However, in this embodiment, the relationship between the first machine learning model and the second machine learning model is reversed compared to FIG. 11(A). Also, as in the second embodiment, the configuration in FIG. 11(B) (wherein the relationship between the first machine learning model and the second machine learning model is swapped) may be used. In this embodiment, after completing the learning of the first machine learning model, the second machine learning model is learned.

[0080] First, in step S101, the acquisition unit 512 acquires one or more pairs of input data and a correct answer image. The input data includes a training image. The method of generating the training image is the same as that of the first embodiment. However, the blur to be added is the defocus of the optical system 521. The defocus amount is determined, and the corresponding defocus blur is added. Training images in a focused state (defocus amount 0) and various defocus amounts are generated and used for learning. The correct answer image is generated by adding the defocus blur to be realized by the conversion to the original image. In the conversion of the blur, the frequency components lost due to the blur are not restored. Therefore, the spread of the defocus blur added to the correct answer image is equal to or larger than the defocus blur of the training image.

[0081] As in the first embodiment, the first machine learning model converts only non-saturated bokeh. The second machine learning model converts all bokeh (saturated bokeh and non-saturated bokeh), but artifacts may occur. In addition to the training image, a parallax image from a viewpoint different from that of the training image may be added to the input data. The parallax makes it easier for the machine learning model to distinguish between an unfocused object that is the target of the conversion and a focused object that is not the target, improving the accuracy of the bokeh conversion. In this case, the parallax image is also input during estimation. Instead of the parallax image, a depth map or a defocus map may be included in the input data.

[0082] Next, in step S102, the calculation unit 513 generates a model output from the input data using the machine learning model. Next, in step S103, the update unit 514 updates the weight of the machine learning model based on the error between the model output and the correct image. As in the second embodiment, when learning the second machine learning model, the weight of the part common to the first machine learning model is fixed. Next, in step S104, the update unit 514 determines the completion of learning. Information on the learned weight is stored in the storage unit 511.

[0083] Although the first and second machine learning models are trained separately here, they can also be trained together. In this case, two correct images corresponding to the first and second machine learning models are prepared for each training image. The loss function is the weighted sum of the error between the first model output and the first correct image (which differs from the training image only in non-saturated blur) and the error between the second model output and the second correct image (which differs from the training image in total blur). This allows the first and second machine learning models to be trained together in a linked state.

[0084] Next, generation of an estimated image in which the blur of a captured image is converted, which is executed by the image processing device 504 and the blur conversion device 503, will be described with reference to Fig. 15. Fig. 15 is a flowchart of generation of an estimated image in this embodiment.

[0085] First, in step S401, the communication unit (transmission means) 542 of the image processing device 504 transmits input data including a captured image and a request for execution of the blur conversion process to the blur conversion device 503. The captured image is acquired in advance from the imaging device 502 and stored in the storage unit 541. Furthermore, if the input data during learning includes information other than the training image, similar information is transmitted together with the captured image.

[0086] Next, in step S402, the communication unit (receiving means) 532 of the blur conversion device 503 acquires (receives) input data including a captured image transmitted from the image processing device 504 and a request for execution of processing. Next, in step S403, the acquisition unit 533 acquires weight information from the storage unit 531. Next, in step S404, the blur conversion unit (first generating means) 534 generates a first model output and a second model output from the input data using each of the first machine learning model and the second machine learning model. The second model output is an image in which all blur has been converted, but artifacts may occur. The first model output is an image in which only non-saturated blur has been converted, and no artifacts exist. Next, in step S405, the communication unit 532 transmits the first model output and the second model output to the image processing device 504.

[0087] Next, in step S406, the communication unit (acquisition means) 542 of the image processing device 504 acquires the first model output and the second model output. Next, in step S407, the calculation unit 543 generates a second map based on a comparison between the first model output and the second model output. In this embodiment, similar to the first embodiment, the difference is binarized using a predetermined threshold value to generate the second map.

[0088] Next, in step S408, the calculation unit 543 generates a first map based on the second map and the positions of the saturated pixels in the captured image. In this embodiment, as in the first embodiment, the saturated blur includes saturated pixels, so the first map is generated by extracting an artifact region that does not include saturated pixels from the second map. In the first map, the artifacts are shown as regions having a second value.

[0089] Next, in step S409, the calculation unit (second generation means) 543 generates an estimated image based on the first model output, the second model output, and the first map. By replacing the artifact area shown in the first map in the second model output with the first model output, it is possible to generate an artifact-free blur-transformed estimated image.

[0090] As described above, according to this embodiment, it is possible to provide an image processing system capable of suppressing the adverse effects inherent to image estimation using a machine learning model.

[0091] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0092] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program capable of suppressing the inherent drawbacks that occur in image estimation using a machine learning model.

[0093] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0094] 123 Image processing unit (image processing device) 123b blur correction unit (acquisition means) 123c Arithmetic unit (generation means)

Claims

1. Obtaining a first model output generated based on the captured image by a first machine learning model; acquiring a second model output generated based on the captured image by a second machine learning model different from the first machine learning model; generating an estimated image using the first model output and the second model output based on a comparison between the second model output and the captured image or the first model output.

2. 2. The method of claim 1, wherein the comparison comprises one of a difference, a ratio, or a correlation.

3. 3. The image processing method according to claim 1, wherein the estimated image is generated based on a first map generated based on the comparison.

4. 4. The image processing method according to claim 3, further comprising the step of determining an area in which the first model output or the second model output is used based on the first map in generating the estimated image.

5. the captured image has a plurality of color components, 5. The image processing method according to claim 3, wherein the first map is common to the plurality of color components.

6. the first machine learning model and the second machine learning model are models for estimating a correction component of a blur of the captured image, The image processing method according to claim 1 , wherein the first machine learning model has a smaller effect of correcting blur on a high-brightness object than the second machine learning model.

7. The estimated image is generated based on a first map generated based on the comparison; 7. The image processing method according to claim 1, wherein the first map is generated based on positions of saturated pixels based on the captured image.

8. 8. The image processing method according to claim 7, wherein the first map is a map indicating an area where the comparison satisfies a predetermined condition and does not include pixels based on the positions of the saturated pixels.

9. generating a second map having second values ​​in areas where the comparison satisfies a predetermined condition; The image processing method according to claim 7 or 8, characterized in that the first map is generated by replacing, among the closed spaces having the second value in the second map, closed spaces including pixels based on the positions of the saturated pixels with a first value.

10. the first machine learning model and the second machine learning model are models for estimating a correction component of a blur of the captured image, and have at least a part of the same processing as each other; 6. The image processing method according to claim 1, wherein the second model output has a spatial frequency strength equal to or greater than the first model output.

11. an acquisition means for acquiring a first model output generated based on a captured image by a first machine learning model, and acquiring a second model output generated based on the captured image by a second machine learning model different from the first machine learning model; a generation means for generating an estimated image using the first model output and the second model output based on a comparison between the second model output and the captured image or the first model output.

12. 1. An image processing system having a first device and a second device, the first device has a transmission means for transmitting a captured image and a request for execution of processing to the second device; The second device comprises: a receiving means for receiving the captured image and the request; a first generating means for generating a first model output based on the captured image using a first machine learning model in response to the request, and generating a second model output based on the captured image using a second machine learning model different from the first machine learning model, The image processing system according to claim 1, wherein the first device has a second generation means for generating an estimated image using the first model output and the second model output based on a comparison between the second model output and the captured image or the first model output.

13. A program causing a computer to execute the image processing method according to any one of claims 1 to 10.

Images