Image processing method, image processing apparatus, and program

Abstract

To reduce the adverse effect of blur correction while maintaining the effect of the blur correction.SOLUTION: An image processing method includes: generating a second image by sharpening a first image; generating a first weight map based on the first image; generating the first weight map based on the first image; and generating a third image based on the first and second images and the first and second weight maps, the first and second weight maps representing weight of the first or second image in generating the third image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing method using a machine learning model. [Background technology]

[0002] There is known an image processing method that performs deconvolution using a filter (e.g., a Wiener filter or an inverse filter) to correct (blur) degradation of a captured image caused by aberration in an optical system. However, when performing blur correction processing using a filter, there is a risk of adverse effects such as ringing and undershoot occurring in a portion of the captured image where pixel values ​​(pixel values) are saturated (brightness saturated portion) and its surrounding portions.

[0003] Patent Document 1 discloses a method for generating an image with fewer drawbacks in brightness saturated areas and their surrounding areas by reducing the amount of blur correction based on the results of determining whether an area in the image is a brightness saturated area or its surrounding areas. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2013-84247 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while the method of Patent Document 1 can generate an image with fewer adverse effects in brightness saturated areas and their surrounding areas, it has a problem in that the effect of blur correction is reduced because the amount of blur correction is reduced.

[0006] SUMMARY OF THE PRESENT DISCLOSURE In view of the above, an object of the present invention is to reduce the adverse effects of blur correction while maintaining the effect of blur correction in image processing. [Means for solving the problem]

[0007] The image processing method of the present invention includes generating a second image by sharpening a first image, generating a first weight map based on the first image, generating a third image based on the first and second images and the first and second weight maps, and the first and second weight maps indicate weights of the first or second image in generating the third image. Effect of the Invention

[0008] According to the present invention, it is possible to reduce the adverse effects of blur correction while maintaining the effect of the blur correction. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a flowchart showing an estimation process in the first embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] FIG. 4 is a flowchart of a learning process in the first embodiment. [Diagram 5] FIG. 1 is a diagram showing a learning flow of a machine learning model in a first embodiment. [Figure 6] FIG. 13 is a diagram showing a conventional correction process. [Figure 7] FIG. 4 is a diagram illustrating a correction process in the first embodiment. [Figure 8] FIG. 1 is a diagram relating to gamma correction. [Figure 9] FIG. 11 is a diagram showing demosaicing processing for G. [Figure 10] FIG. 13 is a schematic diagram of an interpolation filter in the demosaicing process. [Figure 11] FIG. 13 is a diagram showing the demosaicing process for R. [Figure 12] FIG. 4 is a diagram showing a response function in the first embodiment. [Figure 13] FIG. 2 is a schematic diagram of a weight map. [Figure 14] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 15] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 16] FIG. 11 is a flowchart of an estimation process in the second embodiment. [Figure 17] FIG. 11 is a block diagram of an image processing system according to a third embodiment. [Figure 18] FIG. 11 is a flowchart showing an estimation process in Example 3. [Figure 19] FIG. 11 is a diagram showing a response function in Example 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same reference numerals are used to refer to the same components, and duplicated explanations will be omitted.

[0011] First, before describing each example in detail, the gist of the embodiment of the present invention will be described. In the image processing method of the present embodiment, first, blur caused by the optical system in a captured image acquired by imaging using an optical system and an image sensor is sharpened using a machine learning model to generate an estimated image. Then, two weight maps are generated based on pixel values ​​of the captured image and the estimated image, and a corrected image is generated based on the two weight maps, the captured image, and the estimated image. Here, the blur caused by the optical system includes at least one of blur caused by aberration, diffraction, and defocus, blur caused by an optical low-pass filter, and a decrease in the aperture ratio of pixels in the image sensor. As the machine learning model, for example, a neural network, genetic programming, a Bayesian network, etc. can be adopted. In addition, the neural network is, for example, a CNN (Convolutional Neural Network), a GAN (Generative Adversarial Network), an RNN (Recurrent Neural Network), etc.

[0012] By applying the processing of this embodiment, it is possible to reduce the adverse effects of blur correction (sharpening) while maintaining the effect of blur correction (sharpening).

[0013] In addition, each image processing described in the following embodiments can be applied to image processing such as high contrast, brightness improvement, defocus blur conversion, and lighting conversion in addition to sharpening processing, by appropriately changing the learning data used to train the machine learning model.

[0014] [Example 1] First, an image processing system 100 in the first embodiment of the present invention will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a block diagram of the image processing system 100 in this embodiment. Fig. 3 is an external view of the image processing system 100.

[0015] The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107.

[0016] The learning device 101 is a device that executes a learning process (learning phase) and includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and an update unit 101d. The acquisition unit 101b acquires training images and correct images. The generation unit 101c inputs the training images to a multi-layered neural network to generate an output image. The update unit 101d updates (learns) the weights of the machine learning model based on the error between the output image and the correct image calculated by the generation unit 101c. Details of the learning phase will be described later using a flowchart. The learned weights of the machine learning model are stored in the storage unit 101a.

[0017] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives (photoelectrically converts) an optical image (subject image) formed via the optical system 102a to obtain a captured image. The imaging element 102b is, for example, a charge coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. The captured image obtained by the imaging device 102 contains blur due to aberration and diffraction of the optical system 102a and noise due to the imaging element 102b.

[0018] The image estimation device 103 is a device that executes an estimation step (estimation phase), and includes a storage unit 103a, an acquisition unit 103b, a generation unit 103c, and a correction unit 103d. The image estimation device 103 generates an estimated image by performing blur correction on a captured image acquired using a machine learning model. The weights of the machine learning model are read from the storage unit 103a. The weights of the machine learning model are learned by the learning device 101, and the image estimation device 103 reads the weights from the storage unit 101a via the network 107 in advance and stores them in the storage unit 103a. The weights to be stored may be the numerical values ​​themselves or may be in an encoded format. Details regarding the update of the weights and the blur correction process using the machine learning model will be described later. The image estimation device 103 has a function of performing development processing and other image processing as necessary. In this case, an image obtained by performing other image processing on the estimated image may be used as the output image. The generation of the estimated image by the image estimation device 103 can be implemented by one or more processors (processing means) such as CPUs.

[0019] The estimated image (output image) is output to at least one of a display device 104, a recording medium 105, and an output device 106. The display device 104 is, for example, a liquid crystal display or a projector. A user can perform editing work while checking an image being processed through the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, or the like. The output device 106 is, for example, a printer.

[0020] Next, a weight (weight information) learning method (method of manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart related to weight learning. Each step in Fig. 4 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the update unit 101d of the learning device 101. Fig. 5 is a diagram showing the flow of learning of a neural network in this embodiment.

[0021] In step S101, the acquisition unit 101b acquires an original image. In this embodiment, the original image is a high-resolution (high-quality) image with little blurring due to aberration and diffraction of the optical system 102a. A plurality of original images are acquired, and are images having various subjects, that is, edges, textures, gradations, flat parts, etc., of various strengths and directions. The original image may be a real-life image or an image generated by CG (Computer Graphics). In particular, when a real-life image is used as the original image, since blurring has already occurred due to aberration and diffraction, the effect of blurring can be reduced by reducing the image, and a high-resolution (high-quality) image can be obtained. Note that if the original image contains sufficient high-frequency components, reduction may not be required. In addition, the original image may contain noise components. In this case, the noise in the original image is considered to be the subject, so the noise in the original image is not particularly problematic. In addition, it is preferable that the original image has a signal value (luminance value, pixel value) higher than the luminance saturation value of the image sensor 102b. This is because, even in real life, there are subjects whose brightness does not fall within the saturation brightness value when imaged by the image capture device 102 under specific exposure conditions.

[0022] In step S102, the acquisition unit 101b acquires blur used to perform an imaging simulation, which will be described later. The acquisition unit 101b first acquires imaging conditions corresponding to the lens state (zoom, aperture, and focal distance state) of the optical system 102a. Then, information about blur determined by the imaging conditions and screen position is acquired. The information about blur is the PSF (point spread function) or OTF (optical transfer function) of the optical system 102a, and can be acquired by optical simulation or measurement in the optical system 102a. Note that the lens state, image height, and blur due to azimuth aberration and diffraction, which differ for each original image, are acquired. This makes it possible to perform imaging simulation corresponding to a plurality of imaging conditions, image heights, and azimuth. In addition, a component such as an optical low-pass filter included in the imaging device 102 may be added to the blur to be applied, if necessary.

[0023] In step S103, the generating unit 101c generates learning data that combines correct answer data consisting of correct answer patches (correct answer images) and training data consisting of training patches (training images). The correct answer patches and training patches are changed depending on the function or effect to be learned, and corresponding images can be used as the correct answer patches and training patches. One or more correct answer patches are generated corresponding to one original image. Note that a combination of multiple correct answer patches and training patches is used as learning data. In this embodiment, the original image is an undeveloped RAW image, and the correct answer patches and training patches are also RAW images. However, the present invention is not limited to this as long as the correct answer patches and training patches are images that show the same subject. For example, they may be images after development, or feature maps obtained by converting images as described later. In this embodiment, multiple original images stored in the storage unit 101a are used as subjects, and imaging simulation is performed to generate multiple pairs of correct answer images and training images that are relatively different in the influence of blur due to aberration and diffraction.

[0024] Note that a patch refers to an image having a predetermined number of pixels (for example, 64×64 pixels). The number of pixels of the correct patch and the training patch does not necessarily have to match. In this embodiment, mini-batch learning is used to train the machine learning model. For this reason, in step S103, multiple pairs of correct patches and training patches are generated. However, the present invention is not limited to this, and online learning or batch learning may be used.

[0025] In step S104, the generation unit 101c inputs the training patch 11 to the machine learning model to generate an estimated patch (estimated image) 213. Since the learning in this embodiment is mini-batch learning, estimated patches 12 corresponding to a plurality of training patches 11 are generated.

[0026] FIG. 5 shows the flow of the learning phase, which corresponds to steps S104 to S105 in FIG. 4. The estimated patch 12 has a higher sharpness than the training patch 11, and ideally coincides with the correct patch (correct image) 10. In this embodiment, the neural network configuration shown in FIG. 5 is used, but is not limited to this. CN in FIG. 5 represents a convolution layer, and DC represents a deconvolution layer. Whether a convolution layer or a deconvolution layer is used, the sum of the input, the convolution of the filter, and the bias is calculated, and the result is nonlinearly transformed by the activation function. The initial values ​​of each component of the filter and the bias are arbitrary, and are determined by random numbers in this embodiment. The activation function can be, for example, ReLU (Rectified Linear Unit) or a sigmoid function. The output of each layer except the final layer is called a feature map. The skip connections 22 and 23 combine feature maps output from discontinuous layers. The feature maps may be combined by taking the sum of each element, or by concatenation in the channel direction. In this embodiment, the sum of each element is adopted. The skip connection 21 takes the sum of the residual estimated from the training patch 11 and the ground truth patch 10 and the training patch 11 to generate an estimated patch 12. For each of the multiple training patches 11, multiple estimated patches 12 are generated.

[0027] In step S105, the update unit 101d updates the weights of the neural network based on the error between the estimated patch 12 and the ground truth patch 10. Here, the weights include the filter components and biases of each layer. Backpropagation is used to update the weights, but the present invention is not limited to this. In mini-batch learning, the errors between a plurality of ground truth patches 10 and their corresponding estimated patches 12 are calculated, and the weights are updated. For example, the L2 norm or the L1 norm can be used as the loss function.

[0028] In step S106, the update unit 101d determines whether the weight update is complete. Completion of the weight update can be determined by whether the number of iterations of learning (weight update) reaches a specified value, or whether the amount of change in the weight at the time of update is smaller than a specified value. If it is determined that the weight update is not complete, the process returns to step S103, and multiple new answer patches and training patches are obtained. On the other hand, if it is determined that the weight update is complete, the learning device 101 (update unit 101d) ends the learning, and stores the weight information in the storage unit 101a.

[0029] Here, a conventional correction process will be described with reference to FIG. 6. FIG. 6 is a diagram showing a conventional correction process. In FIG. 6, the vertical axis is the signal value, the horizontal axis is the spatial coordinate, and the dotted line represents the saturation luminance value. In this embodiment, the signal value is a pixel value. FIG. 6(a) represents the luminance of objects A and B. FIG. 6(b) is an example of an image (captured image) obtained by capturing objects A and B, and the image is degraded by the aberration of the optical system. Furthermore, since the luminance of object B is equal to or greater than the luminance saturation value, the signal is clipped at the luminance saturation value. FIG. 6(c) is an example of an image in which blur correction is performed on the captured image shown in FIG. 6(b) by deconvolution processing. Also, FIG. 6(d) is an example of an image in which blur correction is performed on the captured image shown in FIG. 6(b) by a machine learning model. In this embodiment, blur correction is performed using a machine learning model learned by the method of FIG. 4.

[0030] In FIG. 6(c), the deconvolution process is performed using a filter such as a Wiener filter or an inverse filter. Instead of the deconvolution process, a process of performing Fourier transform on the image and the filter in the frequency space and multiplying them may be used. In this case, the degradation caused by aberration can be reduced in the case of degradation such as that of the captured image of FIG. 6(b) in the case of the subject A. On the other hand, in the case of the subject B in the case of an area including a luminance saturation area, there is a high possibility that a serious problem such as overshoot or undershoot will occur. The overshoot occurring near the edge of the luminance saturation area in FIG. 6(c) is actually clipped at the luminance saturation value and does not exist in the image, but the undershoot occurring in the dark area does exist in the image. The black level (brightness of the dark area) of the captured image is offset from zero, and when an undershoot occurs, it is clipped so that it does not fall below zero, but this offset value (OB value) is below the black level. In addition, the area below the black level becomes darker than the surroundings, which causes a problem in the image. This is because the pixel value of the subject does not fall within the dynamic range of the image sensor and is clipped, causing information loss, which causes the above problem.

[0031] However, in the case of convolution processing as shown in Figure 6(c), it is difficult to avoid these problems in principle because it is a linear operation. Therefore, a machine learning model can be used to reduce the above-mentioned problems. For example, convolution operations are also used in neural networks, but a nonlinear function can be used for the activation function part. Nonlinear processing by a machine learning model that has been properly trained allows for more flexible processing. In this case, the machine learning model needs to be trained using images that include brightness saturation parts as training data. By using captured images that include brightness saturation parts in the training data and images with little aberration and no problems around the brightness saturation parts as correct images, it becomes possible to perform correction processing that reduces the problems caused by correction while maintaining the effect of aberration correction by the optical system for both subjects A and B as shown in Figure 6(d).

[0032] By learning using captured images that contain brightness saturation areas, the machine learning model determines whether or not there are brightness saturation areas, and changes the processing depending on the result. However, if there is an error in this determination, it may have a negative impact on the result.

[0033] Next, the correction process in this embodiment will be described with reference to FIG. 7. FIG. 7 is a diagram showing the correction process in this embodiment. In FIG. 7, the vertical axis is the signal value, the horizontal axis is the spatial coordinate, and the dotted line represents the saturation luminance value. In this embodiment, the signal value is a pixel value. FIG. 7(a) shows the luminance of objects A, B, and C. In FIG. 7, an example will be described in which three objects are captured: object A with low luminance, object B with a value about half the luminance saturation value, and object C with a value near the luminance saturation value. Note that object C in FIG. 7(a) shows an example in which erroneous determination is likely to occur, and an object that is flat near the luminance saturation value has a value equal to or greater than the luminance saturation value and is difficult to determine as being clipped at the luminance saturation value. FIG. 7(b) is an example of images (captured images) of objects A, B, and C, and the image is degraded by the aberration of the optical system. FIG. 7(c) is an example of an image in which blur correction is performed on the captured image shown in FIG. 6(b) using the above-mentioned machine learning model. FIG. 7(d) is an example of an image obtained by processing the captured image shown in FIG. 6(b) using the method of the present invention.

[0034] As Fig. 6 and Fig. 7 are schematic diagrams, evaluations were performed without noise, but actual captured images are also affected by noise. If an area that was not saturated in brightness is erroneously determined to be saturated in brightness due to the influence of noise and corrected, that area will be significantly corrected. Fig. 7(c) shows the result of the machine learning model determining subject C as saturated in brightness, resulting in overcorrection of subject C. Fig. 7(c) is just one example, but the blur correction process reduces the signal (pixel value), resulting in a lower pixel value than normal, causing problems.

[0035] Moreover, such problems occur not only on the high-luminance side but also on the low-luminance side. The subject A in FIG. 7(c) shows an example. Problems occurring on the low-luminance side are related to gamma correction. Gamma correction will be described with reference to FIG. 8. FIG. 8 shows the relationship between pixel values ​​before and after gamma correction. In FIG. 8, when the pixel values ​​before correction are represented by a straight line with a slope of 1 (the dashed line in FIG. 8), the pixel values ​​after correction are represented by a curve (gamma curve). Gamma correction is a process performed in a development process for acquiring a developed image from a RAW image, which is undeveloped image data output from an imaging element. The actual process is a process of raising the input pixel value to a power, and for example, 1 / 2.2 is used as the exponent of the power. In gamma correction, the pixel values ​​after correction have a value equal to or greater than the pixel values ​​before correction, and the pixel values ​​on the low-luminance side change more significantly than the high-luminance side. In other words, there is a risk that the problems on the low-luminance side in the state of a RAW image will become more noticeable (emphasized) by gamma correction in the state of a developed image.

[0036] In this embodiment, in order to reduce these problems, two gain maps corresponding to the high brightness side and the low brightness side are created, and a final corrected image is generated using the two gain maps. Fig. 7(d) shows a captured image, an estimated image, and a corrected image generated using the above-mentioned gain maps. Even if problems such as undershooting occur in the estimated image, the image processing method of this embodiment can be executed to reduce the problems caused by blur correction while maintaining the effect of blur correction.

[0037] Next, the generation of a corrected image (correction process) executed by the image estimation device 103 in this embodiment will be described with reference to Fig. 1. Fig. 1 is a flowchart related to the generation of a corrected image (third image). Each step in Fig. 1 is mainly executed by the acquisition unit 103b, the generation unit 103c, or the correction unit 103d.

[0038] First, in step S201, the acquisition unit 103b acquires a captured image (first image). The first image is an undeveloped RAW image, similar to the learning, and is transmitted from the imaging device 102 in this embodiment.

[0039] In step S202 (first step), the generating unit 103c executes a sharpening process on the captured image using the machine learning model to obtain an estimated image (second image). Note that the weight information of the machine learning model is transmitted from the learning device 101 and stored in the storage unit 103a.

[0040] In step S203 (second step), the generating unit 103c generates a first gain map (first weight map). In this embodiment, the first gain map is generated based on pixel values ​​of the captured image, and indicates weights applied to an estimated image when generating a corrected image. The first gain map has weight information for each pixel corresponding to the corrected image. The first gain map is used to reduce adverse effects in low brightness areas (dark areas).

[0041] Here, each process applied to the captured image will be described with reference to Figs. 9, 10, and 11. Each process applied to the captured image is a procedure performed on the captured image as necessary when generating the first gain map, such as demosaicing, white balance adjustment, and normalization. Fig. 9 is a diagram related to color components of the captured image. Fig. 10 is a schematic diagram of an interpolation filter in demosaicing. Fig. 11 is a diagram related to demosaicing for R. In this embodiment, the captured image is an image of a Bayer array as shown in Fig. 9(a). Here, RGB represent pixel values ​​(luminance values) of red, green, and blue, respectively. However, the color array of the captured image is not limited to this.

[0042] First, the demosaicing process performed on the captured image represented by the array shown in FIG. 9(a) will be described. Since there are two types of G, G1 and G2, in the Bayer array, the demosaicing process is different for G, R, and B. FIG. 9(b) shows a state in which only G1 and G2 are extracted from FIG. 9(a). When performing demosaicing on G in FIG. 9(a), linear interpolation is performed after extracting G as shown in FIG. 9(b). In this embodiment, linear interpolation is performed using the filter in FIG. 10(a). Note that FIG. 10(a) is a filter used for demosaicing on G, and the value in FIG. 10(a) is not limited to this. By convoluting the filter in FIG. 10(a) on G in FIG. 9(b), it is possible to generate information on G after interpolation as shown in FIG. 9(c).

[0043] Next, the demosaicing process of R and B will be described. Since the same process can be applied to both R and B with only one type, the process for R will be described in this embodiment. FIG. 11(a) shows a state in which only R is extracted from the Bayer array in FIG. 9(a). FIG. 11(b) shows the color difference of R before demosaicing process, and the difference of G after demosaicing process in FIG. 9(c) is taken only for pixels in which R exists in FIG. 11(a). In this embodiment, color difference information is calculated by taking the difference of G with respect to R. Then, in order to interpolate the color difference information, the filter in FIG. 10(b) is convolved with the color difference in FIG. 11(b) to calculate color difference information corresponding to all pixels. Note that (b) is a filter for performing the interpolation process, and the value in FIG. 10(b) is not limited to this. FIG. 11(c) shows information of R after demosaicing process, and can be calculated by adding G (corresponding to FIG. 9(c)) to the color difference after convolution with the filter in FIG. 10(b).

[0044] Although the demosaicing process for R has been described above, the demosaicing process can be similarly performed for B. In this embodiment, the interpolation process is performed using color differences for R and B, but the interpolation process may be performed directly for R and B. Furthermore, in addition to simple interpolation process, the demosaicing process may also be performed by incorporating adaptive processing using the gradient of RGB values ​​or the like.

[0045] Next, a white balance adjustment performed on the captured image will be described. The white balance adjustment is a process of adjusting the luminance level by performing gain processing for each color component on the RGB components output by the image sensor so that the achromatic parts of the subject become achromatic in the output image. By performing the white balance adjustment, the color of the subject is correctly reproduced regardless of the color characteristics of the image sensor, and an image with colors closer to the actual subject can be generated. Furthermore, in this embodiment, a more accurate gain map can be generated by performing the white balance adjustment.

[0046] The coefficients (white balance coefficients) used for white balance adjustment utilize the settings at the time of capturing the captured image. Usually, there are white balance coefficients for each of the RGB color components, so white balance adjustment can be performed by multiplying the coefficients by the corresponding color components of the captured image. If the black level is offset, the offset may be subtracted before white balance adjustment. If the white balance coefficients are separate for G1 and G2, the average of these may be used as the white balance coefficient for G. Note that the white balance coefficients may not be the settings at the time of capturing, but may be a combination of other coefficients. For example, if the imaging device used for capturing images has a function (auto white balance) that automatically determines and corrects the type of light source, the coefficient data is also attached to the captured image and may be used for white balance adjustment.

[0047] Next, the normalization process performed on the captured image will be described. In the normalization process of this embodiment, first, the maximum value of RGB is obtained for each pixel of the captured image having the luminance values ​​of the three colors of RGB. In this embodiment, the maximum value is used, but not limited to this, and an index such as a median value, an average value, or a minimum value may be used. In this way, by setting one value for one pixel, it is possible to prevent the weights from differing between RGB. Scaling is performed on two-dimensional data composed of the maximum values ​​of RGB so that the black level is 0 and the luminance saturation value is 1. Then, a gain map is calculated based on the two-dimensional data after scaling. Note that in this embodiment, scaling is performed based on the black level and the luminance saturation value, but this is not limited to this. For example, conversion may be performed so that 2 bytes become 1 by dividing by 2 bytes.

[0048] FIG. 12 shows the response function in this embodiment. In FIG. 12, the horizontal axis is the two-dimensional data after scaling, and the vertical axis is the adjusted value of the weight map. The response function in this embodiment can calculate the weight of an estimated image used when generating a corrected image by inputting a signal value (pixel value). FIG. 12(a) shows the response function (first response function) used to generate a first gain map, and FIG. 12(b) shows the response function (second response function) used to generate a second gain map.

[0049] In Fig. 12(a), as the pixel values ​​of the 2D data after scaling increase, the output value also increases (the weight of the estimated image increases). On the other hand, in Fig. 12(b), as the pixel values ​​of the 2D data after scaling increase, the output value decreases (the weight of the estimated image decreases).

[0050] In FIG. 12(a), the thresholds are set to 0.2 and 0.3, and when the threshold is 0.2 or less, the weight of the estimated image is 0, and when the threshold is 0.3 or more, the weight of the estimated image is 1. The weight changes linearly between them. In this embodiment, two thresholds are set so that each section changes linearly, but a nonlinear function such as a sigmoid function may be used as the response function without setting a threshold. The two thresholds are not limited to this, and other values ​​may be used. By setting a threshold, the area where the weight of the estimated image is large can be increased, thereby obtaining a corrected image with less aberration. On the other hand, by setting a threshold, the area where the weight of the estimated image is small can be increased, thereby obtaining a corrected image with less adverse effects due to aberration correction. Furthermore, the response function may be generated based on information on the correction effect. Information on the correction intensity indicates the intensity of the aberration correction when generating the corrected image, and is used to determine the threshold in the response function. Information on the correction intensity is set, for example, by a user.

[0051] Fig. 13(a) is a schematic diagram of a first gain map generated using the captured image of Fig. 7(b). In Fig. 13(a), the vertical axis represents the weight of the estimated image, and the horizontal axis represents the area (spatial coordinates) in the image.

[0052] In step S204 (third step), the generating unit 103c generates a second gain map (second weight map). In this embodiment, the second gain map is generated based on pixel values ​​of the estimated image, and indicates weights applied to the estimated image when generating the corrected image. The second gain map has weight information for each pixel corresponding to the corrected image. The second gain map is used to reduce adverse effects in high luminance areas (bright areas).

[0053] The calculation of the second gain map is different from the calculation of the first gain map in that an estimated image is used instead of a captured image, whereas the demosaicing process, white balance adjustment, and normalization process are the same as those in the calculation of the first gain map, and therefore will not be described.

[0054] In FIG. 12(b), the thresholds are set to 0.8 and 0.9, and when the threshold is 0.8 or less, the weight of the estimated image is 1, and when the threshold is 0.9 or more, the weight of the estimated image is 0. The weight changes linearly between them. In this embodiment, two thresholds are set so that each section changes linearly, but a nonlinear function such as a sigmoid function may be used as the response function without setting a threshold. The two thresholds are not limited to this, and other values ​​may be used. By setting a threshold, the area where the weight of the estimated image is large can be increased, thereby obtaining a corrected image with less aberration. On the other hand, by setting a threshold, the area where the weight of the estimated image is small can be increased, thereby obtaining a corrected image with less adverse effects due to aberration correction. Furthermore, the response function may be generated based on information on the correction effect. Information on the correction intensity indicates the intensity of the aberration correction when generating the corrected image, and is used to determine the threshold in the response function. Information on the correction intensity is set, for example, by a user.

[0055] Fig. 13(b) is a schematic diagram of a second gain map generated using the estimated image of Fig. 7(c). In Fig. 13(b), the vertical axis represents the weight of the estimated image, and the horizontal axis represents the area (spatial coordinates) in the image.

[0056] In step S205 (fourth step), the generating unit 103c combines the first gain map and the gain map of the second step to obtain a gain map (third weighting map) used for generating a corrected image.

[0057] 13(c) is a schematic diagram of a gain map generated using the first gain map and the second gain map. In this embodiment, the third gain map is generated by comparing values ​​in corresponding regions of the first gain map and the second gain map and adopting the smaller value. Note that the third gain map may be calculated by multiplying or averaging the two gain maps.

[0058] In step S206, the correction unit 103d generates (synthesizes) a corrected image based on the captured image, the estimated image, and the third gain map. In this embodiment, the corrected image is generated by taking a weighted average of the captured image and the estimated image based on the weight of the estimated image indicated by the third gain map.

[0059] With this configuration, a corrected image as shown in Fig. 7(d) can be generated. In Fig. 7(d), the undershoot that occurred in subjects C and A is reduced compared to the estimated image in Fig. 7(c). In addition, the sharpening effect can be maintained to be comparable to the estimated image in Fig. 7(c).

[0060] By performing the correction process described above and using a gain map to generate a corrected image based on a captured image and an estimated image, it is possible to reduce the adverse effects of blur correction while maintaining the effect of the blur correction.

[0061] In this embodiment, the first gain map and the second gain map are calculated based on pixel values ​​of the captured image and the estimated image, respectively. The low brightness region is highly sensitive because it is greatly affected by noise and has a large gain due to gamma correction. Therefore, it is desirable to base the low brightness region on the captured image before sharpening. On the other hand, in order to maintain the effect of aberration correction in the brightness saturation region and its surroundings, it is desirable to base the estimated image after sharpening.

[0062] In the first response function and the second response function in FIG. 12, the pixel value when the weight of the estimated image (or the captured image) is 0.5 is set as the first threshold and the second threshold, respectively. Specifically, in FIG. 12, the first threshold is 0.25, and the second threshold is 0.85. In this embodiment, the first gain map is a weight for reducing adverse effects on the low luminance side, and the second gain map is a weight for reducing adverse effects on the high luminance side. Therefore, when considering the weight applied to the estimated image as in FIG. 12, the first threshold is smaller than the second threshold. This configuration is preferable in that the influence of ringing originating from the estimated image on the generated weight map can be reduced. However, the first and second thresholds are not limited to these numerical values. In addition, this magnitude relationship is premised on the fact that the reference value when scaling is the same.

[0063] In this embodiment, the gain map is calculated based on the pixel values ​​of the captured image and the estimated image, but other information may be added to calculate the gain map as necessary. The other information is, for example, information about the performance of the optical system during imaging. The performance of the optical system is calculated using, for example, the PSF (point spread function) and OTF (optical transfer function) of the optical system. When the performance of the optical system is low, the amount of correction by blur correction using a machine learning model increases, and adverse effects such as undershoot and ringing are likely to occur. In addition, since the PSF changes depending on the position in the captured image, for example, in a high image height area where the PSF spreads widely and performance is likely to be low, the occurrence of adverse effects can be suppressed by suppressing the amount of correction for the area at the center of the screen.

[0064] Although the embodiment of the blur determination method of the present invention has been described above, various modifications and changes are possible within the scope of the gist of the invention.

[0065] [Example 2] Next, an image processing system 200 according to a second embodiment will be described with reference to Fig. 14 and Fig. 15. The image processing system 200 of this embodiment differs from the first embodiment in that an original image is acquired using an optical system 221 and an image sensor 222 in an imaging device 202, and an image estimation unit 223 processes the image. Furthermore, the generation of a corrected image (third image) in this embodiment differs from the first embodiment in that a third gain map is not calculated.

[0066] Fig. 14 is a block diagram of an image processing system 200 in this embodiment, and Fig. 15 is an external view of the image processing system 200.

[0067] The image processing system 200 includes a learning device 201 and an imaging device 202 , and the learning device 201 and the imaging device 202 are capable of communicating with each other via a network 203 .

[0068] The learning device 201 has a storage unit (storage means) 211, an acquisition unit (acquisition means) 212, a generation unit (generation means) 213, and an update unit (learning means) 214, and learns weights (weight information) for performing blur correction in a neural network. Note that the learning device 201 in this embodiment is similar to the learning device 101 in the first embodiment, and therefore a description thereof will be omitted.

[0069] The imaging device 202 includes an optical system 221, an imaging element 222, an image estimation unit 223, a storage unit 224, a recording medium 225, a display unit 226, and a system controller 227. The imaging device 202 in this embodiment captures an image of a subject space to obtain a captured image, generates an estimated image from the captured image using a machine learning model, and generates a corrected image by taking a weighted average of the captured image and the estimated image.

[0070] The image estimation unit 223 includes an acquisition unit 223a, a generation unit 223b, and a correction unit 223c, which correspond to the acquisition unit 103b, the generation unit 103c, and the correction unit 103d in the first embodiment, respectively. The weight information is learned in advance by the learning device 201 and stored in the storage unit 211. The imaging device 202 reads out the weight information from the storage unit 211 via the network 203 and stores it in the storage unit 224. The corrected image generated by the image estimation unit 223 is stored in the recording medium 225. Note that the captured image already stored in the recording medium 225 may be read out and corrected by the image estimation unit 223. When an instruction on displaying the corrected image is issued by the user, the stored corrected image is read out and displayed on the display unit 226. The above series of controls are performed by the system controller 227.

[0071] The learning method of the machine learning model executed by the learning device 201 of this embodiment is the same as that of the first embodiment, and therefore the description thereof will be omitted.

[0072] Next, generation of a corrected image executed by the image estimation unit 223 in this embodiment will be described with reference to Fig. 16. Fig. 16 is a flowchart related to generation of a corrected image. Each step in Fig. 16 is mainly executed by the acquisition unit 223a, the generation unit 223b, or the correction unit 223c. Steps S301 to S303 are similar to steps S201 to S203 in the first embodiment.

[0073] In step S304, the correction unit 223c generates an intermediate corrected image using the first gain map, the captured image, and the estimated image. The intermediate corrected image generated using the first gain map, the captured image, and the estimated image is as shown in FIG. 7(d) for subjects A and B, but since the estimated image is partially used for the area of ​​subject C, undershoot remains as shown in FIG. 7(c). The intermediate corrected image is an image obtained by taking a weighted average of the captured image and the estimated image using the first gain map, and adverse effects on the low brightness side are reduced.

[0074] Next, in step S305, the generating unit 223b generates a second gain map. The second gain map in the second embodiment is similar to step S114 in the first embodiment, and therefore a description thereof will be omitted.

[0075] Next, in step S306, the correction unit 103d generates a corrected image using the captured image, the intermediate corrected image, and the second gain map. By weighting the captured image and the intermediate corrected image using the second gain map, the undershoot remaining in the area of ​​the high-luminance subject C in the intermediate corrected image is reduced. As a result, as shown in FIG. 7(d), a corrected image can be generated that maintains the correction effect of the estimated image while suppressing adverse effects. In this embodiment, the intermediate corrected image is calculated using the first gain map, and then the corrected image is generated using the intermediate corrected image and the second gain map, but the order of processing the first gain map and the second gain map may be reversed. The same effect can be obtained even if the second gain map is applied first, and then the corrected image is generated using the estimated image, the intermediate corrected image, and the first gain map.

[0076] By performing the correction process described above and using a gain map to generate a corrected image based on a captured image and an estimated image, it is possible to reduce the adverse effects of aberration correction while maintaining the effect of the aberration correction.

[0077] [Example 3] Next, an image processing system 300 according to a third embodiment will be described with reference to Fig. 17 and Fig. 18. The image processing system 300 of this embodiment differs from the first embodiment in that it has a control device (first device) 304 that acquires a captured image from an imaging device 302 and issues a request for image processing on the captured image to an image estimation device (second device) 303. Fig. 17 is a block diagram of the image processing system 300 in this embodiment. Fig. 18 is a flowchart related to generation of a corrected image in the third embodiment.

[0078] The image processing system 300 includes a learning device 301, an imaging device 302, an image estimation device 303, and a control device 304. In this embodiment, the learning device 301 and the image estimation device 303 are servers. The control device 304 is, for example, a user terminal (a personal computer or a smartphone). The control device 304 is connected to the image estimation device 303 via a network 305, and the control device 304 and the image estimation device 303 can communicate with each other. In addition, the image estimation device 303 is connected to the learning device 301 via a network 306, and the image estimation device 303 and the learning device 301 can communicate with each other.

[0079] The configurations of the learning device 301 and the imaging device 302 are similar to those of the learning device 101 and the imaging device 102 in the first embodiment, respectively, and therefore will not be described.

[0080] The image estimation device 303 has a storage unit 303a, an acquisition unit 303b, a generation unit 303c, a correction unit 303d, and a communication unit (reception means) 303e. The storage unit 303a, the acquisition unit 303b, the generation unit 303c, and the correction unit 303d are similar to the storage unit 103a, the acquisition unit 103b, the generation unit 103c, and the correction unit 103d of the first embodiment, respectively. The communication unit 303e has a function of receiving a request transmitted from the control device 304, and a function of transmitting an output image generated by the image estimation device 303 to the control device 304.

[0081] The control device 304 has a communication unit (transmission means) 304a, a display unit 304b, a processing unit 304c, and a recording unit 304d. The communication unit 304a has a function of transmitting a request to the image estimation device 303 to cause the image estimation device 303 to execute processing on the captured image, and a function of receiving an output image processed by the image estimation device 303. The display unit 304b has a function of displaying various information. The information displayed by the display unit 304b includes, for example, a captured image to be transmitted to the image estimation device 303 and an output image received from the image estimation device 303. The processing unit 304c has a function of further performing image processing on the output image received from the image estimation device 303. The recording unit 304d records the captured image acquired from the imaging device 302, the output image received from the image estimation device 303, etc.

[0082] Next, the image processing in this embodiment will be described. The image processing in this embodiment is started by an instruction to start image processing from a user via the control device 304, and is executed in the image estimation device 303.

[0083] First, the operation of the control device 304 will be described. In step S401, the control device 304 transmits a request for processing the captured image to the image estimation device 303. Note that the method of transmitting the captured image to be processed to the image estimation device 303 does not matter. For example, the captured image may be uploaded to the image estimation device 303 simultaneously with S401, or may be uploaded to the image estimation device 303 before step S401. Furthermore, the captured image may be an image stored on a server different from the image estimation device 303. Note that in step S401, the control device 304 may transmit ID information for authenticating a user together with the request for processing the captured image.

[0084] In step S402, the control device 304 receives the output image generated in the image estimation device 303. The output image is a corrected image obtained by taking a weighted average of the captured image and the estimated image, similar to the first embodiment.

[0085] Next, the operation of the image estimation device 303 will be described. In step S501, the image estimation device 303 receives a request for processing the captured image transmitted from the control device 304. The image estimation device 303 executes the processes from step S502 onwards based on the request for processing the captured image. The weight information in this embodiment is information (trained model) learned in the same manner as in embodiment 1 (FIG. 4). The image estimation device 303 may acquire the weight information from the learning device 301, or may acquire weight information acquired in advance from the learning device 301 and stored in the storage unit 303a.

[0086] Steps S502 and S503 are similar to steps S201 and S202 in the first embodiment.

[0087] In S504, the generating unit 303c generates a gain map (fourth weight map). In this embodiment, the gain map is generated based on a captured image or an estimated image. The generating unit 303c performs various processes (e.g., demosaicing process, white balance adjustment, normalization process) on the captured image or the estimated image as necessary in the same manner as in the first embodiment, and generates a gain map using a response function.

[0088] FIG. 19 shows the response function in this embodiment. In FIG. 19, the horizontal axis is the two-dimensional data after scaling, and the vertical axis is the adjusted value of the weight map. The thresholds of the response function in this embodiment are set to 0.2, 0.3, 0.8, and 0.9. As shown in FIG. 19, the weight of the estimated image is 0 when it is 0.2 or less or 0.9 or more, and the weight of the estimated image is 1 when it is 0.3 or more and 0.8 or less. The weight changes linearly in the other sections. The threshold value is not limited to this. By setting the threshold, the area where the weight of the estimated image is large can be increased, thereby obtaining a corrected image with less aberration. On the other hand, by setting the threshold, the area where the weight of the estimated image is small can be increased, thereby obtaining a corrected image with less adverse effects due to aberration correction. By using such a response function, the weight of the estimated image in low brightness and high brightness areas can be reduced in order to reduce adverse effects caused in the corrected image obtained by blur correction. The response function may be generated based on information on the correction effect. In addition, the amount of calculation can be reduced compared to other embodiments in which multiple gain maps are generated.

[0089] In step S505, the image estimation device 303 transmits the output image to the control device 304.

[0090] By executing the correction process described above and generating a corrected image based on a captured image and an estimated image using a gain map, it is possible to reduce the adverse effects of blur correction while maintaining the effect of blur correction. In this embodiment, the control device 304 only requests processing for a specific image. Actual image processing is performed by the image estimation device 303. For this reason, if the control device 304 is a user terminal, it is possible to reduce the processing load on the user terminal. Therefore, the user side can obtain an output image with a low processing load.

[0091] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0092] According to each embodiment, it is possible to provide an image processing method, an image processing device, and a program that can reduce adverse effects caused by blur correction while maintaining the effect of blur correction.

[0093] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0094] The embodiments of the present invention include the following methods, configurations, and programs.

[0095] [Method 1] generating a second image by sharpening the first image; generating a first weight map based on the first image; generating a first weight map based on the first image; generating a third image based on the first and second images and the first and second weight maps; 11. A method of image processing, comprising the steps of: generating a third image from a first image; generating a third image from a second image; and generating a third image from the first image.

[0096] [Method 2] The image processing method of method 1, wherein the second image is generated by sharpening the first image using a machine learning model.

[0097] [Method 3] generating a third weight map from the first and second weight maps; The image processing method described in Method 1, characterized in that the third image is generated by taking a weighted average of the first image and the second image using the third weight map.

[0098] [Method 4] generating an intermediate corrected image based on the first image, the second image, and the first weighting map; The image processing method according to method 1, further comprising generating the third image based on the first image, the second weight map and the intermediate corrected image.

[0099] [Method 5] The image processing method according to method 1, further comprising performing white balance adjustment on the first image and the second image.

[0100] [Method 6] The image processing method according to Method 1, characterized in that a demosaicing process is performed on the first image and the second image.

[0101] [Method 7] generating a second image by sharpening the first image; generating a weight map based on a response function and the pixel values ​​of the first image or the pixel values ​​of the second image; generating a third image based on the first and second images and the weight map; An image processing method, characterized in that the response function is generated based on information regarding correction intensity and indicates a relationship between pixel values ​​of the first image or pixel values ​​of the second image and adjustment values ​​of the weight map.

[0102] [Method 8] the first weighting map is generated using a first response function that indicates a relationship between pixel values ​​of the first image and adjustment values ​​of the first weighting map; An image processing method described in any one of methods 1 to 6, characterized in that the second weighting map is generated using a second response function indicating the relationship between pixel values ​​of the second image and adjustment values ​​of the second weighting map.

[0103] [Method 9] The image processing method described in Method 8, characterized in that the pixel value of the second image when the weights of the first and second images are equal in the second response function is greater than the pixel value of the first image when the weights of the first and second images are equal in the first response function.

[0104] [Method 10] An image processing method described in any one of methods 1 to 6, characterized in that the first and second weighting maps are generated using information regarding the performance of the optical system used in acquiring the first image.

[0105] [Method 11] 11. The image processing method according to method 10, wherein the information about the performance of the optical system is a point spread function or an optical transfer function of the optical system.

[0106] [Program 12] A program for causing a computer to execute the image processing method according to any one of Methods 1 to 11.

[0107] [Configuration 13] A storage medium storing the program described in Program 12.

[0108] [Configuration 14] 12. An image processing device comprising a processing means capable of executing the image processing method according to any one of Methods 1 to 11.

[0109] [Configuration 15] 15. An image processing system having a control device and an image processing device according to configuration 14, which are communicable with each other, 2. An image processing system according to claim 1, wherein the control device has a means for transmitting a request for execution of processing on the first image to the image processing device. [Explanation of symbols]

[0110] S202 First step S203 Second process S204 Third process

Claims

1. sharpening the first image to generate a second image; generating a first weight map based on the first image; generating a second weight map based on the second image; An image processing method comprising generating a third image based on the first and second images and the first and second weight maps.

2. The image processing method of claim 1 , wherein the second image is generated using a machine learning model.

3. generating a third weight map from the first and second weight maps; 2. The image processing method according to claim 1, wherein the third image is generated by taking a weighted average of the first image and the second image using the third weight map.

4. generating an intermediate corrected image based on the first image, the second image, and the first weight map; 2. The image processing method of claim 1, further comprising generating the third image based on the first image, the second weight map, and the intermediate corrected image.

5. 2. The image processing method according to claim 1, further comprising the step of performing white balance adjustment on the first image and the second image.

6. 2. The image processing method according to claim 1, wherein demosaicing is performed on the first image and the second image.

7. sharpening the first image to generate a second image; generating a weight map based on a response function and pixel values ​​of the first image or pixel values ​​of the second image; generating a third image based on the first and second images and the weight map; An image processing method characterized in that the response function is generated based on information regarding correction intensity and indicates the relationship between pixel values ​​of the first image or pixel values ​​of the second image and adjustment values ​​of the weight map.

8. the first weight map is generated using a first response function that indicates a relationship between pixel values ​​of the first image and adjustment values ​​of the first weight map; 7. The image processing method according to claim 1, wherein the second weight map is generated using a second response function that indicates a relationship between pixel values ​​of the second image and adjustment values ​​of the second weight map.

9. 9. The image processing method according to claim 8, wherein the pixel value of the second image when the weights of the first and second images are equal in the second response function is greater than the pixel value of the first image when the weights of the first and second images are equal in the first response function.

10. 7. The image processing method according to claim 1, wherein the first and second weight maps are generated using information about the performance of an optical system used when acquiring the first image.

11. 11. The image processing method according to claim 10, wherein the information about the performance of the optical system is a point spread function or an optical transfer function of the optical system.

12. A program causing a computer to execute the image processing method according to any one of claims 1 to 7.

13. A storage medium storing the program according to claim 12.

14. An image processing apparatus comprising processing means capable of executing the image processing method according to any one of claims 1 to 7.

15. An image processing system comprising a control device and the image processing device according to claim 14, which are capable of communicating with each other, The image processing system according to claim 1, wherein the control device has means for transmitting a request for execution of processing on the first image to the image processing device.

Images