Image processing method, image processing apparatus, image processing system, and program

Abstract

To provide an image processing method that can prevent harmful effects to be conspicuous when correction related to brightness is performed on an image on which correction using a machine learning model is performed.SOLUTION: An image processing method includes: a step of inputting a picked-up image to a machine learning model to create a first image (S112); a step of acquiring correction information related to brightness in performing development processing or correction processing on a second image (S113); and a step of performing weighted addition of the picked-up image and the first image on the basis of the correction information to acquire the second image (S115). The correction information is constant with respect to a position in the image, or changes depending on the position in the image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing method, an image processing device, an image processing system, and a program. [Background technology]

[0002] In image recognition or regression tasks, methods using machine learning models can achieve higher accuracy than theory-based methods that use assumptions and approximations. In theory-based methods, accuracy decreases due to elements ignored by assumptions and approximations. However, in methods using machine learning models, by training the machine learning model using training data that includes those elements, estimation based on training data without assumptions or approximations can be achieved, improving the accuracy of the task.

[0003] For example, in a technology for sharpening a blurred captured image, the element is brightness saturation (also called whiteout) of the captured image. In theory-based methods such as the Wiener filter, it is assumed that there is no brightness saturation, so the blur is not properly sharpened around the brightness saturated area, resulting in problems such as ringing and undershooting.

[0004] In response to this, Patent Document 1 discloses a method for sharpening the blur of a captured image using a convolutional neural network (CNN), which is one of the machine learning models. A training data set is generated by blurring an image having a signal value equal to or greater than the brightness saturation value of the captured image, and a CNN is trained with the training data set, so that the blur can be sharpened while suppressing adverse effects even around the brightness-saturated region. Patent Document 1 also discloses a method for adjusting the strength of sharpening by taking a weighted average of the captured image and an estimated image (blur-sharpened image) based on the brightness-saturated region. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-166628 A Summary of the Invention [Problem to be solved by the invention]

[0006] When performing correction processing using a method using a machine learning model such as a neural network, if the training data and correct answer data are set appropriately, correction is possible even if the above-mentioned information-missing areas are included. However, in the case of a method using a machine learning model, adverse effects may occur even in areas that do not occur in deconvolution processing. In particular, adverse effects such as black sinking and white floating may occur in bright areas close to the brightness saturation value and dark areas close to the black level. Black sinking is a phenomenon in which the brightness value becomes lower than it should be, and white floating is a phenomenon in which the brightness value becomes higher than it should be.

[0007] Such problems tend to occur when the subject is significantly blurred due to optical system aberration, when the blur changes significantly with respect to the screen position, when a machine learning model that can handle a variety of aberrations is used, and when the task is difficult. In addition, problems tend to occur not only near edges but also in bright and dark areas as described above. For this reason, although it is difficult to see in the estimated image, when brightness correction such as peripheral illumination correction and exposure correction or development processing is performed, the above problems may be emphasized to a degree that is visible.

[0008] Therefore, an object of the present invention is to provide an image processing method capable of suppressing the adverse effects that occur when performing brightness correction on an image obtained by performing correction using a machine learning model. [Means for solving the problem]

[0009] An image processing method as one aspect of the present invention includes the steps of generating a first image by inputting a captured image into a machine learning model, acquiring correction information related to brightness when performing a development process or correction process on a second image, and acquiring the second image by weighted addition of the captured image and the first image based on the correction information, wherein the correction information is constant with respect to a position in the image or varies depending on the position in the image.

[0010] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0011] According to the present invention, it is possible to provide an image processing method capable of suppressing adverse effects that occur when performing brightness correction on an image obtained by performing correction using a machine learning model. [Brief description of the drawings]

[0012] [Figure 1] 5 is a flowchart showing a process of generating a blur-corrected image in the first embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 4 is a flowchart showing a learning process in the first embodiment. [Diagram 5] FIG. 2 is an explanatory diagram of a learning flow of the neural network in the first embodiment. [Figure 6] 5A to 5C are diagrams illustrating the effect of the blur correction process in the first embodiment. [Figure 7] 5A to 5C are explanatory diagrams of the advantages and disadvantages of the blur correction process in the first embodiment. [Figure 8] 6 is an explanatory diagram of weighting of a captured image with respect to a correction amount related to brightness in the first embodiment. FIG. [Figure 9] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 10] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 11] 10 is a flowchart showing a process of generating a blur-corrected image in the second embodiment. [Figure 12] FIG. 11 is an explanatory diagram of the demosaicing process for G in the second embodiment. [Figure 13] FIG. 11 is an explanatory diagram of an interpolation filter in the demosaicing process in the second embodiment. [Figure 14] FIG. 11 is an explanatory diagram of the demosaicing process for R in the second embodiment. [Figure 15] FIG. 11 is an explanatory diagram of a response function used when creating a gain map in the second embodiment. [Figure 16] FIG. 11 is an explanatory diagram of a gain map in the second embodiment. [Figure 17] 13A to 13C are explanatory diagrams of the advantages and disadvantages of the blur correction process in the second embodiment. [Figure 18] FIG. 11 is an explanatory diagram of adjustment values ​​of a gain map in the second embodiment. [Figure 19] FIG. 11 is a block diagram of an image processing system according to a third embodiment. [Figure 20] 13 is a flowchart showing image processing in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0014] First, before describing each embodiment in detail, the gist of each embodiment will be described. Each embodiment generates an estimated image (first image) by sharpening blur caused by an optical system from an image captured using the optical system using a machine learning model. Then, weighted addition is performed on the captured image and the estimated image based on correction information related to brightness to generate a final blur-corrected image (second image).

[0015] Here, the blur caused by the optical system includes blur caused by aberration, diffraction, defocus, the action of an optical low-pass filter, deterioration of the pixel aperture of the image sensor, etc. The machine learning model includes, for example, a neural network, genetic programming, a Bayesian network, etc. The neural network includes a CNN (Convolutional Neural Network), a GAN (Generative Adversarial Network), an RNN (Recurrent Neural Network), etc.

[0016] The blur sharpening refers to a process of restoring the frequency components of a subject that have been reduced or lost due to blurring. When the blurring is sharpened, depending on the captured image, it may not be possible to suppress adverse effects such as black sinking and white floating in bright areas close to the luminance saturation value or dark areas close to the black level, and these adverse effects may occur in the estimated image. Specifically, when the subject is significantly blurred due to the aberration of the optical system, when the blur changes significantly with respect to the screen position, when a machine learning model corresponding to various aberrations is used, or when the task is difficult, the above-mentioned adverse effects tend to occur easily. Although the sharpening process (blur correction process) is given as an example, each image processing described in the following embodiment can also be applied to image processing such as high contrast, brightness improvement, defocus blur conversion, and lighting conversion.

[0017] By applying the processing of this method, it is possible to suppress (reduce) adverse effects that occur when brightness correction is performed on an estimated image. Note that the adverse effects in this embodiment are, for example, black sinking and white floating that occur in bright areas close to the brightness saturation value or dark areas close to the black level. In the following, the stage of learning the weights of the machine learning model is referred to as the learning phase, and the stage of sharpening blur using the machine learning model using the learned weights is referred to as the estimation phase. EXAMPLES

[0018] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, blur is sharpened (blur correction processing) for a captured image using a machine learning model. The blur to be sharpened is aberration and diffraction occurring in the optical system, and blur caused by an optical low-pass filter. However, the effects of the invention can be obtained in the same way when blur caused by pixel aperture, defocus, or shaking is sharpened. The invention can also be implemented and the effects obtained in the same way for tasks other than blur sharpening (blur correction processing).

[0019] Fig. 2 is a block diagram of an image processing system 100 in this embodiment. Fig. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107.

[0020] The learning device 101 is an image processing device that executes a learning process, and includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and an update unit 101d. The acquisition unit 101b acquires training images and correct images. The generation unit 101c inputs the training images into a multi-layered neural network to generate an output image. The update unit 101d updates the network parameters of the neural network based on the error between the output image and the correct image calculated by the generation unit 101c. Details of the learning process will be described later using a flowchart. The network parameters obtained by learning are stored in the storage unit 101a.

[0021] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives (photoelectrically converts) an optical image (subject image) formed via the optical system 102a to obtain a captured image. The imaging element 102b is, for example, a charge coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. The captured image obtained by the imaging device 102 contains blur due to aberration and diffraction of the optical system 102a and noise due to the imaging element 102b.

[0022] The image estimation device 103 is a device that executes the estimation process, and includes a storage unit 103a, an acquisition unit 103b, and a correction unit 103c. The image estimation device 103 sharpens the blur of the captured image acquired to generate an estimated image (first image). A multi-layered neural network is used for the blur sharpening, and information on the network parameters is read from the storage unit 103a. The multi-layered neural network is trained by the learning device 101, and the image estimation device 103 reads the network parameters from the storage unit 101a via the network 107 in advance and stores them in the storage unit 103a. The stored network parameters may be in the form of their numerical values ​​or in an encoded format. Details regarding the training of the multi-layered neural network and the blur sharpening process using the multi-layered neural network will be described later.

[0023] The output image is output to at least one of a display device 104, a recording medium 105, and an output device 106. The display device 104 is, for example, a liquid crystal display or a projector. A user can perform editing work while checking an image being processed through the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The output device 106 is, for example, a printer. The image estimation device 103 has a function of performing development processing and other image processing as necessary. (How to learn) Next, a method for updating weights (weight information) (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart related to weight learning (learning process). Each step in Fig. 4 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the update unit 101d of the learning device 101. Fig. 5 is a diagram showing the flow of updating the weights of a neural network (learning a machine learning model).

[0024] First, in step S101 of FIG. 4, the acquisition unit 101b acquires an original image (subject image). In this embodiment, the original image is a high-resolution (high-quality) image with little blurring due to aberration and diffraction of the optical system 102a. A plurality of original images are acquired, and are images of various subjects, that is, images having edges of various strengths and directions, textures, gradations, flat areas, etc. The original image may be a real image or an image generated by CG (Computer Graphics).

[0025] Preferably, the original image has a signal value higher than the brightness saturation value of the image sensor 102b. This is because, even in actual subjects, there are subjects that do not fall within the brightness saturation value when imaged by the image capture device 102 under specific exposure conditions. A high-resolution captured image is generated by reducing the original image and clipping the signal at the brightness saturation value of the image sensor 102b. In particular, when a real image is used as the original image, blurring has already occurred due to aberration and diffraction, so that by reducing the image, the effect of blurring can be reduced and a high-resolution (high-quality) image can be obtained. Note that if the original image contains sufficient high-frequency components, it is not necessary to reduce the image. Also, the original image may contain noise components. In this case, the noise contained in the original image can be considered to be the subject, so the noise in the original image is not particularly problematic.

[0026] Next, in step S102, the acquisition unit 101b acquires the blur used to perform the imaging simulation described later. First, the imaging conditions corresponding to the lens state (zoom, aperture, and focal distance state) of the optical system 102a are acquired. Then, the blur determined by the imaging conditions and the screen position is acquired. Here, the blur is the PSF (point spread function) or OTF (optical transfer function) of the optical system 102a. The blur can be acquired by optical simulation or measurement in the optical system 102a. Note that the blur due to the lens state, image height, aberration of azimuth, and diffraction, which differ for each original image, is acquired. This makes it possible to perform imaging simulation corresponding to a plurality of imaging conditions, image heights, and azimuth. In addition, a component such as an optical low-pass filter included in the imaging device 102 may be added to the blur to be applied as necessary.

[0027] Next, in step S103, the generating unit 101c generates a correct answer patch (correct answer image) and a training patch (training data). A plurality of correct answer patches and a plurality of training patches are generated, and one or more patches are generated corresponding to one original image. In this embodiment, the correct answer patch and the training patch are images of the same subject. In this embodiment, a plurality of combinations of the correct answer patch and the training patch are used as learning data. Note that a patch refers to an image having a predetermined number of pixels (for example, 64×64 pixels). Also, the number of pixels of the correct answer patch and the training patch does not necessarily have to be the same.

[0028] In this embodiment, mini-batch learning is used to learn the weights of the multi-layered neural network. Therefore, in step S103, multiple pairs of ground truth patches and training patches are generated. However, this embodiment is not limited to this, and online learning or batch learning may be used. In this embodiment, the original image is an undeveloped RAW image, and the ground truth patches and training patches are also RAW images. However, the present invention is not limited to this, and may be a developed image or a feature map obtained by converting an image as described later. In addition, the position of a partial region refers to the center of the partial region.

[0029] Next, in step S104, the generation unit 101c inputs the training patches (training data) 212 shown in Fig. 5 to a multi-layer neural network to generate estimated patches (estimated images) 213. For mini-batch learning, estimated patches 213 corresponding to the multiple training patches 212 are generated. Fig. 5 is an explanatory diagram of the flow of neural network learning, showing the flow from step S104 to step S105 in Fig. 4.

[0030] The estimated patch 213 has a higher sharpness than the training patch 212, and ideally matches the correct patch (correct image) 211. In this embodiment, the neural network configuration shown in FIG. 5 is used, but is not limited to this. CN in FIG. 5 represents a convolution layer, and DC represents a deconvolution layer. In both CN and DC, the convolution of the input and the filter and the sum of the bias are calculated, and the result is nonlinearly transformed by the activation function. The initial values ​​of each component of the filter and the bias are arbitrary, and are determined by random numbers in this embodiment. For example, ReLU (Rectified Linear Unit) or a sigmoid function can be used as the activation function. The output of each layer except the final layer is called a feature map.

[0031] The skip connections 222 and 223 combine feature maps output from discontinuous layers. The combination of feature maps may be performed by element-by-element summation or by channel-wise concatenation. In this embodiment, element-by-element summation is adopted. The skip connection 221 takes the sum of the residual estimated from the training patch 212 and the ground truth patch 211, and the training patch 212 to generate an estimated patch 213. An estimated patch 213 is generated for each of the multiple training patches 212.

[0032] Next, in step S105 of FIG. 4, the update unit 101d updates the weights (weight information) of the neural network based on the error between the estimated patch 213 and the correct patch (correct image) 211. Here, the weights include the filter components and biases of each layer. Backpropagation is used to update the weights, but this embodiment is not limited to this. For mini-batch learning, the errors between the multiple correct patches 211 and the corresponding estimated patches 213 are obtained, and the weights are updated. For the loss function, for example, the L2 norm or the L1 norm may be used.

[0033] Next, in step S106, the update unit 101d determines whether the weight update is complete. Completion of the weight update can be determined by whether the number of iterations of learning (weight update) reaches a specified value, or whether the amount of change in weight at the time of update is smaller than a specified value. If it is determined that the weight update is not complete, the process returns to step S103, and multiple new answer patches and training patches are obtained. On the other hand, if it is determined that the weight update is complete, the learning device 101 (update unit 101d) ends learning, and stores the weight information in the storage unit 101a. (Estimation method) Next, the generation of a blur-corrected image (blur correction processing) executed by the image estimation device 103 in this embodiment will be described. Before going into a detailed description of the correction processing, the effects of this embodiment will be described. FIGS. 6(a) to (d) are explanatory diagrams of the blur correction processing using a machine learning model, and each is a schematic diagram showing the luminance value and spatial domain of an image. FIG. 6(a) shows an original image in an ideal state without aberration (blur), FIG. 6(b) shows a captured image degraded by aberration of the optical system, FIG. 6(c) shows an image after blur correction by deconvolution processing, and FIG. 6(d) shows an estimated image after blur correction processing by this method.

[0034] In Figures 6(a) to (d), the dotted lines indicate the brightness saturation value, with subject A having a brightness value below the brightness saturation value and subject B having a brightness value above the brightness saturation value. When subject B, which has a brightness above the brightness saturation value in Figure 6(a), is imaged by an imaging device, the image in Figure 6(b) is degraded by aberration in the optical system, and then the signal is clipped at the brightness saturation value. Figure 6(c) shows an example of applying deconvolution processing to such an image.

[0035] Examples of filters using deconvolution include Wiener filters and inverse filters. Equivalent processing can be performed by multiplying the image and filter by Fourier transforming them in frequency space. In this type of processing, degradation due to aberration can be reduced compared to the captured image in FIG. 6(b) for degradation such as that of object A in FIG. 6(c). However, for areas including saturation areas such as object B, overshoots and undershoots may occur. The overshoot occurring near the edge of the saturation area in FIG. 6(c) is actually clipped at the saturation value and does not exist in the image, but the undershoot occurring in the dark area does exist in the image. The black level of the captured image is offset from zero, and when an undershoot occurs, it is clipped so that it does not fall below zero, but it falls below this offset value (OB value). In this way, the area below the black level becomes darker than the surroundings, and this causes a problem in the image. This is due to the loss of information caused by the luminance value of the object not falling within the dynamic range of the image sensor and being clipped. In the case of convolution processing as shown in Figure 6(c), since it is a linear operation, it is difficult in principle to avoid such issues.

[0036] On the other hand, as shown in Figure 6(d), when a machine learning model is used, it is possible to reduce such adverse effects. For example, even in the case of a neural network, convolution calculations are used, but a nonlinear function can be used for the activation function. When a machine learning model is used, nonlinear processing is used, allowing for more flexible processing. However, when training, it is necessary to use learning data that includes luminance saturation parts. If training is performed using a data set that does not include luminance saturation parts, there is a possibility that serious adverse effects such as overshoot and undershoot may occur, as in Figure 6(c). On the other hand, by using captured images that include luminance saturation parts in the training data and images with little aberration (blur) and no adverse effects around the luminance saturation parts as correct images, it is possible to perform blur correction processing that reduces aberrations while suppressing adverse effects for both subjects A and B, as shown in Figure 6(d).

[0037] So far, we have discussed the advantages of using a machine learning model to perform blur correction, but this is not necessarily a method that can perform blur correction without any drawbacks. Even in the case of captured images that contain brightness saturation parts, it can be considered that blur correction can be performed while suppressing drawbacks by determining whether or not there is a brightness saturation part inside the machine learning model and changing the processing depending on the result. However, if there is an error in this judgment, it will also affect the result.

[0038] With reference to Figs. 7(a)-(d), we will describe the problems that occur when making corrections using a machine learning model and how to deal with them. Figs. 7(a)-(d) are explanatory diagrams of the effects and problems of blur correction processing. Figs. 7(a) and (b) respectively show an original image in an ideal state without aberration (blur) and a captured image degraded by the aberration of the optical system. While Figs. 6(a)-(d) only show two subjects, A and B, Figs. 7(a)-(d) show three subjects: subject A with a low brightness value, subject B with a value about half the brightness saturation value, and subject C with a value close to the brightness saturation value.

[0039] Subject C in Fig. 7(a) shows an example where misjudgment is likely to occur, and it is difficult to judge that a subject that is flat near the brightness saturation value has a value above the brightness saturation value and is clipped at the brightness saturation value. Note that Figs. 6(a)-(d) and Figs. 7(a)-(d) are schematic diagrams and are evaluated without noise, but actual captured images are also affected by noise. When values ​​vary due to noise near the brightness saturation value as in the case of subject C in Fig. 7(a), the noise can cause the value to exceed or fall below the brightness saturation value. If a region that was not saturated is erroneously judged to be a brightness saturation region and corrected, a larger correction will be applied.

[0040] FIG. 7(c) shows this state, and the result of the estimation process using the machine learning model for the captured image of FIG. 7(b) is shown in FIG. 7(c). As shown in FIG. 7(a), the maximum luminance value of the object C coincides with the luminance saturation value in a state without aberration, and FIG. 7(c) shows the result of the machine learning model erroneously determining that the luminance is saturated, resulting in overcorrection for the object C. Basically, such a problem is caused by an attempt to reduce the signal more than necessary, so that the luminance value tends to be lower than it should be due to blackening. In this embodiment, the purpose is to suppress such a problem. In addition, such a problem does not necessarily occur only on the high luminance side, but also appears at other luminance values. The object A in FIG. 7(c) shows one of the patterns.

[0041] As described above, when learning is performed using a RAW image in the learning process, blur correction processing is also performed on the RAW image during estimation. A RAW image is undeveloped image data output from an image sensor, and the light amount and luminance value of each pixel have an approximately linear relationship. Normally, a RAW image is developed and various correction processing is performed before the user views the image, and at that time, brightness corrections such as exposure correction and peripheral illumination correction, and gradation conversions such as gamma correction and bit conversion are applied.

[0042] Brightness correction processes such as exposure compensation and peripheral illumination compensation emphasize the problems in low or high brightness areas. Gamma correction is a process in which the input brightness value is raised to a power, and 1 / 2.2 is used as the exponent. Gamma correction is a process in which the curve showing the relationship between the brightness values ​​before and after correction is at a position equal to or greater than a straight line with a slope of 1, and is a characteristic that depends on the brightness value. During development, tone conversion processes such as gamma correction, black level and white level adjustment, and bit conversion are performed, so low brightness areas tend to be blocked up and high brightness areas tend to be blown out, and in such light and dark areas, the slight brightness differences mentioned above are difficult to see after development.

[0043] In Fig. 7(c), for ease of understanding, the amount of black sinking is shown to be about the same for subject C on the high-brightness side and subject A on the low-brightness side, but even if the amount is slight in the RAW image, the problem can be accentuated by performing brightness correction processing and development processing. The brighter the low-brightness side is made, the more the problem of black crush that was previously invisible after development becomes apparent. If the high-brightness side is made darker, the problem of white blowout that was previously invisible after development becomes apparent.

[0044] In this embodiment, in order to suppress these problems, a final blur-corrected image (second image) is generated by performing weighted addition of the captured image and the estimated image (first image) based on the correction information on brightness. Fig. 7(d) is a simulation diagram of the blur-corrected image generated based on the captured image, the estimated image, and the above-mentioned correction information on brightness. Even if problems such as blackening or whitening occur in the estimated image, the blur correction strength of the estimated image based on the correction information on brightness can be adjusted to suppress the problems that occur when the correction on brightness is performed.

[0045] Next, the generation of a blur-corrected image (blur correction processing) executed by the image estimation device 103 in this embodiment will be described with reference to Fig. 1. Fig. 1 is a flowchart showing the blur-corrected image generation processing. Each step in Fig. 1 is mainly executed by the acquisition unit 103b and the correction unit 103c of the image estimation device 103.

[0046] First, in step S111, the acquisition unit 103b acquires information on a captured image and weights. The captured image is an undeveloped RAW image, similar to learning, and in this embodiment, is transmitted from the imaging device 102. The weight information is the weights of the machine learning model transmitted from the learning device 101 and stored in the storage unit 103a.

[0047] Next, in step S112, the acquisition unit (first means) 103b executes blur sharpening processing based on the weights of the machine learning model acquired in step S111, and acquires an estimated image (first image).

[0048] Next, in step S113, the acquisition unit (second means) 103b acquires correction information related to brightness. The correction information related to brightness is correction information related to brightness that is applied to a blur-corrected image acquired in a later stage, and is parameters (correction amounts) such as exposure correction and peripheral illumination correction. A constant parameter may be acquired regardless of the position on the screen (position in the image) as in exposure correction, or a parameter that changes depending on the position on the screen (position in the image) as in peripheral illumination correction. Alternatively, both may be acquired.

[0049] Furthermore, white balance correction parameters may be acquired as correction information related to brightness. The white balance correction parameters are constant parameters that do not depend on the position on the screen (the position in the image). White balance correction can be regarded as exposure correction that differs for each color of the image, and white balance correction parameters (correction amount) may be acquired for each color. In this case, the parameter (correction amount) with the largest absolute value among the white balance correction parameters (correction amount) for each color may be acquired as correction information related to brightness.

[0050] Furthermore, the correction information regarding brightness may include not only the absolute value of the correction amount but also positive / negative information as the correction direction of whether to brighten or darken. For example, it may be a parameter expressed as +3 steps to brighten and -3 steps to darken, or a parameter regarding a ratio expressed as +30% to brighten and -30% to darken. Even in the case of a parameter based on 100%, it can be considered as a positive parameter to brighten and a negative parameter to darken. Note that the peripheral illumination correction may include not only positive correction to brighten but also negative correction to add peripheral illumination (darken).

[0051] For parameters that depend on the position on the screen, such as peripheral illumination correction, parameters corresponding to the position of the entire screen may be acquired as a two-dimensional map. Alternatively, a coefficient obtained by converting a correction value according to the distance from the center of the screen (correction center) into a function may be acquired, and a parameter (correction amount) corresponding to the distance from the center of the screen (correction center) may be acquired. Furthermore, the correction information regarding brightness is performed on the blur-corrected image (second image) obtained by the processing of this embodiment, and may be a default value, or the correction information regarding brightness specified by the user may be acquired. Alternatively, information obtained by estimating correction information regarding preferred brightness from the captured image, estimated image, or blur-corrected image may be acquired.

[0052] Next, in step S114, the correction unit 103c acquires a weight for weighted addition of the captured image and the estimated image (first image) based on the correction information related to brightness acquired in step S113. As described above, the brightness correction process such as exposure correction and peripheral light amount correction emphasizes adverse effects in low brightness areas or high brightness areas. Therefore, the larger the absolute value of the correction amount in the correction information related to brightness, the larger the weight of the captured image during weighted addition is, thereby suppressing the blur correction strength in the estimated image and suppressing adverse effects.

[0053] For example, the weight of the captured image is set to be larger when the exposure correction is +3 steps than when it is 0 steps. By increasing the weight of the captured image when the captured image and the estimated image are weighted and added, the blur correction strength in the estimated image is suppressed, and adverse effects in low brightness areas that are emphasized when the image is brightened can be suppressed. In addition, by increasing the weight of the captured image when the exposure correction is -3 steps than when it is 0 steps, adverse effects in high brightness areas that are emphasized when the image is darkened can be suppressed. For example, as shown in FIG. 8, the weight of the captured image at the time of weighted addition is set with respect to the correction amount related to brightness. FIG. 8 is an explanatory diagram of the weight of the captured image with respect to the correction amount related to brightness. In FIG. 8, the horizontal axis indicates the correction amount related to brightness, and the vertical axis indicates the weight of the captured image. However, this embodiment is not limited to this, and various modifications are possible. When information on peripheral light correction, which is a correction depending on the screen position, is acquired as the correction information related to brightness, the weight of the captured image at the time of weighted addition may be set for each position on the screen, and the weight of the captured image may be acquired as a gain map that expresses the weight of the captured image in two dimensions.

[0054] Here, the weight for weighted addition is obtained based on the absolute value of the correction amount related to brightness, but the weight for weighted addition does not need to be the same whether the correction amount is positive or negative (brightening or darkening), and different weights may be used for positive and negative corrections related to brightness. However, the weight for the captured image is increased as the absolute value of the correction amount in the correction information related to brightness increases. Also, the weight for the captured image may be increased when the correction amount in the correction information related to brightness is equal to or less than a threshold value or equal to or greater than a threshold value, or when the absolute amount of the correction amount is equal to or greater than a threshold value. Also, the weight for the captured image is obtained as the weight for weighted addition, but the weight for an estimated image may be obtained. The weight for the estimated image is 1-the weight for the captured image.

[0055] Next, in step S115, the correction unit (third means) 103c performs weighted addition of the captured image and the estimated image (first image) based on the weight of the weighted addition acquired in step S114. As a result, the correction unit 103c adjusts the blur correction intensity to acquire a blur-corrected image (second image) in which adverse effects are suppressed. FIG. 7(d) is a simulation diagram of a blur-corrected image after weighted addition of the captured image and the estimated image using the acquired weight of the weighted addition. It can be seen that the darkening that occurred in the low-luminance subject A and the high-luminance subject C has been reduced compared to the estimated image in FIG. 7(c). When a large correction is made to the brightness, an image in which adverse effects are suppressed is acquired by suppressing the blur correction intensity as shown in FIG. 7(d). In this case, the adverse effects are not noticeable even in the output image obtained by performing various corrections and conversion processes including development processing and corrections to the brightness on the blur-corrected image. When the correction of brightness is weak or not performed, a blur-corrected image in which the blur correction strength is suppressed little or not suppressed will be obtained, as shown in Fig. 7(c). In this case, it is difficult to visually recognize the adverse effects in the output image obtained by performing development processing, various corrections, and conversion processing on the blur-corrected image. Therefore, when the correction of brightness is not performed, there is no need to suppress the strength of blur correction.

[0056] Next, in step S116, the correction unit 103c performs various processes including development processing or brightness correction and conversion processing (correction processing) on ​​the blur-corrected image (second image) acquired in step S115 to generate an output image.

[0057] Through the above processing, it is possible to obtain a blur-corrected image or output image in which adverse effects such as blackening and whitening that occur in low and high luminance areas due to correction of brightness are suppressed.

[0058] In this embodiment, the weights of the weighted summation are obtained based on the correction information on the brightness, but other information may be added to obtain the weights of the weighted summation. For example, the weights of the weighted summation or the gain map, which is a two-dimensional weight, may be generated based on the performance of the optical system at the time of image capture, and may be combined with the weights based on the correction information on the brightness. Alternatively, the information of the gain map based on the performance of the optical system may be reflected in the acquired weighted summation weights or gain map. The combined gain map may be generated by comparing these two weighted summation weights or gain maps and adopting the value of the larger weight of the captured image, or the combined gain map may be generated by multiplying or averaging the two weighted summation weights or gain maps.

[0059] When the performance of the optical system is low, the amount of correction by the estimation process also increases, so a large blur correction effect can be obtained, but problems such as black sinking and white floating are more likely to occur. Therefore, the weight of the captured image may be adjusted based on the PSF (point spread spread) and OTF (optical transfer function) of the optical system. Since the PSF changes depending on the position in the captured image, for example, at high image heights where the PSF spreads widely and performance is likely to be poor, the risk of problems occurring can be reduced by suppressing the amount of blur correction for the center of the screen. EXAMPLES

[0060] Next, an image processing system according to a second embodiment of the present invention will be described with reference to Fig. 9. In this embodiment, a blur-corrected image is generated by an image estimation unit 323 in an imaging device. Also, the flow of generating a blur-corrected image is different from that of the first embodiment. In this embodiment, the blur correction strength of an estimated image is adjusted by performing weighted addition based on the luminance values ​​in the captured image and the estimated image in addition to correction information related to brightness, thereby suppressing adverse effects and achieving a good blur correction effect at the same time.

[0061] Fig. 9 is a block diagram of an image processing system 300 in this embodiment. Fig. 10 is an external view of the image processing system 300. The image processing system 300 includes a learning device (image processing device) 301 and an imaging device 302 connected via a network 303. The learning device 301 has a storage unit (storage means) 311, an acquisition unit (acquisition means) 312, a generation unit (generation means) 313, and an update unit (learning means) 314, and updates weights (weight information) in a neural network for performing blur correction.

[0062] The imaging device 302 captures an image of a subject space to obtain a captured image, generates an estimated image (first image) from the captured image using the read weight information, and generates a blur-corrected image (second image) by weighted addition of the captured image and the estimated image. The imaging device 302 has an optical system 321 and an image sensor 322. The image estimation unit 323 has an acquisition unit 323a and a correction unit 323b, and performs estimation processing on the captured image using weight information stored in a storage unit 324.

[0063] The weight information is learned in advance by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads out the weight information from the storage unit 311 via the network 303 and stores it in the storage unit 324. A blur-corrected image (second image) or an output image generated using the captured image and the estimated image (first image) is stored in the recording medium 325. When an instruction is given from the user regarding the display of the blur-corrected image or the output image, the stored blur-corrected image or the output image is read out and displayed on the display unit 326. Note that a captured image already stored in the recording medium 325 may be read out and an estimation process may be performed by the image estimation unit 323. The above series of controls are performed by the system controller 327.

[0064] The learning method of the machine learning model executed by the learning device 301 of this embodiment is the same as that of the first embodiment, and therefore the description thereof will be omitted.

[0065] Next, generation of a blur-corrected image (second image) executed by the image estimation unit 323 in this embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing a blur-corrected image generation process. Each step in Fig. 11 is mainly executed by the acquisition unit 323a and correction unit 323b of the image estimation unit 323. Note that steps S211 to S213 and S216 in Fig. 11 are similar to steps S111 to S113 and S116 in Fig. 1 executed by the acquisition unit 103b and correction unit 103c in the first embodiment, respectively, and therefore description thereof will be omitted.

[0066] In step S214, the correction unit 323b determines the weight of the weighted addition based on the brightness value of the captured image or the estimated image (first image) and the correction information related to brightness. As described above, when the estimated image is brightened, the adverse effects in low brightness areas tend to be large, and when the estimated image is darkened, the adverse effects in high brightness areas tend to be large. Therefore, it is preferable to adjust the blur correction intensity by performing weighted addition based on not only the correction information related to brightness but also the brightness value of the captured image or the estimated image (first image). This makes it possible to achieve both the blur correction effect and the suppression of adverse effects.

[0067] For example, when the correction amount in the brightness correction information is positive (in the direction of making the second image brighter), the weight of the captured image in the weighted addition is increased as the luminance value in the captured image or estimated image (first image) decreases. On the other hand, when the correction amount in the brightness correction information is negative (in the direction of making the second image darker), the weight of the captured image is increased as the luminance value in the captured image or estimated image (first image) increases.

[0068] More preferably, when the correction amount in the brightness correction information is positive (in the direction of making the second image brighter), the weight of the captured image in the weighted addition is increased as the luminance value in the captured image decreases. Also, when the correction amount in the brightness correction information is negative (in the direction of making the second image darker), it is preferable to increase the weight of the captured image as the luminance value in the estimated image (first image) increases. (dark part) First, an example will be described in which the weight of a captured image in weighted addition is increased as the luminance value of the captured image decreases when the correction amount in the correction information regarding brightness is positive (in the direction of making the image brighter).

[0069] 12(a) to 12(c) are explanatory diagrams relating to color components of a captured image, showing demosaicing processing for G. In this embodiment, the captured image is an image of a Bayer array as shown in FIG. 12(a). Here, RGB represent red, green, and blue, respectively. In this embodiment, in generating the weight of the captured image, demosaicing processing, white balance correction, normalization processing, etc. are performed on the captured image, so each processing will be explained in order.

[0070] First, demosaicing is performed on a captured image configured with the array shown in FIG. 12(a). Since there are two types of Bayer arrays for G only, G1 and G2, the demosaicing process differs between G and RB. FIG. 12(b) shows a state in which only G1 and G2 are extracted from FIG. 12(a). When performing demosaicing on G in FIG. 12(a), linear interpolation is performed after extracting G as shown in FIG. 12(b). FIG. 13(a) shows a filter used for linear interpolation, and by convolving the filter in FIG. 13(a) with G in FIG. 12(b), an interpolated G as shown in FIG. 12(c) can be generated.

[0071] Next, the demosaicing process for RB will be described. Similar processes can be applied to both R and B with only one type, so in this embodiment, the process contents will be described only for R. Fig. 14(a) to (c) are explanatory diagrams of the demosaicing process method for R. Fig. 14(a) shows a state in which only R is extracted from the Bayer array in Fig. 12(a), and in this embodiment, color difference information is calculated by taking the difference of G with respect to R. Fig. 14(b) shows the color difference of R before demosaicing process, and the difference of G after demosaicing process in Fig. 12(c) is taken only for pixels where the value of R exists in Fig. 14(a).

[0072] Next, an interpolation process is performed on the color difference in FIG. 14(b). FIG. 13(b) shows an example of a filter for performing the interpolation process, and color difference information corresponding to all pixels can be calculated by convolving such a filter. FIG. 14(c) shows R after demosaicing process, which can be calculated by adding G to the color difference after convolving the filter in FIG. 13(b). Although the demosaicing process for R has been described above, the demosaicing process can be performed similarly for B. In this embodiment, the interpolation process is performed using the color difference for RB, but the interpolation process may be performed directly for RB. In addition to the simple interpolation process, the demosaicing process may be performed by incorporating an adaptive process using the gradient of RGB values ​​or the like.

[0073] Next, white balance correction is performed on the captured image after demosaicing. White balance correction is a process of adjusting the luminance level by performing gain processing for each color component on the RGB components output by the image sensor so that the achromatic parts of the subject become achromatic in the output image. If white balance correction is not performed, the color of the subject is not reproduced correctly due to the color characteristics of the image sensor, and an image with a color different from the actual subject is generated. In this embodiment, by performing white balance correction, a two-dimensional gain map, which is the weight of the captured image, can be generated with a higher degree of composition. The coefficient (white balance coefficient) used for white balance correction can use the settings at the time of capturing the captured image. Normally, since there is a white balance coefficient for each RGB color component, white balance correction can be performed by multiplying the coefficient by the color component of the captured image corresponding to the color component. Here, if the black level is offset, the offset is subtracted before white balance correction is performed. Also, if the white balance coefficients are divided into G1 and G2, the average of them is used as the white balance coefficient for G.

[0074] The white balance coefficients may not be the settings at the time of shooting, but may be a combination of other coefficients. For example, general digital cameras have a function called auto white balance that automatically identifies the type of light source and corrects it, and the coefficient data is also attached to the captured image and can be used for white balance correction.

[0075] Next, normalization processing is performed on the captured image after white balance correction. In the normalization processing of this embodiment, first, the maximum value of RGB is obtained for each pixel of the captured image having three colors of RGB. In this embodiment, the maximum value is used, but it is not limited to this, and an index such as a median value, an average value, or a minimum value may be used. In this manner, in this embodiment, by setting one value for one pixel, the weights are prevented from being different between RGB. This makes it possible to avoid problems such as an estimated image being used for RB and a captured image being used for G, resulting in a broken image. Next, scaling is performed on the two-dimensional data composed of the maximum values ​​of RGB so that the black level is 0 and the brightness saturation value is 1. Here, scaling is performed based on the black level and the brightness saturation value, but scaling may be performed based on other values. For example, conversion may be performed by dividing by 2 bytes so that 2 bytes become 1.

[0076] Then, a two-dimensional first gain map, which is the weight of the weighted addition, is calculated based on the two-dimensional data after scaling. FIG. 15(a) shows a first response function for obtaining the first gain map. The horizontal axis of FIG. 15(a) is the two-dimensional data after scaling that is the input, and the vertical axis is the weight used when performing weighted addition of the captured image and the estimated image. In this embodiment, the weight on the vertical axis of FIG. 15(a) is the weight applied to the estimated image. When the weight of the estimated image increases, the weight of the captured image decreases. Therefore, in FIG. 15(a), when the luminance value of the two-dimensional data after scaling increases, the output value also increases, which means that the estimated image is adopted. On the other hand, when the pixel value of the two-dimensional data after scaling decreases, the output value also decreases, which means that the captured image is used. In FIG. 15(a), the thresholds are set to 0.2 and 0.3, and when it is 0.2 or less, it is the captured image, and when it is 0.3 or more, it is the estimated image, and the weight changes linearly between them.

[0077] In this embodiment, two thresholds are set so that each section changes linearly, but a nonlinear function such as a sigmoid function may be used as the response function without setting a threshold. The two thresholds are not limited to these, and other values ​​may be used. If a higher priority is given to blur correction effect, the threshold can be set smaller, and if a priority is given to suppression of adverse effects, the threshold can be set larger.

[0078] FIG. 16(a) is a schematic diagram of the first gain map calculated using the captured image of FIG. 17(b). FIG. 17(a)-(c) are explanatory diagrams of the effects and disadvantages of the blur correction process, and are similar to FIG. 7(a)-(c), respectively. In FIG. 16(a), the first gain map is also 1 for the areas of subjects B and C in FIG. 17(b), and it can be seen that the estimated image is adopted in this area. On the other hand, for the low-luminance subject A, the first gain map is 0, and it can be seen that the captured image is adopted. Thus, in step S214, the first gain map as shown in FIG. 16(a) is generated.

[0079] When the correction amount in the correction information regarding brightness is positive (in the direction of making the image brighter), the first gain map obtained above is used as a weight for weighted addition. Alternatively, the first gain map generated based on the luminance value of the captured image may be adjusted based on the correction amount of the correction information regarding brightness. That is, the weight of the captured image in the first gain map increases as the absolute value of the correction amount of the correction information regarding brightness increases. In this case, the first gain map may be adjusted by multiplying the first gain map by an adjustment value obtained from the correction amount of the correction information regarding brightness. For example, the adjustment value of the gain map as shown in FIG. 18 is used. FIG. 18 is an explanatory diagram of the adjustment value of the gain map. In FIG. 18, the horizontal axis indicates the correction amount regarding brightness, and the vertical axis indicates the adjustment value of the gain map.

[0080] However, this embodiment is not limited to this, and various modifications are possible. In FIG. 18, the vertical axis of the gain map adjustment value means that the gain map is used as is when it is 1, and that the gain map is not used when it is 0. In other words, the gain map adjustment value is a parameter such that the weight of the captured image in the first gain map increases as the absolute value of the correction amount in the correction information regarding brightness increases. The first gain map after adjustment can be calculated by (first gain map-1) x the gain map adjustment value based on the correction amount regarding brightness + 1. Alternatively, the first response function in FIG. 15(a) may be adjusted in the same manner based on the correction information regarding brightness, and the first gain map may be generated based on the adjusted first response function.

[0081] Furthermore, when the absolute value of the correction amount of the correction information related to brightness is equal to or greater than a threshold value, the first gain map may be used, and when the absolute value is smaller than the threshold value, the first gain map may be set to 0. (Akbe) Next, an example will be described in which, when the correction amount in the correction information regarding brightness is negative (in the direction of darkening), the weight of the captured image in weighted addition is increased as the luminance value in the estimated image (first image) increases.

[0082] In step S214 of FIG. 11, the correction unit 323c generates a second gain map. The second gain map is a weight map used to suppress the amount of adverse effects occurring in high brightness areas. Here, the second gain map represents the weighting applied to the estimated image. In other words, the larger the second gain map is, the heavier the weighting of the estimated image is than that of the captured image when performing weighted addition. The second gain map is generated based on the pixel values ​​of the estimated image. Although the part where the estimated image is used instead of the captured image is different from the calculation of the first gain map described above, the demosaicing process, white balance correction, and normalization process are the same, and therefore description will be omitted.

[0083] FIG. 15(b) shows a second response function for obtaining a second gain map. The horizontal axis of FIG. 15(b) is the two-dimensional data after scaling generated based on the pixel values ​​of the estimated image as input, and the vertical axis is the weight used when performing weighted addition of the captured image and the estimated image. In this embodiment, the weight on the vertical axis of FIG. 15(b) is the weight applied to the estimated image. Therefore, in FIG. 15(b), when the pixel value of the two-dimensional data after scaling increases, the output value decreases, which means that the captured image is adopted. On the other hand, when the pixel value of the two-dimensional data after scaling decreases, the output value increases, which means that the estimated image is adopted. In FIG. 15(b), the thresholds are set to 0.8 and 0.9, and when it is 0.8 or less, it is an estimated image, and when it is 0.9 or more, it is a captured image, and the weight changes linearly between them.

[0084] In this embodiment, two thresholds are set so that each section changes linearly, but a nonlinear function such as a sigmoid function may be used as the response function without setting a threshold. The two thresholds are not limited to these, and other values ​​may be used. If a greater priority is given to blur correction effect, the threshold can be set to a larger value, while if priority is given to suppression of adverse effects, the threshold can be set to a smaller value.

[0085] Fig. 16(b) is a simulation diagram of the second gain map calculated using the estimated image of Fig. 17(c). In Fig. 16(b), the second gain map is also 1 for the areas of subjects A and B in Fig. 17(c), and it is understood that the estimated image is adopted for this area. On the other hand, for the high-luminance subject C, the second gain map is 0, and it is understood that the captured image is adopted. Thus, in step S214, the second gain map as shown in Fig. 16(b) is generated.

[0086] When the correction amount in the correction information regarding brightness is negative (in the direction of darkening), the second gain map obtained above is used as a weight for weighted addition. Alternatively, the second gain map generated based on the luminance value of the estimated image may be adjusted based on the correction amount of the correction information regarding brightness. That is, the weight of the captured image in the second gain map increases as the absolute value of the correction amount of the correction information regarding brightness increases. In this case, the second gain map may be adjusted by multiplying the second gain map by an adjustment value obtained from the correction amount of the correction information regarding brightness. For example, the adjustment value of the gain map as shown in FIG. 18 described above is used.

[0087] The adjustment value of the gain map is a parameter such that the weight of the captured image in the second gain map increases as the absolute value of the correction amount in the correction information regarding brightness increases. The adjusted second gain map can be calculated by (second gain map-1) x adjustment value of the gain map based on the correction amount regarding brightness + 1. Alternatively, the second response function in FIG. 15(b) may be similarly adjusted based on the correction information regarding brightness, and the second gain map may be generated based on the adjusted second response function.

[0088] Furthermore, if the absolute value of the correction amount of the correction information related to brightness is equal to or greater than a threshold value, the second gain map may be used, and if it is smaller than the threshold value, the second gain map may be set to zero.

[0089] In the above, the gain map to be acquired is changed depending on whether the correction amount in the correction information related to the brightness is positive or negative. However, this embodiment is not limited to this, and a first gain map and a second gain map may be generated in advance, and a gain map that is a weight used during weighted addition may be selected based on the correction information related to the brightness. Also, the first gain map and the second gain map may be adjusted according to the correction amount of the correction information related to the brightness, and the first gain map and the second gain map after adjustment may be synthesized. When synthesizing the first gain map and the second gain map, it is preferable to compare the two gain maps and adopt the value of the one with the larger weight of the captured image. Also, a gain map after synthesis may be generated by multiplying or averaging the weights or gain maps of the two weighted additions.

[0090] As a modified example, weighted addition may be performed on pixels whose luminance values ​​of the captured image or estimated image are outside a threshold range (such as a range where black sunkenness or white floating is likely to occur) and whose luminance values ​​fall within the threshold range when the captured image or estimated image is subjected to a correction for brightness (such as exposure correction). As described above, when estimation is performed using a machine learning model, problems occur in high-luminance and low-luminance areas, and are emphasized when a correction for brightness is performed. Therefore, by suppressing the blur correction strength of pixels that fall into a luminance range where problems in high-luminance and low-luminance areas are emphasized when a correction for brightness is performed, it is possible to suppress only major problems after the correction for brightness is performed.

[0091] For example, the threshold range is set to 0.2 to 0.8, and the blur correction intensity of pixels whose luminance value of the captured image or estimated image after scaling is outside the threshold range and falls within the threshold range when brightness correction is performed is suppressed. In addition, when the exposure correction is set to -10%, weighted addition is not performed on pixels whose luminance value is 1 in the captured image or estimated image, and weighted addition is performed on pixels whose luminance value is 0.85. In this case, the weight of the captured image may be increased as the luminance value of the captured image or estimated image increases. When the exposure correction is set to +10%, weighted addition is not performed on pixels whose luminance value is 0 in the captured image or estimated image, and weighted addition is performed on pixels whose luminance value is 0.15. In this case, the weight of the captured image may be increased as the luminance value of the captured image or estimated image decreases.

[0092] In addition, the weights for weighted addition may be determined based on the luminance values ​​of the captured image or estimated image and the luminance values ​​of the captured image or estimated image corrected for brightness. In other words, the gain map may be generated by multiplying a gain map that suppresses the blur correction effect for pixels that are brighter or darker in the captured image or estimated image with a gain map that suppresses the effect for pixels with intermediate luminance values ​​in the captured image or estimated image corrected for brightness.

[0093] Next, in step S215, the correction unit 323c performs weighted addition of the captured image and the estimated image (first image) based on the weight (gain map) of the weighted addition acquired in step S214. In this way, a blur-corrected image (second image) in which adverse effects are suppressed by adjusting the blur correction intensity is acquired.

[0094] FIG. 17(d) is a simulation diagram of a blur-corrected image after weighted addition of a captured image and an estimated image using the acquired gain map when the correction regarding brightness is positive (in the direction of making the image brighter). It can be seen that the darkening that occurred in the low-luminance subject A has been reduced compared to the estimated image in FIG. 17(c). FIG. 17(e) is a simulation diagram of a blur-corrected image after weighted addition of a captured image and an estimated image using the acquired gain map when the correction regarding brightness is negative (in the direction of making the image darker). It can be seen that the darkening that occurred in the high-luminance subject C has been reduced compared to the estimated image in FIG. 17(c). This makes it possible to generate a blur-corrected image with sufficiently little adverse effects such as darkening, even if the correction regarding brightness is performed. It can also be seen that the sharpening effect is comparable to that of the estimated image in FIG. 17(c).

[0095] Next, in step S216, the correction unit 323c performs various correction or conversion processes (correction processes) including development processing and correction regarding brightness on the blur-corrected image (second image) acquired in step S215 to generate an output image.

[0096] By performing the blur correction process described above, a blur-corrected image can be generated that combines suppressing adverse effects with maintaining the correction effect by weighted-adding the captured image and the estimated image using correction information regarding brightness and a gain map based on the captured image or the estimated image.

[0097] In this embodiment, it is preferable to increase the weight of the captured image as the luminance value of the captured image decreases, and to increase the weight of the captured image as the luminance value of the estimated image increases. In this embodiment, the first gain map is calculated based on the luminance value of the captured image. This is because low-luminance areas are highly susceptible to noise and have high gain due to subsequent gamma correction, so sensitivity is high. Therefore, in such areas, it is preferable to use the captured image before sharpening as a reference. For this reason, in this embodiment, the captured image is used when calculating the first gain map.

[0098] On the other hand, the second gain map is calculated based on the estimated image. This is because an estimated image is more desirable than a captured image as a reference image in order to maintain the effect of aberration correction in the brightness saturation area and its surrounding area. If a gain map is created using a response function such as that shown in FIG. 15(b) using a captured image, most of the aberration components in the brightness saturation surrounding area that have been corrected by the estimated image will be replaced by the captured image. For this reason, in this embodiment, an estimated image is used when calculating the second gain map. As described above, the first gain map is generated using a captured image, and the second gain map is generated using an estimated image, thereby achieving both maintenance of the correction effect and suppression of adverse effects in this embodiment.

[0099] In addition, in Fig. 15(a) and (b), the threshold value at which the output of the first response function is 0.5, that is, the weight of the captured image and the estimated image is half and half, is 0.25, and the threshold value at which the output of the second response function is 0.5 is 0.85. In this embodiment, the first gain map is a weight for suppressing the adverse effects of low brightness areas, and the second gain map is a weight for suppressing the adverse effects of high brightness areas. Therefore, when considering the weight applied to the estimated image as in Fig. 15(a) and (b), the first threshold value is smaller than the second threshold value. However, the first threshold value and the second threshold value are not limited to these numerical values, and such a magnitude relationship is also premised on the fact that the reference value when scaling is the same.

[0100] Also in this embodiment, the weight of the weighted average may be obtained by adding information other than the correction information on brightness and the luminance value of the captured image or the estimated image. For example, the weight of the weighted average or the gain map may be generated based on the performance of the optical system at the time of capturing an image, and may be combined with the weight of the weighted average based on the correction information on brightness and the luminance value of the captured image or the estimated image. EXAMPLES

[0101] Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment differs from the first and second embodiments in that it includes a processing device (computer) that transmits a captured image to be subjected to image processing to an image estimation device and receives a processed output image (blur-corrected image) from the image estimation device.

[0102] FIG. 19 is a block diagram of an image processing system 600 in this embodiment. The image processing system 600 includes a learning device 601, an imaging device 602, an image estimation device (second device) 603, and a processing device (computer, first device) 604. The learning device 601 and the image estimation device 603 are, for example, servers. The computer 604 is, for example, a user terminal (personal computer or smartphone). The computer 604 is connected to the image estimation device 603 via a network 605. The image estimation device 603 is connected to the learning device 601 via a network 606. That is, the computer 604 and the image estimation device 603 are configured to be able to communicate with each other, and the image estimation device 603 and the learning device 601 are configured to be able to communicate with each other.

[0103] The configuration of the learning device 601 is similar to that of the learning device 101 in the first embodiment, and therefore a description thereof will be omitted. The configuration of the imaging device 602 is similar to that of the imaging device 102 in the first embodiment, and therefore a description thereof will be omitted.

[0104] The image estimation device 603 has a storage unit 603a, an acquisition unit 603b, a correction unit 603c, and a communication unit (reception means) 603e. The storage unit 603a, the acquisition unit 603b, and the correction unit 603c are similar to the storage unit 103a, the acquisition unit 103b, and the correction unit 103c of the image estimation device 103 of the embodiment 1, respectively. The communication unit 603d has a function of receiving a request transmitted from the computer 604, and a function of transmitting an output image generated by the image estimation device 603 to the computer 604. The computer 604 has a communication unit (transmission means) 604a, a display unit 604b, an image processing unit 604c, and a recording unit 604d. The communication unit 604a has a function of transmitting a request to the image estimation device 603 to cause the image estimation device 603 to execute processing on the captured image, and a function of receiving an output image processed by the image estimation device 603. The display unit 604b has a function of displaying various information. The information displayed by the display unit 604b includes, for example, the captured image to be transmitted to the image estimation device 603 and the output image received from the image estimation device 603. The image processing unit 604c has a function of further performing image processing on the output image received from the image estimation device 603.

[0105] The recording unit 604d records the captured image acquired from the imaging device 602, the output image received from the image estimation device 603, and the like.

[0106] Next, image processing in this embodiment will be described with reference to Fig. 20. Image processing in this embodiment includes processing equivalent to the blur correction processing (Fig. 1) described in embodiment 1. Fig. 20 is a flowchart showing image processing in this embodiment. The image processing shown in Fig. 20 is started when an instruction to start image processing is given by a user via computer 604. First, the operation of computer 604 will be described.

[0107] First, in step S701, the computer 604 transmits a request for processing a captured image to the image estimation device 603. Note that the method of transmitting the captured image to be processed to the image estimation device 603 is not limited. For example, the captured image may be uploaded to the image estimation device 603 simultaneously with step S701, or may be uploaded to the image estimation device 603 before step S701. Furthermore, the captured image may be an image stored on a server different from the image estimation device 603. Note that in step S701, the computer 604 may transmit ID information for authenticating a user together with the request for processing the captured image.

[0108] In step S702, the computer 604 receives the output image generated in the image estimation device 603. The output image is a blur-corrected image obtained by weighted addition of the captured image and the estimated image, similar to the first embodiment.

[0109] Next, a description will be given of the operation of the image estimation device 603. First, in step S801, the image estimation device 603 receives a request for processing a captured image transmitted from the computer 604. The image estimation device 603 determines that a blur correction process for the captured image has been instructed, and executes the processes from step S802 onward.

[0110] Next, in step S802, the image estimation device 603 acquires the captured image and weight information. The weight information is information (trained model) learned by the same method (FIG. 4) as in the first embodiment. The image estimation device 603 may acquire the weight information from the learning device 601, or may acquire weight information previously acquired from the learning device 601 and stored in the storage unit 603a. The following steps S803 to S807 are similar to steps S112 to S116 in the first embodiment, respectively.

[0111] Then, in step S805, the image estimation device 603 transmits the output image to the computer 604.

[0112] Although this embodiment has been described as performing the blur correction process of the first embodiment, it can be similarly applied to the blur correction process of the second embodiment. Moreover, it is possible to similarly implement the invention and obtain the effects with respect to tasks other than the blur correction process. When the correction process is performed within the image estimation device 603 as in this embodiment, the processing load due to the blur correction process can be borne within the image estimation device 603, and therefore the processing capacity required for the computer 604 can be reduced.

[0113] As described above, as in this embodiment, the image estimation device 603 may be configured to be controlled by using the computer 604 communicably connected to the image estimation device 603 . (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0114] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that can suppress adverse effects that occur when performing brightness correction on an image that has been corrected using a machine learning model.

[0115] The disclosure of each embodiment includes the following configurations and methods. (Method 1) generating a first image by inputting the captured image into a machine learning model; acquiring correction information relating to brightness when performing development processing or correction processing on the second image; acquiring the second image by performing weighted addition of the captured image and the first image based on the correction information; An image processing method, wherein the correction information is constant with respect to a position in an image or varies depending on the position in the image. (Method 2) The image processing method according to Method 1, further comprising a step of performing the development process or the correction process on the second image to obtain an output image. (Method 3) 3. The image processing method according to method 1 or 2, wherein the correction information does not depend on the luminance value of the image. (Method 4) The image processing method according to any one of Methods 1 to 3, characterized in that the larger the absolute value of the correction amount in the correction information, the larger the weight of the captured image in the weighted addition. (Method 5) The image processing method according to method 1 or 2, characterized in that weights in the weighted addition are determined based on luminance values ​​in the captured image or the first image and the correction information. (Method 6) When the correction in the correction information is in a direction to brighten the second image, the smaller the luminance value in the captured image or the first image, the larger the weight of the captured image in the weighted addition is, The image processing method according to method 5, characterized in that, when the correction in the correction information is in a direction to darken the second image, the weight of the captured image is increased as the luminance value increases. (Method 7) 7. The image processing method according to method 5 or 6, wherein the smaller the luminance value of the captured image, the greater the weight of the captured image in the weighted addition. (Method 8) 7. The image processing method according to method 5 or 6, wherein the weight of the captured image in the weighted addition is increased as the luminance value in the first image is increased. (Method 9) The image processing method described in any one of Methods 5 to 8, characterized in that the weighted addition is performed on pixels whose luminance values ​​in the captured image or the first image are outside a threshold range and that have luminance values ​​that fall within the threshold range when the captured image or the first image is corrected for brightness. (Method 10) 10. The image processing method according to any one of Methods 1 to 9, wherein the correction information is correction information relating to at least one of exposure correction, peripheral light amount correction, and white balance correction. (Method 11) 11. The image processing method according to any one of Methods 1 to 10, wherein the first image is an image obtained by sharpening the captured image. (Configuration 1) A first means for generating a first image by inputting a captured image into a machine learning model; A second means for acquiring correction information related to brightness when performing a development process or a correction process on the second image; a third means for acquiring a second image by performing weighted addition of the captured image and the first image based on the correction information; The image processing device according to claim 1, wherein the correction information is constant with respect to a position in an image or depends on a position in the image. (Configuration 2) An image processing system including the image processing device according to configuration 1 and a control device capable of communicating with the image processing device, The control device has a means for transmitting a request for executing a process on the image, 2. An image processing system according to claim 1, wherein the image processing device comprises means for executing processing on the image in response to the request. (Configuration 3) A program for causing a computer to execute the image processing method according to any one of Methods 1 to 11.

[0116] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0117] 103 Image Estimation Device (Image Processing Device) 103b Acquisition unit (first means, second means) 103c Correction unit (third means)

Claims

1. generating a first image by inputting the captured image into a machine learning model; acquiring correction information relating to brightness when performing development processing or correction processing on the second image; acquiring the second image by performing weighted addition of the captured image and the first image based on the correction information; An image processing method, wherein the correction information is constant with respect to a position in an image or varies depending on the position in the image.

2. 2. The image processing method according to claim 1, further comprising the step of performing the development process or the correction process on the second image to obtain an output image.

3. 2. The image processing method according to claim 1, wherein the correction information does not depend on a luminance value of the image.

4. 2. The image processing method according to claim 1, wherein the weight of the captured image in the weighted addition is increased as the absolute value of the correction amount in the correction information increases.

5. 2. The image processing method according to claim 1, wherein weights in the weighted addition are determined based on luminance values ​​in the captured image or the first image and the correction information.

6. When the correction in the correction information is in a direction to brighten the second image, the weight of the captured image in the weighted addition is increased as the luminance value in the captured image or the first image is smaller; 6. The image processing method according to claim 5, wherein when the correction in the correction information is in a direction to darken the second image, the weight of the captured image is increased as the luminance value increases.

7. 6. The image processing method according to claim 5, wherein the weight of the captured image in the weighted addition is increased as the luminance value of the captured image is smaller.

8. 6. The image processing method according to claim 5, wherein the weight of the captured image in the weighted addition is increased as the luminance value in the first image is increased.

9. The image processing method according to claim 5, characterized in that the weighted addition is performed on pixels whose luminance values ​​in the captured image or the first image are outside a threshold range and whose luminance values ​​fall within the threshold range when the captured image or the first image is corrected for brightness.

10. 2. The image processing method according to claim 1, wherein the correction information is correction information relating to at least one of exposure correction, peripheral light amount correction, and white balance correction.

11. 2. The image processing method according to claim 1, wherein the first image is an image obtained by sharpening the captured image.

12. A first means for generating a first image by inputting a captured image to a machine learning model; A second means for acquiring correction information related to brightness when performing a development process or a correction process on the second image; a third means for acquiring a second image by performing weighted addition of the captured image and the first image based on the correction information; The image processing device according to claim 1, wherein the correction information is constant with respect to a position in an image or depends on a position in the image.

13. An image processing system including the image processing device according to claim 12 and a control device capable of communicating with the image processing device, The control device has a means for transmitting a request for executing a process on the image, 2. An image processing system according to claim 1, wherein the image processing device comprises means for executing processing on the image in response to the request.

14. A program causing a computer to execute the image processing method according to any one of claims 1 to 11.

Images