Image enhancement device, image enhancement method, and program

Abstract

An image enhancing device that, on the basis of an illumination component that indicates an amount of light that is incident to an image and a shooting object of the image, performs enhancing of the image by performing conversion of irradiance of the shooting object into intensity level of the image and correction of the irradiance in accordance with the illumination component.

Description

[Technical Field]

[0001] The present invention relates to an image enhancement device, an image enhancement method, and a program. [Background technology]

[0002] High-quality input images are essential for many computer vision tasks, including video surveillance, autonomous driving, and computational photography. However, images are often captured under unavoidable environmental and technical constraints, such as poor or uneven lighting and limited exposure times, leaving details hidden in shadows. Therefore, it is necessary to improve the visibility of such low-light images prior to downstream processing and analysis. In recent years, deep image enhancement techniques have attracted much attention to improve visibility, and they generally fall into two major categories: decomposition enhancement methods and illumination estimation-centric methods.

[0003] Decomposition enhancement methods decompose a low-light image (the input image to be enhanced) into two or more image components (illumination and reflection components, low-frequency and high-frequency layers, multi-scale band representation, etc.), improve the image quality of each image component, and finally recombine them to restore an enhanced image (the output image that is the result of image enhancement).Since deep neural networks are used for both decomposition and enhancement, these methods enable high-degree of freedom in image enhancement, but for the same reason, they are often inefficient. For example, the neural network proposed in Non-Patent Document 1 consists of a layer decomposition unit, an illumination adjustment unit, and a reflection restoration unit. The layer decomposition unit is a network for decomposing low-light images into illumination and reflection components, while the illumination adjustment unit and reflection restoration unit are networks for enhancing each component. The layer decomposition unit has a total of 10 convolutional layers, including a deconvolutional layer. This alone imposes a certain amount of computational load, but the illumination adjustment unit and reflection restoration unit further have four and ten or more convolutional layers, respectively, which require a total of enormous memory usage and computational load, making them inefficient.

[0004] On the other hand, the illumination estimation-centered method proposed in Non-Patent Document 2 and elsewhere directly considers the reflection component of an image (a low-illumination image divided by the illumination component on a pixel-by-pixel basis) as an enhanced image, and therefore requires only one deep neural network to convert a low-illumination image to an illumination component (an image component that represents the amount of light source light incident on a scene).While these methods are efficient, they make a strong implicit assumption that the reflection component of an image is the desired enhanced image, and image enhancement is restricted to a nonparametric conversion from a low-illumination image and illumination component, which may make it difficult to easily and flexibly meet the demands of various applications. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. Beyond brightening low-light images. International Journal of Computer Vision, Vol. 129, No. 4, pp. 1013-1037, 2021. [Non-patent document 2] Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In Computer Vision and Pattern Recognition (CVPR), pp. 10561-10570, 2022. [Non-patent document 3] Steve Mann. Comparative equations with practical applications in quantigraphic image processing. IEEE Transactions on Image Processing, Vol. 9, No. 8, pp. 1389-1406, 2000. [Non-patent document 4] Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafal K. Mantiuk, and Jonas Unger. HDR image reconstruction from a single exposure using deep CNNs. ACM Transactions on Graphics, Vol. 36, No. 6, pp. 178:1-178:15, 2017. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention provides efficient and flexible image enhancement. [Means for solving the problem]

[0007] One aspect of the present invention is an image enhancement device that enhances an image by converting the irradiance of an object to an intensity level of the image based on an illumination component that indicates the amount of light incident on the image and the object of the image, and correcting the irradiance using the illumination component.

[0008] One aspect of the present invention is an image enhancement method that enhances an image by converting the irradiance of an object to an intensity level of the image based on an illumination component that indicates the amount of light incident on the image and the object of the image, and correcting the irradiance using the illumination component. [Effects of the Invention]

[0009] The present invention provides efficient and flexible image enhancement. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing the configuration of an image enhancing device 10 according to the present embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an image enhancing unit 14. [Figure 3] 3 is a flowchart showing the operation of the image enhancing device 10. [Figure 4] 1 is a diagram illustrating an example of the configuration of a training device 20 according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] FIG. 1 is a diagram showing the configuration of an image enhancement device 10 according to this embodiment. The image enhancement device 10 adjusts an input image and outputs an enhanced image. The image input to the image enhancement device 10 is, for example, a low-illumination image captured in a low-illumination environment. The image enhancement device 10 includes an illumination estimation unit 12 and an image enhancement unit 14.

[0012] The illumination estimation unit 12 receives an image as input and calculates an illumination component t of the image. The illumination component t is a value that indicates the amount of light incident on the subject of the input image. The illumination component t has, for example, the same spatial resolution as the input image, and indicates the amount of light incident on each pixel corresponding to the image. The illumination estimation unit 12 may be any illumination-based neural network, such as the neural network proposed in Non-Patent Document 2. The parameters of the neural network of the illumination estimation unit 12 are, for example, parameters learned by a training device 20, which will be described later.

[0013] The image enhancing unit 14 enhances the image input to the image enhancing device 10 based on the illumination component t estimated by the illumination estimating unit 12, and generates an enhanced image. Fig. 2 is a diagram showing an example of the configuration of the image enhancing unit 14. The image enhancing unit 14 includes an irradiance estimating unit 140, an exposure correcting unit 142, an enhanced image generating unit 144, and a storage unit 150. The storage unit 150 stores functions and parameters used by the irradiance estimating unit 140, the exposure correcting unit 142, and the enhanced image generating unit 144.

[0014] The irradiance estimation unit 140 estimates the irradiance from the image input to the image enhancing device 10. The irradiance estimation unit 140 estimates the irradiance from the image input to the image enhancing device 10, for example, using a camera response function. The irradiance of the subject of the image is converted into the intensity level of the image by the camera response function when the image is captured by the camera. The camera response function is usually unknown, but Non-Patent Documents 3 and 4 propose several functions as camera response functions. The irradiance estimation unit 140 estimates the irradiance from the intensity level of the image using an inverse function of the camera response function.

[0015] Below are shown three camera response functions proposed in Non-Patent Documents 3 and 4. The first camera response function is expressed by equation (1).

number

number

[0016] where x is the intensity level of the image. The second camera response function is given by equation (3).

number

number

[0017] In equation (4), x is the intensity level of the image. The third camera response function is given by equation (5).

number

number

[0018] The irradiance estimation unit 140 converts the intensity level x of the image into irradiance q using the inverse function of the camera response function (for example, any one of equations (2), (4), and (6)).

[0019] The exposure correction unit 142 corrects the irradiance q by multiplying the irradiance q for each pixel by an exposure correction ratio k defined for each pixel. The corrected irradiance is represented as kq.

[0020] k is defined as a positive real matrix inversely proportional to the illumination component t. k is expressed using t as in equation (7).

number

[0021] The enhanced image generation unit 144 generates an enhanced image based on the corrected irradiance kq. The enhanced image generation unit 144 substitutes the corrected irradiance kq into a camera response function (e.g., any one of Equation (1), Equation (3), and Equation (5)) to generate an enhanced image y=f(kq). The enhanced image generation unit 144 uses a camera response function corresponding to the inverse function of the camera response function used by the irradiance estimation unit 140. That is, when the irradiance estimation unit 140 calculates the irradiance q using Equation (2), the enhanced image generation unit 144 generates an enhanced image using Equation (1); when the irradiance estimation unit 140 calculates the irradiance q using Equation (4), the enhanced image generation unit 144 generates an enhanced image using Equation (3); and when the irradiance estimation unit 140 calculates the irradiance q using Equation (6), the enhanced image generation unit 144 generates an enhanced image using Equation (5).

[0022] 3 is a flowchart showing the operation of the image enhancing device 10. The illumination estimation unit 12 estimates the illumination component t from the image (step S10). The irradiance estimation unit 140 estimates the irradiance q from the image (step S12). Steps S10 and S12 may be performed in parallel, or either step may be performed first. Thereafter, the exposure correction unit 142 corrects the irradiance (step S14). Thereafter, the image enhancing unit 14 generates an enhanced image based on the corrected irradiance (step S16).

[0023] The estimation by the illumination estimation unit 12 and the image enhancement by the image enhancement unit 14, such as the parameters of the neural network in the illumination estimation unit 12 and the parameters of the camera response function in the image enhancement unit 14 (a, b, c in equations (1) to (4) and a, b in equations (5) to (6)), are learned and updated by the training device 20. 4 is a diagram showing an example of the configuration of a training device 20 according to this embodiment. The training device 20 includes an illumination estimation training unit 22, an image enhancement training unit 24, and an output unit . The illumination estimation training unit 22 trains the parameters in the illumination estimation unit 12 based on the image and the illumination component estimated from the image. The illumination estimation training unit 22 trains the parameters of the neural network in the illumination estimation unit 12 by, for example, a gradient method.

[0024] The following is an example of a parameter training method performed by the illumination estimation training unit 22. The lighting estimation training unit 22 calculates the loss function L by using the formula (8). (t) Define

number

[0025] In equation (8), L sm is the smoothness loss, L f is the fidelity loss, ω f is the weighting coefficient of the fidelity loss. sm is introduced to enforce that the illumination component t is spatially smooth while preserving the edges (spatial structure) of the image x. sm is expressed by equation (9).

number

[0026] In equation (9), n is the number of pixels in image x, and i and j are pixel indices. N(i) is the set of indices of neighboring pixels of the i-th pixel, for example, the set of indices of pixels in a 5x5 window centered on the i-th pixel. t(i) is a vector containing the RGB values ​​of the i-th pixel in illumination component t. ||·||1 represents the L1 norm. ω i,j is a weight, which is calculated, for example, by equation (10).

number

[0027] In equation (10), ||||2 represents the L2 norm, σ is the standard deviation of the Gaussian kernel, and x(i) is a vector containing the YUV values ​​of the ith pixel in image x.

[0028] Fidelity loss L f is introduced to ensure pixel-wise consistency between the image x and the illumination component t. f is expressed by equation (11).

number

[0029] In equation (11), n ​​represents the number of pixels in image x, and ||·||2 represents the L2 norm.

[0030] The illumination estimation training unit 22 calculates a gradient for each parameter of the neural network of the illumination estimation unit 12 based on the value of the loss function calculated by equation (8). The illumination estimation training unit 22 updates each parameter of the neural network of the illumination estimation unit 12 based on the calculated gradient.

[0031] The image enhancement training unit 24 trains parameters in the illumination estimation unit 12 and the image enhancement unit 14 based on the image, the illumination components, and the enhanced image. Similar to the illumination estimation training unit 22, the image enhancement training unit 24 trains parameters of the camera response function using a deep learning method such as stochastic gradient descent. The following is an example of a method for training parameters by the image enhancement training unit 24. The image enhancement training unit 24 calculates the loss function L by using the formula (12). (y) Define

number

[0032] In equation (12), L e is the exposure control loss, L sp is the spatial consistency loss, L c is the color constancy loss. ω cis the weighting coefficient of the color constancy loss. e is introduced to mitigate underexposure and overexposure. Exposure control loss L e is expressed by equation (13).

number

[0033] In equation (13), ε is the desired exposure level, n is the number of pixels in image x, and ||||2 is the L2 norm. Note that two-dimensional average pooling (e.g., kernel size 16 × 16) may be applied to the enhanced image y.

[0034] Spatial consistency loss L sp is introduced to promote spatial consistency between image x and enhanced image y with respect to edges and spatial structures. The spatial consistency loss L sp is expressed by equation (14).

number

[0035] ∇ denotes the first derivative. Two-dimensional average pooling (e.g., kernel size 4×4) may be applied to both image x and enhanced image y.

[0036] Color constancy loss L c is introduced to counteract the potential color shift in the enhanced image y. c The Gray World Assumption in color constancy is based on the assumption that in a color-balanced photograph, the average of all colors is neutral gray. Color constancy loss L c is expressed by equation (15).

number

[0037] In equation (15), the overlined y irepresents the average intensity of the i channel of the enhanced image y. (i, j) are (R, G), (R, B), (G, B), and (R, G, B) are the three channels in a color image.

[0038] The image enhancement training unit 24 calculates gradients for the parameters of the neural network of the illumination estimation unit 12 and the parameters of the camera response function of the image enhancement unit 14 based on the value of the loss function calculated by equation (12). The image enhancement training unit 24 updates the parameters of the neural network and the parameters of the camera response function based on the calculated gradients.

[0039] The output unit 26 outputs the parameters updated by the illumination estimation training unit 22 and the image enhancement training unit 24. The output parameters are stored in the image enhancement device 10 and are used to estimate the illumination component and enhance the image.

[0040] From the above, in the image enhancing device 10 according to this embodiment, the image enhancing unit 14 enhances an image from the image x and the illumination component t using a camera response function. For example, in the illumination estimation center method described in Non-Patent Document 2, an enhanced image y is calculated by equation (16) using the image x and the illumination component t.

number

[0041] Since Equation (16) is a non-parametric transformation, it may not be possible to easily and flexibly meet the needs of various applications. In contrast, the method of this embodiment converts an image into an enhanced image using a parametric transformation, which allows for more flexible image enhancement.

[0042] Furthermore, in the method of this embodiment, the illumination estimation unit 12 uses a deep neural network, but the image enhancement unit 14 does not use a neural network. Therefore, compared to the method described in Non-Patent Document 1, which also uses a neural network for image enhancement, it is possible to reduce the amount of memory usage and calculations.

[0043] At least some of the functions of the image enhancement device 10 and the training device 20 may be implemented by a computer. In this case, a program for implementing the functions of the image enhancement device 10 and the training device 20 may be recorded on a computer-readable recording medium, and the program may be read and executed by a computer system. The computer system includes hardware such as a processor, an OS, and peripheral devices. The programs of the image enhancement device 10 and the training device 20 may be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include portable media such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and storage devices such as a hard disk built into a computer system. The programs of the image enhancement device 10 and the training device 20 may be transmitted via a telecommunications line.

[0044] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0045] 10 image enhancement device, 12 illumination estimation unit, 14 image enhancement unit, 140 irradiance estimation unit, 142 exposure correction unit, 144 enhanced image generation unit, 150 memory unit, 20 training device, 22 illumination estimation training unit, 24 image enhancement training unit, 26 output unit

Claims

1. enhancing the image by converting the irradiance of an object into an intensity level of the image based on an illumination component that indicates the amount of light incident on the image and the object of the image, and correcting the irradiance by the illumination component; Image enhancement device.

2. estimating the irradiance of the object from the image using the image and an inverse function of a function representing the transformation; correcting the irradiance based on the illumination component; generating an enhanced image from the corrected irradiance using the corrected irradiance and the function; The image enhancement device of claim 1 .

3. The parameters of the function are determined by a deep learning method.

3. The image enhancement device of claim 2.

4. estimating the illumination component based on the image using a neural network; The image enhancement device of claim 1 .

5. enhancing the image by converting the irradiance of an object into an intensity level of the image based on an illumination component that indicates the amount of light incident on the image and the object of the image, and correcting the irradiance by the illumination component; Image enhancement methods.

6. 6. A method for enhancing an image according to claim 5, program.

Images