Method and device for estimating guided curve for improving low illumination image

Abstract

According to a first aspect of the present invention, a method for estimating a guided curve for improving a low illumination image includes the steps of: extracting an image feature from an image input using an encoder; and calculating a guided curve including an adjustment map representing the brightness of each pixel in the image by inputting the image feature to a decoder and a guide map for assigning a weight to each pixel in the image on the basis of the brightness.

Description

Guided curve estimation method and device for improving low-light images

[0001] The present invention relates to a guided curve estimation method and device for improving low-light image quality. This research was conducted with the support of the National IT Industry Promotion Agency (NIPA) funded by the Ministry of Science and ICT (Government) (Project ID: 1711193159; Project No.: 2021-0-01364-003; Research and Development Project: SW Computing Industry Core Technology Development; Research Project Title: (SW Star Lab) Continuous Real-Time Intelligent Traffic Surveillance System on Edge Devices; Project Period: January 1, 2025 - December 31, 2025).

[0002] For reference, this application claims priority to Korean Patent Application No. 10-2024-0073935, filed on June 5, 2024. The entire contents of that application, which serves as the basis for this priority claim, are incorporated herein by reference.

[0003] Shooting video in low-light environments, especially if the light intensity is inadequate or the subject is facing away from a strong light source, can result in severe backlighting. Videos captured in these environments are not only visually unsightly but also unsuitable for high-level information processing.

[0004] Recently, various methods for improving the image quality of low-light images have been studied. However, conventional AI models for low-light image enhancement use pixel-based mapping methods, which have limitations such as high memory requirements, long inference times, unnatural image processing, and potential object or color distortion. Furthermore, model training requires pairs of training data under different illumination conditions.

[0005] The problem to be solved by the present invention is to provide a guided curve estimation method and device using a guided curve estimation deep learning model for improving low-light images.

[0006] However, the problems to be solved by the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the description below.

[0007] A guided curve estimation method for improving low-light images, performed by a guided curve estimation model including an encoder and a decoder according to a first aspect of the present invention, comprises the steps of extracting image features from an input image using the encoder, and inputting the image features to the decoder to produce a guided curve including an adjustment map expressing the brightness of each pixel in the image and a guide map assigning a weight to each pixel in the image based on the brightness.

[0008] The step of calculating the guided curve may include a step of inputting the image features into a first decoder and outputting an adjustment map expressing the current brightness of each pixel in the image, and a step of inputting the image features into a second decoder and outputting a guide map that assigns a weight to the current brightness of each pixel in the image.

[0009] The above adjustment map may include three channels: red, green and blue.

[0010] In the step of outputting the above guide map, a higher weight can be given to a pixel in the image that has a darker current brightness.

[0011] In the step of outputting the above guide map, if the current brightness of a pixel in the image is greater than or equal to a threshold value, a weight of 0 can be assigned.

[0012] The above threshold value may be determined through learning of the guided curve estimation model.

[0013] The method may further include a step of applying the guided curve to the image to output an adjusted image in which the brightness of each pixel of the image is adjusted.

[0014] The above guided curve estimation model may be pre-learned based on a loss function calculated by reflecting the difference in brightness of each pixel for the image and the adjusted image.

[0015] The above loss function may include a channel consistency loss function. Here, the channel consistency loss function is: can be determined. At this time, KL is a value that numerically represents the difference between the two distributions, is the above adjustment image, may represent the above image.

[0016] The above loss function may include an edge consistency loss function. Here, the edge consistency loss function is: can be determined. At this time, I is the input image, is the Laplace operator, Y is the above adjustment image, can be any constant.

[0017] The above loss function may include a brightness consistency loss function. Here, the brightness consistency loss function is: can be determined. At this time, is the brightness (V, value) channel of image I in HSV (hue, saturation, value) color space format, is a brightness map ( ) may be a variable that adjusts the curvature.

[0018] A guided curve estimation device for improving low-light images according to a second aspect of the present invention comprises a memory capable of storing computer-executable instructions, and a processor for performing a method including a step of extracting image features from an input image using an encoder by executing the instructions, a step of inputting the image features to a decoder to produce a guided curve including an adjustment map expressing the brightness of each pixel in the image and a guide map assigning a weight to each pixel in the image based on the brightness.

[0019] A computer-readable recording medium storing computer-executable instructions according to a third aspect of the present invention, wherein the computer-executable instructions, when executed by a processor, cause the processor to perform a method including the steps of extracting image features from an input image using an encoder, and inputting the image features to a decoder to produce a guided curve including an adjustment map expressing the brightness of each pixel in the image and a guide map assigning a weight to each pixel in the image based on the brightness.

[0020] A computer program stored in a computer-readable recording medium according to a fourth aspect of the present invention, wherein the computer program comprises instructions for causing the processor to perform a method, the method comprising the steps of extracting an image feature from an input image using an encoder, and inputting the image feature to a decoder to produce a guided curve including an adjustment map expressing the brightness of each pixel in the image and a guide map assigning a weight to each pixel in the image based on the brightness.

[0021] According to the present invention, a guided adjustment curve estimation model capable of performing low-light image enhancement while using less computing resources can be provided.

[0022] In addition, according to the present invention, in an image under lighting conditions such as extreme backlighting, a visually natural low-light image can be provided by adjusting the brightness so that the dark area is clearly visible while maintaining the bright area.

[0023] In addition, according to the present invention, a low weight is given to a bright area of ​​an image to prevent excessive saturation, and a high weight is given to a dark area to increase brightness and contrast, thereby preserving the shape and color contained in the image and improving brightness, thereby providing a visually natural low-light image with improved brightness.

[0024] In addition, the guided adjustment curve estimation model according to the present invention can be trained in a manner that processes the entire process from input to output at once with a neural network without a pipeline network.

[0025] In addition, unlike conventional artificial intelligence models for improving low-light images, the guided adjustment curve estimation model according to the present invention can be learned without pairs of learning data with different illumination conditions by using loss functions that do not require references.

[0026] In addition, the guided adjustment curve estimation model according to the present invention can reduce learning and inference time by consuming less computing resources and using fewer variables for learning.

[0027] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0028] FIG. 1 is a flowchart exemplarily showing a guided curve estimation method for improving low-light images according to a first aspect of the present invention.

[0029] FIG. 2 is a block diagram exemplarily showing a guided curve estimation device for improving low-light images according to a second aspect of the present invention.

[0030] Figure 3 is a block diagram exemplifying the function of a guided curve estimation program for improving low-light images.

[0031] FIG. 4 is an exemplary diagram showing a guided curve estimation network (GCE-Net) for low-light image improvement according to the present invention.

[0032] Figure 5 is an example diagram showing the effect of applying a guide map to an input image.

[0033] FIG. 6 is an exemplary diagram showing the structure of a guided curve inference network for improving low-light images according to the present invention.

[0034] Figure 7 is an example diagram showing the results of image enhancement experiments on various low-light image datasets for a conventional low-light image enhancement model and a low-light image enhancement model of the present invention.

[0035] Figure 8 is an exemplary diagram showing quality evaluations of original images and improved images for a conventional low-light image enhancement model and a low-light image enhancement model of the present invention.

[0036] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined solely by the scope of the claims.

[0037] When describing embodiments of the present invention, detailed descriptions of known functions or configurations will be omitted if they are deemed to unnecessarily obscure the gist of the invention. Furthermore, the terms described below are defined in light of their functions in the embodiments of the present invention and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the overall content of this specification.

[0038] The terms used in this specification will be briefly explained, and the present invention will be described in detail.

[0039] The terms used in this specification have been selected from widely used, current terms, taking into account the functions of the present invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should not be defined simply as names, but rather based on their inherent meanings and the overall content of the present invention.

[0040] When a part of a specification is said to 'include' a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0041] Also, the term 'part' used in the specification means a software or hardware component such as an FPGA or ASIC, and the 'part' performs certain functions. However, the 'part' is not limited to software or hardware. The 'part' may be configured to reside on an addressable storage medium or may be configured to play one or more processors. Thus, as an example, the 'part' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or further separated into additional components and 'parts'.

[0042] Below, with reference to the attached drawings, an embodiment of the present invention is described in detail so that a person having ordinary skill in the art to which the present invention pertains can easily practice it.

[0043] FIG. 1 is a flowchart exemplarily illustrating a guided curve estimation method for low-light image enhancement according to a first aspect of the present invention. Hereinafter, the guided curve estimation method for low-light image enhancement will be described on the assumption that it is performed by a guided curve estimation device for low-light image enhancement.

[0044] In this specification, the terms "image" and "video" are used interchangeably. Furthermore, "video" may refer to a still picture, but is not limited thereto. That is, the present invention may also be applied to moving pictures, i.e., video, depending on the embodiment.

[0045] Additionally, in this specification, brightness refers to a term in the field of image correction used in various image editing programs, and illumination refers to the amount of light or lighting conditions in a situation in which a photograph is taken.

[0046] As shown in FIG. 1, a guided curve estimation method for improving low-light images according to a first aspect of the present invention includes a step (S110) of extracting image features from an input image using an encoder, and a step (S120) of inputting the image features into a decoder to produce a guided curve including an adjustment map expressing the brightness of each pixel in the image and a guide map that assigns a weight to each pixel in the image based on the brightness.

[0047] A guided curve may include an adjustment map and a guidance map.

[0048] An adjustment map can refer to a map that represents the brightness of each pixel in an image. In other words, an adjustment map can improve an image by adjusting the dynamic range of the image based on the pixel.

[0049] A guide map can refer to a map that assigns weights to each pixel within an image based on its brightness. Guide maps can adjust the brightness of an image by considering the brightness of each region within the image. For example, by applying a guide map, a lower weight is assigned to a brighter region within the image, preventing excessive saturation. A higher weight is assigned to a darker region, increasing the brightness and contrast of dark regions. By applying a guide map to image enhancement, the shape and color contained within the image are preserved while simultaneously improving brightness, resulting in visually pleasing and natural-looking low-light images.

[0050] FIG. 2 is a block diagram exemplarily showing a guided curve estimation device for improving low-light images according to a second aspect of the present invention.

[0051] As shown in FIG. 2, a guided curve estimation device (200) for improving low-light images may include an input unit (210), an output unit (220), a processor (230), a memory (240), and a communication unit (260).

[0052] Hereinafter, for the convenience of explanation, the guided curve estimation device (200) for low-light image improvement is described as an example including an input unit (210), an output unit (220), a processor (230), a memory (240), and a communication unit (260), but is not limited thereto. That is, each unit configuration may be provided outside the guided curve estimation device (200) for low-light image improvement and may operate in a manner that interacts with the guided curve estimation device (200) for low-light image improvement.

[0053] The input unit (210) may include a user interface for receiving commands, information, etc. used to control the guided curve estimation device (200) for improving low-light images. In addition, the input unit (210) may be a hardware device (e.g., a keyboard, a mouse, a touch pad, etc.) that can directly receive commands, information, etc. used to control the guided curve estimation device (200) for improving low-light images.

[0054] In one embodiment, the input unit (210) may receive information from a user required for a guided curve estimation method for low-light image enhancement. Specifically, the user may input information including training data, an image requiring low-light image enhancement, information related to a loss function, and information related to a guided curve estimation model through the input unit (210).

[0055] The output unit (220) can provide information including learning data, images requiring low-light image improvement, information related to a loss function, information related to a guided curve estimation model, and adjusted images with adjusted brightness to a user as visual information through an interface or display device.

[0056] The processor (230) can control the overall operation of the guided curve estimation device (200) for low-light image improvement to perform the present invention.

[0057] The processor (230) can load the guided curve estimation program (250) for low-light image improvement and the information necessary for executing the guided curve estimation program (250) for low-light image improvement from the memory (240) to execute the guided curve estimation program (250) for low-light image improvement.

[0058] The processor (230) can control to store data received from an external device through the communication unit (260) in the memory (240). In addition, the processor (230) can control to transmit and receive information including learning data, images requiring low-light image improvement, information related to a loss function, information related to a guided curve estimation model, and adjusted images with adjusted brightness to and from the external device through the communication unit (260).

[0059] The processor (230) may refer to a processing device such as a microprocessor, a central processing unit (CPU), a graphic processing unit (GPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a micro controller unit (MCU), but is not limited to the above-described embodiment.

[0060] The memory (240) can store a guided curve estimation program (250) for low-light image improvement and information necessary for executing the guided curve estimation program (250) for low-light image improvement. In addition, the memory (240) can also store processing results by the processor (230).

[0061] A guided curve estimation program (250) for low-light image improvement may mean software including commands programmed to perform a method according to the present invention.

[0062] The memory (240) can store information including learning data, images requiring low-light image enhancement, information related to a loss function, information related to a guided curve estimation model, and adjusted images with adjusted brightness. Furthermore, the memory (240) can store information received from an external device via the communication unit (260).

[0063] Memory (240) may refer to a computer-readable recording medium, such as a magnetic media such as a hard disk, a floppy disk, and a magnetic tape, an optical media such as a CD-ROM, a DVD, a magneto-optical media such as a floptical disk, a random access memory such as a dynamic random access memory (DRAM) and a static random access memory (SRAM), and a hardware device specifically configured to store and execute program instructions such as a flash memory, but is not limited to the above-described embodiment.

[0064] The communication unit (260) may be a wireless communication module capable of performing wireless communication by adopting a communication method such as CDMA, GSM, W-CDMA, TD-SCDMA, WiBro, LTE, EPC, 5G, wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), UWB (Ultra Wide Band), infrared communication (IrDA; infrared data association), BLE (Bluetooth Low Energy), or NFC (Near Field Communication), but is not limited to the above-described embodiment.

[0065] In addition, information input and output through the input unit (210) and output unit (220), information stored in the memory (240), and information transmitted and received through the communication unit (260) include all information related to the present invention, and are not limited to the above-described embodiment.

[0066] The function or operation of the guided curve estimation program (250) for improving low-light images will be examined in detail with reference to FIG. 3.

[0067] Figure 3 is a block diagram exemplifying the function of a guided curve estimation program for improving low-light images.

[0068] As shown in FIG. 3, the guided curve estimation program (250) for improving low-light images may include an image feature extraction unit (310), an adjustment map output unit (320), a guide map output unit (330), an adjustment image output unit (340), and a learning unit (350). The image feature extraction unit (310), the adjustment map output unit (320), the guide map output unit (330), the adjustment image output unit (340), and the learning unit (350) are exemplary divisions of the functions of the guided curve estimation program (250) for improving low-light images, and are not limited thereto.

[0069] According to an embodiment, the functions of the image feature extraction unit (310), the adjustment map output unit (320), the guide map output unit (330), the adjustment image output unit (340), and the learning unit (350) can be merged / separated, and implemented as a series of commands included in at least one program.

[0070] The image feature extraction unit (310), the adjustment map output unit (320), the guide map output unit (330), the adjustment image output unit (340), and the learning unit (350) may be implemented by the processor (230), and may mean a data processing device built into hardware having a physically structured circuit to perform a function expressed by a code or command included in the guided curve estimation program (250) for low-light image improvement stored in the memory (240).

[0071] The image feature extraction unit (310) can extract image features from an input image using an encoder.

[0072] The guided curve output unit (not shown) may include an adjustment map output unit (320) and a guide map output unit (330).

[0073] The adjustment map output unit (320) can input image features to the first decoder and output an adjustment map that expresses the brightness of each pixel in the image.

[0074] A tuning map can contain three channels: red, green, and blue.

[0075] The guide map output unit (330) can input image features to the second decoder and output a guide map that assigns a weight to each pixel in the image based on the brightness of each pixel in the image.

[0076] The guide map output unit (330) can assign a higher weight to a pixel in the image as its current brightness becomes darker.

[0077] The guide map output unit (330) can assign a weight of 0 if the current brightness of a pixel in the image is greater than or equal to a threshold value. Here, the threshold value may be determined through learning of a guided curve estimation model by the learning unit (350).

[0078] The adjusted image output unit (340) can output an adjusted image in which the brightness of each pixel of the image is adjusted by applying a guided curve to the image.

[0079] The learning unit (350) can learn a guided curve estimation model based on a loss function calculated by reflecting the brightness difference of each pixel for the image and the adjusted image.

[0080] The loss function used for learning may include at least one of a spatial consistency loss function, an exposure control loss function, a color persistence loss function, an illuminance smoothing loss function, a channel consistency loss function, an edge consistency loss function, and a brightness consistency loss function. The loss functions are described in detail in Fig. 4.

[0081] FIG. 4 is an exemplary diagram showing a guided curve estimation network (GCE-Net) for low-light image improvement according to the present invention.

[0082] GCE-Net can adjust the brightness of an input image to brighten dark areas and preserve details in bright areas. When an image is input to the network, an adjustment curve and a guidance map can be estimated. The adjustment curve affects the dynamic range of the image, and the guidance map can drive the enhancement process, so that dark areas are enhanced more and bright areas are modified less. Since the adjustment curve and guidance map have sizes corresponding to the image, they can enable pixel-based brightness adjustment. GCE-Net can be trained directly from the input image I and the output image J using a reference-free loss function.

[0083] In GCE-Net, bright regions of the input image are considered to already have high-quality detail and color information, and thus can be determined to require no further correction. This means that in low-light images, illumination enhancement can only be performed on dark regions. Consequently, by preserving color and detail in the adjusted brightness image, GCE-Net can learn improved guided curve estimation for dark regions. GCE-Net has a network architecture that requires less computational power, exhibits fast inference speed, and can utilize a reference-free learning method.

[0084] GCE-Net can be used with an end-to-end learning method that does not require references. This means that training can be performed even if low-light images and images taken under normal lighting are not necessarily paired in the training data.

[0085] The loss function used for training GCE-Net may include at least one of a spatial consistency loss function, an exposure control loss function, a color persistence loss function, an illuminance smoothness loss function, a channel consistency loss function, an edge consistency loss function, and a brightness consistency loss function.

[0086] The total loss L of the loss function used for learning GCE-Net can be expressed as follows.

[0087]

[0088] Here, is a function that constitutes the loss, represents the weight for each loss function.

[0089] Spatial consistency loss function ( ) can preserve the differences between neighboring regions in the input image, thereby maintaining spatial consistency even in images with improved illumination. The spatial consistency loss function can be expressed as follows.

[0090]

[0091] Here, I represents the input image, J represents the improved image, K represents the number of local regions, and Ω(i) represents the eight regions (top, bottom, left, right, top left, top right, bottom left, bottom right) neighboring region i, where each region can be 4x4 in size.

[0092] Exposure control loss function ( ) can suppress underexposed / overexposed areas by adjusting the exposure level. The exposure control loss function can be expressed by calculating the absolute value of the difference between the average intensity value of the local area and the appropriate exposure level as follows.

[0093]

[0094] Here, M represents the number of non-overlapping 16x16 sized regions, Y represents the average intensity value of the local region in the improved image, and E represents the appropriate exposure level, and the appropriate exposure level E can be set to an arbitrary constant.

[0095] Color persistence loss function ( ) can reduce color degradation during the image enhancement process. The color persistence loss function can be expressed as follows.

[0096]

[0097] Here represents the average intensity value for the p channel of the improved image, and a pair of channels can be represented as (p,q).

[0098] Illumination smoothness loss function ( ) can prevent overexposure and underexposure by ensuring that the values ​​of adjacent pixels remain monotonous. The luminance smoothing loss function can be expressed as follows.

[0099]

[0100] Here, N is the total number of iterations, and represent gradient operations in the horizontal and vertical directions, respectively.

[0101] Channel consistency loss function ( ) uses Kullback-Leibler divergence to ensure that the distribution differences in R, G, B values ​​for pixels of the original image remain consistent in the improved image, and can suppress the generation of noise information and invalid features during the image enhancement process. Specifically, the original image has three channels: R, G, and B, and each pixel in the image has an integer value greater than or equal to 0 and less than or equal to 255 for each R, G, and B channel. Here, the number of pixels with values ​​from 0 to 255 in a certain channel of the original image can be generated as a graph for each of the three channels: R, G, and B. For example, if the graphs generated for the R, G, and B channels are Red_1, Green_1, and Blue_1, three distributions can be obtained by calculating (Red_1 - Green_1), (Red_1 - Blue_1), and (Green_1 - Blue_1). At this time, if the above-described process is performed on the image after enhancement, three distributions are produced, which can be expressed as (p, q). In addition, if the above-described process is performed on the image before enhancement, three distributions are produced, which can be expressed as (p', q'). Here, KL divergence can mean calculating the difference between two distributions and expressing it as a number. Since the three distributions of the image after enhancement and the image before enhancement are calculated in the same way, they can correspond to each other, and if the difference between the corresponding distributions is calculated as KL divergence, three KL divergence values ​​can be calculated. Therefore, the three KL divergence values ​​can be interpreted as the quality of the image enhancement being better as they are closer to 0, and the channel consistency loss function can be expressed as the sum of these three KL divergence values. The channel consistency loss function can be expressed as follows.

[0102]

[0103] Here, KL is a value that numerically represents the difference between two distributions, is an adjusted image, represents the input image. If and If the difference between the two distributions is small, the KL value is also small, and if the two distributions are consistent, the KL value can be 0.

[0104] Edge consistency loss function ( ) can sharpen image details by reducing gradient distortion that occurs during the image enhancement process. The edge consistency loss function can be expressed as follows.

[0105]

[0106] At this time, I is the input image, is the Laplace operator, Y is the adjustment image, is an arbitrary constant.

[0107] Brightness consistency loss function ( ) can enable measurement of the guide map G without reference. The brightness consistency loss function can be expressed as follows.

[0108]

[0109] At this time, is the brightness (V, value) channel of image I in HSV (hue, saturation, value) color space format, is a brightness map ( ) can be greater than 1 as a variable that adjusts the curvature.

[0110] To specifically explain the guide map that guides curve adjustment by giving relatively high weights to dark areas in the image, the low-light image I can be expressed by dividing it into bright areas B and dark areas D as follows.

[0111]

[0112] For the input low-light image I, the resulting image J with improved illumination can be expressed as follows.

[0113]

[0114] Here, LE() represents the illumination enhancement process. According to the present invention, the input image is divided into a D map and a B map based on the brightness values ​​of pixels using the guide map G, so mathematical expression 10 can be expressed as follows.

[0115]

[0116] In addition, when dividing the input image into a D map and a B map according to mathematical expression 3 and then applying the adjustment map A to create a guided adjustment curve, the overall illumination reconstruction process can be expressed as follows.

[0117]

[0118] Figure 5 is an example diagram showing the effect of applying a guide map to an input image.

[0119] A guided adjustment curve can increase or decrease the dynamic range of an input image based on pixel brightness. Here, the guide map can accurately display bright and dark areas. Accordingly, even after applying the adjustment map, the colors and details in bright areas, such as the sky, beach, and mountains in Figure 5, remain vivid, while dark areas, such as the trees in Figure 5, are brightened, making them stand out clearly within the image.

[0120] FIG. 6 is an exemplary diagram showing the structure of a guided curve inference network for improving low-light images according to the present invention.

[0121] The GCE-Net can be configured as a multi-task encoder-decoder network. Specifically, the GCE-Net can include an encoder (610), a first decoder (620) that outputs a steering map, and a second decoder (630) that outputs a guide map. The encoder (610) can extract common features from an input image, and the two decoders (620, 630) can estimate the steering map A and the guided enhancement map G, respectively. The network can be configured with 10 convolutional layers-ReLU and convolutional layers-Tanh, except for the last layer of each branch. In addition, through a symmetrical skip connection structure, it is possible to extract multi-scale features while maintaining high-quality details.

[0122] A guided curve composed of a control map A and a guide map G can be directly applied to a low-light image to output an improved result. Through a GCE-Net including one encoder (610) and two decoders (620, 630), the control map and the guide map can be estimated simultaneously from an input image. The encoder (610) extracts common features from the input image to estimate the control map and the guide map, and the extracted features can be connected to the input layers of the decoders (620, 630) that estimate the control map and the guide map, respectively. The number of encoders and decoders of the GCE-Net expressed in Fig. 6 is merely an example and is not limited thereto.

[0123] To reduce the number of variables used, GCE-Net can be configured with 10 convolutional layers. Each layer has 32 kernels with a size of 3x3 and a stride of 1, and a ReLU activation function can be applied after each layer. The last layer of each decoder branch (620, 630) can apply a Tanh activation function, which can determine the values ​​of the adjustment map A and the guide map G within the range of [-1, 1]. Additionally, symmetric skip connections can capture and connect multi-scale features from early layers to later layers. The output of the first decoder is the adjustment map A with three channels of R, G, and B, and the adjustment map A can be used to represent the brightness of each pixel in the image. The output of the second decoder is the guide map G with a single channel, and the guide map G can guide which areas in the image should be made brighter and how much by assigning a higher weight to a darker area and a lower weight to a brighter area. Pixel-based mapping is possible because both the adjustment map and the guide map have the same pixel size as the input image. This network structure allows GCE-Net to use only 104K learnable variables, making it suitable for use on devices with limited computing power.

[0124] Figure 7 is an example diagram showing the results of image enhancement experiments on various low-light image datasets for a conventional low-light image enhancement model and a low-light image enhancement model of the present invention.

[0125] Figure 7 shows the results of a comparative experiment with state-of-the-art methods in the field of low-light image enhancement. The low-light image enhancement methods of LIME and PIE, which are based on conventional methods; IAT, KinD, LCDPNet, LLFlow, RetinexNet, SNR, StableLLVE, URetinexNet, UTVNet, which are based on supervised learning; EnlightenGAN, SCI, which is based on GAN; and Zero-DCE, Zero-DCE++, SGZ, and RUAS, which are based on zero-shot learning, were compared with the method according to the present invention. Quantitative evaluation was performed using IAQ-PyTorch.

[0126] A qualitative comparison was performed using peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), learned perceptual image patch similarity (LPIPS), naturalness image quality evaluator (NIQE), perceptual index (PI), learnable variable P, number of flops (F), and processing speed (PS). As shown in Figure 7, GCE-Net obtained the highest NIQE and PI scores for most validation datasets.

[0127] Figure 8 is an exemplary diagram showing quality evaluations of original images and improved images for a conventional low-light image enhancement model and a low-light image enhancement model of the present invention.

[0128] Figure 8 shows the results of an evaluation that analyzes image quality by comparing the improved image with the original image. The GCE-Net of the present invention shows the second best results, and among methods utilizing zero-shot learning, it shows the best results.

[0129] As described above, according to the present invention, a guided adjustment curve estimation model capable of performing low-light image enhancement while using less computing resources can be provided.

[0130] In addition, according to the present invention, in an image under lighting conditions such as extreme backlighting, a visually natural low-light image can be provided by adjusting the brightness so that the dark area is clearly visible while maintaining the bright area.

[0131] In addition, according to the present invention, a low weight is given to a bright area of ​​an image to prevent excessive saturation, and a high weight is given to a dark area to increase brightness and contrast, thereby preserving the shape and color contained in the image and improving brightness, thereby providing a visually natural low-light image with improved brightness.

[0132] In addition, the guided adjustment curve estimation model according to the present invention can be trained in a manner that processes the entire process from input to output at once with a neural network without a pipeline network.

[0133] In addition, unlike conventional artificial intelligence models for improving low-light images, the guided adjustment curve estimation model according to the present invention can be learned without pairs of learning data with different brightness conditions by using loss functions that do not require references.

[0134] In addition, the guided adjustment curve estimation model according to the present invention can reduce learning and inference time by consuming less computing resources and using fewer variables for learning.

[0135] The embodiments of the present invention described above may be implemented through various means. For example, the embodiments of the present invention may be implemented using hardware, firmware, software, or a combination thereof.

[0136] The combination of each block of the block diagram and each step of the flowchart attached to the present invention may be performed by computer program instructions. These computer program instructions may be installed in an encoding processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, so that the instructions executed by the encoding processor of the computer or other programmable data processing equipment create a means for performing the functions described in each block of the block diagram or each step of the flowchart. These computer program instructions may also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing equipment to implement the functions in a specific manner, so that the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes an instruction means for performing the functions described in each block of the block diagram or each step of the flowchart. Since the computer program instructions can also be installed on a computer or other programmable data processing device, a series of operational steps are performed on the computer or other programmable data processing device to create a computer-executable process, and the instructions that cause the computer or other programmable data processing device to perform the steps for executing the functions described in each block of the block diagram and each step of the flowchart can also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

[0137] Additionally, each block or step may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specific logical function(s). In some embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps depicted in succession may actually be performed substantially simultaneously, or the blocks or steps may sometimes be performed in reverse order depending on the corresponding function.

[0138] The above description is merely an illustrative illustration of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential quality of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate, rather than limit, the technical idea of ​​the present invention, and the scope of the technical idea of ​​the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

Claims

1. A guided curve estimation method for improving low-light images performed by a guided curve estimation model including an encoder and a decoder, A step of extracting image features from an input image using the above encoder; and A step of inputting the image features into the decoder to produce a guided curve including an adjustment map expressing the brightness of each pixel in the image and a guide map that assigns a weight to each pixel in the image based on the brightness, A guided curve estimation method for improving low-light image quality.

2. In paragraph 1, The steps for calculating the above guided curve are: A step of inputting the above image features into a first decoder and outputting an adjustment map expressing the current brightness of each pixel in the image; and A step of inputting the image features into a second decoder and outputting a guide map that assigns weights to the current brightness of each pixel in the image, A guided curve estimation method for improving low-light image quality.

3. In paragraph 2, The above adjustment map is, Containing three channels: red, green and blue. A guided curve estimation method for improving low-light image quality.

4. In paragraph 2, In the step of printing the above guide map, The darker the current brightness of a pixel in the image above, the higher the weight given to it. A guided curve estimation method for improving low-light image quality.

5. In paragraph 2, In the step of printing the above guide map, If the current brightness of a pixel in the image above is greater than a threshold, the weight is given as 0. A guided curve estimation method for improving low-light image quality.

6. In paragraph 5, The above threshold is, It is determined through learning of the above guided curve estimation model, A guided curve estimation method for improving low-light image quality.

7. In paragraph 1, Further comprising a step of applying the guided curve to the image to output an adjusted image in which the brightness of each pixel of the image is adjusted. A guided curve estimation method for improving low-light image quality.

8. In paragraph 7, The above guided curve estimation model is, It is learned based on a loss function calculated by reflecting the difference in brightness of each pixel for the above image and the above adjusted image. A guided curve estimation method for improving low-light image quality.

9. In paragraph 8, The above loss function includes a channel consistency loss function, The above channel consistency loss function is, is decided as, is the channel consistency loss function, KL is a value that numerically represents the difference between the two distributions, is the above adjustment image, represents the above image, A guided curve estimation method for improving low-light image quality.

10. In paragraph 8, The above loss function includes an edge consistency loss function, The above edge consistency loss function is, is decided as, is the edge consistency loss function, Y is the adjusted image, is an arbitrary constant, A guided curve estimation method for improving low-light image quality.

11. In paragraph 8, The above loss function includes a brightness consistency loss function, The above brightness consistency loss function is, is decided as, is the brightness consistency loss function, is the brightness (V, value) channel of image I in HSV (hue, saturation, value) color space format, is a brightness map ( ) is a variable that adjusts the curvature of the A guided curve estimation method for improving low-light image quality.

12. Memory capable of storing computer-executable instructions; and By executing the above command, A step of extracting image features from an input image using an encoder; A processor that performs a method including a step of inputting the image features into a decoder and generating a guided curve including an adjustment map that expresses the brightness of each pixel in the image and a guide map that assigns a weight to each pixel in the image based on the brightness. Guided curve estimation device for low-light image improvement.

13. In paragraph 12, The steps for calculating the above guided curve are: A step of inputting the image features into a first decoder and outputting an adjustment map expressing the brightness of each pixel in the image; and A step of inputting the image features into a second decoder and outputting a guide map that assigns weights to the brightness of each pixel in the image, Guided curve estimation device for low-light image improvement.

14. In paragraph 13, The above adjustment map is, Containing three channels: red, green and blue. Guided curve estimation device for low-light image improvement.

15. In paragraph 13, In the step of printing the above guide map, The darker the brightness of the pixel in the image above, the higher the weight given to it. Guided curve estimation device for low-light image improvement.

16. In paragraph 13, In the step of printing the above guide map, If the brightness of a pixel in the image is greater than a threshold, the weight is given as 0. Guided curve estimation device for low-light image improvement.

17. In paragraph 16, The above threshold is, It is determined through learning of the above guided curve estimation model, Guided curve estimation device for low-light image improvement.

18. In paragraph 12, The above method, Further comprising a step of applying the guided curve to the image to output an adjusted image in which the brightness of each pixel of the image is adjusted. Guided curve estimation device for low-light image improvement.

19. In paragraph 18, The above device includes a guided curve estimation model, The above guided curve estimation model is, Based on the loss function calculated based on the brightness of each pixel of the above image and the above adjusted image, Guided curve estimation device for low-light image improvement.

20. A non-transitory computer-readable recording medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, A step of extracting image features from an input image using an encoder; and A method comprising the step of causing the processor to perform a method including a step of inputting the image features to a decoder and generating a guided curve including an adjustment map representing the brightness of each pixel in the image and a guide map assigning a weight to each pixel in the image based on the brightness. Non-transitory computer-readable recording medium.

Images