Deep learning-based system and method for augmenting various low-light image data, capable of adjusting brightness

Abstract

The present invention relates to a technology for performing low-light image data augmentation by converting a bright image such as a daytime situation into a low-light image such as a nighttime situation, and a low-light image data augmentation system according to an embodiment includes: a training data configuration unit for configuring training data on the basis of a first image generated in a first environment classified on the basis of illuminance and a second image generated in a second environment in low illumination compared to the first environment; an input image conversion unit for converting the first image into the second image and the second image into the first image, by receiving the first image and the second image as inputs; and a calculation unit for calculating a total loss for training a model by comparing an input image with at least one of the converted first image or the converted second image, wherein the input image conversion unit can convert the first image in the first environment into the second image in the second environment on the basis of a brightness degree (b) and a weight (w), through the model trained using the total loss.

Description

A variety of low-light image enhancement systems and methods capable of deep learning-based brightness control

[0001] The present invention relates to a system for performing low-light image enhancement by setting the brightness of a result image based on a user's input brightness level, reducing noise using a self-attention method, and converting a bright image, such as a daytime situation, into a low-light image, such as a nighttime situation, based on a CycleGAN structure deep learning network.

[0002] Most datasets used to train image deep learning models applied to autonomous vehicle recognition are comprised of bright images from daytime environments, resulting in a lack of low-light data from nighttime environments. This results in lower recognition performance in low-light images compared to daytime images.

[0003] Low-light image data augmentation is an image generation method that addresses performance degradation in certain situations due to insufficient existing data sets.

[0004] Most image generation models, like GANs, consist of an encoder and decoder. The encoder extracts features from the input image, and the decoder generates the resulting image using the features obtained from the encoder.

[0005] In particular, the CycleGAN[]-based generative model improves image transformation performance by introducing a generator and discriminator, while also allowing relatively free constraints on input and output image pairs, making it suitable for use in a variety of fields and situations. However, low-light image enhancement using CycleGAN and other image generation models generates significant noise, and the brightness of the resulting image is arbitrarily set.

[0006] Furthermore, Transformer[], the most powerful natural language processing technology, utilizes self-attention to deliver superior results compared to existing methods. Recently, self-attention has been used not only in natural language processing but also in convolutional neural networks (CNNs), through methods such as non-local attention[] and channel attention[], leading to excellent performance in image classification and detection.

[0007] The purpose of the present invention is to reduce noise in a result image by using a self-attention method when a low-light image enhancement system is operated, and to obtain diverse and accurate low-light images by setting the brightness of the result image according to a user's input brightness level, thereby resolving the data shortage phenomenon.

[0008] A low-light image augmentation system according to an embodiment includes a training data configuration unit that configures training data based on a first image generated in a first environment distinguished based on illuminance, and a second image generated in a second environment that is low-illuminance compared to the first environment, an input image conversion unit that receives the first image and the second image as inputs and converts the first image into a second image and the second image into the first image, and a calculation unit that compares the input image with at least one of the converted first image and the converted second image to calculate a total loss (Loss) for training a model, wherein the input image conversion unit can convert the first image of the first environment into a second image of the second environment based on a brightness level (b) and a weight (w) through a model trained by the total loss.

[0009] A low-light image enhancement system according to one embodiment may further include an image determination unit that determines whether each of the converted images is a first image or a second image.

[0010] A low-light image enhancement system according to one embodiment may further include a self-attention unit that performs noise reduction by separating the background of the image for the converted first image or second image.

[0011] The low-light image enhancement system according to one embodiment may further include a brightness setting unit that sets the brightness level of the resulting image when calculating the total loss.

[0012] According to an embodiment, the training data configuration unit extracts N data from each of the first situation data set (x) of the first environment and the second situation data set (y) of the second environment to create a first image ( ) and the second image ( ) can be built.

[0013] The input image conversion unit according to one embodiment uses an image conversion model that adds brightness control and feature concentration functions based on the CycleGAN model to convert N first images. and N second images It takes as input, and constructor G XY Using the generator G, the first image is converted to the second image. YX Using this, the second image can be converted into the first image.

[0014] The above-described converted image determination unit according to one embodiment may perform a task of distinguishing between a first image converted from a second image and a first image in an actual environment, and may perform a task of distinguishing between a second image converted from a first image and a second image in an actual environment.

[0015] The above feature concentration part according to one embodiment is the generator G of the existing CycleGAN model. XY, G YX and discriminator D x , D y Non-local neural networks can be additionally included in the structure.

[0016] According to an embodiment, the above-described operation unit can calculate the total loss including the adversarial loss presented in CycleGAN, the cycle consistency loss, and additional brightness control loss.

[0017] The above operation unit according to one embodiment generates a generator G using the calculated total loss. XY, G YX and discriminator D x , D y You can update the parameters of .

[0018] According to an embodiment, the brightness setting unit can set the brightness level (b) and the weight (w) so that the brightness level of the resulting image converges to a specific value when calculating the brightness control loss.

[0019] A method of operating a low-light image augmentation system according to an embodiment comprises the steps of collecting data including a first image generated in a first environment classified based on illuminance and a second image generated in a second environment that is low-light in comparison with the first environment, setting an environment for training an image conversion model, receiving the collected data as input to the trained image conversion model, dividing the input data into a source domain image (X) and a target domain image (Y), and generating a generator G that converts the source domain image into the target domain image. XY Wow, a constructor G that converts target domain images to source domain images. YX The method may include a step of performing image transformation through a step of calculating a total loss based on a set loss function for training the system, and a step of generating a trained image transformation model using the calculated total loss.

[0020] The step of setting an environment for training the image transformation model according to an embodiment may include a step of setting training parameters and a step of setting a target brightness for a result image.

[0021] According to one embodiment, a bright image in a daytime environment can be converted into a low-light image in a nighttime environment, thereby performing low-light image enhancement with excellent performance and controlling the brightness level of the low-light image.

[0022] The effects of the present invention are not limited to the effects described above, and effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the attached drawings.

[0023] FIG. 1 is a drawing illustrating a variety of low-light image enhancement systems capable of deep learning-based brightness control according to one embodiment of the present invention.

[0024] Figure 2 is a diagram illustrating an image transformation model training process to reduce noise that occurs when a low-light image is augmented and to transform the resulting image so that it has a brightness level input by the user.

[0025] Figure 3 is a diagram illustrating the process of converting a bright image into a low-light image by a trained image conversion model.

[0026] Figure 4 is a drawing showing a graph showing the brightness level of a result image generated according to the brightness level set in the brightness setting section.

[0027] Figure 5 is a drawing showing a graph showing the brightness level of a result image generated according to the weight level set in the brightness setting section.

[0028] Figure 6 is a diagram showing the result image of the performance of the image transformation model that applies the CycleGAN model and the feature focus and brightness control parts.

[0029] Figure 7 is a drawing showing the result images of the image transformation model that applies existing image transformation models (CycleGAN, DRIT, TSIT, MUNIT) and feature concentration and brightness control parts to a bright input image (KITTI).

[0030] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0031] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes modifications, equivalents, or alternatives that fall within the spirit and technical scope of the present invention.

[0032] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component," without departing from the scope of the invention.

[0033] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between," "immediately between," or "directly adjacent to," should be interpreted similarly.

[0034] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0035] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0036]

[0037] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same components.

[0038] FIG. 1 is a block diagram briefly illustrating a variety of low-light image enhancement systems (100) capable of deep learning-based brightness control according to one embodiment of the present invention.

[0039] In the present invention, the first environment and the second environment are distinguished based on the level of illumination. In the following description, the first environment is a bright daytime environment, and the second environment is a nighttime environment with low illumination, as an example, in contrast to the bright daytime environment.

[0040] Additionally, the invention can be explained by interpreting the first image acquired in the first environment as a bright image generated in a daytime environment, and the invention can be explained by interpreting the second image acquired in the second environment as a low-light image generated in a nighttime environment.

[0041] A low-light image enhancement system (100) according to one embodiment performs low-light image enhancement with excellent performance by converting a bright image in a daytime environment into a low-light image in a nighttime environment and can adjust the brightness level of the low-light image.

[0042] To this end, a low-light image enhancement system (100) according to an embodiment may include a training data configuration unit (110), an input image conversion unit (120), a calculation unit (130), a conversion image determination unit (140), a feature concentration unit (150), and a brightness setting unit (160).

[0043] The training data configuration unit (110) can configure training data based on bright images generated in a daytime environment classified based on illuminance, and low-illuminance images generated in a nighttime environment with low illuminance compared to the daytime environment.

[0044] The training data configuration unit (110) extracts N data from each of the first situation data set (X) of the daytime environment and the second situation data set (Y) of the nighttime environment to create a bright image. Wow low light image Build.

[0045] In one example, bright and low-light images are randomly extracted from each context data set.

[0046] The input image conversion unit (120) can receive a bright image and a low-light image as input and convert the bright image into a low-light image and the low-light image into a bright image.

[0047] The calculation unit (130) can compare the input image with at least one of the converted bright image or the converted low-light image to calculate the total loss for training the model.

[0048] For example, the operation unit (130) can calculate the total loss including the adversarial loss presented in CycleGAN, the cycle consistency loss, and additional brightness control loss.

[0049] The operation unit (130) uses the calculated total loss to generate the generator G XY ,G YX and discriminator D X ,D Y You can update parameters.

[0050] For example, the computation unit (130) may include adversarial loss, cycle consistency loss, and brightness control loss, and the generator G may be generated using the calculated loss. XY ,G YX and discriminator D X ,D Y Learning progresses by updating the parameters.

[0051] The input image conversion unit (120) can convert a bright image of a daytime environment into a low-light image of a nighttime environment based on the brightness level (b) and weight (w) through a model trained by the total loss.

[0052] The input image conversion unit (120) uses an image conversion model that adds brightness control and feature concentration functions based on the CycleGAN model to generate N bright images. and N low-light images Takes as input, constructor G XY Using the generator G to convert bright images into low-light images YX Using this, low-light images can be converted into bright images.

[0053] The input image conversion unit (120) adds a brightness control loss function and a feature concentration map based on the CycleGAN generator. Typically, CycleGAN receives unpaired source domain images and target domain images as input and performs training, generating a result image by reflecting the information of the target domain while preserving the information of the source domain. In the present invention, N unpaired bright images are used. and N low-light images Constructor G that takes as input XY is a bright image to a low-light image, generator G YX Low-light images can be converted into bright images.

[0054] For example, constructor G XY The existing CycleGAN generator includes an additional feature concentration map, and learning is performed by a loss function that adds brightness control loss, which can result in the generation of low-light images with a specific brightness level.

[0055] The converted image determination unit (140) can determine whether each converted image is a bright image or a low-light image.

[0056] The converted image determination unit (140) can perform a task of distinguishing between a bright image converted from a low-light image and a bright image in an actual environment, and can perform a task of distinguishing between a low-light image converted from a bright image and a low-light image in an actual environment.

[0057] In one example, the transformed image discriminator adds a feature concentration map based on the CycleGAN discriminator. Typically, the discriminator of CycleGAN determines whether each image is in the corresponding domain for both the input image and the resulting image. In the present invention, the result of the discriminator is D Y (G XY (X)), D X (G YX (Y)) and the loss function is calculated through the result and reflected in the learning of each discriminator, so that the accuracy of discrimination can be improved.

[0058] The feature focus unit (150) can perform noise reduction by separating the background of the image for a converted bright image or low-light image.

[0059] In particular, the feature focus (150) is the generator G of the existing CycleGAN model. XY ,G YX and discriminator D X ,D Y Non-local neural networks can be additionally included in the structure.

[0060] In one example, the feature focus unit (150) may include self-attention. Generally, self-attention is a method used in Transformer, which shows good performance in natural language processing. For a given query, it calculates the similarity with each key and reflects each value mapped to the key using this similarity as a weight.

[0061] In the present invention, the feature focus unit (150) uses a non-local neural network, a method that applies self-attention to a CNN, to separate background and image-specific information. This method reduces noise, such as incorrectly converted artifacts that frequently occur in the resulting image, thereby generating low-light images similar to actual nighttime environments.

[0062] The brightness setting unit (160) can set the brightness level of the resulting image when calculating the total loss.

[0063] The brightness setting unit (160) can set the brightness level (b) and weight (w) so that the brightness level of the resulting image converges to a specific value when calculating the brightness control loss.

[0064] For example, when calculating brightness control loss, the brightness setting unit (160) can set the brightness level (b) and weight (w) input by the user. Through the setting values, the brightness level of the resulting image can converge to a specific value.

[0065] In addition, the low-light image enhancement system (100) according to one embodiment may further include a control unit (170).

[0066] The control unit (170) can be interpreted as a central processing unit (CPU) and can perform various operations and process data within the system.

[0067] In particular, the control unit (170) can read commands from memory, interpret and execute the commands, and can also perform arithmetic operations such as addition, subtraction, multiplication, and division.

[0068] In addition, the control unit (170) can handle data storage and retrieval, and can also perform the function of reading data from memory and storing the results of performing operations back into memory.

[0069] In addition, the control unit (170) can manage the execution flow of the program, and in particular, can control the flow of the program using commands such as conditional statements (if-else) or iterative statements (for, while). In addition, the control unit (170) can have a small and fast memory device called a register placed inside, and this register can be used to temporarily store data or perform operations.

[0070] The control unit (170) can process and take appropriate action when an external event or exceptional situation occurs, and can quickly access data and instructions by using cache memory that is faster than the main memory.

[0071] In addition, the control unit (170) can use a system bus to communicate with memory or input / output devices, and can provide various power management functions to minimize power consumption.

[0072] FIG. 2 is a diagram illustrating an image transformation model training process to reduce noise generated during low-light image augmentation and to transform the resulting image so that it has a brightness level input by a user in one embodiment of the present invention.

[0073] Referring to FIG. 2, in step 210, the data set of the low-light image enhancement system can be acquired from the camera of the autonomous vehicle in a bright situation in a daytime environment and a low-light situation in a nighttime environment.

[0074] In step 220, you can set the training parameters for the low-light image augmentation system. The image transformation model used in the low-light image augmentation system is based on CycleGAN, so you can set the learning rate, optimization method, and number of learning iterations (epochs). You can also set the brightness level (b) and weight (w) using the added brightness control loss function.

[0075] At step 230, the training parameters of the image transformation model can be set to a learning rate of 0.0002, an optimization method of Adam optimizer, and a learning iteration count of 200 epochs.

[0076] In step 240, the brightness level of the resulting image generated by the image transformation model can be set to a specific value.

[0077] The brightness level is typically measured using the Y value in the YCbCr domain of commonly used images, with a range of [0~255]. The formula for converting an RGB domain image to a YCbCr domain image is as follows.

[0078]

[0079] [Mathematical Formula 1]

[0080]

[0081]

[0082] Here, the R, G, and B values ​​represent each channel value in the RGB domain image, and the Y, Cb, and Cr values ​​represent each channel value in YCbCr.

[0083] The user normalizes the desired Y value (p) to [-1 to 1] using the formula below to specify the setting value (q). Here, the smaller the value, the darker the image is created, and the larger the value, the brighter the image is created.

[0084]

[0085] [Equation 2]

[0086]

[0087]

[0088] At step 250, the low-light image enhancement system can input the image obtained at step 210 into the model set at step 220.

[0089] At step 260, the low-light image enhancement system divides the input image at step 250 into a source domain image (X) and a target domain image (Y), and generates a generator G that transforms the source domain image into a target domain image. XY and a constructor G that converts a target domain image into a source domain image. YX Image conversion is performed through the target domain image (G) generated by conversion from the source domain. XY (X)) and the source domain image (G) generated by conversion from the target domain YX (Y)) can be created.

[0090] At step 270, the total loss can be calculated based on the established loss function to train the low-light image augmentation system. The equation for the total loss function is as follows.

[0091]

[0092] [Equation 3]

[0093]

[0094]

[0095] In [Equation 3] is used to adjust the ratio of the result values ​​of each loss function as a weight. The overall loss function consists of adversarial loss, cycle consistency loss, and Y-control loss, and each can be expressed as

[0096] Adversarial loss and circular matching loss are loss functions proposed by CycleGAN that help images in the source domain better transform into images in the target domain. Adversarial loss function is the constructor G XY , discriminator D Y , the loss is calculated through the source domain image X and the target domain image Y. As the image transformation model is trained, the parameters are updated to minimize the adversarial loss function. In this process, the generator G XY is trained to generate output images from source domain image X that are very similar to target domain image Y, and the discriminator D Y is a constructor G XY Image G generated by XY (X) is trained to distinguish Y, which is a target domain image.

[0097] In this process, the generator is trained with the goal of creating realistic images so that the discriminator cannot distinguish between the input image and the generated image. Training is conducted in the opposite situation.

[0098] Circular matching loss function is a constructor G XY ,G YX Compute the loss through the generator G from the source domain image X. XY Create a target domain image through (G XY (X)) This generated image is regenerated by the generator G YX Convert to source domain image via (G YX(G XY After (X))) the loss is calculated from the input source domain image X and the Euclidean distance and the generator G is calculated from the target domain image Y. YX Create a source domain image through (G YX (Y)) This generated image is regenerated by the generator G XY Convert to target domain image via (G XY (G YX After (Y))), the circular matching loss is calculated by adding the loss calculated by the input target domain image Y and the Euclidean distance.

[0099] The brightness adjustment loss function is a generator G from the source domain image X. XY Image G generated as a target domain image via XY (X) is used to calculate the loss. In step 240, the image G is generated with the set value (q) and the target domain image. XY After converting (X) to the YCbCr image domain, the average of Y values ​​for all pixels of the image (Here, the T function means the Y value after converting to a YCbCr domain image.) The loss is calculated by the Euclidean distance.

[0100] In step 280, a trained image transformation model for a low-light image augmentation system is obtained. The resulting model can be used to transform bright images in daytime environments into low-light images in nighttime environments.

[0101] FIG. 3 is a diagram illustrating a process of converting a bright image into a low-light image by a trained image conversion model according to one embodiment of the present invention.

[0102] In one example, an image transformation model may consist of an encoder, a residual block, a self-attention part, and a decoder.

[0103] In one example, an encoder can take as input a bright situation image of a daytime environment and extract a feature map.

[0104] In one example, a feature map is the result of applying a convolutional method to an input image in an encoder.

[0105] In one example, the residual block learns the differences between bright and low-light images during the process of converting a bright image in a daytime environment into a low-light image in a nighttime environment. This allows the residual block to extract low-light features from the bright image and apply them to the converted image. Therefore, the residual block learns the information necessary for the image transformation model to extract low-light features from the bright image and apply them to convert it into a low-light image.

[0106] In one example, the feature focus unit can add self-attention to the image transformation model and pass the feature map obtained from the residual block through the feature focus unit to obtain an attention map.

[0107] In one example, self-attention added to an image transformation model can be set to a non-local neural network.

[0108] In one example, a non-local neural network is applied to the entire space of the image.

[0109] Here, every space means every pixel.

[0110] The working formula of a non-local neural network is as follows.

[0111]

[0112] [Equation 4]

[0113]

[0114]

[0115] Here, i represents the location of a specific pixel in the image, and j represents another pixel location in the image excluding the location of i. Equation u obtains the degree of correlation between the information at the location of pixel i and the information at the location of pixel j, and equation v provides the degree of correlation between the information at the location of pixel j and all other information. In [Mathematical Equation 4], C is the part calculated by normalization. The result of the non-local neural network provides the role of each pixel in the entire image, and this information can reduce noise when generating a low-light image.

[0116] In one example, a decoder can take the results of a feature-focused region as input and generate a low-light image. This generation is performed using an upconvolution method.

[0117] In one example, an image transformation model generates a low-light image with a set brightness level. By setting the brightness level to a different value, the trained model can generate low-light images with that brightness level. This method allows for the generation of low-light images with various brightness levels, and these models can be used to generate low-light images with various brightness levels.

[0118] FIG. 4 is a graph showing the brightness level of a result image generated according to the brightness level set in the brightness setting unit according to one embodiment of the present invention.

[0119] In one example, in step 220 of Fig. 2, the weight w can be set constant and the brightness level b can be set differently.

[0120] Here, the weight w can be set to 10.

[0121] Looking at the results of the graph in Fig. 2, it can be confirmed that the brightness of the low-light images generated when the brightness level b is set to 10 converges to 10, that the brightness of the low-light images generated when the brightness level b is set to 20 converges to 20, and that the brightness of the low-light images generated when the brightness level b is set to 50 converges to 50.

[0122] The results demonstrate that the image transformation model learned by adding brightness control loss can converge the brightness level of the resulting image to a specific value.

[0123] FIG. 5 is a graph showing the brightness level of a result image generated according to the weight level set in the brightness setting unit according to one embodiment of the present invention.

[0124] In one example, in step 220 of Fig. 2, the brightness level b can be set constant and the weight w can be set differently.

[0125] Here, the brightness level b can be set to 10.

[0126] Looking at the graph results in Fig. 5, when the weight w is 1, the average brightness of the generated low-light images converged to approximately 45, and the deviation is very large compared to other weighted results. When the weight w is 5, the average brightness of the generated low-light images converged to approximately 33, and the deviation is large compared to other weighted results. When the weight w is 10, the average brightness of the generated low-light images converged to approximately 10, and the deviation is small compared to other weighted results. When the weight w is 20, the average brightness of the generated low-light images converged to approximately 10, and the deviation is very small compared to other weighted results, but the occurrence of many outliers is very large. Therefore, it can be proven that the case where the weight w is 10 is the case where an ideally bright image is converted into a low-light image.

[0127] FIG. 6 is a diagram illustrating an image of the execution result of an image conversion model that applies a CycleGAN model and a feature concentration unit and a brightness control unit according to an embodiment of the present invention.

[0128] In one example, (a) of Fig. 6 is an image representing a bright image original in a daytime situation and is an input image of an image transformation model.

[0129] Figure 6 (b) shows the result of converting the input image of (a) into a low-light image of a nighttime environment using the image conversion model CycleGAN. Figure 6 (c) shows the result image when only a brightness control unit is added to the image conversion model CycleGAN to converge the brightness level of the converted low-light image to a specific value. Comparing the result with (b), we can see that the brightness level is darker, similar to an actual nighttime environment, and the darkness level is more uniformly distributed.

[0130] Figure 6 (d) shows the resulting image when the CycleGAN image transformation model was converted into a low-light image by adding a brightness control section and a feature focus section. Compared to (c), the resulting image shows reduced noise. Furthermore, compared to (b), which uses CycleGAN alone, (d) more closely resembles a real nighttime environment, demonstrating an improved image transformation model for low-light image enhancement systems.

[0131] FIG. 7 is a diagram illustrating an image of the execution result of an image transformation model that applies existing image transformation models (CycleGAN, DRIT++, TSIT, MUNIT) and a feature concentration part and brightness control part to a bright input image (KITTI) according to one embodiment of the present invention.

[0132] In one example, the raw image in Fig. 7 represents a bright original image in daylight conditions and is the input image for the image transformation model. Ours is a model that adds a brightness control unit and a feature focus unit to CycleGAN, and the Y value represents the brightness level b in the brightness control unit.

[0133] In one example, the results of several image transformation models (CycleGAN, DRIT++, TSIT, MUNIT) in Fig. 7 show that the brightness level does not converge to a specific value, and a lot of noise, such as artifacts, is generated in the image. Compared to these results, Ours allows brightness adjustment through the Y value, and it can be confirmed that artifacts are greatly reduced compared to the results of other image transformation models, obtaining results similar to actual nighttime situations. Through these results, it can be explained that an image transformation model with added brightness control and feature control is suitable for a low-light image enhancement system.

[0134] Ultimately, by using the present invention, it is possible to perform low-light image enhancement with excellent performance by converting a bright image in a daytime environment into a low-light image in a nighttime environment, and to control the brightness level of the low-light image.

[0135]

[0136] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0137] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0138] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0139] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0140] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A training data configuration unit that configures training data based on a first image generated in a first environment distinguished by illuminance, and a second image generated in a second environment with low illuminance compared to the first environment; An input image conversion unit that receives a first image and a second image as input and converts the first image into a second image and the second image into the first image; and A computation unit that compares an input image with at least one of the first image or the second image that has been converted to calculate a total loss for training a model. Including, The above input image conversion unit, A low-light image augmentation system characterized in that it transforms a first image of a first environment into a second image of a second environment based on a brightness level (b) and a weight (w) through a model trained by the above total loss.

2. In paragraph 1, An image determination unit that determines whether each of the above-mentioned converted images is the first image or the second image. A low-light image enhancement system further comprising:

3. In paragraph 1, Self-attention section that performs noise reduction by separating the background of the image for the first or second image that has been converted above. A low-light image enhancement system further comprising:

4. In paragraph 1, A brightness setting section that sets the brightness level of the resulting image when calculating the above total loss. A low-light image enhancement system further comprising:

5. In paragraph 1, The above training data configuration section is, Extract N data from each of the first situation data set (X) of the first environment and the second situation data set (Y) of the second environment to create the first image. Wow, the second image A low-light image enhancement system characterized by constructing a .

6. In paragraph 1, The above input image conversion unit, N first images are generated using an image transformation model that adds brightness control and feature concentration functions based on the CycleGAN model. and N second images It takes as input, and constructor G XY Using the generator G, the first image is converted to the second image. YX A low-light image enhancement system characterized in that a second image is converted into a first image using a.

7. In paragraph 1, The above-mentioned converted image determination unit is, A low-light image enhancement system characterized by performing a task of distinguishing a first image converted from a second image from a first image in a real environment, and performing a task of distinguishing a second image converted from a first image from a second image in a real environment.

8. In paragraph 3, The above feature concentration area is, Generator G of the existing CycleGAN model XY, G YX and discriminator D X , D Y A low-light image enhancement system characterized by additionally including a non-local neural network in its structure.

9. In paragraph 1, The above operation unit, A low-light image augmentation system characterized by calculating the total loss including the adversarial loss presented in CycleGAN, the cycle consistency loss, and additional brightness control loss.

10. In paragraph 1, The above operation unit, Generator G using the calculated total loss XY, G YX and discriminator D X , D Y A low-light image enhancement system characterized by updating parameters.

11. In paragraph 4, The above brightness setting section, When calculating brightness control loss, So that the brightness level of the resulting image converges to a specific value, A low-light image enhancement system characterized by setting a brightness level (b) and a weight (w).

12. A step of collecting data including a first image generated in a first environment distinguished based on illuminance, and a second image generated in a second environment with low illuminance compared to the first environment; Steps to set up an environment for training an image transformation model; A step of inputting the collected data into the trained image transformation model; A generator G that divides the above input data into a source domain image (X) and a target domain image (Y), and converts the source domain image into a target domain image. XY Wow, a constructor G that converts target domain images to source domain images. YX Steps to convert images through; A step of calculating the total loss based on a set loss function for training the system; and A step of creating a trained image transformation model using the above calculated total loss. A method of operating a low-light image enhancement system, characterized in that it includes:

13. In paragraph 12, The steps for setting up an environment for training the above image transformation model are: Step of setting training parameters; and Step 1: Set the target brightness for the resulting image A method of operating a low-light image enhancement system, characterized in that it includes:

Images