Method and apparatus for improving low-illuminance image using resizing

Abstract

The present invention relates to an apparatus and method for improving a low-illuminance image using resizing, whereby the low-illuminance image can be improved in near real time by using a resizing mechanism and a convolutional network in order to solve the conventional problem that excessive resources and calculation time are required to improve the brightness of a high-resolution image.

Description

Method and apparatus for improving low-light images using resizing

[0001] The present invention relates to a technology for improving the brightness of low-light images using resizing, and more specifically, to a technology capable of improving low-light images in near real-time by utilizing a resizing mechanism and a convolutional network to solve the conventional problem of requiring excessive resources and computation time to improve the brightness of high-resolution images.

[0002] A low-light image enhancement algorithm refers to an image transformation algorithm that adjusts the brightness of at least some areas of an image to facilitate the identification of a target object by the user in an image taken in a low-light environment.

[0003] Recently, deep learning-based low-light image enhancement algorithms have been actively researched, and deep learning algorithms based on CNN (Convolutional Neural Network), such as DCE-Net (Deep Curve Estimation Network), U-Net, and PENet (Pulmonary Embolism Network), have been proposed.

[0004] These conventional low-light image enhancement algorithms have the disadvantage that they operate only on input images of a fixed size, requiring a preprocessing step to handle cases where the input image sizes vary.

[0005] Furthermore, since these illumination enhancement algorithms perform convolution with an output size equal to the input image size, improving low-light in high-resolution images requires processing a large number of pixels, which consumes a significant amount of time. This not only makes real-time processing difficult but also necessitates high-spec computing resources.

[0006] Korean Registered Patent Publication No. 10-2611606 (Prior Art) discloses a technology for improving noise generated in a low-light improved image by applying a Generative Adversarial Network when improving pixel-based low-light images for which actual measurement data (Ground Truth) is unavailable.

[0007] However, although the prior art improves the illuminance of the low-light image through a low-light noise improvement model learned by unsupervised learning, the need for improvement is still raised in that it requires significant computation time and excessive resources due to the large number of pixels to be processed.

[0008] The problem that the present invention aims to solve is to provide an apparatus and method for improving low-light images using resizing, which solves the problem of computational speed of high-resolution images by converting a low-light high-resolution image to low resolution, improving the illumination of the low-resolution image, upscaling the low-resolution image with improved illumination, and then improving the illumination of the original high-resolution image using the pixel values ​​of the upscaled image as a hint.

[0009] One of the means of solution of the present invention proposes a method for improving low-light images of an image processing device comprising a preprocessor and a convolutional processor.

[0010] A low-light image improvement method according to one embodiment comprises: a step in which a preprocessor converts a low-light image of a first resolution to a second resolution lower than the first resolution, corrects the brightness of the low-light image with reduced resolution, and then converts the image with corrected brightness back to the first resolution to generate a hint image; and a step in which a convolutional processor improves the brightness of the low-light image of the first resolution by adjusting the pixel values ​​of the original image using the adjusted pixel values ​​of the hint image as a hint.

[0011] In addition, as another means of solution of the present invention, a device for improving low-light images using resizing is proposed.

[0012] The low-light image enhancement device described above includes: a resizer that generates a low-resolution image by converting a high-resolution low-light original image into a low-resolution image; an illuminance enhancer that generates an illuminance enhanced image with increased brightness by adjusting the value of each pixel of the low-resolution image using a pre-established illuminance enhancement algorithm; an upscaler that generates an enlarged image by increasing the illuminance enhanced image to the same size as the original image; and an image processor that improves the brightness of the high-resolution original image by adjusting the pixel value of the original image using the adjusted pixel value of the enlarged image as a hint.

[0013] In the above method or device, a convolutional processor of one embodiment may extract each pixel value for illumination using the pixel intensity and gradient of the low-illumination image of the first resolution and the hint image, construct a neural network by stacking the extracted each pixel value of the low-illumination image of the first resolution and the hint image, and generate an image with improved brightness by adjusting the pixel value of the low-illumination image of the first resolution through the constructed neural network.

[0014] In the above method or device, a convolutional processor of one embodiment may determine whether a pixel of a low-light image of the first resolution has an illuminance level below a preset standard based on the constructed stacked neural network.

[0015] In the above method or device, a convolution processor of one embodiment may, when the pixel value of the low-light image of the first resolution is an illuminance level lower than or equal to a preset standard, convert the pixel value of the low-light image of the first resolution and the pixel value of the hint image into a matrix and match them by the same location, and perform a convolution operation between the pixel value of the low-light image of the first resolution and the pixel value of the hint image by sector to adjust the pixel value of the low-light image of the first resolution.

[0016] In the above method or device, if the result of performing the convolution operation and adjusting the pixel value of the low-light image of the first resolution falls short of a preset standard, the convolution processor of one embodiment may repeat the convolution operation until the adjusted pixel value reaches the standard.

[0017] In the above method or device, an image processor of one embodiment may extract each pixel value for illuminance using the pixel-by-pixel intensity and gradient of the original image, build a neural network by stacking the extracted each pixel value of the original image and the enlarged image into a stack, and generate an image with improved brightness by adjusting the pixel value of the original image through the built neural network.

[0018] In the above method or device, an image processor of one embodiment may determine whether a pixel of the original image has an illumination level below a reference based on the constructed stacked neural network.

[0019] In the above method or device, an image processor of one embodiment may, when it is determined whether the pixel value of the original image is an illuminance below a reference, convert the pixel values ​​of the original image and the enlarged image into a matrix and match them by the same location, and perform a convolution operation between the pixel value of the original image and the pixel value of the enlarged image by sector to adjust the pixel value of the original image.

[0020] In the above method or device, if the result of adjusting the pixel values ​​of the original image by performing the convolution operation falls short of a standard, the image processor of one embodiment may repeat the convolution operation until the result reaches a standard.

[0021] In the above method or device, one of Unet, DCE-Net ("Dual-Channel Enhanced Network"), and PENet (Progressive Enhancement Network) may be used as the illumination improvement algorithm of one embodiment.

[0022] According to an embodiment of the present invention, in order to solve the problem of computational speed of high-resolution images, a high-resolution image with low illumination is converted to a low resolution, the illumination of the low-resolution image is improved, and the low-resolution image with improved illumination is upscaled, and the illumination of the original high-resolution image is improved using the pixel values ​​of the upscaled image as a hint. This significantly reduces the amount of computation compared to improving the illumination of the original image as is, enabling real-time processing, and resolves the problem of requiring high hardware when the amount of computation is large.

[0023] FIG. 1 is a schematic diagram illustrating an apparatus for improving low-light images using resizing according to Example 1 of the present invention.

[0024] FIG. 2 is a block diagram showing the configuration of a device for improving low-light images using resizing according to Example 1 of the present invention.

[0025] FIG. 3 is a flowchart illustrating a method for improving a low-light image using resizing according to Example 2 of the present invention.

[0026] Several embodiments of the present invention will be described in detail below with reference to the drawings. However, this is not intended to limit the present invention to any specific embodiment, and it should be understood that all transformations, equivalents, and substitutions including the technical concept of the present invention are included within the scope of the present invention.

[0027] In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0028] Where in this specification it is stated that one component “have” or “comprise” a sub-component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0029] In this specification, the terms "...Unit," "...Module," and "Component" refer to a unit that processes at least one function or operation, and may be implemented in hardware, software, or a combination of hardware and software.

[0030] In this specification, the term “connect” may mean that two components are directly connected, but is not necessarily limited thereto, and may also mean that they are connected via one or more other components positioned between the components.

[0031] <Example 1>

[0032] FIG. 1 is a schematic diagram illustrating an apparatus for improving low-light images using resizing according to Example 1 of the present invention.

[0033] The device of the present embodiment converts a low-illumination high-resolution image into a low-resolution image, improves the illumination of the low-resolution image, enlarges the low-resolution image (11) with improved illumination, and then improves the illumination of the high-resolution original image (10) using the pixel values ​​of the enlarged image as a hint.

[0034] For example, if this process is omitted to improve low-light performance in high-resolution images, there is a problem in that real-time processing is difficult because it takes a considerable amount of time due to the large number of pixels to be processed.

[0035] Since low-resolution images have a low amount of computation and can be processed quickly, the device of the present embodiment first converts a high-resolution image to a low resolution and then improves the illumination, and then improves the illumination of the original image (10) based on the pixel values ​​of the low-resolution image with improved illumination (hint). Therefore, compared to improving the illumination of the original image (10) as is, the amount of computation is significantly reduced, enabling real-time processing, and the problem of requiring high hardware when the amount of computation is high can be resolved.

[0036] FIG. 2 is a block diagram showing the configuration of a device for improving low-light images using resizing according to Example 1 of the present invention.

[0037] As shown in FIGS. 1 and 2, the device includes a resizer (110), an illuminance improver (120), an upscaler (130), and an image processor (140).

[0038] The resizer (110) receives a high-resolution, low-light original image (10) and reduces the original image (10) to a predefined size.

[0039] Here, reducing the size of the image means reducing the width and height of the original image (10), and the predefined size may mean a size that can be used in the illumination improvement algorithm described later.

[0040] For reference, since the U-Net among the illumination improvement algorithms operates only on input images of a fixed size, a preprocessing step may be required to handle cases where the size of the input image varies, and to omit the preprocessing step, the resizer (110) reduces the size of the original image (10) to a size that can be used in the U-Net.

[0041] When the width and height, i.e., the size, of the original image (10) is reduced through the resizer (110), the number of pixels is reduced, and the resolution of the original image (10) is reduced.

[0042] Additionally, the resizer (110) may reduce the resolution of the original image (10) by adjusting pixel values ​​during the process of the width and height of the original image (10).

[0043] Pixel values ​​refer to the values ​​assigned to each pixel within an image, and it is also possible to reduce the image resolution by adjusting pixel values.

[0044] The illumination improver (120) generates an illumination improved image (21) with increased brightness by adjusting the value of each pixel of the low-resolution image (11).

[0045] For reference, the values ​​assigned to the pixels of an image include color and brightness, and the brightness, contrast, and color of the image can be adjusted by controlling the pixel values.

[0046] The illumination improver (120) may generate an illumination improvement image (21) with increased brightness by adjusting the value of each pixel using a pre-established illumination improvement algorithm.

[0047] For example, the illumination improvement algorithm that can be applied in this embodiment may use any one of the following methods: a Retinex-based illumination improvement algorithm, a deep learning-based illumination improvement algorithm, noise removal, and back-projection (Deconvolution).

[0048] Retinex-based illumination enhancement algorithms improve image contrast and sharpness by adjusting image color and brightness. Retinex-based algorithms improve images by separating the adjustment of overall brightness from the adjustment of local lighting variations.

[0049] Deep learning-based illumination enhancement algorithms primarily use Convolutional Neural Networks (CNNs) to learn pairs of low-light and high-light images and improve low-light images. Representative examples include Unet, DCE-Net ("Dual-Channel Enhanced Network"), and PENet (Progressive Enhancement Network).

[0050] Deconvolution algorithms use deconvolution and backprojection techniques to remove noise from noisy, low-light images and make the images sharper.

[0051] The upscaler (130) generates an enlarged image (20) that increases the brightness-enhanced image (21) to the same size as the original image (10).

[0052] Specifically, the upscaler (130) expands the width and height of the illumination-enhanced image (21) to be equal to the original image (10), calculates new pixel values ​​using surrounding pixel values, and expands the image by placing new pixels between existing pixels. That is, when the width and height of the image are expanded by the upscaler (130), the total number of pixels increases due to the new pixels.

[0053] For example, when an upscaler (130) enlarges an image, it undergoes a process of spreading existing pixels into the enlarged area. At this time, to enlarge the image, it determines new pixel values ​​using pixel value interpolation. The new pixels have the same size as the existing pixels.

[0054] Pixel value interpolation techniques create new pixels by calculating new pixel values ​​from existing pixel values ​​when an image is greatly enlarged.

[0055] Pixel value interpolation techniques include bicubic interpolation, nearest neighbor interpolation, bilinear interpolation, and trilinear interpolation. Since these interpolation techniques enlarge the image size using mathematically simple methods, they have the advantage of being able to be executed with fast computation speeds and low hardware requirements.

[0056] Bicubic interpolation is one of the interpolation techniques used in image processing that calculates a new pixel value by considering 16 surrounding pixels. Bicubic interpolation can generate smoother images by interpolating values ​​between neighboring pixels using a curve. The bicubic algorithm provides more sophisticated interpolation than bilinear (considering 4 neighboring pixels) and trilinear (considering 8 neighboring pixels) interpolation methods.

[0057] Nearest neighbor interpolation creates a new pixel by using the value of the nearest neighbor pixel when determining the value of a new pixel. This method has the advantage of being simple and fast to calculate.

[0058] Bilinear interpolation creates a new pixel by calculating a weighted average of the values ​​of the four surrounding pixels to calculate a new pixel value. This method enables the generation of smoother images compared to nearest neighbor interpolation.

[0059] Trilinear interpolation is used for 3D image data and creates a new pixel by calculating a weighted average of the values ​​of the eight surrounding pixels. This method provides more precise interpolation than bilinear interpolation.

[0060] The image processor (140) adjusts the pixel values ​​of the original image (10) using the adjusted pixel values ​​of the enlarged image (20) as a hint to generate an image (30) with improved brightness of the original image (10) of high resolution.

[0061] For example, the image enlarged in the upscaler (130) using the interpolation technique described above has its size increased in a mathematically simple way, so it is low resolution and thus image distortion occurs. However, since each pixel has a pixel value with improved illumination, the image processor (140) uses the pixel value with improved illumination of the enlarged image (20) as a hint to adjust the pixel value of the original image (10) to generate an image (30) with improved brightness.

[0062] As a specific example, the image processor (140) extracts each pixel value for illumination using the pixel intensity and gradient of the original image. Using the pixel intensity and gradient means edge detection in the image, that is, finding points with strong changes (edges), where strong changes refer to changes in pixel intensity. And at the points with edges, each pixel value for illumination can be extracted.

[0063] The image processor (140) must build a neural network by stacking the extracted pixel values ​​of the original image and the enlarged image into a stack. That is, it creates a total of 6 layers of data by stacking an additional RGB layer with improved pixel values ​​on the 3 layers (RGB) of data, and generates an image (30) with improved brightness by adjusting the pixel values ​​of the original image (10) through the pixel values ​​of the original image and improved illumination information.

[0064] For reference, a stack refers to a structure formed by stacking multiple models or layers.

[0065] The image processor (140) determines whether the pixel value of the original image is below a reference illuminance based on the constructed stacked neural network.

[0066] The image processor (140) places the original image (10) and the enlarged image (20) side by side to improve the illumination level below a standard in the original image (10), and stacks the pixels of the original image (10) and the adjusted (improved) pixels of the enlarged image (20) to adjust the pixel values ​​of the original image (10) to generate an image (30) with improved brightness.

[0067] Specifically, the image processor (140) converts each pixel value of the original image (10) and the enlarged image (20) into a matrix and matches them by position, and performs a convolution operation (convolution) on the pixel values ​​of the original image (10) and the enlarged image (20) by sector to adjust the pixel values ​​of the original image (10). For convenience of explanation regarding the matrix form of the pixel values, a black and white image is used as an example. In the case of a black and white image, it has a matrix form of (horizontal, vertical, 1), and when the two images are placed side by side, it has a matrix form of (horizontal, vertical, 2).

[0068] Here, the convolution operation moves sector by sector over the original image (10) and the enlarged image (20) placed side by side, multiplies the pixel values ​​of the parts that overlap with the convolution filter, and then sums them all to generate a new pixel value, and the new pixel value is adjusted to the corresponding pixel value of the original image (10). And when this process is performed for all pixels, an image with improved illumination of the original image (10) is generated.

[0069] If the image processor (140) performs a convolution operation and the result of adjusting the pixel values ​​of the original image falls short of the standard, the convolution operation can be repeated until the result reaches the standard.

[0070] <Example 2>

[0071] FIG. 3 is a flowchart illustrating a method for improving a low-light image using resizing according to Example 2 of the present invention.

[0072] As shown in FIG. 3, the method of the present embodiment includes the steps of generating a low-resolution image (S100), generating an image with improved illumination (S200), generating an enlarged image (S300), and improving the brightness of the original image (S400).

[0073] The step of generating a low-resolution image (S100) involves a resizer receiving a high-resolution, low-light original image and reducing the original image to a predefined size. Here, the predefined size may refer to a size that can be used in the illumination improvement algorithm described later.

[0074] For reference, among illumination enhancement algorithms, UNET operates only on fixed-size input images; therefore, if the input image sizes vary, a preprocessing step may be required to handle them. To skip this preprocessing step, the resizer reduces the size of the original image to a size usable by UNET.

[0075] When the width and height of the original image—that is, its size—are reduced through a resizer, the number of pixels decreases, and the resolution of the original image is reduced.

[0076] In addition, the resizer can reduce the resolution of the original image by adjusting pixel values ​​for the width and height of the original image during the process.

[0077] Pixel values ​​refer to the values ​​assigned to each pixel within an image, and it is also possible to reduce the image resolution by adjusting pixel values.

[0078] The step of generating an illumination improvement image (S200) involves the illumination improver adjusting the value of each pixel of a low-resolution image to generate an illumination improvement image with increased brightness.

[0079] For reference, the values ​​assigned to the pixels of an image include color and brightness, and the brightness, contrast, and color of the image can be adjusted by controlling the pixel values.

[0080] The illuminance enhancer can also generate an illuminance-enhanced image with increased brightness by adjusting the value of each pixel using a pre-established illuminance enhancement algorithm.

[0081] For example, the illumination improvement algorithm that can be applied in this embodiment may use any one of the following methods: a Retinex-based algorithm, a deep learning-based illumination improvement algorithm, noise removal, and back-projection (Deconvolution).

[0082] Retinex-based algorithms enhance image contrast and sharpness by adjusting image color and brightness. Retinex-based algorithms improve images by separating the adjustment of overall brightness from the adjustment of local lighting variations.

[0083] Deep learning-based illumination enhancement algorithms primarily use Convolutional Neural Networks (CNNs) to learn pairs of low-light and high-light images and improve low-light images. Representative examples include Unet, DCE-Net ("Dual-Channel Enhanced Network"), and PENet (Progressive Enhancement Network).

[0084] Deconvolution algorithms use deconvolution and backprojection techniques to remove noise from noisy, low-light images and make the images sharper.

[0085] The step of generating an enlarged image (S300) involves the upscaler generating an enlarged image in which the illumination-enhanced image is increased to the same size as the original image.

[0086] Specifically, the upscaler enlarges the width and height of the illumination-enhanced image to be the same as the original image, but it expands the image by calculating new pixel values ​​using surrounding pixel values ​​and placing new pixels between existing pixels. In other words, when the width and height of the image are enlarged by the upscaler, the total number of pixels increases due to the new pixels.

[0087] For example, when an upscaler enlarges an image, it undergoes a process of spreading existing pixels into the enlarged area; in this process, it determines new pixel values ​​using pixel value interpolation to enlarge the image. The new pixels have the same size as the existing pixels.

[0088] Pixel value interpolation techniques create new pixels by calculating new pixel values ​​from existing pixel values ​​when an image is greatly enlarged.

[0089] Pixel value interpolation techniques include bicubic interpolation, nearest neighbor interpolation, bilinear interpolation, and trilinear interpolation. Since these interpolation techniques enlarge the image size using mathematically simple methods, they have the advantage of being able to be executed with fast computation speeds and low hardware requirements.

[0090] Bicubic interpolation is one of the interpolation techniques used in image processing that calculates a new pixel value by considering 16 surrounding pixels. Bicubic interpolation can generate smoother images by interpolating values ​​between neighboring pixels using a curve. The bicubic algorithm provides more sophisticated interpolation than bilinear (considering 4 neighboring pixels) and trilinear (considering 8 neighboring pixels) interpolation methods.

[0091] Nearest neighbor interpolation creates a new pixel by using the value of the nearest neighbor pixel when determining the value of a new pixel. This method has the advantage of being simple and fast to calculate.

[0092] Bilinear interpolation creates a new pixel by calculating a weighted average of the values ​​of the four surrounding pixels to calculate a new pixel value. This method enables the generation of smoother images compared to nearest neighbor interpolation.

[0093] Trilinear interpolation is used for 3D image data and creates a new pixel by calculating a weighted average of the values ​​of the eight surrounding pixels. This method provides more precise interpolation than bilinear interpolation.

[0094] The step of improving the brightness of the original image (S400) involves the image processor adjusting the pixel values ​​of the original image using the adjusted pixel values ​​of the enlarged image as a hint to improve the brightness of the high-resolution original image.

[0095] For example, an image enlarged by an upscaler using the interpolation technique described above is enlarged in a mathematically simple way, but because it is low resolution, image pixelation occurs. However, since each pixel has a pixel value with improved illumination, the image processor adjusts the pixel values ​​of the original image using the pixel value with improved illumination of the enlarged image as a hint.

[0096] As a specific example, the image processor extracts each pixel value for illuminance using the pixel-by-pixel intensity and gradient of the original image. Using pixel-by-pixel intensity and gradient means edge detection in the image, that is, finding points with strong changes (edges), where strong changes refer to changes in pixel intensity. And at the points with edges, each pixel value for illuminance can be extracted.

[0097] The image processor builds a neural network by stacking each extracted pixel value into a stack. That is, it creates a total of 6 layers of data by stacking an additional 3 layers of RGB with improved pixel values ​​on the 3 layers (RGB) of data, and generates an image with improved brightness by adjusting the pixel values ​​of the original image (10) through the pixel values ​​of the original image and improved illumination information.

[0098] For reference, a stack refers to a structure formed by stacking multiple models or layers.

[0099] The image processor determines whether the pixel value of the original image is below a reference illuminance based on the constructed stacked neural network.

[0100] An image processor places the original image (10) and the enlarged image side by side to improve the illumination level below a standard in the original image, and stacks the pixels of the original image and the adjusted (improved) pixels of the enlarged image to adjust the pixel values ​​of the original image and generates an image with improved brightness.

[0101] Specifically, the image processor converts each pixel value of the original image and the enlarged image (20) into a matrix and matches them by position, and performs a convolution operation between the pixel values ​​of the original image and the pixel values ​​of the enlarged image (20) by sector to adjust the pixel values ​​of the original image. For convenience of explanation regarding the matrix form of the pixel values, a black and white image is used as an example. In the case of a black and white image, it has a matrix form of (horizontal, vertical, 1), and when the two images are placed side by side, it has a matrix form of (horizontal, vertical, 2).

[0102] Here, the convolution operation moves sector by sector over the original image and the enlarged image (20) placed side by side, multiplies the pixel values ​​of the parts that overlap with the convolution filter, and then sums them all to generate a new pixel value, and the new pixel value is adjusted to the corresponding pixel value of the original image. And when this process is performed for all pixels, an image with improved illumination of the original image is generated.

[0103] If the image processor performs a convolution operation and the result of adjusting the pixel values ​​of the original image falls short of a standard, the convolution operation may be repeated until the result reaches the standard.

[0104] <Example 3>

[0105] Example 3 relates to a method for improving low-light images of an image processing device comprising a preprocessor and a convolution processor.

[0106] The image enhancement method of Example 3 includes the step of generating a hint image and the step of enhancing the brightness of the image.

[0107] The step of generating a hint image involves a preprocessor converting a low-light image of a first resolution to a second resolution lower than the first resolution, correcting the brightness of the low-light image with reduced resolution, and then converting the brightness-corrected image back to the first resolution to generate a hint image.

[0108] The preprocessor converts a low-light image of the first resolution into a second resolution lower than the first resolution.

[0109] For example, the preprocessor reduces a low-light image of a first resolution to a predefined size, but when the image size is reduced, the number of pixels decreases and the resolution may decrease.

[0110] For reference, the preprocessor can also reduce the image resolution by adjusting pixel values.

[0111] The preprocessor corrects the brightness of low-light images with reduced resolution using a pre-established illumination enhancement algorithm.

[0112] Then, the preprocessor converts the brightness-corrected image back to the first resolution to generate a hint image.

[0113] The step of improving the brightness of a low-light image of the first resolution involves a convolutional processor adjusting the pixel values ​​of the low-light image of the first resolution using the adjusted pixel values ​​of the hint image as a hint to improve the brightness of the low-light image of the first resolution of the high resolution.

[0114] The convolution processor extracts each pixel value for illuminance using the pixel-by-pixel intensity and gradient of the low-light image and hint image of the first resolution. Using pixel-by-pixel intensity and gradient means edge detection in the image, that is, finding points with strong changes (edges), where strong changes refer to changes in pixel intensity. And at the points with edges, each pixel value for illuminance can be extracted.

[0115] The convolution processor constructs a neural network by stacking each pixel value extracted from a low-light image of resolution 1 and the hint image. That is, it creates a total of 6 layers of data by stacking 3 layers of RGB data with improved pixel values ​​to the 3 layers (RGB) of data, and generates an image with improved brightness by adjusting the pixel values ​​of the low-light image of resolution 1 using the pixel values ​​of the low-light image of resolution 1 and improved illumination information.

[0116] The convolution processor determines whether the pixel value of the low-light image of the first resolution is an illumination level below a reference based on the constructed stacked neural network.

[0117] A convolution processor places a low-light image of a first resolution and a hint image side by side to improve the illumination below a reference level in a low-light image of a first resolution, and stacks the pixels of the low-light image of the first resolution and the adjusted (improved) pixels of the hint image to adjust the pixel values ​​of the low-light image of the first resolution to generate an image with improved brightness.

[0118] Specifically, the convolution processor converts each pixel value of the low-light image of the first resolution and the hint image into a matrix and matches them by position, and performs a convolution operation (convolution) between the pixel values ​​of the low-light image of the first resolution and the pixel values ​​of the hint image (20) by sector to adjust the pixel values ​​of the low-light image of the first resolution. For convenience of explanation regarding the matrix form of the pixel values, a black and white image is used as an example. In the case of the black and white image, it has a matrix form of (horizontal, vertical, 1), and when the two images are placed side by side, it has a matrix form of (horizontal, vertical, 2).

[0119] Here, the convolution operation moves sector by sector over the first low-light image and the hint image placed side by side, multiplies the pixel values ​​of the parts overlapping with the convolution filter, and then sums them all to generate a new pixel value, and the new pixel value is adjusted to the corresponding pixel value of the first low-light image. And when this process is performed for all pixels, an image with improved illumination of the first low-light image (10) is generated.

[0120] If the result of the convolution processor performing a convolution operation to adjust the pixel values ​​of the low-light image of the first resolution falls short of the standard, the convolution operation may be repeated until the result reaches the standard.

[0121] Although the present invention has been described above with reference to several embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

[0122] In addition, among the embodiments described above, the invention relating to the method may be implemented as a program or as a computer-readable recording medium on which the program is stored.

[0123] That is, the present invention can be implemented in the form of an application, and can be implemented as a software program that runs on a mobile terminal such as a smartphone or tablet PC running on Google’s Android or Apple’s iOS, or as a software program that runs on a wearable device such as Google Glass, Apple Watch, Samsung Galaxy Watch, or smartwatch, or as a software program that runs on a laptop PC or desktop PC running on Microsoft’s Windows or Google’s Chrome OS.

[0124] In addition, partial functions of the device or system described above may be provided by being included in a computer-readable recording medium by tangibly implementing a program of instructions for implementing them. A computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and USB memory.

Claims

1. A method for improving low-light images of an image processing device including a preprocessor and a convolutional processor, wherein A step in which the above preprocessor converts a low-light image of a first resolution to a second resolution lower than the first resolution, corrects the brightness of the low-light image with reduced resolution, and then converts the brightness-corrected image back to the first resolution to generate a hint image; A step in which the convolutional processor adjusts the pixel values ​​of the original image using the adjusted pixel values ​​of the hint image as a hint to improve the brightness of the high-resolution low-light image of the first resolution. A method for improving low-light images using resizing including 2. In Paragraph 1, The above convolutional processor is, Each pixel value for illuminance is extracted using the pixel intensity and gradient of the low-light image of the first resolution and the hint image, and A neural network is constructed by stacking the extracted pixel values ​​of the low-light image of the first resolution and the hint image, and A method for improving a low-light image using resizing, characterized by generating an image with improved brightness by adjusting the pixel values ​​of the low-light image of the first resolution through the neural network constructed above.

3. In Paragraph 2, The above convolutional processor is, Based on the stacked neural network constructed above, it is determined whether the pixels of the low-light image of the first resolution have an illumination level below a standard, and A method for improving a low-light image using resizing, characterized in that when the pixel value of the low-light image of the first resolution is an illuminance level below a reference, the pixel values ​​of the low-light image of the first resolution and the hint image are converted into a matrix and matched by the same location, and the pixel values ​​of the low-light image of the first resolution and the hint image are adjusted by performing a convolution operation between the pixel values ​​of the low-light image of the first resolution and the pixel values ​​of the hint image by sector.

4. In Paragraph 3, The above convolutional processor is, A method for improving a low-light image using resizing, characterized by performing the above convolution operation to adjust the pixel values ​​of the low-light image of the first resolution, and if the adjusted pixel values ​​fall below a standard, repeating the above convolution operation until the adjusted pixel values ​​reach a preset standard.

5. In Paragraph 4, With the above illumination improvement algorithm, A method for improving low-light images using resizing with any one of Unet, DCE-Net ("Dual-Channel Enhanced Network"), or PENet (Progressive Enhancement Network).

6. A resizer that generates a low-resolution image by converting a high-resolution, low-light original image into a low-resolution image; An illuminance enhancer that generates an illuminance-enhanced image with increased brightness by adjusting the value of each pixel using a pre-established illuminance enhancement algorithm for the above low-resolution image; An upscaler that generates an enlarged image by increasing the above illumination improvement image to the same size as the original image; and An image processor that improves the brightness of the high-resolution original image by adjusting the pixel values ​​of the original image using the adjusted pixel values ​​of the enlarged image as a hint. A device for improving low-light images using resizing including 7. In Paragraph 6, The above image processor is, Each pixel value for illuminance is extracted using the pixel-by-pixel intensity and gradient of the original image above, and A neural network is constructed by stacking each extracted pixel value of the original image and the enlarged image into a stack, and A device for improving low-light images using resizing, characterized by generating an image with improved brightness by adjusting the pixel values ​​of the original image through the neural network constructed above.

8. In Paragraph 7, The above image processor is, Based on the stacked neural network constructed above, it is determined whether the pixels of the original image have an illumination level below a standard, and A device for improving low-light images using resizing, characterized by converting the pixel values ​​of the original image and the enlarged image into a matrix and matching them by position when the pixel values ​​of the original image are below a reference illuminance, and performing a convolution operation between the pixel values ​​of the original image and the pixel values ​​of the enlarged image by sector to adjust the pixel values ​​of the original image.

9. In Paragraph 8, The above image processor is, A device for improving low-light images using resizing, characterized by performing the convolution operation to adjust the pixel values ​​of the original image, and if the adjusted pixel values ​​fall short of a preset standard, repeating the convolution operation until the adjusted pixel values ​​reach the standard.

10. In Paragraph 9, With the above illumination improvement algorithm, A device for enhancing low-light images using resizing that uses any one of Unet, DCE-Net ("Dual-Channel Enhanced Network"), or PENet (Progressive Enhancement Network).

Images