Deep learning-based image noise reduction device and method

Abstract

The present invention relates to a deep learning-based image noise reduction device and method. The deep learning-based image noise reduction device according to an embodiment of the present invention comprises: a first noise reduction module for acquiring a first image in which noise of an input image having an RGB color space is reduced; a first color space conversion module for acquiring a second image converted from the first image, so as to have a YCbCr color space; a second color space conversion module for acquiring a third image converted from the input image, so as to have only a luminance attribute from among attributes of the YCbCr color space; a second noise reduction module for acquiring a fourth image in which noise of the third image is reduced; an attribute combination module for generating a fifth image in which additional noise is reduced from the first image by using the luminance attribute and chrominance attribute of the second image and the luminance attribute of the fourth image; and a third color space conversion module for acquiring an output image converted from the fifth image, so as to have the RGB color space.

Description

Deep learning-based image noise reduction device and method

[0001] The present invention relates to a deep learning-based image noise reduction device and method, and more particularly, to a deep learning-based image noise reduction device and method that outputs an image in which color noise and luminance noise included in the image are reduced using deep learning.

[0002] As screen resolution and color expression capabilities improve, the demand for improved image quality is increasing, and this demand is being met by image enhancements through denoising, super-resolution, and color correction.

[0003] In particular, denoising is the process of reducing noise in images or videos. It reduces color noise, which is noise that appears randomly in the RGB color space and distorts the color of the image, luminance noise, which mainly occurs under low-light conditions and is expressed as random fluctuations in brightness, and time-dependent noise, which occurs when brightness, color, etc. between adjacent frames do not match overall.

[0004] However, this image enhancement requires a lot of manual work by video editing experts, as experts must manually reduce image noise using video editing tools.

[0005] Accordingly, various studies have recently been conducted to automate image enhancement using deep learning to denoise images and videos. Among them, SwinIR performs image denoising by applying the transformer structure, a module that has been actively researched in the field of deep learning, and NAFNet performs image denoising by removing unnecessary activation functions and modifying the convolutional model structure to suit denoising. However, the transformer model takes a long time to train and inference, and tends to fail to properly handle grayscale / luminance noise, while NAFNet and others suffer from excessive loss of detail.

[0006] The present invention relates to a deep learning-based image noise reduction device and method, and aims to provide a deep learning-based image noise reduction device and method that outputs an image in which color noise and luminance noise included in the image are reduced by using deep learning.

[0007] A deep learning-based image noise reduction device according to one aspect includes: a first noise reduction module for obtaining a first image in which noise of an input image having an RGB color space is reduced; a first color space conversion module for obtaining a second image converted from the first image to have a YCbCr color space; a second color space conversion module for obtaining a third image converted from the input image to have only a luminance attribute among the attributes of the YCbCr color space; a second noise reduction module for obtaining a fourth image in which noise of the third image is reduced; an attribute combination module for generating a fifth image in which additional noise is reduced from the first image by using luminance attributes and chrominance attributes of the second image and luminance attributes of the fourth image; and a third color space conversion module for obtaining an output image in which the fifth image is converted to have an RGB color space.

[0008] At this time, the attribute combination module can produce a new luminance attribute by combining the luminance attribute of the second image and the luminance attribute of the fourth image according to preset conditions, and generate the fifth image that is generated to have the new luminance attribute and the color difference attribute of the second image.

[0009] Additionally, the new luminance property may be a value that is greater than or equal to a smaller value and less than or equal to a larger value among the luminance property of the second image and the luminance property of the fourth image.

[0010] In addition, the attribute combination module can determine the new luminance attribute through the following mathematical expression 1. (Y3 is the new luminance attribute, p is a real number having a value of 0≤p≤1, Y1 is the luminance attribute of the second image, and Y2 is the luminance attribute of the fourth image)

[0011] [Mathematical Formula 1]

[0012] Y3=pY1+(1-p)Y 2.

[0013] Additionally, the first noise reduction module and the second noise reduction module may be models that use the structure of a DRUNet (Dilated-Residual U-Net) model as a backbone.

[0014] In addition, the first noise reduction module may be a module learned using a learning data set that pairs “an original image having an RGB color space and including noise” and “an image from which color noise and luminance noise are reduced from the original image.”

[0015] In addition, the second noise reduction module may be a module trained using a training data set comprising a pair of "an image in which an original image including noise is converted to have only a luminance attribute in a YCbCr color space" and "an image in which luminance noise from the original image is reduced to have only a luminance attribute in a YCbCr color space."

[0016] Alternatively, the first noise reduction module may be a module trained using a training data set that pairs “an image with noise added to an original image having an RGB color space and including noise” and “an image with color noise and luminance noise reduced from the original image.”

[0017] In addition, the second noise reduction module may be a module trained using a training data set comprising a pair of "an image in which noise is added to an original image containing noise and converted to have only a luminance attribute in a YCbCr color space" and "an image in which luminance noise is reduced from the original image and converted to have only a luminance attribute in a YCbCr color space."

[0018] In addition, at least one of the first noise reduction module and the second noise reduction module may be a module learned using a learning data set that pairs “an image with noise added to an original image including noise” and “the original image.”

[0019] According to another aspect, a deep learning-based image noise reduction method is performed by a computing device including a first noise reduction module, a second noise reduction module, a first color space conversion module, a second color space conversion module, an attribute combining module, and a third color space conversion module, the method comprising: obtaining, in the first noise reduction module, a first image in which noise of an input image having an RGB color space is reduced; obtaining, in the first color space conversion module, a second image in which the first image is converted to have a YCbCr color space; obtaining, in the second color space conversion module, a third image in which the input image is converted to have only a luminance attribute among attributes of the YCbCr color space; obtaining, in the second noise reduction module, a fourth image in which noise of the third image is reduced; generating, in the attribute combining module, a fifth image in which additional noise is reduced from the first image by using a luminance attribute and a chrominance attribute of the second image and a luminance attribute of the fourth image; And in the third color space conversion module, a step of obtaining an output image converted from the fifth image to have an RGB color space is included.

[0020] At this time, in the step of acquiring the fifth image, the method may further include a step of calculating a new luminance property by combining the luminance property of the second image and the luminance property of the fourth image according to preset conditions; and a step of generating the fifth image so as to have the new luminance property and the chrominance property of the second image.

[0021] Additionally, in the step of determining the new luminance property, the new luminance property may be a value that is greater than or equal to a smaller value and less than or equal to a larger value among the luminance property of the second image and the luminance property of the fourth image.

[0022] In addition, in the step of determining the new luminance property, the new luminance property can be determined through the following mathematical expression 1 (Y3 is the new luminance property, p is a real number having a value of 0≤p≤1, Y1 is the luminance property of the second image, and Y2 is the luminance property of the fourth image).

[0023] [Mathematical Formula 1]

[0024] Y3=pY1+(1-p)Y 2.

[0025] Additionally, the first noise reduction module and the second noise reduction module may be models that use the structure of a DRUNet (Dilated-Residual U-Net) model as a backbone.

[0026] In addition, the first noise reduction module may be a module learned using a learning data set that pairs “an original image having an RGB color space and including noise” and “an image from which color noise and luminance noise are reduced from the original image.”

[0027] In addition, the second noise reduction module may be a module trained using a training data set comprising a pair of "an image in which an original image including noise is converted to have only a luminance attribute in a YCbCr color space" and "an image in which luminance noise from the original image is reduced to have only a luminance attribute in a YCbCr color space."

[0028] Alternatively, the first noise reduction module may be a module trained using a training data set that pairs “an image with noise added to an original image having an RGB color space and including noise” and “an image with color noise and luminance noise reduced from the original image.”

[0029] In addition, the second noise reduction module may be a module trained using a training data set comprising a pair of "an image converted from an original image containing noise to which other noise has been added so as to have only a luminance attribute in a YCbCr color space" and "an image converted from an image with reduced luminance noise from the original image to have only a luminance attribute in a YCbCr color space."

[0030] In addition, at least one of the first noise reduction module and the second noise reduction module may be a module learned using a learning data set that pairs “an image with noise added to an original image including noise” and “the original image.”

[0031] The present invention has the effect of outputting an image with reduced color noise and luminance noise included in the image.

[0032] In addition, by using the present invention, there is an effect of improving the image quality by further reducing color noise and luminance noise included in the image compared to a structure using a single model.

[0033] FIG. 1 is a diagram illustrating a deep learning-based image noise reduction system according to an embodiment of the present invention;

[0034] Figure 2 is a diagram showing the architecture of the DRUNet model.

[0035] Figure 3 is a diagram showing the architecture of the residual block.

[0036] FIG. 4 is a diagram defining a color space and attribute type corresponding to an image type according to an embodiment of the present invention;

[0037] FIG. 5 is a diagram showing an input image and noise reduction results according to an embodiment of the present invention;

[0038] FIG. 6 is a diagram illustrating a deep learning-based image noise reduction method according to an embodiment of the present invention.

[0039] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the present invention, and the present invention is defined solely by the scope of the claims.

[0040] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like components throughout the specification, and "and / or" includes each and any combination of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another. Therefore, it should be understood that a first component mentioned below may also be a second component within the technical spirit of the present invention.

[0041] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those skilled in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0042] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0043]

[0044] FIG. 1 is a diagram illustrating a deep learning-based image noise reduction system according to an embodiment of the present invention.

[0045] Referring to FIG. 1, a deep learning-based image noise reduction device (1) according to an embodiment of the present invention may include a first noise reduction module (100), a first color space conversion module (200), a second color space conversion module (300), a second noise reduction module (400), an attribute combination module (500), and a third color space conversion module (600).

[0046] A deep learning-based image noise reduction device (1) according to an embodiment of the present invention can input an image corresponding to each frame of an image, and obtain an image with improved image quality by reducing noise included in the input image. Here, the noise included in the image may include color noise and luminance noise, and color noise may be noise in which a color other than the color that should originally appear appears among colors that can be expressed in an RGB color space, and luminance noise may be noise that occurs due to a difference in brightness in a color that should originally appear.

[0047] A deep learning-based image noise reduction device (1) can obtain an image in which color noise and luminance noise are effectively reduced by first reducing color noise and luminance noise from an image and secondarily supplementing the reduced luminance noise.

[0048] The first noise reduction module (100) can obtain a first image with reduced noise of an input image having an RGB color space. The input image has an RGB color space, and the color space is a space that expresses a color system in three dimensions. The RGB color space may be a space in which all colors of the color system are expressed as coordinates for red (RED), green (GREEN), and blue (BLUE), respectively.

[0049] Here, the input image may be an image that has been converted to have a preset number of pixels, such that the target image for noise reduction has a preset number of pixels. For example, the input image may have a pixel size of 128×128 in width×height, but is not limited thereto. That is, the first noise reduction module (100) may be provided with an input image having a preset number of pixels.

[0050] For example, if the number of pixels of the target image for noise reduction is greater than the preset number of pixels, an input image can be generated to match the preset number of pixels by removing some pixels of the target image for noise reduction. Here, some pixels can be removed from the edge of the target image based on the center of the target image to convert the target image to have the preset number of pixels. In addition, some pixels that deviate from a point corresponding to the number of horizontal and vertical pixels based on a vertex of the target image can be removed to convert the target image to have the preset pixel size.

[0051] In addition, if the number of pixels of the target image for noise reduction is greater than the preset number of pixels, the target image can be divided into multiple patches according to the preset number of pixels, and each of the multiple patches can be provided as an input image.

[0052] A patch refers to an individual image that is divided into a target image, and the number of pixels in a patch may be smaller than the number of pixels in the target image. Furthermore, a patch may include a portion of the target image. Here, a patch may be divided from the target image so as not to overlap with at least one other patch, but it may also be divided from the target image so as to overlap with another patch. Furthermore, each of the multiple patches generated by dividing the target image may have the same number of pixels, or may have different numbers of pixels.

[0053] Meanwhile, if the number of pixels of the target image for noise reduction is smaller than the preset number of pixels, the number of pixels of the target image for noise reduction may be increased using the zero padding technique. For example, pixels may be extended from pixels corresponding to the edge of the target image to increase the number of pixels, and a value of 0 may be added to each of the increased pixels. In this way, by increasing the number of pixels of the target image using the zero padding technique, the target image may be converted to have the preset number of pixels. Here, the number of pixels increased may vary depending on the number of pixels of the target image.

[0054] However, although it is described that the input image has a preset number of pixels, the first noise reduction module (100) may be provided with an input image with an unlimited number of pixels. For example, the resolution of the image that can be processed by the first noise reduction module (100) may vary depending on the model applied to the first noise reduction module (100). Accordingly, the number of pixels that the input image has is not limited and may be set in various ways.

[0055] The first noise reduction module (100) can be trained using a learning data set that pairs “an original image with RGB color space and noise” and “an image with reduced color noise and luminance noise from the original image.”

[0056] In another embodiment, the first noise reduction module (100) can be trained using a training data set that pairs “an image with noise added to the original image” and “an original image.”

[0057] In another embodiment, the first noise reduction module (100) can be trained using a training data set that pairs “an image with noise added to the original image” and “an image with color noise and luminance noise reduced from the original image.”

[0058] In particular, the noise added to the original image can be structural noise (e.g., structural noise whose size is determined based on the standard deviation between the "original image" and the "image with noise reduced from the original image").

[0059] Traditionally, random noise was added to original images by adding noise of a specific variance to each channel of the original image. However, in real-world images, structural noise is more prevalent than random noise. Therefore, methods that add random noise often fail to reflect the structural noise patterns of real-world images.

[0060] Accordingly, according to another embodiment of the present invention, structural noise can be added to an original image by adjusting the noise size based on the standard deviation of the original image and the image from which noise has been reduced from the original image.

[0061] In this way, when creating an image with structural noise added to the original image, the performance of the noise reduction module can be expected to be significantly improved compared to creating an image with random noise added using conventional techniques.

[0062] The size of noise added to the original image can be determined using the following mathematical expression 1.

[0063] [Mathematical Formula 1]

[0064] N=α(xy) / σ+y.

[0065] Here, N is the noise added to the original image, α is a value randomly extracted within a specific range, σ is the standard deviation of xy, x can be the RGB attribute value of the original image, and y can be the RGB attribute value of the image with reduced noise from the original image. Here, as the value of α increases, the size of the noise increases, and more noise can be added to the original image.

[0066] Here, the first noise reduction module (100) can respond to various noise levels by learning an image with additional noise added to the original image as input data. That is, even when a low-quality image is input, the learned result can be used to effectively output an image with reduced noise.

[0067] When an input image having an RGB color space is input, the first noise reduction module (100) can output a first image having an RGB color space in which color noise and luminance noise are reduced from the input image using the learned result.

[0068] Additionally, the first noise reduction module (100) may be a model that uses the structure of a DRUNet (Dilated-Residual U-Net) model as a backbone.

[0069] Here, DRUNet (Dilated-Residual U-Net) may be a model with a structure in which some of the standard blocks used in the existing U-NET model are replaced with residual blocks.

[0070] Figure 2 is a diagram showing the architecture of the DRUNet model, and Figure 3 is a diagram showing the architecture of the residual block.

[0071] Referring to FIGS. 2 and 3, the U-NET structure can be a model that can complement the problems of the Transformer model. In the case of the Transformer model, the number of channels is increased and the dimensionality is reduced so that the characteristics of the input image can be captured during the encoding stage, and in the decoding stage, only the information encoded in low dimensions is used to reduce the number of channels and increase the dimensionality to restore the high-dimensional image. However, during the dimensionality reduction in the encoding stage, detailed location information about the image object is lost, and in the decoding stage, there is a problem in that the lost location information cannot be recovered because only low-dimensional information is used.

[0072] The U-NET architecture can complement the shortcomings of the Transformer model by utilizing both low-dimensional and high-dimensional information to accurately identify the location of image objects. Specifically, the U-NET architecture utilizes skip connections, which merge features obtained from each encoding layer into each decoding layer. This reduces the image size while still accurately defining the location of objects, thereby improving image quality with greater accuracy.

[0073] Specifically, the U-NET structure is a neural network structure configured in a U shape, with layers arranged symmetrically on both sides with the center as the reference, and layers in the downsampling process and layers in the upsampling process can be connected to each other through skip connections. Here, each stage of downsampling and upsampling can be composed of standard blocks.

[0074] The DRUNet model is a model that replaces some of the standard blocks of the above U-Net structure with residual blocks. A residual block may be a block that has connections that not only input the result value of one layer to the immediately following layer, but also to the layers several steps later.

[0075] For example, referring to FIG. 3, the input value (x) can be input to the first layer and the third layer. In addition, the output value (F(x)) output from the second layer through the first and second layers can also be input to the third layer. That is, the input value (x) and the output value (F(x)) output from the second layer are input to the third layer, and the third layer can use the sum of the input value (x) and the output value (F(x)) output from the second layer as input and output the result value. In addition, the first noise reduction module (100) reuses the value output from the third layer as an input value for learning, thereby enabling learning with a small amount of data, simplifying the learning structure, and enabling image output with a small amount of computation.

[0076] The first noise reduction module (100) can learn output data with reduced noise for input data containing noise according to the method described above, and when an input image is input, can output a first image with reduced color noise and luminance noise from the input image according to the learned result.

[0077] The first color space conversion module (200) can obtain a second image by converting the first image to have a YCbCr color space. The first image has an RGB color space, and thus may have values ​​for each of the coordinates of R, G, and B. The first color space conversion module (200) converts the color space of the first image from the RGB color space to the YCbCr color space, and the first image converted to the YCbCr color space can be defined as the second image. The YCbCr color space can express a color with one luminance attribute (brightness) and two chrominance attributes (chrominance attribute for blue and chrominance attribute for red).

[0078] For example, the first color space conversion module (200) can convert the color space of the first image from the RGB color space to the YCbCr color space according to the following mathematical expression 2, but is not limited thereto.

[0079] [Equation 2]

[0080] Y = K R ·R + K G ·G + K B ·B,

[0081] C b = ½·(B - Y) / (1 - K B ),

[0082] C r = ½·(R - Y) / (1 - K R ).

[0083] Y is the luminance attribute, and C b is the chrominance attribute for blue, and C r is the chrominance property for red, and K R is the coefficient for red, and K G is the coefficient for green, and K B may be a coefficient for blue. However, the method by which the first color space conversion module (200) converts the RGB color space into the YCbCr color space is not limited thereto.

[0084] The second color space conversion module (300) can obtain a third image by converting the input image to have only the luminance attribute among the attributes of the YCbCr color space. The input image has an RGB color space, and thus may have values ​​for the coordinates of each of R, G, and B. The second color space conversion module (300) can convert the color space of the input image from the RGB color space to the YCbCr color space, but can convert it to have only the luminance attribute (Y), and can define the input image converted to have only the luminance attribute as the third image.

[0085] For example, the second color space conversion module (300) can convert the input image to have only the luminance property according to the following mathematical expression 3, but is not limited thereto.

[0086] [Equation 3]

[0087] Y = K R ·R + K G ·G + K B ·B.

[0088] Y is the luminance attribute, and K R is the coefficient for red, and K G is the coefficient for green, and K B may be a coefficient for blue. However, the method by which the second color space conversion module (300) converts an input image having an RGB color space to have only a luminance attribute is not limited thereto.

[0089] Meanwhile, the first color space conversion module (200) and the second color space conversion module (300) are described as having separate configurations, but may have the same configuration.

[0090]

[0091] The second noise reduction module (400) can obtain a fourth image with reduced noise from the third image.

[0092] The third image is an image that has only luminance properties in the YCbCr color space.

[0093] The second noise reduction module (400) can output a fourth image with reduced luminance noise in the YCbCr color space for the input third image.

[0094] Accordingly, the fourth image also has only luminance properties in the YCbCr color space.

[0095] The second noise reduction module (400) can be trained using a training data set that pairs “an image in which an original image containing noise is converted to have only a luminance attribute in the YCbCr color space” and “an image in which luminance noise from the original image is reduced to have only a luminance attribute in the YCbCr color space.”

[0096] In another embodiment, the second noise reduction module (400) can be trained using a training data set that pairs “an image with noise added to the original image” and “an original image.”

[0097] In another embodiment, the second noise reduction module (400) can be trained using a training data set that pairs “an image in which noise is added to an original image containing noise and converted to have only a luminance attribute in the YCbCr color space” and “an image in which luminance noise is reduced from the original image and converted to have only a luminance attribute in the YCbCr color space.”

[0098] In particular, the noise added to the original image can be structural noise (e.g., structural noise whose size is determined based on the standard deviation between the "original image" and the "image with noise reduced from the original image").

[0099] Traditionally, random noise was added to original images by adding noise of a specific variance to each channel of the original image. However, in real-world images, structural noise is more prevalent than random noise. Therefore, methods that add random noise often fail to reflect the structural noise patterns of real-world images.

[0100] Accordingly, according to another embodiment of the present invention, structural noise can be added to an original image by adjusting the noise size based on the standard deviation of the original image and the image from which noise has been reduced from the original image.

[0101] In the real world, structured noise is common in images. Therefore, models trained with training data degraded by adding structured noise can effectively reduce noise that may actually be present in images. In contrast, adding random noise, as in conventional techniques, does not significantly improve noise removal performance.

[0102] Therefore, when generating an image with structural noise added to the original image in this way, it is expected that the performance of the noise reduction module will be significantly improved compared to generating an image with random noise added using conventional techniques.

[0103] The size of the noise added to the original image can be determined using the following mathematical formula 4.

[0104] [Equation 4]

[0105] N=α(xy) / σ+y.

[0106] Here, N is the noise added to the original image, α is a value randomly extracted within a specific range, σ is the standard deviation of xy, x can be the luminance attribute value of the original image, and y can be the luminance attribute value of the image with reduced noise from the original image. Here, as the value of α increases, the size of the noise increases, and more noise can be added to the original image.

[0107] Here, the second noise reduction module (100) can respond to various noise levels by learning an image with additional noise added to the original image as input data. That is, even when a low-quality image is input, the learned result can be used to effectively output an image with reduced noise.

[0108] When a third image having only a luminance attribute is input, the second noise reduction module (400) can output a fourth image having only a luminance attribute, with luminance noise reduced from the third image, using the learned result.

[0109] Additionally, the second noise reduction module (400) may be a model that uses the structure of the DRUNet (Dilated-Residual U-Net) model as a backbone.

[0110] Meanwhile, there is a difference in that the first noise reduction module (100) is a noise reduction module learned to reduce image noise in the RGB color space, while the second noise reduction module (400) is a noise reduction module learned to reduce image noise (particularly, luminance noise) in the YCbCr color space.

[0111] The attribute combination module (500) can generate a fifth image with reduced additional noise from the first image by using the luminance attribute and chrominance attribute of the second image and the luminance attribute of the fourth image.

[0112] The attribute combination module (500) can combine the luminance attribute of the second image and the luminance attribute of the fourth image according to preset conditions to produce a new luminance attribute, and can generate a fifth image to have the produced new luminance attribute and the chrominance attribute of the second image.

[0113] Specifically, the attribute combination module (500) can produce a new luminance attribute through the following mathematical expression 5.

[0114] [Equation 5]

[0115] Y3=pY1+(1-p)Y2.

[0116] Y3 may be a new luminance attribute, p may be a real number having a value of 0≤p≤1, Y1 may be a luminance attribute of the second image, and Y2 may be a luminance attribute of the fourth image. That is, the new luminance attribute may be a value that is greater than or equal to a smaller value and less than or equal to a larger value among the luminance attribute of the second image and the luminance attribute of the fourth image.

[0117] Here, p is a coefficient determined between 0 and 1, and can have a value of 0 or 1.

[0118] When p=0, the new luminance property can have the same luminance property as the luminance property of the fourth image, and when p=1, the new luminance property can have the same luminance property as the luminance property of the second image.

[0119] Additionally, if p is a real number between 0 and 1 rather than 0 or 1, the new luminance attribute may have a luminance attribute that is a combination of the luminance attribute of the second image and the luminance attribute of the fourth image in a certain ratio.

[0120] Meanwhile, the p value can be determined by the user, but is not limited thereto.

[0121] The third color space conversion module (600) can obtain an output image converted from the fifth image to have an RGB color space. The fifth image has a YCbCr color space, and thus may have values ​​for each of the Y, Cb, and Cr coordinates. The third color space conversion module (600) converts the color space of the fifth image from the YCbCr color space to the RGB color space, and the fifth image converted to the RGB color space can be defined as an output image.

[0122] For example, the third color space conversion module (600) can convert the color space of the fifth image from the YCbCr color space to the RGB color space according to the following mathematical expression 6, but is not limited thereto.

[0123] [Equation 6]

[0124] B = Y + 2·C b ·(1 - K B ),

[0125] R = Y + 2·C r ·(1 - K R ),

[0126] G = (Y - K R ·R - K B ·B) / K G .

[0127] Y is the luminance attribute, Cb is the chrominance attribute for blue, Cr is the chrominance attribute for red, and K R is the coefficient for red, and K G is the coefficient for green, and K B may be a coefficient for blue. However, the method by which the third color space conversion module (600) converts the YCbCr color space into the RGB color space is not limited thereto.

[0128] Meanwhile, the first color space conversion module (200), the second color space conversion module (300), and the third color space conversion module (600) are described as separate configurations, but may be the same configuration.

[0129] FIG. 4 is a diagram defining a color space and attribute type corresponding to an image type according to an embodiment of the present invention.

[0130] Referring to Fig. 4, the input image may have an RGR color space as an image input to the first noise reduction module (100). That is, the input image may have values ​​of R, G, and B as attributes, and each value may be R 1, G 1, It could be B1.

[0131] The first image is an image output from the first noise reduction module (100) with noise reduced from the input image, and the first image may have an RGB color space that is the same as the color space of the input image. That is, the input image may have values ​​of R, G, and B as attributes, and each value may be R 2, G 2, It could be B2.

[0132] The second image is an image output from the first color space conversion module (200), and the first color space conversion module (200) can output the second image by converting the first image to have the YCbCr color space. That is, the second image can have the values ​​of Y, Cb, and Cr as attributes, and each value is Y 1, Cb 1, It could be Cr1.

[0133] The third image is an image output from the second color space conversion module (300), and the second color space conversion module (300) can output the third image by converting the input image to have only the luminance attribute among the attributes of the YCbCr color space. That is, the third image can have the Y value as an attribute, and the corresponding value can be Y2.

[0134] The fourth image is an image output from the third image with reduced noise from the second noise reduction module (400), and the fourth image may have a YCbCr color space identical to the color space of the third image. That is, the fourth image may have a Y value as an attribute, and the corresponding value may be Y3.

[0135] The fifth image is an image output from the attribute combination module (500), and the attribute combination module (500) can calculate a new luminance attribute using the luminance attribute of the second image and the luminance attribute of the fourth image, and can generate and output a fifth image having the new luminance attribute and the chrominance attribute of the second image. That is, the fifth image can have values ​​of Y, Cb, and Cr as attributes, and each value can be Y 4,Cb 1, It could be Cr1.

[0136] The output image is an image output from the third color space conversion module (600), and the third color space conversion module (600) can output the output image by converting the fifth image to have an RGB color space. That is, the output image can have each of the values ​​of R, G, and B as attributes, and each value can be R 3, G 3, It could be B3.

[0137] Figure 5 is a diagram showing an input image and noise reduction results according to an embodiment of the present invention.

[0138] At this time, Fig. 5(a) is a drawing showing an input image, Fig. 5(b) is a drawing showing a first image, and Fig. 5(c) is a drawing showing an output image.

[0139] Referring to Figure 5, it can be seen that the input image contains color noise and luminance noise, and that the first image has reduced color noise and luminance noise compared to the input image. In addition, it can be seen that the output image has further reduced luminance noise compared to the first image.

[0140] Therefore, according to an embodiment of the present invention, it is possible to obtain an image in which color noise and luminance noise are effectively reduced by first reducing color noise and luminance noise from an input image and secondarily supplementing the reduced luminance noise.

[0141] FIG. 6 is a diagram illustrating a deep learning-based image noise reduction method according to an embodiment of the present invention.

[0142] Referring to FIG. 6, the first noise reduction module (100) can obtain a first image with reduced noise of an input image having an RGB color space (S100). The first noise reduction module (100) may be a model that uses the structure of a DRUNet (Dilated-Residual U-Net) model as its backbone.

[0143] The first noise reduction module (100) can be trained using a learning data set that pairs “an original image with RGB color space and noise” and “an image with reduced color noise and luminance noise from the original image.”

[0144] In addition, the first noise reduction module (100) can be trained using a learning data set that pairs “an original image with noise added to the original image” and “an image with color noise and luminance noise reduced from the original image.”

[0145] In another embodiment, the first noise reduction module (100) can be trained using a training data set that pairs “an image with noise added to the original image” and “an original image.”

[0146] In another embodiment, the first noise reduction module (100) can be trained using a training data set that pairs “an image with noise added to the original image” and “an image with color noise and luminance noise reduced from the original image.”

[0147] The input image has an RGB color space, and when an input image having an RGB color space is input, the first noise reduction module (100) can output a first image having reduced color noise and luminance noise in the input image having an RGB color space using the learned result.

[0148] The first color space conversion module (200) can obtain a second image by converting the first image to have a YCbCr color space (S200). The first image has an RGB color space, and the first color space conversion module (200) can convert the color space of the first image from the RGB color space to the YCbCr color space.

[0149] The second color space conversion module (300) can obtain a third image by converting the input image to have only the luminance attribute among the attributes of the YCbCr color space (S300). The input image has an RGB color space, and the second color space conversion module (300) can convert the color space of the input image from the RGB color space to the YCbCr color space, but convert it to have only the luminance attribute.

[0150] The second noise reduction module (400) can obtain a fourth image with reduced noise from the third image (S400). The second noise reduction module (400) may be a model that uses the structure of the DRUNet (Dilated-Residual U-Net) model as its backbone.

[0151] The second noise reduction module (400) can be trained using a training data set that pairs “an image in which an original image containing noise is converted to have only a luminance attribute in the YCbCr color space” and “an image in which luminance noise from the original image is reduced to have only a luminance attribute in the YCbCr color space.”

[0152] In another embodiment, the second noise reduction module (400) can be trained using a training data set that pairs “an image with noise added to the original image” and “an original image.”

[0153] In another embodiment, the second noise reduction module (400) can be trained using a training data set that pairs “an image in which noise is added to an original image containing noise and converted to have only a luminance attribute in the YCbCr color space” and “an image in which luminance noise is reduced from the original image and converted to have only a luminance attribute in the YCbCr color space.”

[0154] The third image has a YCbCr color space, and when the second noise reduction module (400) inputs a third image having a YCbCr color space, it can output a fourth image having reduced luminance noise in the third image having a YCbCr color space using the learned result.

[0155] The attribute combination module (500) can obtain a fifth image with reduced additional noise from the first image by using the luminance attribute and chrominance attribute of the second image and the luminance attribute of the fourth image (S500).

[0156] The attribute combination module (500) can combine the luminance attribute of the second image and the second luminance attribute of the fourth image according to preset conditions to produce a new luminance attribute, and the new luminance attribute can be determined through the following mathematical expression 7.

[0157] [Equation 7]

[0158] Y3=pY1+(1-p)Y2.

[0159] Y3 is a new luminance attribute, p is a real number having a value of 0≤p≤1, Y1 is a luminance attribute of the second image, and Y2 can be a luminance attribute of the fourth image. That is, the new luminance attribute can be a value that is greater than or equal to a smaller value and less than or equal to a larger value among the luminance attribute of the second image and the luminance attribute of the fourth image. Here, p is a coefficient determined between 0 and 1 and can have a value of 0 or 1. When p=0, the new luminance attribute can have the same luminance attribute as the luminance attribute of the fourth image, and when p=1, the new luminance attribute can have the same luminance attribute as the luminance attribute of the second image. In addition, when p is a real number between 0 and 1 rather than 0 or 1, the new luminance attribute can have a luminance attribute that is a combination of the luminance attribute of the second image and the luminance attribute of the fourth image according to a certain ratio. Meanwhile, the value of p can be determined by the user, but is not limited thereto.

[0160] The third color space conversion module (600) can obtain an output image converted from the fifth image to have an RGB color space (S600). The fifth image has a YCbCr color space, and the third color space conversion module (600) can convert the color space of the fifth image from the YCbCr color space to the RGB color space.

[0161] As described above, according to an embodiment of the present invention, a deep learning-based image noise reduction method and device that outputs an image with reduced color noise and luminance noise included in the image can be realized using deep learning.

[0162] While the embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

[0163]

[0164] [Explanation of symbols]

[0165] 100: 1st noise reduction module

[0166] 200: First color space conversion module

[0167] 300: Second color space conversion module

[0168] 400: Second noise reduction module

[0169] 500: Attribute Combination Module

[0170] 600: Third color space conversion module

Claims

1. A first noise reduction module for obtaining a first image with reduced noise of an input image having an RGB color space; A first color space conversion module for obtaining a second image by converting the first image to have a YCbCr color space; A second color space conversion module that obtains a third image by converting the input image to have only the luminance attribute among the attributes of the YCbCr color space; A second noise reduction module for obtaining a fourth image with reduced noise of the third image; An attribute combination module that generates a fifth image with reduced additional noise from the first image by using the luminance attribute and chrominance attribute of the second image and the luminance attribute of the fourth image; and A deep learning-based image noise reduction device, comprising a third color space conversion module that obtains an output image converted from the fifth image to have an RGB color space.

2. In paragraph 1, The above attribute binding module is, A new luminance property is derived by combining the luminance property of the second image and the luminance property of the fourth image according to preset conditions, A deep learning-based image noise reduction device that generates the fifth image that has the new luminance property and the color difference property of the second image.

3. In paragraph 2, A deep learning-based image noise reduction device, wherein the new luminance attribute is a value greater than or equal to a smaller value and less than or equal to a larger value among the luminance attribute of the second image and the luminance attribute of the fourth image.

4. In paragraph 2, The above attribute binding module is, The new luminance property is determined using the following mathematical expression 1. [Mathematical Formula 1] Y3=pY1+(1-p)Y 2. (Y3 is the new luminance attribute, p is a real number having a value of 0≤p≤1, Y1 is the luminance attribute of the second image, and Y2 is the luminance attribute of the fourth image) A deep learning-based image noise reduction device.

5. In paragraph 1, A deep learning-based image noise reduction device, wherein the first noise reduction module and the second noise reduction module are models that use the structure of a DRUNet (Dilated-Residual U-Net) model as a backbone.

6. In paragraph 1, The above first noise reduction module, A deep learning-based image noise reduction device, which is a module trained using a learning data set comprising a pair of "an original image having an RGB color space and including noise" and "an image from which color noise and luminance noise are reduced from the original image." 7. In paragraph 1, The above second noise reduction module, A deep learning-based image noise reduction device, which is a module trained using a training data set comprising a pair of "an image converted from an original image containing noise to have only a luminance attribute in a YCbCr color space" and "an image converted from an image with reduced luminance noise from the original image to have only a luminance attribute in a YCbCr color space." 8. In paragraph 1, The above first noise reduction module, A deep learning-based image noise reduction device, which is a module trained using a learning data set comprising a pair of "an image with noise added to an original image having an RGB color space" and "an image with reduced color noise and luminance noise from the original image." 9. In paragraph 1, The above second noise reduction module, A deep learning-based image noise reduction device, which is a module trained using a training data set comprising a pair of "an image converted from an original image containing noise to an image with added noise so that it has only a luminance attribute in a YCbCr color space" and "an image converted from an image with reduced luminance noise from the original image to have only a luminance attribute in a YCbCr color space." 10. In paragraph 1, At least one of the first noise reduction module and the second noise reduction module, A deep learning-based image noise reduction device, which is a module trained using a learning data set that pairs “an image with noise added to an original image containing noise” and “the original image.” 11. A deep learning-based image noise reduction method performed by a computing device including a first noise reduction module, a second noise reduction module, a first color space conversion module, a second color space conversion module, an attribute combination module, and a third color space conversion module, In the first noise reduction module, a step of obtaining a first image with reduced noise of an input image having an RGB color space; In the first color space conversion module, a step of obtaining a second image converted into the first image to have a YCbCr color space; In the second color space conversion module, a step of obtaining a third image by converting the input image to have only the luminance attribute among the attributes of the YCbCr color space; In the second noise reduction module, a step of obtaining a fourth image with reduced noise of the third image; In the above attribute combination module, a step of generating a fifth image with reduced additional noise from the first image by using the luminance attribute and chrominance attribute of the second image and the luminance attribute of the fourth image; and A deep learning-based image noise reduction method, comprising: a step of obtaining an output image obtained by converting the fifth image to have an RGB color space in the third color space conversion module.

12. In paragraph 11, In the step of acquiring the fifth image, A step of calculating a new luminance property by combining the luminance property of the second image and the luminance property of the fourth image according to preset conditions; and A deep learning-based image noise reduction method, further comprising: a step of generating the fifth image so as to have the new luminance property and the chrominance property of the second image.

13. In paragraph 12, In the step of determining the new luminance property, A deep learning-based image noise reduction method, wherein the new luminance attribute is a value greater than or equal to a smaller value and less than or equal to a larger value among the luminance attribute of the second image and the luminance attribute of the fourth image.

14. In paragraph 11, In the step of determining the new luminance property, The new luminance property is determined using the following mathematical expression 1. [Mathematical Formula 1] Y3=pY1+(1-p)Y 2. (Y3 is the new luminance attribute, p is a real number having a value of 0≤p≤1, Y1 is the luminance attribute of the second image, and Y2 is the luminance attribute of the fourth image) A deep learning-based image noise reduction method.

15. In paragraph 11, A deep learning-based image noise reduction method, wherein the first noise reduction module and the second noise reduction module are models that use the structure of a DRUNet (Dilated-Residual U-Net) model as a backbone.

16. In paragraph 11, The above first noise reduction module, A deep learning-based image noise reduction method, which is a module learned using a learning data set comprising a pair of "original images with RGB color space and noise" and "images from which color noise and luminance noise are reduced from the original images." 17. In paragraph 11, The above second noise reduction module, A deep learning-based image noise reduction method, which is a module learned using a learning data set comprising a pair of "an image converted from an original image containing noise to have only a luminance attribute in a YCbCr color space" and "an image converted from an image with reduced luminance noise from the original image to have only a luminance attribute in a YCbCr color space." 18. In paragraph 11, The above first noise reduction module, A deep learning-based image noise reduction method, which is a module learned using a learning data set comprising a pair of "an image with noise added to an original image having an RGB color space" and "an image with reduced color noise and luminance noise from the original image." 19. In paragraph 11, The above second noise reduction module, A deep learning-based image noise reduction method, which is a module trained using a training data set comprising a pair of "an image converted to have only a luminance attribute in the YCbCr color space by adding other noise to an original image containing noise" and "an image converted to have only a luminance attribute in the YCbCr color space by reducing luminance noise from the original image." 20. In paragraph 11, At least one of the first noise reduction module and the second noise reduction module, A deep learning-based image noise reduction method, which is a module learned using a learning data set that pairs “an image with noise added to an original image containing noise” and “the original image.”

Images