Method for removing rgbir image noise and apparatus thereof

Abstract

A method for removing RGBIR image noise according to an embodiment of the present disclosure comprises the steps of: acquiring an IR image and an RGB image including an IR component by demosaicing an RGBIR image; reconstructing a first RGB image from which the IR component is removed by removing the IR component from the RGB image including the IR component; and removing noise from the first RGB image on the basis of the first RGB image to which a predetermined RGB weight is applied and the IR image to which a predetermined IR weight is applied.

Description

RGBIR image noise removal method and device thereof

[0001] The present disclosure relates to a technology for removing noise from an RGBIR image, and more specifically, to a method and device for removing noise from an RGBIR image that reduces or removes noise generated during a color restoration process of an RGBIR image.

[0002] Unlike typical RGB cameras, RGBIR cameras capture near-infrared (NIR, hereinafter referred to as IR) images in addition to visible-light RGB images. To achieve this, RGBIR sensors feature IR pixels in addition to the standard R, G, and B pixels. Specifically, these R, G, and B pixels are optically designed to simultaneously respond to IR light in addition to the standard R, G, and B color bands.

[0003] In the case of an RGBIR camera that has a spectral response component with correlation between RGBIR channels, the IR (Infrared) component is added to the pixel values ​​of each R, G, and B channel of the captured raw input image.

[0004] In the case of channels 1, 2, and 3 of a 4-channel input raw image input through an RGBIR color filter array, since they contain IR components in addition to each of the R, G, and B components, they can be expressed as R+IR, G+IR, B+IR channels, or R+, G+, and B+ channels. However, for the convenience of the following explanation, if the R, G, and B channels contain IR components, they will be expressed as R+, G+, and B+, and in the case of a color image consisting of three channels of R+, G+, and B+, they will be expressed as an RGB+ image.

[0005] In order to obtain a general RGB image from such raw input RGB+ image, the RGB+ image undergoes a color restoration process that removes the IR component and removes the correlation between the R, G, and B channels. At this time, when the IR component of the lighting is weak, the image quality is relatively not bad compared to the existing RGB camera, but when the IR component of the lighting is strong, there is a problem that the noise of the image becomes worse during the process of removing the low-resolution IR component and the color restoration and signal amplification process using the relatively small (or dark) RGB components remaining after the IR component is removed.

[0006] Conventional techniques to solve this problem have used methods such as performing complex NR (Noise Reduction) at the ISP (image signal processor) stage, appropriately merging RGB+ images or IR images interpolated from the original image with images converted from color-restored dark noisy RGB images to HSV color space, YCbCr color space, etc., or performing guided filtering using the color-restored noisy RGB image as a guide image and the RGB+ images or IR images interpolated from the original image.

[0007] However, conventional technologies using complex NR have the problem that the hardware (HW) size increases when implemented as an ISP, it is difficult to integrate it as a post processor at the sensor level, and the chip unit cost increases when implementing a product.

[0008] Conventional techniques that process in the HSV, YCbCr domain require an additional color space conversion process, are difficult to determine and tune intuitively and quantitatively weights, have problems with Hue or Cb, Cr values ​​being distorted during the processing, and have problems with natural processing linked to the WB processing unit because it is difficult to define the quantitative relationship between white balance (WB) and the lighting components of the scene.

[0009] Conventional techniques using guided filtering have the disadvantages of being difficult to determine tuning parameters such as kernel size and epsilon for each scene, the optimal parameter values ​​are locally different even within a scene, resulting in non-uniform quality of the resulting image, and even if the parameters are well adjusted in complex texture areas such as leaves, the image tends to be relatively blurry, requiring a lot of additional logic, and requiring a relatively large hardware size.

[0010] The technical problem of the present disclosure is to provide a method and device for removing RGBIR image noise that reduces or removes noise occurring in the process of restoring color of an RGBIR image.

[0011] The technical problem of the present disclosure is to provide a method and device for removing RGBIR image noise, which can remove or reduce color noise in an RGBRIR image by performing tuning intuitively and quantitatively without an unnecessary color space conversion process by processing in the RGB domain.

[0012] The technical problem of the present disclosure is to provide an RGBIR image noise removal method and device capable of removing false colors included in an image by calculating pixel weights for each pixel of an RGB image and an IR image and applying the pixel weights to remove noise in the RGB image.

[0013] The technical problem of the present disclosure is to provide an RGBIR image noise removal method and device capable of removing noise from an RGBIR image according to a white balance result by removing noise by adjusting the weights of IR components for each R, G, and B channel according to the illumination component evaluated or estimated from the white balance result when performing white balance, thereby removing noise.

[0014] A method for removing noise from an RGBIR image according to one embodiment of the present disclosure comprises the steps of: demosaicing an RGBIR image to obtain an RGB image including an IR component and an IR image; removing the IR component from the RGB image including the IR component to restore a first RGB image from which the IR component has been removed; and removing noise from the first RGB image based on the first RGB image to which a predetermined RGB weight is applied and the IR image to which a predetermined IR weight is applied.

[0015] According to one embodiment, the RGB weight and the IR weight may be determined based on the brightness of an RGB image including the IR component, the brightness of the IR image, and the brightness of the first RGB image.

[0016] In one embodiment, the RGB weights and the IR weights are gain RGB = -(IR norm / RGB norm ) × gain IR + gain max is determined by a point on a straight line expressed by the formula of the gain RGB represents the RGB weights, and the gain IR represents the above IR weight, and the above IR norm represents the brightness of the IR image, and the RGB normrepresents the brightness of the first RGB image, and the gain max can represent the ratio of the brightness of the RGB image including the IR component and the brightness of the first RGB image.

[0017] According to one embodiment, the RGB weight and the IR weight are determined by a preset tuning parameter alpha corresponding to a ratio of an IR weight determined by a point on the straight line and a maximum IR weight on the straight line, and the tuning parameter alpha can be tuned to a value between '0' and '1'.

[0018] According to one embodiment, the removing step may remove noise from the first RGB image by duplicating the IR image into a three-channel IR image for each of the R, G, and B channels of the first RGB image, applying the IR weight to the three-channel IR image, and then adding the three-channel IR image to which the IR weight is applied to the first RGB image to which the RGB weight is applied.

[0019] According to one embodiment, the removing step may include estimating an illumination component of the RGBIR image, determining an illumination weight for each of the R, G, and B channels based on the estimated illumination component, applying the illumination weight for each of the R, G, and B channels to each of the three-channel IR images, and then applying the IR weight.

[0020] According to one embodiment, the removing step may calculate weights for each pixel of the first RGB image and the IR image using the RGB weights, the IR weights, and a preset IR factor, and remove noise from the first RGB image based on the first RGB image to which the pixel-wise weights of the first RGB image are applied and the IR image to which the pixel-wise weights of the IR image are applied.

[0021] According to one embodiment, the IR factor may be determined by a first value determined based on the IR component and the brightness component of each pixel of an RGB image including the IR component, and predetermined tuning parameters.

[0022] According to one embodiment, the IR factor is determined by adding a second tuning parameter to a product of the first value and the first tuning parameter, and may be determined as '0' when the IR factor is less than '0' and '1' when the IR factor is greater than '1'.

[0023] According to another embodiment of the present disclosure, an RGBIR image noise removal device includes: an acquisition unit that demosaices an RGBIR image to obtain an RGB image including an IR component and an IR image; a restoration unit that removes the IR component from the RGB image including the IR component to restore a first RGB image from which the IR component has been removed; and a removal unit that removes noise from the first RGB image based on the first RGB image to which a predetermined RGB weight is applied and the IR image to which a predetermined IR weight is applied.

[0024] According to one embodiment, the RGB weight and the IR weight may be determined based on the brightness of an RGB image including the IR component, the brightness of the IR image, and the brightness of the first RGB image.

[0025] In one embodiment, the RGB weights and the IR weights are gainRGB = -(IR norm / RGB norm ) × gain IR + gain max is determined by a point on a straight line expressed by the formula of the gain RGB represents the RGB weights, and the gain IR represents the above IR weight, and the above IR normrepresents the brightness of the IR image, and the RGB norm represents the brightness of the first RGB image, and the gain max can represent the ratio of the brightness of the RGB image including the IR component and the brightness of the first RGB image.

[0026] According to one embodiment, the RGB weight and the IR weight are determined by a preset tuning parameter alpha corresponding to a ratio of an IR weight determined by a point on the straight line and a maximum IR weight on the straight line, and the tuning parameter alpha can be tuned to a value between '0' and '1'.

[0027] According to one embodiment, the removal unit can remove noise from the first RGB image by replicating the IR image into three-channel IR images for each of the R, G, and B channels of the first RGB image, applying the IR weight to the three-channel IR images, and then adding the three-channel IR images to which the IR weights are applied to the first RGB image to which the RGB weights are applied.

[0028] According to one embodiment, the removal unit estimates an illumination component of the RGBIR image, determines an illumination weight for each of the R, G, and B channels based on the estimated illumination component, and applies the illumination weight for each of the R, G, and B channels to each of the three-channel IR images, and then applies the IR weight.

[0029] According to one embodiment, the removal unit may calculate weights for each pixel of the first RGB image and the IR image using the RGB weights, the IR weights, and a preset IR factor, and remove noise from the first RGB image based on the first RGB image to which the pixel-wise weights of the first RGB image are applied and the IR image to which the pixel-wise weights of the IR image are applied.

[0030] According to one embodiment, the IR factor may be determined by a first value determined based on the IR component and the brightness component of each pixel of an RGB image including the IR component, and predetermined tuning parameters.

[0031] According to one embodiment, the IR factor is determined by adding a second tuning parameter to a product of the first value and the first tuning parameter, and may be determined as '0' when the IR factor is less than '0' and '1' when the IR factor is greater than '1'.

[0032] According to the present disclosure, noise occurring in the process of color restoration of an RGBIR image can be reduced or eliminated, and color noise can be removed or reduced in an RGBRIR image by performing tuning intuitively and quantitatively without an unnecessary color space conversion process by processing in the RGB domain.

[0033] According to the present disclosure, noise can be removed from an RGBIR image according to a white balance result by adjusting the weights of IR components for each R, G, and B channel according to an illumination component evaluated or estimated from a white balance result when performing white balance, thereby removing noise.

[0034] According to the present disclosure, by calculating pixel weights for each pixel of an RGB image and an IR image and applying the pixel weights to remove noise in the RGB image, false colors included in the image can be removed.

[0035] According to the present disclosure, it is possible to implement with a small hardware size, maintain the Hue (color) value of the color restored in the noise mitigation process as much as possible, and when processed in the RGB domain and integrated into the sensor as a post-processing block, it is easy to re-mosaic the RGB Bayer so that it can be used in an existing RGB ISP.

[0036] According to the present disclosure, a trade-off between color and noise is possible in RGBIR color reproduction, and tuning is possible to an optimal setting in the trade-off relationship between color and noise by adjusting the global parameter (alpha) and local parameters (Beta1, Beta2).

[0037] According to the present disclosure, the reduction from 4 channels to 3 channels does not require major modifications to the subsequent 3-channel ISP pipeline, and the structure is simpler than that of an RGBIR ISP solution requiring separate RGB and IR passes, thereby reducing the gate count.

[0038] According to the present disclosure, since an area with a lot of IR can be expressed as gray and an area with visible light can be expressed as color, visibility can be improved by seamlessly expressing RGB local areas and IR local areas, and transition between IR images and RGB images can be performed naturally without distinction between RGB mode and IR mode.

[0039] FIG. 1 is a flowchart illustrating an operation of an RGBIR image noise removal method according to an embodiment of the present disclosure.

[0040] FIG. 2 illustrates an operational flowchart of one embodiment of step S140 illustrated in FIG. 1.

[0041] Figure 3 shows a straight line graph for RGB weights and IR weights.

[0042] Figure 4 shows an example diagram for explaining the tuning parameter alpha in the straight line graph of Figure 3.

[0043] Figure 5 shows an example diagram to explain false colors generated by applying global weights.

[0044] Figures 6a and 6b illustrate an example diagram for explaining local RGB weights and local IR weights.

[0045] Figure 7 shows an example of an image in which noise has been removed through application of local weights.

[0046] Figure 8 shows an example diagram for explaining airlight.

[0047] Figure 9 shows an example of a Noisy RGB image with white balance performed and a CNA RGB image with noise removed.

[0048] Figure 10 shows an example of a Noisy RGB image without white balance and a CNA RGB image with noise removed.

[0049] FIG. 11 illustrates a configuration of an RGBIR image noise removal device according to another embodiment of the present disclosure.

[0050] Fig. 12 shows the configuration of one embodiment for explaining the removal part of Fig. 11.

[0051] FIG. 13 is a block diagram of a computing system for executing an RGBIR image noise removal method according to an embodiment of the present disclosure.

[0052] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0053] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.

[0054] In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.

[0055] In this disclosure, terms such as first, second, etc. are used solely to distinguish one component from another, and do not limit the order or importance of components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0056] In this disclosure, distinct components are used only to clearly illustrate their respective characteristics and do not necessarily imply separation. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0057] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also within the scope of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also within the scope of the present disclosure.

[0058] In the present disclosure, expressions of positional relationships used in the present specification, such as top, bottom, left, right, etc., are described for convenience of explanation, and when the drawings illustrated in the present specification are viewed in reverse, the positional relationships described in the specification may be interpreted in the opposite way.

[0059] In the present disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0060] The embodiments of the present disclosure are intended to reduce or remove noise occurring in the process of color restoration of an RGBIR image, and the gist thereof is to remove color noise from an RGBRIR image by performing tuning intuitively and quantitatively without an unnecessary color space conversion process by processing in the RGB domain.

[0061] Embodiments of the present disclosure can remove false colors included in an image by calculating pixel weights for each pixel of an RGB image (Noisy RGB image) and an IR image, and applying the pixel weights to remove noise in the RGB image.

[0062] Embodiments of the present disclosure can remove noise from an RGBIR image according to a white balance result by adjusting the weights of IR components for each R, G, and B channel according to an illumination component evaluated or estimated from a white balance result when performing white balance, thereby removing noise.

[0063] Methods and devices according to embodiments of the present disclosure are described with reference to FIGS. 1 to 12.

[0064] FIG. 1 is a flowchart illustrating an operation of an RGBIR image noise removal method according to an embodiment of the present disclosure, and is a flowchart illustrating an operation in an RGBIR image noise removal device.

[0065] Referring to FIG. 1, a method for removing noise from an RGBIR image according to an embodiment of the present disclosure demosaicing an RGBIR image, for example, an RGBIR Bayer image, to obtain an RGB+ image including an IR component and an IR image, and removing the IR component from the RGB+ image using the IR image, thereby restoring an RGB color-restored Noisy RGB image (S110, S120).

[0066] Here, step S110 can obtain an RGB+ image by separating the RGB+ image and IR components from the RGBIR Bayer image, demosaicing the separated RGB+ image, and then restoring the separated IR image to obtain an IR image. These technical details are self-evident to those skilled in the art, so a detailed description thereof is omitted.

[0067] The RGB+ image obtained by step S110 may include images of the R+ channel, the G+ channel, and the B+ channel, and the IR component is removed by step S120 and the RGB color is restored, thereby restoring or obtaining a Noisy RGB image.

[0068] When the Noisy RGB image and the IR image are acquired or restored by steps S110 and S120, a preset RGB weight is applied to the Noisy RGB image, a preset IR weight is applied to the IR image, and then noise is removed from the Noisy RGB image using the Noisy RGB image to which the RGB weight is applied and the IR image to which the IR weight is applied (S130, S140).

[0069] According to an embodiment, step S130 may equally apply preset RGB weights to all pixels of the Noisy RGB image, and may equally apply preset IR weights to all pixels of the IR image. The RGB weights and IR weights applied in this manner may be understood as global weights applied to all pixels. For example, the RGB weights applied in step S130 may be expressed as global RGB weights, and the IR weights may be expressed as global IR weights. Here, the RGB weights and IR weights may be determined based on the brightness of the RGB+ image, the brightness of the IR image, and the brightness of the Noisy RGB image.

[0070] According to an embodiment, the RGB weight and the IR weight may be determined by a point on a predetermined straight line with the RGB weight and the IR weight as variables, and may be determined by a preset tuning parameter alpha corresponding to a ratio of the IR weight determined by the point on the straight line to the maximum IR weight on the straight line, and the tuning parameter alpha may be tuned to a value between '0' and '1'. A detailed description of these RGB weights and IR weights will be described later.

[0071] According to an embodiment, step S140 may remove noise from the Noisy RGB image by duplicating the IR image into a three-channel IR image for each of the R, G, and B channels of the Noisy RGB image, applying IR weights to the three-channel IR images, and then adding the three-channel IR images to which the IR weights are applied to the Noisy RGB image to which the RGB weights are applied, thereby obtaining an RGB image (Color Noise Adaptation image, hereinafter referred to as a 'CNA RGB image') from which noise has been removed.

[0072] According to an embodiment, the method of the present disclosure is not limited or restricted to applying global RGB weights and global IR weights to obtain a CNA RGB image, and may apply RGB weights (local RGB weights) and IR weights (local IR weights) for each pixel, and may also apply weights according to illumination components (illumination weights) according to illumination components of the RGBIR image. This will be described with reference to FIG. 2.

[0073] FIG. 2 is a flowchart illustrating an operation of one embodiment of step S140 illustrated in FIG. 1, and illustrates a flowchart illustrating a process of obtaining a CNA RGB image by applying an illumination component and local weights.

[0074] Referring to FIG. 2, step S140 of obtaining a CNA RGB image by removing noise from a Noisy RGB image estimates a lighting component according to white balance performance, and determines weights (lighting weights) of R, G, and B channels according to the lighting component (S210, S220).

[0075] According to an embodiment, step S220 may determine the lighting weights of the R, G, and B channels to be the same or different depending on the lighting component of the white balance. For example, if the lighting component is white light, the lighting weights may be determined to be the same as '1' for all channels, and if the lighting component is not white light, the lighting weights may be determined differently depending on the lighting component so that the average of the lighting weights becomes 1.

[0076] When the lighting weight is determined by step S220, the lighting weight of each channel is applied to each IR image corresponding to the R, G, and B channels, and the RGB weight to be applied to each pixel of the Noisy RGB image and the IR image, that is, the local RGB weight, and the IR weight, that is, the local IR weight, are calculated (S230, S240).

[0077] According to an embodiment, step S240 calculates local RGB weights and local IR weights using global RGB weights, global IR weights, and preset IR factors, and adds an IR image to which a lighting weight and a local IR weight are applied to a Noisy RGB image to which a local RGB weight is applied, thereby removing noise and false color from the Noisy RGB image, thereby obtaining a CNA RGB image from which noise has been removed (S250).

[0078] Here, the IR factor can be determined by a first value determined based on the brightness component and IR component of each pixel of the RGB+ image and predetermined tuning parameters (Beta1, Beta2). Specifically, the IR factor can be determined by adding a second tuning parameter (Beta2) to a value obtained by multiplying the first value by the first tuning parameter (Beta1), and if the value is less than '0', it can be determined as '0', and if the value is greater than '1', it can be determined as '1'.

[0079] Although the local RGB weight and the local IR weight are calculated and applied in Fig. 2, it is not limited or restricted thereto, and the global RGB weight and the global IR weight may be applied after applying the lighting weight.

[0080] A method according to an embodiment of the present disclosure is described in detail using FIGS. 3 to 10 as follows.

[0081] First, AE target = RGB + norm Assuming this, gain is used to increase the brightness of the color-restored Noisy RGB image to the AE (Auto Exposure) target brightness. max = RGB + norm / (RGB + -IR) norm = RGB + norm / RGB norm There was a problem with noise being amplified when applying .

[0082] A method according to an embodiment of the present disclosure comprises: a gain for a Noisy RGB image; max Without applying it as is, gain is an appropriate RGB weight for Noisy RGB images. RGB Multiply by gain, which is an appropriate IR weighting factor for the demosaiced IR image. IR Multiply the two images to obtain the weighted sum.

[0083] At this time, the weight gain RGB Wow gain IR can be obtained or determined as follows:

[0084] RGB + norm = RGB norm × gain RGB + IR norm × gain IR This is called gain RGB Wow gain IR If organized into a first-order equation, it can be expressed as in <Mathematical Equation 1> below.

[0085] [Mathematical Formula 1]

[0086] gain RGB = (RGB + norm - IR norm × gain IR ) / RGB norm

[0087] gain RGB = -(IR norm / RGB norm ) × gain IR + gain max

[0088] Here, IR norm means the brightness of the IR image, and RGB norm means the brightness of the Noisy RGB image, and gain max can mean the ratio of the brightness of the RGB+ image including the IR component and the brightness of the Noisy RGB image. The RGB weight (gain) of the above mathematical expression 1 RGB ) and IR weighting (gain IR ) can be drawn as in Fig. 3.

[0089] RGB weighting (gain) RGB ) and IR weighting (gain IR ) is determined on the straight line of Fig. 3, and the brightness of the weighted sum image is RGB + norm This is it.

[0090] Here, alpha (α) can be defined as a normalized tuning parameter as follows.

[0091] Alpha (α) can be defined as A / B in Fig. 3, i.e. α=A / B.

[0092] This tuning parameter alpha is, as shown in Fig. 4, 0≤α≤1, and gain RGB =1 and gain IR When =1, that is, α=IR norm / RGB + norm When the best noise and best image quality are achieved, this is intuitively called gain. RGB =1, so the noise amplification of the RGB image is minimized, and the gain IR =1, so the image quality degradation that occurs in the process of removing low-quality IR images is mostly restored by adding the low-quality IR images again.

[0093] Also, when alpha (α) is '0', it is the best color (noise don't care), and α≥IR norm / RGB + norm In this case, that is, in the hatched area of ​​Fig. 4, RGB noise is not amplified, but the ratio of low-quality IR images increases, so the image quality deteriorates as a result.

[0094] Also, when alpha (α) is '1', it becomes an IR only image.

[0095] If alpha (α) is greater than '0', i.e. gain RGB Ga gain max If it is smaller than the global RGB weight (gain RGB ) and global IR weighting (gain IR) is used to remove noise from the RGB image, but as shown in the example in Fig. 5, false color occurs in some areas where RGB+ is large and IR is small.

[0096] To remove or mitigate these false colors, a local CNA can be applied. Here, a local CNA can mean calculating the weights of each pixel, i.e., local RGB weights and local IR weights, based on the global RGB weights and global IR weights applied to the entire image, and applying these to the CNA.

[0097] The process of calculating local RGB weights and local IR weights is described as follows.

[0098] First, an IR factor can be defined to calculate the local RGB weight and the local IR weight. At this time, the IR factor can be defined as “beta1×A + beta2”, and beta1 and beta2 can be the first tuning parameter and the second tuning parameter. In addition, A can be defined as (1 + IR - C+) or IR / C+, and C+ can be the brightness of each channel R+, G+, B+ channel of the input RGB+ image normalized between 0 and 1 and interpolated to full resolution, or the brightness component Y+ calculated using the R+, G+, B+ channels.

[0099] The method of the present disclosure can calculate local RGB weights and local IR weights using the IR factors, and is described as follows.

[0100] if ir_factor < 0 : ir_factor = 0

[0101] else if ir_factor > 1 : ir_factor = 1

[0102] gain_rgb_local = 1 - (1 - gain_rgb_global) * ir_factor

[0103] gain_ir_local = ir_factor * gain_ir_global

[0104] As can be seen from the above algorithm, the IR factor is set to '0' when the IR factor value is less than '0', and is set to '1' when it is greater than '1'. In addition, the RGB weight and IR weight applied to the entire image mean gain_rgb_global and gain_ir_global, and the local RGB weight (gain_rgb_local) and local IR weight (gain_ir_local) of each pixel can be determined or calculated through the formula in the above-described algorithm.

[0105] Also, instead of tuning the local RGB weights and local IR weights using alpha, you can fix alpha to 1 and tune them using beta1 and beta2.

[0106] For example, beta1 =1 beta2 =0, and gain max is 1.59, and α=IR norm / RGB + norm For example, in the case where both the global RGB weight and the global IR weight are 1, a local map (local RGB gain map) for the local RGB weight can be obtained as shown in Fig. 6a through the algorithm for calculating the local weight described above, and a local map (local IR gain map) for the local IR weight can be obtained as shown in Fig. 6b.

[0107] That is, in the area where RGB is large and IR is small, which is the area where false color occurs due to CNA using global RGB weights and global IR weights, the gain when CNA is not performed with a local RGB weight value of 1.46 maxAs the value approaches 1.59, we can see that the local IR weighting in that area linearly decreases by that ratio. This could mean that the CNA is locally weakened in the direction of alpha closer to '0' only in the region where false colors occur.

[0108] By applying the above-described local RGB weights and local IR weights to the method of the present disclosure, the false color generated in FIG. 5 can be removed, thereby obtaining an RGB image (CNA RGB image) with the false color removed, as shown in FIG. 7.

[0109] Furthermore, the method of the present disclosure applies IR weighting to the IR image. IR ) in the process of adding weights, the IR image is copied into three channels to add the IR image to each of the R, G, and B three channels. If the illumination component of the scene is estimated to be white light, the weights (illumination weights) for each channel of the copied IR image are all the same.

[0110] At this time, the reason why the color components of the resulting image, Cb, Cr, or Hue, are not distorted and appear natural is because the IR image replicated in three channels acts like air light in which white light is scattered, as shown in Fig. 8.

[0111] According to an embodiment, in the method of the present disclosure, when the illumination component of a color-restored RGB image is not white light and the illumination component is not corrected to white light through a white balance process, the illumination component of the scene normalized to have an average of '1' must be multiplied by a weight (illumination weight) as in <Mathematical Formula 2> below during the 3-channel duplication process of the IR image.

[0112] [Equation 2]

[0113]

[0114] Here, R_illumination + G_illumination + B_illumination = 3, and R_illumination, G_illumination, and B_illumination can mean R illumination weight, G illumination weight, and B illumination weight.

[0115] FIG. 9 is an example diagram showing a Noisy RGB image that has undergone white balance and a CNA RGB image from which noise has been removed. FIG. 9a shows a color-restored Noisy RGB image from which white balance has been performed for light source A (incandescent lamp), and FIG. 9b shows a CNA RGB image from which noise has been removed by the method of the present disclosure from which white balance has been performed for light source A (incandescent lamp).

[0116] As can be seen in Fig. 9, in a situation where the illuminance of visible light is below a certain level and the color temperature of visible light is low, the color is restored and white balance is performed to compensate for the low color temperature lighting compared to the color-restored Noisy RGB image, and it can be seen that the noise is reduced while securing the appropriate image brightness through the CNA RGB image as a result of noise removal using the method of the present disclosure, that is, the IR signal, RGB weight, and IR weight.

[0117] FIG. 10 illustrates an example of a Noisy RGB image without performing white balance and a CNA RGB image with noise removed. FIG. 10a illustrates a color-restored Noisy RGB image without performing white balance for light source A (incandescent lamp), and FIG. 10b illustrates a CNA RGB image with noise removed by the method of the present disclosure without performing white balance for light source A (incandescent lamp).

[0118] As can be seen in Fig. 10, even in a situation where white balance is not performed for the A light source (incandescent lamp), the noise is clearly reduced in the CNA RGB image as a result of noise removal using the method of the present disclosure, that is, the IR signal, RGB weights, and IR weights, and reflecting the low color temperature lighting component into the IR image, compared to the color-restored Noisy RGB image.

[0119] In this way, the RGBIR image noise removal method according to one embodiment of the present disclosure can reduce or eliminate noise occurring in the color restoration process of an RGBIR image, and can remove or reduce color noise in an RGBIR image by performing tuning intuitively and quantitatively without an unnecessary color space conversion process by processing in the RGB domain.

[0120] In addition, the RGBIR image noise removal method according to one embodiment of the present disclosure can remove noise from an RGBIR image according to a white balance result by adjusting the weights of IR components for each R, G, and B channel according to the illumination component evaluated or estimated from the white balance result when performing white balance, thereby removing noise.

[0121] In addition, a RGBIR image noise removal method according to one embodiment of the present disclosure can remove false colors included in an image by calculating pixel weights for each pixel of an RGB image and an IR image and removing noise in the RGB image by applying the pixel weights.

[0122] In addition, the RGBIR image noise removal method according to one embodiment of the present disclosure can be implemented with a small hardware size, can maintain the Hue (color) value of the color restored in the noise mitigation process as much as possible, and when processed in the RGB domain and integrated into the sensor as a post-processing block, the process of creating RGB Bayer again so that it can be used in an existing RGB ISP can be easy.

[0123] In addition, the RGBIR image noise removal method according to one embodiment of the present disclosure enables a trade-off between color and noise in RGBIR color reproduction, and tuning to an optimal setting in the trade-off relationship between color and noise is possible by adjusting the global parameter (alpha) and local parameters (Beta1, Beta2).

[0124] In addition, the RGBIR image noise removal method according to one embodiment of the present disclosure does not require a major modification to the subsequent 3-channel ISP pipeline due to the reduction from 4 channels to 3 channels, and has a simpler structure compared to an RGBIR ISP solution that requires separate RGB passes and IR passes, thereby reducing the gate count.

[0125] Furthermore, the RGBIR image noise removal method according to one embodiment of the present disclosure can express a portion with a lot of IR as a result of CNA as a gray color close to IR-dominated gray, and can express a portion with a lot of visible light as color. For example, since the IR image of a conventional RGBIR camera is 100% gray in IR mode at night, an area where visible light exists and can be expressed as color is also expressed as gray. On the other hand, the method according to the present disclosure can express an area where visible light exists as color, so it can seamlessly express the RGB local area and the IR local area to improve visibility.

[0126] Furthermore, the RGBIR image noise removal method according to one embodiment of the present disclosure can naturally switch between IR images and RGB images without distinguishing between RGB mode and IR mode.

[0127] FIG. 11 illustrates a configuration of an RGBIR image noise removal device according to another embodiment of the present disclosure, and illustrates a configuration of a device that performs the methods of FIGS. 1 to 10.

[0128] Referring to FIG. 11, an RGBIR image noise removal device (1100) according to another embodiment of the present disclosure includes an acquisition unit (1110), a restoration unit (1120), and a removal unit (1130).

[0129] The acquisition unit (1110) demosaics the RGBIR image to acquire an RGB image (RGB+ image) containing an IR component and an IR image.

[0130] Here, the acquisition unit (1110) can acquire an RGB+ image by separating an RGB+ image and an IR component from an RGBIR Bayer image and then demosaicing the separated RGB+ image, and can acquire an IR image by restoring the separated IR image.

[0131] The restoration unit (1120) removes the IR component from an RGB image (RGB+ image) including an IR component, thereby restoring (or obtaining) an RGB image (Noisy RGB image) from which the IR component has been removed.

[0132] The removal unit (1130) obtains or generates a CNA RGB image by removing noise from the Noisy RGB image by applying a preset RGB weight to the Noisy RGB image and applying a preset IR weight to the IR image, and then using the Noisy RGB image to which the RGB weight is applied and the IR image to which the IR weight is applied.

[0133] According to an embodiment, the removal unit (1130) may equally apply preset RGB weights to all pixels of the Noisy RGB image, and may equally apply preset IR weights to all pixels of the IR image. Here, the RGB weights and the IR weights may be determined based on the brightness of the RGB+ image, the brightness of the IR image, and the brightness of the Noisy RGB image.

[0134] According to an embodiment, the RGB weight and the IR weight may be determined by points on a predetermined straight line with the RGB weight and the IR weight as variables, and may be determined by a preset tuning parameter alpha corresponding to a ratio of the IR weight determined by the points on the straight line to the maximum IR weight on the straight line, and the tuning parameter alpha may be tuned to a value between '0' and '1'.

[0135] According to an embodiment, the removal unit (1130) can obtain a CNA RGB image from which noise has been removed by copying an IR image into a three-channel IR image for each of the R, G, and B channels of a Noisy RGB image, applying an IR weight to the three-channel IR image, and then adding the three-channel IR image to which the IR weight has been applied to the Noisy RGB image to which the RGB weight has been applied.

[0136] Furthermore, the removal unit (1130) is not limited or restricted to applying global RGB weights and global IR weights to obtain a CNA RGB image, and may apply local RGB weights and local IR weights to each pixel, and may obtain a CNA RGB image by applying lighting weights according to lighting components according to lighting components.

[0137] That is, the removal unit (1130), as illustrated in FIG. 12, applies an IR weight (IR gain) to the IR image and adds the IR image to the RGB image to which the RGB weight (RGB gain) is applied, and applies an illumination weight (Illumination), for example, an R illumination weight, a G illumination weight, and a B illumination weight, to the replicated three-channel IR image to add the IR image to each of the three channels of R, G, and B, thereby taking the illumination component into consideration and obtaining a CNA RGB image.

[0138] Of course, the RGB weights and IR weights applied in Fig. 12 may be global RGB weights and global IR weights, or local RGB weights and local IR weights.

[0139] Although the description is omitted in the device according to another embodiment of the present disclosure, the device according to another embodiment of the present disclosure may include all the contents described in the method of FIGS. 1 to 10, which is obvious to a person skilled in the art engaged in the technical field of the present disclosure.

[0140] FIG. 13 is a block diagram of a computing system for executing an RGBIR image noise removal method according to an embodiment of the present disclosure.

[0141] Referring to FIG. 13, the RGBIR image noise removal method according to an embodiment of the present disclosure described above may also be implemented through a computing system. The computing system (2000) may include at least one processor (2100), memory (2300), user interface input device (2400), user interface output device (2500), storage (2600), and network interface (2700) connected via a system bus (2200).

[0142] The processor (2100) may be a central processing unit (CPU) or a semiconductor device that executes processing on instructions stored in memory (2300) and / or storage (2600). The memory (2300) and storage (2600) may include various types of volatile or non-volatile storage media. For example, the memory (2300) may include a read-only memory (ROM) (2310) and a random access memory (RAM) (2320).

[0143] Accordingly, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be implemented directly in hardware, a software module, or a combination of the two executed by the processor (2100). The software module may reside in a storage medium (i.e., memory (2300) and / or storage (2600)), such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM. An exemplary storage medium is coupled to the processor (2100), such that the processor (2100) can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral with the processor (2100). The processor (2100) and the storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within a user terminal. Alternatively, the processor (2100) and the storage medium may reside as discrete components within the user terminal.

[0144] The above description is merely an illustrative description of the technical idea of ​​the present disclosure, and those skilled in the art to which the present disclosure pertains will appreciate that various modifications and variations may be made without departing from the essential characteristics of the present disclosure. Therefore, the embodiments disclosed in the present disclosure are not intended to limit the technical idea of ​​the present disclosure but to explain it, and the scope of the technical idea of ​​the present disclosure is not limited by these embodiments. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present disclosure.

Claims

A step of demosaicing an RGBIR image to obtain an RGB image and an IR image including an IR component; A step of restoring a first RGB image from which the IR component has been removed by removing the IR component from an RGB image including the IR component; and A step of removing noise from the first RGB image based on the first RGB image to which a predetermined RGB weight is applied and the IR image to which a predetermined IR weight is applied. A method for removing RGBIR image noise, comprising: In the first paragraph, The above RGB weights and the above IR weights are, A method for removing RGBIR image noise, wherein the brightness of an RGB image including the IR component is determined based on the brightness of the IR image and the brightness of the first RGB image. In the second paragraph, The above RGB weights and the above IR weights are, gain RGB = -(IR norm / RGB norm ) × gain IR + gain max It is determined by a point on a straight line expressed by the formula, The above gain RGB represents the RGB weights, and the gain IR represents the above IR weight, and the above IR norm represents the brightness of the IR image, and the RGB norm represents the brightness of the first RGB image, and the gain max A method for removing RGBIR image noise, which represents a ratio of the brightness of an RGB image including the IR component and the brightness of the first RGB image. In the third paragraph, The above RGB weights and the above IR weights are, It is determined by a preset tuning parameter alpha corresponding to the ratio of the IR weight determined by the point on the straight line and the maximum IR weight on the straight line, The above tuning parameter alpha is, A method for removing RGBIR image noise, tuned to values ​​between '0' and '1'. In the first paragraph, The above removal steps are: An RGBIR image noise removal method, wherein noise is removed from the first RGB image by duplicating the IR image into a three-channel IR image for each of the R, G, and B channels of the first RGB image, applying the IR weight to the three-channel IR image, and then adding the three-channel IR image to which the IR weight is applied to the first RGB image to which the RGB weight is applied. In paragraph 5, The above removal steps are: A RGBIR image noise removal method, which estimates the illumination component of the RGBIR image, determines an illumination weight for each of the R, G, and B channels based on the estimated illumination component, applies the illumination weight for each of the R, G, and B channels to each of the three-channel IR images, and then applies the IR weight. In the first paragraph, The above removal steps are: An RGBIR image noise removal method, which calculates pixel-by-pixel weights of each of the first RGB image and the IR image using the RGB weights, the IR weights, and a preset IR factor, and removes noise from the first RGB image based on the first RGB image to which the pixel-by-pixel weights of the first RGB image are applied and the IR image to which the pixel-by-pixel weights of the IR image are applied. In paragraph 7, The above IR factor is, A method for removing RGBIR image noise, wherein the RGBIR image noise is determined based on a first value determined based on the brightness component and IR component of each pixel of the RGB image including the IR component and predetermined tuning parameters. In paragraph 8, The above IR factor is, A method for removing RGBIR image noise, wherein the second tuning parameter is added to the product of the first value and the first tuning parameter, and is determined as '0' when the value is less than '0' and '1' when the value is greater than '1'. An acquisition unit that demosaics an RGBIR image to acquire an RGB image and an IR image containing an IR component; A restoration unit that restores a first RGB image from which the IR component has been removed by removing the IR component from an RGB image including the IR component; and A removal unit that removes noise from the first RGB image based on the first RGB image to which a predetermined RGB weight is applied and the IR image to which a predetermined IR weight is applied. An RGBIR image noise removal device comprising: In Article 10, The above RGB weights and the above IR weights are, An RGBIR image noise removal device, wherein the brightness of an RGB image including the IR component is determined based on the brightness of the IR image and the brightness of the first RGB image. In Article 11, The above RGB weights and the above IR weights are, gain RGB = -(IR norm / RGB norm ) × gain IR + gain max It is determined by a point on a straight line expressed by the formula, The above gain RGB represents the RGB weights, and the gain IR represents the above IR weight, and the above IR norm represents the brightness of the IR image, and the RGB norm represents the brightness of the first RGB image, and the gain max An RGBIR image noise removal device, which represents a ratio of the brightness of an RGB image including the IR component and the brightness of the first RGB image. In Article 12, The above RGB weights and the above IR weights are, It is determined by a preset tuning parameter alpha corresponding to the ratio of the IR weight determined by the point on the straight line and the maximum IR weight on the straight line, The above tuning parameter alpha is, An RGBIR image noise removal device tuned to values ​​between '0' and '1'. In Article 10, The above removal part, An RGBIR image noise removal device that removes noise from the first RGB image by replicating the IR image into a three-channel IR image for each of the R, G, and B channels of the first RGB image, applying the IR weight to the three-channel IR image, and then adding the three-channel IR image to which the IR weight is applied to the first RGB image to which the RGB weight is applied. In Article 14, The above removal part, An RGBIR image noise removal device that estimates an illumination component of the RGBIR image, determines an illumination weight for each of the R, G, and B channels based on the estimated illumination component, applies the illumination weight for each of the R, G, and B channels to each of the three-channel IR images, and then applies the IR weight. In Article 10, The above removal part, An RGBIR image noise removal device that calculates pixel-by-pixel weights of each of the first RGB image and the IR image using the RGB weights, the IR weights, and a preset IR factor, and removes noise from the first RGB image based on the first RGB image to which the pixel-by-pixel weights of the first RGB image are applied and the IR image to which the pixel-by-pixel weights of the IR image are applied. In Article 16, The above IR factor is, An RGBIR image noise removal device, wherein the RGBIR image noise removal device is determined based on a first value determined based on the brightness component and IR component of each pixel of the RGB image including the IR component and predetermined tuning parameters. In Article 17, The above IR factor is, An RGBIR image noise removal device, wherein the second tuning parameter is added to the product of the first value and the first tuning parameter, and is determined as '0' when the value is less than '0' and '1' when the value is greater than '1'.

Images