Image processing apparatus and image processing method thereof

Abstract

An image processing apparatus and an image processing method thereof are disclosed. The image processing apparatus includes a discrete wavelet transform (DWT) device that performs DWT and down-sampling on a first source image to split the first source image into a low-frequency sub-image including low-frequency components in horizontal and vertical directions and a plurality of high-frequency sub-images including high-frequency components in at least one of the horizontal or vertical directions; a frame memory that stores a low-frequency sub-image of a second source image; a first noise reduction device that reduces noise in the low-frequency sub-image of the first source image using the low-frequency sub-image of the second source image; and an inverse discrete wavelet transform (IDWT) device that applies IDWT to the low-frequency sub-image of the first sub-image and the high-frequency sub-images of the first sub-image, which are noise-reduced by the first noise reduction device, to restore the first source image.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of priority to Korean Patent Application No. 10-2021-0185187, filed December 22, 2021, and Korean Patent Application No. 10-2022-0126603, filed October 4, 2022, in the Korean Intellectual Property Office, the contents of which are incorporated herein in their entirety. TECHNICAL FIELD

[0003] The present disclosure relates to an image processing apparatus for performing three-dimensional (3D) noise reduction (3DNR) on a source image using a frame memory of a small size (capacity). BACKGROUND

[0004] An image sensor can amplify its output to amplify a video signal at low illumination. When the output of the image sensor is amplified, gain noise can occur on a screen. Such noise can be removed by a 3DNR method to obtain a clearer image.

[0005] However, because a previous frame (screen) should be stored to remove noise using the 3DNR method, a frame memory having a large capacity is required. This can be a factor in reducing product competitiveness. SUMMARY

[0006] The present disclosure aims to solve the above-described problems occurring in the related art while maintaining the advantages achieved by the related art unchanged.

[0007] One aspect of the present disclosure provides an image processing apparatus and an image processing method thereof for reducing noise in a source image using a 3DNR method even if a frame memory of a small size is used.

[0008] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned herein will be clearly understood by a person skilled in the art from the following description.

[0009] According to an aspect of the disclosure, an image processing apparatus can include a discrete wavelet transform (DWT) device that performs DWT and downsampling on a first source image to segment the first source image into a low-frequency sub-image including low-frequency components in horizontal and vertical directions and a plurality of high-frequency sub-images each including high-frequency components in at least one of the horizontal or vertical directions; a frame memory that stores a low-frequency sub-image of a second source image that is input before the first source image is input and is denoised; a first denoising device that reduces noise in the low-frequency sub-image of the first source image using the low-frequency sub-image of the second source image stored in the frame memory; and an inverse discrete wavelet transform (IDWT) device that applies IDWT to the low-frequency sub-image of the first sub-image denoised by the first denoising device and the high-frequency sub-images of the first sub-image not passed through the first denoising device to restore the first source image.

[0010] According to another aspect of the disclosure, an image processing method can include performing DWT and downsampling on a first source image to segment the first source image into a low-frequency sub-image including low-frequency components in horizontal and vertical directions and a plurality of high-frequency sub-images including high-frequency components in at least one of the horizontal or vertical directions, reducing noise in the low-frequency sub-image of the first source image using a low-frequency sub-image of a second source image stored in a memory, and performing IDWT on the low-frequency sub-image of the first sub-image reduced in noise by performing DWT and downsampling and reducing noise and the high-frequency sub-images of the first image not passed through performing DWT and downsampling and reducing noise to restore the first source image.

[0011] The technical problems to be solved by the disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other objects, features and advantages of the disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 is a block diagram schematically illustrating a configuration of an image processing apparatus according to an embodiment of the disclosure;

[0014] Figure 2 is a diagram for describing a process of segmenting a source image into sub-images in a discrete wavelet transform (DWT) device of Figure 1

[0015] Figure 3 is a diagram showing sub-images in a hierarchical structure generated through a two-step segmentation process by DWT;

[0016] ​Figure 4 is a flowchart for describing an image processing method according to an embodiment of the disclosure;

[0017] Figure 5A and Figure 5B is a block diagram schematically illustrating a configuration of an image processing apparatus according to other embodiments of the disclosure; and

[0018] Figure 6 is a block diagram schematically illustrating a configuration of an image processing apparatus according to other embodiments of the disclosure. DETAILED DESCRIPTION

[0019] Hereinafter, some embodiments of the disclosure will be described in detail with reference to the exemplary drawings. In the drawings, the same reference numerals will always be used to designate the same or equivalent elements. Also, detailed descriptions of well-known features or functions will be omitted to avoid unnecessarily obscuring the point of the disclosure.

[0020] Figure 1 is a block diagram schematically illustrating a configuration of an image processing apparatus according to an embodiment of the disclosure.

[0021] Figure 2 is a graph for describing a process of segmenting a source image into sub-images in a discrete wavelet transform (DWT) apparatus of Figure 1 . Also, Figure 3 is a graph showing sub-images LH1, HL1, HH1, LL2, LH2, HL2, and HH2 in a hierarchical structure generated through a two-step segmentation process of DWT.

[0022] Referring to Figures 1 to 3 , the image processing apparatus 100 can include a DWT apparatus 110, a three-dimensional noise reduction (3DNR) apparatus 120, a frame memory 130, an image adder 140, and an inverse discrete wavelet transform (IDWT) apparatus 150.

[0023] The DWT apparatus 110 can perform DWT on a source image to segment the source image into a plurality of sub-images LL, LH, HL, and HH. For example, as Figure 2As illustrated, the DWT device 110 can apply a low-pass filter X-LPF and a high-pass filter X-HPF in a horizontal direction (e.g., an X direction) to a source image to split one source image (e.g., one source frame) into a sub-image L having low frequency components in the horizontal direction and a sub-image H having high frequency components in the horizontal direction, and can apply a low-pass filter Y-LPF and a high-pass filter Y-HPF in a vertical direction (e.g., a Y direction) to each of the two sub-images L and H to split one source image into four sub-images LL, LH, HL, and HH. At this time, LL can be a sub-image including low frequency components in the horizontal and vertical directions, which can have important information with high energy concentration and important information about the source image. LH, HL, and HH can be sub-images each having high frequency components with edge components in the horizontal, vertical, and diagonal directions of the source image, respectively, which can have details with low energy concentration and correspond to outline portions of the source image.

[0024] Here, for signals of low frequency components and signals of high frequency components obtained by filtering a source image in a horizontal direction, the DWT device 110 can perform a first down-sampling (2↓) to obtain two sub-images L and H, which takes only half of input data. Also, for signals of low frequency components and signals of high frequency components obtained by filtering each of the two sub-images L and H, the DWT device 110 can perform a second down-sampling (2↓) to obtain four sub-images LL, LH, HL, and HH, which takes only half of input data, which performs the first down-sampling in the horizontal direction. As a result, the DWT device 110 can split a source image into four sub-images LL, LH, HL, and HH. The sub-images LL, LH, HL, and HH can be images whose size (e.g., capacity) is reduced to 1 / 4 compared to the source image.

[0025] When the DWT device 110 repeatedly performs the DWT and the down-sampling N times, the size (e.g., capacity) of the sub-image is reduced (1 / 4) N times compared to the source image.

[0026] The DWT device 110 can transfer only a signal of the sub-image LL having low frequency components in the horizontal and vertical directions among the split sub-images LL, LH, HL, and HH to the 3D NR device 120, and can transfer signals of the other sub-images LH, HL, and HH to the image adder 140. As Figure 3 illustrated, the DWT device 110 can additionally perform the DWT and the down-sampling only on the sub-image LL having high energy concentration to obtain a sub-image LL' having a smaller size (e.g., (1 / 4) 2two-step sub-images LL2, LH2, HL2, and HH2. In this case, the DWT device 110 can transfer only the two-step sub-image LL2 to the 3DNR device 120.

[0027] In Figure 3 , the sub-images LH1, HL1, and HH1 represent one-step sub-images obtained by performing DWT and downsampling once in the horizontal and vertical directions of the source image. The sub-images LL2, LH2, HL2, and HH2 represent two-step sub-images obtained by performing DWT and downsampling again on the one-step sub-image LL1 in the horizontal and vertical directions.

[0028] The 3DNR device 120 can perform noise cancellation on the sub-image LL provided from the DWT device 110. The 3DNR device 120 can perform a 3DNR method for reducing noise in a spatial domain with respect to the sub-image LL and reducing noise again in a temporal domain with respect to the sub-image LL. For example, the 3DNR device 120 can apply a two-dimensional noise reduction (2DNR) filter to the current sub-image LL provided from the DWT device 110 t to mainly perform noise reduction (e.g., noise reduction in the spatial domain). Next, the 3DNR device 120 can compare the sub-image LL' from which noise is primarily removed with a previous sub-image LL t-1 previously stored in the frame memory 130, and can perform noise cancellation again (e.g., noise cancellation in the temporal domain) to generate a sub-image LL' t '.

[0029] The 3DNR device 120 can store the current sub-image LL t ' from which noise is reduced in the 3DNR method in the frame memory 130. In addition, the 3DNR device 120 can transfer the current sub-image LL t ' from which noise is reduced to the image adder 140. The current sub-image LL t ' stored in the frame memory 130 can be used as a new previous sub-image for 3DNR of a next sub-image LL t+1 When the two-step sub-image LL2 is received from the DWT device 110, the 3DNR device 120 can perform 3DNR on the two-step sub-image LL2, and can store a two-step sub-image LL2' from which noise is reduced in the frame memory 130.

[0030] The frame memory 130 can store necessary image information when the 3DNR device 120 performs 3DNR on the current sub-image LL. For example, when the 3DNR device 120 performs 3DNR on the current sub-image LL tWhen performing 3DNR, the frame memory 130 can store a previous sub-image LL t-1 for comparison. At this time, the previous sub-image LL t stored in the frame memory 130 can be a sub-image LL' whose noise has been reduced immediately before by the 3DNR device 120. t-1

[0031] In the present embodiment, since the 3DNR device 120 performs 3DNR only on the sub-image LL of the low-frequency components among the sub-images LL, LH, HL, and HH, and since the sub-image LL is an image whose size (e.g., capacity) is reduced to 1 / 4 compared to the source image by downsampling, the size of the frame memory 130 required for 3DNR can be significantly reduced. When the DWT device 110 divides the source image in two steps to generate the sub-images LL2, LH2, HL2, and HH2, and when the 3DNR device 120 performs 3DNR only on the two-step sub-image LL2, the size of the frame memory 130 required for 3DNR can be further reduced.

[0032] The image adder 140 can add the sub-image LL' whose noise has been reduced by the 3DNR device 120 to the sub-images LH, HL, and HH from the DWT device 110, and can output the added sub-images LL', LH, HL, and HH to the IDWT device 150. In other words, for one source image, the image adder 140 can add the sub-image LL' on which 3DNR has been performed to the sub-images LH, HL, and HH on which 3DNR has not been performed, and can output the added sub-images LL', LH, HL, and HH to the IDWT device 150.

[0033] The IDWT device 150 can synthesize the sub-images LL', LH, HL, and HH added by the image adder 140 to restore the source image. For example, the IDWT device 150 can perform IDWT on the sub-images LL', LH, HL, and HH divided by DWT to synthesize the sub-images LL', LH, HL, and HH, thereby restoring the source image. At this time, because the sub-image LL' is an image on which noise has been reduced in the sub-image LL, the restored image (hereinafter referred to as a "restored image") can be an image on which noise has been reduced in the original source image.

[0034] Figure 4 is a flowchart for describing an image processing method according to an embodiment of the disclosure, which is sequentially illustrated in Figure 1 the image processing apparatus of FIG. 1.

[0035] Referring to Figure 4 , when an input source image is input in S410, Figure 1 ​The DWT device 110 can perform DWT and down-sampling on the source image to split the source image into a plurality of sub-images LL, LH, HL, and HH.

[0036] For example, the DWT device 110 can apply a low-pass filter X-LPF and a high-pass filter X-HPF on the source image in a horizontal direction (e.g., X direction) to split one source image into a sub-image having a low frequency component in the horizontal direction and a sub-image having a high frequency component in the horizontal direction. Next, as shown in Figure 2 For each of the sub-images split into the low frequency component and the high frequency component in the horizontal direction, the DWT device 110 can perform a first down-sampling (2↓) that takes only half of the input data in the horizontal direction, thereby obtaining sub-images L and H whose size in the horizontal direction is reduced to 1 / 2 compared to the source image.

[0037] Next, the DWT device 110 can apply a low-pass filter Y-LPF and a high-pass filter Y-HPF on each of the sub-images L and H in a vertical direction (e.g., Y direction) to split each of the sub-images L and H into a low frequency component and a high frequency component in the vertical direction, and can perform a second down-sampling (2↓) on the split sub-images in the vertical direction that takes only half of the input data, thereby obtaining four sub-images LL, LH, HL, and HH whose size of each is reduced by 1 / 2 in the horizontal and vertical directions compared to the source image. In other words, one source image (e.g., one source frame) can be split into four sub-images LL, LH, HL, and HH whose size (e.g., capacity) is reduced to 1 / 4 by the DWT device 110.

[0038] At this time, LL can indicate a sub-image having a low frequency component in the horizontal and vertical directions, and LH, HL, and HH can indicate sub-images each having a high frequency component, which have edge information about horizontal, vertical, and diagonal directions of the source image, respectively.

[0039] The DWT device 110 can transfer the sub-image LL having a low frequency component in the horizontal and vertical directions among the four sub-images LL, LH, HL, and HH to the 3D NR device 120, and can transfer the other sub-images LH, HL, HH, i.e., the sub-images LH, HL, and HH having high frequency components in the horizontal, vertical, and diagonal directions. Figure 1

[0040] When receiving a signal of the sub-image LL from the DWT device 110, the 3D NR device 120 can perform noise reduction in a spatial domain with respect to the received sub-image LL in S420.

[0041] ​For example, the 3D NR device 120 can apply a 2D NR filter to the received sub-image LL to reduce noise in the sub-image LL.

[0042] In S430, the 3D NR device 120 can perform noise reduction in the time domain with respect to the sub-image LL initially reduced in the spatial domain.

[0043] For example, the 3D NR device 120 can compare the current sub-image LL t initially reduced in the spatial domain with the previous sub-image LL Figure 1 previously stored in the frame memory 130 of the DWT device 110 to perform noise reduction again. The previous sub-image LL t-1 may indicate that it is stored in the frame memory 130 after performing 3D NR on the immediately previous sub-image LL (e.g., after reducing noise of the immediately previous sub-image LL in the spatial domain and the time domain). t-1

[0044] In S440, the 3D NR device 120 can store the current sub-image LL t on which 3D NR is performed in the frame memory 130 to use the current sub-image LL t ' as a new previous sub-image for the next 3D NR, and can deliver the current sub-image LL t ' to the image adder 140 of the DWT device 110. Figure 1

[0045] In the present embodiment, since the 3D NR device 120 performs 3D NR only on the sub-image LL, the sub-image LL can be an image whose size is reduced to 1 / 4 compared to the source image. In other words, the size of the sub-image (e.g., the previous sub-image) that should be stored in the frame memory 130 for 3D NR can also be reduced to 1 / 4 compared to the source image. Accordingly, the size of the frame memory 130 can be reduced to 1 / 4 in the present embodiment compared to performing 3D NR on the source image.

[0046] When the DWT device 110 divides the source image into two steps to generate the sub-images LL2, LH2, HL2, and HH2, and when the 3D NR device 120 performs 3D NR only on the two-step sub-image LL2, the necessary size of the frame memory 130 can be further reduced.

[0047] The image adder 140 can add the sub-image LL' reduced in noise by the 3D NR device 120 to the sub-images LH, HL, and HH from the DWT device 110, and can output the added sub-images LL', LH, HL, HH to the image decoder 150. Figure 1 ​​IDWT device 150. In S450, the IDWT device 150 can synthesize the sub-images LL', LH, HL, and HH received from the image adder 140 to restore the source image.

[0048] For example, the IDWT device 150 can perform the IDWT on the sub-images LL', LH, HL, and HH to restore the source image. At this time, because the sub-image LL' is the image denoised in the sub-image LL, the restored image can be the image denoised in the original source image.

[0049] Figure 5A and Figure 5B is a block diagram schematically illustrating a configuration of an image processing apparatus according to other embodiments of the disclosure.

[0050] Referring to Figure 5A and Figure 5B , compared to the image processing apparatus 100 described above with reference to Figure 1 , the image processing apparatus 200a or 200b can additionally include a 2DNR device 210 located at the input end of the DWT device 110 or the output end of the IDWT device 150. The 2DNR device 220 can perform 2DNR for applying a 2DNR filter to the source image or the restored image to reduce noise in the spatial domain.

[0051] Although the human eye is relatively insensitive to motion information of high-frequency components unlike motion of low-frequency components, 2D noise can be amplified in the process of amplifying a signal in an image sensor, thereby causing deterioration of image quality.

[0052] Since Figure 1 the image processing apparatus 100 in performs denoising only on the sub-image LL including low-frequency components, in the present embodiment, the 2DNR device 210 can be formed at the input end of the DWT device 110 or at the output end of the IDWT device 150 to first reduce 2D high-frequency noise in the source image before performing the DWT of the source image. As a result, the image processing apparatus 200a or 200b can reduce high-frequency noise and low-frequency noise while using a small-sized frame memory 130.

[0053] Figure 5A and Figure 5B , the DWT device 110, the 3DNR device 120, the frame memory 130, the image adder 140, and the IDWT device 150 can perform the same functions as the above-described corresponding components in Figure 1 . Therefore, the same reference numerals as in Figure 1 are used for the corresponding components in Figure 5A and Figure 5B , and the description of the corresponding components will be omitted.

[0054] Figure 6 is a block diagram schematically illustrating a configuration of an image processing apparatus according to other embodiments of the disclosure.

[0055] Reference Figure 6 , compared to the image processing apparatus 100 of Figure 1 , the image processing apparatus 300 can further include an encoder 310a and a decoder 310b located between the 3DNR device 120 and the frame memory 130.

[0056] The encoder 310a can compress the sub-images from the 3DNR device 120, and can transfer the compressed sub-images to the frame memory 130. In other words, when storing the current sub-image LL t ’ in the frame memory 130, which performs 3DNR thereof by the 3DNR device 130, the encoder 310a can compress the current sub-image LL t ’ to be stored, and can transfer the compressed sub-image to the frame memory 130.

[0057] The decoder 310b can decompress the compressed sub-images stored in the frame memory 130, and can transfer the decompressed sub-images to the 3DNR device 120. In other words, when the 3DNR device 120 reads the previous sub-image LL t-1 stored in the frame memory 130 to perform 3DNR, the decoder 310b can decompress the sub-image LL t-1 , which has been compressed and stored, and can transfer the decompressed sub-image to the 3DNR device 120.

[0058] In Figure 6 , the DWT device 110, the 3DNR device 120, the frame memory 130, the image adder 140, and the IDWT device 150 can perform the same functions as the above-described corresponding components in Figure 1 . Therefore, the same reference numerals as in Figure 1 are used for the corresponding components in Figure 6 , and the description of the corresponding components will be omitted.

[0059] Even if the frame memory having a small capacity is used, one embodiment of the disclosure can remove noise in an image signal to reduce the cost of an image processing apparatus.

[0060] In the foregoing, although the disclosure has been described with reference to the example embodiments and the accompanying drawings, the disclosure is not limited thereto, but can be variously modified and changed by those skilled in the art to which the disclosure belongs without departing from the spirit and scope of the disclosure claimed in the following claims.

[0061] Therefore, embodiments of the disclosure are not intended to limit the technical spirit of the disclosure, but are provided only for illustrative purposes. The scope of the disclosure should be interpreted based on the appended claims, and all technical ideas equivalent thereto within the scope of the claims should be included in the scope of the disclosure.

Claims

1. An image processing apparatus comprising: a discrete wavelet transform (DWT) device configured to perform a DWT and down-sampling on a first source image to divide the first source image into a low frequency sub-image comprising low frequency components in a horizontal direction and a vertical direction and a plurality of high frequency sub-images each comprising high frequency components in at least one of the horizontal direction or the vertical direction; a frame memory storing a low frequency sub-image of a second source image, the second source image being input prior to input of the first source image and being denoised; a first denoising device configured to reduce noise in the low frequency sub-image of the first source image using the low frequency sub-image of the second source image stored in the frame memory; and an inverse discrete wavelet transform (IDWT) device configured to apply an IDWT to the low frequency sub-image of the first source image denoised by the first denoising device and to high frequency sub-images of the first source image not passed through the first denoising device to restore the first source image. The DWT device applies a low pass filter and a high pass filter on the first source image in the horizontal direction and performs down-sampling on the first source image in the horizontal direction to generate first and second sub-images and applies a low pass filter and a high pass filter on each of the first and second sub-images in the vertical direction and performs down-sampling on each of the first and second sub-images in the vertical direction to generate the low frequency sub-image and high frequency sub-images of the first source image.

2. The image processing apparatus according to claim 1, wherein The first denoising device denoises in a spatial domain for the low frequency sub-image of the first source image and in a temporal domain by comparing the low frequency sub-image of the first source image with the low frequency sub-image of the second source image.

3. The image processing apparatus according to claim 1, wherein 4. The image processing apparatus according to claim 1, further comprising: a second denoising device located at an input of the DWT device and configured to apply a two-dimensional denoising (2DNR) filter to the first source image.

5. The image processing apparatus according to claim 1, further comprising: a third denoising device located at an output of the IDWT device and configured to apply a 2DNR filter to the first source image restored by the IDWT device. The first denoising device stores the low frequency sub-image of the first source image in the frame memory, the low frequency sub-image of the first source image being denoised.

6. The image processing apparatus according to claim 1, wherein 7. The image processing apparatus according to claim 6, further comprising: an encoder configured to compress the low frequency sub-image of the first source image denoised by the first denoising device and store the compressed low frequency sub-image in the frame memory; and an IDWT device configured to apply an IDWT to the low frequency sub-image of the first source image denoised by the first denoising device and to high frequency sub-images of the first source image not passed through the first denoising device to restore the first source image. ​ a decoder configured to decompress a low frequency sub-image of the second source image, the low frequency sub-image of the second source image being compressed and stored in the frame memory, and to transmit the decompressed low frequency sub-image to the first noise reduction device.

8. An image processing method comprising: performing a DWT and a down-sampling on a first source image to segment the first source image into a low frequency sub-image comprising low frequency components in a horizontal direction and a vertical direction and a plurality of high frequency sub-images comprising high frequency components in at least one of the horizontal direction or the vertical direction; reducing noise in the low frequency sub-image of the first source image using a low frequency sub-image of a second source image stored in a memory; and performing an IDWT on the low frequency sub-image of the first source image and high frequency sub-images of the first source image to recover the first source image, the low frequency sub-image of the first source image being noise reduced by the performing of the DWT and the down-sampling and the reducing of the noise, the high frequency sub-images not being noise reduced by the performing of the DWT and the down-sampling and the reducing of the noise.

9. The image processing method of claim 8, wherein, the segmenting of the first source image comprises: applying a low pass filter and a high pass filter on the first source image in the horizontal direction and performing a down-sampling on the first source image in the horizontal direction to generate a first sub-image and a second sub-image; and applying a low pass filter and a high pass filter on each of the first sub-image and the second sub-image in the vertical direction and performing a down-sampling on each of the first sub-image and the second sub-image in the vertical direction to generate a low frequency sub-image and high frequency sub-images of the first source image.

10. The image processing method of claim 8, wherein, the low frequency sub-image of the second source image is a low frequency sub-image separated and noise reduced from the second source image being input prior to inputting the first source image.

11. The image processing method of claim 10, wherein, the noise reducing comprises: noise reducing in a spatial domain for the low frequency sub-image of the first source image, comparing the low frequency sub-image of the first source image with the low frequency sub-image of the second source image, and noise reducing in a temporal domain.

12. The image processing method of claim 8, further comprising: applying a 2DNR filter to the first source image prior to segmenting the first source image.

13. The image processing method of claim 8, further comprising: applying a 2DNR filter to the recovered first source image after recovering the first source image.

14. The image processing method of claim 8, wherein, the noise reducing comprises: storing the low frequency sub-image of the first source image in the memory, the low frequency sub-image being noise reduced.

15. The image processing method of claim 14, wherein, the noise reducing comprises: compressing and storing the low frequency sub-image of the first source image when storing the low frequency sub-image of the first source image in the memory.

Images