Image processing device

Abstract

The present invention provides an image processing device that can reduce random noise while keeping the size of the device under control. [Solution] The system includes: a wavelet transform unit that receives an image signal corresponding to the input image for each frame and generates a high-frequency component image signal and a low-frequency component image signal by performing a wavelet transform on the image signal; a coring processing unit that performs coring on the high-frequency component image signal; a smoothing processing unit that performs frame-cycle smoothing on the low-frequency component image signal using a frame memory; and an inverse transform unit that performs the inverse transform of the wavelet transform on the high-frequency component image signal that has undergone coring and the low-frequency component image signal that has undergone frame-cycle smoothing.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus.

Background Art

[0002] Systems that capture images of the vehicle surroundings with a camera and display them on a display to check for blind spots, and systems such as drive recorders that record the video during driving to check the situation at the time of an accident, are increasing in number, in which cameras are mounted on automobiles.

[0003] Due to the influence of noise generated inside, random noise is superimposed on the output image of the image sensors used in these cameras. Especially in a dark environment, a large gain is applied, so random noise is also amplified, and visibility deteriorates significantly. Therefore, in an ISP (Image Signal Processor) that performs image processing, an image processing apparatus having a function of noise removal processing is used (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] [Summary] As a noise removal processing technique, frame cycling type NR (noise removal) processing, which is a noise removal processing in the time direction with a high random noise removal effect, is widely used. However, frame cycling type NR processing has a problem that since a frame memory is required, the size and cost of the image processing apparatus increase.

[0006] An image processing apparatus according to one aspect of the present disclosure includes: a wavelet transform unit that receives an image signal corresponding to an input image for each frame and generates a high-frequency component image signal and a low-frequency component image signal by performing a wavelet transform on the image signal; a coring processing unit that performs coring on the high-frequency component image signal; a smoothing processing unit that performs frame-cycle smoothing on the low-frequency component image signal using a frame memory; and an inverse transform unit that performs an inverse transform of the wavelet transform on the high-frequency component image signal that has undergone the coring process and the low-frequency component image signal that has undergone the frame-cycle smoothing process.

[0007] Furthermore, an image processing method according to one aspect of the present disclosure is an image processing method performed by an image processing device that receives an image signal corresponding to an input image for each frame and performs image processing on the image signal, comprising the steps of: generating an image signal of high frequency components and an image signal of low frequency components by performing a wavelet transform on the image signal; performing a coring process on the image signal of high frequency components; performing a frame-cycle smoothing process using a frame memory on the image signal of low frequency components; and performing an inverse transform of the wavelet transform on the image signal of high frequency components that has undergone the coring process and the image signal of low frequency components that has undergone the frame-cycle smoothing process. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present disclosure. [Figure 2] This is a block diagram showing the configuration of the NR processing unit. [Figure 3] This diagram shows the processing blocks in the NR processing unit. [Figure 4] This diagram shows the wavelet transforms performed by the first and second WLT transform units. [Figure 5] This is a block diagram showing a modified configuration of an image processing device. [Figure 6]This is a block diagram showing a modified configuration of an image processing device. [Figure 7] This is a block diagram showing a modified configuration of the frame cyclic smoothing processing unit.

[0009] [Detailed explanation] Preferred embodiments of the present disclosure are described in detail below. In the following descriptions and accompanying drawings, substantially identical or equivalent parts are denoted by the same reference numerals.

[0010] Figure 1 is a block diagram showing the configuration of an image processing device 100 according to an embodiment of the present disclosure. The image processing device 100 is mounted on a vehicle such as an automobile and performs image processing on captured images of the area around the vehicle. The image processing device 100 receives input of captured images supplied from an image sensor IS and performs image processing on the Bayer signal BS.

[0011] The image processing device 100 includes a pre-processing unit 11, an NR processing unit 12, and a post-processing unit 13.

[0012] The pre-processing unit 11 performs adjustment processing on the Bayer signal BS, such as black level adjustment and white balance adjustment. The pre-processing unit 11 then performs demosaicing on the adjusted Bayer signal BS to generate an RGB image. Furthermore, in this embodiment, the pre-processing unit 11 generates a luminance signal Y, chrominance signals Cb and Cr by applying YC conversion processing to the RGB pixels. In the following description, the luminance signal Y, chrominance signals Cb and Cr will be collectively referred to as the image signal.

[0013] The NR processing unit 12 performs processing (hereinafter referred to as NR processing) to reduce random noise on the image signal that has been processed by the preceding processing unit 11. The NR processing unit 12 consists of an NR processing unit 12A that performs NR processing on the luminance signal Y, an NR processing unit 12B that performs NR processing on the chrominance signal Cb, and an NR processing unit 12C that performs NR processing on the chrominance signal Cr.

[0014] The subsequent processing unit 13 performs processing such as edge enhancement and YC image quality adjustment on the image signal that has undergone NR processing, and outputs it as the output signal OS.

[0015] Figure 2 is a block diagram showing the configuration of the NR processing unit 12A. The NR processing units 12B and 12C have a similar configuration. The NR processing unit 12A includes a WLT conversion unit 21, a frame memory 22, a frame cyclic smoothing processing unit 23, a coring processing unit 24, and a WLT inverse conversion unit 25.

[0016] The WLT conversion unit 21 performs a wavelet transform (WLT transform) on the input image signal S1, which consists of a luminance signal Y, and decomposes it into low-frequency components and high-frequency components. The WLT conversion unit 21 supplies the low-frequency components of the input image signal S1 to the frame cyclic smoothing processing unit 23. The WLT conversion unit 21 also supplies the high-frequency components of the input image signal S1 to the coring processing unit 24.

[0017] The frame memory 22 is composed of, for example, RAM (Random Access Memory). The frame memory 22 has a storage area capable of holding the low-frequency components of the input image signal S1 corresponding to at least one frame of captured image (input image).

[0018] The frame cyclic smoothing processing unit 23 performs frame cyclic smoothing on the low-frequency components of the input image signal S1 obtained by the wavelet transform of the WLT transform unit 21, using the frame memory 22.

[0019] The coring processing unit 24 performs coring on the high-frequency components of the input image signal S1 obtained by the wavelet transform of the WLT transform unit 21.

[0020] The WLT inverse transform unit 25 performs a wavelet inverse transform (WLT inverse transform) on the low-frequency component of the input image signal S1 that has undergone frame-by-frame smoothing processing and the high-frequency component of the input image signal S1 that has undergone core-ringing processing. The WLT inverse transform unit 25 outputs the result of the wavelet inverse transform as an output signal S2.

[0021] FIG. 3 is a diagram showing the processing blocks in the NR processing unit 12A. The WLT transform unit 21 and the WLT inverse transform unit 25 perform two-stage wavelet transform and wavelet inverse transform, respectively.

[0022] The WLT transform unit 21 includes a first WLT transform unit 211, which is a processing block that performs a wavelet transform at level 1 (LV0→LV) (first wavelet transform), and a second WLT transform unit 212, which is a processing block that performs a wavelet transform at level 2 (LV1→LV2) (second wavelet transform).

[0023] The first WLT transform unit 211 performs a wavelet transform at level 1 on the input image signal S1. As a result, the input image signal S1 is decomposed by frequency, and four signals, namely, an LL component (LV1), an HL component (LV1), a LH component (LV1), and an HH component (LV1), are generated.

[0024] The second WLT transform unit 212 performs a wavelet transform at level 2 on the LL component (LV1) signal generated through the wavelet transform by the first WLT transform unit 211. As a result, four signals, namely, an LL component (LV2), an HL component (LV2), a LH component (LV2), and an HH component (LV2), are generated.

[0025] Figure 4 shows the wavelet transforms performed by the first WLT transform unit 211 and the second WLT transform unit 212. The first WLT transform unit 211 performs a level 1 wavelet transform on the input image signal S1, thereby decomposing the input image signal S1 into LL component (LV1), HL component (LV1), LH component (LV1), and HH component (LV1) signals. Furthermore, the second WLT transform unit 212 performs a level 2 wavelet transform on the LL component (LV1) signal, thereby decomposing the LL component (LV1) signal into LL component (LV2), HL component (LV2), LH component (LV2), and HH component (LV2) signals.

[0026] Referring again to Figure 3, the frame cyclic smoothing processing unit 23 performs frame cyclic smoothing on the LL component (LV2), which is a low-frequency component generated by the wavelet transform of the second WLT transform unit 212. The frame cyclic smoothing processing unit 23 includes a subtractor 31, an alpha coefficient calculation unit 32, and an alpha blend unit 33.

[0027] The subtractor 31 receives input from the WLT conversion unit 21, which is the LL component (LV2) signal corresponding to the captured image of the current frame, and the processed signal of the LL component (LV2) corresponding to the captured image of the previous frame held in the frame memory 22 (i.e., the LL component signal corresponding to the previous frame after alpha synthesis), and outputs the difference value DV.

[0028] The alpha coefficient calculation unit 32 calculates the alpha coefficient, which is the coefficient used for alpha synthesis, based on the difference value DV output from the subtractor 31.

[0029] The α blending unit 33 performs α synthesis using the α coefficient calculated by the α coefficient calculation unit 32, the LL component (LV2) signal of the current frame supplied from the WLT conversion unit 21, and the LL component (LV2) signal of the previous frame held in the frame memory 22, and outputs the synthesis result as a smoothed signal SS (LL component Lv2).

[0030] The coring processing unit 24 consists of coring processing units 24A to 24F. Coring processing unit 24A performs coring on the HL component (LV2) signal generated by the wavelet transform of the second WLT transform unit 212. Coring processing unit 24B performs coring on the LH component (LV2) signal generated by the wavelet transform of the second WLT transform unit 212. Coring processing unit 24C performs coring on the HH component (LV2) signal generated by the wavelet transform of the second WLT transform unit 212.

[0031] The coring processing unit 24D performs coring on the HL component (LV1) signal generated by the wavelet transform of the first WLT transform unit 211. The coring processing unit 24E performs coring on the LH component (LV1) signal generated by the wavelet transform of the first WLT transform unit 211. The coring processing unit 24F performs coring on the HH component (LV1) signal generated by the wavelet transform of the first WLT transform unit 211.

[0032] The WLT inverse transform unit 25 includes a first WLT inverse transform unit 251, which is a processing block that performs a level 2 (LV2→LV1) wavelet inverse transform, and a second WLT inverse transform unit 252, which is a processing block that performs a level 1 (LV1→LV0) wavelet inverse transform.

[0033] The first wavelet inverse transform unit 251 performs a wavelet inverse transform on the smoothed signal SS (LL component Lv2) obtained by frame cyclic smoothing processing by the frame cyclic smoothing processing unit 23, and on the HL component (Lv2), LH component (Lv2), and HH component (Lv2) signals obtained by coring processing by coring processing units 24A to 24C. This generates a signal with a level 1 LL component.

[0034] The second WLT inverse transform unit 252 performs an inverse wavelet transform on the LL component (LV1) signal obtained by the first WLT inverse transform unit 251, and on the HL component (Lv1), LH component (Lv1), and HH component (Lv1) signals obtained by the coring processing units 24D to 24F. The second WLT inverse transform unit 252 outputs the result of the inverse wavelet transform as the output signal S2.

[0035] As described above, the image processing apparatus 100 of this embodiment includes a WLT transform unit 21 and a WLT inverse transform unit 25 that perform wavelet transform and inverse transform, a coring processing unit 24 that performs coring on the high-frequency component signal generated by the wavelet transform, and a frame cyclic smoothing processing unit 23 that performs frame cyclic smoothing on the low-frequency component signal. Random noise is removed by combining frame cyclic smoothing, which is a time-direction NR processing, and coring, which is a spatial NR processing.

[0036] With this configuration, unlike when frame-cycle smoothing is performed on the entire input image, the size of the frame memory 22 used for frame-cycle smoothing can be reduced.

[0037] For example, in conventional frame-cyclic smoothing performed on the entire image signal, if the image size is 1920 x 1080 pixels and the bit length per pixel is 10 bits each for Y, Cb, and Cr, then a memory size of 1920 x 1080 x 10 x 3 = approximately 62.2 M [bits] is required. In contrast, in this embodiment, where frame-cyclic smoothing is performed only on the LL component (Lv2) signal that has undergone a two-stage wavelet transform, the image size becomes 480 x 270, so the required memory size is 480 x 270 x 14 x 3 = approximately 5.44 M [bits].

[0038] Therefore, according to the image processing apparatus 100 of this embodiment, it is possible to reduce random noise while suppressing an increase in the size of the apparatus.

[0039] In particular, since in-vehicle cameras used in drive recorders and electronic mirrors are required to be small in size and cost, using the image processing device 100 of this embodiment makes it possible to improve visibility in dark environments such as at night while minimizing the impact on size and cost.

[0040] However, this disclosure is not limited to the embodiments shown above. For example, in the embodiments described above, the case in which NR processing is performed on the luminance signal Y, chrominance signals Cb and Cr obtained by the YC conversion processing of the pre-processing unit 11 was described as an example, but the target of NR processing is not limited to this.

[0041] Figure 5 is a block diagram showing the configuration of an image processing apparatus 100A in a modified example where the target of NR processing is the signal corresponding to RGB pixels (i.e., the target image signal is the RGB pixel signal).

[0042] The pre-processing unit 11 generates an RGB image by demosaicing the Bayer signal BS, which has undergone adjustments such as black level adjustment and white balance adjustment. The pre-processing unit 11 supplies the R pixel signal, G pixel signal, and B pixel signal to the NR processing unit 12.

[0043] NR processing unit 12A performs NR processing on the R pixel signal. NR processing unit 12B performs NR processing on the G pixel signal. NR processing unit 12C performs NR processing on the B pixel signal.

[0044] The subsequent processing unit 13 performs edge enhancement and other processing on the RGB pixel signals that have undergone NR processing, and outputs them as the output signal OS.

[0045] Even in this configuration, similar to when NR processing is performed on the luminance signal Y and the chrominance signals Cb and Cr, frame-cycle smoothing is performed on the low-frequency components of the signal generated by wavelet transform, and coring is performed on the high-frequency components of the signal, thereby reducing random noise while keeping the size of the device small.

[0046] Figure 6 is a block diagram showing the configuration of an image processing apparatus 100B in a modified example where the target of NR processing is a Bayer signal BS.

[0047] The pre-processing unit 11 supplies the Bayer signal BS, which has undergone adjustment processing such as black level adjustment and white balance adjustment, to the NR processing unit 12.

[0048] The NR processing unit 12 includes an NR processing unit 12X that performs NR processing on the Bayer signal BS. The NR processing unit 12X has the same configuration as the NR processing unit 12A in the above embodiment.

[0049] The subsequent processing unit 13 performs contour enhancement and other processing on the Bayer signal BS, which has undergone NR processing by the NR processing unit 12X, and outputs it as the output signal OS.

[0050] Furthermore, in the above embodiment, the frame-cycle smoothing processing unit 23 was described as performing smoothing by alpha synthesis according to the difference between the low-frequency component image signal corresponding to the captured image of the current frame and the processed low-frequency component image signal corresponding to the captured image of the previous frame held in the frame memory 22. However, the method of smoothing by the frame-cycle smoothing processing unit 23 is not limited to this.

[0051] Figure 7 is a block diagram showing the configuration of a modified frame cyclic smoothing processing unit 23A. The frame cyclic smoothing processing unit 23A includes a first subtractor 41, a clipping processing unit 42, an NR effectiveness setting unit 43, and a second subtractor 44.

[0052] The first subtractor 41 receives input from the WLT conversion unit 21, which is the LL component (LV2) signal corresponding to the captured image of the current frame, and the processed signal of the LL component (LV2) corresponding to the captured image of the previous frame held in the frame memory 22, and outputs the difference value DV.

[0053] The clipping processing unit 42 cuts off the portion of the difference value DV that exceeds the upper and lower threshold limits, and generates the difference value CDV.

[0054] The NR effectiveness setting unit 43 supplies the second subtractor 44 with a difference value EDV, which is obtained by multiplying the difference value CDV by a predetermined effectiveness setting ES as a coefficient.

[0055] The second subtractor 44 outputs a smoothed signal SS (LL component Lv2) obtained by subtracting the difference value EDV from the LL component (LV2) signal corresponding to the captured image of the current frame.

[0056] This configuration also enables frame cyclic smoothing processing using the frame memory 22.

[0057] Furthermore, in the above embodiment, the case of performing a two-stage wavelet transform by the first WLT transform unit 211 and the second WLT transform unit 212 was described as an example. However, the number of wavelet transforms is not limited to this, and wavelet transforms may be performed three or more times (three or more stages). That is, the WLT transform unit 21 may be configured to perform a first process of performing a first wavelet transform on the input image signal S1, a second process of performing a second wavelet transform on the separated LL component signals, and a third process of performing a second wavelet transform on the LL component signals separated by the second wavelet transform. In that case, the frame cyclic smoothing processing unit 23 performs frame cyclic smoothing on the LL components separated after one or more third processes performed following the second process. With such a configuration, it is possible to further reduce the size of the frame memory 22 used for frame cyclic smoothing.

[0058] [Note] This specification discloses the following configuration:

[0059] (Composition 1) An image processing apparatus comprising: a wavelet transform unit that receives an image signal corresponding to an input image for each frame and generates a high-frequency component image signal and a low-frequency component image signal by performing a wavelet transform on the image signal; a coring processing unit that performs coring on the high-frequency component image signal; a smoothing processing unit that performs frame-cycle smoothing on the low-frequency component image signal using a frame memory; and an inverse transform unit that performs an inverse transform of the wavelet transform on the high-frequency component image signal that has undergone the coring process and the low-frequency component image signal that has undergone the frame-cycle smoothing process.

[0060] (Configuration 2) The image processing apparatus according to configuration 1, wherein the wavelet transform unit performs a first wavelet transform on the image signal to separate the image signal into high-frequency components and low-frequency components, and a second wavelet transform on the separated low-frequency components to separate them into high-frequency components and low-frequency components.

[0061] (Composition 3) The image processing apparatus according to configuration 2, wherein the smoothing processing unit applies frame-cyclic smoothing processing using a frame memory to the low-frequency components separated by the second processing.

[0062] (Composition 4) The image processing apparatus according to configuration 2, wherein the wavelet transform unit performs a second wavelet transform on the low-frequency components separated by the second wavelet transform to perform a third process that separates them into high-frequency components and low-frequency components, and the smoothing unit performs a frame-cycle smoothing process using a frame memory on the low-frequency components separated after one or more third processes performed following the second process.

[0063] (Composition 5) The image processing apparatus according to configuration 2, wherein the coring processing unit performs coring on the high-frequency components separated by the wavelet transform of the first processing and the high-frequency components separated by one or more wavelet transforms of the second processing.

[0064] (Composition 6) The image processing apparatus according to any one of configurations 1 to 5, wherein the image signal is a Bayer signal.

[0065] (Composition 7) The image processing apparatus according to any one of configurations 1 to 5, wherein the image signal is a luminance signal Y, a chrominance signal Cb, and a chrominance signal Cr.

[0066] (Composition 8) The image processing apparatus according to any one of configurations 1 to 5, wherein the image signal is an RGB pixel signal obtained by demosaicing the input image.

[0067] (Composition 9) The image processing apparatus according to any one of configurations 1 to 8, wherein the smoothing processing unit performs alpha synthesis according to the difference between the low-frequency component image signal corresponding to the current frame of the input image and the low-frequency component image signal corresponding to the processed image of the previous frame held in the frame memory, thereby performing the frame cyclic smoothing process.

[0068] (Composition 10) The image processing apparatus according to any one of configurations 1 to 9, wherein the smoothing processing unit performs the frame cyclic smoothing process by subtracting the difference between the image signal of the low-frequency component corresponding to the current frame of the input image and the image signal of the low-frequency component corresponding to the processed image of the previous frame held in the frame memory from the image signal of the low-frequency component corresponding to the current frame of the input image.

[0069] (Composition 11) An image processing method performed by an image processing device that receives an image signal corresponding to an input image for each frame and performs image processing on the image signal, comprising the steps of: generating an image signal with high frequency components and an image signal with low frequency components by performing a wavelet transform on the image signal; performing a coring process on the image signal with high frequency components; performing a frame-cycle smoothing process using a frame memory on the image signal with low frequency components; and performing an inverse transform of the wavelet transform on the image signal with high frequency components that has undergone the coring process and the image signal with low frequency components that has undergone the frame-cycle smoothing process. [Explanation of Symbols]

[0070] 100 Image Processing Devices 11 Pre-processing stage 12 NR Processing Unit 13. Subsequent Processing Unit 21 WLT conversion section 22 frame memory 23. Frame cyclic smoothing processing unit 24 Coring Processing Unit 25 WLT Inverse Transformer 31 Subtractor 32 Alpha coefficient calculation unit 33 α Blend Section 211 First WLT Conversion Unit 212 Second WLT Conversion Unit 251 First WLT Inverse Transformer 252 Second WLT Inverse Transformer

Claims

1. A wavelet transform unit receives an image signal corresponding to the input image for each frame, and generates a high-frequency component image signal and a low-frequency component image signal by performing a wavelet transform on the said image signal. A core processing unit that performs core processing on the aforementioned high-frequency component image signal, A smoothing processing unit that applies frame-cycle smoothing processing using a frame memory to the image signal of the low-frequency components, An inverse transformer unit performs an inverse wavelet transform on the high-frequency component image signal obtained through the coring process and the low-frequency component image signal obtained through the frame cyclic smoothing process. An image processing device having

2. The wavelet transform unit is A first process involves performing a first wavelet transform on the image signal to separate the image signal into high-frequency components and low-frequency components. A second process is performed on the separated low-frequency components to separate them into high-frequency components and low-frequency components. An image processing apparatus according to claim 1, which performs the following:

3. The image processing apparatus according to claim 2, wherein the smoothing processing unit performs frame-cycle smoothing processing using a frame memory on the low-frequency components separated by the second processing.

4. The wavelet transform unit is A third process is performed to separate the low-frequency components separated by the second wavelet transform into high-frequency components and low-frequency components by further performing the second wavelet transform on the low-frequency components separated by the second wavelet transform. The image processing apparatus according to claim 2, wherein the smoothing processing unit performs frame-cycle smoothing using a frame memory on the low-frequency components separated after one or more third processing steps performed following the second processing.

5. The image processing apparatus according to claim 2, wherein the coring processing unit performs coring on the high-frequency components separated by the first wavelet transform and the high-frequency components separated by one or more second wavelet transforms.

6. The image processing apparatus according to claim 1, wherein the image signal is a Bayer signal.

7. The image processing apparatus according to claim 1, wherein the image signal comprises a luminance signal Y, a chrominance signal Cb, and a chrominance signal Cr.

8. The image processing apparatus according to claim 1, wherein the image signal is an RGB pixel signal obtained by demosaicing the input image.

9. The image processing apparatus according to claim 1, wherein the smoothing processing unit performs frame cyclic smoothing by performing α synthesis according to the difference between the low-frequency component image signal corresponding to the current frame of the input image and the low-frequency component image signal corresponding to the processed image of the previous frame held in the frame memory.

10. The image processing apparatus according to claim 1, wherein the smoothing processing unit performs the frame cyclic smoothing process by subtracting the difference between the image signal of the low-frequency component corresponding to the current frame of the input image and the image signal of the low-frequency component corresponding to the processed image of the previous frame held in the frame memory from the image signal of the low-frequency component corresponding to the current frame of the input image.

11. An image processing method performed by an image processing device that receives an image signal corresponding to an input image for each frame and performs image processing on said image signal, The steps include: generating an image signal with high-frequency components and an image signal with low-frequency components by performing a wavelet transform on the aforementioned image signal; The steps include: applying a coring process to the image signal of the high-frequency components; The steps include applying frame-cycle smoothing processing using a frame memory to the image signal of the low-frequency components, The process involves performing an inverse wavelet transform on the high-frequency component image signal obtained through the coring process and the low-frequency component image signal obtained through the frame-cycle smoothing process. Image processing methods including [specific details omitted].

Images