Image noise reduction method and image processing device

Abstract

The embodiments of the disclosure provide an image noise reduction method and an image processing device. The method includes: obtaining a t-th input image and a t-th reference image; determining a first weight map based on the t-th input image and the t-th reference image; using the t-th input image as a guiding image to perform a guided filtering operation on the first weight map to generate a second weight map; and performing a temporal filtering operation on the t-th input image and the t-th reference image based on the second weight map to generate a t-th output image.

Description

[Technical Field]

[0001] This invention relates to an image processing method, and more particularly to an image noise reduction method and an image processing apparatus. [Previous Technology]

[0002] Currently, many video analytics applications (such as video surveillance, facial recognition, and target tracking) have increasingly higher requirements for image quality to ensure analytical accuracy and meet the needs of human visual observation. However, in high-noise environments, especially in low-light scenes, image quality often degrades significantly, thus affecting the reliability of the analysis results. To improve image quality, temporal noise reduction (TNR) processing of video has become a crucial technology.

[0003] When performing temporal denoising, accurate motion detection is the foundation for achieving the following two key objectives: (1) to achieve efficient denoising in static areas; (2) to avoid ghost effects on moving objects, especially when dealing with objects with minute movements.

[0004] Therefore, for those skilled in the art, how to design a means to improve the image quality and analysis accuracy of TNR through accurate motion detection is an important issue. [Summary of the Invention]

[0005] In view of the above, the present invention provides an image noise reduction method and an image processing apparatus, which can be used to solve the above-mentioned technical problems.

[0006] This embodiment of the invention provides an image denoising method, executed by an image processing device, comprising: acquiring a t-th input image and a t-th reference image, where t is a time index value; determining a first weight map based on the t-th input image and the t-th reference image; performing a guided filtering operation on the first weight map using the t-th input image as a guiding image to generate a second weight map; and performing a temporal filtering operation on the t-th input image and the t-th reference image based on the second weight map to generate a t-th output image.

[0007] This embodiment of the invention provides an image processing apparatus, including a storage circuit and an image processor. The storage circuit stores program code. The image processor is coupled to the storage circuit and accesses the program code to execute: acquiring a t-th input image and a t-th reference image, where t is a time index value; determining a first weight map based on the t-th input image and the t-th reference image; performing a guided filtering operation on the first weight map using the t-th input image as a guiding image to generate a second weight map; and performing a temporal filtering operation on the t-th input image and the t-th reference image based on the second weight map to generate a t-th output image.

Implementation Method

[0009] Please refer to FIG1, which is a schematic diagram of an image processing apparatus according to one embodiment of the present invention. In different embodiments, the image processing apparatus 100 may be implemented as various smart devices and / or computer devices, but is not limited thereto.

[0010] In Figure 1, the image processing apparatus 100 includes a storage circuit 102 and an image processor 104.

[0011] The storage circuit 102 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar device or combination of these devices, and can be used to record multiple code or modules.

[0012] The image processor 104 is coupled to the storage circuit 102 and may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, multiple microprocessors, one or more microprocessors incorporating a digital signal processor core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), any other type of integrated circuit, a state machine, a processor based on an advanced RISC machine (ARM), and the like.

[0013] Please refer to Figure 2, which is a functional block diagram of the image processor drawn according to Figure 1. In Figure 2, the image processor 104 includes a temporal filtering circuit 202, a weight determination circuit 204, and a guide filter 206, wherein the guide filter 206 is coupled between the temporal filtering circuit 202 and the weight determination circuit 204.

[0014] In an embodiment of the present invention, the image processor 104 can access the modules and program code recorded in the storage circuit 102 to implement the image noise reduction method proposed in the present invention, the details of which are described below.

[0015] Please refer to Figure 3, which is a flowchart illustrating an image noise reduction method according to one embodiment of the present invention. The method of this embodiment can be executed collaboratively by the various components in the image processing apparatus 100 of Figure 1. The details of each step in Figure 3 are explained below with reference to the components shown in Figures 1 and 2.

[0016] In step S310, the image processor 104 obtains the t-th input image I(t) and the t-th reference image IR(t), where t is the time index value.

[0017] In embodiments of the present invention, the image processor 104 may acquire various images from a considered image source. In different embodiments, the image source may be, for example, a monitor capable of capturing images of the monitored area in real time and / or storage space (e.g., an image database) storing images to be analyzed and / or processed, but is not limited thereto.

[0018] In an embodiment of the present invention, the t-th input image is, for example, the t-th image among a plurality of images obtained by the image processor 104 from the aforementioned image source.

[0019] In addition, the t-th reference image IR(t) is, for example, a reference image corresponding to the t-th input image I(t). In one embodiment, the t-th reference image IR(t) is, for example, the (t-1)-th input image (hereinafter referred to as I(t-1)), that is, the previous input image of the t-th input image I(t), but it is not limited to this.

[0020] In an embodiment of the present invention, after obtaining the t-th input image I(t) and the t-th reference image IR(t), the image processor 104 may continue to execute steps S320 to S340 to generate the corresponding t-th output image I'(t). In one embodiment, the t-th output image I'(t) may serve as a reference image corresponding to the (t+1)-th input image (i.e., the (t+1)-th reference image).

[0021] Based on the principle of similarity, in some embodiments, the t-th reference image IR(t) can also be the (t-1)-th output image (hereinafter referred to as I'(t-1)), that is, the previous output image of the t-th output image I'(t), but it is not limited to this.

[0022] In step S320, the weight determination circuit 204 of the image processor 104 determines the first weight map W(t) based on the t-th input image I(t) and the t-th reference image IR(t). The details of step S320 will be explained below with the aid of FIG4.

[0023] Please refer to Figure 4, which is an application scenario diagram of the first weight map determined by the weight determination circuit based on Figure 2.

[0024] In Figure 4, the weight determination circuit 204 includes a downsampling circuit 402, a motion detection circuit 404, a weight calculation circuit 406 and an upsampling circuit 408 that are sequentially coupled.

[0025] In this embodiment, the downsampling circuit 402 can downsample the t-th input image I(t) as the first image Id(t) and downsample the t-th reference image IR(t) as the second image Id(t-1).

[0026] In one embodiment, the downsampling circuit 402 can perform the aforementioned downsampling on the t-th input image I(t) and the t-th reference image IR(t) based on preset downsampling parameters. For example, assuming the selected downsampling parameter is S (e.g., 8), the size of the first image Id(t) obtained by downsampling the t-th input image I(t) is, for example, 1 / 64 of the size of the t-th input image I(t). Similarly, the size of the second image Id(t-1) obtained by downsampling the t-th reference image IR(t) is, for example, 1 / 64 of the size of the t-th reference image IR(t).

[0027] Subsequently, the motion detection circuit 404 can determine the motion image Im based on the pixel-by-pixel absolute difference between the first image Id(t) and the second image Id(t-1).

[0028] In one embodiment, the motion detection circuit 404 may determine an absolute difference map based on the pixel-by-pixel absolute difference between the first image Id(t) and the second image Id(t-1).

[0029] In one embodiment, the pixel with coordinates (i, j) in the absolute difference map can be represented as di,j (i, j are positive integers), which is, for example, the absolute difference between the pixel with coordinates (i, j) in the first image I d(t) and the pixel with coordinates (i, j) in the second image I d(t-1).

[0030] Subsequently, the motion detection circuit 404 can convert the absolute difference map into a motion image Im based on a first monotonically increasing function (hereinafter referred to as f(.)). In one embodiment, the pixel with coordinates (i, j) in the motion image Im can be represented as mi,j, where mi,j = f(di,j).

[0031] Please refer to Figure 5, which is a schematic diagram of a first monotonically increasing function according to one embodiment of the present invention. In this embodiment, the designer can select appropriate thresholds th1 and th2 as needed.

[0032] In the scenario of Figure 5, for di,j less than the threshold th1, the corresponding mi,j is, for example, 0 (or other values ​​preferred by the designer). Furthermore, for di,j greater than the threshold th2, the corresponding mi,j is, for example, a fixed value (which can be determined by the designer according to requirements). For di,j between the threshold th1 and the threshold th2, the corresponding mi,j may, for example, monotonically increase with the increase of di,j, but is not limited to this.

[0033] Subsequently, the motion detection circuit 404 can convert the motion image Im into a first reference weight map Iw.

[0034] In one embodiment, the weighting circuit 406 can convert the motion image Im into a first reference weight map Iw based on a second monotonically increasing function (hereinafter referred to as g(.)). In one embodiment, the pixel with coordinates (i, j) in the first reference weight map Iw can be represented as wi,j, where wi,j = g(mi,j).

[0035] Please refer to Figure 6, which is a schematic diagram of a second monotonically increasing function according to one embodiment of the present invention. In this embodiment, the designer can select appropriate thresholds th1' and th2' as needed.

[0036] In the scenario of Figure 6, for mi,j less than the threshold th1', the corresponding wi,j is, for example, the value w1 (which can be determined by the designer according to requirements). Additionally, for mi,j greater than the threshold th2', the corresponding wi,j is, for example, the value w2 (which can be determined by the designer according to requirements). For mi,j between the threshold th1' and the threshold th2', the corresponding wi,j may, for example, monotonically increase with the increase of mi,j, but is not limited to this.

[0037] Next, the upsampling circuit 408 can upsample the first reference weight map I w into the first weight map W(t).

[0038] In one embodiment, the downsampling parameter (i.e., S) corresponding to the first image Id(t) and the second image Id(t-1) is the reciprocal of the upsampling parameter (e.g., 1 / S) corresponding to the first weight map W(t).

[0039] Accordingly, the first weight map W(t) obtained by sampling the first reference weight map I w can have the same size as the t-th input image I(t) and the t-th reference image IR(t).

[0040] Referring again to Figure 2, after the weight determination circuit 204 determines the first weight map W(t), in step S330, the guiding filter 206 of the image processor 104 uses the t-th input image as I(t) as the guiding image to perform a guiding filtering operation on the first weight map W(t) to generate a second weight map. In one embodiment, the second weight map can be represented as "".

[0041] In one embodiment, a pixel with coordinates (i, j) can be characterized, for example, as follows.

[0042] In one embodiment, for example, it can be calculated based on the following formula: (1), where, .

[0043] In some embodiments, , .

[0044] The calculation methods and definitions of the parameters related to equation (1) can be summarized in Table 1 below. parameter definition A window ω containing (i, j) k coefficient A window ω containing (i, j) k coefficient The guiding image I(t) is in a window centered at (i, j). Window ω k Total number of pixels Guided images in window ω k The number of variants in Guided images in window ω k 中的像素數平均 The first weighted graph W(t) is in window ω k 中的權重平均 Regularization parameters Table 1

[0045] In other embodiments, details of the guided filtering operation can be found in the literature "Guided image filtering", which will not be repeated here.

[0046] In step S340, the temporal filtering circuit 202 of the image processor 104 performs temporal filtering operation on the t-th input image I(t) and the t-th reference image IR(t) based on the second weight map to generate the t-th output image I'(t).

[0047] In one embodiment, the pixel with coordinates (i, j) in the t-th output image I'(t) is represented as, for example, by the following formula: Equation (2), where is the pixel with coordinates (i, j) in the t-th input image I(t), is the pixel with coordinates (i, j) in the t-th reference image IR(t), and is the pixel with coordinates (i, j) in the second weight map.

[0048] As previously stated, in different embodiments, the t-th reference image IR(t) can be the (t-1)-th input image I(t-1) or the (t-1)-th output image I'(t-1).

[0049] Based on this, in the embodiment where the t-th reference image IR(t) is the (t-1)-th input image I(t-1), equation (2) can be rewritten as, for example, the following equation (3): (3), where ϵ [0.0, 1.0] and is the pixel with coordinates (i, j) in the (t-1)-th input image I(t-1).

[0050] In addition, in the embodiment where the t-th reference image IR(t) is the (t-1)-th output image I'(t-1), equation (2) can be rewritten, for example, as equation (4): where ϵ [0.0, 1.0] and is the pixel with coordinates (i, j) in the (t-1)-th output image I'(t-1).

[0051] Please refer to FIG7, which is a schematic diagram of an input image, a first weight map and a second weight map according to one embodiment of the present invention.

[0052] In Figure 7, the t-th input image I(t) under consideration is, for example, an image of a rider riding a motorcycle, and the image processor 104 can execute steps S320 and S330 of Figure 2 accordingly to generate a first weight map W(t) and a second weight map.

[0053] As can be seen from Figure 7, compared with the first weight map W(t), the pixels corresponding to the moving region are more obvious in the second weight map, and the noise of the pixels corresponding to the stationary region is better suppressed.

[0054] In other words, the second weight map can accurately present the subtle movement of the moving object in the detected t-th input image I(t), thereby improving the subsequent TNR effect.

[0055] From another perspective, the second weight map generated based on the guided filtering concept can better emphasize the pixels of the t-th input image I(t) corresponding to the motion region. Therefore, when the second weight map is used for temporal filtering operations, a better TNR effect can be achieved accordingly.

[0056] In some embodiments, after generating the t-th output image I'(t), the image processor 104 may further perform, for example, object detection or other subsequent image analysis / processing on the t-th output image I'(t). Since the noise in the t-th output image I'(t) is low, the performance of subsequent image analysis / processing can be improved.

[0057] In summary, the methods of the embodiments of the present invention can accurately detect subtle movements of moving objects in images, significantly improving the TNR effect of the images. The method of the embodiments of the present invention improves the accuracy of motion detection through guided filters, enabling it to effectively capture subtle movements of objects, thereby achieving excellent noise reduction performance for stationary areas in the detected images and avoiding artifacts caused by moving objects (especially subtle moving objects) in the detected images.

[0058] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0008] Figure 1 is a schematic diagram of an image processing apparatus according to one embodiment of the present invention. Figure 2 is a functional block diagram of an image processor according to Figure 1. Figure 3 is a flowchart of an image noise reduction method according to one embodiment of the present invention. Figure 4 is an application scenario diagram of determining a first weight map by a weight determination circuit according to Figure 2. Figure 5 is a schematic diagram of a first monotonically increasing function according to one embodiment of the present invention. Figure 6 is a schematic diagram of a second monotonically increasing function according to one embodiment of the present invention. Figure 7 is a schematic diagram of an input image, a first weight map, and a second weight map according to one embodiment of the present invention.

Claims

1. An image noise reduction method, performed by an image processing device, comprising: Obtain the t-th input image and the t-th reference image, where t is a time index value; determine a first weight map based on the t-th input image and the t-th reference image; use the t-th input image as a guide image to perform a guiding filtering operation on the first weight map to generate a second weight map; and perform a temporal filtering operation on the t-th input image and the t-th reference image based on the second weight map to generate a t-th output image.

2. The method as described in claim 1, wherein the t-th reference image includes the (t-1)-th input image or the (t-1)-th output image.

3. The method as described in claim 1, wherein determining the first weight map based on the t-th input image and the t-th reference image includes: The t-th input image is downsampled to form a first image; the t-th reference image is downsampled to form a second image; a moving image is determined based on the pixel-wise absolute difference between the first image and the second image; the moving image is converted into a first reference weight map; and the first reference weight map is upsampled to form the first weight map.

4. The method as described in claim 3, wherein determining the moving image based on the pixel-by-pixel absolute difference between the first image and the second image includes: The absolute difference map is determined based on the pixel-by-pixel absolute difference between the first image and the second image; And the absolute difference map is converted into the motion image based on the first monotonically increasing function.

5. The method as described in claim 3, wherein converting the motion image into the first reference weight map comprises: The motion image is converted into the first reference weight map based on the second monotonically increasing function.

6. The method as described in claim 3, wherein the downsampling parameters corresponding to the first image and the second image are the reciprocal of the upsampling parameters corresponding to the first weight map.

7. The method as described in claim 1, wherein the pixel with coordinates (i, j) in the t-th output image is represented as, where: Wherein, is the pixel with coordinates (i, j) in the t-th input image, is the pixel with coordinates (i, j) in the t-th reference image, and is the pixel with coordinates (i, j) in the second weighted image.

8. An image processing apparatus, comprising: A storage circuit that stores program code; and an image processor coupled to the storage circuit and accessing the program code to execute: acquiring a t-th input image and a t-th reference image, where t is a time index value; determining a first weight map based on the t-th input image and the t-th reference image; performing a guided filtering operation on the first weight map using the t-th input image as a guide image to generate a second weight map; and performing a temporal filtering operation on the t-th input image and the t-th reference image based on the second weight map to generate a t-th output image.

Images