Image denoising method and image processing device

Abstract

Embodiments of the present application provide an image denoising method and an image processing device. The method comprises: obtaining a tth input image and a tth reference image; determining a first weight map based on the tth input image and the tth reference image; performing a guided filtering operation on the first weight map with the tth input image as a guide image to generate a second weight map; and performing a temporal filtering operation on the tth input image and the tth reference image based on the second weight map to generate a tth output image.

Description

Technical Field

[0001] This invention relates to an image processing method, and more particularly to an image noise reduction method and an image processing apparatus. Background Technology

[0002] Currently, many video analytics applications (such as video surveillance, facial recognition, and target tracking) have increasingly higher requirements for image quality to ensure analytical accuracy and meet the needs of human visual observation. However, in high-noise environments, especially in low-light scenes, image quality often degrades significantly, thus affecting the reliability of the analysis results. To improve image quality, temporal noise reduction (TNR) processing of video has become a crucial technique.

[0003] Accurate motion detection is fundamental to achieving the following two key objectives when performing temporal denoising: (1) achieving efficient denoising in static areas; and (2) avoiding ghost effects on moving objects, especially when dealing with objects with minute movements.

[0004] Therefore, for those skilled in the art, how to design a means to improve the image quality and analysis accuracy of TNR through accurate motion detection is an important issue. Summary of the Invention

[0005] In view of this, the present invention provides an image noise reduction method and an image processing apparatus, which can be used to solve the above-mentioned technical problems.

[0006] This invention provides an image denoising method, executed by an image processing device, comprising: acquiring a t-th input image and a t-th reference image, where t is a time index value; determining a first weight map based on the t-th input image and the t-th reference image; performing a guided filtering operation on the first weight map using the t-th input image as a guiding image to generate a second weight map; and performing a temporal filtering operation on the t-th input image and the t-th reference image based on the second weight map to generate a t-th output image.

[0007] This invention provides an image processing apparatus, including a storage circuit and an image processor. The storage circuit stores program code. The image processor is coupled to the storage circuit and accesses the program code to execute: acquiring a t-th input image and a t-th reference image, where t is a time index value; determining a first weight map based on the t-th input image and the t-th reference image; performing a guided filtering operation on the first weight map using the t-th input image as a guiding image to generate a second weight map; and performing a temporal filtering operation on the t-th input image and the t-th reference image based on the second weight map to generate a t-th output image. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of an image processing apparatus according to an embodiment of the present invention.

[0009] Figure 2 It is based on Figure 1 The diagram shows the functional block diagram of the image processor.

[0010] Figure 3 This is a flowchart illustrating an image noise reduction method according to an embodiment of the present invention.

[0011] Figure 4 It is based on Figure 2 The diagram illustrates an application scenario where the weight determination circuit determines the first weight map.

[0012] Figure 5 This is a schematic diagram of a first monotonically increasing function according to an embodiment of the present invention.

[0013] Figure 6 This is a schematic diagram of the second monotonically increasing function according to an embodiment of the present invention.

[0014] Figure 7 This is a schematic diagram illustrating the input image, the first weight map, and the second weight map according to an embodiment of the present invention. Detailed Implementation

[0015] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same element symbols are used in the drawings and description to denote the same or similar parts.

[0016] Please refer to Figure 1 This is a schematic diagram of an image processing apparatus according to an embodiment of the present invention. In different embodiments, the image processing apparatus 100 may be implemented as various intelligent devices and / or computer devices, but is not limited thereto.

[0017] exist Figure 1 In the image processing device 100, there are storage circuits 102 and image processors 104.

[0018] The storage circuit 102 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar device or combination of these devices, and may be used to record multiple program codes or modules.

[0019] The image processor 104 is coupled to the storage circuit 102 and may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, multiple microprocessors, one or more microprocessors incorporating a digital signal processor core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), any other type of integrated circuit, a state machine, an advanced RISC-based processor (ARM), and the like.

[0020] Please refer to Figure 2 It is based on Figure 1 The diagram illustrates the functional block diagram of an image processor. Figure 2 In the image processor 104, there are a temporal filtering circuit 202, a weight determination circuit 204 and a guide filter 206, wherein the guide filter 206 is coupled between the temporal filtering circuit 202 and the weight determination circuit 204.

[0021] In an embodiment of the present invention, the image processor 104 may access the modules and program code recorded in the storage circuit 102 to implement the image noise reduction method proposed in the present invention, the details of which are described below.

[0022] Please refer to Figure 3 This is a flowchart illustrating an image noise reduction method according to an embodiment of the present invention. The method of this embodiment can be derived from... Figure 1 The various components in the image processing device 100 work together, as described below. Figure 1 , Figure 2 Component description shown Figure 3 Details of each step.

[0023] In step S310, the image processor 104 obtains the t-th input image I(t) and the t-th reference image IR(t), where t is the time index value.

[0024] In embodiments of the present invention, the image processor 104 may acquire various images from a considered image source. In different embodiments, the image source may be, for example, a monitor capable of capturing images of the monitored area in real time and / or storage space (e.g., an image database) storing images to be analyzed and / or processed, but is not limited thereto.

[0025] In an embodiment of the present invention, the t-th input image is, for example, the t-th image among a plurality of images obtained by the image processor 104 from the aforementioned image source.

[0026] Additionally, the t-th reference image IR(t) is, for example, a reference image corresponding to the t-th input image I(t). In one embodiment, the t-th reference image IR(t) is, for example, the (t-1)-th input image (hereinafter referred to as I(t-1)), that is, the previous input image of the t-th input image I(t), but it is not limited to this.

[0027] In an embodiment of the present invention, after obtaining the t-th input image I(t) and the t-th reference image IR(t), the image processor 104 may subsequently execute steps S320 to S340 to generate the corresponding t-th output image I'(t). In one embodiment, the t-th output image I'(t) may serve as a reference image corresponding to the (t+1)-th input image (i.e., the (t+1)-th reference image).

[0028] Based on the principle of similarity, in some embodiments, the t-th reference image IR(t) can also be the (t-1)-th output image (hereinafter referred to as I'(t-1)), that is, the previous output image of the t-th output image I'(t), but it is not limited to this.

[0029] In step S320, the weight determination circuit 204 of the image processor 104 determines the first weight map W(t) based on the t-th input image I(t) and the t-th reference image IR(t). The following will supplement... Figure 4 Explain the details of step S320.

[0030] Please refer to Figure 4 ,in Figure 4 It is based on Figure 2 The diagram illustrates an application scenario where the weight determination circuit determines the first weight map.

[0031] exist Figure 4 In the process, the weight determination circuit 204 includes a downsampling circuit 402, a motion detection circuit 404, a weight calculation circuit 406, and an upsampling circuit 408 that are sequentially coupled.

[0032] In this embodiment, the downsampling circuit 402 can downsample the t-th input image I(t) into the first image I. d(t), and downsampling the t-th reference image IR(t) into the second image I. d (t-1).

[0033] In one embodiment, the downsampling circuit 402 can perform the aforementioned downsampling on the t-th input image I(t) and the t-th reference image IR(t) based on preset downsampling parameters. For example, assuming the selected downsampling parameter is S (e.g., 8), the first image I obtained by downsampling the t-th input image I(t) is... d The size of I(t) is, for example, 1 / S of the t-th input image I(t). 2 (For example, 1 / 64). Similarly, the second image I obtained by downsampling the t-th reference image IR(t) d The size of (t-1) is, for example, 1 / S of the t-th reference image IR(t). 2 (For example, 1 / 64).

[0034] Subsequently, the motion detection circuit 404 can, based on the first image I, d (t) and the second image I d The pixel-wise absolute difference between (t-1) determines the motion image I. m .

[0035] In one embodiment, the motion detection circuit 404 may be based on the first image I d (t) and the second image I d The pixel-by-pixel absolute difference determination absolute difference map between (t-1).

[0036] In one embodiment, the pixel with coordinates (i,j) in the absolute difference map can be represented as d. i,j (i, j are positive integers), for example, the first image I. d The pixel at coordinates (i,j) in (t) and the second image I d The absolute difference between pixels with coordinates (i,j) in (t-1).

[0037] Subsequently, the motion detection circuit 404 can convert the absolute difference map into a motion image I based on a first monotonically increasing function (hereinafter referred to as f(.)). m In one embodiment, the moving image I m The pixel with coordinates (i,j) in the middle can be represented as m. i,j , where m i,j =f(d i,j ).

[0038] Please refer to Figure 5This is a schematic diagram of a first monotonically increasing function according to an embodiment of the present invention. In this embodiment, the designer can select appropriate thresholds th1 and th2 as needed.

[0039] exist Figure 5 In the context of d being less than the threshold th1 i,j In other words, the corresponding m i,j For example, it could be 0 (or another value preferred by the designer). Additionally, for d values ​​greater than the threshold th2... i,j In other words, the corresponding m i,j For example, it can be a fixed value (which can be determined by the designer according to requirements). For d between threshold th1 and threshold th2... i,j In other words, the corresponding m i,j For example, it can be along with d i,j It increases monotonically with the increase of , but it is not limited to this.

[0040] Then, the motion detection circuit 404 can detect the motion image I m Convert to first reference weight map I w .

[0041] In one embodiment, the weighting circuit 406 may weight the motion image I based on a second monotonically increasing function (hereinafter referred to as g(.)). m Convert to first reference weight map I w In one embodiment, the first reference weight map I w The pixel with coordinates (i,j) in the middle can be represented as w i,j , where w i,j =g(m i,j ).

[0042] Please refer to Figure 6 This is a schematic diagram of a second monotonically increasing function according to an embodiment of the present invention. In this embodiment, the designer can select appropriate thresholds th1' and th2' as needed.

[0043] exist Figure 6 In the context of m being less than the threshold th1', i,j In other words, the corresponding w i,j For example, the value w1 (which can be determined by the designer according to requirements). Additionally, for m greater than the threshold th2'... i,j In other words, the corresponding w i,j For example, the value w2 (which can be determined by the designer according to requirements). For m between threshold th1' and threshold th2'... i,j In other words, the corresponding w i,j For example, it can be along with m i,j It increases monotonically with the increase of , but it is not limited to this.

[0044] Next, the upsampling circuit 408 can convert the first reference weight map I w The upsampling is the first weighted graph W(t).

[0045] In one embodiment, the first image I d (t) and the second image I d The downsampling parameter (i.e., S) corresponding to (t-1) is the reciprocal of the upsampling parameter (e.g., 1 / S) corresponding to the first weight map W(t).

[0046] Therefore, based on the first reference weight map I w The first weighted image W(t) obtained by upsampling can have the same size as the t-th input image I(t) and the t-th reference image IR(t).

[0047] Please refer to this again. Figure 2 After the weight determination circuit 204 determines the first weight map W(t), in step S330, the guiding filter 206 of the image processor 104 uses the t-th input image as I(t) as the guiding image to perform a guiding filtering operation on the first weight map W(t) to generate the second weight map W. gf (t). In one embodiment, the second weight map W gf (t) can be characterized as "W gf (t)=Guided_Filtering(I(t),W(t))".

[0048] In one embodiment, W gf The pixel with coordinates (i,j) in (t) can be characterized as follows:

[0049]

[0050] In one embodiment, For example, it can be calculated based on the following formula:

[0051]

[0052] ,in

[0053] In some embodiments,

[0054]

[0055] The calculation methods and definitions of the parameters related to equation (1) can be summarized in Table 1 below.

[0056]

[0057] Table 1

[0058] In other embodiments, details of the guided filtering operation can be found in the literature "Guided image filtering", which will not be repeated here.

[0059] In step S340, the temporal filtering circuit 202 of the image processor 104 is based on the second weight map W. gf (t) Perform a time-domain filtering operation on the t-th input image I(t) and the t-th reference image IR(t) to produce the t-th output image I'(t).

[0060] In one embodiment, the pixel with coordinates (i,j) in the t-th output image I'(t) is represented as I. i ′ ,j (t), for example, can be calculated using the following formula:

[0061]

[0062] , where I i,j (t) represents the pixel with coordinates (i,j) in the t-th input image I(t), IR i,j (t) represents the pixel with coordinates (i,j) in the t-th reference image IR(t). For the second weighted graph W gf The pixel with coordinates (i,j) in (t).

[0063] As previously mentioned, in different embodiments, the t-th reference image IR(t) can be the (t-1)-th input image I(t-1) or the (t-1)-th output image I'(t-1).

[0064] Therefore, in an embodiment where the t-th reference image IR(t) is the (t-1)-th input image I(t-1), equation (2) can be rewritten as, for example, the following equation (3):

[0065]

[0066] ,in And I i,j (t-1) represents the pixel with coordinates (i,j) in the (t-1)th input image I(t-1).

[0067] Furthermore, in an embodiment where the t-th reference image IR(t) is the (t-1)-th output image I'(t-1), equation (2) can be rewritten, for example, as equation (4):

[0068]

[0069] ,in And I i′ ,j (t-1) represents the pixel with coordinates (i,j) in the (t-1)th output image I'(t-1).

[0070] Please refer to Figure 7 It is a schematic diagram of the input image, the first weight map and the second weight map according to an embodiment of the present invention.

[0071] exist Figure 7 In this context, the t-th input image I(t) under consideration is, for example, an image of a rider on a motorcycle, and the image processor 104 can execute [the necessary actions]. Figure 2 Steps S320 and S330 are performed to generate the first weight map W(t) and the second weight map W(t) accordingly. gf (t).

[0072] Depend on Figure 7 It can be seen that, compared to the first weighted graph W(t), the second weighted graph W gf In (t), the pixels corresponding to the moving region are more prominent, and the noise of the pixels corresponding to the stationary region is better suppressed.

[0073] In other words, the second weighted graph W gf (t) can accurately represent the subtle movement of moving objects in the detected t-th input image I(t), thereby improving the subsequent TNR effect.

[0074] From another perspective, the second weight graph W generated based on the concept of guided filtering... gf (t) can better emphasize the pixels in the t-th input image I(t) corresponding to the motion region. Therefore, when the second weight map W... gf When (t) is used for time-domain filtering, it can achieve better TNR results.

[0075] In some embodiments, after generating the t-th output image I'(t), the image processor 104 may perform further image analysis / processing on the t-th output image I'(t), such as object detection or other subsequent image analysis / processing. Since the noise in the t-th output image I'(t) is low, the performance of subsequent image analysis / processing can be improved.

[0076] In summary, the methods of this invention can accurately detect subtle movements of moving objects in images, significantly improving the TNR (Time-to-Noise Reduction) performance of the images. The methods of this invention improve the accuracy of motion detection through guided filters, enabling them to effectively capture subtle movements of objects. This results in superior noise reduction performance for stationary regions in the detected images and avoids artifacts caused by moving objects (especially subtly moving objects) in the detected images.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image noise reduction method, performed by an image processing device, characterized in that, include: Obtain the t-th input image and the t-th reference image, where t is the time index value; The first weight map is determined based on the t-th input image and the t-th reference image; Using the t-th input image as a guide image, a guided filtering operation is performed on the first weight map to generate a second weight map; as well as Based on the second weight map, a temporal filtering operation is performed on the t-th input image and the t-th reference image to generate the t-th output image.

2. The method of claim 1, wherein the t-th reference image includes the (t-1)-th input image or the (t-1)-th output image.

3. The method of claim 1, wherein determining the first weight map based on the t-th input image and the t-th reference image includes: The t-th input image is downsampled to become the first image; The t-th reference image is downsampled into a second image; The motion image is determined based on the pixel-by-pixel absolute difference between the first image and the second image; The motion image is converted into a first reference weight map; The first reference weight map is upsampled to become the first weight map.

4. The method of claim 3, wherein determining the moving image based on the pixel-by-pixel absolute difference between the first image and the second image comprises: The absolute difference map is determined based on the pixel-by-pixel absolute difference between the first image and the second image; as well as The absolute difference map is converted into the motion image based on the first monotonically increasing function.

5. The method of claim 3, wherein converting the motion image into the first reference weight map comprises: The motion image is converted into the first reference weight map based on the second monotonically increasing function.

6. The method of claim 3, wherein the downsampling parameters corresponding to the first image and the second image are the reciprocal of the upsampling parameters corresponding to the first weight map.

7. The method of claim 1, wherein the pixel with coordinates (i,j) in the t-th output image is represented as I. , (t), where: Where I , (t) represents the pixel with coordinates (i,j) in the t-th input image, IR , (t) represents the pixel with coordinates (i,j) in the t-th reference image. The pixel with coordinates (i,j) in the second weighted graph.

8. An image processing apparatus, characterized in that, include: Storage circuitry that stores program code; as well as An image processor, which is coupled to the storage circuit and accesses the program code to execute: Obtain the t-th input image and the t-th reference image, where t is the time index value; The first weight map is determined based on the t-th input image and the t-th reference image; Using the t-th input image as a guide image, a guided filtering operation is performed on the first weight map to generate a second weight map; as well as Based on the second weight map, a temporal filtering operation is performed on the t-th input image and the t-th reference image to generate the t-th output image.

Images