A low-latency architecture for all-frequency noise reduction in image processing
The multi-scale filtering technique with hardware-based architecture addresses the challenge of high latency and complexity in real-time image processing by employing intermediate data storage and resolution scaling, achieving efficient full-frequency noise filtering for real-time applications.
Patent Information
- Application Number
- JP2024535704
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-24
- Filing Date
- 2022-12-07
- Publication Date
- 2024-12-05
AI Technical Summary
Real-time image capture and processing applications, such as video streaming, face challenges in performing full-frequency noise filtering due to high hardware complexity, power consumption, and latency, especially when using large filtering windows.
A low-latency, low-cost image processing system utilizing a multi-scale filtering technique with hardware-based architecture that leverages intermediate data storage and targeted partitioning of memory data flows, applying different resolution scales for low-, mid-, and high-frequency filtering through a multi-scale filtering process involving denoising, downsampling, upsampling, and fusion operations.
The system achieves low-latency, full-frequency noise filtering with reduced complexity and power consumption, enabling efficient real-time image processing by minimizing memory bandwidth requirements and latency through optimized data flow management.
Smart Images

Figure 2024545232000001_ABST
Abstract
Description
[Background technology]
[0001] Image processing for captured images often involves filtering the image for low-, mid-, and high-frequency noise. However, low-frequency noise filtering typically requires a relatively large filtering window and therefore incurs a relatively high cost in terms of complexity, power, and latency in hardware implementations. Real-time image capture applications, such as video streaming, that require high-speed image processing further impede the ability to effectively perform full-frequency noise filtering at a reasonable cost.
[0002] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief description of the drawings]
[0003] [Figure 1] FIG. 1 is a block diagram of an image processing system that performs a low-latency multi-scale full-frequency filtering process in accordance with some embodiments. [Diagram 2] 2 is a flow diagram illustrating a high-level view of the image capture and filtering process of the system of FIG. 1 in accordance with some embodiments. [Diagram 3] FIG. 2 illustrates a hard-coded hardware architecture that implements the low-latency multi-scale full-frequency filtering process of the system of FIG. 1 in accordance with some embodiments. [Figure 4] FIG. 4 illustrates an example operation of a multi-scale full-frequency filtering process using the hardware architecture of FIG. 3 in accordance with some embodiments. [Diagram 5]FIG. 4 illustrates an example operation of a multi-scale full-frequency filtering process using the hardware architecture of FIG. 3 in accordance with some embodiments. [Figure 6] FIG. 4 illustrates an example operation of a multi-scale full-frequency filtering process using the hardware architecture of FIG. 3 in accordance with some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0004] Real-time image capture and processing, such as that found in live video streaming applications, requires low-latency filtering processes to achieve a commanded frame rate. Conventional approaches either rely on heavy software computational loads and thus typically result in significant latency or require large filter windows in hardware-based solutions, and thus are often impractical for implementation in low-cost or low-power systems such as mobile phones. Accordingly, the present disclosure describes systems and techniques for low-latency, low-cost full-frequency noise filtering of images by using image scaling-based filtering techniques, i.e., "multi-scale filtering techniques," that can provide low-, mid-, and / or high-frequency (i.e., "full-frequency") filtering for one or more components of an image (e.g., chrominance and / or luminance), such that different resolution scales at each level of the multi-scale filtering technique provide a larger receptive field for the denoising process employed at each level than conventional denoising frameworks. The present disclosure further provides a hardware architecture for performing multi-scale filtering techniques in hardware (i.e., hard-coded and / or programmable logic) that leverages intermediate data storage in memory and target partitioning of in-out memory data flow to provide lower latency and lower complexity.
[0005] As described in more detail below, this multi-scale filtering involves receiving an input image to be filtered (e.g., the chrominance portion of a captured image) and then performing a multi-scale filtering process in which the input image is denoised, downscaled, upscaled, and fused with the results of the lower resolution scales at different resolution scales to produce a filtered image. In some embodiments, this involves temporarily buffering intermediate image data for some of the resolution scales in memory using direct memory access (DMA) operations.
[0006] More specifically, in some embodiments, the process includes, recursively, for each level of a plurality of levels (e.g., resolution scales) in a first sequence, performing a denoising operation on an image received from a previous level in the first sequence, then performing a downsampling operation on the resulting denoised image, then performing an upsampling operation on the resulting downsampled image, then performing a residual operation to subtract the resulting upsampled image at the level from the denoised image at the level to generate a residual image of the level. The process then further includes, recursively, for each level of a plurality of levels in a second sequence opposite to the first sequence, performing an upscaling operation to upscale a fused image from the previous level, then performing a fusion operation to combine the resulting upscaling image with the residual image of the next level in the second sequence to generate a fused image of the level. The resulting filtered image is then provided for at least one of storage, transmission or encoding. To illustrate, for a given level of the plurality of levels in the first sequence, the recursive process may include receiving an output image from a previous level in the first sequence, performing a denoising operation to generate a denoised image, downsampling the denoised image to generate a downsampled image, providing the downsampled image as an output image to a next level in the first sequence, upsampling the downsampled image to generate an upsampled image, and subtracting the upsampled image from the denoised image to generate a residual image for the level.Further, in some embodiments, for a given level, the process may further include storing a denoised image of the level in memory using at least one direct memory access operation, storing a downsampled image of the level in memory using at least one direct memory access operation, accessing the downsampled image of the level from memory using at least one DMA operation for use in upsampling the downsampled image to generate the upsampled image, and accessing the denoised image of the level from memory using at least one DMA operation for use in subtracting the upsampled image from the denoised image. Receiving the output image from a previous level may include accessing a denoised image of the previous level from memory using at least one DMA operation, the accessed denoised image being the output image from the previous level. Conversely, for a given level of the plurality of levels in the second sequence, the recursive process includes upscaling a fused image from a previous level in the second sequence to generate an upscaled fused image, and combining the upscaled fused image with a residual image of the level to generate a fused image for use in the next level in the second sequence.
[0007] 1 illustrates an image capture and processing system 100 (hereinafter "system 100") using a multi-scale full-frequency filtering process, according to some embodiments. System 100 may be implemented in any of a variety of electronic devices for providing real-time image capture and / or video streaming. Such devices include, for example, mobile phones, laptop computers, desktop computers, video cameras, still image cameras, vehicle navigation systems, and the like.
[0008] System 100 includes a system on a chip (SOC) 102 or other collection of one or more integrated circuit (IC) dies. SOC 102 is connected to an imaging camera 104 and memories 106 and 108 (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), or a combination thereof). Note that while FIG. 1 illustrates memories 106 and 108 as separate memories, in some embodiments, as described below, memory 106 and memory 108 comprise the same memory. SOC 102 includes one or more processors 110 (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), or a combination thereof) and other hardware for executing an image processing pipeline 112. The image processing pipeline 112 includes an image pre-processing stage 114, a multi-scale full frequency filter stage 116 (hereafter "multi-scale filter stage 116" for simplicity), and an image post-processing stage 118. Stages 114-118 may be executed via one or more processors 110 executing software to perform their functions described herein as hard-coded and / or programmable logic of the SOC 102, or a combination thereof. As described in more detail below with reference to FIG. 3, in at least one embodiment, the multi-scale filter stage 116 is executed using a hardware micro-architecture that uses memory 106 as intermediate storage accessed via one or more direct memory access (DMA) engines to perform the multi-scale noise filtering process via an efficient data flow and reduce latency in the data flow.
[0009] 2 illustrates a method 200 illustrating a high-level operation of the image processing pipeline 112 of the system 100 of FIG. 1 according to some embodiments. An iteration of the method 200 begins in block 202 with the capture of an image 120 (FIG. 1) by the imaging camera 104, either as a still image or as an image in a sequence of images for a video stream. Image data for the image 120 is read from the imaging camera 104 into a frame buffer (not shown) implemented in the SRAM or other internal memory of the SOC 102, or into one of the memories 106, 108. Typically, the image data represented in the captured image 120 is presented in a raw format in which the pixel data represents the radiometric properties of the scene captured in the image 120. Thus, in block 204, the image pre-processing stage 114 converts the image data from the raw format to a luminance-chrominance format more suitable for noise filtering to generate a pre-processed image 122 (FIG. 1). In the following, luminance-chrominance formats are described in the exemplary context of YUV (or YCrCb) format, where "Y" represents the luma or luminance component and UV represents the chrominance format. However, other luminance-chrominance formats may be used instead, using the guidelines provided herein. Additionally, although not shown, the image pre-processing stage 114 may use additional image processing techniques, such as color correction or anti-aliasing, in preparing the resulting pre-processed image 122.
[0010] In block 206, the pre-processed image 122 is input to a multi-scale filtering stage 116, which then performs a multi-scale full-frequency noise filtering (or "denoising") process to filter low, medium and high frequency noise in the pre-processed image 122. This process is described in more detail below with reference to Figures 3-6. The resulting filtered image 124 is then provided to an image post-processing stage 118 for post-processing in block 208. This post-processing may include, for example, format or color space conversion of the filtered image 124 (e.g., from YUV to red-blue-green (RGB) format), upscaling, downscaling, cropping, etc. In block 210, the resulting processed image 126 (FIG. 1) is provided to downstream components within the system 100, such as by being stored in memory 108 in an unencoded format for further post-processing, being encoded and then stored in memory 108 in an encoded format, and then being encoded and then transmitted over a network to another device.
[0011] FIG. 3 illustrates a hardware architecture 300 implemented by the multi-scale filtering stage 116 to perform a full-frequency multi-scale noise filtering process as described in more detail below with reference to FIGS. 4-6. However, as a general overview to facilitate understanding of the functionality of the components of the hardware architecture 300 at this point, the multi-scale filtering process employs multi-scale modeling in which multiple models of a system (in this case, the image to be filtered) are processed at different scales (in this case, different image resolution scales). In this approach, the multi-scale filtering process executes a first sequence of levels, then a second sequence of levels opposite to the first sequence, the first sequence starting at two or more scaled filtering levels, and at each scaled filtering level in the first sequence, a noise filtering process (such as a non-local means noise filtering process) is applied to a downsampled version of the image from the previous scaled filtering level (or to the original input image, if a level is at the first scaled filtering level) to generate a noise filtered image at that level, and the noise filtered image is downscaled to generate a downscaled image at that level, so that the downscaled image preserves low frequency content. The downscaled image is then upscaled back to the same resolution as the noise filtered image at that level, and the resulting upscaled image at that level is then subtracted from the noise filtered image at that level to generate residual data at that level that represents the high frequency content of the image at the scale of that level. In the second sequence (called the fusion level), the noise filtered image of the last scaled filtering level is upscaled to the scale of the previous level, and then the upscaled version of that image is combined or "fused" with the residual data generated at the corresponding level.The resulting fused image is then upscaled again to the scale of the next higher level in the second sequence, and the resulting upscaled image is then fused with the residual data generated at the next higher level. This process repeats recursively for each successively higher scaled filtering level in the second sequence until the residual data is combined with an upscaled version of the fused image from the next lower level to generate a final fused image, which is then output as an output filtered image (e.g., filtered image 124).
[0012] Thus, as this overview shows, the multi-scale filtering process has a data flow that utilizes intermediate data buffered between operations and between levels in the scaled sequence. Thus, to improve data flow efficiency and latency, the multi-scale filtering stage 116 uses the memory 106 as intermediate storage for such data during the multi-scale filtering process, and thus the hardware architecture 300 uses a write DMA (WDMA) engine 301 and a read DMA (RDMA) engine 303 coupled to the memory 106. The WDMA engine 301 is configured to receive image data from various functional blocks described herein and write the received image data to the memory 106 via corresponding DMA operations. The RDMA engine 303 is configured to access image data stored in the memory 106 via corresponding DMA operations and provide the accessed image data to the various functional blocks described herein. In some embodiments, each of the WDMA engine 301 and the RDMA engine 303 is a separate DMA engine that operates independently to write image data to the memory 106 and read data from the memory 106. In other embodiments, WDMA engine 301 and RDMA engine 303 are implemented within the same DMA engine (i.e., a single DMA engine), in which case references to WDMA engine 301 or RDMA engine 303 are references to the write DMA configuration or read DMA configuration of that single DMA engine.
[0013] The hardware architecture 300 of the multi-scale filtering stage 116 further includes a number of functional blocks interconnected with each other and with the WDMA engine 301 and the RDMA engine 303 to perform the multi-scale filtering process briefly described above and described in more detail below with reference to Figures 3-6, each functional block implementing hard-coded and / or programmable logic and associated circuitry to perform a corresponding operation of a corresponding level of the multi-scale filtering process. In the following example, the multi-scale filtering process uses three scaled filtering levels (levels 0-2) and a final fusion (FS) level.
[0014] The following naming convention is used for these functional blocks and the resulting image data generated by them: For all blocks, the first "C" stands for "chrominance" since the filtering process is applied to the chrominance component of the image data in this example. For the blocks labeled CNRX, CDSX, CUSX, CRESX, "NR" stands for noise reduction operation, "DS" stands for "downsampling" operation, "US" stands for "upsampling" operation, "RES" stands for "residue calculation" operation, and the number represented by "X" represents the level at which the corresponding operation is performed, with level 0 being the initial scaled filtering level, level 1 being the intermediate scaled filtering level, and level 2 being the final scaled filtering level. Thus, the functional block labeled CNR0 indicates that the functional block performs a noise reduction operation for the initial scaled filtering level, and the functional block labeled CRES1 indicates that the functional block performs a residual calculation operation for the intermediate scaled filtering level. For blocks labeled CFUSX or CFSX, "FUS" refers to a "fusion upscale" operation for upscaling the image from the next lower level, "FS" refers to a "fusion" operation, and X represents the number of fusion upscale operations or fusion operations that have been performed so far. Furthermore, in this naming convention, image data output by a given function block is referenced by appending "_D" to the name given to the function block that generated the output image data. For example, "CRES0_D" refers to residual data generated by the residual operation performed by the CRES0 function block. To further facilitate the distinction between function blocks and the data generated by the function blocks, the function blocks are represented in FIG. 3 using blocks with square corners, while the data generated by the function blocks is represented using blocks with round corners.
[0015] As shown, the hardware architecture 300 includes functional blocks 302, 304, 306, 308, 310, 312, 314, 316, 318, 320, 322, 324, 326. In the following example, the functional blocks operate on the chrominance information of the preprocessed image 122 as a reflection of the observation that the human eye is typically more perceptible to digital noise in the chrominance information of an image than in the luminance information of the same image, and thus, full-frequency noise removal is typically more effective in the chrominance domain than in the luminance domain. Under this approach, the hardware architecture 300 may be replicated such that each of the chrominance channels (e.g., the U and V channels) is processed by a corresponding instance of the hardware architecture 300 in parallel with the other channels, or the hardware architecture 300 may be used to sequentially perform the multi-scale filtering process once for the U channel and then again for the V channel (or vice versa). In yet other embodiments, this same process may be used on the luminance information, either individually or in combination with the processing of the chrominance information.
[0016] Starting from the input, a functional block 302 (denoted as "CNR0 302") is implemented at an initial scaled filtering level (level 0) and has an input that receives the chrominance information of the preprocessed image 122. This chrominance information itself can be seen as a kind of image and is therefore referred to herein as a chrominance image 328. The chrominance image 328 can be provided as an input to the CNR0 302 either directly from the output of the image pre-processing stage 114 or via any of the memories 106, 108, either via DMA access or normal memory controller read operations. The CNR0 302 further includes an output that provides a noise-reduced (or "denoised") image CDNR0_D 330 based on one or more noise reduction operations performed by the CNR0 302 on the chrominance image 328 to remove high frequency components before proceeding to the next sub-process. For example, a non-local means (NLM) denoising operation can be used as the denoising operation. However, other denoising operations may also be used, such as local average denoising, bilateral denoising, guided filter denoising, wavelet denoising, etc. The output of CNR0 302 is coupled to an input of WDMA engine 301 such that the denoised image CNR0_D 330 is stored in memory 106 via one or more write DMA operations. The output of CNR0 302 is also coupled to an input of a functional block 304 (denoted CDS0 304), also implemented at level 0, which operates to apply a downscaling operation to the denoised image CNR0_D 330 to produce a downscaled image CDS0_D 332 at its output, which is coupled to WDMA engine 301 and thus stored in memory 106 via one or more write DMA operations. The original resolution of image 122 is identified herein as OR, and the degree of downscaling implemented by this operation at level 0 is denoted as DS0. For example, DS0 may represent a 50% downscaling of the original resolution, or DS0=0.5*OR (for both vertical and horizontal resolution).
[0017] Functional block 306 (denoted "CNR1 306") has an input for receiving downscaled image CDS0_D 332 implemented at an intermediate scaled filtering level (Level 1) and accessed by RDMA engine 303 from memory 106 via one or more DMA operations at the appropriate time. CNR1 306 further includes an output for providing a noise-reduced (or "de-noised") CNR1_D image based on one or more noise reduction operations performed by CNR1 306 on downscaled image CDS0_D 332. This de-noising operation may be the same type of operation (e.g., NLM noise reduction) as that performed in CNR0 302, or a different type of noise reduction operation. The output of CNR1 306 is coupled to an input of WDMA engine 301 such that de-noised image CNR1_D 334 is stored in memory 106 via one or more write DMA operations. The output of CNR1 306 is also coupled to the input of a functional block 308 (denoted CDS1 308), also implemented at level 1, which operates to apply a downscaling operation to the denoised image CNR1_336 to produce a downscaled image CDS1_D 336 at its output, which is coupled to the WDMA engine 301 and thus stored in the memory 106 via one or more write DMA operations. The degree of downscaling implemented by this operation at level 1 is denoted as DS1, where DS1>DS0. For example, DS1 may represent a 50% downscaling of the downscaled denoised image 334, and thus a 75% downscaling of the original resolution OR, or DS1=0.25*OR (for both vertical and horizontal resolution).
[0018] Functional block 310 (denoted CNR2 310) is implemented with the final scaled filtering level (level 2) and has an input coupled to an output of RDMA engine 303 to receive downscaled image CDS1_D 336 retrieved by RDMA engine 303 from memory 106 via one or more DMA operations at the appropriate time. CNR2 310 further includes an output for providing a noise reduced (or "denoised") image 338 (denoted "CNR2_D 338") based on one or more noise reduction operations performed by CNR2 310 on downscaled image CDS1_D 336. This denoising operation may be the same type of operation as performed by CNR0 302, or may be a different type of noise reduction operation. A functional block 312 (denoted CFSUS1 312) is implemented as a fusion level, has an input coupled to the output of CNR2 310 to receive the denoised image CNR2_D 338, and operates to apply an upscaling operation to the denoised image CNR2_D 338 to result in an upscaled image CFSUS1_D 340 at its output. The degree of upscaling is denoted as FUS1 and is configured to effectively reverse the degree of downscaling DS1 applied to the downscaled image CDS1_D 336 (i.e. FUS1=1 / (1-DS1)). Thus, if downscaling DS1 results in a 50% reduction in resolution, upscaling FUS1 provides a 200% increase in resolution in the resulting upscaled image CFSUS1_D 340.
[0019] Functional block 314 (denoted CUS1 314) is implemented at an intermediate filtering level (level 1) and has an input coupled to the output of RDMA engine 303 to receive in parallel downscaled image CDS1_D 336 retrieved by RDMA engine 303 from memory 106 for input to CNR2 310. CUS1 314 operates to apply an upscaling operation to downscaled image CDS1_D 336 to result in upscaled image CUS1_D 342 at its output. The degree of upscaling is denoted as US1 and is configured to effectively reverse the degree of downscaling DS1 applied to downscaled image CDS1_D 336 (i.e. US1=FS1=1 / (1-DS1)). Thus, if downscaling DS1 results in a 50% reduction in resolution, upscaling US1 provides a 200% increase in resolution in the resulting upscaled image CUS1_D 342. Functional block 316 (denoted CRES1 316) is implemented at level 1 and has an input coupled to the output of CUS1 314 to receive the upscaled image CUS1_D 342 and an input coupled to the output of RDMA Engine 303 to receive the denoised image CNR1_D 334 retrieved by RDMA Engine 303 from Memory 106 via one or more DMA operations at the appropriate time. CRES1 316 operates to subtract the upscaled image CUS1_D 342 from the denoised image CNR1_D 334 to generate a residual image CRES1_D 344 for output (thus preserving high frequency signals but removing lower frequency signals).
[0020] A functional block 318 (denoted CFS1 318) is implemented at the fusion level, has an input coupled to the output of CFSUS1 312 to receive the upscaled image CFSUS1_D 340, and an input for receiving the residual image CRES1_D 344, and is operative to superimpose or combine the images 340 and 344 to generate a fused image CFS1_D 346. A functional block 320 (denoted CFSUS2 320) is also implemented at the fusion level, has an input coupled to the output of CFS1 318 to receive the fused image CFS1_D 346, and is operative to apply an upscaling operation to the fused image CFS1_D 346 to result at its output an upscaled image CFSUS2_D 348. The degree of upscaling is denoted as FUS2 and is configured to effectively invert the degree of downscaling DS0 applied to the downscaled image CDS0_D 332 (i.e. FUS2=1 / (1-DS0)). Thus, if downscaling DS0 results in a 50% reduction in resolution, upscaling FUS1 provides a 200% increase in resolution in the resulting upscaled image CFSUS2_D348.
[0021] Functional block 322 (denoted CUS0 322) is implemented with an initial scaled filtering level (level 0) and has an input coupled to an output of RDMA engine 303 to receive downscaled image CDS0_D 332 retrieved by RDMA engine 303 from memory 106 at the appropriate time. CUS0 322 operates to apply an upscaling operation to downscaled image CDS0_D 332 to result in upscaled image CUS0_D 350 at its output. The degree of upscaling is denoted as US0 and is configured to effectively reverse the degree of downscaling DS0 applied to downscaled image CDS0_D 332 (i.e., US0=FUS0=1 / (1-DS0)). Thus, if downscaling DS0 results in a 50% reduction in resolution, upscaling US0 provides a 200% increase in resolution in the resulting upscaled image CUS0_D 350. Functional block 324 (denoted CRES0 316) is implemented at Level 0 and has an input coupled to an output of CUS0 322 to receive upscaled image CUS0_D 350 and an input coupled to an output of RDMA Engine 303 to receive denoised image CNR0_D 330 retrieved by RDMA Engine 303 from memory 106 via one or more DMA operations at the appropriate time. CRES0 324 operates to subtract upscaled image CUS0_D 350 from denoised image CNR0_D 330 to generate residual image CRES0_D 352 for output (thus removing lower frequency signals while preserving high frequency signals).
[0022] A functional block 326 (denoted CFS2 326) is implemented at the fusion level and has an input coupled to the output of CFSUS2 312 to receive the upscaled image CFSUS2_D 348, and an input for receiving the residual image CRES0_D 352, and operates to overlay or combine the images 348 and 352 to generate a filtered chrominance image 354, which contains the chrominance components of the filtered image (124), which is then output to the image post-processing stage 118 either directly or via intermediate storage in memory 106 or memory 108.
[0023] The data flow and operational flow employed using the hardware architecture 300 to perform the multi-scale filtering process 400, together with the individual connections and operation of the functional blocks 302-326 and DMA engines 301, 303 of the hardware architecture 300 for the described multi-scale filtering stage 116, are shown in Figures 4-6.
[0024] 4, in a first time slice (times T0-T2), the chrominance image 328 representing the chrominance information of the preprocessed image 122 is input to a level 0 in the initial scaled filtering level, after which it is input to the CNR0 302, which performs an NLM denoising operation or other denoising operation to generate a denoised image CNR0_D 330 (times T0-T1). The WDMA engine 301 performs a write operation 402 to write the denoised image CNR0_D 330 to the memory 106 for subsequent access. The CDS0 304 downscales the denoised image CNR0_D 330 output by the CNR0 302 by a downscaling factor DS0 (e.g., 50%) to generate a downscaled image CDS0_D 332 (times T1-T2). The WDMA engine 301 performs a write DMA operation 404 to write the downscaled image CDS0_D 332 to the memory 106 for subsequent access.
[0025] 5, in a subsequent second time slice (times T2-T4), the RDMA engine 303 performs a read DMA operation 502 to access the stored downscaled image CDS0_D 332 from memory 106 and provide it to CNR1 306, which applies a denoising operation to the downscaled image CDS0_D 332 (times T2-T3) to generate a denoised image CNR1_D 334. The WDMA engine 301 performs a write DMA operation 504 to buffer the denoised image CNR1_D 334 in memory 106. CDS1 308 then downscales the denoised image CNR1_D334 output by CNR1 306 by a downscaling factor DS1 (e.g., a further 50%) to generate a downscaled image CDS1_D336 (times T3 to T4), and the WDMA engine 301 performs a write DMA operation 506 to buffer the downscaled image CDS1_D336 in memory 106.
[0026] In a subsequent third time slice (T4-T5), the RDMA engine 303 performs a read DMA operation 508 to access the downscaled image CDS1_D 336 from the memory 106 and provides the accessed downscaled image CDS1_D 336 at an intermediate scaled filtering level (level 2) to CRN2 310. CRN2 310 performs an NLM denoising operation (or other denoising operation) to generate a denoised image CNR2_D 338.
[0027] Briefly referring to Figure 6, in a subsequent fourth time slice (T5-T6), the denoised image CNR2_D338 is provided to the fusion level, after which CFSUS1 312 performs an upscaling operation using an upscaling factor FUS1 to upscale the denoised image CNR2_D338 to generate an upscaled image CFSUS1_D340. Referring again to Figure 5, during this same time slice, the RDMA engine 303 performs a read DMA operation 510 to access the stored downscaled image CDS1_D336 from the memory 106 and provides it to CUS1 314, which performs an upscaling operation using an upscaling factor US1 to upscale the downscaled image CDS1_D336 to generate an upscaled image CUS1_D342 having the same resolution as the upscaled image CFSUS1_D340, and the upscaled image CUS1_D342 is input to CRES1 316. Concurrently, the RDMA engine 303 performs a read DMA operation 512 to access the denoised image CNR1_D 334 from memory 106 and provides it as an input to CRES1 316. CRES1 316 then subtracts the upscaled image CUS1_D 342 from the denoised image CNR1_D 334 to generate a residual image 344. Returning to Figure 5, during this same time slice, the upscaled image CFSUS1_D 340 and the residual image CRES1_D 344 are input to CFS1 318, which overlays or "fuse" these input images to generate a fused image CFS1_D 346.
[0028] Referring again to FIG. 6, in a subsequent fifth time slice (times T6 to T7), the output fused image CFS1_D346 is input to CFSUS2 340, which performs an upscaling operation to upscale the fused image CFS1_D346 by upscaling factor FUS1 to generate an upscaled image CFSUS2_D348 having the original resolution OR of the chrominance image 328. Returning to FIG. 4, simultaneously, in the fifth time slice (T6-T7), the WDMA engine 301 performs a read DMA operation 406 to access the stored downscaled image CDS0_D332 from the memory 106 and provides the downscaled image CDS0_D332 to CUS0 332, which performs an upscale operation to upscale the image CDS0_D332 by an upscale factor US0 to generate an upscaled image CUS0_D350 having the same resolution as the original resolution OR of the chrominance image 328.
[0029] For the sixth and final time slice (times T7-T8), the RDMA engine 303 performs a read DMA operation 408 to access the denoised image CNR0_D 330 from the memory 106 and provides the accessed image to an input of CRES0 324. CRES0 324 subtracts the upscaled image CUS0_D 350 from the accessed denoised image CNR0_D 330 to generate a residual image CRES0_D 352. Referring again to FIG. 6, the residual image CRES0_D 352 and the upscaled image CFSUS2_D image are input to CFS2 326 at the fusion level, which performs a registration operation to register the residual image CRES0_D 352 with the upscaled image CFSUS2_D 348 to generate an output filtered chrominance image 354.
[0030] Thus, as described above, the multi-scale filtering process 400 and the hardware architecture 300 executing this process together provide low latency full frequency image filtering by using temporary DMA-based storage and subsequent access of intermediate images from memory in an end-to-end flow to provide improved latency and bandwidth. In particular, note the number of intermediate images in the multi-scale full frequency filtering process described above, such as three denoised images, two downscaled images, two residual images, four upscaled images, and two fused images, the last of which serves as the final output. Using the architecture and process described above, only two downscaled images and two denoised images are utilized, which provides a minimal amount of data to store in and access from memory, leading to significantly reduced memory bandwidth requirements. Furthermore, the intermediate images are processed by the next level immediately after they are generated in the current level. Thus, the latency mainly depends on the filter window size and memory latency of the noise reduction, downscale and upscale processes, thus resulting in an overall lower latency compared to conventional approaches.
[0031] In some embodiments, the above apparatus and techniques are implemented in a system that includes one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as SOC 102 described above with reference to FIGS. 1-6. Electronic design automation (EDA) and computer aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. The one or more software programs include code executable by a computer system for operating the computer system to operate on code representing circuits of one or more IC devices to perform at least a portion of a process for designing or adapting a manufacturing system for manufacturing the circuits. This code may include instructions, data, or a combination of instructions and data. The software instructions representing the design or manufacturing tools are typically stored in a computer readable storage medium accessible to the computing system. Similarly, code representing one or more stages of the design or manufacture of the IC devices is stored in and accessed from the same computer readable storage medium or a different computer readable storage medium.
[0032] A computer-readable storage medium includes any non-transitory storage medium or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tape, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMS) based storage media. The computer-readable storage medium (e.g., system RAM or ROM) may be internal to the computing system, the computer-readable storage medium (e.g., a magnetic hard drive) may be permanently attached to the computing system, the computer-readable storage medium (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory) may be removably attached to the computing system, or the computer-readable storage medium (e.g., network-accessible storage (NAS)) may be coupled to the computer system via a wired or wireless network.
[0033] In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by the one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s), etc. The executable instructions stored in the non-transitory computer-readable storage medium may be implemented as source code, assembly language code, object code, or other form of instructions that can be interpreted or otherwise executed by one or more processors.
[0034] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, some of the specific activities or devices may not be required, one or more additional activities may be performed, and one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will appreciate that various changes and modifications can be made without departing from the scope of the invention as set forth in the claims. Thus, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the invention.
[0035] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and features from which any benefit, advantage, or solution may arise or be manifested are not to be construed as critical, essential, or essential features of any or all of the claims. Moreover, the specific embodiments described above are illustrative only, as the disclosed invention may be modified and practiced in different but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and all such variations are considered to be within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.
Claims
1. 1. A method for denoising an image, comprising: performing a multi-scale filtering process in which the input image is denoised, downscaled, upscaled and merged with the result of the lower resolution scale to generate a filtered image; method.
2. performing the multi-scale filtering process includes temporarily buffering intermediate image data of some of the resolution scales in a memory using direct memory access (DMA) operations; 2. The method of claim 1.
3. Each operation at a corresponding resolution scale is performed by a corresponding separate functional block of a hard-coded hardware architecture of a system on chip (SOC) coupled to the memory, the hardware architecture including at least one DMA engine; The method of claim 2.
4. The input image is the chrominance portion of an image captured by an imaging camera.
2. The method of claim 1.
5. 1. A method for multi-scale noise filtering of an image, comprising: Receiving an input image to be filtered; for each level of a plurality of levels in a first sequence, performing a denoising operation on an image received from a previous level in the first sequence, performing a downsampling operation on the resulting denoised image, performing an upsampling operation on the resulting downsampled image, and recursively performing a residual operation to subtract the resulting upsampled image from the denoised image at that level to generate a residual image for that level; performing, for each level of the plurality of levels in a second sequence opposite to the first sequence, an upscaling operation to upscale a fused image of a previous level, and recursively performing a fusion operation to combine the resulting upscaled fused image with the residual image of a next level in the second sequence to generate a fused image of that level; and providing the resulting final fused image as a filtered image for at least one of storage, transmission, or encoding. method.
6. For any one of the plurality of levels in the first sequence, receiving an output image from a previous level in the first sequence; performing a denoising operation to generate a denoised image; downsampling the denoised image to generate a downsampled image; providing the downsampled image as an output image to the next level in the first sequence; upsampling the downsampled image to generate an upsampled image; and subtracting the upsampled image from the denoised image to generate a residual image at the level. The method of claim 5.
7. For any one of the plurality of levels in the first sequence, storing said denoised image at said levels in a memory using at least one direct memory access operation; storing said downsampled image at said level in said memory using at least one direct memory access operation; accessing the downsampled image at the level of the memory using at least one DMA operation for use in upsampling the downsampled image to generate the upsampled image; and accessing the denoised image at said level of said memory using at least one DMA operation for use in subtracting the upsampled image from the denoised image. The method of claim 6.
8. receiving an output image from the previous level includes accessing the denoised image of the previous level in the memory using at least one DMA operation, the accessed denoised image being the output image from the previous level. The method of claim 7.
9. Each operation performed at each of the plurality of levels is performed by a corresponding separate functional block of a hard-coded hardware architecture of a system on a chip (SOC) coupled to a memory, the hardware architecture including at least one DMA engine that performs DMA operations for the functional blocks of the hard-coded hardware architecture.
9. The method of claim 8.
10. For any one of the plurality of levels in the second sequence, upscaling a fused image from a previous level in the second sequence to generate an upscaled fused image; and combining the upscaled fusion image with the residual image of the level to generate a fusion image for use in a next level in the second sequence.
9. The method of claim 8.
11. For any one of the plurality of levels in the second sequence, upscaling a fused image from a previous level in the second sequence to generate an upscaled fused image; and combining the upscaled fusion image with the residual image of the level to generate a fusion image for use in a next level in the second sequence. The method of claim 6.
12. The input image is a chrominance image of an image captured by an imaging camera. The method of claim 5.
13. 1. An image processing system, comprising: an input for receiving an input image; a multi-scale filtering stage adapted to carry out the method of any of claims 1 to 12. Image processing system.
14. a memory coupled to the multi-scale filtering stage; the multi-scale filtering stage comprises at least one direct memory access (DMA) engine for temporarily buffering intermediate image data of several resolution scales in the memory; 14. The image processing system of claim 13.
15. each operation at a corresponding resolution scale is performed by a corresponding separate functional block of a hard-coded hardware architecture of a system-on-chip (SOC) coupled to said memory; 15. The image processing system of claim 14.
16. an imaging camera connected to the input and configured to capture an image of a scene, the input image being a chrominance portion of the image captured by the imaging camera; 14. The image processing system of claim 13.
17. the multi-scale filtering stage comprises a system on chip (SoC) having a hardware architecture implementing a set of hardware functional blocks; 14. The image processing system of claim 13.
18. The SOC is configured to perform the method of claim 5 , The set of hardware function blocks includes: For any one of the plurality of levels in the first sequence, receiving an output image from a previous level in the first sequence; performing a denoising operation to generate a denoised image; downsampling the denoised image to generate a downsampled image; providing the downsampled image as an output image to the next level in the first sequence; upsampling the downsampled image to generate an upsampled image; subtracting the upsampled image from the denoised image to generate a residual image at said levels; [0023] 20. The image processing system of claim 17.
19. The set of hardware function blocks includes: For any one of the plurality of levels in the first sequence, storing the denoised image at said levels in a memory using at least one direct memory access (DMA) operation by a DMA engine; storing said downsampled image at said level in said memory using at least one direct memory access operation by a DMA engine; accessing the downsampled image at the level of the memory using at least one DMA operation by a DMA engine for use in upsampling the downsampled image to generate the upsampled image; accessing the denoised image at said level of said memory using at least one DMA operation by a DMA engine for use in subtracting the upsampled image from the denoised image; [0023] 20. The image processing system of claim 18.
20. the set of hardware functional blocks is configured to receive the output image from the previous level by accessing the denoised image of the previous level of the memory using at least one DMA operation of a DMA engine, the accessed denoised image being the output image from the previous level; 20. The image processing system of claim 19.
21. The set of hardware function blocks includes: For any one of the plurality of levels in the second sequence, upscaling a fused image from a previous level in the second sequence to generate an upscaled fused image; combining the upscaled fusion image with the residual image of the level to generate a fusion image for use in a next level in the second sequence; [0023] 20. The image processing system of claim 18.
22. an imaging camera connected to the input and configured to capture an image of a scene, the input image being a chrominance portion of the image captured by the imaging camera; 20. The image processing system of claim 17.
Citation Information
Patent Citations
Image processing apparatus, program, method and image capturing system
JP2010183182A
Parallel scaling
JP2015513837A
