Noise removal for SDR-HDR local reconstruction

Abstract

An example method relates to generating an image in a first dynamic range from an input image in a second dynamic range lower than the first dynamic range. A first array of co-occurrence statistics for the input image in the second dynamic range is generated. A second array of reconstruction function index values ​​for selecting a reconstruction function from among a plurality of candidate reconstruction functions is generated. A denoising operation is performed based at least in part on the first array of co-occurrence statistics while a reconstruction operation is performed based at least in part on the second array of reconstruction function index values ​​to generate a reconstructed image in the first dynamic range.

Description

[Technical field]

[0001] [REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 63 / 283,684, filed November 29, 2021, and European Patent Application No. 21210983.9, filed November 29, 2021, the disclosures of both of which are incorporated herein by reference in their entirety.

[0002] FIELD OF THE DISCLOSURE The present disclosure relates generally to image processing operations. More particularly, certain embodiments of the present disclosure relate to video codecs. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest gray (black) to brightest white (highlight). In this sense, DR relates to scene-referred intensities. DR may also relate to the ability of a display device to reasonably or approximately render an intensity range having a certain width. In this sense, DR relates to display-referred intensities. Anywhere in this specification, the term may be used in either sense (e.g., interchangeably) unless a particular meaning is specifically and explicitly specified.

[0004] In this specification, the term high dynamic range (HDR) refers to a DR width that spans about 14-15 orders of magnitude in the human visual system (HVS). In practice, the DR that humans can simultaneously perceive a wide range of intensities may be somewhat truncated for HDR. In this specification, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR), individually or indistinguishably, refer to the DR that the human visual system (HVS) (including eye movements and allowing some degree of photopic change across a scene or image) can perceive in a scene or image. In this specification, EDR may refer to a DR that spans 5-6 orders of magnitude. Although somewhat narrower for true scene-referred HDR, EDR represents a wide DR width and may also be referred to as HDR.

[0005] In practice, an image has one or more color components of a color space (e.g. luma Y and chroma Cb and Cr), each color component being represented with n bits of precision per pixel (e.g. n = 8). When using non-linear luminance coding (e.g. gamma coding), images with n < 8 (e.g. color 24-bit JPEG images) may be considered standard dynamic range, and images with n > 8 may be considered enhanced dynamic range.

[0006] The reference electro-optical transfer function (EOTF) for a display characterizes the relationship from the color values ​​(e.g., luminance) of the input video signal to the output screen luminance (e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT. 1886, “Reference electro-optical transfer function for flat panel displays used in HDTV studio production” (March 2011), the disclosure of which is incorporated herein in its entirety, defines the reference EOTF for flat panel displays. Given a video stream, information about its EOTF is embedded in the bitstream as (image) metadata. In this specification, the term “metadata” refers to any auxiliary information that is transmitted as part of the encoded bitstream and helps a decoder to render the decoded image. Such metadata includes, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein.

[0007] In this specification, the term "PQ" refers to the quantization of perceptual luminance amplitude. The human visual system responds to increasing light levels in a highly nonlinear manner. A human's ability to see a stimulus is affected by the luminance of the stimulus, the size of the stimulus, the spatial frequencies that compose the stimulus, and the luminance level to which the eye has adapted up to the moment the stimulus is viewed. In some embodiments, the perceptual quantizer function maps linear input gray levels to output gray levels that are more closely matched to the contrast sensitivity thresholds in the human visual system. An example of a PQ mapping function is described in SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays" (hereinafter "SMPTE"), the disclosure of which is incorporated herein in its entirety. Here, for a fixed stimulus size, for each luminance level (e.g., stimulus level), the minimum visible contrast step at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model). Summary of the Invention [Problem to be solved by the invention]

[0008] 200~1000cd / m 2A display supporting a luminance of 1000 nits or more is an example of a lower dynamic range (LDR) (also called standard dynamic range (SDR)) as opposed to EDR (or EDR). EDR content may be displayed on an EDR display supporting a higher dynamic range (e.g., 1000 nits to 5000 nits or more). Such displays may be defined using an alternative EOTF supporting high luminance capabilities (e.g., 0 to 10,000 nits or more). An example of such an EOTF is defined in SMPTE ST 2084:2014 and Rec. ITU-R BT.2100, "Image parameter values ​​for high dynamic range television for use in production and international programme exchange" (06 / 2027). As appreciated by the present inventors, improved techniques for generating video content having a high dynamic range and little or no visual artifacts are desired.

[0009] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their description in this section. Similarly, unless otherwise indicated, it should not be assumed based on this section that a problem identified with one or more approaches has been recognized in any prior art. [Brief description of the drawings]

[0010] Certain embodiments of the present invention are illustrated, without limitation, in the figures of the accompanying drawings, in which like elements are given like references and in which:

[0011] [Figure 1] FIG. 1 shows an example process for a video delivery pipeline.

[0012] [Figure 2A] FIG. 2A illustrates an example flow for applying denoising for local reconstruction. [Figure 2B] FIG. 2B illustrates an example flow for applying denoising for local reconstruction. [Figure 2C] FIG. 2C illustrates an example flow for applying denoising for local reconstruction. [Figure 2D] FIG. 2D illustrates an example flow for applying denoising for local reconstruction. [Figure 2E] FIG. 2E illustrates an example flow for applying denoising for local reconstruction. [Figure 2F] 1 illustrates an example operation for collecting co-occurrence statistics. [Figure 2G] FIG. 2G illustrates an example clipping and quantization operation. [Figure 2H] FIG. 2H illustrates an example downsampling operation. [Figure 2I] FIG. 2I illustrates an example process flow for obtaining a binary cluster map. [Figure 2J] FIG. 2J illustrates an example denoising reconstruction function index map with co-occurrence statistics. [Figure 2K] FIG. 2K illustrates an example process flow for generating a weight mask.

[0013] [Figure 3A] FIG. 3A illustrates an example co-occurrence matrix or array of co-occurrence statistics. [Figure 3B] FIG. 3B illustrates an example of clipping and normalization of a standard deviation map.

[0014] [Figure 4] FIG. 4 illustrates an example process flow.

[0015] [Diagram 5]FIG. 5 illustrates a simplified block diagram of an example hardware platform upon which a computer or computing device described herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In the following description, for the purposes of convenience, numerous details are set forth in order to provide a thorough understanding of the present invention, although it will be apparent that the present invention may be practiced without these details. In other instances, detailed details of well-known structures and devices are not set forth in order to unnecessarily obscure, obscure or obfuscate the description of the present invention.

[0017] [overview] The denoising techniques described herein can be implemented in conjunction with reconstruction techniques that (inversely) reconstruct or upconvert image data of a relatively narrow dynamic range, such as SDR image data, into image data of a higher dynamic range, such as HDR image data with enhanced local contrast and saturation. These techniques can help to improve image quality and prevent or reduce visual artifacts (induced by noise in SDR image data) in the resulting HDR image data.

[0018] As used herein, "up-conversion" or "(inverse) reconstruction" refers to converting lower dynamic range image data to higher dynamic range image data via a reconstruction operation, such as a local or global reconstruction operation. Local reconstruction refers to an up-conversion or inverse reconstruction operation that applies different reconstruction functions or mappings to different pixels of an input image according to a local reconstruction function index map. Thus, in a local reconstruction, a first reconstruction function specified in the local reconstruction function index map and applied to a first pixel of the input image may be a different function than a second reconstruction function specified in the same local reconstruction function index map and applied to a second, different pixel of the input image.

[0019] The local reconstruction function index map includes index values ​​(e.g., a 2-dimensional array, etc.) that are used to identify a particular reconstruction function or mapping. The index values ​​may include, but are not necessarily limited to, (local) L1-mid values ​​calculated from spatial filtering (edge-preserving, boundary-preserving, etc.) performed on the input SDR image.

[0020] Exemplary local reconstruction operations and local reconstruction function index maps (e.g., including L1-mid values, etc.) are described in PCT Application No. PCT / US2021 / 053241, filed October 1, 2021, the entire contents of which are incorporated herein by reference as if fully set forth herein. Exemplary global operations are described in U.S. Provisional Patent Application No. 62 / 136,402, filed March 20, 2015 (also U.S. Patent Application Publication No. 2018 / 0020224, published January 18, 2018) and PCT Application No. PCT / US2019 / 031620, filed May 9, 2019, the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0021] Reconstruction operations, such as local reconstruction operations, may be used in SDR-HDR upconversion to enhance visual quality in image content in terms of dynamic range (from darkest to brightest in an image portion to the entire image), local or global contrast, local or global saturation, etc.

[0022] However, noises such as film grain noise, camera noise, etc. may be present in the original or input SDR image. In addition, noises may also be introduced from image processing / encoding operations such as video compression. These noises may be introduced into the HDR image generated from the step of reconstructing the input SDR image and may be further accentuated or magnified in the HDR image, resulting in visually noticeable visual artifacts. Such visual artifacts may be particularly problematic on large-screen image displays because the human visual system (HVS) can resolve more and finer details on large-screen image displays compared to small-screen image displays such as mobile devices.

[0023] The techniques described herein can be implemented to denoise a reconstruction function index map using co-occurrence filtering. The resulting denoised reconstruction function index map can then be used to perform reconstruction operations, such as local reconstruction operations, on input image content, such as SDR image content, to generate relatively high quality image content, such as HDR image content, that is relatively free of noise-causing visual artifacts. The HDR image content can be rendered on a variety of HDR image displays, including but not limited to large screen image displays.

[0024] Exemplary embodiments described herein relate to generating an image in a first dynamic range from an input image in a second dynamic range lower than the first dynamic range. A first array of co-occurrence statistics associated with the input image in the second dynamic range is generated. A second array of reconstruction function index values ​​for selecting a reconstruction function from among a plurality of candidate reconstruction functions is generated. A denoising operation is performed based at least in part on the first array of co-occurrence statistics while a reconstruction operation is performed based at least in part on the second array of reconstruction function index values ​​to generate a reconstructed image in the first dynamic range.

[0025] [Video supply processing pipeline example] FIG. 1 illustrates an example process of a video delivery pipeline (100) showing various stages from video capture / generation to HDR or SDR display. Examples of HDR displays may include, but are not limited to, image displays operating with TVs, mobile devices, home theaters, etc. Examples of SDR displays may include, but are not limited to, SDR TVs, mobile devices, home theater displays, head-mounted display devices, wearable display devices, etc. It should be noted that reconstruction denoising can occur either at the encoder / server side (before video compression) or at the decoder / playback side (after video decompression). Different system configurations than those illustrated in FIG. 1 may be used to support playback-side reconstruction denoising. Different image metadata formats than those used by the processing components illustrated in FIG. 1 may be used to transmit image metadata in these different system configurations. For illustrative purposes only, FIG. 1 is used to illustrate or illustrate a reconstruction denoising process that occurs at the server side to generate a denoised reconstruction function index map, such as a denoised local reconstruction function index map.

[0026] Video frames such as a sequence of consecutive input SDR images 102 may be received by the SDR-HDR upconverter 115 or the denoising block 105 therein. These SDR images (102) may be received from a video source or captured from a video data store. Some or all of the SDR images (102) may be generated from source images, for example, via video editing or conversion operations (by automatic means without human input, manual, automatic means with human input, etc.), color grading operations, etc. The source images may be digitally captured (e.g., by a digital camera, etc.), analog camera images captured on film converted to a digital format, or computer generated (e.g., using computer animation, image rendering, etc.). The SDR images (102) may be images related to one or more of a movie release, archived media programming, a media programming library, video recordings / clips, media programming, television programs, user-generated video content, etc. Additionally, optionally, or alternatively, in some operating scenarios, the SDR image (102) may be a reconstructed SDR image generated by forward reconstruction of a source image, such as an original pre-reconstructed HDR image or an original pre-constructed SDR image.

[0027] The denoising block (105) performs a denoising operation on each SDR image in the sequence of consecutive SDR images (102). For example, the denoising block (105) may generate a denoised local reconstruction function index map that is used to apply a denoised local reconstruction operation to each SDR image in the sequence of consecutive SDR images (102) to generate a respective (reconstructed) HDR image of a sequence of corresponding consecutive (reconstructed) HDR images that portrays the same visual semantic content as the SDR image (102) but has a higher dynamic range, more vibrant colours, less noise, less visible visual artifacts, etc., compared to the SDR image (102).

[0028] Some or all of the SDR image (102) and / or the corresponding denoised local reconstruction function index map can be used by the SDR-HDR up-conversion block (115) to generate an inverse reconstruction mapping (e.g., a BLUT, an inverse reconstruction function / curve or polynomial set, MMR coefficients, etc.) that, when applied to the input SDR image (102), generates a corresponding HDR image 107.

[0029] The HDR image (107) may be encoded into a video signal 122 (e.g., an encoded bitstream, etc.) or a set of consecutive video segments by an HDR video encoder 120. Given the video signal (122), a receiving device, such as a mobile phone, may decide, as part of internal or post-processing on the device, to use the metadata along with the SDR image data to generate and render an image with higher dynamic range, such as HDR, more vivid colors, less noise, fewer visual artifacts, etc., within the display capabilities of the receiving device.

[0030] Examples of video signals or video segments may include, but are not necessarily limited to, single-layer video signals / segments, etc. In some embodiments, the HDR video encoder (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other distribution formats, to generate the video signal (122) or video segments.

[0031] The video signal (122) or video segments are then provided downstream to receivers such as mobile devices, tablet computers, decoding and playback devices, media source devices, media streaming client devices, television sets (e.g., smart TVs, big screen TVs, wall-mounted TVs, etc.), set-top boxes, movie theaters, etc.

[0032] In some operation scenarios, once an HDR image is generated via denoising local reconstruction, a video encoder can encode an HDR image in the video signal (e.g., a base layer, etc.) or a version of an HDR image derived from the locally reconstructed HDR image, instead of encoding an SDR image in the video signal (e.g., a base layer, etc.). An HDR image decoded from such a video signal may be directly viewable on an HDR display.

[0033] [Noise removal for local reconstruction] 2A illustrates a first example flow for applying a local reconstruction operation (e.g., 212, etc.) using a denoised reconstruction function index map to generate a corresponding HDR image from an input SDR image. An image processing system, a processing block therein (e.g., 105, etc. in FIG. 1), may implement or perform this process flow. In some operation scenarios, given an input SDR image 202, a global reconstruction function 204 is selected to generate a global reconstructed HDR image. The global reconstructed HDR image is used to determine or represent a main look of HDR for an output HDR image 214 generated by the local reconstruction operation (212) from the input SDR image (202).

[0034] In some operation scenarios, the (inverse) reconstruction function used to generate an HDR image from an SDR image may be specified by a reconstruction function index value. In a non-limiting example, the reconstruction function index value may be specified using an L1-mid value. Examples of reconstruction function index values ​​such as the L1-mid value are described in the aforementioned U.S. Provisional Patent Application No. 63 / 086,699.

[0035] The purpose of applying the local reconstruction operation in combination with a denoising operation down to the pixel-by-pixel level to generate an HDR image (214) from an input SDR image (202) is to enhance the local contrast ratio in a local region of the HDR image (214) without changing the local brightness level of that local region of the HDR image (214) and without having visible artifacts resulting from noise present in the input SDR image (202), while maintaining the same main HDR appearance of the global reconstructed HDR image generated from the global reconstruction function (204) in the HDR image (214) generated from the local reconstruction operation combined with the denoising operation.

[0036] To help prevent or reduce common artifacts such as halo artifacts that may be generated from varying local brightness levels adjacent to edges / boundaries of visual objects / characters, high-precision filtering, such as multi-level edge-preserving filtering 206, may be applied to the input SDR image (202) resulting in a filtered image. The filtered image may be used to derive or estimate local brightness levels (or local region-specific brightness levels) in different local regions (up to pixel-by-pixel accuracy or up to local regions around each pixel) of the input SDR image (202). The estimated local brightness levels in different local regions (up to pixel-by-pixel accuracy or up to local regions around each pixel) of the input SDR image (202) may be used to estimate or predict local reconstruction function index values, such as local L1-mid values ​​(up to pixel-by-pixel accuracy or up to local regions around each pixel) in the HDR image (214). Examples of predictions of reconstruction function index values, such as L1-mid values, are described in the aforementioned U.S. Provisional Patent Application No. 63 / 086,699.

[0037] To improve the efficiency or response time of the local reconstruction operation, a family of local reconstruction functions 208 may be loaded or constructed (e.g., initially, in advance, etc.) in the image processing system described herein during system startup. The loading or construction of the family of local reconstruction functions (208) may be performed before applying the local reconstruction operation combined with the denoising operation to each input image (e.g., 202, etc.) of a sequence of consecutive input SDR images into a corresponding HDR image (e.g., 214, etc.) of a sequence of corresponding consecutive output HDR images based at least in part on the family of local reconstruction functions (208). The family of local reconstruction functions (208) may be generated, for example, but not limited to, via extrapolation and / or interpolation from a plurality of basic reconstruction functions used to select the global reconstruction function (204). Each local reconstruction function of the family of local reconstruction functions (208) may be indexed or identified, in whole or in part, using a respective reconstruction function index value, such as the L1-mid value, as is the case for the global reconstruction function (204) or each base reconstruction function of the plurality of base reconstruction functions from which the global reconstruction function (204) is selected.

[0038] As described above, a filtered image generated by applying multi-level edge-preserving filtering (206) to an input SDR image (202) may be used to generate or estimate local brightness levels. The local brightness levels may be used in combination with or with reference to a global reconstruction function in a predictive model to generate or predict local reconstruction function index values. These local reconstruction function index values ​​form the indices of the local reconstruction function index map 210 (before being denoised).

[0039] In the techniques described herein, instead of using the (pre-denoised) local reconstruction function index map (210) to refer to a pixel-specific local reconstruction function in a family of local reconstruction functions (208), the index map (210) is first denoised by a denoising block (denoted the “proposed denoising module”) 252 to generate a denoised local reconstruction function index map 254.

[0040] The pixel-specific local reconstruction functions referenced by indexes (or values ​​thereof) in the denoised local reconstruction function index map (254) may be applied on a pixel-by-pixel basis to the input SDR image (202) by a local reconstruction operation (212) to inversely reconstruct the SDR codewords in the luma and chroma channels into reconstructed HDR codewords in the luma and chroma channels of the HDR image (214). Each of these pixel-specific local reconstruction functions in the form of a fine-tuned (to pixel-by-pixel accuracy) nonlinear function may be used to enhance local contrast and / or saturation in the HDR image (214).

[0041] The denoising block (252) avoids increasing noise coming from the input SDR image (202) by reducing or removing noise in the reconstruction function index map (210) from the denoised local reconstruction function index map (254).

[0042] As illustrated in Fig. 2A, the denoising block (252) includes a first processing block 256 for collecting co-occurrence statistics. The first processing block (256) receives as input the input SDR image (202) and the (undenoised) reconstruction function index map (210), and collects co-occurrence statistics of pixels from the input SDR image (202) and co-occurrence statistics of pixels in a binary cluster map representing background and texture areas of the input SDR image (202). The terms binary cluster map and binary cluster mask are both used synonymously to refer to what is commonly referred to as a binary cluster map.

[0043] The denoising block (252) includes a second processing block (258) for denoising the reconstruction function index map. The second processing block (258) receives as input the two co-occurrence statistics from the first processing block (256) and uses a combination of the two co-occurrence statistics to smooth the (pre-denoised) reconstruction function index map (210) into a denoised reconstruction function index map (254). The denoised reconstruction function index map (254) may then be used in the local reconstruction operation (212) to perform SDR-HDR local reconstruction and generate the HDR image (214). The smoothing performed by the second processing block (258) of the denoising block (252) serves to reduce noise in the HDR image (214). Additionally, optionally, or alternatively, a weight mask may be derived from the standard deviation map and applied in the smoothing operation to avoid unintentional reduction in the emphasis of non-noise image content in the HDR image (214) that may result from over-smoothing of non-noise image content in the HDR image (214).

[0044] FIG. 2B illustrates a second example flow for applying a local reconstruction operation (e.g., 212, etc.) to generate a corresponding HDR image from a denoised SDR image derived from an input SDR image. An image processing system, a processing block therein (e.g., 105, etc. in FIG. 1 ), may implement or perform this process flow. In some operation scenarios, given an input SDR image 202, a denoising block 260 performs a denoising operation on the input SDR image (202) to derive a denoised SDR image 262. A global reconstruction function 204 is selected to use the denoised SDR image (262) as an input to generate a global reconstructed HDR image from the step of performing global reconstruction on the denoised SDR image (262). The global reconstructed HDR image is used to determine or represent the main HDR appearance of the output HDR image 214 to be generated by the local reconstruction operation (212) from the input SDR image (202).

[0045] To help prevent or reduce common artifacts such as halo artifacts that may be generated from varying local brightness levels adjacent to edges / boundaries of visual objects / characters, high-precision filtering, such as multi-level edge-preserving filtering 206, may be applied to the denoised SDR image (262) resulting in a filtered image. The filtered image may be used to derive or estimate local brightness levels (or local region-specific brightness levels) in different local regions (up to pixel-by-pixel accuracy or up to local regions around each pixel) of the input SDR image (202). The estimated local brightness levels in different local regions (up to pixel-by-pixel accuracy or up to local regions around each pixel) of the denoised SDR image (262) may be used to estimate or predict local reconstruction function index values, such as local L1-mid values ​​(up to pixel-by-pixel accuracy or up to local regions around each pixel) in the HDR image (214). These local reconstruction function index values ​​form the indices of the local reconstruction function index map 210.

[0046] The pixel-specific local reconstruction functions referenced by the index (or its value) in the local reconstruction function index map (210) may be applied on a pixel-by-pixel basis to the input SDR image (202) by a local reconstruction operation (212) to inversely reconstruct the input SDR codeword in the luma and chroma channels into a reconstructed HDR codeword of the HDR image (214) in the luma and chroma channels. Each of these pixel-specific local reconstruction functions in the form of a fine-tuned (to pixel-by-pixel accuracy) nonlinear function may be used to enhance local contrast and / or saturation in the HDR image (214).

[0047] The denoising block (260) avoids increasing noise originating from the input SDR image (202) by reducing or removing noise in the input SDR image (202) from the denoised SDR image (262). Additionally, optionally or alternatively, a mask may be designed to limit the reduction or denoising in the input SDR image (202) only in image content affected by noise or in spatial domains (such as highlight or bright regions) to avoid smoothing of non-noised image content.

[0048] Additionally, optionally, or alternatively, instead of applying a local reconstruction operation (212) to an input SDR image (202) as shown in FIG. 2B, in a third example flow for applying a local reconstruction operation (e.g., 212) with a denoising operation, as illustrated in FIG. 2C, a pixel-specific local reconstruction function referenced by an index (or its value) in the local reconstruction function index map (210) may be applied to the SDR image (262) denoised on a pixel-by-pixel basis by the local reconstruction operation (212) to inversely reconstruct the denoised SDR codeword in the luma and chroma channels of the denoised SDR image (262) into a reconstructed HDR codeword of the HDR image (214) in the luma and chroma channels.

[0049] FIG. 2D illustrates an example of operations performed using the denoising block (260) of FIG. 2B or FIG. 2C. As shown, the denoising block (260) includes a first processing block 266 for collecting co-occurrence statistics. The first processing block (266) receives the input SDR image (202) as input and collects pixel co-occurrence statistics from the input SDR image (202). The denoising block (260) includes a second processing block 268 for denoising the input SDR image. The second processing block (268) receives the co-occurrence statistics from the first processing block (266) as input and uses the co-occurrence statistics to smooth the input SDR image (202) into a denoised SDR image (262). The denoised SDR image (262) may then be used in subsequent processing, such as SDR-HDR local reconstruction, to generate the HDR image (214). The smoothing performed by the second processing block (268) of the noise removal block (260) serves to reduce noise in the HDR image (214). Additionally, optionally, or alternatively, a weight mask may be obtained from the standard deviation map and applied in the smoothing operation to avoid unintentional reduction in the emphasis of non-noise image content in the HDR image (214), which may result from over-smoothing non-noise image content in the HDR image (214).

[0050] 2E illustrates a fourth example flow for applying a local reconstruction operation (such as 212) in combination with a denoising operation to generate a corresponding HDR image from an input SDR image. In addition to the processing blocks illustrated in FIG. 2A for the first example flow, in this fourth example flow, a denoising block (260) is added to perform a denoising operation on an input SDR image (262) to generate a denoised SDR image. Further, instead of applying the local reconstruction operation (212) to the input SDR image (202) as shown in Fig. 2A, in this fourth example flow illustrated in Fig. 2E, a pixel-specific local reconstruction function referenced by an index (or its value) in the local reconstruction function index map (210) may be applied to the denoised SDR image (262) on a pixel-by-pixel basis by the local reconstruction operation (212) to inversely reconstruct the denoised SDR codeword in the luma and chroma channels of the denoised SDR image (262) into a reconstructed HDR codeword in the luma and chroma channels of the HDR image (214). In the fourth example, both the denoising of the input SDR image (202) and the denoising of the (pre-denoised) reconstruction function map index (210) may be used to achieve a relatively strong denoising.

[0051] [Collecting co-occurrence statistics] In some operation scenarios, the image smoothing operation can be performed using a co-occurrence filter or co-occurrence filtering (CoF). Image smoothing using CoF involves collecting co-occurrence statistics and smoothing the image using CoF. An example of denoising using CoF can be found in Jevnisek, Roy J., and Shai, Avidan, “Co-occurrence filter,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.3184-3192 (2017), which is incorporated herein by reference in its entirety.

[0052] Given a single-channel image I, the co-occurrence statistics C(a,b) of pixel values ​​a and b can be obtained from:

number

number

[0053] The matrix elements of the co-occurrence matrix M may be used to store co-occurrence statistics of pixel pairs, each representing the same pair of pixel values ​​a and b, as follows:

number

number

[0054] Using the co-occurrence matrix M, an input image I of size H×W can be smoothed to obtain an (output, smoothed) image J as follows:

number

number

number

[0055] By way of example only, for a full high detail or HD (1920×1080) input image I, an exemplary standard deviation is σ s = 10, and an exemplary half-window size is k s =ceil(σ s )=10. For input images with the same content but different dimensions, the standard deviation may be set to be proportional to the image size.

[0056] Co-occurrence filtering can be compared to bilateral filtering in Table 1 below. In bilateral filtering,

number

number

[0057] An example procedure for calculating or collecting co-occurrence statistics using a sliding window is illustrated in Table 2 below. [Table 2]

[0058] An example procedure for smoothing an input image using CoF is illustrated below in Table 3. The boundaries of the input image are preserved relatively well in the smoothed image, and relatively frequent co-occurring pixel values ​​may be smoothed relatively strongly. [Table 3]

[0059] 2F illustrates an example operation for collecting co-occurrence statistics for the denoising operations described herein. As shown, a first instance of processing block 270 may receive an input SDR image 202 and collect / obtain co-occurrence statistics of pixels in the input SDR image (202). The co-occurrence statistics of pixels in the input SDR image (202) may be calculated as M 1 The co-occurrences may be stored or collected in a first co-occurrence matrix denoted as:

[0060] Additionally, optionally, or alternatively, the processing block 272 may receive the reconstruction function index map 210 and generate or obtain a binary cluster map from the reconstruction function index map (210). The binary cluster map partitions, divides, or divides the reconstruction function index map (210) into background and texture areas, co-occurrence statistics of pixels in the input SDR image (202). A second instance of the processing block (270) collects / obtains co-occurrence statistics of clusters in the binary cluster map. The co-occurrence statistics of clusters in the binary cluster map are calculated as M 2 The co-occurrences may be stored or collected in a second co-occurrence matrix denoted as:

[0061] In some operation scenarios, co-occurrence statistics of an input SDR image (e.g., 202 in FIG. 2A to FIG. 2C and FIG. 2E) may be collected and used to determine or control (e.g., filtering strength, etc.) whether and how the pixel values ​​or index values ​​of pixel positions in the corresponding reconstruction function index map should be smoothed. In comparison, the co-occurrence statistics of the reconstruction function index map may not be collected and may not be used to determine or control (e.g., filtering strength, etc.) whether and how the pixel values ​​or index values ​​of pixel positions in the corresponding reconstruction function index map should be smoothed. This is because similar pixel values ​​or index values ​​at pixel positions in the reconstruction function index map may correspond to very different pixel values ​​at the same pixel position of the input SDR image. Therefore, the co-occurrence statistics of pixel values ​​or index values ​​may be inaccurate to determine or control (e.g., filtering strength, etc.) whether and how the pixel values ​​or index values ​​of pixel positions in the reconstruction function index map should be smoothed.

[0062] For purposes of example only, the input (e.g., YCbCr, SDR, etc.) image may be an image with a bit depth of 8 in each channel of the color space in which the input image is represented. The total number of possible values ​​in a codeword space with a bit depth of 8 in each channel of a three channel color space, such as a color space (e.g., YCbCr, RGB, etc.), is 256. 3 ×256 3 Therefore, the size or dimensions of the co-occurrence matrix M for all pixel value pairs is 256. 3 ×256 3 This size may be too large to be practically usable due to its relatively high memory consumption and computational cost, and for a particular pixel value pair (a,b), there may not be a sufficient number of pixels to obtain or collect co-occurrence statistics for (a,b) with a relatively high degree of confidence.

[0063] In some operational scenarios, an 8-bit per channel input image may be quantized to help ensure relatively low memory usage, relatively high computational efficiency, and a relatively sufficient number of pixels for each pixel value pair. The quantized input image may then be used to collect co-occurrence statistics.

[0064] A possible way to quantize the input image could be to apply K-means based clustering among the pixel values ​​in the input image, however this method can still be time consuming.

[0065] In some operational scenarios, a more computationally effective or efficient quantization method than K-means may be applied or implemented. As illustrated in FIG. 2G, a single channel image S includes the luma or Y channel codeword of an input image, denoted as S. Y may be clipped or quantized. For example, a single channel image S Y The 5th and 95th percentiles (denoted as a and b) of the luma or Y channel codewords of a single channel image S YAs used herein, the term "letterbox" may refer to one or more of a padded dark, solid color, colored, or white area of ​​any shape, such as a rectangle, letterbox, pillarbox, oval, or other shape, or an inactive (static) region of any shape, such as a letterbox, pillarbox, oval, or other shape. Y The pixel values ​​(or luma / Y codewords) of are clipped using these percentiles a and b as lower and upper bounds, as follows: Yc We can obtain a clipped single channel image, denoted as:

number

[0066] The clipped single channel image S Yc is further quantized to nBins=8 uniform intervals as follows: Q It is possible to generate a quantized clipped single channel image denoted as

number

[0067] Clipped single channel image S Yc Quantize the clipped single channel image S from Q After obtaining S QCo-occurrence statistics may be collected from. In some operation scenarios, a region of interest map P (e.g., complementing a letterbox probability map, surrounding area probability map, etc.) may be used to filter out the influence of pixels in any letterbox areas that may be present in the input image. Examples of letterbox probability maps or surrounding area probability maps can be found in U.S. Provisional Patent Application Serial No. 63 / 209,602, filed June 11, 2021, by TW Huang and GM Su, the entire contents of which are incorporated herein by reference as if fully set forth herein. Thus, Equation (1) and Equation (2) may be modified or adjusted using the region of interest map P as follows:

number

[0068] Since the quantization level corresponding to nBins is 8, the co-occurrence matrix M 1 has a size of 8 x 8. As above, for a full HD (1920 x 1080) input image, σ may be selected or set to be, for example, a value of 10, and the half-window size k may be selected or set to be, for example, a value of 10. For different input image sizes, σ and k may be selected or set to be proportional to the image size or to scale with the image size.

[0069] In some operation scenarios, the quantized image S Q can be downsampled to accelerate the collection of co-occurrence statistics. For example, a quantized image S of size H × W Q is downsampled by selecting one pixel out of every D pixels (or downsampling factor) in each row and each column, resulting in a pixel of size

number

number

number

number

[0070] Fig. 3A shows the co-occurrence matrix M of size 8 × 8 calculated from the quantized downsampled single-channel image generated from the input image. 1 As shown, elements along or near the diagonal of the matrix are relatively large values ​​indicating relatively strong co-occurrence between similar pixel values, while elements off or away from the diagonal of the matrix are relatively small values ​​indicating relatively weak co-occurrence between pixel values ​​with relatively large differences.

[0071] In some operation scenarios, co-occurrence statistics may be collected from clusters identified in the reconstruction function index map to prevent or reduce smoothing in textured areas of the input image, or to help limit or restrict smoothing in background areas of the input image, even if the co-occurrence statistics in the textured areas in the reconstruction function index map may be relatively large. The reconstruction function index map may be partitioned or divided into a binary map K having two clusters, one for background areas and one for textured areas. The binary map K may be used to help limit or restrict the smoothing operation to be performed on pixels in the background areas, provided that the co-occurrence statistics from these background areas of the input image are relatively large. On the other hand, the binary map K may be used to help avoid smoothing with pixels in textured areas, even if the co-occurrence statistics from these textured areas of the input image happen to be relatively large. Additionally, optionally, or alternatively, a weight mask may be further used to avoid smoothing in textured areas of the input image, as will be described in more detail below.

[0072] Because the standard deviation of pixel values ​​in background areas is relatively small compared to the standard deviation of pixel values ​​in texture areas, a local standard deviation, such as from weighted standard deviation filtering, can be used to determine whether a pixel belongs to a background area or a texture area.

[0073] A standard deviation map (or std map) can be obtained by applying a standard deviation filter to an input image. Given a pixel location, the pixel values ​​in a pixel patch or neighborhood around that pixel are given by: 1 ,…,x N}, where x i is the value of the i-th pixel. The variance Var(X) of these pixel values ​​of patch X can be obtained as follows:

number

[0074] However, if a letterbox exists in the input image, the (local) variance at the boundary of the letterbox existing in the input image may be relatively large. Weighted normal filtering may be performed to obtain the local variance and / or the local standard deviation at different pixel positions of the input image. For example, the local variance Var'(X) of the pixel values ​​of patch X can be obtained as follows instead of the above formula (10):

number

[0075] A small (spatial filtering) kernel size for weighted standard deviation filtering may result in a variance or standard deviation map with finer details such as edges, whereas a large kernel size may result in a blurrier variance or standard deviation map.

[0076] FIG. 2I illustrates an example process flow for obtaining a binary cluster map K from a standard deviation map derived from a reconstruction function index map. 1 = 3 and k 2 Apply weighted standard deviation filtering to the reconstruction function index map with two different kernel sizes, e.g., std_map 1 and std_map 2 Using the weighted standard deviation filtering with a smaller kernel size, we can derive two standard deviation maps, denoted as std_map1 The standard deviation map std_map can detect or store edges in the reconstruction function index map. 1 Furthermore, σ b The blurred std_map is created by applying a Gaussian filter with standard deviation (having values ​​of 3, 4, 5, etc.) 1 You can generate a blurred std_map like this: 1 and (not blurred) std_map 2 Both, t 1 (having values ​​of 18, 20, 22, etc.), t 2 The two binary maps can be compared (or thresholded) separately with a threshold having values ​​(such as 65, 70, 75, 80, etc.) to obtain two binary maps.

number

[0077] Binary map img 1 andimg 2 can be combined or multiplied to obtain the binary cluster map K as follows:

number

[0078] The pixel position is the binary map img 1 andimg 2 It may be determined to be within a background area if it is within the background area in both the first and second pixel regions. The parameters used for the threshold, kernel, standard deviation, etc. for blurring may be empirically selected to be different for video frames of different spatial resolutions.

[0079] The second co-occurrence matrix can be used together with the region of interest map P (or the corresponding surrounding area probability map) to generate and store co-occurrence statistics for the binary cluster map K as follows:

number

[0080] In some operation scenarios, the binary cluster map K is downsampled to generate a downsampled binary cluster map from the downsampling step, and the downsampled binary cluster map is used to calculate the co-occurrence statistics into a second co-occurrence matrix M 2 can be collected.

[0081] An example of co-occurrence statistics from the binary cluster map K is illustrated in Table 5 below. As shown, the co-occurrence statistic C(1,1) for the background area is relatively large and can help smooth the pixel values ​​in the background area. A second co-occurrence matrix M of size 2×2 2 is a (first) co-occurrence matrix M 1 can be combined with [Table 5]

[0082] [Co-occurrence filtering and noise removal] 2J illustrates an example denoised reconstruction function index map with co-occurrence statistics. A weight mask W is generated or obtained from the (pre-denoised) reconstruction function index map, denoted as L, to help confine the denoising to background areas and to help preserve the highlighting from the local reconstruction step in textured areas. The reconstruction function index map L is calculated based on the pre-computed co-occurrence matrix M 1 and M. 2 It is smoothed using L d We generate an intermediate denoised reconstruction function index map, denoted as L f The final denoised reconstruction function index map, denoted as L, is calculated by multiplying the (pre-denoised) reconstruction function index map L and the intermediate denoised reconstruction function index map L by the weighting factors (or weighting coefficients) specified in the weight mask.d It may be generated as a weighted sum of

number

[0083] FIG. 2K illustrates an example process flow for generating a weight mask W. The second co-occurrence matrix M is generated from the binary cluster map K of the reconstructed function index map L. 2 The co-occurrence statistics collected within may have relatively large co-occurrence statistics, and if used without the weight mask W, the pixels or pixel values ​​in the textured areas may be smoothed like the pixels or pixel values ​​in the background areas, thereby causing undesirable blurring of image details rendered in the textured areas and reducing / negating the desired enhancement of image details by the local reconstruction operation.

[0084] The weight mask W may be used to preserve image detail or a desired enhancement thereof in textured areas. A variance (e.g., weighted) or standard deviation map of the reconstruction function index map L may be generated using standard deviation (e.g., weighted) filtering and used to obtain the weight mask W. Since textured areas have a relatively large spatial frequency visual content, the variance or standard deviation map has relatively large values ​​in textured areas.

[0085] In the non-limiting example illustrated in FIG. 2K and FIG. 2I, weighted standard deviation filtering is performed using k 2 = 51, apply it to the index map to obtain the standard deviation map std_map 2 The standard deviation map std_map can be derived. 2 is σ 0 The blurred standard deviation map may be blurred using a Gaussian filter with a standard deviation such as ∇ = 5. As illustrated in FIG. 3B, the blurred standard deviation map may be clipped to lower and upper limits [l,h]. The clipped values ​​in the lower and upper limits [l,h] may then be normalized to a normalized value range such as [0,1] to obtain the (final) weight mask W. An example of a reconstruction function index value range may be part or all of the value range [0,4095]. The lower and upper limits [l,h] are empirically selected and may be any of [60,200], [50,210], [40,220], etc. The clipping and normalization operation applies a function f to each pixel in the blurred standard deviation map as follows: cn This may be equivalent to applying the following:

number

[0086] An example procedure for generating a weight mask W using clipping and normalization operations on a blurred standard deviation map is illustrated in Table 6 below. [Table 6]

[0087] For pixel locations that may be in texture areas, the blur standard deviation at the pixel locations may be relatively large. Correspondingly, the weighting factor for the pixel locations in the weighting mask W may be relatively small, and the final denoised reconstruction function index map L f The contribution to the final reconstruction function index value for that pixel location in is the intermediate denoised reconstruction function index map L d from the original (pre-denoising) reconstruction function index map L than from the local reconstruction operation identified in the original reconstruction function index map L. As a result, the enhancement of textured areas from the local reconstruction operations identified in the original reconstruction function index map can be better preserved and less affected by the smoothing operation than would otherwise be the case.

[0088] As shown in FIG. 2J, the intermediate denoised reconstruction function index map L d The (undenoised) reconstruction function index map L is converted into a co-occurrence matrix M 1 and M. 2 By way of example, and not by way of limitation, in one example, the co-occurrence matrix M 1 and M. 2 The average of the co-occurrence values ​​in can be taken as the final co-occurrence value.

number

[0089] Alternatively, instead of the average,

number

[0090] co-occurrence matrix M 1 and M. 2 The final combined co-occurrence values ​​from can be used to denoise the (pre-denoised) reconstruction function index map L as follows:

number

[0091] Standard deviation σ s A relatively large value for may make the filtered image relatively smoother. The noise removal results in background areas may be visually better, but texture areas may be more blurred.

[0092] An example procedure for denoising the reconstruction function index map is illustrated in Table 7 below. [Table 7]

[0093] The denoising of the reconstruction function index map can be accelerated using downsampling. For example, the D s One out of every 100 pixels (or downsampling factor) may be selected or used to perform calculations related to the denoising operation. In a non-limiting example, to achieve fast processing speeds while preserving the visual quality of the output image (e.g., 214 in FIGS. 2A-2C and 2E), a downsampling factor D s= 4. At pixel location p, the intermediate smoothed or (denoised) function index map L d The index value in may be calculated as follows:

number

[0094] An example procedure for denoising a reconstruction function index map using downsampling is illustrated in Table 8 below. [Table 8]

[0095] In some operation scenarios, linear segment-based structures may be used in denoising for local reconstruction. An example of a linear segment-based structure is described in U.S. Patent No. 10,397,576, "Reshaping curve optimization in HDR coding," by H. Kadu and GM. Su, the entire contents of which are incorporated herein as if fully set forth herein.

[0096] Some or all of the techniques described herein may be implemented and / or performed as part of real-time operations for broadcast video applications, real-time streaming applications, etc. Additionally, optionally, or alternatively, some or all of the techniques described herein may be implemented and / or performed as part of time-delayed or offline operations for non-real-time streaming applications, movie applications, etc.

[0097] [Processing flow example] 4 illustrates an example process flow for generating an image in a first dynamic range from an input image in a second dynamic range lower than the first dynamic range, according to an embodiment. In some embodiments, the process flow may be performed by one or more computing devices or components (e.g., an encoding device / module, a transcoding device / module, a decoding device / module, an inverse tone mapping device / module, a tone mapping device / module, a media device / module, an inverse mapping generation and application system, etc.). At block 402, the image processing system generates a first array of co-occurrence statistics for the input image in the second dynamic range.

[0098] In block 404, the image processing system generates a second array of reconstruction function index values ​​for selecting a reconstruction function from among a plurality of candidate reconstruction functions.

[0099] In block 406, the image processing system performs (a) a denoising operation based at least in part on the first array of co-occurrence statistics and (b) a reconstruction operation based at least in part on the second array of reconstruction function index values ​​to generate a reconstructed image in a first dynamic range.

[0100] In one embodiment, the second array of reconstruction function index values ​​is an array of denoised reconstruction function index values ​​generated by performing a denoising operation on a third array of pre-denoised reconstruction function index values.

[0101] In one embodiment, the first array of co-occurrence statistics is a combination of a fourth array of co-occurrence statistics collected from a binary cluster mask derived from a third array of reconstruction function index values ​​before denoising, and a fifth array of co-occurrence statistics collected from a clipped quantized image derived from an input image. The first array of co-occurrence statistics being a combination of the fourth array of co-occurrence statistics and the fifth array of co-occurrence statistics means that the first array of co-occurrence statistics includes a fourth array and a fifth array. The fourth array and the fifth array are not added together to form one array. In one example, in other words, for an input image of the second dynamic range, a reconstruction function index map before denoising (third array) is first calculated. Then, a binary cluster map is calculated using the reconstruction function map before denoising (third array), and then a co-occurrence statistics of the binary cluster map (fourth array) is calculated. On the other hand, the input image is clipped and quantized to calculate the co-occurrence statistics (fifth array). Then, both co-occurrence statistics (fourth array and fifth array) are combined to form a combined co-occurrence statistics (first array). Then, the combined co-occurrence statistics (first array) is used to denoise the undenoised reconstruction function index map (third array) to obtain a denoised co-occurrence statistics (second array). Then, the denoised co-occurrence statistics (second array) is used to generate a reconstructed image of the first dynamic range.

[0102] In one embodiment, the binary cluster mask is generated from applying standard deviation filtering to a third array of pre-denoised reconstruction function index values.

[0103] In one embodiment, standard deviation filtering is applied to the third array of pre-denoised reconstruction function index values ​​with at least two different spatial filtering kernel sizes.

[0104] In one embodiment, the fifth array of co-occurrence statistics is collected from a clipped quantized image derived from the input image after either the clipped quantized image or the input image have been downsampled.

[0105] In one embodiment, the fourth array of co-occurrence statistics is stored in a first co-occurrence matrix, where the fifth array of co-occurrence statistics is stored in a second co-occurrence matrix, and where the co-occurrence statistic at a pixel location in the first array represents one of the average or minimum of the first co-occurrence statistic at the pixel location in the first co-occurrence matrix and the second co-occurrence statistic at the pixel location in the second co-occurrence matrix.

[0106] In one embodiment, each reconstruction function index value at a pixel location represented in the second array of reconstruction function index values ​​is generated as a weighted sum of a first reconstruction function index value at the same pixel location represented in the array of intermediate denoised reconstruction function index values ​​and a second reconstruction function index value at the same pixel location represented in the array of pre-denoised reconstruction function index values.

[0107] In one embodiment, the weighted sum is determined using weight elements at the same pixel location in an array of weight masks, the array of weight mask values ​​being obtained from applying standard deviation filtering to the array of pre-denoised reconstruction function index values.

[0108] In one embodiment, the second array of reconstruction function index values ​​is generated based at least in part on denoised codewords in a denoised image of the second dynamic range, the denoised codewords in the denoised image of the second dynamic range being generated by performing a denoising operation on pre-denoised codewords in the input image of the second dynamic range.

[0109] In one embodiment, the reconstructed image in the first dynamic range is generated from performing a reconstruction operation on one of the input image or the denoised image.

[0110] In one embodiment, a first array of co-occurrence statistics is collected from pre-denoised codewords in an input image.

[0111] In one embodiment, the first dynamic range represents a high dynamic range and the second dynamic range represents a standard dynamic range that is lower than the high dynamic range.

[0112] In some embodiments, a computing device, such as a display device, a mobile device, a set-top box, a multimedia device, etc., is configured to perform any of the above methods. In some embodiments, an apparatus includes a processor and is configured to perform any of the above methods. In some embodiments, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause any of the above methods to be performed.

[0113] In one embodiment, a computing device includes one or more processors and one or more storage media containing a set of instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods described above.

[0114] It should be noted that although separate embodiments are described herein, any combination of the embodiments and / or sub-embodiments described herein may form further embodiments.

[0115] [Example of computer system implementation] Embodiments of the present invention include computer systems, systems made up of electronic circuits and components, microcontrollers, field programmable gate arrays (FPGs), and A) or other configurable or programmable logic device (PLD), discrete time or digital signal processor (DSP), application specific IC (ASIC), or other integrated circuit (IC) device, and / or an apparatus including one or more such systems, devices, or components. The computer and / or IC may perform, control, or execute instructions for the adaptive perceptual quantization of images having enhanced dynamic range described herein. The computer and / or IC may compute any of the various parameters or values ​​for the adaptive perceptual quantization process described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0116] Certain aspects of the present invention include a computer processor that executes software instructions to cause the processor to perform the methods of the present disclosure. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc., may implement the above-described method for adaptive perceptual quantization of HDR images by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention may be provided in a program product form. The program product may include any non-transitory medium that stores a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the method of an embodiment of the present invention. A program product according to an embodiment of the present invention may take various forms. For example, the program product may include a physical medium, such as a magnetic data storage medium, including a floppy disk, a hard disk drive, an optical data storage medium, including a CD ROM, a DVD, an electronic data storage medium, including a ROM, a flash RAM, etc. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0117] When a component (e.g., a software module, a processor, an assembly, a device, a circuit, etc.) is referred to above, any reference to that component (including any reference to a "means") should be construed to include, unless expressly stated otherwise, all components that perform the function of the component (e.g., are functionally equivalent) as equivalents of the component (including components that are not structurally equivalent to the disclosed structures that perform the functions appearing in the example embodiments of the invention described above).

[0118] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hardwired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are permanently programmed to perform the techniques, or may include one or more general-purpose hardware processors that are programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination thereof. Such special-purpose computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to achieve the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices that include hardwired and / or program logic to implement the techniques.

[0119] For example, Figure 5 is a block diagram illustrating a computer system 500 in which an embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for transmitting and receiving information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0120] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, make computer system 500 a special-purpose machine customized to perform operations specified in the instructions.

[0121] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0122] The computer system 500 may be connected via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric or other keys, is connected to bus 502 for sending and receiving information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for sending and receiving directional information and command selections to the processor 504 and for controlling cursor movement on the display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), allowing the device to be located in a plane.

[0123] Computer system 500 may implement the techniques described above using hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that combine with the computer system to make or program computer system 500 into a special purpose machine. According to one embodiment, the techniques described herein are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. By executing the sequences of instructions contained in main memory 506, processor 504 performs the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0124] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0125] Storage media is distinct from, but may be used in conjunction with, transmission media. Transmission media involves transferring information from one storage medium to another. For example, transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0126] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions into its dynamic memory or send the instructions over a telephone line using a modem. A modem local to the computer system 500 may receive the data on the telephone line and convert the data to an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal. Appropriate circuitry may then place the data on the bus 502. The bus 502 carries the data to a main memory 506. The processor 504 retrieves and executes the instructions from the main memory 506. The instructions received by the main memory 506 may optionally be stored on a storage device 510 either before or after execution by the processor 504.

[0127] Computer system 500 also includes a communications interface 518 coupled to bus 502. Communications interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, communications interface 518 may be an Integrated Services Digital Network (ISDN) card, a cable modem, satellite modem, or a modem to provide a data communications connection to a corresponding type of telephone line. As another example, communications interface 518 may be a local area network (LAN) card to provide a data communications connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, communications interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0128] The network link 520 typically provides data communication through one or more networks to other data devices. For example, the network link 520 may provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. The ISP 526 in turn provides data communication services through the world wide packet data communications network 528, now commonly referred to as the "Internet." Both the local network 522 and the Internet 528 use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518, which carry the digital data to and from the computer system 500, are exemplary forms of transmission media.

[0129] Computer system 500 may send messages and receive data, such as program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0130] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0131] [Equivalents, extensions, substitutes, etc.] In this specification, the embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what the claimed embodiments of the invention are and what the applicant intends the claimed embodiments of the invention to be is the set of claims arising from this application in the specific form in which they arise, including any subsequent amendments. Any definitions expressly set forth in this specification for terms contained in the claims shall determine the meaning of such terms used in the claims. Thus, any limitations, elements, characteristics, features, advantages or attributes not expressly recited in the claims shall not limit the scope of the claims in any way. Thus, the specification and drawings are to be regarded as illustrative, not restrictive.

[0132] [Examples of enumerated embodiments] The present invention may be embodied in any of the forms described herein, including, but not limited to, the following enumerated example embodiments (EEE) that illustrate the structure, features, and functions of some portions of the present invention. EEE1. 1. A method for generating an image in a first dynamic range from an input image in a second dynamic range, comprising: generating a first array of co-occurrence statistics for the second dynamic range input image; generating a second array of reconstruction function index values ​​for selecting a reconstruction function from among a plurality of reconstruction function candidates; (a) performing a denoising operation based at least in part on the first array of co-occurrence statistics, and (b) performing a reconstruction operation based at least in part on the second array of reconstruction function index values ​​to generate a reconstructed image in the first dynamic range; The method includes: EEE2. the second array of reconstruction function index values ​​represents an array of denoised reconstruction function index values ​​generated by performing the denoising operation on a third array of pre-denoised reconstruction function index values. EEE1 method. EEE3. the first array of co-occurrence statistics is a combination of a fourth array of co-occurrence statistics collected from a binary cluster mask derived from the third array of pre-denoised reconstruction function index values ​​and a fifth array of co-occurrence statistics collected from a clipped quantised image derived from the input image. EEE2 method. EEE4. the binary cluster mask is generated from applying standard deviation filtering to the third array of undenoised reconstruction function index values. EEE3 method. EEE5. the standard deviation filtering is applied to the third array of undenoised reconstruction function index values ​​using at least two different spatial filtering kernel sizes. EEE4 method. EEE6. the fifth array of co-occurrence statistics is collected from a clipped quantized image derived from the input image or from the clipped quantized image after one of the input images has been downsampled. Any of the methods EEE3 to 5. EEE7. the fourth array of co-occurrence statistics is stored in a first co-occurrence matrix and the fifth array of co-occurrence statistics is stored in a second co-occurrence matrix, a co-occurrence statistic at a pixel location in the first array representing one of an average or a minimum of a first co-occurrence statistic at the pixel location in the first co-occurrence matrix and a second co-occurrence statistic at the pixel location in the second co-occurrence matrix. Any of the methods EEE3 to 6. EEE8. each reconstruction function index value at a pixel location represented in the second array of reconstruction function index values ​​is generated as a weighted sum of a first reconstruction function index value at the same pixel location represented in the array of intermediate denoised reconstruction function index values ​​and a second reconstruction function index value at the same pixel location represented in the array of pre-denoised reconstruction function index values; Any of the methods EEE1 to 7. EEE9. the weighted sum is determined using weighting elements at the same pixel location in an array of weight mask values, the array of weight mask values ​​being obtained from applying standard deviation filtering to the array of pre-denoised reconstruction function index values; EEE8 method. EEE10. the first array of co-occurrence statistics is generated using a region of interest map excluding co-occurrence statistics of pixel locations within a letterbox; Any of the methods EEE1 to 9. EEE11. the first array of co-occurrence statistics is downsampled from an unsampled array of co-occurrence statistics. Any of the methods EEE1 to 10. EEE12. the second array of reconstruction function index values ​​is generated based at least in part on denoised codewords in a denoised image of the second dynamic range, the denoised codewords in the denoised image of the second dynamic range being generated by performing the denoising operation on pre-denoised codewords in the input image of the second dynamic range. EEE1 method. EEE13. the reconstructed image in the first dynamic range is generated from performing the reconstruction operation on one of the input image or the denoised image. EEE12 method. EEE14. the first array of co-occurrence statistics is collected from the pre-denoised codewords in the input image; EEE12 or EEE13 method. EEE15. the first dynamic range represents a high dynamic range and the second dynamic range represents a standard dynamic range that is lower than the high dynamic range; Any of the methods EEE1 to 14. EEE16. 16. An apparatus comprising a processor and configured to perform any one of the methods described in EEE1 to 15. EEE17. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing a method using one or more processors according to any of the methods described in EEE1 to 15. EEE18. A computer system configured to perform any one of the methods described in EEE1 to 15.

Claims

1. 1. A method for generating an image in a first dynamic range from an input image in a second dynamic range, comprising: generating a first array of co-occurrence statistics for the second dynamic range input image, where a co-occurrence statistic is a measure of the frequency of a pair of values ​​appearing together in an image or an array of values, the first array of co-occurrence statistics being a combination of a fourth array of co-occurrence statistics collected from a binary cluster map derived from a third array of pre-denoised reconstruction function index values ​​and a fifth array of co-occurrence statistics collected from a clipped quantised image derived from the input image; generating a second array of reconstruction function index values ​​for selecting a reconstruction function from among a plurality of reconstruction function candidates, the second array of reconstruction function index values ​​representing an array of denoised reconstruction function index values ​​generated by performing a denoising operation on the third array of pre-denoised reconstruction function index values; (a) performing a denoising operation based at least in part on the first array of co-occurrence statistics, and (b) performing a reconstruction operation based at least in part on the second array of reconstruction function index values ​​to generate a reconstructed image in the first dynamic range; The method includes:

2. the binary cluster map being generated from applying standard deviation filtering to the third array of pre-denoised reconstruction function index values. The method of claim 1.

3. the standard deviation filtering is applied to the third array of undenoised reconstruction function index values ​​using at least two different spatial filtering kernel sizes. The method of claim 2.

4. the fifth array of co-occurrence statistics is collected from a clipped quantized image derived from the input image or after one of the input images has been downsampled. The method of claim 1.

5. a fourth array of the co-occurrence statistics is stored in a first co-occurrence matrix and a fifth array of the co-occurrence statistics is stored in a second co-occurrence matrix, the co-occurrence statistic at a pixel position in the first array representing one of an average or a minimum of a first co-occurrence statistic at the pixel position in the first co-occurrence matrix and a second co-occurrence statistic at the pixel position in the second co-occurrence matrix. The method of claim 1.

6. each reconstruction function index value at a pixel location represented in the second array of reconstruction function index values ​​is generated as a weighted sum of a first reconstruction function index value at the same pixel location represented in the array of intermediate denoised reconstruction function index values ​​and a second reconstruction function index value at the same pixel location represented in the array of pre-denoised reconstruction function index values; The method of claim 1.

7. the weighted sum is determined using weighting elements at the same pixel location in an array of weight mask values, the array of weight mask values ​​being obtained from applying standard deviation filtering to the array of pre-denoised reconstruction function index values; The method according to claim 6.

8. the first array of co-occurrence statistics is generated using a region of interest map excluding co-occurrence statistics of pixel locations within a letterbox, the letterbox referring to one or more static regions of any shape, such as a letterbox, a pillarbox, an oval, or other shape; The method of claim 1.

9. the first array of co-occurrence statistics is downsampled from an unsampled array of co-occurrence statistics. The method of claim 1.

10. the second array of reconstruction function index values ​​is generated based at least in part on denoised codewords in a denoised image of the second dynamic range, the denoised codewords in the denoised image of the second dynamic range being generated by performing the denoising operation on pre-denoised codewords in the input image of the second dynamic range, the codewords referring to pixel values. The method of claim 1.

11. the reconstructed image in the first dynamic range is generated from performing the reconstruction operation on one of the input image or the denoised image. The method of claim 10.

12. the first array of co-occurrence statistics is collected from the pre-denoised codewords in the input image; The method of claim 10.

13. the first dynamic range represents a high dynamic range and the second dynamic range represents a standard dynamic range that is lower than the high dynamic range; The method of claim 1.

14. An apparatus comprising a processor and configured to perform any one of the methods according to claims 1 to 13.

15. A non-transitory computer readable storage medium having stored thereon computer executable instructions for performing, with one or more processors, a method according to any of the methods recited in claims 1 to 13.

Images