Blind local shaping in HDR imaging

Abstract

In an encoder, a high dynamic range (HDR) image is encoded using a family of local forward shaping functions selected according to a forward mapping index (FMI) array, the forward mapping index indicating which local forward shaping function needs to be used for each pixel in the HDR image to generate a shaped standard dynamic range (SDR) image. Given a shaped SDR image, a decoder iteratively generates a reconstructed HDR image and an estimated shaped SDR image by adjusting a local FMI array and a local backward mapping index (BMI) array until an error metric associated with the difference between the local BMI and the FMI array and the difference between the estimated SDR image and the shaped SDR image satisfies a convergence criterion. Techniques for generating a family of local forward shaping functions and a family of local backward shaping functions based on a global forward shaping function are also presented.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 004,609 and European Patent Application No. 20167908.1, both filed on April 3, 2020, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to images and, more particularly, to improving the encoding and decoding efficiency and image quality of high dynamic range (HDR) images reconstructed from standard dynamic range (SDR) images using a local reshaping function. Background Art

[0004] As used herein, the term "dynamic range (DR)" may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, such as from the darkest gray (black) to the brightest white (highlight). In this sense, DR is related to "scene-referred" intensities. DR may also relate to the ability of a display device to adequately or approximately render a range of intensities of a particular breadth. In this sense, DR is related to "display-referred" intensities. Unless a particular sense is explicitly designated at any point in the description herein as having a particular meaning, it should be inferred that the terms may be used in either sense, e.g., interchangeably.

[0005] As used herein, the term "high dynamic range (HDR)" refers to a DR width that spans 14 to 15 orders of magnitude of the human visual system (HVS). In practice, the DR, over a wide range of intensities that humans can simultaneously perceive, may be slightly truncated relative to HDR. As used herein, the terms "visual dynamic range (VDR) or enhanced dynamic range (EDR)" may relate individually or interchangeably to the DR that can be perceived within a scene or image by the human visual system (HVS) including eye movement, allowing for some photometric variations across the scene or image. As used herein, VDR may refer to a DR that spans 5 to 6 orders of magnitude. Thus, while perhaps slightly narrower relative to real-scene-referenced HDR, VDR or EDR may represent a wide DR width and may also be referred to as HDR.

[0006] In practice, an image includes one or more color components (e.g., luma Y and chroma Cb and Cr), where each color component is represented by n bits of precision per pixel (e.g., n=8). For example, using gamma luminance codec, images where n≤8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images where n≥10 can be considered enhanced dynamic range images. HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0007] Most consumer desktop monitors currently support 200 to 300 cd / m 2 Most consumer HDTVs range from 300 to 500 nits, with newer models reaching 1000 nits (cd / m 2 ). Thus, such conventional displays represent a lower dynamic range (LDR) relative to HDR, also known as standard dynamic range (SDR). As the availability of HDR content increases due to advances in both capture devices (e.g., cameras) and HDR displays (e.g., Dolby Laboratories' PRM-4200 Professional Reference Monitor), HDR content can be color graded and displayed on HDR displays that support a higher dynamic range (e.g., from 1,000 nits to 5,000 nits or more).

[0008] As used herein, the term "forward reshaping" refers to the process of sample-to-sample mapping or codeword-to-codeword mapping of a digital image from its original bit depth and original codeword distribution or representation (e.g., gamma, PQ, HLG, etc.) to an image of the same or different bit depth and different codeword distribution or representation. Reshaping allows for improved compressibility or improved image quality at a fixed bit rate. For example, and without limitation, reshaping can be applied to HDR video encoded with 10-bit or 12-bit PQ to improve codec efficiency in a 10-bit video codec architecture. In a receiver, after decompressing the received signal (which may or may not be reshaped), the receiver can apply an "inverse (or backward) reshaping function" to restore the signal to its original codeword distribution and / or achieve a higher dynamic range.

[0009] In conventional shaping techniques, a single global forward shaping function may be applied to all pixels in an input HDR image to generate a shaped SDR image to be compressed and transmitted to a decoder. Information related to the forward shaping function (e.g., parameter representation of the backward shaping function) may then be sent to the decoder as metadata along with the shaped SDR image to aid the decoder in reconstructing the input HDR image. As appreciated by the inventors herein, improved techniques for image shaping are desired to reduce coding artifacts in HDR encoding and decoding and to improve the quality of the corresponding shaped SDR image.

[0010] As used herein, the term “local shaping” means that the encoder may use a family of shaping functions and may encode each pixel in an input image using a different shaping function selected from the family of shaping functions according to local spatial information.

[0011] As used herein, the term "blind local shaping" refers to a method where no information is sent to the decoder about how the individual forward shaping function was selected for each input HDR image pixel in the encoder, and the decoder is therefore required to autonomously reconstruct this information from the encoded SDR data.

[0012] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying figures and in which like reference numerals refer to similar elements and in which:

[0014] FIG1A depicts an example single-layer encoder for HDR data using a global shaping function according to the prior art;

[0015] FIG1B depicts an example HDR decoder corresponding to the encoder of FIG1A according to the prior art;

[0016] Figure 2A depicts an example single-layer encoder for HDR data using local forward shaping according to an embodiment of the present invention;

[0017] Figure 2B Describes an embodiment of the present invention and Figure 2A An example HDR decoder corresponding to the encoder;

[0018] Figure 3Adepicts an example iterative process for blind local backward shaping according to a first embodiment of the present invention;

[0019] Figure 3B depicts an example iterative process for blind local backward shaping according to a second embodiment of the present invention;

[0020] Figure 4A Depicts an example graph of the global forward shaping function;

[0021] Figure 4B Depicts the Figure 4A Example graph of the template forward shaping function corresponding to the global forward shaping function;

[0022] Figure 4C Depicts the Figure 4B Example graph of the template backward shaping function corresponding to the template forward shaping function;

[0023] Figure 4D Depicts the Figure 4B A set of example graphs of local forward shaping functions corresponding to the template forward shaping function;

[0024] Figure 4E Depicts the Figure 4C An example graph of a set of local backward shaping functions corresponding to the template backward shaping function; and

[0025] Figure 5 An example process of generating a family of local forward shaping functions and a family of local backward shaping functions based on a global forward shaping function according to an embodiment of the present invention is described. DETAILED DESCRIPTION

[0026] This document describes a method for blind local reshaping for encoding and decoding HDR images and video content. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be practiced without these specific details. In other instances, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring, or obfuscating the present invention. Summary of the Invention

[0027] The example embodiments described herein relate to blind local shaping for encoding and decoding HDR images. In an embodiment, in an encoder, given an input HDR image, the encoder uses a family of forward shaping functions and a forward mapping index (FMI) array, wherein the FMI array indicates which forward shaping function to use for each input HDR pixel to shape the input HDR image and generate a shaped SDR image. In a decoder, given a received shaped SDR image and a family of backward shaping functions, the decoder applies an iterative technique to generate a backward mapping index (BMI) array, wherein the BMI array indicates which backward shaping function to use for each SDR pixel to generate a reconstructed HDR image that is closest to the input HDR image.

[0028] In an embodiment, in an apparatus comprising one or more processors, a decoder receives an input reshaped image (156) in a first dynamic range, the decoder initializes (305) a back mapping index (BMI) array, wherein each element of the BMI array indicates an index of a local back shaping function in a set of two or more local back shaping functions, wherein the local back shaping function maps pixel values ​​from the first dynamic range to pixel values ​​in a second dynamic range, and performs one or more iterations to generate an output reconstructed image in the second dynamic range, wherein the iterations comprise:

[0029] generating (310) a first reconstructed image in a second dynamic range by applying a set of local backward shaping functions to the input shaped image according to the BMI array;

[0030] generating (315) a first forward mapping index (FMI) array based on the first reconstructed image and a forward mapping index (FMI) generation function, wherein each element of the FMI array indicates an index of a local forward shaping function in a set of two or more local forward shaping functions, wherein the local forward shaping function maps pixel values ​​from the second dynamic range to pixel values ​​in the first dynamic range;

[0031] measuring a difference between the BMI array and the first FMI array according to a first error metric to generate a mapping index error;

[0032] generating (325) an updated BMI array based on the mapped index errors;

[0033] generating (330) a second reconstructed image in a second dynamic range by applying a set of local backward shaping functions to the input shaped image according to the updated BMI array;

[0034] generating (335) an updated FMI array based on the second reconstructed image and the FMI generation function;

[0035] generating (340) an estimated shaped image in the first dynamic range by applying a set of local forward shaping functions to the second reconstructed image according to the updated FMI array;

[0036] measuring a difference between the estimated shaped image and the input shaped image according to a second error metric to generate a shaped image error; and

[0037] Testing whether the combination of the shaped image error and the mapping index error meets the convergence criteria:

[0038] If the convergence criterion is met, terminating the iteration and outputting the second reconstructed image as the output reconstructed image;

[0039] otherwise:

[0040] replacing (360) elements of the BMI array based on the updated elements of the BMI array and the shaped image error; and

[0041] Perform another iteration.

[0042] In a second embodiment, in an apparatus comprising one or more processors, the processor performs the following operations: accessing a global forward shaping function;

[0043] generating property values ​​for the global forward shaping function, wherein the property values ​​include one or more of: a non-flat region of the function in the second dynamic range (x-range), a midpoint of the x-range, and flat regions of shadows and highlights in the second dynamic range;

[0044] Generates a template forward shaping function based on the global forward shaping function, the attribute value, and the x-range scaling factor;

[0045] generating a template backward shaping function by calculating an inverse function of the template forward shaping function; and, for a local forward shaping function associated with a target luminance value in a second dynamic range, generating a local forward shaping function by shifting the template forward shaping function by an x-axis shift value, wherein the x-axis shift value is determined such that, for the target luminance value, corresponding average shaped output values ​​using the local forward shaping function and the template forward shaping function are approximately equal.

[0046] Example HDR codec system

[0047] As described in U.S. Patent Serial No. 10,032,262, “Block-based content-adaptive reshaping for high dynamic range images” by A. Kheradmand et al. (referred to as the '262 patent), the entire contents of which are incorporated herein by reference in their entirety, FIG1A and FIG1B illustrate an example single-layer backward-compatible codec framework using image reshaping. More specifically, FIG1A illustrates an example encoder-side codec architecture that can be implemented with one or more computing processors in an upstream video encoder. FIG1B illustrates an example decoder-side codec architecture that can also be implemented with one or more computing processors in one or more downstream video decoders.

[0048] In this framework, given a reference HDR content (120), the corresponding SDR content (134) (also referred to as the base layer (BL) or shaped content) is encoded and transmitted in a single layer of an encoded video signal (144) by an upstream encoding device that implements an encoder-side codec architecture. The SDR content is received and decoded in a single layer of a video signal by a downstream decoding device that implements a decoder-side codec architecture. Backward shaping metadata (152) is also encoded and transmitted in the video signal along with the SDR content, so that an HDR display device can reconstruct the HDR content based on the SDR content and the backward shaping metadata. Without loss of generality, in some embodiments, such as in non-backward compatible systems, the SDR content itself may not be viewable, but must be viewed in conjunction with a backward shaping function that will generate viewable SDR or HDR content. In other embodiments that support backward compatibility, a legacy SDR decoder can still play back the received SDR content without employing a backward shaping function.

[0049] As illustrated in FIG1A , given an HDR image (120) and a target dynamic range, after generating a forward shaping function (132) in step 130, a forward shaping mapping step (132) is applied to the HDR image (120) to generate a shaped SDR base layer (134) given the forward shaping function. A compression block (142) (e.g., an encoder implemented according to any known video codec algorithm, such as AVC, HEVC, AV1, etc.) compresses / encodes the SDR image (134) into a single layer (144) of a video signal. Additionally, a backward shaping function generator (150) may generate a backward shaping function that may be transmitted to a decoder as metadata 152. In some embodiments, the metadata (152) may represent the forward shaping function (130), so that the decoder will generate a backward shaping function (not shown).

[0050] Examples of backward shaping metadata representing / specifying an optimal backward shaping function may include, but are not necessarily limited to, any of the following: an inverse tone mapping function, an inverse luma mapping function, an inverse chroma mapping function, a lookup table (LUT), a polynomial, inverse display management coefficients / parameters, etc. In various embodiments, the luma backward shaping function and the chroma backward shaping function may be derived / optimized jointly or separately, and may be derived using various techniques, such as, but not limited to, as described in the '262 patent.

[0051] The backward shaping metadata (152) generated by the backward shaping function generator (150) based on the SDR image (134) and the target HDR image (120) can be multiplexed as part of the video signal 144, for example, as a supplemental enhancement information (SEI) message.

[0052] In some embodiments, the backward shaping metadata (152) is carried in the video signal as part of the overall image metadata that is carried in the video signal separately from the individual layers in the video signal in which the SDR images are encoded. For example, the backward shaping metadata (152) may be encoded in a component stream in the coded bitstream that may or may not be separate from the individual layers (of the coded bitstream) in which the SDR images (134) are encoded.

[0053] Therefore, the backward shaping metadata (152) can be generated or pre-generated on the encoder side to take advantage of the powerful computing resources and offline encoding processes available on the encoder side (including but not limited to content-adaptive multi-pass, look-ahead operations, inverse luma mapping, inverse chroma mapping, CDF-based histogram approximation and / or transfer, etc.).

[0054] The encoder-side architecture of Figure 1A can be used to avoid encoding the target HDR image (120) directly as an encoded / compressed HDR image in the video signal; instead, backward shaping metadata (152) in the video signal can be used to enable downstream decoding devices to backward shape the SDR image (134) (encoded in the video signal) into a reconstructed image that is identical to or close to / best approximation of the reference HDR image (120).

[0055] In some embodiments, as illustrated in FIG1B , a video signal encoded with an SDR image in a single layer (144) and backward shaping metadata (152) as part of the overall image metadata are received as input at the decoder side of the codec framework. A decompression block (154) decompresses / decodes the compressed video data in the single layer (144) of the video signal into a decoded SDR image (156). Decompression 154 generally corresponds to the inverse process of compression 142. The decoded SDR image (156) may be identical to the SDR image (134), but has undergone quantization errors in the compression block (142) and the decompression block (154), which have been optimized for SDR display devices. In backward compatible systems, the decoded SDR image (156) may be output in an output SDR video signal (e.g., via an HDMI interface, via a video link, etc.) for presentation on an SDR display device.

[0056] Optionally, alternatively, or additionally, in the same or another embodiment, a backward shaping block 158 extracts backward (or forward) shaping metadata (152) from the input video signal, constructs a backward shaping function based on the shaping metadata (152), and performs a backward shaping operation on the decoded SDR image (156) based on the optimal backward shaping function to generate a backward shaped image (160) (or a reconstructed HDR image). In some embodiments, the backward shaped image represents a production quality or near-production quality HDR image that is the same as or close to / best approximates the reference HDR image (120). The backward shaped image (160) can be output in the form of an output HDR video signal (e.g., via an HDMI interface, via a video link, etc.) for presentation on an HDR display device.

[0057] In some embodiments, as part of an HDR image rendering operation to render the back-shaped image (160) on an HDR display device, display management operations specific to the HDR display device may be performed on the back-shaped image (160).

[0058] Blind local plastic surgery

[0059] Embodiments based on the system depicted in FIG. 1A and FIG. 1B may be considered to use "global shaping" because the same forward shaping function is used to remap all pixels in a frame. Figure 2A An example of an embodiment of "local shaping" is depicted, where the encoder can apply multiple forward shaping functions, where each function is selected based on local image characteristics.

[0060] Compared with Figure 1A, Figure 2AAs depicted, there is now a new forward mapping index (FMI) table 205 which indicates which forward shaping function from a finite set of such functions should be used for each pixel in the reference HDR image (120). Given the forward mapping table 205, forward shaping (215) is performed as before, except that the shaper engine (215) uses more than one forward shaping function, as provided by unit (210), thereby supporting local shaping. In order to provide backward compatibility with decoders that are unable to support locally shaped SDR images, in an embodiment, as shown in FIG1A, the optional unit 150 may also generate a global backward shaping function or other metadata related to the global forward shaping function, which may be transmitted as metadata (152) together with the encoded shaped SDR image (144). Note that the FMI array (205) does not need to be transmitted to the decoder. As will be explained later, in an example embodiment, a decoder (e.g., such as Figure 2B (depicted) can apply local inverse or backward shaping to generate a reconstructed version of the input HDR image using an iterative decoding process (160). This type of shaping that allows the decoder to perform local backward shaping without any metadata from the encoder can be referred to as "blind" shaping.

[0061] Notation

[0062] Given an input reference HDR(120) sequence, let its bit depth be denoted as B v Let the bit depth of the corresponding shaped SDR sequence (134) be denoted as B s . Make v t，i represents the i-th pixel of the t-th frame in the input reference HDR signal. In an embodiment, all P pixels in the t-th frame can be collected together as V t Array of .

[0063] Make t，i represents the i-th pixel in the t-th frame of the reference SDR signal. All P pixels in the t-th frame can be collected together as a vector S t In the embodiment, without loss of generality, for ease of discussion, the pixel value (v t，i and s t，i ) are not normalized (e.g., in [0, 1]) and are assumed in this document to be in their original bit depth range (e.g., for each color plane in Although the methods presented herein may be discussed for a single color plane (such as luma Y), these methods are applicable to all available color planes in the input signal, regardless of its color space (e.g., RGB, YCbCr, etc.).

[0064] Denote the forward shaping function family as {F <l>< / l>()|l=0,...,L-1}, and the backward shaping function family is represented as {B <l>< / l> ()|l=0, ..., L-1}, where L represents the number of functions in each family. In an embodiment, the two families are reversible within a reasonable approximation with respect to quantization errors due to bit depth differences and non-uniform intervals, so:

[0065]

[0066] Expressed as

[0067] M t =G(V t ) (2)

[0068] The process of generating a forward mapping index (FMI) (205), wherein M t The elements in are denoted as m t，i , where m t，i is between 0 and L-1 and indicates the forward shaping function to be used for the i-th pixel.

[0069] As mentioned before, because M t is not transmitted to the decoder, so the decoder can use an iterative process to estimate M t and the backward mapping index n with the estimate t，i Correspondence table (N t )(240). For notational purposes, at iteration k, the shaping function selection index in the forward path is denoted as And the shaping function selection index in the backward path is expressed as in, and has a value between 0 and L-1. Note that these two choices may not be the same due to estimation accuracy and convergence criteria. Furthermore, in the decoder, the selected forward shaping function for the i-th pixel is expressed as And the selected backward shaping function is expressed as Local forward shaping

[0070] like Figure 2A As depicted, in the encoder, given M t , in block 215, the shaped SDR signal (134) is generated as

[0071]

[0072] As an example, in an embodiment, without limitation, the equation (2) used to generate M t The mapping function G() can be expressed as

[0073] G()=G2(G1()). (4)

[0074] For example, the first operator G1() may be a Gaussian blur operator that, when applied to the input HDR image, generates blurred pixel values

[0075]

[0076] The weights are defined as:

[0077]

[0078] where x and y represent the distance of the weighted pixel from the filter center, 2W+1 represents the width and height of the filter (e.g., W=12), and σ represents the standard deviation of the blur (e.g., σ=50 for 1080p content).

[0079] As an example, in an embodiment, the second operator G2() is a uniform quantizer that assigns one of 0 to L-1 values ​​to the output of the blur filter.

[0080]

[0081] Where U represents the quantization interval, and z=clip3(x, a, b) represents the clipping function, where

[0082]

[0083] For L uniform intervals, we have HDR codewords,

[0084]

[0085] In some embodiments, the uniform quantizer may be replaced by a non-uniform quantizer or some other mapping function. In some embodiments, equation (4) does not apply to any letterbox regions that may surround the active region of the frame. Instead, when a letterbox region is detected, a single constant value (e.g., m t，i = C, corresponding to the cth forward shaping function) is applied to all pixels in the letterbox area. In an embodiment, instead of using a blurring filter, a low-pass filter may be used.

[0086] Local back shaping

[0087] The goal of local backshaping is to generate a reconstructed HDR image (160) that is as close as possible to the original reference HDR image (120). During the reconstruction process, for example, Figure 2B As described, it is necessary to evaluate the forward function selection (225) and the backward function selection (240), each with a corresponding index and , where k represents the kth iteration in the iterative estimation process.

[0088] One of the convergence conditions is that the two function choices need to be approximately equal, i.e. However, satisfying this convergence condition does not necessarily mean that the reconstructed HDR image will be close to the original HDR image, because for all values ​​of i, any trivial constant value (such as ) can easily meet such requirements but cannot predict the correct HDR signal. In an embodiment, assuming that the shaped SDR signal (156) is known, another convergence condition is to check whether the shaped SDR image (denoted as SDR) generated using the reconstructed HDR image (160) (k) )(232) is close enough to the original input SDR image (156). Therefore, if these two convergence conditions are met, the estimated HDR image should be close to the original HDR image.

[0089] In an embodiment, the estimation process may be formulated as an optimization problem to minimize a) the difference between the two shaping function selection indices and b) the difference between the input shaped SDR and the estimated shaped SDR. (k) To achieve this, an iterative algorithm is proposed. In each iteration, the difference between the two estimated values ​​under the convergence conditions is tested, and if it fails, the difference between the two estimated values ​​under the convergence conditions is tested. and The steps of this iterative process refer to Figure 2B and Figure 3A The example process flow is discussed in more detail in .

[0090] Step 1 (305): Initialization

[0091] 1. k = 0;

[0092] 2. Initialization (e.g., to a constant c = L / 2) for selecting the local backward shaping function (B <l>< / l> Backward Map Index (BMI) array for ()) (240).

[0093] Step 2 (310): Perform local backshaping (220)

[0094] For each pixel in the input shaped SDR signal (156) Find the corresponding integer function index And apply the corresponding backward shaping function to generate the reconstructed HDR signal

[0095]

[0096] The reconstructed HDR image at the kth iteration can be represented as a vector V t (k) .

[0097] Step 3 (315): Generate an estimated forward mapping index (FMI) array (225) based on the reconstructed HDR image. The G() function may be the same as that used by the encoder.

[0098]

[0099] Step 4 (325): Update the BMI array (240) based on the difference between the current estimated FMI and the BMI array.

[0100] 1. First calculate the difference between the existing estimated FMI and BMI arrays

[0101]

[0102] And calculate the average difference over the entire frame

[0103]

[0104] 2. Next, update the BMI array based on the average difference

[0105]

[0106] In an embodiment, h() is a rounding function and / or a clipping function, and α1 controls the convergence speed (eg, α1=1).

[0107] Step 5 (330): Given the updated BMI value array (240), perform the local backshaping (220) again. For each input shaped SDR pixel, find the updated backshaping function index and applying local backshaping (220) to obtain an updated (second) version of the reconstructed HDR signal

[0108]

[0109] The updated (second) version of the reconstructed image at iteration k can be expressed as

[0110] Step 6 (335): Estimate the second FMI array (225) based on the updated HDR reconstructed signal

[0111]

[0112] Step 7 (340): Perform local forward shaping (230) based on the estimated HDR reconstruction signal to obtain the shaped SDR image SDR (k) Estimates.

[0113] For each pixel from Find the forward shaping function index in and applying (230) the corresponding forward shaping function to generate a shaped SDR signal The estimated version of (232).

[0114]

[0115] Step 8 (345): Calculate the difference between the input SDR reshaped image and the estimated SDR reshaped image

[0116]

[0117]

[0118] Note that in step 4 and this step, the L1 error can be replaced by the mean square error metric or other error metrics known in the art (such as signal-to-noise ratio (SNR) or peak SNR). This metric is more computationally intensive but can improve the convergence speed.

[0119] Step 9 (350): Determine convergence and update the BMI array

[0120] 1. Calculate the overall error

[0121]

[0122] Where w1+w2=1 is a weighting factor used to weight the two types of errors. For example, if the two errors are considered equally important, then w1=w2=0.5.

[0123] 2. If the overall error If the value is less than a threshold Δ (e.g., Δ=0.5 for unnormalized 10-bit data), the decoder has converged. Current BMI array is considered the final array and in an output step 355 the output of step 5 (330) (Equation (14)) is considered the final reconstructed HDR output.

[0124] 3. If This means that a better solution may not be found. is considered as the final array and in the output step 355, the previously generated signal is considered the final reconstructed HDR output.

[0125] 4. Otherwise (step 360) and ), update (235) the BMI array based on the shaped SDR difference and return to step 2 (310), that is:

[0126]

[0127] Return to step 2 (310),

[0128] Here, α2 is another variable used to manage convergence (eg, α2=1).

[0129] and Figure 2A By comparison, it can be noted that in each iteration, the process includes two separate updates of the FMI array: a) in step 3 (315) and b) in step 6 (335). When the FMI array is updated for the first time, the BMI array is updated based on the error between the current BMI and the FMI value (step 325). When the FMI array is updated for the second time, the decoder uses the second FMI array to generate a more accurate estimate of the shaped SDR before calculating the shaped SDR error in step 8 (345). Experimental results show that such a process iterates faster.

[0130] In another embodiment, this repetition step can be eliminated to reduce complexity, but overall convergence may be slower. Figure 3B Depicted in, among others, Figure 3A Steps 5 (330) and 6 (335) in are now removed. This means that the previous step 8 (345) for calculating the difference of the shaped SDR will now be replaced by step 345B, which also calculates the BMI-FMI error:

[0131]

[0132] Among them, (340B), SDR (k) is given by:

[0133]

[0134] Additionally, updating the BMI array (325) based on the error between the current BMI and the FMI difference is absorbed into step 360B because if there is convergence, then there is no need to update the BMI array.

[0135]

[0136] Return to step 2 (310),

[0137] Here, the error metric can be calculated using equation (20).

[0138] Now, if there is convergence (eg, step 355B), the current BMI array is considered the final array, and the output of step 2 (310) (Equation (9)) is considered the final reconstructed HDR output (160).

[0139] Constructing a local shaping function

[0140] The methods described so far apply to any type of reversible shaping function. This section discusses some concrete examples of local shaping functions and their properties.

[0141] Local shaping function based on global shaping function

[0142] The global forward shaping function and the global backward shaping function are expressed as F() and B() respectively. Ideally, when the quantization error F() = B -1 In an embodiment, the selection of the local shaping function may depend on a property of the luma pixel (eg, the value of the luma pixel calculated based on neighboring pixels or a value such as an average value, a standard deviation, etc.).

[0143] In an embodiment, the following properties may be desirable for a set of local shaping functions:

[0144] Use local shaping to locally increase the sharpness and contrast of the shaped SDR image. This can be achieved by increasing the slope of each local forward shaping function while compressing the global forward shaping function on the x-axis; that is, by reducing the width (range) of the non-flat region of the global shaping function on the HDR x-axis while maintaining the same range on the SDR y-axis. Denote this x-axis compression ratio as α, where 0<α<1.

[0145] Keep the global brightness constant. This can be achieved if, given an HDR codeword, each local shaping function produces a shaped SDR value close to the value provided by the global shaping function.

[0146] Given a local forward shaping function, a local backward shaping function can be constructed by inverting the local forward shaping function. As discussed before, it is necessary to construct L forward local shaping functions and L backward local shaping functions.

[0147] For the first property above, if the x-axis compression ratio is fixed for all L forward / backward shaping functions, then the template forward function F can be constructed T () and template backward shaping function B T() to serve as the basis function. Next, the template forward shaping function can be shifted across the x-axis to generate the desired set of local forward shaping functions. These steps are described below.

[0148] Step 1. Identify properties of the global shaping function

[0149] The first step is to identify properties of the global forward shaping function F(), such as its input and output ranges (e.g., its minimum and maximum input and output codewords), its median values ​​in the non-flat range, etc. Without limitation, Figure 4A An example of a global forward shaping function that maps 16-bit HDR data (4,000 nits, PQ) to 10-bit SDR data (100 nits, gamma) is shown in FIG. Figure 4A As depicted, a typical global shaping function has an S-shaped characteristic with two flat regions in the dark and bright regions, due to the constraints on valid input and output values. To construct the template forward shaping function, those flat regions can be excluded and only regions with (scaled) slopes > 0 can be considered. As an example, to identify the start and end points of non-flat regions, the pseudocode in Table 1 can be applied, where v represents a very small threshold greater than zero that is related to the numerical accuracy of our calculations (e.g., δ = 0.0001 for floating-point operations).

[0150] Table 1: Searching for flat segments in the global shaping function

[0151]

[0152] The entire valid input HDR range can then be calculated as follows:

[0153] v R =v H -v L . (twenty three)

[0154] The midpoint of the valid input HDR range can be calculated as

[0155]

[0156] Step 2. Generate template forward shaping function

[0157] Given the x-axis compression ratio α, the new scaling range of the HDR signal is

[0158]

[0159] Due to the ceiling operator, the value of α can be updated as

[0160]

[0161] Then, the template forward shaping function F T() can be constructed as described in Table 2.

[0162] Table 2: Example of generating a template function based on a global shaping function

[0163]

[0164] Figure 4B Describes the Figure 4A Example of a template forward shaping function generated using α=0.8 for the global forward shaping function.

[0165] Step 3. Generate template backward shaping function

[0166] Template backward shaping function B T () can be reconstructed by inverting the template forward shaping function. An example process is depicted in Table 3. Given Figure 4B The template forward shaping function in Figure 4C The corresponding template backward shaping function is presented.

[0167] Table 3: Example process of generating a template backward shaping function from a template forward shaping function

[0168]

[0169]

[0170] Step 4: Generate a family of local forward shaping functions

[0171] Given a template forward shaping function and a template backward shaping function, the entire family of local shaping functions can be constructed as follows. Construct a shifted version of each local shaping function

[0172] Consider constructing L local forward shaping functions F <l>< / l> () and B <l>< / l> (). When m t，i = 1, the lth forward shaping function is used to shape the i-th HDR pixel. In an embodiment, the first step is to divide the input codeword range into L uniform intervals and find the center of each interval.

[0173]

[0174] Given the goal of maintaining global brightness, in an embodiment, one solution consists in Force the original global shaping function and local shaping functions The scaled and shifted versions of have the same mapping values. Note that a horizontal shift of the local forward shaping function will result in a vertical shift of the corresponding local backward shaping function. In other words, this property can be used to determine the "shift" of the lth local shaping function from the template function without recalculating the function. This is depicted in Table 4, where the main idea is to generate the lth local shaping function based on the template function, shifting the template forward shaping function so that it is centered at the lth bin. In other words, at the center of the lth bin At , the lth local shaping function and the global shaping function are mapped to the same value. Therefore, in order to generate the lth local forward shaping function and the local backward shaping function:

[0175] Table 4: Example of generating a local shaping function by shifting the template forward shaping function to match its brightness

[0176]

[0177]

[0178] For local functions that have invalid flags (e.g., because their start or end points are in the valid range (except for ), a local function can be generated by interpolating from the nearest local shaping function or by copying its nearest local shaping function. An example of this process is depicted in Table 5, but is not limited thereto.

[0179] Table 5: Example of interpolating a local shaping function from its neighboring functions when no valid function is found

[0180]

[0181] As an example, Figure 4D Four of the L = 1,024 local forward shaping functions are depicted, for α = 0.8, along with the original global forward shaping function. Smaller values ​​of α result in steeper slopes and larger shifts. Steeper slopes result in higher contrast. Unless both luminance and chrominance are adjusted, larger shifts can distort the image's colors; for example, colors may appear washed out. Figure 4E Eight of the corresponding 1,024 local backward shaping functions are depicted.

[0182] Variants and Annotations

[0183] A potential problem in the scaling and shifting-based derivation of the proposed local shaping function is hard bright / dark clipping in local regions. As the slope becomes higher, highlights and shadows saturate earlier. To avoid the premature saturation problem, the following alternative solution can be applied.

[0184] a) Apply different scaling and shifting in each local reshaping function

[0185] In such an embodiment, for example, scaling in highlights and shadows can be avoided to avoid clipping. To smooth the transition in the brightness domain, the scaling factor can be gradually changed from 1 to a target value (e.g., 0.8) from dark to mid-tones, kept constant in mid-tones, and gradually increased to 1 in highlights. This approach can delay premature saturation. On the other hand, this also means that contrast will be reduced in this approach.

[0186] b) Fusion of local shaping function and global shaping function

[0187] In this embodiment, a weighted linear combination of a local shaping function and a global shaping function can be applied to remove saturated portions. The weighting factor can be a function of brightness. When the local shaping function is in shadows or highlights, the weight will tend toward the global function; when the local shaping function is in mid-tones, the weight will tend toward the local function.

[0188] The weighting factors of global function and local function are expressed as and in,

[0189]

[0190] The final fused local shaping function can be expressed as

[0191]

[0192] For example, in an embodiment, for l=0 to L=1,023:

[0193]

[0194] For l = 0 to 1,023.

[0195] c) Use local shaping as a sharpening operator

[0196] As the inventors appreciate, when the global shaping function is a simple one-to-one mapping, then a set of local forward shaping functions constructed using the previously discussed algorithm can be used as a sharpening operator. Thus, in an embodiment, a set of local shaping functions can be used for SDR-to-SDR mapping, where the ultimate goal is to improve the perceived clarity of the transmitted SDR image.

[0197] d) Backward compatibility

[0198] Local shaping does not require any additional metadata, but may require predefined knowledge of the local forward and backward shaping functions, or at least characteristics of a global forward shaping function and / or a global backward shaping function, such that the local shaping functions can be locally reconstructed, e.g., as defined by metadata 152. A legacy decoder that is not capable of applying local backward shaping may still apply global shaping (e.g., as shown in FIG1B ) even if the incoming SDR signal was encoded with a local shaper.

[0199] Figure 5 Describes an example overview of the process of generating a family of local shaping functions based on a global shaping function. Figure 5 As depicted, given a global forward shaping function (502), step 505 finds its properties (e.g., flat region, range, and midpoint). Given an x-axis scaling parameter α and the collected properties of the global forward shaping function, in step 510: a) a template forward function is generated, and b) after generating the forward function, a corresponding template backward shaping function is generated. Finally, given a desired number of functions (L) and optimization constraints (such as maintaining global brightness), step 515 generates a family of local forward and backward shaping functions based on the template forward and backward shaping functions.

[0200] Example Computer System Implementation

[0201] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete-time or digital signal processor (DSP), an application-specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to blind local shaping, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to blind local shaping described herein. The image and video dynamic range extension embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0202] Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. can implement the method for blind local shaping as described above by executing software instructions in a program memory accessible to the processor. The present invention can also be provided in the form of a program product. The program product may include any non-transient and tangible medium that carries a set of computer-readable signals, wherein the set of computer-readable signals includes instructions that, when executed by a data processor, cause the data processor to perform the method of the present invention. The program product according to the present invention can be in any of various non-transient and tangible forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash memory RAM, etc. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0203] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents (e.g., functionally equivalent) to the components, including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

[0204] Equivalents, extensions, alternatives, and miscellaneous

[0205] Thus, example embodiments involving blind local shaping of HDR images have been described. In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary depending on the implementation. Accordingly, the sole and exclusive indicator of the invention, and the applicant's inventive intent, is the set of claims issuing from this application in the specific form in which such claims issue, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Accordingly, no limitation, element, property, feature, advantage, or attribute that is not expressly recited in a claim should in any way limit the scope of such claim. Accordingly, the specification and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A method for generating a reconstructed image using a set of two or more local backward shaping functions and a set of two or more local forward shaping functions, the method comprising: receiving an input reshaped image in a first dynamic range (156); initializing (305) a first backward mapping index BMI array, wherein each element of the first BMI array corresponds to a pixel of the input shaped image and indicates an index of a local backward shaping function in the set of two or more local backward shaping functions, wherein the local backward shaping function maps a pixel value from the first dynamic range to a pixel value in a second dynamic range; and performing one or more iterations to generate an output reconstructed image in the second dynamic range, wherein the iterations comprise: i) generating (310) a first reconstructed image in the second dynamic range by applying the set of two or more local backward shaping functions to the input shaped image according to the first BMI array; ii) generating (315) a first FMI array using the first reconstructed image as input to a forward mapping index FMI generation function, wherein each element of the first FMI array corresponds to a pixel in the first reconstructed image and indicates an index of a local forward shaping function in the set of two or more local forward shaping functions, wherein the local forward shaping function maps pixel values ​​from the second dynamic range to pixel values ​​in the first dynamic range; iii) generating a mapping index error according to a first error metric based on a difference between elements of the first BMI array and the first FMI array corresponding to the same pixel in both the input shaped image and the first reconstructed image; iv) generating (325) a second BMI array based on the first BMI array and the mapping index errors; v) generating (330) a second reconstructed image in the second dynamic range by applying the set of two or more local backward shaping functions to the input shaped image according to the second BMI array; vi) generating (335) a second FMI array using the second reconstructed image as input to the FMI generation function; vii) generating (340) an estimated shaped image in the first dynamic range by applying the set of two or more local forward shaping functions to the second reconstructed image according to the second FMI array; viii) generating a shaped image error according to a second error metric, said second error metric being based on a difference between said estimated shaped image and said input shaped image; and ix) testing whether the combination of the shaped image error and the mapping index error meets a convergence criterion: If the convergence criterion is met, terminating the iteration and outputting the second reconstructed image as the output reconstructed image; otherwise: replacing (360) elements of the BMI array based on elements of the second BMI array and the shaped image error; and Perform another iteration.

2. The method of claim 1, wherein: All elements of the first BMI array are initialized to constant values.

3. The method of claim 2, wherein: For a set of L local backward shaping functions, all elements of the first BMI array are initialized to L / 2.

4. The method according to any one of claims 1 to 3, wherein: For a pixel value in the first reconstructed image, the FMI generation function assigns a value to a corresponding element of the FMI array according to brightness characteristics of one or more pixels surrounding the pixel value in the first reconstructed image.

5. The method according to any one of claims 1 to 3, wherein: The FMI generation function includes calculating where, for the kth iteration, t represents the time-dependent index, represents the i-th FMI array value, represents the i-th pixel value of the first reconstructed image, include is a function of the ambient light brightness value, and Including The output of the function is limited to a quantizer between 0 and L-1, where L represents the number of local forward shaping functions.

6. The method of claim 5, wherein: G1() includes a low-pass filter or a Gaussian blur filter.

7. The method according to any one of claims 1 to 3, wherein: The first error metric is the mean absolute error or the mean square error of the differences between elements of the first BMI array and the first FMI array corresponding to the same pixels in both the input reshaped image and the first reconstructed image.

8. The method according to any one of claims 1 to 3, wherein: The second error metric is the mean absolute error or the mean square error of the differences between corresponding pixels of the estimated reshaped image and the input reshaped image.

9. The method according to any one of claims 1 to 3, wherein: At the kth iteration, generate the elements of the second BMI array Including calculation in, represents the elements of the first BMI array, represents the mapping index error, α1 represents the scaling factor, and h() represents the rounding function.

10. The method according to any one of claims 1 to 3, wherein: At the kth iteration, the convergence criterion comprises calculating a weighted sum of the mapping index error and the shaped image error and testing whether the weighted sum is below an iteration threshold.

11. The method of any one of claims 1 to 3, further comprising generating the set of two or more local forward shaping functions by: Access the global forward shaping function; Generate property values ​​for the global forward shaping function, where The property value comprises one or more of: a non-flat region of the function in the second dynamic range, a midpoint of the non-flat region of the function in the second dynamic range, and a flat region of shadows and highlights in the second dynamic range; Generate a template forward shaping function based on the global forward shaping function, the attribute value and the non-flat area scaling factor; Generate a template backward shaping function by calculating the inverse function of the template forward shaping function; as well as For a local forward shaping function associated with a target range of luminance values ​​in said second dynamic range, The local forward shaping function is generated by shifting the template forward shaping function by an x-axis shift value, wherein the x-axis shift value is determined so that for the target range of luminance values, corresponding average shaped output values ​​using the local forward shaping function and the global forward shaping function are approximately equal. 12 . A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing the method according to claim 1 with one or more processors.

13. An apparatus for generating a reconstructed image, the apparatus using a set of two or more local backward shaping functions and a set of two or more local forward shaping functions to generate the reconstructed image, the apparatus comprising: processor; as well as A memory storing computer executable instructions configured to cause the processor to perform the method according to any one of claims 1 to 11.

14. A computer program product comprising computer executable instructions for performing the method according to any one of claims 1 to 11 with one or more processors.

Images