Block-based content adaptive reshaping for HDR images

DE602017086341T2Active Publication Date: 2024-11-27DOLBY LABORATORIES LICENSING CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE602017086341
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2016-05-10
Filing Date
2017-02-02
Publication Date
2024-11-27
Estimated Expiration
2037-02-02

AI Technical Summary

Technical Problem

Current technologies face challenges in efficiently reshaping high dynamic range (HDR) images to accommodate lower bit depths for compression and display on conventional devices, which limits the ability to effectively distribute and display HDR content.

Method used

The implementation of block-based, content-adaptive reshaping techniques that divide images into blocks, compute their complexity using standard deviation, and apply a codeword mapping function to allocate minimal bit depths, allowing for efficient compression and subsequent restoration of HDR images.

Benefits of technology

This approach enables the efficient encoding and decoding of HDR content within existing infrastructure, preserving image quality and supporting dynamic ranges from 0.001 to 10,000 nits, while minimizing artifacts and ensuring perceptually lossless representation.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of the U.S. Provisional Patent Application Nos. 62 / 290,399, filed on February 2, 2016, and 62 / 334,099, filed May 10, 2016.TECHNOLOGY

[0002] The present invention relates generally to images. More particularly, an embodiment of the present invention relates to block-based content-adaptive reshaping of images with high dynamic range.BACKGROUND

[0003] As used herein, the term `dynamic range' (DR) may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest grays (blacks) to brightest whites (highlights). In this sense, DR relates to a 'scene-referred' intensity. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth. In this sense, DR relates to a 'display-referred' intensity. Unless a particular sense is explicitly specified to have particular significance at any point in the description herein, it should be inferred that the term may be used in either sense, e.g. interchangeably.

[0004] As used herein, the term high dynamic range (HDR) relates to a DR breadth that spans the some 14-15 orders of magnitude of the human visual system (HVS). In practice, the DR over which a human may simultaneously perceive an extensive breadth in intensity range may be somewhat truncated, in relation to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may individually or interchangeably relate to the DR that is perceivable within a scene or image by a human visual system (HVS) that includes eye movements, allowing for some light adaptation changes across the scene or image. As used herein, EDR may relate to a DR that spans 5 to 6 orders of magnitude. Thus while perhaps somewhat narrower in relation to true scene referred HDR, EDR nonetheless represents a wide DR breadth and may also be referred to as HDR.

[0005] In practice, images comprise one or more color components (e.g., luma Y and chroma Cb and Cr) wherein each color component is represented by a precision of n-bits per pixel (e.g., n=8). Using linear luminance coding, images where n ≤ 8 (e.g., color 24-bit JPEG images) are considered images of standard dynamic range, while images where n > 8 may be considered images of enhanced dynamic range. EDR and HDR images may also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0006] A reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between color values (e.g., luminance) of an input video signal to output screen color values (e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT. 1886, "Reference electro-optical transfer function for flat panel displays used in HDTV studio production," (03 / 2011), which is included herein by reference in its entity, defines the reference EOTF for flat panel displays based on measured characteristics of the Cathode Ray Tube (CRT). Given a video stream, information about its EOTF is typically embedded in the bit stream as metadata. As used herein, the term "metadata" relates to any auxiliary information that is transmitted as part of the coded bitstream and assists a decoder to render a decoded image. Such metadata may include, but are not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as those described herein.

[0007] Most consumer desktop displays currently support luminance of 200 to 300 cd / m 2< or nits. Most consumer HDTVs range from 300 to 500 nits with new models reaching 1000 nits (cd / m 2< ). Such conventional displays thus typify a lower dynamic range (LDR), also referred to as a standard dynamic range (SDR), in relation to HDR or EDR. As the availability of HDR content grows due to advances in both capture equipment (e.g., cameras) and HDR displays (e.g., the PRM-4200 professional reference monitor from Dolby Laboratories), HDR content may be color graded and displayed on HDR displays that support higher dynamic ranges (e.g., from 1,000 nits to 5,000 nits or more). Such displays may be defined using alternative EOTFs that support high luminance capability (e.g., 0 to 10,000 nits). An example of such an EOTF is defined in SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays," which is incorporated herein by reference in its entirety. In general, without limitation, the methods of the present disclosure relate to any dynamic range higher than SDR.

[0008] As used herein, the term "forward reshaping" denotes the process of mapping (or quantizing) an HDR image from its original bit depth to an image of a lower or the same bit depth to allow compressing the image using existing coding standards or devices. In a receiver, after decompressing the reshaped signal, the receiver may apply an inverse reshaping function to restore the signal to its original high dynamic range. As appreciated by the inventors here, improved techniques for image reshaping of high-dynamic range images are desired.

[0009] WO 2014 / 204865 A1 discloses an encoder that receives an input enhanced dynamic range (EDR) image to be coded in a layered representation. Input images may be gamma-coded or perceptually-coded using a bit-depth format not supported by one or more video encoders. The input image is remapped to one or more quantized layers to generate output code words suitable for compression using the available video encoders. Algorithms to determine optimum function parameters for linear and non-linear mapping functions are presented. Given a mapping function, the reverse mapping function may be transmitted to a decoder as a look-up table or it may be approximated using a piecewise polynomial approximation. A polynomial approximation technique for representing reverse-mapping functions and chromaticity translation schemes to reduce color shifts are also presented.

[0010] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated.SUMMARY OF THE INVENTION

[0011] The present invention is defined by the appended independent claims. The dependent claims concern optional features of some embodiments of the present invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] An embodiment of the present invention is illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which: FIG. 1A depicts an example process for a video delivery pipeline; FIG. 1B depicts an example process for data compression using content-adaptive quantization or reshaping according to an embodiment of this invention; FIG. 2 depicts an example process for block-based, content-adaptive reshaping according to an embodiment of this invention; FIG. 3 depicts an example of a mapping function which maps average standard deviation values to required signal bit depth according to an embodiment of this invention; FIG. 4 depicts examples of computed normalized codeword allocations according to embodiments of this invention; and FIG. 5 depicts an example of a forward reshaping function computed according to an embodiment of this invention. DESCRIPTION OF EXAMPLE EMBODIMENTS

[0013] Block-based, content-adaptive techniques for image reshaping of high dynamic range (HDR) images are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present invention.OVERVIEW

[0014] Example embodiments described herein relate to block-based image reshaping of HDR images. A processor for signal reshaping receives an input image with an input bit depth. The image is divided in to blocks and a metric of complexity of each block (such as, its standard deviation) is computed. The input codewords are divided into codeword bins and each bin is assigned a metric of complexity value. For each bin, a metric of complexity to bit-depth function is applied to the bin values to generate minimal bit depth values for each codeword bin. An output codeword mapping function is generated based on the input bit depth, a target bit depth, and the minimal bit depth values. The codeword mapping function is applied to the input image to generate an output image in the target bit depth.

[0015] The measure of complexity comprises computing the standard deviation of image blocks.EXAMPLE VIDEO DELIVERY PROCESSING PIPELINE

[0016] FIG. 1A depicts an example process of a conventional video delivery pipeline (100) showing various stages from video capture to video content display. A sequence of video frames (102) is captured or generated using image generation block (105). Video frames (102) may be digitally captured (e.g. by a digital camera) or generated by a computer (e.g. using computer animation) to provide video data (107). Alternatively, video frames (102) may be captured on film by a film camera. The film is converted to a digital format to provide video data (107). In a production phase (110), video data (107) is edited to provide a video production stream (112).

[0017] The video data of production stream (112) is then provided to a processor at block (115) for post-production editing. Block (115) post-production editing may include adjusting or modifying colors or brightness in particular areas of an image to enhance the image quality or achieve a particular appearance for the image in accordance with the video creator's creative intent. This is sometimes called "color timing" or "color grading." Other editing (e.g. scene selection and sequencing, image cropping, addition of computer-generated visual special effects, etc.) may be performed at block (115) to yield a final version (117) of the production for distribution. During post-production editing (115), video images are viewed on a reference display (125).

[0018] Following post-production (115), video data of final production (117) may be delivered to encoding block (120) for delivering downstream to decoding and playback devices such as television sets, set-top boxes, movie theaters, and the like. In some embodiments, coding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate coded bit stream (122). In a receiver, the coded bit stream (122) is decoded by decoding unit (130) to generate a decoded signal (132) representing an identical or close approximation of signal (117). The receiver may be attached to a target display (140) which may have completely different characteristics than the reference display (125). In that case, a display management block (135) may be used to map the dynamic range of decoded signal (132) to the characteristics of the target display (140) by generating display-mapped signal (137).Signal Reshaping

[0019] Currently, most digital interfaces for video delivery, such as the Serial Digital Interface (SDI) are limited to 12 bits per pixel per component. Furthermore, most compression standards, such as H.264 (or AVC) and H.265 (or HEVC), are limited to 10-bits per pixel per component. Therefore efficient encoding and / or quantization is required to support HDR content, with dynamic range from approximately 0.001 to 10,000 cd / m 2< (or nits), within existing infrastructures and compression standards.

[0020] The term "PQ" as used herein refers to perceptual luminance amplitude quantization. The human visual system responds to increasing light levels in a very non-linear way. A human's ability to see a stimulus is affected by the luminance of that stimulus, the size of the stimulus, the spatial frequencies making up the stimulus, and the luminance level that the eyes have adapted to at the particular moment one is viewing the stimulus. In a preferred embodiment, a perceptual quantizer function maps linear input gray levels to output gray levels that better match the contrast sensitivity thresholds in the human visual system. An example PQ mapping function is described in SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays," which is incorporated herein by reference in its entirety, where given a fixed stimulus size, for every luminance level (i.e., the stimulus level), a minimum visible contrast step at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to HVS models). Compared to the traditional gamma curve, which represents the response curve of a physical cathode ray tube (CRT) device and coincidently may have a very rough similarity to the way the human visual system responds, a PQ curve imitates the true visual response of the human visual system using a relatively simple functional model.

[0021] For example, under SMPTE ST 2084, at 1 cd / m 2< , one 12-bit code value corresponds to a relative change of approximately 0.0048 cd / m 2< ; however, at 1,000 cd / m 2< , one 12-bit code value corresponds to a relative change of approximately 2.24 cd / m 2< . This non-linear quantization is needed to accommodate for the non-linear contrast sensitivity of the human visual system (HVS).

[0022] Another example of a perceptually-quantized EOTF is presented in "Chromaticity based color signals for wide color gamut and high dynamic range," by J. Stessen et al., ISO / IEC JTC1 / SC29 / WG11 MPEG2014 / M35065, Oct. 2014, which is incorporated herein by reference in its entirety.

[0023] Contrast sensitivity of the HVS does not only depend on luminance but also on masking characteristics of the image content (most particularly noise and texture), as well as the adaptation state of the HVS. In other words, depending on the noise level or the texture characteristics of an image, image content can be quantized with larger quantization steps than those predicted by PQ or gamma quantizers, because texture and noise mask quantization artifacts. The PQ quantization describes the best the HVS can do, which occurs when there is no noise or masking in the image. However, for many images (frames of a video), there is significant masking.

[0024] In addition to noise and texture masking, other characteristics of visual behavior, such as optical flare and local adaptation may also be taken into consideration to increase the level of quantization and allow representing HDR images at 10-bits or lower per color component. As used herein, the terms "Content-Adaptive PQ" or "Content-adaptive reshaping" denote methods to adaptively adjust the perceptually quantization of images based on their content.

[0025] FIG. 1B depicts an example process for content-adaptive reshaping according to an embodiment. Given input frames (117), a forward reshaping block (150) analyzes the input and the coding constrains and generates codeword mapping functions which map input frames (117) to re-quantized output frames (152). For example, input (117) may be gamma-coded or PQ-coded according to certain EOTF. In some embodiments, information about the reshaping process may be communicated to downstream devices (such as decoders) using metadata. Following coding (120) and decoding (130), decoded frames (132) may be processed by a backward reshaping function (160), which converts the re-quantized frames (132) back to the original EOTF domain (e.g., gamma or PQ), for further downstream processing, such as the display management process (135) discussed earlier. In some embodiments, the backward or inverse reshaping function (160) may be integrated with a de-quantizer in decoder (130), e.g., as part of the de-quantizer in an AVC or HEVC video decoder.

[0026] An example of a signal reshaping function was presented in U.S. Provisional Patent Application Ser. No. 62 / 126,925, filed on March 2, 2015, "Content-adaptive perceptual quantizer for high dynamic range images," by J. Froehlich et al., which is incorporated herein by reference in its entirety, to be referred to as the `925 Application, or the CAQ method. According to the CAQ method, the reshaping function is computed according to a noise mask histogram of the input image. Alternative, block-based reshaping techniques are described next.Block-based Reshaping

[0027] FIG. 2 depicts an example process for block-based content-adaptive reshaping according to an embodiment of this invention. As depicted in FIG. 2, given a sequence of images (e.g., video frames) (117), in block (205), each image is divided into non overlapping blocks and the standard deviation (σ) for pixel values in each block is computed. In step (210), input codewords are divided into bins and the average standard deviation for each bin is computed. Block (215) computes the number of minimum bits required for each bin. Based on this information, in block (220), output codewords are assigned to each bin. Finally, in (225), a forward reshaping function is constructed based on the output codeword assignment. Each of these steps is described in more detail next.

[0028] In an embodiment, without limitation, let I j (p) denote the p-th pixel of a color component under quantization (e.g., luminance) in the j-th frame in the input sequence (117), normalized to [0 1). Let v Lj and v Hj denote the minimum and maximum pixel values in this frame, or ν L j = min I j p , ν H j = max I j p .

[0029] Frame I j is divided into rectangular, non-overlapping blocks (e.g., into n × n blocks, where n = 16). Denote the set of pixels within the k-th block as θ j,k = {p | p ∈ block(k)}. Then, the mean and standard deviation of the pixel values within the k-th, n × n, block may be computed as: mean j , k = ∑ p ∈ θ j , k I j p n 2 , and σ j , k = std j , k = ∑ p ∈ θ j , k I j p n 2 2 − mean j , k 2 .

[0030] All pixels in the k-th block of the j-th frame are associated with the same standard deviation, std j , k . For all pixels in the j-th frame, let H j (p) denote the standard deviation associated with pixel p in the j-th frame. In the process of designing a reshaping function for frame j some pixels in H j (p) may be excluded. These pixels may include: Pixels of a letterbox area within the input image (that is, dark areas added to a picture so its size matches a certain aspect ratio). Pixels in letterbox areas have constant values, hence zero standard deviation. Pixels at the right- and bottom-ends of the frame. If the size of the picture does not accommodate an integer number of the n × n blocks, then pixel areas at the right- and bottom-ends of the frame, where their corresponding blocks have a smaller size, may be excluded. Let Ωj denote the set of valid (not excluded) pixel indices for the j-th frame. Let i be an index inside Ωj. Then, Hji,i∈Ωj denotes, for each valid pixel in the j-th frame, the standard deviation associated with pixel i in the j-th frame. Experimental results indicate that standard deviation maps provide a good, and simple to compute, representation of the underlying frequency content in each frame and thus can be used to decide how to allocate different bit depths to different regions in each frame. However, the methods described herein are equally applicable using alternative metrics of the complexity of an image block, such as their variance, their entropy, frequency response, and the like.

[0031] Let B I denote the bit-depth of the input image (117) (e.g., B I = 16) and let K = 2 BI< , then the dynamic range 0 to K-1 may be partitioned into AT bins of equal number (W) of input codewords, that is W = KIM. Let b j,m . denote the average standard deviation of all valid pixels with input values belonging to the m-th bin, where m = 0, 1, ..., M-1. That is, for I j (i),i ∈ Ω j , if Ψ j , m = i | m M ≤ I j i < m + 1 M , then b j , m = mean H j i | i ∈ Ψ j , m . Note that for specific images there might exist some codeword bins with no pixels inside them. These bins may be marked so that they can be properly accounted for and / or adjusted in the next step.

[0032] Signal reshaping functions may be generated at the frame level or at the scene level. As used herein, the terms 'scene' or 'shot' for a video sequence may relate to a series of consecutive frames in the video signal sharing similar color and dynamic range characteristics. Because of the consecutive nature of video prediction in most video compression formats, it may be preferable to adjust the quantization parameters only at boundaries that match the typical boundaries of a video encoder, such as scene changes or a new group of pictures (GOP). Thus, given a scene with F frames and frame-based b j,m values, a scene-based measure of the standard deviation in each bin may be derived as b m = mean b j , m | j = 0,1 , … , F − 1 . For bins with no valid pixel values within a whole scene, in an embodiment, their standard deviation is set to a large number (say, 1 or higher) to guarantee that that no codewords are assigned to them. In another embodiment, let v L and v H denote the minimum and maximum pixel values within a scene, that is: ν L = min ν Lj , ν H = max ν Hj , then for empty bins, the following process may be employed: If the luminance values in a bin are lower than v L , then assign to this bin the same average standard deviation value as the closest bin to the right Else, if the luminance values in a bin are higher than v H , then assign to this bin the same average standard deviation value as the closest bin to the left Else, interpolate the bin value from its neighbor bins Minimum bit-depth per bin

[0033] Given the computed b m levels for a scene, the next step would be to determine the number of bits that need to be allocated for each bin. In an embodiment, such a mapping may be determined based on experimental user study results. For example, in one such study, users were shown to evaluate test images constructed as follows: a) A selection of 16-bit HDR frames was converted from the original RGB space to the color space to be used during the compression step (120) b) The color-transformed images were truncated (starting from the least significant bit (LSB)) to bit depths ranging from 7 bits to 11 bits c) The LSB-truncated images were converted back to the RGB color space d) Then, for each image, the minimum required bit-depth for which the output image does not exhibit any artifacts, such as false contouring, was determined

[0034] FIG. 3 depicts an example of such results for a set of video data coded according to ST 2084. In FIG. 3, curve (305) represents the worst case results for the tested images, thus representing the worst-case bit-depth requirements as a function of computed average standard deviation. From FIG. 3, for example, for an average standard deviation of 2 -8< , the required bit depth is about 9.75 bits. As expected, FIG. 3 indicates that image regions with higher levels of complexity can be coded at smaller bit depths. Alternatively, the smoother the image, the more bit depth is needed for an accurate and perceptually lossless representation.

[0035] Given a set of average standard deviation values, std i , i = 0, 1, 2, ..., N-1, where std i ≤ std i+1 , let Q̃ i denote the corresponding computed minimum bit depth. Based on these samples, one may derive a standard deviation to bit-depth allocation function f B (), so that Q m = f B b m ,

[0036] In an embodiment, f B () can be a piece-wise linear function computed using linear interpolation. For example, for std n ≤ b m ≤ std n + 1 Q m = Q ˜ n − Q ˜ n − Q ˜ n + 1 b m − std n std n + 1 − std n . However, other interpolation techniques known in the art, such as bilinear interpolation, spline interpolation, and the like, could also be used.

[0037] In an embodiment, it may be more convenient to perform codeword mapping (220) based on the number of required codewords within each bin instead of using the bit depth data directly. This is examined in the next section.Codeword Mapping Generation

[0038] Let B T denote the target bit-depth for the re-quantized signal (152) (e.g., B T = 10 bits / pixel per color component), then the output will be mapped using 2 BT< codewords. In an embodiment, the range of codewords is normalized to be in [0, 1), hence let D m = 2 Q m 2 B T / 2 B I . denote the number of required normalized codewords per bin m. For example, if Q m = 9, B I = 16 and B T = 10, then D m = 2 -17< .

[0039] Let d i = D m for m − 1 W ≤ 1 < mW , denote the number of normalized codewords per input i ∈ (0, 2 BI< - 1), then d i can be considered a lower bound for the number of required codewords per input codeword. The total number of normalized codewords for all output codewords, D, is bounded by 1, or D = ∑ i = ν L ν H d i ≤ 1 .

[0040] Let U = 1- D denote the number of unused normalized codewords after applying equation (12). These unused codewords need to be reassigned to output codewords according to a given criterion. In an embodiment, using a constant allocation scheme, for input v L ≤ i ≤ v H , the same amount of additional codewords are assigned based on the existing distribution, or d ˜ i = d i + U ν H − ν L , where d̃ i denotes the updated number of normalized codeword values. Additional reallocation techniques were also presented in the `925 Application. In this invention, another codeword allocation scheme based on computed entropy is also proposed as follows.

[0041] Consider a region Ω i surrounding the i-th pixel in the j-th frame (e.g., a 5 x 5 area with the i-th pixel in the center), then the entropy of the i-th pixel may be computed as E j i = ∑ k ∈ Ω i − p k log p k , where p(k) denotes an empirical probability for the occurrence of pixel value k. For example: p k = y = k | y ∈ Ω i Ω i , where |X| denotes the number of elements in set X.

[0042] Without loss of generality, in an embodiment, let E j NL i = E j i 1 / 2 represent a non-linear mapping of the entropy values so that more codewords are allocating to regions of higher complexity while smoother areas, with low entropy, are also guaranteed not to get too few codewords, thus preventing coding artifacts. After normalizing with the maximum entropy value within each frame, the final entropy of the j-th frame is given by E j NL i = E j NL i / max i E j NL i .

[0043] Let E j,m denote the mean entropy within each codeword bin m in the j-th frame, and let E m = mean(E j,m | j = 0, 1, ... , F - 1) denote the average entropy in each codeword bin within a scene, then, in an embodiment, for v L ≤ i ≤ v H , unused codewords are allocated based on d ˜ i = d i + a U ν H − ν L + 1 − a e i S e , where a is a constant between 0 and 1 (e.g., a = 0. 1) that determines what proportion of the unused codewords is assigned using a constant allocation scheme, e.g., as in equation (14), and what proportion takes into consideration the computed entropy values, e i = E m for m − 1 W ≤ i < mW , and S e = ∑ i e i for ν L ≤ i ≤ ν H .

[0044] In another embodiment, let u i = U ⋅ i − ν L ν H − ν L α , for ν L ≤ i ≤ ν H , where a is a constant (e.g. α = 1.4), . Then, unused codewords are allocated based on d ˜ i = d i + u i − u i − 1 , for ν L + 1 ≤ i ≤ ν H . Experimental results have indicated that this allocation scheme tends to preserve better film grain, camera noise, or details at certain high bit rates.

[0045] In an embodiment, the allocation curves may also be smoothed using a low-pass filter or a moving average filter to ensure the curve is smooth and the inverse reshaping function is easy to approximate using a piece-wise polynomial. For example, in an embodiment s i ˜ = ∑ k = − W W a k ⋅ d ˜ i + k , where a k = 1 2 W + 1 .

[0046] In an embodiment, no codewords are assigned to values outside the (v L , v H ) range, or s l ˜ = 0 for i < ν L and i > ν H .

[0047] Furthermore, in case the sum of smoothed values exceeds 1, the codeword assignment needs to be normalized. For example, if S = ∑ i = ν L ν H s i ˜ > 1 , then s i = s i ˜ S .

[0048] Given the final allocation of codewords in each codeword bin, a forward reshaping function may be generated as FL i = ∑ k = 0 i s k . Given FL(i), then, in the j-th frame, for a given input pixel I j (p) = i (i ∈ (0, 2 BI< -1)) the final, normalized, re-quantized pixel value s jp may be expressed as: s jp = { 0 , if i < v L 1 , if i > v H FL i , if v L ≤ i ≤ v H .

[0049] In an embodiment, FL(i) values may be stored in a pre-computed look-up table (LUT). Alternatively, normalized s jp values may also be mapped to de-normalized values in the range 0 to 2 BT< - 1.

[0050] FIG. 4 depicts example plots for codeword allocation according to embodiments. Plot (610) shows the original allocation based on the computed lower bit-depth values (e.g., using equation (12)) for a scene, for B I = 16 (input codewords range from 0 to 65,535). FIG. 4 also shows the number of normalized codewords after the final allocation, using both smoothed (620) and unsmoothed (615) curves. FIG. 5, depicts an example look-up table for reshaping 16-bit input data into 10-bit output data according to embodiments of this invention.

[0051] People skilled in the art will appreciate that while the methods discussed herein are based on a block-based measure of standard deviation, other measures of block complexity, such as their variance, entropy (for example, as computed in equation (15)), frequency transform components, and the like, may also be applied.Backward or Inverse Reshaping

[0052] In some embodiments, backward reshaping (160) may be applied to reverse the effects of forward reshaping (150). In an embodiment, a look-up table for inverse quantization may be constructed as follows: a) For each codeword in the quantized domain (s c ), identify all input codewords (v i ) for which FL(v i ) = s c . Let this group be denoted as ω(s c ) = {v i | FL(v i ) = s c }; then b) Construct the backward reshaping function (BL(s c )) as a function of ω(s c ) For example, in an embodiment, without limitation, BL(s c ) may be constructed as the average of all codewords that belong to ω(s c ), or if |ω(s c )| > 0, then BLsc=∑i∈ωscνiωsc, where |ω(s c )| denotes the number of elements in the set ω(s c ). If |ω(s c )|=0 for any s c values, in an embodiment, these values may be interpolated from its neighbor non-zero values. In another embodiment, backward reshaping may be defined as min(ω(s c )).EXAMPLE COMPUTER SYSTEM IMPLEMENTATION

[0053] Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or apparatus that includes one or more of such systems, devices or components. The computer and / or IC may perform, control, or execute instructions relating to the block-based reshaping of images with enhanced dynamic range, such as those described herein. The computer and / or IC may compute any of a variety of parameters or values that relate to the block-based reshaping processes described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

[0054] Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder or the like may implement methods related to block-based reshaping of HDR images as described above by executing software instructions in a program memory accessible to the processors. The invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0055] Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

[0056] Example embodiments that relate to the efficient block-based reshaping of HDR images are thus described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including their equivalents. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for adaptive image reshaping of a high dynamic range (HDR) input image in an input bit depth to generate a corresponding output image in a target bit depth lower than or equal to the input bit depth with a processor, the method comprising: accessing with a processor the HDR input image in the input bit depth; dividing the HDR input image into a plurality of non-overlapping blocks and computing, for each of the plurality of blocks, a block measure of complexity of their pixel values, wherein computing the respective block measure of complexity of pixel values comprises computing the standard deviation of valid pixel values in the corresponding block and assigning the respective block measure of complexity of pixel values to each valid pixel of the corresponding block, wherein valid pixels are all pixels of the block excluding those pixels in edge areas of the HDR input image that are deemed not essential in image reshaping; dividing a range of input codewords related to the entire HDR input image into a plurality of codeword bins; for one or more of the codeword bins: computing a bin measure of complexity value, wherein computing the bin measure of complexity value comprises computing the average value of the block measures of complexity associated with those pixels of the HDR input image having pixel values belonging to the respective codeword bin; generating a bin measure of complexity to bit-depth function related to the entire HDR input image according to results from a perceptual user study, wherein the bin measure of complexity to bit-depth function maps a given bin measure of complexity value to a corresponding minimal bit-depth value representing the minimum bit depth required for perceptually lossless image reshaping; generating a codeword mapping function based on the input bit depth, the minimal bit depth values, and a target bit depth, wherein the codeword mapping function maps input codewords in the input bit depth to output codewords in the target bit depth, wherein generating the codeword mapping function further comprises: for each codeword bin: determining a lower bound for a normalized number of required codewords in the target bit depth based on the minimal bit depth values, the input bit depth and the target bit depth, wherein the normalized number of required codewords in the target bit depth is the proportion of the number of required codewords to the number of codewords available in the target bit depth; determining unused codewords as the remaining codewords in the target bit depth that are not part of the required codewords in the target bit depth; allocating the unused codewords to each bin image according to an allocation scheme, thereby increasing the normalized numbers of required codewords to updated normalized numbers of required codewords; generating the codeword mapping function based on computing a cumulative sum of the updated normalized numbers of required codewords; and applying the codeword mapping function to the input codewords of the input image in the input bit depth to generate the output codewords of a corresponding output image in the target bit depth.

2. The method of claim 1, wherein dividing a range of input codewords into a plurality of codeword bins comprises dividing the range of K input codewords into M codeword bins, each having an equal number W of input codewords, wherein BI denotes the input bit depth, K = 2Bi, and W = K / M.

3. The method of claim 1 or claim 2, wherein the standard deviation is computed with respect to the mean value of the pixel values in the block.

4. The method of any of claims 1-3, wherein the HDR input image is encoded according to gamma encoding or SMPTE ST 2084.

5. The method of any of claims 1-4, wherein generating the bin measure of complexity to bit-depth function according to results of the perceptual user study comprises: accessing a plurality of HDR test images in the input bit depth; for each test image in the plurality of HDR test images: converting the test image from its original color space to a second color space; truncating the image in the second color space to generate truncated images at bit depths lower than the input bit depth; converting the truncated images to the original color space to generate reconstructed images; and determining, for each of a plurality of different bin measure of complexity values, the corresponding minimal bit-depth value for which one of the truncated images in the original color space best matches the test image.

6. The method of any of claims 1-5, further comprising, filtering the updated normalized numbers of required codewords by a low-pass filter before generating the codeword mapping function.

7. The method of any of claims 1-6, wherein the allocation scheme comprises a constant offset allocation scheme, where for the i-th input pixel value d ˜ i = d i + U ν H − ν L , wherein d̃i denotes the updated normalized number of codeword values, di denotes the normalized number of required codewords, if D denotes the sum of di values, then U=1-D denotes the normalized number of unused codewords, vH denotes a maximum input pixel value, and vL denotes a minimum input pixel value.

8. The method of any of claims 1-6, wherein the allocation scheme comprises computing d ˜ i = d i + u i − u i − 1 , for ν L + 1 ≤ i ≤ ν H . wherein d̃i denotes the updated normalized number of codeword values, di denotes the normalized number of required codewords, if D denotes the sum of di values, then U=1-D denotes the normalized number of unused codewords, vH denotes a maximum input pixel value, vL denotes a minimum input pixel value, and u i = U ⋅ i − ν L ν H − ν L α , for ν L ≤ i ≤ ν H , where a is a constant.

9. The method of any of claims 1-6, wherein the allocation scheme comprises an entropy-based allocation scheme, where for the i-th input pixel value d ˜ i = d i + a U ν H − ν L + 1 − a e i S e , wherein d̃i denotes the updated normalized number of codeword values, di denotes the normalized number of required codewords, if D denotes the sum of di values, then U=1-D denotes the normalized number of unused codewords, a is a scalar between 0 and 1, vH denotes a maximum input pixel value and vL denotes a minimum input pixel value, ei denotes a measure of local entropy for the i-th codeword, and Se = Σiei for vL ≤ i ≤ vH.

10. The method of any of claims 1-9, wherein generating the codeword mapping function for an input pixel value i comprises computing FL i = ∑ k = 0 i s k , wherein sk values are derived based on the updated normalized number of codeword values.

11. The method of claim 10, wherein generating the output codewords of the corresponding output image comprises computing: s p = { 0 , if i < v L 1 , if i > v H FL i , if v L ≤ i ≤ v H , wherein for a pixel Ip = i in the input image, sp denotes the corresponding, normalized, quantized output pixel, vH denotes a maximum input pixel value, and vL denotes a minimum input pixel value.

12. The method of any of claims 1-11, wherein the bin measures of complexity values are computed based on block complexity values computed across multiple consecutive frames in a video scene that includes the HDR input image.

13. An apparatus comprising a processor and configured to perform the method recited in any of claims 1-12.

14. A computer-readable storage medium having stored thereon computer-executable instruction for executing the method recited in any of claims 1-12.