Image enhancement via global and local reshaping

Abstract

A first re-shaping mapping is performed on the first image represented in the first domain to generate a second image represented in a second domain, the first domain having a first dynamic range different from a second dynamic range of the second domain. A second re-shaping mapping is performed on the second image represented in the second domain to generate a third image represented in the first domain, the third image perceptually different from the first image in at least one of global contrast, global saturation, local contrast, local saturation, etc. A display image is derived from the third image and rendered on a display device.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 142,270, both filed on January 27, 2021, and European Patent Application No. 21153722.0, each of which is incorporated by reference in its entirety.

[0002] technology The present disclosure relates generally to image processing operations. More particularly, certain embodiments of the present disclosure relate to video codecs. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest black (shadows) to brightest white (highlights). In this sense, DR relates to "scene-referred" intensity. DR may also relate to the ability of a display device to fully or approximately render a certain width of intensity range. In this sense, DR relates to "display-referred" intensity. Unless a particular meaning is explicitly specified as having a particular meaning at any point in the description herein, it should be inferred that the term may be used in either sense, e.g., interchangeably.

[0004] As used herein, the term "high dynamic range (HDR)" refers to a DR width that spans about 14-15 orders of magnitude or more of the human visual system (HVS). In practice, the DR where humans can simultaneously perceive a wide range of intensity ranges may be somewhat truncated compared to HDR. As used herein, the terms "enhanced dynamic range (EDR)" or "visual dynamic range (VDR)" refer individually or interchangeably to the DR perceivable in a scene or image by the human visual system (HVS), including eye movements, allowing for some light adaptation changes across the scene or image. As used herein, EDR can refer to a DR that spans 5-6 orders of magnitude. Although perhaps somewhat narrower in relation to true scene-based HDR, EDR still represents a wide DR width and may be referred to as HDR.

[0005] In practice, an image comprises one or more color components of a color space (e.g. luma Y and chroma Cb and Cr), each color component being represented with a precision of n bits per pixel (e.g. n=8). Using non-linear luminance coding (e.g. gamma encoding), images with n≦8 (e.g. color 24-bit JPEG images) are considered standard dynamic range images, while images with n>8 are considered enhanced dynamic range images.

[0006] A reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between the color values ​​(e.g., luminance) of the input video signal and the output screen color values ​​(e.g., screen luminance) produced by the display. For example, “Video Signal Processing: A Reference for Color Space and Gamut,” IEEE Transactions on Image Processing, vol. 13, No. 1, pp. 1171-1175, 2002, which is incorporated herein by reference in its entirety, defines a reference EOTF for flat panel displays. Given a video stream, information about its EOTF can be embedded in the bitstream as (image) metadata. The term “metadata” in this specification relates to any auxiliary information transmitted as part of the encoded bitstream that assists a decoder to render a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein. [Non-Patent Document 1] ITU Rec. ITU-R BT.1886 "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011)

[0007] As used herein, the term "PQ" refers to perceptual luminance amplitude quantization. The human visual system responds highly nonlinearly to increasing light levels. A human's ability to see a stimulus is affected by the luminance of the stimulus, the magnitude of the stimulus, the spatial frequencies that make up the stimulus, and the luminance level to which the eye is adapted at the particular moment the stimulus is viewed. In some embodiments, the perceptual quantization function maps linear input gray levels to output gray levels that better match contrast sensitivity thresholds in the human visual system. Exemplary PQ mapping functions are described in SMPTE, "Perceptual Quantization of Brightness and Brightness," which is incorporated herein by reference in its entirety. Here, for all luminance levels (e.g., stimulus levels), given a fixed stimulus size, the smallest visible contrast increment at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model). [Non-Patent Document 2] SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays"

[0008] 200~1,000cd / m 2 Displays supporting luminance of 1,000 or 1,000 nits represent low dynamic range (LDR), also called standard dynamic range (SDR), as opposed to EDR (or HDR). EDR content may be displayed on EDR displays supporting higher dynamic range (e.g., 1,000 nits to 5,000 nits or more). Such displays may be defined using alternative EOTFs supporting higher luminance capabilities (e.g., 0 to 10,000 nits or more). Examples of such EOTFs are defined in SMPTE 2084 and Non-Patent Document 3. As recognized herein by the inventors, improved techniques for converting input video content data into output video content having high dynamic range, high local contrast, and vivid colors are desired. [Non-Patent Document 3] Rec.ITU-R BT.2100,"Image parameter values ​​for high dynamic range television for use in production and international program exchange,"(06 / 2017)

[0009] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art by mere inclusion in this section. Similarly, it should not be assumed that problems identified with one or more approaches have been recognized in any prior art based on this section, unless otherwise indicated. [Brief description of the drawings]

[0010] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols refer to similar elements and in which:

[0011] [Figure 1] 1 illustrates an example process of a video delivery pipeline. [Figure 2A] 1 illustrates an example architecture for applying global and / or local reshaping operations. [Figure 2B] 1 illustrates an example architecture for generating an output HDR video signal from an input HDR video signal. [Figure 2C] 1 illustrates an example architecture for generating an output HDR video signal from an input HDR video signal. [Figure 2D] 1 illustrates an example architecture for generating an output HDR video signal from an input HDR video signal. [Figure 2E]1 illustrates an example architecture for generating an output HDR video signal from an input HDR video signal. [Figure 2F] 1 illustrates an example architecture for generating an output HDR video signal from an input HDR video signal. [Figure 2G] 1 illustrates an example architecture for generating an output HDR video signal from an input HDR video signal. [Figure 2H] 1 illustrates an example architecture for generating an output HDR video signal from an input HDR video signal. [Figure 2I] 1 illustrates an example architecture for generating an output HDR video signal from an input HDR video signal. [Figure 2J] 1 illustrates an exemplary video application in which a source image is captured as an SDR image by a mobile device at an input SDR video signal. [Figure 2K] 1 illustrates an example flow for using unpaired forward and backward reshape functions to modify the luma and / or chroma of an input video signal. [Figure 2L] 1 illustrates exemplary luma and chroma local reshaping that can be performed on an input image to generate a locally reshaped output image. [Figure 2M] 13 shows an example flow for self-derived single-channel local function generation. [Figure 2N] 1 illustrates an example flow for cross-channel local function generation. [Figure 2O] 1 illustrates an example flow for cross-channel local function generation. [Figure 2P] 1 illustrates an example flow for cross-channel local function generation. [Figure 2Q] 1 illustrates an example flow for local reshape function selection dithering. [Figure 2R] 13 illustrates an example mapping flow for generating coefficients for a chroma local reshape function. [Figure 2S]13 illustrates an example mapping flow for generating coefficients for a chroma local reshape function. [Figure 2T] 13 illustrates an example mapping flow for generating coefficients for a chroma local reshape function.

[0012] [Figure 3A] 13 shows an example indexed luminance global forward reshape function. [Figure 3B] 13 shows an exemplary indexed luma global backward reshaping function. [Figure 3C] 3C and 3F show an example luma codeword mapping using a pair of global forward and backward reshaping functions, each having the same index value. [Figure 3D] 3D and 3E show an example of an exemplary chroma codeword mapping using a pair of global forward and backward reshaping functions, each with the same index value. [Figure 3E] 3D and 3E show an example of an exemplary chroma codeword mapping using a pair of global forward and backward reshaping functions, each with the same index value. [Figure 3F] 3C and 3F show an example luma codeword mapping using a pair of global forward and backward reshaping functions, each having the same index value. [Figure 3G] 1 illustrates an example luma backward function with different index values. [Figure 3H] 4 illustrates an example luma forward function with different index values. [Figure 3I] 4 illustrates an exemplary non-linear function used to achieve saturation variation.

[0013] [Figure 4] An exemplary process flow is shown.

[0014] [Diagram 5]FIG. 1 illustrates a simplified block diagram of an exemplary hardware platform upon which a computer or computing device described herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, burying, or obscuring the present disclosure.

[0016] overview To reshape or transform the input image data into the output image data, forward or backward reshaping techniques described herein can be implemented. Exemplary reshaping or transformation operations can include one or more combinations of a local reshaping operation, a global reshaping operation, a local forward reshaping operation, a global forward reshaping operation, a local backward reshaping operation, a global backward reshaping operation, combinations thereof, and the like.

[0017] Global reshaping refers to a transformation or reshaping operation that applies the same global (forward and / or backward) reshaping function / mapping to every pixel of an input image to produce a corresponding output image -- a reshaped (e.g. forward, backward, circular, etc.) image -- that depicts the same visual semantic content as the input image.

[0018] In contrast to global reshaping, which applies the same reshape function or mapping to all pixels of an input image, local reshaping refers to a transformation or reshaping operation that applies different (forward and / or backward) reshape functions or mappings to different pixels of the input image. Thus, in local reshaping, a first reshape function applied to a first pixel of the input image may be different from a second reshape function applied to a second, different pixel of the input image.

[0019] Exemplary global reshaping operations are described in U.S. Provisional Patent Application No. 62 / 136,402, filed March 20, 2015 (also published as U.S. Patent Application Publication No. 2018 / 0020224 on January 18, 2018), and PCT Application No. PCT / US2019 / 031620, filed May 9, 2019, PCT Application No. PCT / US2019 / 063796, filed November 27, 2019, entitled “Interpolation of Reshaping Functions,” also published on June 11, 2020 as WO2020 / 117603; U.S. Provisional Patent Application No. 63 / 013,063, filed April 21, 2020 by G. M. Su, entitled “Reshaping functions for HDR imaging with continuity and reversibility,” and U.S. Provisional Patent Application No. 63 / 013,807, filed April 22, 2020, by GM. Su and H. Kadu, entitled "Iterative optimization of reshaping functions in single-layer HDR image codec," all of which are incorporated by reference in their entirety as if fully set forth herein. Exemplary local reshaping operations are described in U.S. Provisional Patent Application No. 63 / 086,699, filed October 2, 2020, the entirety of which is incorporated by reference in its entirety as if fully set forth herein.

[0020] The reshaping techniques described herein can be used to implement a unified image enhancement method for selecting, combining, and / or blending various global and / or local reshaping operations to produce a reshaped or reconstructed image that has better image quality compared to the input image used to generate the reshaped or reconstructed image.

[0021] For example, HDR video may provide a significantly better viewing experience than the corresponding SDR video in a TV viewing application, but HDR video may exhibit a much more subdued viewing experience that is comparable or similar to the corresponding SDR video in a mobile device viewing application.

[0022] Under the techniques described herein, HDR images in HDR video can be further enhanced in addition to the existing SDR-HDR dynamic range enhancement. The further enhanced HDR images can appear much more appealing in TV and mobile device viewing applications than the SDR images in the corresponding SDR video. Additionally, optionally or alternatively, even for SDR video as carried in the base layer of an SDR backward-compatible video signal or a layered video signal, the SDR images in the SDR video can be enhanced with better local contrast and / or color saturation to enrich or improve the viewing experience of the base layer.

[0023] As used herein, "circular reshaping" or "circular reshaped" refers to a combination of anterior and posterior reshaping.

[0024] Circular reshaping composed of global reshaping operations can be classified into paired forward / backward reshaping and unpaired forward / backward reshaping. An input image received by a paired forward / backward global reshaping operation can be reconstructed (e.g., perfectly, faithfully, subject to quantization errors, etc.) using an output image generated by a paired forward / backward global reshaping operation. The luminance, tone, and / or saturation of an input image received by an unpaired forward / backward global reshaping operation can be adjusted in an output image generated by an unpaired forward / backward global reshaping operation.

[0025] Local reshaping operations, such as forward, backward and / or circular local reshaping, may not fall into paired / unpaired categories: Regardless of whether a correspondence or relationship exists between a local forward reshaping operation and a local backward reshaping operation, the local forward reshaping operation and the local backward reshaping operation may not be used as a pair to generate a reconstructed image that is identical (e.g., perfectly, faithfully, subject to quantization error, etc.) to the input image used to generate the reconstructed image by the local forward reshaping operation and the local backward reshaping operation.

[0026] Local reshaping operations as described herein, whether paired or unpaired, can be used to increase local contrast ratio and color saturation to achieve better local appearance in the reshaped and / or reconstructed image produced by the local reshaping operation.

[0027] In some operating scenarios, incoming image metadata received along with an input image from an image source (e.g., an upstream encoder, a media content server, a media streaming server, a production studio, etc.) is preserved or substantially undisturbed in the outgoing image metadata of an output video signal that encodes the reshaped or reconstructed image produced by the reshaping operations described herein.

[0028] An unpaired global reshaping operation may excessively change the codeword statistics or distribution in the output image relative to the input image, thereby partially or totally invalidating the incoming image metadata, or at least causing inaccuracies in the outgoing image metadata due to the change in codeword statistics or distribution caused by the unpaired global reshaping operation. In some operating scenarios, a local reshaping operation may be applied in whole or in part instead of the unpaired global reshaping operation to minimize or reduce the change in the codeword statistics / distribution in the output image relative to the input image, thereby maintaining the validity and / or accuracy of the incoming image metadata. As a result, some or all of the incoming image metadata may be preserved or undisturbed in the outgoing image metadata of the output video signal encoded with the output image.

[0029] Depending on the use case, different combinations of global / local reshaping functions / operations to enhance an image will produce an output image that has a better appearance, for example, in terms of contrast ratio, hue and / or saturation, than the input image received by the reshaping functions / operations.

[0030] A variety of methods can be used to generate the reshaping functions as described herein. For example, the local reshaping functions can be generated without limitation by using one or more of the following: (1) self-derivation methods, (2) pre-construction methods, (3) hybrid methods that combine self-derivation and pre-construction methods, etc. Some or all of these methods can be deployed in online / real-time reshaping use cases, offline / non-real-time reshaping use cases. For example, in some operational scenarios, the local reshaping functions can be generated in real-time from a family of global reshaping functions that are trained offline.

[0031] In some operating scenarios, local reshaping may cause false contouring or banding artifacts to be generated. Local reshaping increases the slope or contrast in the mapping function, thus widening the gap between consecutive codewords in the codeword space of available codewords for encoding image content in the image. To mitigate or mask the false contouring or banding artifacts, local reshaping function index dithering may be applied. Additionally, optionally or alternatively, film grain injection may be performed on the reshaped codewords to further mitigate or mask the false contouring or banding artifacts.

[0032] Exemplary embodiments described herein relate to generating an output image from an input image. A first reshaping mapping is performed on a first image represented in a first domain to generate a second image represented in a second domain. The first domain has a first dynamic range that is different from a second dynamic range of the second domain. A second reshaping mapping is performed on the second image represented in the second domain to generate a third image represented in the first domain. The third image is perceptually different from the first image in at least one of global contrast, global saturation, local contrast, local saturation, etc. A display image is derived from the third image to be rendered on a display device.

[0033] Exemplary Video Delivery Processing Pipeline 1 illustrates an example process of a video delivery pipeline (100) showing various stages from video capture / generation to an HDR or SDR display. Exemplary HDR displays may include, but are not limited to, image displays operating in conjunction with a TV, a mobile device, a home theater, etc. Exemplary SDR displays may include, but are not limited to, an SDR TV, a mobile device, a home theater display, a head mounted display device, a wearable display device, etc.

[0034] It should be noted that image processing operations described herein, such as reshaping operations, can be performed either at the encoder / server side (before video compression) or at the decoder / playback side (after video decompression), as well as in video pre-processing systems or blocks that provide input images for video encoding. To support playback-side reshaping operations, the same system configurations as shown in Figure 1 may be used, or different system configurations may be used in addition to those shown in Figure 1. A wide variety of different image metadata formats may be used to convey image metadata in these system configurations other than those used by the processing components as shown in Figure 1.

[0035] Video frames, such as a sequence of successive input (e.g., SDR, HDR, etc.) images 102, can be received by the image generation block 105. These images (102) can be received from a video source, generated by a video pre-processing system / block external or internal to the image generation block (105), or retrieved from a video data store. Some or all of the images (102) can be generated from source images, for example, through video editing or conversion operations (e.g., automatic without human input, manual, automatic with human input, etc.), color grading operations, etc. The source images may be digitally captured (e.g., by a digital camera), generated by converting analog camera pictures captured on film to a digital format, generated by a computer (e.g., using computer animation, image rendering, etc.), etc. The images (102) can be images associated with one or more of a movie release, an archived media program, a media program library, a video recording / clips, a media program, a TV program, user-generated video content, etc.

[0036] The image generation block (105) applies a reshaping operation (e.g., local, global, local and global, etc.) to each input image in a sequence of consecutive input images (102) to generate a respective (e.g., reshaped, reconstructed, enhanced, further enhanced, etc.) output image in a sequence of corresponding consecutive (e.g., reshaped, reconstructed, enhanced, further enhanced, etc.) output images that depict the same visual semantic content as the input image (102) but have the same or a different dynamic range, higher local contrast, sharper colors, etc. compared to the input image (102).

[0037] More specifically, based on the code words present in the input image of the input image (102), the image generation block (105) can select or construct a global reshaping function 142 and / or a local reshaping function 146 that is general or specific to the input image. The image generation block (105) can perform a global reshaping operation based on the global reshaping function (142) and / or a local reshaping operation based on the local reshaping function (146) to reshape or transform the input image into a corresponding output (e.g., reshaped, reconstructed, enhanced, further enhanced, etc.) image that portrays the same visual semantic information as the input image. The image generation block (105) can perform only a global reshaping operation, only a local reshaping operation, or a combination of a global reshaping operation and / or a local reshaping operation, etc., to generate a corresponding output image from the input image. The output image can have the same or a different dynamic range, higher local contrast, vivid colors, etc., compared to the input image.

[0038] In some operating scenarios, some of the reshaping operations may be performed offline to generate pre-constructed reshaping mappings / functions based on training image data such as training HDR images, training SDR images, training HDR-SDR image combinations / pairs, etc. These pre-constructed reshaping mappings / functions may be used or adapted, directly or indirectly, for the purpose of performing dynamic online reshaping operations in the image generation block (105) at run-time.

[0039] In some operating scenarios, the output image generated by the image generation block (105) may be provided to the composer metadata generation block 115 to generate the forward reshaped SDR image 112 as well as to generate image metadata 177 (e.g., composer metadata, backward reshaped mapping, etc.). The image metadata (177) may include composer data for generating a backward reshaped mapping that, when applied to the forward reshaped SDR image, generates a corresponding HDR image that portrays the same visual semantic content as the forward reshaped SDR image. Some or all of the input image (102) may be provided to the composer metadata generation block (115) to facilitate or assist in the forward reshaped operation that generates the forward reshaped SDR image (112) as well as to facilitate or assist in the generation of the image metadata or the backward reshaped mapping therein.

[0040] Exemplary reshaping mappings / operations include, but are not limited to, backward mappings / operations, forward mappings / operations, and / or circular mappings / operations, which may include, but need not be limited to, those specified as one or more of a backward lookup table (BLUT), a forward lookup table (FLUT), backward and forward reshaping functions / curves or polynomial sets, multivariate multiple regression (MMR) coefficients, tensor product B-spline (TPB) coefficients, any combination thereof, etc. An example of an MMR operation is described in U.S. Patent No. 8,811,490, the entire contents of which are incorporated by reference in its entirety as if fully set forth herein. Examples of TPB operation are described in U.S. Provisional Application No. 62 / 908,770, entitled "TENSOR-PRODUCT B-SPLINE PREDICTOR," filed on October 1, 2019 (Attorney Docket No. 60175-0417), the entire contents of which are incorporated by reference in their entirety as if fully set forth herein.

[0041] The reshaped SDR image (112) and image metadata (177) can be encoded by the coding block 120 in a video signal 122 (e.g., an encoded bitstream, etc.) or a collection of consecutive video segments. Given the video signal (122), a receiving device, such as a mobile phone, can decide to use the metadata along with the SDR image data as part of internally processing or post-processing the video signal (122) on the device to generate and render an image having a higher dynamic range, such as HDR, and more vivid colors within the display capabilities of the receiving device. Additionally, optionally, or alternatively, the video signal (122) or video segments allows backward compatibility with legacy SDR displays that can ignore the image metadata (177) and simply display the SDR image represented in the SDR image data.

[0042] Exemplary video signals or video segments may include, but are not necessarily limited to, single layer video signals / segments, etc. In some embodiments, the encoding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other distribution formats, to generate the video signal (122) or video segment.

[0043] The video signal (122) or video segment is delivered downstream to a receiver, such as a mobile device, a tablet computer, a decoding and playback device, a media source device, a media streaming client device, a television set (e.g., a smart TV, etc.), a set-top box, a movie theater, etc. At the downstream device, the video signal (122) or video segment is decoded by a decoding block (130) to generate a decoded image 182, which may be similar or identical to the reshaped SDR image (112), albeit affected by quantization errors and / or transmission errors and / or synchronization errors and / or errors caused by packet loss generated in the compression performed by the encoding block (120) and the decompression performed by the decoding block (130).

[0044] In a non-limiting example, the video signal (122) (or video segment) may be a backward-compatible SDR video signal (or video segment), where "backward compatible" refers to a video signal or video segment that carries an SDR image that is optimized for an SDR display (e.g., where a particular artistic intent is preserved, etc.).

[0045] The decode block (130) can also obtain or decode image metadata (177) from the video signal (122) or video segment. The image metadata (177) specifies a backward reshaping mapping that can be used by a downstream decoder to perform backward reshaping on the decoded SDR image (182) and generate a backward reshaped HDR image for rendering on an HDR display (e.g., a target display, a reference display, etc.). The backward reshaping mapping expressed in the image metadata (177) can be generated by the composer metadata generation block (115) through minimizing an error or difference between a backward reshaped HDR image generated using the image metadata (177) and an output image generated by the image generation block (115) using global and / or local reshaping operations. As a result, the image metadata (177) helps ensure that the backward reshaped HDR image generated by the receiver using the image metadata (177) is a relatively good and accurate approximation of the output image generated by the image generation block (115) using global and / or local reshaping operations.

[0046] Additionally, optionally, or alternatively, the image metadata (177) may include display management (DM) metadata that can be used by a downstream decoder to perform display management operations on the rear reshaped image to generate a display image (e.g., an HDR display image) optimized for rendering on an HDR display device.

[0047] In an operational scenario in which the receiver operates with (or is attached to) an SDR display 140 that supports a standard or relatively narrow dynamic range, the receiver can render the decoded SDR image directly or indirectly on the target display (140).

[0048] In an operational scenario in which the receiver operates with (or is attached to) an HDR display 140-1 that supports a high dynamic range (e.g., 400 nits, 1000 nits, 4000 nits, 10000 nits or more, etc.), the receiver can extract composer metadata from (e.g., from a metadata container therein) the video signal (122) or video segment and use the composer metadata to compose an HDR image (132), which may be a backward reshaped image generated from backward reshaping an SDR image based on the composer metadata. Additionally, the receiver can extract DM metadata from the video signal (122) or video segment and apply DM operations (135) to the HDR image (132) based on the DM metadata to generate a display image (137) optimized for rendering on the HDR display device (140-1) and render the display image (137) on the HDR display device (140-1).

[0049] For purposes of illustration only, it has been described that the global and / or local reshaping operations described herein can be performed by an upstream device, such as a video encoder, that generates output images from input images. These output images are then used as target or reference images by the video encoder to generate backward reshaping metadata that assists a receiving device in generating backward reshaped HDR images that relatively closely or accurately approximate the output images generated from the global and / or local reshaping operations.

[0050] It should be noted that in various embodiments, some or all of the global and / or local reshaping operations may be performed by a video encoder alone, a video decoder alone, a video transcoder alone, or a combination thereof.

[0051] Global and / or local reshaping options 2A shows an example architecture for applying global and / or local reshaping operations, some of which may be implemented or performed by an image processing system, such as one or more processing blocks of FIG. 1 on the encoder side, one or more processing blocks of FIG. 1 on the encoder side, a combination of the processing blocks of FIG. 1 on the encoder and decoder side, etc.

[0052] As shown in FIG. 2A, block 202 represents an HDR video signal input / output interface. In various operating scenarios, the HDR video signal input / output interface (202) may be an HDR video signal input interface alone, an HDR video signal output interface alone, or a combination of an HDR video signal input interface and an HDR video signal output interface. In some operating scenarios, an input HDR video signal, or an input HDR image therein, may be input or received using an HDR video input interface of the HDR video signal input / output interface (202). In some operating scenarios, an output HDR video signal, or an output HDR image therein, may be output or transmitted using an HDR video output interface of the HDR video signal input / output interface (202).

[0053] Similarly, block 208 represents an SDR video signal input / output interface. In various operating scenarios, the SDR video signal input / output interface (208) may be an SDR video signal input interface alone, an SDR video signal output interface alone, or a combination of an SDR video signal input interface and an SDR video signal output interface. In some operating scenarios, an input SDR video signal, or an input SDR image therein, may be input or received using an SDR video input interface of the SDR video signal input / output interface (208). In some operating scenarios, an output SDR video signal, or an output SDR image therein, may be output or transmitted using an SDR video output interface of the SDR video signal input / output interface (208).

[0054] Blocks 204 and 206 represent options for forward reshaping operations, methods and / or functions in the forward reshaping path. Blocks 210 and 212 represent options for backward reshaping operations, methods and / or functions in the backward reshaping path. Each of the forward and backward reshaping paths includes options for global reshaping operations, methods and / or functions that may be designed to maintain the overall (SDR or HDR) appearance of the SDR or HDR image generated by these global reshaping operations, methods and / or functions.

[0055] In some operating scenarios, a forward global reshaping operation, method and / or function in the forward reshaping path forms a reversible reshaping pair (e.g., perfectly, faithfully, subject to quantization error, etc., etc.) with a backward global reshaping operation, method and / or function in the backward reshaping path. In these operating scenarios, an (e.g., input, received, etc.) HDR image is forward reshaped by a forward global reshaping operation, method and / or function in the forward reshaping path to generate a forward reshaped (e.g., SDR, etc.) image, and then a reconstructed or backward reshaped image generated by applying a backward global reshaping operation, method and / or function in the backward reshaping path is the same as the (e.g., input, received, etc.) HDR image or closely approximates the (e.g., input, received, etc.) HDR image subject to possible quantization error.

[0056] Similarly, an SDR image (e.g., an input, received, etc.) is backward reshaped by a backward global reshaping operation, method and / or function in a backward reshaping path to generate a backward reshaped (e.g., HDR, etc.) image, and then the reconstructed or forward reshaped image generated by applying a forward global reshaping operation, method and / or function in the forward reshaping path is the same as the SDR image (e.g., the input, received, etc.) or closely approximates the SDR image (e.g., the input, received, etc.) subject to possible quantization errors.

[0057] While capable of enhancing local visually perceptible characteristics, local reshaping operations, methods and / or functions in the forward or backward path, in contrast to global reshaping operations, methods and / or functions, are not revertible by or cannot form a revertible pair with other global or local reshaping operations in the backward or forward path.

[0058] As shown in Figure 2A, in various operating scenarios, depending on what specific reshaping operations, methods and / or functions are used and what specific input and / or output image interfaces are used (e.g., input and / or output SDR image interfaces, input and / or output HDR image interfaces, etc.), the input video signal, or an input image therein, may be HDR or SDR. Similarly, as shown in Figure 2A, depending on what specific reshaping operations, methods and / or functions are used and what specific input and / or output interfaces are used, the output video signal, or an output (e.g., reconstructed, etc.) image therein, may be HDR or SDR.

[0059] Various different combinations are shown below. First, the input signal is an HDR input signal and the output signal is an HDR output signal. In this case, an input HDR image in the HDR input signal is first processed by forward reshaping and then by backward reshaping to generate a corresponding output HDR image in the HDR output signal (which portrays the same visual semantic content as the input HDR image). There are four (2*2=4) combinations, formed by two options using global or local reshaping in the forward path by two options using global or local reshaping in the backward path.

[0060] Second, the input signal is an HDR input signal and the output signal is an SDR output signal. In this case, an input HDR image in the HDR input signal is processed by forward reshaping to generate a corresponding output SDR image in the SDR output signal (which portrays the same visual semantic content as the input HDR image). There are two combinations formed by two options of using global or local reshaping in the forward path.

[0061] Third, the input signal is an SDR input signal and the output signal is an HDR output signal. In this case, an input SDR image in the SDR input signal is processed by backward reshaping to generate a corresponding output HDR image in the HDR output signal (exhibiting the same visual semantic content as the input SDR image). There are two combinations formed by two options of using global or local reshaping in the backward path.

[0062] Fourth, the input signal is an SDR input signal and the output signal is an SDR output signal. In this case, an input SDR image in the SDR input signal is first processed by backward reshaping and then processed by forward reshaping to generate a corresponding output SDR image in the SDR output signal (which represents the same visual semantic content as the input SDR image). There are four (2*2=4) combinations formed by two options using global or local reshaping in the backward path times two options using global or local reshaping in the forward path.

[0063] Improved HDR To support HDR enhancement in an output HDR video signal shaped from an input HDR video signal, several combinations of the above can be used.

[0064] 2B illustrates an example of generating an output HDR video signal from an input HDR video signal using global reshaping operations, methods and / or functions in both the forward and backward paths. As shown, an input HDR image in an input HDR video signal received via an input HDR interface of the HDR input / output interface (202) may be converted to an output HDR image in an output HDR video signal sent via an output HDR interface of the HDR input / output interface (202) using global forward reshaping (204) in the forward path and global backward reshaping (210) in the backward path.

[0065] In operating scenarios where the global forward reshaping (204) in the forward path and the global backward reshaping (210) in the backward path are paired or symmetrical to each other (e.g., revertible, mathematically inverse, mathematically inverse under quantization error, etc.), the output HDR image is identical (potentially within the limits of quantization error) to the (original) input HDR image without HDR enhancement. In some operating scenarios, the reshaped SDR image generated from performing global forward reshaping (204) on the input HDR image may be a watchable unenhanced SDR image.

[0066] In an operating scenario where the global forward reshaping in the forward path (204) and the global backward reshaping in the backward path (210) are unpaired or asymmetric with respect to each other (e.g., non-reversible, not mathematically inverse, not mathematically inverse with differences not attributable to quantization errors, etc.), the output HDR image is modified from the (original) input HDR image with a different HDR appearance. For example, the global forward reshaping in the forward path (204) and the global backward reshaping in the backward path (210) can generate a different HDR appearance with luminance and saturation that differs from the luminance and saturation in the (original) appearance of the input HDR image. Additionally, optionally, or alternatively, the reshaped SDR image generated from performing the global forward reshaping (204) on the input HDR image can be a viewable non-enhanced SDR image.

[0067] 2C illustrates an example of generating an output HDR video signal from an input HDR video signal using a global forward reshaping operation, method, and / or function in the forward path and a local backward reshaping operation, method, and / or function in the backward path. As shown, an input HDR image in the input HDR video signal received via an input HDR interface of the HDR input / output interface (202) may be converted to an output HDR image in the output HDR video signal sent via an output HDR interface of the HDR input / output interface (202) using global forward reshaping (204) in the forward path and local backward reshaping (212) in the backward path.

[0068] The output HDR image is enhanced thanks to the use of local backward reshaping. Additionally, optionally, or alternatively, the reshaped SDR image generated by performing global forward reshaping (204) on the input HDR image may be a non-locally enhanced SDR image.

[0069] 2D illustrates an example of generating an output HDR video signal from an input HDR video signal using a local forward reshaping operation, method, and / or function in the forward path and a global backward reshaping operation, method, and / or function in the backward path. As shown, an input HDR image in the input HDR video signal received via an input HDR interface of the HDR input / output interface (202) may be converted to an output HDR image in the output HDR video signal sent via an output HDR interface of the HDR input / output interface (202) using local forward reshaping (206) in the forward path and global backward reshaping (210) in the backward path.

[0070] The output HDR image is enhanced thanks to the use of local forward reshaping. The reshaped SDR image generated by performing local forward reshaping (206) on the input HDR image is a locally enhanced SDR image.

[0071] 2E illustrates an example of generating an output HDR video signal from an input HDR video signal using a local forward reshaping operation, method, and / or function in the forward path and a local backward reshaping operation, method, and / or function in the backward path. As shown, an input HDR image in the input HDR video signal received via an input HDR interface of the HDR input / output interface (202) may be converted to an output HDR image in the output HDR video signal sent via an output HDR interface of the HDR input / output interface (202) using local forward reshaping (206) in the forward path and local backward reshaping (212) in the backward path.

[0072] The output HDR image is doubly enhanced thanks to the use of local forward reshaping and local backward reshaping in both the forward and backward paths. The reshaped SDR image generated by performing local forward reshaping (206) on the input HDR image is an enhanced SDR. Additionally, optionally, or alternatively, the global settings in one or both of the forward and backward paths may be asymmetric, thereby further modifying the (original) HDR appearance of the input HDR image to a different HDR appearance of different global luminance and saturation compared to the input HDR image.

[0073] Additionally, optionally, or alternatively, in some operating scenarios, some or all of the aforementioned operations in combination with one or more options described above may be performed iteratively in multiple rounds of global / local forward and global / local backward reshaping to generate further HDR and / or SDR enhancements in the output HDR image and the reshaped SDR image.

[0074] Improved SDR To support SDR enhancement in an output SDR video signal that is reshaped from an input SDR video signal, several combinations of the above may be used.

[0075] 2F illustrates an example of generating an output SDR video signal from an input SDR video signal using global reshaping operations, methods and / or functions in both the forward and backward paths. As shown, an input SDR image in an input SDR video signal received via an input SDR interface of an SDR input / output interface (208) may be converted to an output SDR image in an output SDR video signal sent via an output SDR interface of the SDR input / output interface (208) using global forward reshaping (204) in the forward path and global backward reshaping (210) in the backward path.

[0076] In operating scenarios where the global forward reshaping (204) in the forward path and the global backward reshaping (210) in the backward path are paired or symmetrical to each other (e.g., restorable, mathematically inverse, mathematically inverse under quantization error, etc.), the output SDR image is identical to the (original) input SDR image without SDR enhancement (possibly subject to quantization error). In some operating scenarios, the reshaped HDR image generated from performing global backward reshaping (210) on the input SDR image may be a viewable unenhanced HDR image.

[0077] In operating scenarios where the global forward reshaping (204) in the forward path and the global backward reshaping (210) in the backward path are not paired or asymmetric with respect to each other (e.g., non-reversible, not mathematically inverse, not mathematically inverse with differences not attributable to quantization errors, etc.), the output SDR image is modified from the (original) input SDR image with a different SDR appearance. For example, the global forward reshaping (204) in the forward path and the global backward reshaping (210) in the backward path can be used to generate a different SDR appearance with luminance and saturation that differs from the luminance and saturation in the (original) appearance of the input SDR image. Additionally, optionally, or alternatively, the reshaped HDR image generated from performing the global backward reshaping (210) on the input SDR image can be a viewable non-enhanced HDR image.

[0078] 2G illustrates an example of generating an output SDR video signal from an input SDR video signal using a global forward reshaping operation, method and / or function in the forward path and a local backward reshaping operation, method and / or function in the backward path. As shown, an input SDR image in an input SDR video signal received via an input SDR interface of an SDR input / output interface (208) may be converted to an output SDR image in an output SDR video signal sent via an output SDR interface of the SDR input / output interface (208) using global forward reshaping (204) in the forward path and local backward reshaping (212) in the backward path.

[0079] The output SDR image is enhanced thanks to the use of local backward reshaping. Additionally, optionally, or alternatively, the reshaped HDR image generated by performing global backward reshaping (212) on the input SDR image may be a locally enhanced HDR image.

[0080] 2H illustrates an example of generating an output SDR video signal from an input SDR video signal using a local forward reshaping operation, method and / or function in the forward path and a global backward reshaping operation, method and / or function in the backward path. As shown, an input SDR image in an input SDR video signal received via an input SDR interface of an SDR input / output interface (208) may be converted to an output SDR image in an output SDR video signal sent via an output SDR interface of the SDR input / output interface (208) using local forward reshaping (206) in the forward path and global backward reshaping (210) in the backward path.

[0081] The output SDR image is enhanced thanks to the use of local forward reshaping. Additionally, optionally, or alternatively, the reshaped HDR image generated by performing global backward reshaping (210) on the input SDR image may be a non-locally enhanced HDR image.

[0082] 2I illustrates an example of generating an output SDR video signal from an input SDR video signal using a local forward reshaping operation, method, and / or function in the forward path and a local backward reshaping operation, method, and / or function in the backward path. As shown, an input SDR image in an input SDR video signal received via an input SDR interface of an SDR input / output interface (208) may be converted to an output SDR image in an output SDR video signal sent via an output SDR interface of the SDR input / output interface (208) using local forward reshaping (206) in the forward path and local backward reshaping (212) in the backward path.

[0083] The output SDR image is doubly enhanced thanks to the use of local forward reshaping and local backward reshaping in both the forward and backward paths. The reshaped HDR image generated by performing local backward reshaping (212) on the input SDR image is an enhanced HDR. Additionally, optionally, or alternatively, the global settings in one or both of the forward and backward paths can be asymmetric, which further changes the (original) SDR look of the input SDR image to a different SDR look with different global luminance and saturation compared to the input SDR image.

[0084] Additionally, optionally, or alternatively, in some operating scenarios, some or all of the aforementioned operations in combination with one or more options described above may be performed iteratively in multiple rounds of global / local forward and global / local backward reshaping to generate further HDR and / or SDR enhancements in the output SDR image and the reshaped HDR image.

[0085] Reversible Improvement Revertible enhancements such as those based on global reshaping in both forward and backward paths can be applied in many video applications. FIG. 2J shows an example video application in which a source image is captured as an SDR image by a mobile device in an input SDR video signal. Upconversion from SDR to HDR can be performed or executed using global backward reshaping (e.g., 210). An upconverted (or reshaped) HDR image having at least dynamic range enhancement can be generated by performing global backward reshaping (210) on the (source or input) SDR image. Compared to global reshaping, which can be performed with relatively little or modest computational power, local reshaping may not be affordable or feasible due to the relatively limited computational power in a mobile phone. Instead, the global backward reshaping (210) can be used to generate an (upconverted or reshaped) HDR image. The HDR image can be encoded as an output version of an HDR coded bitstream. This version of the HDR encoded bitstream can be shared, previewed and / or displayed on the display screen of a mobile device.

[0086] The HDR image can be uploaded to another device, such as a more powerful machine (e.g., a PC, a laptop, a cloud-based storage and / or computing system, a server, a media or video sharing system, etc.) via the HDR encoded bitstream. As shown in FIG. 2C, the HDR encoded bitstream can be received as an input HDR video signal. This version of the HDR encoded bitstream's HDR image, generated from the global backward reshaping (210) performed by the mobile device on the (original) SDR image, can be further enhanced by first restoring the (original) SDR image generated by the mobile device (e.g., by a camera(s) operating in association with the mobile device, etc.) using a restorable global forward reshaping (204), which forms a restorable pair or corresponds to the global backward reshaping (210) of FIG. 2J. As shown in FIG. 2C, local backward reshaping (212) can be applied or performed on the restored SDR image to generate a (new) HDR image with improved HDR appearance compared to the HDR appearance of the HDR image of this version of the HDR encoded bitstream.

[0087] Thus, in this video application involving a mobile device acquiring an original or input SDR image, the original or input SDR image can be restored from the HDR image of this version of the HDR encoded bitstream generated by the mobile device and viewed. Also, an HDR image with at least the dynamic range enhancement generated by the mobile device can be received from, for example, the mobile device and viewed. Furthermore, an HDR image with further enhancement can be generated and viewed from performing local reshaping as described herein as shown in FIG. 2C.

[0088] Similarly, in video applications where an input HDR image is received in an input HDR video signal (e.g., generated by a mobile device, generated by a non-mobile device, generated by a camera, etc.), a global forward reshaping can be performed on the HDR image to generate an output SDR image. The output SDR image can be reverted to the (original) input HDR image using a global backward reshaping that forms a restorable pair or corresponds to the global forward reshaping. Additionally, optionally, or alternatively, a local forward reshaping can be performed on the HDR image to generate an enhanced output SDR image, e.g., with local enhancement.

[0089] Global Reshaping In some operating scenarios, the global reshaping function or mapping used in global reshaping can be designed based on a scalable static framework for single-layer backward compatible (SLBC) video coding. An algorithm that ensures perfect restorability from one domain, such as one of the SDR and HDR domains, to another domain, such as the other of the SDR and HDR domains, can be used in global reshaping as described herein. There may be different usages of global reshaping depending on whether the forward and backward (global reshaping) functions used by global reshaping form a reversible pair or not.

[0090] In some operating scenarios, paired forward and backward global reshaping functions are used in a scalable static framework for SLBC video encoding. In these operating scenarios, there may be a total number G (e.g., G=4096; covering all possible codeword values ​​in the 12-bit codeword space) of perfectly reversible global forward and global backward reshaping function pairs. Each perfectly reversible global forward and global backward reshaping function pair includes a global forward reshaping function and a corresponding global backward reshaping function.

[0091] Denote the global forward reshaping functions in the above restorable pairs as follows (each of the global forward reshaping functions transforms or forward reshapes codewords for the luma and chroma channels of a first color space into forward reshaped codewords for the luma and chroma channels of a second color space):

number

[0092] Similarly, the global backward reshaping functions for the above restorable pairs are expressed as follows (each of the global backward reshaping functions transforms or forward reshapes codewords for the luma and chroma channels of the second color space into backward reshaped codewords for the luma and chroma channels of the first color space):

number

[0093] As shown in Figure 3A and in equation (1) above, the luma global forward reshaping function operates on different index values ​​g F,Y Similarly, as shown in FIG. 3B and equation (2) above, the luma global backward reshaping function may be indexed or looked up using different index values ​​g B,Y may be indexed or looked up using

[0094] A reconstructable pair in the multiple reconstructable pairs formed by the global forward reshaping function in equation (1) and the global backward reshaping function in equation (2) may be formed by a global forward reshaping function in the global forward reshaping function in equation (1) and a global backward reshaping function (in the global backward reshaping function in equation (2)) that corresponds to this global forward reshaping function.

[0095] The global forward reshaping function and the global backward reshaping function in the same restorable pair may be indexed with the same index value. The same pair of global forward and global backward reshaping functions can be used to restore the original input video signal or an input image therein in the output video signal or output image. For example, given a g-th pair in the plurality of restorable pairs, g F,Y =g B,Y =g F,Cx =g B,Cx If =g, the input HDR image may be forward reshaped to a reshaped SDR image and then backward reshaped to an output HDR that is identical (e.g., perfectly, faithfully, under quantization error, etc.) to the input HDR image, as follows:

number

[0096] Given the same gth pair, the input SDR image may be backward reshaped to a reshaped HDR image and then forward reshaped to an output SDR image that is the same (e.g., perfectly, faithfully, under quantization error, etc.) as the input SDR image.

number

[0097] Quantization errors or losses may occur when the reshaping function performs a bit depth conversion from a pre-reshaping bit depth (e.g., the total number of bits encoding each codeword in a pre-reshaping image in a pre-reshaping color space channel) to a post-reshaping bit depth (e.g., the total number of bits encoding each codeword in a post-reshaping image in a post-reshaping color space channel), e.g., from a 16-bit pre-reshaping video signal / image to a 10-bit post-reshaping video signal / image. If the difference between an input image received by a restorable pair and an output image generated by the restorable pair is attributable to the quantization errors or losses caused by the bit depth conversion(s), the input image and the output image can still be considered the same.

[0098] FIG. 3C shows an example luma codeword mapping using the same (restorable) pair of global forward and backward reshaping functions indexed by the same index value g to convert input luma codewords of an input HDR image (input bit depth=16 bits) to output luma codewords of an output HDR image (output bit depth=16 bits) using equation (3-1) above.

[0099] As shown, the input-to-output codeword mapping in the luma channel is approximately linear (e.g., within an SMPTE-specified range that clips at both ends of a defined luma value range). This means that the reconstructed or output luma codewords in the output HDR image generated by the restorable pair are the same as or very close to the luma codewords received in the input HDR image.

[0100] 3D and 3E show an example chroma codeword mapping using the same (restorable) pair of global forward and backward reshaping functions indexed by the same index value g to convert input chroma codewords of an input HDR image (16 bits) to output chroma codewords of an output HDR image (16 bits) using equation (3-2) above.

[0101] The input chroma codewords have fixed Cb (eg, 35000) and Cr (eg, 28000) values, but can form a group of colors with varying luma codeword values.

[0102] As shown, the input to output codeword mapping in the chroma channels for this color group is approximately linear, meaning that the chroma codewords reconstructed or output in the output HDR image generated by the restorable pair are the same or very close to the chroma codewords received in the input HDR image for this color group for varying luma codeword values.

[0103] Similarly, another group of colors with another set of fixed Cb and Cr codeword values ​​(e.g., Cb is 25000, Cr is 35000, etc.) but varying luma codeword values ​​can be used in chroma mapping using a restorable pair of global forward and backward reshaping functions. The input to output codeword mapping in the chroma channels for such another group of colors can be shown to be approximately linear, similar to Figures 3D and 3E. This means that the reconstructed or output chroma codewords in the output HDR image generated by the restorable pair are the same or very close to the chroma codewords received in the input HDR image for the other group of colors, for varying luma codeword values.

[0104] FIG. 3F shows an example luma codeword mapping using the same (restorable) pair of global forward and backward reshaping functions indexed by the same index value g to convert input luma codewords of an input SDR image (input bit depth=10 bits) to output luma codewords of an output SDR image (output bit depth=16 bits) using equation (4-1) above.

[0105] As shown, the input-to-output codeword mapping in the luma channel is approximately linear (e.g., within the entire codeword range specified by an industry standard, etc.), which means that the reconstructed or output luma codeword in the output SDR image generated by the restorable pair is the same as, or very close to, the luma codeword received in the input SDR image.

[0106] Similarly, chroma codeword mapping can be performed on the restorable pairs to map input SDR chroma codewords in an input SDR image to output SDR chroma codewords in an output SDR image.

[0107] In some operating scenarios, the global forward and backward reshaping functions in the restorable pair may be, but are not necessarily limited to, MMR-based forward and backward mapping / functions for chroma mapping, and FLUT / BLUT-based forward and backward mapping / functions for luma mapping.

[0108] An MMR-based backward mapping / function in the restorable pair may be used to map input luma and chroma codewords in the input SDR image to backward reshaped chroma codewords in the backward reshaped HDR image. A BLUT-based backward mapping / function in the restorable pair may be used to map input luma codewords in the input SDR image to backward reshaped luma codewords in the backward reshaped HDR image.

[0109] An MMR-based forward mapping / function in the restorable pair may be used to map the backward reshaped luma and chroma codewords in the backward reshaped HDR image to output chroma codewords in the output SDR image. A FLUT-based forward mapping / function in the restorable pair may be used to map the backward reshaped luma codewords in the backward reshaped HDR image to output luma codewords in the output SDR image.

[0110] As in the case of input HDR-output HDR conversion performed using restorable pairs, in input SDR-output SDR conversion performed using restorable pairs, the input chroma codeword in the output SDR image can be restored or can be the same as the output chroma codeword in the input SDR image, subject to possible quantization errors.

[0111] Unpaired anterior and posterior global reshaping Reshaping functions from different restorable pairs may not reproduce the input video signal (or input image therein) in an output signal (or output image therein) generated from a reshaping operation performed on the input video signal (or input image therein) based on these functions.

[0112] For illustrative purposes only, the forward luma and chroma reshaping functions are respectively represented by a first index value g F,Y and g F,Cx where the first index values ​​may or may not be equal to each other. Similarly, the backward luma and chroma reshaping functions may be indexed by second index values ​​g B,C and g B,Cx where the second index values ​​may or may not be equal to each other.

[0113] In operating scenarios in which the first and second index values ​​differ between themselves, the input video signal (or input image therein) may not be reproduced in an output signal (or output image therein) generated from a reshaping operation performed on the input video signal (or input image therein) based on these functions, as described below.

number

[0114] However, in these operating scenarios, these reshaping functions, which may be referred to as unpaired forward and backward reshaping functions, can be applied or used as a global image enhancement, as a tool to adjust the input luminance and chroma of the input image to different luminance and chroma of the output image.

[0115] Figure 3G shows the results for various different index values ​​g ranging from 512 to 3328. B,C , which shows an example luma (or luminance) backward function with a fixed forward index g F,Y = 1792, etc.). Figure 3H shows the results for different index values ​​g ranging from 512 to 3328. F,C , an exemplary luma (or luminance) forward function with a fixed forward index g F,Y =1792, etc.).

[0116] For luma forward and backward reshaping, the index value is g B,Y >g F,Y , the reconstructed HDR image, which is generated at least in part by performing luma forward and backward reshaping functions with these index values ​​on the input HDR image, will be brighter than the input HDR image. B,Y <g F,Y If the gap Δg between the two index values ​​is selected to be Δg, then the reconstructed HDR image, which is generated at least in part by performing the luma forward and backward reshaping functions with these index values ​​on the input HDR image, will be darker than the input HDR image. Y =g B,Y -g F,Y affects the luminance change. In some operating scenarios, the larger the gap size, the larger the luminance change between the input and output images.

[0117] Similarly, for chroma forward and backward reshaping, g B,Cx >g F,Cx(where Cx can be Cb or Cr), the reconstructed HDR image, generated at least in part by performing chroma forward and backward reshaping functions with these index values ​​on the input HDR image, will be more saturated than the input HDR image. On the other hand, if the index values ​​are selected such that g B,Cx <g F,Cx If the gap Δg between the two index values ​​is selected to be 0, then the reconstructed HDR image, which is generated at least in part by performing the chroma forward and backward reshaping functions with these index values ​​on the input HDR image, will be less saturated than the input HDR image. Cx =g B,Cx -g F,Cx affects the saturation change. In some operating scenarios, the larger the gap size, the larger the saturation change between the input and output images.

[0118] FIG. 2K illustrates an example flow using unpaired forward and backward reshaping functions to change the luminance and / or chroma of an input video signal to a different luminance and / or chroma of an output video signal. As with other flows illustrated herein, the flow of FIG. 2K may be performed by one or more computing devices, such as a video encoder, a video transcoder, a video decoder, or a combination thereof. For purposes of illustration only, the input video signal and the output video signal are an input HDR video signal and an output HDR video signal, respectively.

[0119] Block 222 includes receiving an input HDR image in an input HDR video signal. Blocks 224-1 and 224-2 include selecting first luma and chroma index values ​​for a forward reshaping function. Blocks 226-1 and 226-2 include performing luma and chroma forward reshaping on the input HDR image based on the luma and chroma forward reshaping function indexed by the first index value to generate a forward reshaped image. Blocks 228-1 and 228-2 include selecting second luma and chroma index values ​​for backward reshaping. Blocks 230-1 and 230-2 include performing luma and chroma backward reshaping on the forward reshaped image based on the luma and chroma backward reshaping function indexed by the second index value to generate a reconstructed HDR image having a luminance and / or chroma different from the luminance and / or chroma of the input HDR image. Block 232 includes outputting the reconstructed HDR image.

[0120] Local Reshaping and Guided Images Figure 2L shows example luma and chroma local reshaping 246 that may be performed on an input image 242 to generate a (locally reshaped) output image 250. The processing blocks of Figure 2L (including but not limited to luma and chroma local reshaping (246)) may be implemented or performed by one or more computing devices, such as a video encoder, a video transcoder, a video decoder, or a combination thereof.

[0121] As shown in FIG. 2L, two components may be used to assist or support luma and chroma local reshaping (246).

[0122] The first component is F <l>< / l>A family of local reshaping functions 248 includes a set (or total number L) of local reshaping functions denoted as ( ), where l is 0, 1, ... L-1. These local reshaping functions can be used or called as part of luma and chroma local reshaping operations. The family of local reshaping functions is F for luma local reshaping. Y <l>< / l> We consider a family of luma local reshaping functions denoted as F Cx <l>< / l> and a family of chroma-local reshaping functions denoted as

[0123] The second component is guided image generation, which generates a guided image (denoted as M) from an input image (denoted as V) using a guided image generation operator (denoted as G) as follows:

number

[0124] The guided image M includes a distinct (reshape function) index value (in the value range of [0 L-1]) for each input pixel in some or all of the input images (242) to select which local reshape function in a family of local reshape functions performs local reshaping on the input pixel to generate a corresponding output pixel in the output image (250). The guided image M includes a distinct (reshape function) index value (in the value range of [0 L-1]) for luma and chroma local reshaping in different luma and chroma channels in M Y and M. Cx The different guided images may include:

[0125] More specifically, given a guided image generated by the guided image generation (244), for each input pixel in the input image (242), the local reshape (246) can find a particular reshape function index value(s) stored in the guided image in the same row and in the same column as the image frame that contains the input image (242). The particular reshape function index value can then be used to select or identify a particular local reshape function(s) from among the reshape functions that make up the family of reshape functions (248). The particular local reshape function can then perform luma and chroma local reshaping on the input pixel to generate a corresponding output pixel in the output image (250).

[0126] For purposes of explanation only, local forward reshaping is discussed in detail. Note that local backward reshaping may be derived, implemented, or performed in a similar manner. For simplicity, in the discussion herein, superscripts such as "Y" and "Cx" may be removed.

[0127] Single Channel Luma Local Reshape In some operating scenarios, the output or reshaped luma codeword of the output image can be generated from performing a single-channel luma local reshaping on the input or pre-reshaped luma codeword of the input image. More specifically, an input luma codeword (e.g., suffices, etc.) for a pixel of the input image allows the local reshaping function to determine a mapped or reshaped luma codeword for a corresponding pixel of the output image without using the input chroma codeword for that pixel of the input image.

[0128] The family of luma local reshaping functions for single-channel luma local reshaping can be created in many different ways.

[0129] In a first example, self-derived local reshaping may be used to generate a family of luma local reshaping. Self-derived local reshaping refers to an approach where a family of luma local reshaping is derived from a luma global reshaping function. This family of luma local reshaping functions can be generated on the fly at run-time in response to each input image or a specific codeword distribution therein. The family of local reshaping functions can include luma local reshaping functions customized for each frame in different content.

[0130] In a second example, offline training may be used to generate a family of luma local reshaping functions that includes a pre-constructed luma local reshaping function, which may be applied to all input images in all content.

[0131] In a third example, a hybrid offline and online operation may be used to generate a family of luma local reshaping functions that include luma local reshaping functions generated by performing some or all of the self-derived local reshaping from a preconstructed global reshaping function. The benefit of this hybrid method is that it saves or reduces computational cost in that the luma reshaping functions can be constructed using a dynamic global function that is constructed at run-time in response to the input image and preconstructed local functions that are generated from offline training.

[0132] Self-derived single-channel local reshaping The two goals of luma local reshaping are (a) to increase the local contrast ratio while (2) maintaining similar luminance in an output image generated from performing luma local reshaping on an input image. The first goal can be achieved by increasing the slope of the local reshaping function as expressed in a tone curve. The second goal mentioned above can be achieved by intersecting the (e.g., new, etc.) local reshaping function with the global reshaping function at various locations. An exemplary generation of a local reshaping function given a global reshaping function is described in U.S. Provisional Application No. 63 / 004,609, entitled “BLIND LOCAL RESHAPING IN HDR IMAGING,” filed April 3, 2020, the entire contents of which are incorporated by reference in their entirety as if fully set forth herein.

[0133] FIG. 2M illustrates an example flow for generation of local functions that are self-derived from a global reshape function, denoted as F(). In some operating scenarios, the global reshape function may be calculated in online operation by a video codec, e.g., in response to receiving an input image for local reshaping. Such a global reshape function may be referred to as a dynamic global reshape function (e.g., adaptive, run-time, etc.). In some operating scenarios, the global reshape function may be calculated or pre-constructed by an image processing system in offline operation, e.g., not necessarily in response to receiving an input image for local reshaping. Such a global reshape function may be referred to as a static global reshape function (e.g., scalable, pre-constructed, etc.).

[0134] Block 252 includes constructing a template reshape function. To do so, flat regions in the global reshape function F() may first be removed to generate a modified reshape curve or function. These flat regions may correspond to subranges outside the full range of available codewords (e.g., valid, SMPTE, etc.) in the codeword space. The flat regions may be added back into a local reshape function that is constructed later (e.g., eventually, etc.).

[0135] The modified reshape curve or function can be shifted until it is tangent to the y-axis. As used herein, plots or curves representing local or global reshape functions (e.g., as shown in Figures 3A, 3B, 3G, and 3H, etc.) may be expressed in a two-dimensional coordinate system where the y-axis represents the reshaped or output codeword and the x-axis represents the pre-reshape or input codeword. The shifting of the modified reshape curve or function can be used to avoid or eliminate the need to deal with edge cases and to facilitate the scaling and shifting operations that generate the local reshape function. Thus, the shifted modified reshape function (denoted as F') can be constructed from the global reshape function with the removal of flat regions and shifting to the y-axis as follows: F'=shift_to_Yaxis(remove_flat(F())) (7)

[0136] To achieve the higher local contrast ratio mentioned as the first goal above, α <l>< / l> More specifically, at some input codeword values, the x-axis scaling factor is set to be greater than 1, or α <l>< / l> >1, then the local neighborhood around the input codeword value along the x-axis is expanded (e.g., proportional to the x-axis scaling factor), thereby lowering or decreasing the local contrast ratio (or increasing blurriness) at the x-axis location represented by that input codeword value. In other words, the x-axis scaling factor α <l>< / l>is greater than 1, the curve is scaled along the x-axis. If the range along the x-axis (e.g., interval or difference used for the derivative calculation) is increased and the mapping range along the y-axis (e.g., interval or difference used for the derivative calculation) remains the same, the x-axis scaled curve will have a smaller slope than the unscaled curve, and therefore the contrast ratio of the x-axis scaled curve will be smaller compared to the unscaled curve. On the other hand, if the x-axis scaling factor is set to less than 1 or α is set to 0 at the input codeword value, the x-axis scaled curve will have a smaller slope than the unscaled curve, and therefore the contrast ratio of the x-axis scaled curve will be smaller compared to the unscaled curve. <l>< / l> <1, the local neighborhood around the input codeword value along the x-axis is compressed (e.g., proportional to the x-axis scaling factor), thereby increasing or enhancing the local contrast ratio (or sharpness) at the x-axis location represented by the input codeword value. In other words, the x-axis scaling factor α <l>< / l> If is less than 1, the curve is compressed along the x-axis. Since the range along the x-axis (e.g., the interval or difference used for the derivative calculation) is reduced and the mapped range along the y-axis (e.g., the interval or difference used for the derivative calculation) remains the same, the x-compressed curve has a larger slope than the uncompressed curve, and therefore the contrast ratio in the x-compressed curve is larger than in the uncompressed curve.

[0137] x-axis scaling factor (α <l>< / l> <1) can be used to scale the shifted modified reshape function at various input codeword values, for example along the x-axis as specified by the x-axis scaling factor to generate a scaled (forward) function, which increases the local contrast ratio (or sharpness) in the scaled function at these input codeword values.

[0138] Template reshaping function (F T <l>< / l>()) can be constructed or constructed by first translating or shifting the template reshape function into a scaled (or shifted) forward function, and then resampling the scaled function at various input codeword values ​​that correspond to the input codeword values ​​used to encode the input image: F T <l>< / l> ( )=resample(F'(),α <l>< / l> ) (8)

[0139] In some operating scenarios, unequal local contrast ratio enhancements can be achieved between different local reshaping functions in a local reshaping family by using different values ​​of the x-axis scaling factor in equation (8) above for different l values ​​of the local reshaping functions.

[0140] In some operating scenarios, equal local contrast ratio enhancement can be achieved among some or all local reshaping functions in a local reshaping family by using the same value of the x-axis scaling factor in equation (8) above for the l values ​​of the local reshaping functions.

[0141] For purposes of illustration only, the same x-axis scaling factor α that is assigned or applied in the shifting and resampling operations in equation (8) above will be used. <l>< / l> The same local contrast ratio enhancement is used with all the local reshaping functions in the local reshaping family (F for all l). T <l>< / l> ()=F T Thus, only one scaled and resampled function can be generated for the template reshaping function () using the same x-axis scaling factor α <l>< / l> From a global reshaping function having the following formula, .times. ...

[0142] Block 254 includes shifting the template reshape function in equation (8) above to generate a pre-fused local reshape function that can be fused with the global reshape function into the l-th local reshape function in the local reshape family. This shift of the template reshape function achieves a second goal, namely, maintaining similar (e.g., locally average, locally filtered, global, etc.) intensity between the pre-fused local reshape function and the global reshape function. The term "fusion" as used throughout this disclosure refers to the interpolation or blending of two specified functions. In a preferred embodiment, blending of functions involves computing a linear combination of the functions involved. The weights in the linear combination may be normalized to a given value. For example, the sum of the weights may be 1. The term "pre-fusing" is used throughout this disclosure in reference to a reshape function. For example, a local reshape function may be fused with a global reshape function. The resulting fused reshape function is still a local reshape function despite the fusion with the global reshape function because the fused reshape function still maintains different functional characteristics at the pixel level from the pre-fused local reshape function. To distinguish between the local reshape function before fusion and the local reshape function after fusion with the global reshape function, the initial local reshape function before fusion is referred to as the "pre-fused local reshape function" and the fusion result is referred to as the "fused local reshape function", the "fused local reshape function" or simply the "local reshape function". The term "pre-fusion" may also be applied to other reshaping functions; for example, when fusing two global reshaping functions, if they remain more global and therefore a distinction is necessary, one should refer to the global reshaping function before the fusion ("pre-fusion") or the global reshaping function after the fusion.

[0143] More specifically, when an input pixel such as the i-th input codeword in an input image is reshaped to an output pixel such as the i-th output codeword, the pre-merged local reshaping function for the l-th local reshaping function is - in the local reshaping family - the i-th element (m i , where l is the local reshaping function index value stored in the local reshaping function tree.

[0144] The bit depth of the input image is B. v The input codeword range of this bit depth (total) may be partitioned into L uniform intervals with corresponding interval centers as follows:

number

[0145] In order to maintain similar (e.g., locally averaged, locally filtered, global, etc.) intensities around the input pixels between the global reshaping function and the pre-merged local reshaping function for the l-th local reshaping function, the (original) global reshaping function F(C l v ) for the globally reshaped values ​​and the pre-merged local reshaping function F generated by shifting the template reshaping function for the l-th local reshaping function at the same center of the l-th interval of the input codeword range. <l>< / l> (C l v ) may be constrained to satisfy the following conditions / constraints: F(C l v )=F <l>< / l> (C l v ) (11)

[0146] This constraint is the pre-fusion local reshaping function F <l>< / l> (C l v) and apply the shift (value) to the template reshaping function to obtain a pre-fused local reshaping function F <l>< / l> (C l v ) can be used to generate the lth local reshaping function or the pre-fusion local reshaping function F <l>< / l> (C l v ) without having to recalculate it from scratch.

[0147] Therefore, the input codeword is C l v Given a corresponding mapped (or globally reshaped) value F <l>< / l> (C l v ) (for example, the global reshaping function F <l>< / l> (C l v ) can be obtained or looked up from a curve, function and / or lookup table representing the corresponding mapped value F <l>< / l> (C l v ) is the input codeword value φ such that the following equality conditions (e.g., equal luminance) are satisfied: l v Can be used to determine: F <l>< / l> T (φ l v )=F <l>< / l> (C l v ) (12)

[0148] Furthermore, the input codeword value φ l v can be used to determine the shift as follows: a l v =C l v -φ l v (13)

[0149] Therefore, the pre-fusion local reshaping function for the l-th local reshaping function (for all l between 0 and L-1) is F <l>< / l> (C l v ) can be obtained as a shifted version of the template reshaping function, as follows: F <l>< / l> (v)=F <l>< / l> T (va l v ) (14)

[0150] After scaling and shifting operations are performed to increase or change the slope of the global reshape function to that of the template reshape function, the template reshape function or the local reshape functions generated from the template reshape function may generate values ​​outside the range of allowable codeword values ​​that are hard clipped, thereby generating visual artifacts. To avoid or reduce hard clipping, soft clipping may be used by blending the global and local functions together with their respective weighting factors.

[0151] Block 256 is a pre-fusion local reshaping function F <l>< / l> (C l v ) with the global reshaping function to obtain the pre-fusion local reshaping function F <l>< / l> (C l v ) is soft-clipped. The pre-blended local reshaping function F generated for the l-th local reshaping function in block 254 is <l>< / l> (C l v ), then θ <l>< / l> Y,G The global reshaping function weighting factor and θ <l>< / l> Y,L The local reshaping function weighting factors denoted as F <l>< / l> (C lv ) respectively. In some operating scenarios, the two weighting factors satisfy the following constraints / conditions: θ <l>< / l> Y,G +θ <l>< / l> Y,L =1 (15)

[0152] The l-th local reshaping function (after fusion) is given by: F <l>< / l> ( )=θ <l>< / l> Y,L F <l>< / l> ( )+θ <l>< / l> Y,G ·F( ) (16)

[0153] It should be noted that the above operations can be implemented or performed to generate both a local forward reshaping function and a local backward reshaping function.

[0154] Pre-built single channel local reshaping In some operating scenarios, in addition to or instead of the self-derived single-channel local reshaping as described above, multiple pre-constructed single-channel global reshaping functions F, each indexed by a global reshaping function index value denoted as g, may be used. <g>< / g> Y ( ) can be obtained through offline training. These pre-constructed global reshaping functions can be used, for example in the forward path, to generate local reshaping functions, each indexed by a local reshaping function index value l, as follows: F <l>< / l> ( )=F <g>< / g> Y () (17-1) Where: g=l (17-2)

[0155] Similarly, in the posterior pathway, B <l>< / l> The local reshaping function, denoted as ( ), is <g>< / g>Y It can be generated from a preconstructed global reshaping function, denoted as ( ): B <l>< / l> ( )=B <g>< / g> Y () (18)

[0156] An exemplary derivation of a local reshaping function from a global reshaping function is described in U.S. Provisional Application No. 63 / 086,699, “ADAPTIVE LOCAL RESHAPING FOR SDR-TO-HDR UP-CONVERSION,” filed on October 2, 2020, the entire contents of which are incorporated by reference as if fully set forth herein.

[0157] Hybrid of predefined and self-derived single channel local reshaping In some operating scenarios, a hybrid approach combining predefined single-channel local reshaping and self-derived single-channel local reshaping may be used to generate the local reshaping functions. More specifically, the self-derived local reshaping as described above may be performed with respect to an existing pre-constructed global reshaping function instead of a global reshaping function constructed (e.g., dynamically, etc.) in response to receiving an input image. For example, a particular pre-constructed global reshaping function may be selected from among multiple pre-constructed global reshaping functions (e.g., obtained through offline training, etc.) based on the codeword distribution of the input image. The particular pre-constructed global reshaping function may be used in the self-derived local reshaping described above instead of a global reshaping function constructed in response to receiving an input image.

[0158] Cross-Channel Luma Local Reshaping In some operating scenarios, the output or reshaped luma codeword of the output image can be generated from performing cross-channel luma local reshaping on the input or pre-reshaped luma and chroma codewords of the input image. More specifically, the input luma and chroma codewords (e.g., sufficient, etc.) for a pixel of the input image collectively enable a local reshaping mapping (e.g., TPB-based, MMR-based, etc.) to determine a mapped or reshaped luma codeword for a corresponding pixel of the output image.

[0159] The family of luma local reshaping functions for cross-channel luma local reshaping can be generated in many different ways. Similar to single-channel luma local reshaping, cross-channel luma local reshaping includes at least (1) using self-derived local reshaping functions, (2) using preconstructed reshaping functions, and (3) a hybrid that uses both self-derived and preconstructed reshaping functions.

[0160] Self-derived cross-channel luma local reshaping In various operating scenarios, any of a variety of different (e.g., TPB, MMR, etc.) types of cross-channel luma reshaping functions can be used. For illustrative purposes only, the cross-channel luma reshaping function may be Tensor-Product B-Spline (TPB)-based. A TPB-based reshaping function can capture wide nonlinearities in luma reshaping.

[0161] FIG. 2N illustrates an example flow for self-derived cross-channel local function generation based on an image pair including a first image and a second image. The flow may be used to generate a TPB-based local reshaping function to map an input codeword of a first image to a locally reshaped codeword that approximates or reconstructs a codeword of a reference image represented by a second image. For illustrative purposes only, a first image in an image pair may be an HDR image, while a second image (or reference image) may be an SDR image. In one example, the SDR image may be one that was used to generate an HDR image (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.). In another example, the SDR image may be one that was generated from the HDR image (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.).

[0162] Block 262 involves constructing or building a 3D mapping table (3DMT) from the image pairs. Each pixel of the HDR image is represented by a corresponding one of the y, c0, and c1 (or alternatively, y, C1) color coordinates in the HDR color space or domain. 0 , C 1 ), v i =[v i y v i c0 v i c1 ] T Each pixel of an SDR image may include luma and chroma codewords denoted as y, c0, and c1, respectively, in the (e.g., three) color channels of the SDR color space or domain. i =[s i y s i c0 s i c1 ] T The image may include luma and chroma codewords represented as:

[0163] The HDR color space, which contains the available codewords in each of the three channels, has a corresponding fixed number of Q y ,Q c0 ,Q c1 The histogram can be quantized or partitioned (e.g., uniformly) using one-dimensional bins of Q y ×Q c0 ×Q c1 ) cubes or 3D histogram bins.

[0164] 3D histogram Ω Q,v Here, Q=[Q y ,Q c0 ,Q c1 ]. 3D histogram Ω Q,v is the total number (Q y Q c0 Q c1 ) 3D histogram bins. 3D histogram Ω Q,v The 3D histogram bins in are denoted by 3D bin index q=(q y ,q c0 ,q c1 ), where each ch={y,C 0 ,C 1}, q ch =0,…,Q ch-1 and may be used to represent pixels that have quantized values ​​of these three channels within a 3D histogram bin - or to store a count (e.g., total number) of such pixels.

[0165] The sum of the (e.g., reference, mapped, etc.) codeword values ​​in the SDR image may be calculated for each 3D histogram bin in the 3D histogram. y Q,s , Ψ C0 Q,s , Ψ C1 Q,s are the luma and chroma channels in the output domain, {y,C 0 ,C 1} is the sum of each of the codewords in

[0166] Let the HDR and SDR images each contain P pixels. An exemplary procedure for computing the count of HDR pixels in the 3D histogram bins of a 3D histogram and the sum of the SDR codeword values ​​for those 3D histogram bins is shown in Table 1 below. [Table 1]

[0167] (v q y,(B) ,v q C0,(B) , v q C1,(B) Let q, q = q(HDR) 3D histogram bin center in the 3D histogram. An exemplary procedure for pre-computing some or all of the 3D histogram bin centers in a 3D histogram is illustrated in Table 2 below. [Table 2]

[0168] Next, among the 3D histogram bins in the 3D histogram, those that have a non-zero total pixel count can be identified. All other 3D histogram bins that have no pixels -- or in some other operating scenarios, have a total pixel count below a minimum pixel count threshold -- can be discarded from further processing.

[0169] q 0 , q 1 , …, q k-1 But the HDR pixel count is Ω q Q,v Let k be the number of 3D histogram bins for which ≠ 0. For these k 3D histogram bins, the mean value of the SDR codeword is

number

number

[0170] The result is multiple mapping pairs from a first image (HDR image in this example) to a second image (SDR image in this example), where each such mapping pair may include the center of the 3D histogram bin indexed by a valid q with a non-zero HDR pixel count and the average of the SDR codeword values ​​for the SDR pixels mapped from the counted HDR pixels in the 3D histogram bin, as follows:

number

[0171] In some operating scenarios,

number

[0172] Block 264 involves constructing a modified 3DMT for each local reshaping function. To achieve a change in local contrast ratio, h Y A function, denoted by (), is designed to calculate the average SDR or mapped value in each 3D histogram bin (indexed by a valid q with a non-zero HDR pixel count) of the 3D histogram.

number

number

number

[0173] The point of intersection (e.g., center) of the global reshape function and the lth local reshape function is given by, for example, to maintain similar brightness between the global reshape function and the lth local reshape function in a local neighborhood where the local contrast ratio is increased: C l v =l / L (21)

[0174] In some operating scenarios, the above function h used to realize the change in local contrast ratio Y () is the function α for the l-th local reshaping function. <l>< / l> (To increase the local contrast ratio, α <l>< / l> < 1, α to reduce local contrast ratio <l>< / l> >1). Therefore, the above equation (21) can be rewritten as follows:

number

[0175] Block 266 includes constructing TPB coefficients for the global reshape function and each of the local reshape functions (eg, the lth local reshape function, etc.).

[0176] Based on the (original) 3DMT pair defined above in equation (19), the first TPB coefficients for the global reshaping function

number

number

number

[0177] Vector

number

number

number

[0178] Optimized global reshaping solution or first TPB coefficients

number

number

[0179] 1st TPB coefficient

number

number

number

number

number

[0180] Similar to the case of the global reshaping function, for each of the L local reshaping functions (l=0,…,L-1), we obtain the optimized local reshaping solution or the second TPB coefficients

number

number

[0181] Block 268 includes constructing a (e.g., fused) TPB-based local reshaping function. Similar to single-channel local reshaping / prediction, local reshaping functions with higher local contrast may generate values ​​outside the range of acceptable codeword values ​​that are hard clipped, thereby generating visual artifacts. To avoid or reduce hard clipping, soft clipping may be used by blending the global and local functions together with their respective weighting factors.

[0182] Given the l-th local reshaping function generated in block 266, θ <l>< / l> Y,G The global reshaping function weighting factor and θ <l>< / l> Y,L A local reshape function weighting factor, denoted as l th local reshape function, may be assigned to the global reshape function and the l th local reshape function, respectively. In some operating scenarios, the two weighting factors satisfy the constraints / conditions as shown in equation (15) above. Thus, the local reshaping described herein may be performed using a fused local reshape function generated by fusing the first TPB coefficient for the global reshape function and the second TPB coefficient for the local reshape function, as follows:

number

[0183] Prebuilt cross-channel luma local reshaping In some operating scenarios, in addition to or instead of the self-derived cross-channel local reshaping as described above, multiple pre-constructed cross-channel global reshaping functions F, each indexed by a global reshaping function index value denoted as g, may be used. <g>< / g> Y() can be obtained through offline training. These pre-constructed global reshaping functions can be used, for example in the forward path, to generate local reshaping functions, each indexed by a local reshaping function index value l, as follows: F <l>< / l> ( )=F <g>< / g> Y ( ) (29-1) Where: g=l (29-2)

[0184] Similarly, in the posterior pathway, B <l>< / l> The local reshaping function, denoted as (), is <g>< / g> Y From the preconstructed global reshape function denoted by (), it can be generated as follows: B <l>< / l> ( )=B <g>< / g> Y ( ) (30)

[0185] Hybrid of predefined and self-derived cross-channel luma local reshaping In some operating scenarios, a hybrid approach combining predefined and self-derived single-channel local reshaping may be used to generate the local reshaping function.

[0186] In some embodiments, one or more 3DMTs can be generated in offline training using one or more sets of training image pairs in the training data, respectively. Each training image pair in the one or more respective sets of training image pairs may include an input HDR image and a corresponding input SDR image exhibiting the same visual semantic content as the input HDR image. In one example, the corresponding input SDR image may be one that was used to generate the input HDR image (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.). In another example, the corresponding input SDR image may be one that was generated from the input HDR image (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.).

[0187] Each 3DMT in the one or more 3DMTs can be generated using a respective set of training image pairs in the training data. Image data, such as luma and chroma codewords, in each set of training image pairs can be used to generate statistics collected in the corresponding 3DMT. These statistics can be used to form a plurality of 3DMT mapping pairs for the corresponding 3DMT. Some or all of the one or more 3DMTs, the statistics in the one or more 3DMTs, and / or the 3DMT mapping pairs can be stored, cached, and / or loaded at system startup.

[0188] Each 3DMT in the one or more 3DMTs can be indexed by a respective index value (e.g., g, etc.) for lookup purposes. In some operating scenarios, an index value (e.g., g, etc.) that uniquely identifies a corresponding 3DMT in the one or more 3DMTs may be an L1 mid value. Exemplary mid L values ​​can be found in the previously mentioned U.S. Provisional Patent Application No. 63 / 086,699.

[0189] FIG. 2O shows an example flow for combining self-derived and pre-constructed cross-channel local function generation.

[0190] Block 262' includes reusing or selecting a 3DMT generated using a corresponding set of corresponding training image pairs. In some operating scenarios, a distribution of input codeword values ​​determined from an input HDR image can be used to calculate or predict an index value (denoted as g), such as a mid-L value. The index value can be used to look up, identify, or select the 3DMT from among the one or more 3DMTs. The selected 3DMT can be generated based on image data in the gth set of training image pairs that includes the gth SDR data.

[0191] As a result, a plurality of 3DMT mapping pairs are generated from the 3DMT.

number

number

[0192] Block 264' involves constructing an individual modified 3DMT for each local reshape function (eg, the lth local reshape function, etc.).

[0193] To achieve local contrast variation, h Y The function denoted by () can be used to modify the average of the SDR codeword values ​​in the 3DMT mapping pair as follows:

number

[0194] The point of intersection of the global reshape function and the lth local reshape function may be given as follows, for example, to maintain similar brightness between the global reshape function and the lth local reshape function in a local neighborhood where the local contrast ratio is increased: C l v =l / L (33)

[0195] For the l-th local reshaping function, α <l>< / l> (α is used to increase the local contrast ratio) <l>< / l> < 1, α to reduce local contrast ratio <l>< / l> >1) may be used in the following formula to change the average or mapped value of the SDR codeword in the 3DMT mapping pair:

number

[0196] Block 266' involves constructing a first TPB coefficient for the global reshape function and a second TPB coefficient for the lth local reshape function.

[0197] More specifically, based on the original 3DMT mapping pairs generated using the training data, the first TPB coefficients are designed as follows: Y and vector

number

number

number

[0198] Optimized global reshaping solution or optimized values ​​for the first TPB coefficients

number

number

[0199] 1st TPB coefficient

number

number

number

number

number

[0200] As in the case of the global reshaping function, the optimized local reshaping solution for each of the L local reshaping functions (l=0,…,L-1) or the optimized value for the second TPB coefficient

number

number

[0201] Block 268' includes constructing a (e.g., fused, etc.) TPB-based local reshaping function. Similar to single-channel local reshaping / prediction, local reshaping functions with higher local contrast may generate values ​​that are hard-clipped out of range of acceptable codeword values, thereby generating visual artifacts. To avoid or reduce hard clipping, soft clipping may be used by fusing the global and local functions together with their respective weighting factors.

[0202] Given the l-th local reshaping function generated in block 266′, θ <l>< / l> Y,G and a global reshaping function weighting factor, θ <l>< / l> Y,L A local reshape function weighting factor, denoted as l th local reshape function, may be assigned to the global reshape function and the l th local reshape function, respectively. In some operating scenarios, the two weighting factors satisfy the constraints / conditions as shown in equation (15) above. Thus, the local reshaping described herein may be performed using a fused local reshape function generated by fusing the first TPB coefficient for the global reshape function and the second TPB coefficient for the local reshape function as follows:

number

[0203] FIG. 2P illustrates an example flow for using 3DMT to generate multiple cross-channel local reshaping functions for performing (e.g., forward) local reshaping on an input HDR image 272. The cross-channel local reshaping functions may be implemented, at least in part, using α <l>< / l> , θ <l>< / l> Y,G , θ <l>< / l> Y,L It can be generated based on the improvement parameters such as:

[0204] Block 276 includes constructing a 3DMT using a 3D histogram having mapping pairs (or entries) that store counts of HDR pixels in the input HDR image (272) and the average of the corresponding input SDR image in the image pair formed by the input HDR image and the input SDR image.

[0205] In one example, the corresponding input SDR image may have been used to generate the HDR image (272) (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.). In another example, the corresponding input SDR image may have been generated from the HDR image (272) (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.).

[0206] Block 278 includes using 3DMT to construct a global reshape function. For example, the first TPB coefficients may be generated to specify the global reshape function using a complete set of tensor product B-spline basis functions.

[0207] Block 280 determines an enhancement parameter α for local contrast changes. <l>< / l> to modify the 3DMT using a modified 3D histogram for each cross-channel local reshaping function in a plurality of cross-channel local reshaping functions used for local reshaping.

[0208] Block 282 includes constructing a pre-fusion local reshaping function using modified 3DMT. The pre-fusion local reshaping function can be fused with a global reshaping function to the cross-channel local reshaping function in the multiple cross-channel local reshaping functions used for local reshaping. For example, second TPB coefficients may be generated to specify the pre-fusion local reshaping function using tensor product B-spline basis functions.

[0209] Block 284 determines an enhancement parameter θ for soft clipping. <l>< / l> Y,G and θ <l>< / l> Y,L Block 286 includes performing a weighted fusion with both the global reshaping function and the pre-fusion local reshaping functions using a weighted fusion function to generate a corresponding fused cross-channel local reshaping function. Block 286 includes outputting the multiple (fused) cross-channel local reshaping functions.

[0210] Guide image for luma local reshaping The guide image M=G(V) may be generated in one or more different ways to provide local reshaping function index values ​​for pixels of the input image. In some operational scenarios, a multi-level edge-preserving filter may be applied to generate the guide image to avoid or reduce visual artifacts such as halo artifacts (e.g., near edges of image features / objects, near or around background-foreground boundaries, etc.). Exemplary guide image generation with multi-level edge-preserving filtering may be found in the aforementioned U.S. Provisional Patent Application No. 63 / 086,699.

[0211] Local reshaping can increase the local contrast ratio, but it can also increase the likelihood of introducing false contouring or banding artifacts, especially when local reshaping is performed from a higher bit-depth input domain (or input color space) to a lower bit-depth domain (or output color space) and the output image has a lower bit-depth than the input image.

[0212] Film grain injection into a high bit depth video signal that encodes an input image, as described in U.S. Provisional Application No. 63 / 061,937, filed August 6, 2020, entitled "ADAPTIVE STREAMING WITH FALSE CONTOURING ALLEVIATION," the entire contents of which are incorporated by reference in their entirety as if fully set forth herein, may not be sufficient to avoid or significantly reduce banding artifacts because the output domain has fewer available code words than the input domain. This banding artifact problem may be exacerbated when the local contrast ratio is increased by local reshaping. Additionally, optionally, or alternatively, film grain noise may be significantly introduced at the cost of creating an unpleasant visual appearance.

[0213] To overcome or ameliorate the banding artifact problem, local reshape function selection dithering may be implemented or performed to avoid or reduce the banding artifacts.

[0214] FIG. 2Q illustrates an example flow for local reshape function selection dithering. Block 288 includes performing local reshape index noise injection. For example, Gaussian noise is injected into the guided image M with local reshape function index m i The noise injected local reshaping function index is

number

number

[0215] Block 290 includes a step of:

number

[0216] The local (e.g., forward) reshaped codeword value is, for each pixel (e.g., the i-th pixel) of the input image, the respective noise-injected index value in the noise-injected guided image.

number

number

number

[0217] In some operating scenarios, the luma modulation function can be based on a global reshape function. In some operating scenarios, the luma modulation function can be based on a local reshape function. The entire luma range of the input domain (or color space) is reshaped by Δ v The discrete slope in interval k is given by: About (or based on) global reshaping

number

number

[0218] The largest or highest slope among all k

number

number

[0219] The slopes for all bins can be normalized by the maximum slope as follows:

number

number

number

[0220] For a bin, the value

number

number

[0221] In some operating scenarios, the normalized slope

number

[0222] The maximum and minimum noise powers (e.g., user-specified, pre-configured, dynamically configurable, etc.) to be added to the above equation (42) are denoted by ψ max and ψ min Each input codeword v in the input image is expressed as i For , the applicable noise power represented by the luma modulation function can be designed or derived as follows:

number

number

[0223] To avoid or reduce computational costs, the luma modulation function can be derived in dependence on the slope calculated using the global reshape function in equation (46) above.

[0224] In some operating scenarios, a combination of local reshape function index dithering and film grain noise injection in the locally reshaped codeword values ​​can effectively avoid or reduce banding artifacts, especially when the output image is an 8-bit image.

[0225] Chroma Local Reshape In some operating scenarios, the output or reshaped chroma codewords of the output image can be generated from performing chroma local reshaping (e.g., across channels) on the input or pre-reshaped chroma codewords of the input image based on the input or pre-reshaped luma and chroma codewords of the input image.

[0226] The family of chroma local reshaping functions for chroma local reshaping can be generated in a number of different ways, such as self-derived local reshaping, pre-constructed local reshaping, or a combination or hybrid of the above.

[0227] To construct the chroma local reshaping mapping / function, MMR or TPB based techniques can be implemented or performed. The methods and / or processes used to construct or generate the operating parameters for the chroma local reshaping, such as the coefficients for both MMR based or TPB based local reshaping functions, are the same or similar to each other, with the difference (e.g., the only difference, the main difference, etc.) being the basis functions used in the optimization problem to obtain optimized values ​​for the coefficients. For MMR, polynomial basis functions / terms are used as basis functions in conjunction with the MMR coefficients. For TPB, tensor product B-spline basis functions are used as basis functions in conjunction with the TPB coefficients. Both MMR or TPB reshaping functions involve the construction of the 3DMT and the modification of the 3DMT.

[0228] Self-derived chroma local reshaping It may not be straightforward to directly derive local chroma reshaping functions from a global chroma reshaping function (e.g., free-form) because the chroma reshaping may use a cross-channel color predictor (e.g., MMR-based, TPB-based, etc.). Similar to the cross-channel luma local reshaping, the cross-channel chroma local reshaping for the input image can return to the original 3DMT associated with the global reshaping function or mapping and apply modifications to the original 3DMT to obtain local reshaping functions or mappings for local changes (e.g., locally increased saturation, locally decreased saturation, etc.). The 3DMT can be constructed with online and / or offline computation. In some operating scenarios, two global chroma reshaping functions for each of the two chroma channels Cb and Cr can be generated and fused to generate a local reshaping function for each of the two (e.g., fused) local reshaping functions for each of the two chroma channels Cb and Cr.

[0229] FIG. 2R shows an example flow for generating coefficients (e.g., MMR-based, TPB-based, etc.) for a chroma local reshaping function / mapping using self-derived chroma local reshaping.

[0230] Block 292 includes constructing a 3DMT for the global reshaping function or mapping, for example based on an image pair (e.g., an HDR image to be locally reshaped and a corresponding SDR image, etc.). The 3DMT can be constructed using the same or similar operations as those used for cross-channel luma local reshaping, such as those described in Tables (1)-(3) above.

[0231] As a result, for 3DMT, multiple 3DMT mapping pairs can be generated as shown in equation (19), which can then be used to derive a global reshaping function or mapping and to be modified or enhanced for the purpose of generating a second global chroma reshaping function or mapping.

[0232] Block 294 involves modifying the 3DMT or mapped values ​​in the mapping pair. C A function, denoted as (), is designed to map values ​​in each 3D histogram bin (indexed by a valid q with non-zero HDR pixel count) of the 3D histogram.

number

number

number

[0233] Function h C () first, Cb(C 0 ) and Cr (C 1 (e.g., relative to a neutral color point (0.5,0.5) instead of a non-neutral color origin (0,0)) in a Cartesian coordinate system of Cb / Cr (denoted as Cb / Cr=Cb / Cr+Cr+Cr+Cr+Cr+Cr+Cr+Cr+Cr+Cb / ...

number

[0234] Function h C () is expressed as follows: C,q Q,S

[0036] The method further includes applying a non-linear function or mapping to the saturation represented by:

number

[0235] As shown in equation (49) above, near-neutral colors (e.g., ρ C,q Q,s <δ C For colors such as RGB (e.g., RGB, RGB, RGBW, RGB, RGB, RGBW), linear mapping is used to avoid or prevent modification of near-neutral colors, and more specifically, their hues.

[0236] FIG. 3I illustrates an exemplary nonlinear function h that may be used to achieve the saturation change as specified in equation (49) above. C () is indicated.

[0237] Nonlinear function h C The modified shifted Cb / Cr values ​​generated from () can be transformed from the polar coordinate system back to the Cartesian coordinate system. The modified shifted Cb / Cr values ​​transformed to the Cartesian coordinate system can be inversely shifted to generate modified Cb / Cr values ​​by inversely adding an offset of 0.5 as follows:

number

[0238] The modified Cb / Cr values ​​in equation (50) above are used in part to form a number of modified 3DMT mapping pairs for the modified 3DMT as follows: can be done.

number

[0239] Block 296 includes constructing or generating first (e.g., MMR, TPB, etc.) coefficients for a global reshaping function from a plurality of (original) 3DMT mapping pairs, and constructing or generating second (e.g., MMR, TPB, etc.) coefficients for a second global reshaping function from a plurality of modified 3DMT mapping pairs in the modified 3DMT.

[0240] For example, based on the original 3DMT pairs constructed using the original colors, the first coefficient is the design matrix S Cx and vector

number

number

number

[0241] The optimized global reshaping solution or the optimized value for the first coefficient

number

number

[0242] Optimal value for the first coefficient of the global reshaping function

number

number

[0243] In these extended operations, the vector in (54) above for the global reshaping function

number

number

number

[0244] Similar to the global reshape function, the second coefficient for the second global reshape function

number

number

[0245] Block 298 involves constructing a local reshaping function (eg, fused, MMR-based, TPB-based, etc.).

[0246] Given the first and second global reshaping functions generated in block 296, θ <l>< / l> Cx,G1 a first global reshaping function weighting factor, denoted as θ <l>< / l> Cx,G2 A second global reshape function weighting factor, denoted as x, may be assigned to the first global reshape function and the second global reshape function, respectively. In some operating scenarios, the two weighting factors satisfy the following constraints / conditions:

number

[0247] Thus, the local reshaping described herein may be performed using a fused local reshape function generated by fusing a first coefficient for a global reshape function and a second coefficient for a second global reshape function as follows:

number

[0248] Pre-built cross-channel chroma local reshaping In some operating scenarios, in addition to or instead of the self-derived cross-channel local reshaping as described above, multiple pre-constructed cross-channel global reshaping functions F, each indexed by a global reshaping function index value denoted as g, may be used. <g>< / g> Cx () are obtained through offline training using the gth training dataset, which contains each SDR-HDR image pair.

[0249] For example, multiple 3DMT mapping pairs

number

number

[0250] The gth global reshape function can be obtained, for example using the least squares solution approach described above, as follows:

number

[0251] These pre-constructed global reshaping functions may be used, for example in the forward pass, to generate local reshaping functions, each indexed by a local reshaping function index value l, as follows: F <l>< / l> Cx ()=F <g>< / g> Cx () (62-1) Where: g=l (62-2)

[0252] Similarly, in the posterior pathway, B <l>< / l> Cx The local reshaping function, denoted as <g>< / g> Cx can be generated from a preconstructed global reshaping function denoted as: B <l>< / l> Cx ()=B <g>< / g> Cx () (63)

[0253] Hybrid of predefined and self-derived chroma local reshaping In some operating scenarios, a hybrid approach combining predefined and self-derived cross-channel chroma local reshaping may be used to generate the local chroma reshaping functions.

[0254] In some embodiments, two or more 3DMTs can be generated in offline training, each using one or more sets of training image pairs in the training data. Each training image pair in the two or more respective sets of training image pairs can include an input HDR image and a corresponding input SDR image that exhibits the same visual semantic content as the input HDR image. In one example, the corresponding input SDR image can be one that was used to generate the input HDR image (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.). In another example, the corresponding input SDR image can be one that was generated from the input HDR image (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.).

[0255] Each 3DMT in the two or more 3DMTs can be generated using a respective set of training image pairs in the training data. Image data, such as luma and chroma codewords, in each set of training image pairs can be used to generate statistics collected in the corresponding 3DMT. These statistics can be used to form a plurality of 3DMT mapping pairs for the corresponding 3DMT.

[0256] Each 3DMT in the two or more 3DMTs may be indexed by a respective index value (e.g., g) for lookup purposes. In some operating scenarios, the index value (e.g., g) that uniquely identifies a corresponding 3DMT in the two or more 3DMTs may be an L1 mid value.

[0257] Two or more pre-constructed global chroma reshaping functions can be constructed from the two or more (original, unmodified) 3DMTs, respectively, using the self-derived local reshaping approach as described above. The two or more 3DMTs can be enhanced or modified with the same saturation factor or two or more different saturation factors to generate two or more modified 3DMTs. Two or more pre-fused second global chroma reshaping functions can be constructed from the two or more modified 3DMTs, respectively, using the self-derived local reshaping approach as described above. Furthermore, the two or more global chroma reshaping functions from the two or more (original, unmodified) 3DMTs can be respectively fused with the two or more pre-fused second global chroma reshaping functions generated from the two or more modified 3DMTs to generate two or more pre-constructed local chroma reshaping functions, respectively, using the self-derived local reshaping approach as described above.

[0258] FIG. 2S shows an example flow for generating coefficients for chroma local reshaping functions / mappings (e.g., MMR-based, TPB-based, etc.) using a combination of predefined and self-derived chroma local reshaping.

[0259] Block 2092 includes constructing a 3DMT for a global reshaping function or mapping based on an image data subset, such as the gth set of training image pairs selected from the multiple sets of training image pairs. The 3DMT can be constructed using operations the same or similar to those used for cross-channel luma local reshaping, such as those described in Tables (1)-(3) above.

[0260] As a result, multiple 3DMT mapping pairs as shown in equation (19) can be generated for the 3DMT as follows:

number

[0261] The multiple 3DMT mapping pairs in equation (64) can then be used to derive a global reshaping function or mapping, as well as to be modified or extended for purposes of generating a second chroma reshaping function or mapping. Block 2094 includes modifying the 3DMTs or mapped values ​​in the mapping pairs to generate modified 3DMTs.

[0262] To achieve a change in saturation, h C A function, denoted as (), is designed to map values ​​at each 3D histogram bin (indexed by a valid q with non-zero HDR pixel count) of the 3D histogram as follows:

number

number

number

[0263] Function h C () first, Cb(C 0 ) and Cr(C 1 (e.g., relative to a neutral color point at (0.5,0.5) instead of the non-neutral color origin (0,0)) to a polar coordinate system with polar radius ρ and polar angle θ as follows:

number

[0264] Function h C () is the polar radius ρ as shown below. C,q Q,s <g>< / g>

[0036] The method further includes applying a non-linear function or mapping to the saturation represented by:

number

[0265] As shown in equation (67) above, for near-neutral colors (e.g., ρ C,q Q,S, <g>< / g> <δ C etc.), a linear mapping is used to avoid or prevent altering (or, more specifically, altering) colors that are close to neutral.

[0266] Nonlinear function h C The modified shifted Cb / Cr values ​​produced from () can be transformed from the polar coordinate system back to the Cartesian coordinate system. The modified shifted Cb / Cr values ​​transformed to the Cartesian coordinate system can be inversely shifted by inversely adding an offset of 0.5 to produce the modified Cb / Cr values, as follows:

number

[0267] The modified Cb / Cr values ​​in equation (68) above can be used to form multiple modified 3DMT mapping pairs within the modified 3DMT as follows:

number

[0268] Block 2096 includes constructing or generating first (e.g., MMR, TPB, etc.) coefficients for a global reshaping function from the plurality of (original) 3DMT mapping pairs, and constructing or generating second (e.g., MMR, TPB, etc.) coefficients for a second global reshaping function from the plurality of modified 3DMT mapping pairs.

[0269] For example, based on the original 3DMT pairs constructed using the original colors, the first coefficient is the design matrix S Cx and vector

number

number

number

[0270] The optimized global reshaping solution or the optimized value for the first coefficient

number

number

[0271] Optimal value for the first coefficient of the global reshaping function

number

number

[0272] In these extended operations, the vector in equation (24) for the global reshaping function

number

number

number

[0273] Optimized local reshaping solution or optimized value for the second coefficient

number

number

[0274] Block 2098 involves constructing a local reshaping function (eg, fused, MMR-based, TPB-based, etc.).

[0275] Given the two global reshaping functions generated in block 2096, θ <l>< / l> Cx,G1 The first global reshaping function weighting factor, θ <l>< / l> Cx,G2A second global reshape function weighting factor, denoted as x, may be assigned to the first global reshape function and the second global reshape function, respectively. In some operating scenarios, the two weighting factors satisfy the following constraints / conditions:

number

[0276] Thus, the local reshaping described herein may be performed using a fused local reshape function generated by fusing a first coefficient for a global reshape function and a second coefficient for a second global reshape function as follows:

number

[0277] For illustrative purposes only, the mid-L1 values ​​predicted or estimated from a 12-bit (offline) training input image are used in the above equations (64) to (76). <g>Given a bit depth of 12 bits, the index value <g>can be a value selected from the value range [0,4095]. Similarly, given a bit depth of 10 bits, the index value <g>can be a value selected from the value range [0,1023].

[0278] Similarly, the mid-L1 values ​​predicted or estimated from a 12-bit (e.g., actual, non-training, to-be-enhanced, online-processed, etc.) input image are expressed as <l>Given a bit depth of 12 bits, the index value <l>can be a value selected from the value range [0,4095]. Similarly, given a bit depth of 10 bits, the index value <l>can be a value selected from the value range [0,1023].

[0279] Index Value <g>and / or <l>The memory space, volatile or non-volatile storage space, etc. used to store operational parameters such as MMR or TPB coefficients for all reshaping functions / mappings for all possible combinations of can be relatively large and expensive.

[0280] In some operating scenarios, the index value <g>and / or <l>Only operational parameters such as MMR or TPB coefficients for a proper subset of index values ​​(e.g., representative, etc.) in all possible combinations of are calculated or generated offline and stored / cached in memory space or storage. For example, the entire value range of all possible combinations can be partitioned or divided into multiple sub-ranges (e.g., 16, 64, 128, 256, etc.). Representative index values <g>or <l>is selected for each subrange of the multiple subranges (e.g., evenly, at a particular position, every 16, every 64, every 128, every 256, etc.). Representative index values <g>(and the corresponding <l>Operational parameters such as MMR or TPB coefficients for

number

[0281] During system boot-up of an image processing system that performs a local reshaping operation on an input image, a representative index value <g>(and the corresponding <l>Operational parameters such as MMR or TPB coefficients for

number

[0282] At run time, local reshaping as described herein can be applied to any index value in a relatively wide range of values, such as [0,4095], [0,1023], etc. <l>This can be done for an input image with pre-generated index values <g>(and the corresponding <l>For index values ​​not covered by the representative index values, the operating parameters such as MMR or TPB coefficients may not be readily available from memory space or storage at the beginning of system boot-up. <l>For the pre-generated index value <g>and / or <l>The interpolation operation may be performed after available operational parameters, such as MMR or TPB coefficients for the index values ​​covered by (e.g., the two closest, etc.) have been retrieved, loaded, or otherwise made available.

[0283] FIG. 2T shows an example flow for 3DMT-based chroma local reshaping function or mapping generation. For illustrative purposes, an MMR or TPB-based chroma local reshaping function is generated to enhance the local saturation of an input HDR image 2192. The chroma local reshaping function (or mapping) uses δ to achieve local saturation changes and soft clipping. C , M C , β, θ <l>< / l> Cx,G1 The parameter may be generated at least in part based on the enhancement parameters 2194 such as:

[0284] Block 2196 includes constructing a 3DMT using a 3D histogram having bins that store counts of HDR pixels in the input HDR image (2192) and averages of corresponding input SDR images in an image pair formed by the input HDR image and the input SDR image.

[0285] In one example, the corresponding input SDR image may have been used to generate the HDR image (2192) (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.). In another example, the corresponding input SDR image may have been generated from the HDR image (2192) (e.g., through previously performed content mapping, tone mapping, and / or reshaping operations, etc.).

[0286] Block 2198 includes constructing a global chroma reshape function using 3DMT. For example, first MMR or TPB coefficients may be generated to specify the global chroma reshape function using a complete set of MMR terms or tensor product B-spline basis functions.

[0287] Block 2200 calculates an enhancement parameter δ for local saturation changes. C , M C , β to turn the 3DMT into a respective modified 3DMT that includes the modified mapping pairs.

[0288] Block 2202 includes constructing a second global chroma reshape function using the modified 3DMT. The second global chroma reshape function can be fused with the first global reshape function. For example, second MMR or TPB coefficients may be generated to specify the global reshape function using MMR terms or tensor product B-spline basis functions.

[0289] Block 2204 calculates an improvement parameter θ <l>< / l> Cx,G1 , θ <l>< / l> Cx,G2 Block 2205 includes performing a weighted fusion using both the global chroma reshaping functions derived from the original 3DMT and the modified 3DMT to generate a corresponding fused cross-channel local reshaping function. Block 2206 includes outputting the (fused) cross-channel local reshaping function.

[0290] Similar to local luma reshaping, local chroma reshaping can use a guidance image with index values ​​to identify a specific local chroma reshaping function / mapping for specific input pixels of the input image. There are different ways to construct a guidance image for local chroma reshaping. In a first example, the guide image can be a luma independent guide image constructed based on the chroma codeword in the input image, or a saturation image calculated from the input image. In a second example, the guide image can be constructed based on the luma and chroma codewords in the input image. In a third example, the same guide image for local luma reshaping can be used as or to derive the guidance image for local chroma reshaping. Depending on the color sampling format, such as 444 or 420, the guide image for local luma reshaping can be downsampled accordingly to fit the chroma channel image size. In many operating scenarios, the luma guided image can be reused for chroma reshaping, providing a better appearance, including enhanced local color saturation, in the locally reshaped image without introducing visual artifacts.

[0291] Exemplary Process Flow 4 illustrates an exemplary process flow according to an embodiment. In some embodiments, one or more computing devices or components (e.g., an encoding device / module, a transcoding device / module, a decoding device / module, an inverse tone mapping device / module, a tone mapping device / module, a media device / module, an inverse mapping generation and application system, etc.) may perform this process flow. At block 402, an image processing system performs a first reshaping mapping on a first image represented in a first domain to generate a second image represented in a second domain. The first domain has a first dynamic range that is different from a second dynamic range of the second domain.

[0292] At block 404, the image processing system performs a second reshaping mapping on the second image represented in the second domain to generate a third image represented in the first domain, the third image being perceptually different from the first image in at least one of global contrast, global saturation, local contrast, local saturation, etc.

[0293] In block 406, the image processing system causes a display image derived from the third image to be rendered on a display device.

[0294] In one embodiment, the first reshape mapping and the second reshape mapping form one of the following: (a) a combination of a global forward reshape mapping and a global backward reshape mapping; (b) a combination of a global backward reshape mapping and a global forward reshape mapping; (c) a combination of a global forward reshape mapping and a local backward reshape mapping; (d) a combination of a global backward reshape mapping and a local forward reshape mapping; (e) a combination of a local backward reshape mapping and a global forward reshape mapping; (f) a combination of a local forward reshape mapping and a global backward reshape mapping; (g) a combination of a local forward reshape mapping and a local backward reshape mapping; (h) a combination of a local backward reshape mapping and a local forward reshape mapping; etc.

[0295] In an embodiment, the first dynamic range for the first image and the second dynamic range for the second image form one of: a combination of high dynamic range (HDR) for the first image and standard dynamic range (SDR) for the second image; a combination of SDR for the first image and HDR for the second image; and the like.

[0296] In one embodiment, an encoded image is generated from a third reshaping mapping performed on the second image; the encoded image is encoded in a video signal received by a receiving device; and the receiving device generates a display image from a decoded version of the encoded image received with the video signal.

[0297] In one embodiment, a first image represented in a first domain is generated and uploaded by a mobile device.

[0298] In an embodiment, at least one of the first reshape mapping and the second reshape mapping includes a luma local reshape mapping.

[0299] In one embodiment, the luma local reshaping mapping represents one of: (a) a single-channel luma local reshaping mapping that generates an output luma codeword from an input luma codeword independently of the input chroma codeword; (b) a cross-channel luma local reshaping mapping that generates an output luma codeword from both the input luma codeword and the input chroma codeword;

[0300] In one embodiment, the luma local reshaping mapping represents a cross-channel luma local reshaping mapping; the cross-channel luma local reshaping mapping is generated by fusing the cross-channel luma global reshaping mapping and the pre-fused cross-channel luma local reshaping mapping; the cross-channel luma global reshaping is generated using a three-dimensional mapping table (3DMT) computed using the codewords in the image pair; and the pre-fused cross-channel luma local reshaping mapping is generated using a modified 3DMT obtained by modifying the 3DMT with a local contrast enhancement function.

[0301] In an embodiment, at least one of the first reshape mapping and the second reshape mapping includes a chroma-local reshape mapping.

[0302] In one embodiment, the chroma local reshaping mapping represents one of: (a) a cross-channel MMR chroma local reshaping mapping that generates an output chroma codeword from an input luma codeword and an input chroma codeword; (b) a cross-channel TPB chroma local reshaping mapping that generates an output chroma codeword from both an input luma codeword and an input chroma codeword;

[0303] In one embodiment, the chroma local reshaping mapping is generated by blending a first cross-channel chroma global reshaping mapping and a second cross-channel chroma global reshaping mapping; the first cross-channel chroma global reshaping mapping is generated using a three-dimensional mapping table (3DMT) computed using codewords in the image pair; and the second cross-channel chroma global reshaping mapping is generated using a modified 3DMT derived by modifying the 3DMT with a saturation enhancement function.

[0304] In one embodiment, image filtering is applied to at least one of a first image, a second image, or an image derived from the second image using a noise-injected guide image to reduce banding artifacts; the noise-guided image includes per-pixel noise-injected local reshape function indices; a filtered image includes codewords generated from the image filtering with the noise-injected guide image; and the codewords in the filtered image are applied with further noise injection to generate noise-injected codewords.

[0305] In an embodiment, the second image is received by a video encoder as an input image in a sequence of input images; the sequence of input images received by the video encoder is encoded by the video encoder into a video signal.

[0306] In some embodiments, a computing device, such as a display device, a mobile device, a set-top box, a multimedia device, etc., is configured to perform any of the aforementioned methods. In some embodiments, an apparatus includes a processor and is configured to perform any of the aforementioned methods. In some embodiments, a non-transitory computer-readable storage medium storing software instructions that, when executed by one or more processors, cause the execution of any of the aforementioned methods.

[0307] In one embodiment, a computing device comprises one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, cause performance of any of the aforementioned methods.

[0308] Although separate embodiments are discussed herein, it should be noted that any combination of the embodiments and / or sub-embodiments discussed herein may be combined to form further embodiments.

[0309] Exemplary Computer System Implementation Embodiments of the present invention may be implemented by a computer system, a system configured in electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices or components. The computer and / or IC may execute, control, or execute instructions relating to adaptive perceptual quantization of images having enhanced dynamic range as described herein. The computer and / or IC may calculate any of the various parameters or values ​​associated with the adaptive perceptual quantization process described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0310] Certain implementations of the present invention include a computer processor executing software instructions that cause the processor to perform the methods of the present disclosure. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. may implement methods related to adaptive perceptual quantization of HDR images as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention may be provided in the form of a program product. The program product may include any non-transitory medium carrying a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the methods of the embodiments of the present invention. A program product according to an embodiment of the present invention may be in any of a wide variety of forms. The program product may include a physical medium, such as, for example, a magnetic data storage medium including a floppy diskette, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0311] When a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, the reference to that component (including the reference to a "means") should be interpreted as including any component that performs the function of the described component (e.g., is functionally equivalent) as an equivalent of that component, including components that are not structurally equivalent to the disclosed structures that perform that function in exemplary embodiments of the invention.

[0312] According to an embodiment, the techniques described herein are implemented by one or more special purpose computing devices. The special purpose computing devices may be fixedly configured to perform the techniques, or may include digital electronic devices such as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) permanently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or combinations. Such special purpose computing devices may also combine custom fixed configuration logic, ASICs, or FPGAs with custom programming to achieve the techniques. The special purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices incorporating fixed configuration and / or program logic to implement these techniques.

[0313] 5 is a block diagram illustrating a computer system 500 upon which an exemplary embodiment of the present disclosure may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0314] Computer system 500 also includes a main memory 506, coupled to bus 502, such as a random access memory (RAM) or other dynamic storage device, for storing information and instructions to be executed by processor 504. Main memory 506 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, render computer system 500 into a special purpose machine customized to perform the operations specified in the instructions.

[0315] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic or optical disk, semiconductor RAM, is provided and coupled to bus 502 for storing information and instructions.

[0316] Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to bus 502 for communicating information and command selections to processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 504 and for controlling cursor movement on display 512. The input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), allowing the device to specify a position in a plane.

[0317] Computer system 500 may use customized fixed logic, one or more ASICs or FPGAs, firmware and / or program logic that in combination with the computer system configures or programs computer system 500 as a special purpose machine to implement the techniques described herein. According to one embodiment, the techniques are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, fixed circuitry may be used in place of or in combination with software instructions.

[0318] The term "storage media" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs and EPROMs, flash EPROMs, NVRAMs, any other memory chips or cartridges.

[0319] Storage media is distinct from, but may be used in conjunction with, transmission media, which participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0320] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 can receive the data on the telephone line and convert the data to an infrared signal using an infrared transmitter. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus 502. The bus 502 carries the data to main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by processor 504.

[0321] Computer system 500 also includes a communication interface 518 coupled to bus 502. The communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, the communication interface 518 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 may be a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0322] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526 provides data communication services through the world-wide packet data communication network now commonly referred to as the "Internet" 528. Local network 522 and Internet 528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 520 and through communication interface 518, which carry the digital data to and from computer system 500, are exemplary forms of transmission media.

[0323] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0324] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0325] 7. Equivalents, Extensions, Substitutions, etc. In the foregoing specification, embodiments of the present disclosure have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what is an embodiment of the present invention and what is intended by the applicant to be a claimed embodiment of the present invention is the specific form in which the claims of the patent granted on this application, including any subsequent amendments thereto, are patented. The definitions, if any, expressly set forth herein for the terms contained in such claims shall govern the meaning of such terms used in the claims. Thus, any limitations, elements, characteristics, features, advantages or attributes not expressly set forth in the claims should not limit the scope of such claims in any manner. Thus, the specification and drawings should be regarded in an illustrative and not restrictive sense.

[0326] LISTED ILLUSTRATIVE EMBODIMENTS The present invention may be embodied in any of the forms described herein, including, but not limited to, the following enumerated example embodiments (EEE) that describe the structure, features, and functions of some portions of the embodiments of the present invention.

[0327] [EEE1] 1. A method of enhancing an image, comprising: performing a first reshaping mapping on a first image represented in a first domain to generate a second image represented in a second domain, the first domain having a first dynamic range different from a second dynamic range of the second domain; performing a second reshaping mapping on the second image represented in the second domain to generate a third image represented in the first domain, the third image perceptually different from the first image in at least one of global contrast, global saturation, local contrast, or local saturation; and causing a display image derived from the third image to be rendered on a display device. method.

[0328] [EEE2] The method of EEE1, wherein the first reshaping mapping and the second reshaping mapping form one of: (a) a combination of a global forward reshaping mapping and a global backward reshaping mapping; (b) a combination of a global backward reshaping mapping and a global forward reshaping mapping; (c) a combination of a global forward reshaping mapping and a local backward reshaping mapping; (d) a combination of a global backward reshaping mapping and a local forward reshaping mapping; (e) a combination of a local backward reshaping mapping and a global forward reshaping mapping; (f) a combination of a local forward reshaping mapping and a global backward reshaping mapping; (g) a combination of a local forward reshaping mapping and a local backward reshaping mapping; (h) a combination of a local backward reshaping mapping and a local forward reshaping mapping.

[0329] [EEE3] The method of claim 1 or 2, wherein the first dynamic range for the first image and the second dynamic range for the second image form one of: a combination of a high dynamic range (HDR) for the first image and a standard dynamic range (SDR) for the second image; or a combination of SDR for the first image and HDR for the second image.

[0330] [EEE4] The method of any one of EEE1 to 3, wherein an encoded image is generated from a third reshaping mapping performed on the second image; the encoded image is encoded in a video signal received by a receiving device; and the receiving device generates the display image from a decoded version of the encoded image received with the video signal.

[0331] [EEE5] The method of any one of EEE1 to 4, wherein the first image represented in the first domain is generated and uploaded by a mobile device.

[0332] [EEE6] The method of any one of EEE1 to EEE5, wherein at least one of the first reshaping mapping and the second reshaping mapping comprises a luma local reshaping mapping.

[0333] [EEE7] The method of claim 6, wherein the luma local reshaping mapping represents one of: (a) a single-channel luma local reshaping mapping that generates output luma codewords from input luma codewords independent of the input chroma codewords; or (b) a cross-channel luma local reshaping mapping that generates output luma codewords from both input luma codewords and input chroma codewords.

[0334] [EEE8] The method of claim 7, wherein the luma local reshaping mapping represents the cross-channel luma local reshaping mapping; the cross-channel luma local reshaping mapping is generated by fusing a cross-channel luma global reshaping mapping and a pre-fused cross-channel luma local reshaping mapping; the cross-channel luma global reshaping is generated using a three-dimensional mapping table (3DMT) computed using codewords in an image pair; and the pre-fused cross-channel luma local reshaping mapping is generated using a modified 3DMT derived from modifying the 3DMT with a local contrast enhancement function.

[0335] [EEE9] The method of any one of EEE1 to 8, wherein at least one of the first reshaping mapping and the second reshaping mapping comprises a chroma-local reshaping mapping.

[0336] [EEE10] The method of claim 9, wherein the chroma local reshaping mapping represents one of: (a) a cross-channel multivariate multiple regression (MMR) chroma local reshaping mapping that generates output chroma codewords from input luma codewords and input chroma codewords; or (b) a cross-channel tensor product B-spline (TPB) chroma local reshaping mapping that generates output chroma codewords from both input luma codewords and input chroma codewords.

[0337] [EEE11] The method of any one of claims 9 to 10, wherein the chroma local reshaping mapping is generated by blending a first cross-channel chroma global reshaping mapping and a second cross-channel chroma global reshaping mapping; the first cross-channel chroma global reshaping mapping is generated using a three-dimensional mapping table (3DMT) calculated using codewords in an image pair; and the second cross-channel chroma global reshaping mapping is generated using a modified 3DMT derived from modifying the 3DMT with a local saturation enhancement function.

[0338] [EEE12] 12. The method of any one of EEE1 to 11, wherein image filtering is applied to at least one of the first image, the second image, or an image derived from the second image using a noise-injected guide image to reduce banding artifacts; the noise-guided image comprises per-pixel noise-injected local reshaping function indices; a filtered image comprises codewords generated from the image filtering using the noise-injected guide image; and the codewords in the filtered image are applied with further noise injection to generate noise-injected codewords.

[0339] [EEE13] The method of any one of EEE1 to 12, wherein the second image is received by a video encoder as an input image in a sequence of input images; and the sequence of input images received by the video encoder is encoded by the video encoder into a video signal.

[0340] [EEE14] A method for enhancing an image, comprising: constructing a three-dimensional mapping table (3DMT) from one or more first images encoded with a first codeword in a first codeword space and one or more second images encoded with a second codeword in a second codeword space, the one or more first images corresponding to the one or more second images, the first codeword space being divided into a plurality of bins, the 3DMT storing a plurality of mapping pairs, each of the mapping pairs corresponding to a respective bin in the plurality of bins, the 3DMT including: (a) a count value of first codewords located in the respective bins in the one or more first images; and (b) an average second luma codeword value of second codewords to which the first codewords correspond in the one or more second images; applying a local contrast enhancement operation to the average second luma codeword value in each mapping pair in the plurality of mapping pairs to generate a modified 3DMT; generating a global luma reshaping function from the 3DMT and a local luma reshaping function from the modified 3DMT; generating a final local luma reshape function from the global luma reshape function and the local luma reshape function; applying the final local luma reshaping function to an input image to generate a locally contrast enhanced image, from which a display image is derived and rendered on a display device. method.

[0341] [EEE14] A method for enhancing an image, comprising: constructing a three-dimensional mapping table (3DMT) from one or more first images encoded with a first codeword in a first codeword space and one or more second images encoded with a second codeword in a second codeword space, the one or more first images corresponding to the one or more second images, the first codeword space being divided into a plurality of bins, the 3DMT storing a plurality of mapping pairs, each of the mapping pairs corresponding to a respective bin in the plurality of bins, the mapping pairs including: (a) a count value of a first codeword located in the respective bin in the one or more first images; and (b) an average chroma codeword value of a second codeword to which the first codeword corresponds in the one or more second images; applying one or more local saturation enhancement operations to the average chroma codeword value in each mapping pair in the plurality of mapping pairs to generate a modified 3DMT; generating, for each chroma channel, a global chroma reshaping function from the 3DMT and a second global chroma reshaping function from the modified 3DMT; generating, for each chroma channel, a final local chroma reshaping function from the global chroma reshaping function and the second global chroma reshaping function; and applying, for each chroma channel, the final local chroma reshaping function to an input image to generate a locally desaturated image, from which a display image is derived and rendered on a display device. method.

[0342] [EEE15] The method of any one of EEE1 to EEE14, wherein the image filtering represents guide image filtering applied using a guide image, the guide image including high frequency feature values ​​calculated for a plurality of pixel locations, each weighted by an inverse of an image gradient calculated for a plurality of pixel locations in one of the one or more channels of the guide image.

[0343] [EEE16] 16. The method of any one of EEE1 to 15, wherein the guide image comprises guide image values ​​derived based at least in part on a set of halo reduction operational parameters, the method further comprising: calculating, for each image pair of a plurality of image pairs, a region-based texture measure in one or more local foreground areas using a plurality of sets of candidate values ​​for the set of halo reduction operational parameters as a ratio of variances of a training image of the first dynamic range and a corresponding training image of the second dynamic range in the image pair, the one or more local foreground areas being sampled from a foreground mask identified in the training image of the first dynamic range in the image pair; calculating, for each image pair of the plurality of image pairs, a halo measure for one or more local halo artifact areas as a correlation coefficient between a training image of the first dynamic range and a corresponding training image of the second dynamic range in the image pair using the plurality of sets of candidate values ​​for the set of halo reduction operational parameters, the one or more local halo artifact areas being sampled from a halo mask identified in the corresponding training image of the second dynamic range; calculating, for the plurality of sets of candidate values ​​for the set of halo reduction operational parameters, a respective weighted sum of: (a) all region-based texture measures for all local foreground areas sampled from a foreground mask identified in a training image of the first dynamic range in the plurality of image pairs; and (b) all halo measures for all local halo artifact areas sampled from a halo mask identified in a corresponding training image of the second dynamic range in the plurality of image pairs; determining a set of optimized values ​​for the set of halo reduction operational parameters, the set of optimized values ​​being used to generate a minimized weighted sum of (a) all of the region-based texture measures for all local foreground areas sampled from foreground masks identified in training images of the first dynamic range in the plurality of image pairs, and (b) all of the halo measures for all local halo artifact areas sampled from halo masks identified in corresponding training images of the second dynamic range in the plurality of image pairs; applying the image filtering at a plurality of spatial kernel size levels to the input image using the set of optimized values ​​for the set of halo reduction operating parameters. method.

[0344] [EEE16] A computer system configured to perform any one of the methods described in any one of claims EEE1 to 15.

[0345] [EEE17] An apparatus having a processor and configured to carry out a method according to any one of EEE1 to EEE15.

[0346] [EEE18] A non-transitory computer-readable storage medium storing computer-executable instructions for performing the methods described in any one of EEE1 to EEE15.< / l> < / g> < / l> < / l> < / g> < / l> < / l> < / g> < / l> < / g> < / l> < / g> < / l> < / g> < / l> < / g> < / l> < / l> < / l> < / g> < / g> < / g>

Claims

1. 1. A method for enhancing an image, comprising: performing (402) a first reshape mapping on a first image represented in a first domain to generate a second image represented in a second domain, the first domain having a first dynamic range that is different from a second dynamic range of the second domain; performing (404) a second reshape mapping on the second image represented in the second domain to generate a third image represented in the first domain, the third image differing from the first image in at least one of local contrast or local saturation, the difference in local contrast and / or local saturation being caused by at least one of the first and second reshape mappings represented by a function selectable at a pixel level of the respective image; and causing (406) a display image derived from the third image to be rendered on a display device (140, 140-1); At least one of the first reshaping mapping and the second reshaping mapping includes a luma local reshaping mapping; the luma local reshaping mapping represents a cross-channel luma local reshaping mapping that generates output luma codewords from both input luma codewords and input chroma codewords; the cross-channel luma local reshaping mapping is generated by blending (268, 268', 284) a cross-channel luma global reshaping mapping and an initial cross-channel luma local reshaping mapping; blending a cross-channel luma local reshaping mapping includes computing a linear combination of the cross-channel luma global reshaping mapping and an initial cross-channel luma local reshaping mapping; the cross-channel luma global reshaping mapping is generated using a three-dimensional mapping table (3DMT) computed using codewords in an image pair represented by the first and second images; and the initial cross-channel luma local reshaping mapping is generated using a modified 3DMT derived from modifying the 3DMT with a local contrast enhancement function. method.

2. At least one of the first reshape mapping and the second reshape mapping includes a chroma local reshape mapping; the chroma local reshape mapping is generated by blending (298, 2098, 2204) a first cross-channel chroma global reshape mapping and a second cross-channel chroma global reshape mapping; blending the first cross-channel chroma global reshaping mapping and the second cross-channel chroma global reshaping mapping includes computing a linear combination of the first cross-channel chroma global reshaping mapping and the second cross-channel chroma global reshaping mapping; 2. The method of claim 1, wherein the first cross-channel chroma global reshape mapping is generated using a 3DMT computed using codewords in an image pair represented by the first and second images; and the second cross-channel chroma global reshape mapping is generated using a modified 3DMT derived from modifying the 3DMT with a local saturation enhancement function.

3. A method for enhancing an image, comprising: performing (402) a first reshape mapping on a first image represented in a first domain to generate a second image represented in a second domain, the first domain having a first dynamic range that is different from a second dynamic range of the second domain; performing (404) a second reshape mapping on the second image represented in the second domain to generate a third image represented in the first domain, the third image differing from the first image in at least one of local contrast or local saturation, the difference in local contrast and / or local saturation being caused by at least one of the first and second reshape mappings represented by a function selectable at a pixel level of the respective image; and causing (406) a display image derived from the third image to be rendered on a display device (140, 140-1); At least one of the first reshape mapping and the second reshape mapping includes a chroma local reshape mapping; the chroma local reshape mapping is generated by blending (298, 2098, 2204) a first cross-channel chroma global reshape mapping and a second cross-channel chroma global reshape mapping; blending the first cross-channel chroma global reshaping mapping and the second cross-channel chroma global reshaping mapping includes computing a linear combination of the first cross-channel chroma global reshaping mapping and the second cross-channel chroma global reshaping mapping; the first cross-channel chroma global reshape mapping is generated using a 3DMT computed using codewords in an image pair represented by the first and second images; and the second cross-channel chroma global reshape mapping is generated using a modified 3DMT derived from modifying the 3DMT with a local saturation enhancement function. method.

4. and constructing (262) the 3DMT used to generate the cross-channel luma global reshaping mapping from one or more first images encoded with a first codeword in a first codeword space and one or more second images encoded with a second codeword in a second codeword space, the one or more first images corresponding to the one or more second images, the first codeword space being divided into a plurality of bins, the 3DMT storing a plurality of mapping pairs, each of the mapping pairs corresponding to a respective bin in the plurality of bins, the 3DMT including: (a) a count value of first codewords located in the respective bins in the one or more first images; and (b) an average second luma codeword value of second codewords to which the first codewords correspond in the one or more second images.

4. The method according to any one of claims 1 to 3.

5. and constructing the 3DMT used to generate the first cross-channel chroma global reshaping mapping from one or more first images encoded with a first codeword in a first codeword space and one or more second images encoded with a second codeword in a second codeword space, the one or more first images corresponding to the one or more second images, the first codeword space being divided into a plurality of bins, the 3DMT storing a plurality of mapping pairs, each of the mapping pairs corresponding to a respective bin in the plurality of bins, the 3DMT including: (a) a count value of first codewords located in the respective bins in the one or more first images; and (b) an average chroma codeword value of second codewords to which the first codewords correspond in the one or more second images. The method according to claim 2 or 3.

6. 6. The method of claim 5, wherein the chroma local reshaping mapping represents one of: (a) a cross-channel multivariate multiple regression (MMR) chroma local reshaping mapping that generates output chroma codewords from input luma codewords and input chroma codewords; or (b) a cross-channel tensor product B-spline (TPB) chroma local reshaping mapping that generates output chroma codewords from both input luma codewords and input chroma codewords.

7. The first reshape mapping and the second reshape mapping are: (a) a combination of a global forward reshape mapping and a global backward reshape mapping; (b) a combination of a global backward reshape mapping and a global forward reshape mapping; (c) a combination of a global forward reshape mapping and a local backward reshape mapping; (d) a combination of global posterior reshaping mapping and local anterior reshaping mapping; (e) a combination of local posterior reshaping mapping and global anterior reshaping mapping; (f) combination of local forward reshaping mapping and global backward reshaping mapping; 7. A method according to claim 1, further comprising forming one of the following: (g) a combination of a local forward reshaping mapping and a local backward reshaping mapping; (h) a combination of a local backward reshaping mapping and a local forward reshaping mapping.

8. 8. The method according to claim 1 , wherein the first dynamic range for the first image and the second dynamic range for the second image form one of: a combination of a high dynamic range (HDR) for the first image and a standard dynamic range (SDR) for the second image; or a combination of SDR for the first image and HDR for the second image.

9. 9. The method of claim 1, wherein an encoded image is generated from a third reshaping mapping performed on the second image; the encoded image is encoded in a video signal received by a receiving device; and the receiving device generates the display image from a decoded version of the encoded image received with the video signal.

10. The method of claim 1 , wherein the first image represented in the first domain is generated and uploaded by a mobile device.

11. 11. The method of claim 1, further comprising: applying image filtering to at least one of the first image, the second image, or an image derived from the second image using a noise-injected guide image to reduce banding artifacts; the image filtering comprises generating a guide image from the image to be filtered, the guide image providing local reshape function index values ​​for pixels of the image to be filtered; the noise-injected guide image comprises pixel-by-pixel noise-injected local reshape function indexes calculated from the guide image; a filtered image comprises code words generated from the image filtering using the noise-injected guide image; and the code words in the filtered image are applied with further noise injection to generate noise-injected code words.

12. 12. A method according to any one of claims 1 to 11, wherein the second image is received by a video encoder as an input image in a sequence of input images; and the sequence of input images received by the video encoder is encoded by the video encoder into a video signal.

13. An apparatus (500) comprising one or more processors (504) and configured to perform the method of any one of claims 1 to 12.

14. A non-transitory computer readable storage medium (506, 510) storing computer executable instructions for executing the method of any one of claims 1 to 12 on one or more processors (504).

Images