Cascade prediction

Abstract

A first predictor is applied to an input image to generate a first level prediction codeword that is approximately a prediction target codeword of the prediction target image. A second level prediction target value is created by performing an inverse cascade operation on the prediction target codeword and the first level prediction codeword. A second predictor is applied to the input image to generate a second level prediction value that is approximately the second level prediction target value. Multiple sets of cascaded prediction coefficients are generated to include a first set of cascaded prediction coefficients and a second set of cascaded prediction coefficients that specify the first predictor and the second predictor. The multiple sets of cascaded prediction coefficients are encoded into a video signal as image metadata. The video signal is further encoded with the input image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to European Patent Application No. 19214124.0 filed on December 6, 2019 and U.S. Provisional Patent Application No. 62 / 944,636 filed on December 6, 2019, both of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to image processing operations. More specifically, embodiments of the present disclosure relate to video codecs. Background Art

[0004] As used herein, the term "dynamic range (DR)" may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., brightness, luminance) in an image, e.g., from the darkest black (dark colors) to the brightest white (highlights). In this sense, DR is related to "scene-referred" intensities. DR may also relate to the ability of a display device to adequately or approximately render a range of intensities of a particular breadth. In this sense, DR is related to "display-referred" intensities. Unless a particular sense is expressly specified at any point in the description herein to have a particular meaning, it should be inferred that the terms may be used in either sense, e.g., interchangeably.

[0005] As used herein, the term "high dynamic range (HDR)" refers to a DR width that spans approximately 14 to 15 or more orders of magnitude across the human visual system (HVS). In practice, the DR, over a wide range of intensities that humans can simultaneously perceive, may be slightly truncated relative to HDR. As used herein, the terms "enhanced dynamic range (EDR) or visual dynamic range (VDR)" may be individually or interchangeably related to such a DR: the DR that can be perceived by the human visual system (HVS) including eye movement within a scene or image, allowing for some light-adaptive changes in the scene or image. As used herein, EDR may refer to a DR that spans 5 to 6 orders of magnitude. Therefore, while HDR may be slightly narrower relative to a real scene reference, EDR may represent a wide DR width and may also be referred to as HDR.

[0006] In practice, an image includes one or more color components (e.g., luminance Y and chrominance Cb and Cr) of a color space, wherein each color component is represented by n bits of precision per pixel (e.g., n=8). Using nonlinear light intensity coding (e.g., gamma coding), an image where n≤8 (e.g., a color 24-bit JPEG image) is considered an image of standard dynamic range, while an image where n>8 can be considered an image of enhanced dynamic range.

[0007] The reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between the color values ​​(e.g., luminance) of an input video signal and the output screen color values ​​(e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT.1886, "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011), defines a reference EOTF for flat panel displays, the contents of which are incorporated herein by reference in their entirety. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. The term "metadata" herein refers to any auxiliary information that is transmitted as part of the encoded bitstream and that assists a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters as described herein.

[0008] As used herein, the term "PQ" refers to the quantization of the perceived brightness amplitude. The human visual system responds to the increased light level in a very nonlinear manner. The ability of humans to observe stimuli is affected by the following factors: the brightness of the stimuli, the size of the stimuli, the spatial frequency that constitutes the stimuli, and the brightness level that the eyes adapt to at the specific moment of viewing the stimuli. In some embodiments, the perception quantizer function maps the linear input grayscale to the output grayscale that better matches the contrast sensitivity threshold in the human visual system. In SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays" (hereinafter referred to as "SMPTE"), an example PQ mapping function is described, which is incorporated herein by reference in its entirety, wherein, given a fixed stimulus size, for each brightness level (e.g., stimulus level, etc.), the minimum visible contrast step at the brightness level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

[0009] Supports 200 to 1,000 cd / m 2 A display with a brightness of 1,000 nits or less represents a lower dynamic range (LDR) associated with EDR (or HDR), also known as standard dynamic range (SDR). EDR content can be displayed on an EDR display that supports a higher dynamic range (e.g., from 1,000 nits to 5,000 nits or more). Such a display can be defined using an alternative EOTF that supports high brightness capabilities (e.g., 0 to 10,000 nits or more). Examples of such EOTFs are defined in SMPTE 2084 and Rec.ITU-RBT.2100, "Image parameter values ​​for high dynamic range television for use inproduction and international programme exchange" (06 / 2017). As the inventors understand herein, it is expected that improved techniques for synthesizing video content data can be used to support the display capabilities of various SDR and HDR display devices.

[0010] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0012] Figure 1A Depicts an example process of a video transmission pipeline; Figures 1B to 1F illustrates an example codec implementing a cascaded prediction operation;

[0013] FIG. 2A to FIG. 2D illustrates example distributions and associations of input codewords and predicted codewords;

[0014] Figure 3A An example single layer backward compatible (SLBC) codec is illustrated; Figure 3B An example single layer inverse display management (SLiDM) codec is illustrated;

[0015] Figure 4A and Figure 4Billustrates an example process flow; and

[0016] Figure 5 Illustrated is a simplified block diagram of an example hardware platform upon which a computer or computing device as described herein may be implemented. DETAILED DESCRIPTION

[0017] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is apparent that the present disclosure can be practiced without these specific details. In other cases, in order to avoid unnecessarily obscuring, obscuring or confusing the present disclosure, well-known structures and devices are not described in detail.

[0018] Overview

[0019] The cascade prediction method as described herein can be implemented and executed in various video transmission and display applications, including but not limited to video transmission and display applications supporting backward compatibility. In some operating scenarios, base layer (BL) image data can be encoded in a first domain (e.g., having a first dynamic range, in a first color space, corresponding to a first electro-optical transfer function or EOTF, etc.) together with image metadata into a coded bitstream, and the image metadata is used to generate / reconstruct image data encoded in a second domain (e.g., having a second dynamic range, in a second color space, corresponding to a second EOTF, etc.) from the BL image data encoded in the first domain. The BL image data and image metadata can be provided to a receiving decoding and playback device, which can render the BL image data (e.g., in a first dynamic range, in a first color space, corresponding to a first EOTF, etc.) directly on a display that supports rendering images in the first domain after decoding, or can reconstruct image data in the second domain from the decoded BL image data and image metadata, and render the reconstructed image data (e.g., in a second dynamic range, in a second color space, corresponding to a second EOTF, etc.) on a display that supports rendering images in the second domain. The BL or input image data in the first domain and the reconstructed or output image data in the second domain may have different dynamic ranges. Additionally, optionally or alternatively, the BL or input image data in the first domain and the reconstructed or output image data in the second domain may have the same dynamic range; however, the BL or input image data and the reconstructed or output image data may be in different color spaces and / or may correspond to different EOTFs. For example, the BL or input image data in the first domain may have a dynamic range with a maximum brightness of 1000 nits, in the R.2020 color space, corresponding to the HLG EOTF, and the reconstructed or output image data may have the same dynamic range with a maximum brightness of 1000 nits, in the R.2020 color space, corresponding to the PQ EOTF. In this example, both the first domain and the second domain have the same dynamic range with a maximum brightness of 1000 nits and are also in the same color space; however, the BL or input image data and the reconstructed or output image data may be encoded according to or corresponding to different EOTFs. Therefore, the cascaded prediction method described herein can be applied as long as at least one property of the domain representing / encoding the image data - such as dynamic range, color space or EOTF - is different between the BL or input image data and the reconstructed or output image data. Thus, image data of a domain supported by some display devices encoded in the coded bitstream can be used to generate reconstructed image data of one or more other domains supported by some other display devices without requiring other image data of the other domains in the coded bitstream.In some operating scenarios, various display device types supporting display or image processing operations in various domains may be supported by the encoded streams generated under the techniques described herein.

[0020] In some operating scenarios, different groups of cascaded prediction coefficients can be generated to support various video quality levels of reconstructed image data. For example, a first group of cascaded prediction coefficients can be used to construct a first predicted video signal, and the reconstructed image data of the first predicted video signal supports a first video quality level. The second group of cascaded prediction coefficients can be used to construct a refinement signal, which will produce a second prediction signal when combined with the first prediction signal, and the reconstructed image data of the second prediction signal supports a second video quality level higher than the first video quality level. Different groups of cascaded prediction coefficients can be used to support different versions of video decoders or playback devices. Some video decoders or playback devices that can be widely deployed in the field may only recognize the first group of cascaded prediction coefficients and apply the first group of cascaded prediction coefficients in the prediction operation accordingly. When a relatively advanced video decoder or playback device recognizes an additional set (or multiple sets) of prediction coefficients and applies them to the prediction operation, the video quality level can be gradually improved.

[0021] The example embodiments described herein relate to generating and encoding cascade prediction metadata for image reconstruction. A first predictor is applied to an input image encoded in a first domain to generate a first level prediction codeword, the first level prediction codeword being approximately a prediction target codeword of a prediction target image encoded in a second domain different from the first domain. A second level prediction target value is created by performing an inverse cascade operation on the prediction target codeword and the first level prediction codeword. A second predictor is applied to the input image in the first domain to generate a second level prediction value that is approximately the second level prediction target value. Multiple sets of cascade prediction coefficients are generated to include a first set of cascade prediction coefficients that specify a first predictor and a second set of different cascade prediction coefficients that specify a second predictor. Multiple sets of cascade prediction coefficients are encoded into a video signal as image metadata. The video signal is further encoded with an input image in the first domain. A receiving device of the video signal uses the multiple sets of cascade prediction coefficients to apply the first predictor and the second predictor to the input image, thereby generating a prediction image in the second domain. The prediction image in the second domain is approximately the prediction target image in the second domain.

[0022] The example embodiments described herein relate to decoding cascade prediction metadata for image reconstruction and rendering. An input image encoded in a first domain is decoded from a video signal. Image metadata including multiple sets of cascade prediction coefficients is decoded from the video signal. An upstream image processing device generates multiple sets of cascade prediction coefficients to include a first set of cascade prediction coefficients, which are generated at least in part based on a predicted target image encoded in a second domain and specify a first predictor, and a second set of cascade prediction coefficients, which are generated at least in part based on a predicted target image in the second domain and specify a second predictor. The first domain is different from the second domain. The first predictor and the second predictor combined by the cascade operation are applied to the input image in the first domain to generate a reconstructed image in the second domain to approximate the predicted target image in the second domain. A display image derived from the reconstructed image in the second domain is rendered using a display device.

[0023] Example Video Transmission Processing Pipeline

[0024] Figure 1A An example process of a video transmission pipeline (100) is depicted, showing various stages from video capture to video content display. An image generation block (105) is used to capture or generate a sequence of video frames (102). The video frames (102) may be digitally captured (e.g., by a digital camera, etc.) or generated by a computer (e.g., using computer animation, etc.) to provide video data (107). Additionally, optionally, or alternatively, the video frames (102) may be captured on film by a film camera. The film may be converted to a digital format to provide video data (107). In a production stage (110), the video data (107) is edited to provide a video production stream (112).

[0025] The video data of the production stream (112) is then provided to a processor for post-production editing (115). Post-production editing (115) may include adjusting or modifying the color or brightness in specific areas of the image to enhance the image quality or achieve a specific look of the image in accordance with the creative intent of the video creator. This is sometimes referred to as "color timing" or "color grading". Other edits (e.g., scene selection and sequencing, manual and / or automatic scene cut information generation, image cropping, adding computer-generated visual effects, etc.) may be performed at the post-production editing (115) to produce a published version of the HDR image (117-1) or an SDR (or relatively narrow dynamic range) image (117) (e.g., SDR, etc.). In some embodiments, during post-production editing (115), the HDR image (117-1) is viewed on a reference HDR display that supports high dynamic range by a colorist who is performing post-production editing operations on the HDR image (117-1). Additionally, optionally, or alternatively, during post-production editing (115), the SDR image (117) is viewed on a reference display (125) supporting a standard dynamic range (or a relatively narrow dynamic range) by a colorist who is performing post-production editing operations on the SDR image (117). Additionally, optionally, or alternatively, the SDR image (117) may be content mapped from the HDR image (117-1).

[0026] In some embodiments, the encoding block (120) may implement some or all of the cascaded prediction operations to generate multiple sets of cascaded prediction coefficients. The encoding block (120) receives the HDR image (117-1) from the post-production editor (115) and forward shapes the HDR image (117-1) into a (forward) shaped SDR image. The forward shaped SDR image can closely approximate the SDR image (117) from the color grading operation (if available). For illustrative purposes only, the encoding block (120) will generate an encoded bitstream, which includes the SDR image (e.g., the forward shaped SDR image, etc.) and image metadata used by a receiving device of the encoded bitstream to reconstruct the HDR image from the SDR image. However, it should be noted that in various embodiments, the image data encoded / represented in the first domain can be encoded into the encoded bitstream together with the image metadata, which is used by the receiving device of the encoded bitstream to reconstruct the image data encoded in the second domain from the image data in the first domain.

[0027] In some embodiments, the encoding block (120) may include audio and video encoders such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats to generate an encoded bitstream (122).

[0028] The forward shaped SDR image may be compressed / encoded by the encoding block (120) into a coded bitstream (122). The multiple sets of concatenated prediction coefficients may be included as part of the image metadata or encoded into the same coded bitstream.

[0029] The encoded bitstream (122) may represent a video signal (e.g., an 8-bit SDR video signal, a 10-bit SDR video signal, etc.) that is backward compatible with a variety of SDR display devices (e.g., an SDR display, etc.). In a non-limiting example, the video signal encoded with the shaped SDR image may be a single-layer backward compatible video signal. Here, a "single-layer backward compatible video signal" may refer to a video signal that carries an SDR image that is specifically optimized or color graded for an SDR display in a single signal layer. Example single-layer video encoding operations are described in U.S. Provisional Patent Application No. 62 / 312,450 filed on March 23, 2016 (also published on April 11, 2019 as U.S. Patent Application Publication Serial No. 2019 / 011054), the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0030] Multiple sets of cascaded prediction coefficients can be decoded by a receiving device of a video signal or coded bitstream and used for cascaded prediction operations to generate a reconstructed HDR image from a decoded forward-shaped SDR image. The reconstructed HDR image can be approximated to the corresponding input or source HDR image in the color-graded HDR image (117-1). Cascaded prediction operations (e.g., backward shaping operations, inverse tone mapping operations, etc.) and multiple sets of cascaded prediction coefficients generated by the (upstream) coding block (120) can be used to generate different video quality levels for the reconstructed HDR image. As used herein, backward shaping refers to an image processing operation that converts an image of a first dynamic range, e.g., an image to be or already encoded in a coded bitstream, into an image of a second higher dynamic range. Forward shaping refers to an image processing operation that converts an image of a second dynamic range into an image of a first higher dynamic range, e.g., an image to be or already encoded in a coded bitstream. Example shaping operations are described in U.S. Provisional Application Serial No. 62 / 136,402, filed on March 20, 2015 (also published on January 18, 2018, as U.S. Patent Application Publication Serial No. 2018 / 0020224), the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0031] Additionally, optionally or alternatively, the coded bitstream (122) is encoded with additional image metadata, including but not limited to display management (DM) metadata, which can be used by a downstream decoder to perform display management operations on the backward-shaped image for the HDR reference display to generate a display image optimized for rendering on other displays such as a non-reference HDR display.

[0032] The encoded bitstream (122) is then transmitted downstream to a receiver such as a decoding and playback device, a media source device, a media streaming client device, a television (e.g., a smart TV, etc.), a set-top box, a movie theater, etc. In the receiver (or downstream device), a decoding block (130) decodes the encoded bitstream (122) to generate a decoded image 182, which may be the same as the shaped SDR image, but is subject to quantization errors generated in the compression performed by the encoding block (120) and the decompression performed by the decoding block (130).

[0033] In an operating scenario where the receiver operates with (or is attached to or operably linked to) a target display 140 that supports a standard dynamic range or a relatively narrow dynamic range that is comparable to or less than the standard dynamic range, the decoding block (130) can decode a shaped SDR image from the coded bitstream (122) (e.g., a single layer in the coded bitstream, etc.) and use the decoded shaped SDR image directly or indirectly for rendering on the target display (140). In an embodiment where the target display (140) has similar characteristics to the SDR reference display (125), the shaped SDR image can be viewed directly on the target display (140).

[0034] In some embodiments, the receiver operates with (or is attached to or operably linked to) an HDR target display 140-1 that supports a high dynamic range (e.g., 400 nits, 1000 nits, 4000 nits, 10000 nits, or higher), can extract image metadata from an encoded bitstream (122) (e.g., (multiple) metadata containers in the encoded bitstream), and synthesize a backward-shaped image 132-1 from the shaped SDR image by backward-shaping the shaped SDR image based at least in part on the image metadata (synthesizer metadata) using two or more of the multiple sets of cascaded prediction coefficients in the image metadata (or synthesizer metadata), including but not limited to performing a cascaded prediction operation on the shaped SDR image.

[0035] In some operational scenarios, the back-shaped image (132-1) represents a reconstructed HDR image optimized for viewing on an HDR (e.g., reference, etc.) display that is the same as or comparable to an HDR target display with which the receiver operates. The receiver may directly use the back-shaped image (132-1) for rendering on the HDR target display.

[0036] In some operational scenarios, the back-shaped image (132-1) represents a reconstructed HDR image optimized for viewing on an HDR (e.g., reference, etc.) display different from an HDR target display (140-1) operating in conjunction with the receiver. A display management block (e.g., 135-1, etc.), which may be located in the receiver, in the HDR target display (140-1), or in a separate device, further adjusts the back-shaped image (132-1) to accommodate the characteristics of the HDR target display (140-1) by generating a display mapping signal (137-1) adapted to the characteristics of the HDR target display (140-1). The display image or the adjusted back-shaped image may be rendered on the HDR target display (140-1).

[0037] Video codecs and cascade prediction

[0038] Figure 1B An example two-stage cascade prediction operation on the decoder side is illustrated. As shown, an input signal is fed as input to a first prediction block (denoted as "prediction 1"), which generates a first prediction (video) signal (denoted as "prediction signal 1") as an output. The input signal is also fed as input to a second prediction block (denoted as "prediction 2") to generate a residual residual (video) signal as an output. The first prediction signal output from the first prediction block and the prediction residual signal generated by the second prediction block are fed as input to a cascade operation block, which performs a merging operation (e.g., addition, multiplication, etc.) on the input signal to generate a second prediction (video) signal (denoted as "prediction signal 2") as an output. The second prediction signal represents a reconstructed video signal with higher accuracy and better quality than the first prediction signal relative to the source or input (video) signal to which each of the first prediction signal and the second prediction signal is entrusted to approximate.

[0039] In some operating scenarios, Figure 1B The first prediction block performs a prediction operation using an 8-segment 2nd-order polynomial in the luma channel (expressed as "Y") of a color space (e.g., YCbCr, etc.), and performs a prediction operation using MMR coefficients in the chroma channel (expressed as "Cb" or "C0", "Cr" or "Cl") of the color space. Figure 1BThe second prediction block uses MMR (multi-color channel multivariate regression) coefficients or TPB (tensor product B-spline) prediction coefficients in some or all luminance and chrominance channels to perform prediction operations. In a scenario where both the first prediction block and the second prediction block use MMR-based predictions, some or all of the processing logic for implementing MMR-based predictions can be reused in these prediction blocks. Example cumulative density function (CDF) matching operations are described in the following items: PCT Application No. PCT / US 2017 / 50980 filed on September 11, 2017; U.S. Provisional Application Serial No. 62 / 404,307 filed on October 5, 2016 (also published on April 5, 2018, as U.S. Patent Application Publication Serial No. 2018 / 0098094), the entire contents of which are incorporated herein by reference as fully set forth herein. Examples of MMR operations are described in U.S. Patent No. 8,811,490, which is incorporated herein by reference in its entirety as fully set forth herein. Examples of TPB operations are described in U.S. Provisional Application Serial No. 62 / 908,770, filed on October 1, 2019, entitled “TENSOR-PRODUCT B-SPLINE PREDICTOR” (Agent Docket No. 60175-0417), which is incorporated herein by reference in its entirety as if fully set forth herein.

[0040] Figure 1C An example two-level prediction of a function based on a function model is shown. In this model, the output of the first prediction block is directly fed to the second prediction block. In some operating scenarios, the prediction operation in this model may have little or no access to additional information beyond that already obtained by the first prediction block. In contrast, in Figure 1B In the illustrated cascade model, the second prediction block receives the original image data directly from the input (or source) signal. Therefore, the second prediction block is better positioned to correct the prediction error or inaccuracy generated by the first prediction block.

[0041] Example advantages and benefits resulting from cascaded prediction as described herein include, but are not necessarily limited to: (1) backward compatibility with existing codec architectures; and (2) parallel processing of a first prediction block and a second prediction block, which can be utilized or implemented to provide relatively low (e.g., processing, overall, etc.) latency and relatively independent prediction processing.

[0042] Figure 1D An example encoder-side architecture for cascade prediction is illustrated. A video codec (e.g., a video encoder, a video transcoder, etc.) receives a source (video) signal and a reference (video) signal. For purposes of illustration only, the source signal includes an SDR image (e.g., a video to be encoded into a video signal such as a video frame). Figure 1A The reference signal includes reference HDR images, and the reconstructed HDR images generated by the receiving device from the SDR images will approximate these reference HDR images. The goal of the codec is to generate multiple sets of concatenated prediction coefficients so that when one, some or all of the multiple sets of concatenated prediction coefficients are applied to the SDR images in the source signal, the reconstructed HDR images generated in the prediction (video) signal approximate (e.g., closely, at a specific quality level among multiple quality levels, as closely as possible, etc.) the reference HDR images in the reference signal.

[0043] A first set of cascaded prediction coefficients among multiple sets of cascaded prediction coefficients (e.g., specifying 8 segments of 2nd-order polynomials and MMR coefficients, etc.) can be used by a first group of receiving decoding or playback devices that are relatively widely deployed in the field to generate a first reconstructed HDR image that approximates a reference HDR image at a first quality level.

[0044] like Figure 1D As illustrated, the codec may receive a reference signal and a source signal using a first prediction block (denoted as “prediction 1”), generate a first set of concatenated prediction coefficients based at least in part on the received reference signal and the source signal, and output the first set of concatenated prediction coefficients (denoted as “predictor 1 coefficients”) as a block to be encoded with the source signal into a coded bitstream (e.g., Figure 1A 122, etc.), and generates and outputs a first prediction (video) signal generated from a source image in a source signal based at least in part on a first set of cascaded prediction coefficients.

[0045] A second set of cascaded prediction coefficients among the multiple sets of cascaded prediction coefficients (e.g., specified MMR coefficients or TPB prediction coefficients, etc.) can be used together with the first set by a relatively advanced second group of receiving decoding or playback devices to generate a second reconstructed HDR image that approximates the reference HDR image at a second quality level that is higher than the first quality level.

[0046] like Figure 1DAs illustrated, the codec can use two stages to generate a second set of cascaded prediction coefficients. The first stage is implemented by an inverse cascade operation block to generate a second reference signal and output the second reference signal to a second prediction block (denoted as "Prediction 2"). The inverse cascade operation block can implement an inverse cascade operation, such as an inversion of the cascade operation to be applied by a receiving decoder or playback device (or the opposite of the cascade operation). In addition to receiving the second reference signal generated by the inverse cascade operation block, the second prediction block also receives the same source signal received by the first prediction block. The second prediction block generates a second set of cascaded prediction coefficients based at least in part on the source signal and the second reference signal, so that a prediction residual signal can be generated from the source signal based at least in part on the second set of cascaded prediction coefficients to approximate the second reference signal output from the inverse cascade operation block as much as possible. Figure 1D A codec may (e.g., on command, dynamically, statically, etc.) include a second set of cascaded prediction coefficients (denoted “Predictor 2 coefficients”) as part of image metadata in the coded bitstream to be transmitted to (multiple) recipient decoders or playback devices.

[0047] Figure 1E An example decoder-side architecture for cascade prediction is illustrated. A video codec (eg, a video decoder, a video transcoder, etc.) receives a coded bitstream (eg, Figure 1A 122, etc.) into a decoded version of the source signal (which may be the same as the source signal subject to quantization / coding errors caused by encoding / decoding or compression / decompression) and encoded by an upstream codec (e.g., Figure 1D One or more groups of cascaded prediction coefficients among the multiple groups of cascaded prediction coefficients generated by the coding block (120) in the coding block (120), etc.

[0048] One or more sets of cascaded prediction coefficients among the plurality of sets of cascaded prediction coefficients include a first set of cascaded prediction coefficients. Figure 1E A first prediction block (denoted as "Prediction 1") of the codec may receive a source signal (or a decoded version thereof) and a first set of concatenated prediction coefficients (denoted as "Predictor 1 coefficients") and generate a first reconstructed HDR image (in a first prediction reference signal) from the SDR image in the source signal based at least in part on the first set of concatenated prediction coefficients. The first reconstructed HDR image approximates ( Figure 1D A reference HDR image in the reference signal.

[0049] In some operating scenarios, Figure 1E The codec obtains a first display image based on the first reconstructed HDR image and renders the first display image on an HDR display device without involving any other sets of cascaded prediction coefficients except the first set of cascaded prediction coefficients.

[0050] In some operating scenarios, Figure 1E The codec decodes a second set of concatenated prediction coefficients (denoted as "Predictor 2 coefficients") from the received encoded bitstream, and uses Prediction 2 to receive as input a source signal (or a decoded version thereof) and the second set of concatenated prediction coefficients, and generates a prediction residual signal from the source signal based at least in part on the second set of concatenated coefficients. The prediction residual signal may be output by Prediction 2 to a concatenated operation block (denoted as "Prediction 2"). The first prediction signal may be output by the first prediction block to the concatenated operation block, which combines the prediction residual signal with the first prediction signal to generate a second prediction signal, the second prediction signal comprising a second reconstructed HDR image that approximates the reference HDR image at a second quality level that is higher than the first quality level.

[0051] Cascaded prediction in forward and backward mapping at the encoder

[0052] In some operating scenarios, as part of the overall image processing operations performed by the encoder, a video encoder as described herein may perform both (HDR to SDR) forward mapping and (SDR to HDR) backward mapping. For example, an encoder may receive a reference (or input) HDR video signal and a content-mapped SDR video signal. One or more of these video signals may be the result of a color grading operation performed, for example, by a (multiple) colorist using an SDR reference display, an HDR reference display, both an SDR and HDR reference display, and the like. Additionally, optionally, or alternatively, image data in one video signal may be content-mapped from corresponding image data in another video signal (e.g., fully automatically without user intervention or manual input, substantially automatically with user intervention and manual input, and the like).

[0053] Cascaded prediction can be applied to both forward mapping and backward mapping in the encoder. By applying cascaded prediction in the encoder, the accuracy and quality of forward mapping and backward mapping can be improved.

[0054] In order to apply the cascaded prediction as described herein to the forward mapping or the backward mapping at the encoder side, the same method as Figure 1D The same or similar multi-level (or cascaded) prediction as depicted in .

[0055] Cascaded prediction in forward mapping at the encoder

[0056] Assume that there are P pixels in each of the j-th input SDR image in the input SDR video signal (received by the video encoder) and the j-th input HDR image in the input HDR video signal (received by the video encoder) (depicting the same visual semantic content as the input SDR image but with a higher luminance dynamic range).

[0057] Let triples and denote the normalized Y, C0, and C1 values ​​(in a color space such as YCbCr or YC0C1) of the i-th pixel in the j-th SDR and HDR images, respectively.

[0058] In the luminance or brightness channel of the color space (denoted as Y), a single-channel luminance prediction operation can be performed to use the luminance forward shaping function HDR codeword Mapping or forward shaping into a predicted or mapped SDR codeword (denoted as ), to approximate the SDR codeword As shown below:

[0059]

[0060] The luma forward shaping function may be a one-dimensional lookup table (1D-LUT) — or FLUT — that provides a one-to-one mapping from HDR luma codewords / values ​​to predicted or mapped SDR luma codewords / values. The input HDR signal may be used as a prediction input (e.g., including the HDR codewords to be mapped). etc.) and uses the input SDR signal as a prediction reference (e.g., including the SDR codeword to be predicted To approximate the SDR codeword etc.) to generate the brightness forward shaping function.

[0061] However, the luma codewords / values / intensities of pixels are often affected by the chroma components of those pixels. A single-channel luma forward shaping function may not be good at capturing cross-color information or the effects between the chroma and luma components of a pixel. Therefore, predicting SDR codewords may deviate from the input SDR codeword (Predict SDR codeword The expected SDR codeword to be approximated).

[0062] In order to overcome the disadvantages associated with the single-channel luma forward shaping function, the single-channel luma forward shaping function may be (e.g., simply, etc.) viewed as a first luma forward prediction function generated with the input SDR signal as a first prediction target signal including a first prediction target value. Can be used to further incorporate (or better capture) chrominance information or its impact on the luma component.

[0063] A second predicted target signal (expressed as ) to include a second prediction target value, and the second prediction value generated using the second brightness forward prediction function is to be approximated to the second prediction target value, as shown below:

[0064]

[0065] The operator “-” on the right side of expression (2) represents the inverse cascade operator OP -1 (). For illustration purposes only, the inverse concatenation operator OP -1 () is represented as a subtraction operator in expression (2). It should be noted that in various embodiments, the inverse cascade operator as described herein can be represented as a subtraction operator, a division operator, a reversible operator, a functional-based operator, etc.

[0066] The HDR luma and / or chroma codewords in the input HDR image may be used (e.g. etc.)—or its specific function—as input and uses the second predicted target signal As the prediction target, the second brightness forward prediction function (expressed as ). Once the second brightness forward prediction function has been generated, which can be used to convert the HDR luminance and chrominance codewords in the input HDR image — or its specific function — is mapped to the predicted codeword residual As shown below:

[0067]

[0068] Because the first brightness forward prediction function and the second brightness forward prediction function are combined with the cascade operator OP(), the overall brightness forward prediction function (expressed as ), as follows:

[0069]

[0070] The operator "+" on the right side of expression (4) represents the inverse concatenation operator OP in expression (2) -1 () is the inverse cascade operator OP(). For illustrative purposes only, the cascade operator OP() is represented as an addition operator in expression (4). It should be noted that in various embodiments, as long as the cascade operation represented in expression (4) is conjugate with the inverse cascade operation in expression (2), the cascade operator as described herein can be represented as an addition operator, a multiplication operator, a reversible operator, a functional-based operator, etc.

[0071] More specifically, by adding the second mapping value - generated by applying the second luma forward prediction function to the input HDR signal - as a prediction residual to the first mapping value generated by applying the first luma forward prediction function to the input HDR signal, a forward shaped SDR signal including an overall mapped (or overall forward shaped) SDR codeword / value in the SDR luma channel can be obtained, as shown below:

[0072]

[0073] The forward prediction in the chroma channel (or non-luminance channel) (or the prediction operation in the forward shaping) can follow a cascade process similar to the cascade process described herein for the forward prediction in the luminance channel. In various embodiments, the forward prediction in the chroma channel can use a prediction function selection similar to or different from the prediction function used in the forward prediction in the luminance channel. As an illustration and not limitation, the forward prediction in the chroma channel can select the first chroma prediction function as a second-order MMR operator (specified by the second-order MMR coefficients) to predict the SDR chroma codeword / value in the chroma channel, and further select the second chroma prediction function as a third-order MMR operator (specified by the third-order MMR coefficients) or some other operator (e.g., in addition to the second-order MMR operator, etc.) to predict the residual SDR chroma codeword / value in the chroma channel.

[0074] An SDR signal comprising a forward shaped SDR image generated using cascaded forward prediction as described herein may be encoded into a coded bitstream (e.g., Figure 1A 122, etc.), and transmitted to a downstream decoder and / or playback device, and the backward shaping metadata is used to generate and render a reconstructed HDR image from the forward shaped SDR image.

[0075] Cascaded prediction in backward mapping at the encoder

[0076] Assume that there are P pixels in each of the j-th SDR image (to be backward shaped) in the SDR video signal and the (corresponding prediction target) j-th HDR image (depicting the same visual semantic content as the SDR image but with a higher luminance dynamic range) in the HDR video signal.

[0077] Let triples and denote the normalized Y, C0, and C1 values ​​(in a color space such as YCbCr or YC0C1) of the i-th pixel in the j-th SDR and HDR images, respectively.

[0078] In the luminance or brightness channel of the color space (denoted as Y), a single-channel luminance prediction operation can be performed to use the luminance backshaping function SDR codeword Mapping or backward shaping into a predicted or mapped HDR codeword (denoted as ), to approximate the HDR codeword As shown below:

[0079]

[0080] The luma back-shaping function may be a one-dimensional look-up table (1D-LUT) — or BLUT — that provides a one-to-one mapping from SDR luma codewords / values ​​to predicted or mapped HDR luma codewords / values. The SDR image (to be back-shaped) may be used as a prediction input (e.g., including the SDR codewords to be mapped). etc.) and uses the (prediction target) HDR image as a prediction reference (e.g., including the HDR codeword to be predicted To approximate HDR codeword etc.) to generate the brightness backward shaping function.

[0081] However, the luma codewords / values / intensities of pixels are often affected by the chroma components of those pixels. Single-channel luma backshaping functions may not be good at capturing cross-color information or the effects between the chroma and luma components of pixels. Therefore, predicting HDR codewords May deviate from the HDR codeword (Predict HDR codeword The expected HDR codeword to be approximated).

[0082] In order to overcome the disadvantages associated with the single-channel luma backward shaping function, the single-channel luma backward shaping function may be (e.g., simply, etc.) viewed as a first luma backward prediction function generated with the (prediction target) HDR image as a prediction target including a first prediction target value. Can be used to further incorporate (or better capture) chrominance information or its impact on the luma component.

[0083] A second predicted target signal (expressed as ) to include a second prediction target value, and the second prediction value generated using the second brightness backward prediction function is to be approximated to the second prediction target value, as shown below:

[0084]

[0085] The operator “-” on the right side of expression (7) represents the inverse cascade operator OP -1 (). For illustration purposes only, the inverse concatenation operator OP -1 () is represented as a subtraction operator in expression (7). It should be noted that in various embodiments, the inverse cascade operator as described herein can be represented as a subtraction operator, a division operator, a reversible operator, a functional-based operator, etc. Additionally, optionally, or alternatively, in various embodiments, similar or different types of inverse cascade operators can be used for forward shaping and backward shaping. Therefore, it is not necessary to use the same type of inverse cascade operation in forward shaping and backward shaping.

[0086] The SDR luminance and chrominance codewords (e.g. etc.)—or its specific function—as input and uses the second predicted target signal As the prediction target, the second brightness backward prediction function (expressed as ). Once the second brightness backward prediction function has been generated, which can be used to convert the (to be backward shaped) SDR luminance and chrominance codewords in the (to be backward shaped) SDR image — or its specific function — is mapped to the predicted codeword residual As shown below:

[0087]

[0088] Because the first brightness backward prediction function and the second brightness backward prediction function are combined with the inverse cascade operator OP(), the overall brightness backward prediction function (expressed as ), as follows:

[0089]

[0090] The operator "+" on the right side of expression (9) represents the inverse concatenation operator OP in expression (7) -1 () is an inverse cascade operator OP(). For illustrative purposes only, the cascade operator OP() is represented as an addition operator in expression (9). It should be noted that in various embodiments, the cascade operator as described herein can be represented as an addition operator, a multiplication operator, a reversible operator, an operator based on a functional, etc., as long as the cascade operation represented in expression (9) is conjugate with the inverse cascade operation in expression (7). Additionally, optionally or alternatively, in various embodiments, similar or different types of cascade operators can be used for forward shaping and backward shaping. Therefore, it is not necessary to use the same type of cascade operation in forward shaping and backward shaping.

[0091] More specifically, by adding the second mapping value - generated by applying the second luma backward prediction function to the SDR signal (to be backward shaped) or the SDR image therein - as a prediction residual to the first mapping value generated by applying the first luma backward prediction function to the SDR signal (to be backward shaped) or the SDR image therein, a backward shaped HDR signal including an overall mapped (or overall backward shaped) HDR codeword / value in the HDR luma channel can be obtained, as shown below:

[0092]

[0093] Backward prediction in a chroma channel (or non-luminance channel) (or prediction operation in backward shaping) may follow a cascade process similar to the cascade process described herein for backward prediction in a luminance channel. In various embodiments, backward prediction in a chroma channel may use a prediction function selection that is similar to or different from the prediction function used in backward prediction in a luminance channel. As an illustration and not limitation, backward prediction in a chroma channel may select a first chroma prediction function as a second-order MMR operator (specified by a second-order MMR coefficient) to predict an HDR chroma codeword / value in a chroma channel, and further select a second chroma prediction function as a third-order MMR operator (specified by a third-order MMR coefficient) or some other operator (e.g., in addition to a second-order MMR operator, etc.) to predict a residual HDR chroma codeword / value in a chroma channel.

[0094] A first set of cascaded prediction coefficients (or prediction operation parameters) specifying a first backward shaping map (e.g., first luma and chroma backward prediction functions) and a second set of cascaded prediction coefficients (or prediction operation parameters) specifying a second backward shaping map (e.g., second luma and chroma backward prediction functions) may be included in the encoded bitstream as part of the picture metadata (e.g., Figure 1A 122, etc.), the encoded bitstream is encoded with the SDR image (to be backward shaped) and is transmitted to a downstream decoder and / or playback device to generate and render a reconstructed HDR image from the SDR image.

[0095] Cascaded prediction at the decoder

[0096] In some operating scenarios, such as Figure 1E As shown, a codec such as a video decoder only performs a backward mapping from SDR to HDR. For illustration purposes only, the backward mapping on the decoder side may first be explained for luma backward shaping (or backward shaping in the luma channel). The decoder receives the coded bitstream (e.g., Figure 1A 122, etc.) or an image metadata container thereof, decode or read multiple sets of cascaded prediction coefficients.

[0097] The plurality of sets of cascaded prediction coefficients include a first set of cascaded prediction coefficients for a first prediction operation and a second set of cascaded prediction coefficients for a second prediction operation. The first set of cascaded prediction coefficients may specify a first luma backward prediction function For example, the decoded SDR codeword Mapping to predicted HDR codeword These predicted HDR codewords approximate the reference HDR image in the predicted target (or reference) HDR signal in the upstream device that generates multiple sets of cascaded prediction coefficients, as shown below:

[0098]

[0099] In some operating scenarios, the decoder may be configured to generate a predicted HDR luminance codeword from a first luminance backward prediction function. The first predicted / reconstructed HDR image is used to generate a display image, and the display image is rendered using a display device operating with the decoder.

[0100] In some operating scenarios, the decoder may (e.g., simultaneously, in parallel, etc.) use a second luma backward prediction function specified by a second set of concatenated prediction coefficients. From the decoded SDR luminance and chrominance codewords (e.g. etc.) or its specific functions to predict or estimate the difference signal (e.g., prediction residual) As shown below:

[0101]

[0102] The estimated difference signal generated using the second luma backward prediction function is added to the predicted HDR codeword generated using the first luma backward prediction function using a cascade operation to obtain a further improved prediction signal (expressed as ) is as follows:

[0103]

[0104] The chrominance components of the first and second predicted / reconstructed HDR images may be obtained similarly. In some operating scenarios, the first chrominance backward prediction function / mapping may be a three-channel MMR predictor or function. The second chrominance backward prediction function / mapping may be an MMR predictor or function of a different order or type, or another predictor / function that is different or distinct (or more advanced than) from the first chrominance backward prediction function / mapping.

[0105] Example cascade predictors and operation types / categories

[0106] In some operating scenarios, the first cascade predictor among multiple cascade predictors may be, but is not necessarily limited to, a relatively simple predictor (e.g., a single-channel predictor, a low-order polynomial, a low-order MMR operator, etc.), while the second or additional cascade predictors among the multiple cascade predictors may be, but are not necessarily limited to, a relatively complex or advanced predictor (e.g., a cross-channel predictor, a high-order polynomial, a high-order MMR operator, etc.).

[0107] In some operational scenarios, two or more of the plurality of cascade predictors may be, but are not necessarily limited to, predictors of comparable complexity. The two or more cascade predictors may be diverse or different in type or category.

[0108] For illustration only, HDR to SDR mapping (e.g., backward shaping mapping, etc.) may be used as an example to explain different types or categories of cascade predictors or operations. It should be noted that in various embodiments, cascade predictors or operations similar to or different from the types or categories of cascade predictors or operations used for HDR to SDR mapping may be used for SDR to HDR mapping (e.g., forward shaping mapping, etc.).

[0109] A first example type or class of cascade predictors is a univariate function using a polynomial with a single input variable consisting of the codewords in the chroma channel(s). Since only one input variable is used, chroma saturation is one candidate among many different candidates. Chroma saturation (denoted as ) can be defined or calculated as follows:

[0110]

[0111] A first example type of cascade predictor can be represented by a polynomial with chroma saturation as a single input variable in expression (14) and can be expressed as The polynomial coefficients of such a polynomial representing the cascaded predictor may be obtained by minimizing an error metric / cost based on a predicted target signal (e.g., a signal to be approximated, etc.) and a predicted input signal (e.g., a signal to be forward shaped, a signal to be backward shaped, etc.).

[0112] A second example type or category of cascaded predictors are MMR predictors / functions that take as input the codewords in the chroma channels.There are many options when choosing the input variables.

[0113] As a first example option, two input variables and can be used for the MMR predictor / function as described herein (denoted as ).

[0114] As a second example option, two input variables and can be used for the MMR predictor / function as described herein (denoted as ).

[0115] As a third example option, three input variables and can be used for the MMR predictor / function as described herein (denoted as ).

[0116] As a fourth example option, three input variables i and j can be used in the MMR predictor / function as described herein (denoted as ). The input variables i and j can be used to efficiently capture local variations at (multiple) specific pixel locations.

[0117] The MMR prediction coefficients of the MMR predictor representing the cascade predictor can be obtained by minimizing the error metric / cost based on the predicted target signal (e.g., the signal to be approximated, etc.) and the predicted input signal (e.g., the signal to be forward shaped, the signal to be backward shaped, etc.).

[0118] A third example type or class of cascade predictors are TPB predictors / functions that take as input the codewords in the chroma channels.There are many options when choosing the input variables.

[0119] As a first example option, two input variables and can be used for the TPB predictor / function as described herein (expressed as ).

[0120] As a second example option, two input variables and can be used for the MMR predictor / function as described herein (denoted as ).

[0121] As a third example option, three input variables and can be used for the MMR predictor / function as described herein (denoted as ).

[0122] As a fourth example option, three input variables i and j can be used in the MMR predictor / function as described herein (denoted as ). The input variables i and j can be used to efficiently capture local variations at (multiple) specific pixel locations.

[0123] The TPB prediction coefficient of the TPB predictor representing the cascade predictor can be obtained by minimizing the error metric / cost based on the prediction target signal (e.g., the signal to be approximated, etc.) and the prediction input signal (e.g., the signal to be forward shaped, the signal to be backward shaped, etc.).

[0124] Additionally, optionally, or alternatively, types or classes of cascade predictors other than those described herein may be used in the types or classes of cascade predictors (e.g., non-polynomial, non-MMR, non-TPB, etc.). Additionally, optionally, or alternatively, input variables other than those described herein may be used in a type or class of cascade predictors.

[0125] In various embodiments, similar or different types or categories (eg, reversible, invertible, etc.) of cascade operations and their conjugate or inverse cascade operations may be used.

[0126] A first example cascade operation is (e.g., in A second example concatenation operation is (e.g., in (Arithmetic) multiplication in .

[0127] Additionally, optionally, or alternatively, types of cascade operations other than those described herein (arithmetic addition and multiplication) may be used in a type or class of cascade predictors.

[0128] Multiple Cascade Prediction

[0129] The two-level cascade prediction can be extended to multiple (or multi-level) cascade prediction, such as three-level cascade prediction. In some operating scenarios, multiple segmented MMR operators can be generated as multiple predictors in the cascade prediction as described herein. Additionally, optionally or alternatively, instead of or in addition to creating a 1D-LUT to guide MMR prediction (e.g., single-channel multi-segment MMR prediction, multi-channel single-segment MMR prediction, etc.), cascade prediction can be used to support multi-channel multi-segment MMR prediction without performing special processing at segment partition boundaries, such as fusing multiple MMR prediction results at segment partition boundaries. For example, different types of multi-channel multi-segment MMR predictions can be implemented in different continuous or cascaded prediction levels with relatively high computational efficiency and relatively high performance. As Figure 1F As shown, in addition to the first and second prediction operations (represented as "prediction 1" and "prediction 2" respectively) using the first predictor and the second predictor In addition, a third prediction operation (denoted as "prediction 3") can use a third predictor

[0130] As previously mentioned, different types of concatenation operations may be used by the codecs described herein in different operating scenarios. As an illustration and not limitation, two different types of concatenation operators may be used.

[0131] First, in some operational scenarios, the (arithmetic) addition operator can be used to combine Figure 1F The cascade operation of the predicted values ​​of the three prediction operations is as follows:

[0132]

[0133] Second, in some operational scenarios, the (arithmetic) multiplication operator can be used to combine Figure 1F The cascade operation of the predicted values ​​of the three prediction operations is as follows:

[0134]

[0135] With the addition of a third, new level (compared to a two-level cascaded prediction), some additional structure, information, etc. in the image data that is missed by other prediction operations in other levels can be captured. (Multiple) later prediction operations - which can be performed completely simultaneously with (multiple) earlier prediction operations in the decoder described herein - can be selected to be different or distinct from, or higher in level than, (multiple) earlier prediction operations. With the addition of new levels, subsequent residuals to be predicted by subsequent prediction operations may become more like random noise, thereby providing almost negligible benefits, while increasing computational complexity as more and more new levels are added.

[0136] Derivation of predictor coefficients

[0137] The first predictor may be calculated or represented by a polynomial, MMR, TPB, etc. By way of illustration and not limitation, example calculations of concatenated prediction coefficients (or operating parameters) of the second predictor are described herein for forward shaping (from HDR to SDR).

[0138] For simplicity, the target signal of the second predictor is denoted as s ji , and from the first predictor The (first) predicted value of is expressed as therefore,

[0139]

[0140] The first predictor (or first prediction operation) may be based on a single-channel polynomial or MMR with input in the form of a single input variable or multiple input variables. The prediction coefficients (constituting a first set of cascaded prediction coefficients) may be obtained via an optimization process to minimize an error metric / cost, such as an error metric / cost expressed as mean square error (MSE), as shown below:

[0141]

[0142] At the encoder side, the residual signal (expressed as ), such as the difference between the original prediction target (or reference) signal and the output prediction signal from the first predictor. Depending on the type of cascade operator or cascade operation to be used in conjunction with the first predictor and the second predictor (e.g., reversible, etc.), a different prediction target (or reference) value to be predicted by the second predictor in the second prediction operation may be determined.

[0143] In the example, the concatenation operation is arithmetic addition. Therefore, the corresponding inverse concatenation operation (or inverse concatenation operator) is (arithmetic) subtraction. The inverse concatenation operator can be used to calculate the residual signal in the luma channel from the original prediction target signal and the first prediction signal as follows:

[0144]

[0145] In another example, the concatenation operation is an arithmetic multiplication. Therefore, the corresponding inverse concatenation operation (or inverse concatenation operator) is an (arithmetic) division. The inverse concatenation operator can be used to calculate the residual signal in the luma channel from the original prediction target signal and the first prediction signal as follows:

[0146]

[0147] The second predictor (or prediction operation) can be represented by the second prediction function as follows:

[0148]

[0149] The second predictor (or prediction operation) is designed to use the original input value predict As an example and not limitation, the type of the second prediction operator may be MMR, TPB, or another type having an input in the form of a plurality of input variables. The prediction coefficients of the second predictor (constituting a second set of cascaded prediction coefficients) may be obtained via an optimization process to minimize an error metric / cost, such as an error metric / cost expressed as a mean square error (MSE), as shown below:

[0150]

[0151] Taking 3-channel MMR as an example, the design vector can be constructed as a multivariate polynomial vector of the input source (e.g., to be forward shaped, etc.) signal as follows:

[0152]

[0153] The design matrix including the design vector in expression (23) for all pixels (or SDR and HDR pixel pairs) can be constructed as follows:

[0154]

[0155] A reference (or prediction target) signal comprising values ​​that approximate the values ​​generated by the second predictor for all pixels (or SDR and HDR pixel pairs) for the Y, Cb, and Cr channels The Y, Cb or C0, and Cr or C1 components can be constructed as follows:

[0156]

[0157]

[0158]

[0159] The second predictor can be specified using the MMR coefficients of the luma and chroma channels as follows:

[0160]

[0161]

[0162]

[0163] Where M is the total number of MMR coefficients, for example, 7*MMR_ORDER+1.

[0164] These MMR coefficients can be used to generate a reference (or predicted target) signal that approximates expression (25) The predicted values ​​for each of the Y, Cb or C0, and Cr or C1 components of are as follows:

[0165]

[0166]

[0167]

[0168] For the purpose of deriving the MMR coefficients, the optimization problem of minimizing the overall approximation error for all pixels (or pixel bins) can be solved as follows:

[0169] For channel y:

[0170] For channel c0:

[0171] For channel c1:

[0172] This optimization problem can be solved using a linear least squares solution as follows:

[0173]

[0174]

[0175]

[0176] Figure 2AAn example scatter plot of luma (Y) codewords / values ​​in an SDR image (as indicated on the vertical axis) and corresponding luma (Y) codewords / values ​​in a corresponding HDR image (as indicated on the horizontal axis) is illustrated, wherein the HDR image depicts the same visual semantic content as the SDR image.

[0177] Figure 2B An example scatter plot of predicted luma (Y) codewords / values ​​(as indicated on the vertical axis) and corresponding luma (Y) codewords / values ​​in an SDR image (as indicated on the horizontal axis) is illustrated, which are generated by a first predictor or prediction operation (denoted as "Predictor #1") using the HDR image as input and using the SDR image as a prediction target / reference to be approximated by the predicted luma (Y) codewords / values.

[0178] Figure 2C An example scatter plot is shown of second level prediction values ​​(as indicated on the vertical axis; denoted “Prediction Signal #2”) generated by a second predictor or prediction operation using the HDR image as input and using target / reference values ​​generated by an inverse cascade operation and approximated by the second level prediction values, versus predicted luminance (Y) codewords / values ​​(as indicated on the horizontal axis) generated by a first predictor or prediction operation.

[0179] Figure 2D An example scatter plot of overall predicted luma (Y) codewords / values ​​(as indicated by the vertical axis) and corresponding luma (Y) codewords / values ​​in an SDR image (as indicated by the horizontal axis) is illustrated, which are generated by combining the predicted values ​​from the first and second predictors or prediction operations.

[0180] Compared with using only the first predictor or prediction operation Figure 2B Compared to the scatter plot of , the overall predicted luminance (Y) codeword / value generated by both the first and second predictors or prediction operations under the cascaded prediction as described herein shows a relatively strong correlation with the luminance (Y) codeword / value in the SDR image, thereby achieving a relatively high accuracy and quality level in the predicted image including the overall predicted luminance (Y) codeword / value.

[0181] Application Architecture

[0182] Cascaded prediction as described herein may be integrated into a variety of application architectures implemented with video codecs, including but not limited to application architectures supporting single layer backward compatible (SLBC) and single layer inverse display management (SLiDM) video applications.

[0183] SLBC Architecture

[0184] Figure 3AAn example SLBC codec (e.g., SLBC video encoder, SLBC video transcoder) including a forward shaping path and a backward shaping path is illustrated. Cascade prediction, such as two-stage cascade prediction combined with an inverse cascade operation, can be performed in each of the forward shaping path and the backward shaping path (e.g., additionally, optionally, or alternatively, etc.).

[0185] As shown in the figure, the cascade prediction in the forward shaping path includes a first forward prediction operation (denoted as "forward prediction 1") and a second forward prediction operation (denoted as "forward prediction 2"). The cascade prediction in the backward shaping path includes a first backward prediction operation (denoted as "backward prediction 1") and a second backward prediction operation (denoted as "backward prediction 2").

[0186] It should be noted that the predictor or prediction operation in the forward shaping path need not be symmetrical or otherwise determined relative to the predictor or prediction operation in the backward shaping path. For example, the second forward prediction need not be identical or conjugate (e.g., inverse, transposed, etc.) to the first backward prediction. Similarly, the first forward prediction need not be identical or conjugate (e.g., inverse, transposed, etc.) to the second backward prediction.

[0187] Additionally, optionally, or alternatively, different types of cascaded predictions may be used for the two-stage predictions for each of the forward shaping path and the backward shaping path. For example, for each of the forward shaping path and the backward shaping path, a relatively simple predictor or prediction operation may be placed in the first stage, whereas a different, relatively advanced predictor or prediction operation may be placed in the second stage after the simpler predictor or prediction operation to obtain additional prediction accuracy and performance gain.

[0188] More specifically, in some operating scenarios, the second forward predictor or prediction operation may be selected to be different or distinct from (or more advanced than) the first forward predictor or prediction operation. Similarly, the second backward predictor or prediction operation may be selected to be different or distinct from (or more advanced than) the first backward predictor or prediction operation.

[0189] In a forward shaping path, luma and chroma HDR codewords / values ​​constituting each HDR image in an input (or source) HDR signal (denoted as "HDR signal") may be forward shaped into luma and chroma SDR codewords / values ​​constituting a corresponding forward shaped SDR image in a forward shaped signal (denoted as "forward shaped SDR signal"). The forward shaped luma and chroma SDR codewords / values ​​approximate (e.g., closely, at various quality levels, etc.) the luma and chroma SDR codewords / values ​​in a corresponding reference (or target) SDR image in a reference (or target) SDR signal (denoted as "SDR signal").

[0190] By way of example and not limitation, in the forward (shaping) path, a first forward predictor or prediction operation includes a first luma forward predictor (e.g., function, curve, lookup table, FLUT, etc.) constructed according to a CDF matching technique as follows:

[0191]

[0192] in, Represents the first predicted luma codeword / value output by the first luma forward predictor.

[0193] An inverse concatenation operation (which may be luma-specific, first-stage-specific, encoder-specific, etc.) may be used to generate a first predicted luma codeword / value and a target luma codeword / value in a reference (or target) SDR signal to be approximated by the first predicted luma codeword / value. The prediction error between is as follows:

[0194]

[0195] The prediction error may be used as a prediction target value to be predicted or approximated using a second forward predictor or a second luma forward predictor in a prediction operation (eg, MMR mapping), as shown below:

[0196]

[0197] The final prediction value of the luma channel is generated by adding the prediction values ​​output by both the forward predictor and the second luma predictor. As shown below:

[0198]

[0199] Similarly, in the forward (shaping) path, for the chroma channels Cb / C0 and Cr / Cl, the first forward predictor or prediction operation comprises a first chroma forward predictor as follows:

[0200]

[0201]

[0202] in, and Represents the first predicted chroma value of the chroma channel based on a single-stage MMR operation.

[0203] A first predicted chroma codeword / value may be generated for each chroma channel using an inverse concatenation operation (which may be luma-specific, first stage-specific, encoder-specific, etc.) The target chroma codeword / value in the reference SDR signal to be approximated by the first predicted chroma codeword / value The prediction error between is as follows:

[0204]

[0205]

[0206] The prediction error may be used as a predicted target value to be predicted or approximated using a second chroma forward predictor in a second forward predictor or prediction operation (e.g., an MMR map that is more advanced than the MMR map used in the first chroma forward predictor in terms of the type of number of levels used, the order or degree of construction, the mapping and functions used, the operation, the number of input variables, the type(s) of input variables, etc.), as shown below:

[0207]

[0208]

[0209] The final prediction value of the chroma channel is generated by adding the prediction values ​​output by both the second chroma forward predictor and the second chroma forward predictor. and As shown below:

[0210]

[0211]

[0212] In some operational scenarios, the final prediction values ​​in expressions (33) and (37) may be output as part of a forward shaped SDR image / signal as forward shaped luminance and chrominance codewords / values. Figure 1A The forward shaped image / signal is encoded in 122, etc.) and provided as input to the backward shaping path.

[0213] In the backward shaping path, the forward shaped luminance and chrominance SDR codewords / values ​​constituting each forward shaped SDR image in the forward shaped SDR signal can be backward shaped into reconstructed luminance and chrominance HDR codewords / values ​​constituting the corresponding reconstructed HDR image in the reconstructed HDR signal. The reconstructed or backward shaped luminance and chrominance HDR codewords / values ​​are similar (e.g., closely, at various quality levels, etc.) to the luminance and chrominance HDR codewords / values ​​in the corresponding input HDR image in the input HDR signal. Neither the input HDR signal nor the reconstructed HDR signal is encoded into the coded bitstream. Instead, image metadata (or synthesizer metadata) specifying the backward shaping mapping (e.g., multiple groups of cascaded prediction operations) is included in the coded bitstream. Instead of decoding the HDR image from the coded bitstream, the receiving decoder or playback device decodes the forward shaped SDR image and generates a reconstructed HDR image from the forward shaped SDR image by applying the backward shaping mapping to the forward shaped SDR image.

[0214] By way of example and not limitation, in the backward (shaping) path, a first backward predictor or prediction operation includes a first luma backward predictor (e.g., function, curve, lookup table, BLUT, etc.) constructed according to a CDF matching technique as follows:

[0215]

[0216] in, Represents the first predicted luma codeword / value output by the first luma backward predictor.

[0217] An inverse concatenation operation (which may be luma-specific, first-stage-specific, encoder-specific, etc.) may be used to generate a first predicted luma codeword / value and a target luma codeword / value in the input HDR signal to be approximated by the first predicted luma codeword / value. The prediction error between is as follows:

[0218]

[0219] The prediction error may be used as a prediction target value to be predicted or approximated using a second backward predictor or a second luma backward predictor in a prediction operation (eg, MMR mapping), as shown below:

[0220]

[0221] The final prediction value of the luma channel is generated by adding the prediction values ​​output by the first and second luma backward predictors As shown below:

[0222]

[0223] Similarly, in the backward (shaping) path, for the chroma channels Cb / C0 and Cr / Cl, the first backward predictor or prediction operation comprises a first chroma backward predictor as follows:

[0224]

[0225]

[0226] in, and Represents the first predicted chroma value of the chroma channel based on a single-stage MMR operation.

[0227] A first predicted chroma codeword / value may be generated for each chroma channel using an inverse concatenation operation (which may be luma-specific, first stage-specific, encoder-specific, etc.) The target chroma codeword / value in the reference (or target) HDR signal to be approximated by the first predicted chroma codeword / value The prediction error between is as follows:

[0228]

[0229]

[0230] The prediction error may be used as a predicted target value to be predicted or approximated using a second chroma backward predictor in a second backward predictor or prediction operation (e.g., an MMR map that is higher level than the MMR map used in the first chroma backward predictor in terms of the type of number of levels used, the order or degree of construction, the mapping and functions used, the operation, the number of input variables, the type(s) of input variables, etc.), as shown below:

[0231]

[0232]

[0233] The final prediction value of the chroma channel is generated by adding the prediction values ​​output by both the second chroma backward predictor and the second chroma backward predictor. and As shown below:

[0234]

[0235]

[0236] It should be noted that the final predicted luminance and chrominance values ​​will be generated by a receiving device that receives the image metadata (or synthesizer metadata) including multiple sets of cascaded prediction coefficients. and The plurality of sets of cascaded prediction coefficients include specifying first luma and chroma backward predictors The first set of cascaded prediction coefficients and the second luma and chroma backward predictors are specified These backward predictors may be applied by a receiving device to a forward-shaped SDR image (including forward-shaped SDR codewords / values) decoded from an encoded bitstream. ) to generate the final predicted luminance and chrominance values ​​of the reconstructed HDR image and SLiDM Architecture

[0237] Figure 3B An example SLiDM codec (eg, SLiDM video encoder, SLiDM video transcoder) including a backward shaping path is illustrated. Cascade prediction, such as two-stage cascade prediction combined with an inverse cascade operation, may be performed in the backward shaping path (eg, additionally, optionally, alternatively, etc.).

[0238] As shown in the figure, the cascade prediction in the backward shaping path includes a first backward prediction operation (denoted as "backward prediction 1") and a second backward prediction operation (denoted as "backward prediction 2").

[0239] Different types of cascaded predictions can be used for the two-stage prediction of the backward shaping path. For example, a relatively simple predictor or prediction operation can be placed in the first stage, while a different, relatively advanced predictor or prediction operation can be placed in the second stage after the simpler predictor or prediction operation to obtain additional prediction accuracy and performance gain.

[0240] In some operating scenarios, the luma and chroma codewords / values ​​for each SDR image in the input SDR signal are encoded into a coded bitstream (e.g., Figure 1A 122, etc.), and is provided as input to the backward shaping path.

[0241] The luminance and chrominance SDR codewords / values ​​constituting each input SDR image in the input SDR signal can be backward shaped into reconstructed luminance and chrominance HDR codewords / values ​​constituting the corresponding reconstructed HDR image in the reconstructed HDR signal. The reconstructed or backward shaped luminance and chrominance HDR codewords / values ​​are similar (e.g., closely, at various quality levels, etc.) to the luminance and chrominance HDR codewords / values ​​in the corresponding input HDR image in the input HDR signal. Neither the input HDR signal nor the reconstructed HDR signal is encoded into the coded bitstream. Instead, image metadata (or synthesizer metadata) specifying a backward shaping mapping (e.g., multiple groups of cascaded prediction operations) is included in the coded bitstream. Instead of decoding the HDR image from the coded bitstream, the receiving decoder or playback device decodes the SDR image and generates a reconstructed HDR image from the SDR image by applying the backward shaping mapping to the SDR image.

[0242] By way of example and not limitation, the first backward predictor or prediction operation includes a first luma backward predictor (e.g., function, curve, lookup table, BLUT, etc.) constructed according to a CDF matching technique as follows:

[0243]

[0244] in, Represents the first predicted luma codeword / value output by the first luma backward predictor.

[0245] An inverse concatenation operation (which may be luma-specific, first-stage-specific, encoder-specific, etc.) may be used to generate a first predicted luma codeword / value and a target luma codeword / value in the input HDR signal to be approximated by the first predicted luma codeword / value. The prediction error between is as follows:

[0246]

[0247] The prediction error may be used as a prediction target value to be predicted or approximated using a second backward predictor or a second luma backward predictor in a prediction operation (eg, MMR mapping), as shown below:

[0248]

[0249] The final prediction value of the luma channel can be generated by adding the prediction values ​​output by both the luma backward predictor and the second luma backward predictor. As shown below:

[0250]

[0251] Similarly, in the backward (shaping) path, for the chroma channels Cb / C0 and Cr / Cl, the first backward predictor or prediction operation comprises a first chroma backward predictor as follows:

[0252]

[0253]

[0254] in, and Represents the first predicted chroma value of the chroma channel based on a single-stage MMR operation.

[0255] A first predicted chroma codeword / value may be generated for each chroma channel using an inverse concatenation operation (which may be luma-specific, first stage-specific, encoder-specific, etc.) The target chroma codeword / value in the reference (or target) HDR signal to be approximated by the first predicted chroma codeword / value The prediction error between is as follows:

[0256]

[0257]

[0258] The prediction error may be used as a predicted target value to be predicted or approximated using a second chroma backward predictor in a second backward predictor or prediction operation (e.g., an MMR map that is higher level than the MMR map used in the first chroma backward predictor in terms of the type of number of levels used, the order or degree of construction, the mapping and functions used, the operation, the number of input variables, the type(s) of input variables, etc.), as shown below:

[0259]

[0260]

[0261] The final prediction value of the chroma channel can be generated by adding the prediction values ​​output by both the forward and reverse predictors. and As shown below:

[0262]

[0263]

[0264] It should be noted that the final predicted luminance and chrominance values ​​will be generated by a receiving device that receives the image metadata (or synthesizer metadata) including multiple sets of cascaded prediction coefficients. and The plurality of sets of cascaded prediction coefficients include specifying first luma and chroma backward predictors The first set of cascaded prediction coefficients and the second luma and chroma backward predictors are specified These backward predictors can be applied by the receiving device to decode the SDR image (including SDR codewords / values) decoded from the encoded bitstream ) to generate the final predicted luminance and chrominance values ​​of the reconstructed HDR image and

[0265] In some operational scenarios, a linear segment-based structure may be used to compute / generate / include a backward shaping map in the image metadata to maintain temporal stability of the image metadata. Example linear segment-based structures are described in U.S. Patent Application Publication Serial No. 2018 / 0007356, published on January 4, 2018, the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0266] Some or all of the techniques as described herein may be implemented and / or performed as part of a real-time operation to produce a video content color grade suitable for a broadcast video application, a real-time streaming application, etc. Additionally, optionally, or alternatively, some or all of the techniques as described herein may be implemented and / or performed as part of a time-delayed or offline operation to produce a video content color grade suitable for a non-real-time streaming application, a cinema application, etc.

[0267] Example process flow

[0268] Figure 4A An example process flow according to an embodiment is illustrated. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, reverse mapping generation and application system, etc.) can perform this process flow. In block 402, an image processing system applies a first predictor to an input image encoded in a first domain to generate a first level prediction codeword that approximates a prediction target codeword of a prediction target image encoded in a second domain different from the first domain.

[0269] In block 404 , the image processing system creates a second level prediction target value by performing an inverse concatenation operation on the prediction target codeword and the first level prediction codeword.

[0270] In block 406, the image processing system applies a second predictor to the input image in the first domain to generate a second level prediction value that approximates a second level prediction target value;

[0271] In block 408, the image processing system generates a plurality of sets of cascaded prediction coefficients. The plurality of sets of cascaded prediction coefficients include a first set of cascaded prediction coefficients that specifies a first predictor. The plurality of sets of cascaded prediction coefficients include a second, different set of cascaded prediction coefficients that specifies a second predictor.

[0272] In block 410, an image processing system encodes multiple sets of concatenated prediction coefficients as image metadata into a video signal. The video signal is further encoded with an input image in a first domain. A receiving device of the video signal uses the multiple sets of concatenated prediction coefficients to apply a first predictor and a second predictor to the input image to generate a predicted image in a second domain. The predicted image in the second domain approximates a predicted target image in the second domain.

[0273] In an embodiment, the video signal represents one of: a single layer backward compatible signal or a single layer inverse display management signal; the second domain has one of: a higher dynamic range than the first domain, the same dynamic range as the first domain, or a lower dynamic range than the first domain.

[0274] In an embodiment, the first predictor and the second predictor represent an overall backward shaping mapping mapping an input image in the first domain to a predicted image in the second domain.

[0275] In an embodiment, the first predictor comprises a first luma predictor and a first chroma predictor; the second predictor comprises a second luma predictor and a second chroma predictor; the first luma predictor and the second luma predictor are different types of luma predictors, and wherein the first chroma predictor and the second chroma predictor are different types of chroma predictors.

[0276] In an embodiment, the second brightness predictor is more advanced than the first brightness predictor.

[0277] In an embodiment, the first brightness predictor represents a single-channel polynomial predictor; the second brightness predictor represents a multi-channel multivariate multiple regression (MMR) predictor.

[0278] In an embodiment, the second chrominance predictor is more advanced than the first chrominance predictor.

[0279] In an embodiment, the first chrominance predictor represents a first-order multivariate multiple regression (MMR) predictor; the second chrominance predictor represents a second-order MMR predictor higher than the first order.

[0280] In an embodiment, the input image in the first domain is a forward shaped image generated by applying a forward shaping map to a predicted target image in the second domain.

[0281] In an embodiment, the forward shaping mapping means one of: a mapping not based on multi-level cascade prediction, or a mapping based on multi-level cascade prediction.

[0282] In an embodiment, the concatenated prediction represents one of: an arithmetic addition operation, an arithmetic multiplication operation, a reversible arithmetic operation, a reversible functional operation, another reversible operation, and the like.

[0283] Figure 4B An example process flow according to an embodiment of the present invention is illustrated. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, prediction model and feature selection system, reverse mapping generation and application system, etc.) can perform this process flow. In block 452, a video decoding system decodes an input image encoded in a first domain from a video signal.

[0284] In block 454, the video decoding system decodes picture metadata including sets of concatenated prediction coefficients from the video signal.

[0285] The multiple groups of cascade prediction coefficients are generated by an upstream image processing device.

[0286] The plurality of sets of cascaded prediction coefficients include a first set of cascaded prediction coefficients, the first set of cascaded prediction coefficients specifying a first predictor, the first set of cascaded prediction coefficients being generated at least in part based on a prediction target image encoded in a second domain. The first domain is different from the second domain.

[0287] The plurality of sets of cascaded prediction coefficients include a second set of cascaded prediction coefficients specifying a second predictor generated based at least in part on a prediction target image in a second domain.

[0288] In block 456, the video decoding system applies the first predictor and the second predictor combined by the cascade operation to the input image in the first domain to generate a reconstructed image in the second domain. The reconstructed image in the second domain approximates the predicted target image in the second domain.

[0289] In block 458 , the video decoding system renders, using a display device, a display image derived from the reconstructed image in the second domain.

[0290] In an embodiment, a computing device such as a display device, a mobile device, a set-top box, a multimedia device, etc. is configured to perform any of the aforementioned methods. In an embodiment, an apparatus includes a processor and is configured to perform any of the aforementioned methods. In an embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause any of the aforementioned methods to be performed.

[0291] In an embodiment, a computing device includes one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, causes performance of any of the foregoing methods.

[0292] Note that although separate embodiments are discussed herein, any combination of embodiments and / or portions of embodiments discussed herein may be combined to form further embodiments.

[0293] Example Computer System Implementation

[0294] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or implement instructions related to adaptive perceptual quantization of images with enhanced dynamic range, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to the adaptive perceptual quantization process described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0295] Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present disclosure. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement the method related to the adaptive perceptual quantization of HDR images as described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention can also be provided in the form of a program product. The program product may include any non-transient medium carrying a set of computer-readable signals, the set of computer-readable signals including instructions that, when executed by a data processor, cause the data processor to perform the method of an embodiment of the present invention. The program product according to an embodiment of the present invention may take any of a variety of forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash memory RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

[0296] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents (e.g., functionally equivalent) to the components, including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

[0297] According to one embodiment, the technology described herein is implemented by one or more special computing devices. Special computing devices can be hard-wired for performing these technologies, or can include digital electronic devices that are permanently programmed to perform these technologies, such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), or can include one or more general-purpose hardware processors that are programmed to perform these technologies according to program instructions in firmware, memory, other storage devices, or combinations. This special computing device can also combine customized hard-wired logic, ASICs or FPGAs with custom programming to implement these technologies. Special computing devices can be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices that merge hard wiring and / or program logic to implement the technology.

[0298] For example, Figure 5 is a block diagram illustrating a computer system 500 upon which embodiments of the present invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general purpose microprocessor.

[0299] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, render computer system 500 into a special-purpose machine customized to perform the operations specified in the instructions.

[0300] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0301] The computer system 500 may be coupled to a display 512, such as a liquid crystal display, via the bus 502 for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 504 and for controlling cursor movement on the display 512. Typically, this input device has two degrees of freedom in two axes, a first axis (e.g., an x-axis) and a second axis (e.g., a y-axis), allowing the device to be positioned in a plane.

[0302] The computer system 500 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, enables or programs the computer system 500 to be a special purpose machine. According to one embodiment, the computer system 500 performs the techniques described herein in response to the processor 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium, such as the storage device 510. Execution of the sequences of instructions contained in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0303] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, diskettes, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs and EPROMs, flash EPROMs, NVRAMs, any other memory chips or storage cartridges.

[0304] Storage media are distinct from but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0305] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 may receive the data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may optionally be stored on the storage device 510 before or after execution by the processor 504.

[0306] The computer system 500 also includes a communication interface 518 coupled to the bus 502. The communication interface 518 provides a two-way data communication coupled to a network link 520, which is connected to a local network 522. For example, the communication interface 518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 can be a local area network (LAN) card to provide a data communication connection with a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 518 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

[0307] The network link 520 typically provides data communication to other data devices through one or more networks. For example, the network link 520 can provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. The ISP 526 in turn provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 528. Both the local network 522 and the Internet 528 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518 that carry the digital data to and from the computer system 500 are example forms of transmission media.

[0308] Computer system 500 can send messages and receive data, including program code, through network(s), network link 520 and communication interface 518. In the Internet example, server 530 can transmit the requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0309] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage, for later execution.

[0310] Equivalents, extensions, alternatives and miscellaneous

[0311] In the foregoing description, embodiments of the present invention have been described with reference to many specific details, which may vary depending on the implementation. Therefore, the only and exclusive indication of the claimed embodiments of the present invention and the claimed embodiments of the present invention considered by the applicant is the set of claims issued in a specific form according to the present application, wherein such claim issuance includes any subsequent amendments. Any definitions expressly set forth herein for the terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, characteristics, features, advantages or attributes not expressly cited in a claim should not limit the scope of such claim in any way. Therefore, this specification and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A method for image processing, comprising: applying a first predictor to an input image encoded in a first domain to generate a first level prediction codeword that approximates a prediction target codeword of a prediction target image encoded in a second domain different from the first domain; Creating a second-level prediction target value by performing an inverse concatenation operation on the prediction target codeword and the first-level prediction codeword; applying a second predictor to the input image in the first domain to generate a second level prediction value that approximates the second level prediction target value; generating a plurality of sets of cascaded prediction coefficients, wherein the plurality of sets of cascaded prediction coefficients include a first set of cascaded prediction coefficients specifying the first predictor, wherein the plurality of sets of cascaded prediction coefficients include a second, different set of cascaded prediction coefficients specifying the second predictor; encoding the plurality of sets of cascaded prediction coefficients as image metadata into a video signal, wherein the video signal is further encoded with an input image in the first domain, wherein a receiving device of the video signal is caused to use the plurality of sets of cascaded prediction coefficients to apply the first predictor and the second predictor to the input image to generate a predicted image in the second domain, wherein the predicted image in the second domain approximates a predicted target image in the second domain, wherein the first predictor comprises a first luminance predictor and a first chrominance predictor, wherein the second predictor comprises a second luminance predictor and a second chrominance predictor, wherein the first luminance predictor and the second luminance predictor are different types of luminance predictors, and wherein the first chrominance predictor and the second chrominance predictor are different types of chrominance predictors, and wherein the first luminance predictor represents a single-channel polynomial predictor, and wherein the second luminance predictor represents a multi-channel multivariate multivariate regression (MMR) predictor.

2. The method of claim 1, wherein: The video signal represents one of: a single layer backward compatible signal, or a single layer inverse display management signal; and wherein the second domain has one of: a higher dynamic range than the first domain, the same dynamic range as the first domain, or a lower dynamic range than the first domain.

3. The method according to claim 1 or 2, wherein: The first predictor and the second predictor represent an overall backward shaping mapping that maps an input image in the first domain to a predicted image in the second domain.

4. The method according to claim 1 or 2, wherein: The second chrominance predictor is higher level than the first chrominance predictor.

5. The method according to claim 1 or 2, wherein: The first chrominance predictor represents a second order multivariate multiple regression MMR predictor, and wherein the second chrominance predictor represents a third order MMR predictor, higher than the first order.

6. The method according to claim 1 or 2, wherein: The input image in the first domain is a forward shaped image generated by applying a forward shaping map to a predicted target image in the second domain.

7. The method of claim 6, wherein: The forward shaping mapping means one of the following: a mapping not based on multi-level cascade prediction, or a mapping based on multi-level cascade prediction.

8. The method according to claim 1 or 2, wherein: The cascade prediction represents one of: a reversible arithmetic operation or a reversible functional operation.

9. A method for image processing, comprising: decoding an input image encoded in a first domain from a video signal; decoding image metadata including a plurality of sets of concatenated prediction coefficients from the video signal; Wherein, the multiple groups of cascade prediction coefficients are generated by an upstream image processing device; wherein the plurality of sets of cascaded prediction coefficients include a first set of cascaded prediction coefficients, the first set of cascaded prediction coefficients specifying a first predictor, the first set of cascaded prediction coefficients being generated based at least in part on a prediction target image encoded in a second domain; wherein the first domain is different from the second domain; wherein the plurality of sets of cascaded prediction coefficients include a second set of cascaded prediction coefficients, the second set of cascaded prediction coefficients specifying a second predictor, the second set of cascaded prediction coefficients being generated based at least in part on a predicted target image in the second domain, wherein the first predictor includes a first luma predictor and a first chroma predictor, wherein the second predictor includes a second luma predictor and a second chroma predictor, wherein the first luma predictor and the second luma predictor are different types of luma predictors, and wherein the first chroma predictor and the second chroma predictor are different types of chroma predictors, and wherein the first luma predictor represents a single-channel polynomial predictor, and wherein the second luma predictor represents a multi-channel multivariate multivariate regression (MMR) predictor; applying the first predictor and the second predictor to the input image in the first domain to generate a reconstructed image in the second domain, wherein the reconstructed image in the second domain approximates a predicted target image in the second domain, wherein a concatenation operation combines a first level prediction codeword generated by applying the first predictor to the input image with a second level prediction value generated by applying the second predictor to the input image; A display image derived from the reconstructed image in the second domain is caused to be rendered using a display device.

10. The method of claim 9, wherein: The first predictor and the second predictor represent an overall backward shaping mapping that maps an input image in the first domain to a reconstructed image in the second domain.

11. The method according to claim 9 or 10, wherein: The display image is derived directly from a second reconstructed image encoded in a third domain, wherein the second reconstructed image of the third domain is approximate to a second predicted target image of the third domain; wherein the second reconstructed image in the third domain is generated in part by combining a third level prediction value with a reconstructed codeword in the reconstructed image in the second domain using a second cascade operation, wherein the third level prediction value is generated by applying a third predictor to the input image.

12. An apparatus for image processing, the apparatus comprising a processor and configured to execute the method according to any one of claims 1 to 11.

13. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing the method according to any one of claims 1 to 11 using one or more processors.

Images