B-smooth curve predictor based on tensor product

Abstract

A set of tensor product B-smooth curve (TPB) basis functions are determined. A set of selected TPB prediction parameters are generated for use with the set of TPB basis functions to generate predicted image data in a mapped image from source image data in a source image at source color levels. The set of selected TPB prediction parameters are generated by minimizing the difference between the predicted image data in the mapped image and reference image data in a reference image at reference color levels. The reference image corresponds to the source image and depicts the same visual content as depicted by the source image. The set of selected TPB prediction parameters are encoded in a video signal as part of image metadata along with the source image data in the source image. The mapped image is reconstructed and reproduced by a receiving device of the video signal.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 908,770, filed on October 1, 2019, and European Patent Application No. 19200793.8, filed on October 1, 2019, the entire contents of which are hereby incorporated by reference herein. Technical Field

[0003] The present disclosure relates generally to images. More particularly, embodiments of the present disclosure relate to a B-smooth curve predictor of a tensor product. Background Art

[0004] As used herein, the term "dynamic range (DR)" may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., brightness, luminance) of an image, e.g., from the darkest black (darkness) to the brightest white (bright light). In this sense, DR relates to "scene-referenced" intensity. DR may also relate to the ability of a display device to properly or approximately reproduce a range of intensities of a particular width. In this sense, DR relates to "display-referenced" intensity. Unless it is explicitly stated at any point in the description herein that a particular meaning has a particular meaning, it should be inferred that the term may be used in either sense (e.g., interchangeably).

[0005] As used herein, the term "high dynamic range (HDR)" relates to a DR width that spans about 14 to 15 or more orders of magnitude across the human visual system (HVS). In practice, the DR of the broad width within which humans can simultaneously perceive an intensity range may be slightly narrowed relative to HDR. As used herein, the terms "enhanced dynamic range (EDR)" or "visual dynamic range (VDR)" may individually or interchangeably relate to the DR that can be perceived in a scene or image by the human visual system (HVS) including eye movement to allow for some light adaptation changes across the scene or image. As used herein, EDR may relate to a DR that spans 5 to 6 orders of magnitude. Thus, although it may be slightly narrowed relative to a real scene reference HDR, EDR still represents a wide DR width and may also be referred to as HDR.

[0006] In practice, an image includes one or more color components (e.g., luminance Y and chrominance Cb and Cr) of a color space, wherein each color component is represented by a precision of n bits per pixel (e.g., n=8). Using nonlinear luminance coding (e.g., gamma coding), images where n≤8 (e.g., color 24-bit JPEG images) are considered as images of standard dynamic range, while images where n>8 can be considered as images of enhanced dynamic range.

[0007] The reference electro-optical transfer function (EOTF) of a given display characterizes the relationship between the color values ​​(e.g., brightness) of an input video signal and the output screen color values ​​(e.g., screen brightness) produced by the display. For example, ITU Rec. ITU-R BT.1886, "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011), which is incorporated herein by reference in its entirety, defines a reference EOTF for a flat panel display. Given that a video is streamed, information about its EOTF may be embedded in the bitstream as (image) metadata. The term "metadata" is herein related to any auxiliary information that is transmitted as part of the encoded bitstream and facilitates the decoder to reproduce the decoded image. This metadata may include, but is not limited to, color space or color gamut information, reference display parameters, and auxiliary signal parameters, as described herein.

[0008] The term "PQ" used herein refers to the quantization of the perceived brightness amplitude. The human visual system responds to increasing brightness in a very nonlinear manner. The ability of humans to see a stimulus is affected by the brightness of the stimulus, the size of the stimulus, the spatial frequency of the stimulus, and the brightness level to which the eye has adapted at a particular moment when the human is viewing the stimulus. In some embodiments, the perceptual quantization function maps a linear input grayscale to an output grayscale that better matches the contrast sensitivity threshold of the human visual system. An exemplary PQ mapping function is described in "High Dynamic Range EOTF of Mastering Reference Displays" (hereinafter referred to as "SMPTE") of SMPTE 2084:2014 (hereinafter referred to as "SMPTE"), which is incorporated herein by reference in its entirety, wherein in view of a fixed stimulus size, for each brightness level (e.g., stimulus level, etc.), the minimum visible contrast step at the brightness level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

[0009] Support 200cd / m 2 or nits to 1,000cd / m 2A display with a brightness of 0.16 nits or nits represents a lower dynamic range (LDR) relative to EDR (or HDR), also known as standard dynamic range (SDR). EDR content can be displayed on an EDR display that supports a higher dynamic range (e.g., from 1,000 nits to 5,000 nits or more). Such a display may be defined using an alternative EOFT that supports high brightness capabilities (e.g., 0 nits to 10,000 or more nits). An example of this EOFT is defined in SMPTE 2084 and Rec. ITU-R BT.2100, "Image parameter values ​​for high dynamic range television for use in production and international programme exchange" (06 / 2017). The inventors should appreciate that we desire improved techniques for combining video content data that can be used to support the display capabilities of various SDR and HDR display devices.

[0010] The methods described in this section are methods that can be pursued, but not necessarily methods that have been previously conceived or pursued. Therefore, unless otherwise indicated, any method described in this section should not be considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, problems identified with respect to one or more methods should not be considered to have been identified in any prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings in which like reference numerals refer to similar elements and in which:

[0012] Figure 1A Describes an example process of a video delivery pipeline; Figure 1B Describe an example processing block for TPB cross-channel prediction; Figure 1C and Figure 1D An exemplary implementation of a TPB base generation operation is described; Figure 1E and Figure 1F An exemplary implementation of the TPB cross product operation is described; Figure 1G and Figure 1H An example flow chart illustrating the use of TPB prediction;

[0013] Figure 2A to Figure 2C Describe an example codec architecture;

[0014] Figures 3A to 3D A complete set of examples of B-smooth curve basis functions illustrating uniformly distributed knots;

[0015] Figure 4A and Figure 4B Describe an example process flow; and

[0016] Figure 5 A simplified block diagram illustrating an example hardware platform upon which the computers or computing devices described herein may be implemented. DETAILED DESCRIPTION

[0017] In the following description, many specific details are set forth for explanation to provide a thorough understanding of the present disclosure. However, it should be understood that the present disclosure can be practiced without these specific details. In other examples, well-known structures and devices are not described in detail to avoid unnecessary obstruction, obscurity or confusion of the present disclosure.

[0018] Overview

[0019] A tensor product B-smooth curve (TPB) predictor is described herein. The techniques described herein may be used to perform prediction operations in a video content processing pipeline and use an upstream video content processor to generate combiner metadata to enable a downstream video content processor to reconstruct video content of a first different dynamic range (e.g., HDR, SDR, etc.) from decoded video content of a second dynamic range (e.g., SDR, HDR, etc.) carried in a video signal. In some operating schemes, a single-channel luma predictor and a single-piece (e.g., applicable to some or all possible input luma and chroma codewords, etc.) multi-channel multiple regression (MMR) chroma predictor may be used to generate combiner metadata. The single-channel luma predictor may limit the color saturation of pixels with the same luma. The single-piece MMR limits local color prediction by imposing a global mapping. The limitation may cause less color precision and greater color inaccuracy of the reconstructed or mapped image and produce relatively large color differences in some portions (e.g., bright colors or highly saturated colors, etc.) of a colorimetric chart (e.g., containing some or all possible colors to be represented or measured, etc.). Example single-channel luma prediction and MMR chroma prediction operations are described in U.S. Provisional Patent Application No. 62 / 781,185, filed on December 18, 2018, the entire contents of which are incorporated herein as if fully set forth by reference.

[0020] In some operating scenarios, the TPB predictor described herein may be used in a video content processing pipeline to generate combiner metadata by an upstream video content processor to enable a downstream video content processor to reconstruct images with better image quality and relatively higher color accuracy.

[0021] B-smooth curves use polynomials with continuity constraints of a specific order to process the properties used to approximate a given curve or its segment. B-smooth curves can be used in regression processes by performing data fitting only on the top of the B-smooth curve model. In order to capture multiple input variables of relatively high dimensions in video processing, a B-smooth curve (or TPB) of a tensor product can be specified by multiplying multiple B-smooth curve functions to achieve a relatively high-dimensional approximate representation of mappings, curves, etc. to associate multiple input variables with target values ​​(or output variables). Compared with combining a single-channel brightness predictor and a single-piece MMR to generate combiner metadata, using TPB to generate combiner metadata provides a much better tool to model the transformation between different color levels of different dynamic ranges. In addition, optionally or alternatively, TPB can be used to provide a natural continuous curve (whose inherent ability ensures continuity up to a specific order) to avoid or significantly reduce the curve fitting operations after the initial generation of predictions, transformations and / or mappings. For example, the computationally intensive polynomial approximation algorithm used to ensure the continuity of multiple polynomial segments applied to a single-channel brightness predictor can be avoided according to the techniques described herein.

[0022] TPB prediction can be used in different coding schemes such as scene-based and linear-based scenarios. A three-dimensional mapping table (3DMT) can be used in conjunction with TPB prediction to reduce visually perceptible color artifacts. Experimental results show that the prediction accuracy of brightness can be improved by 10 to 40 times and the prediction accuracy of chroma can be improved by 1 to 4 times.

[0023] The exemplary embodiments described herein relate to generating and encoding TPB prediction parameters for image reconstruction. A set of tensor product B smooth curve (TPB) basis functions are determined. A set of selected TPB prediction parameters for use with the set of TPB basis functions are generated to generate predicted image data in one or more mapped images from source image data in one or more source images of a source color level. The set of selected TPB prediction parameters is generated by minimizing the difference between the predicted image data in the one or more mapped images and the reference image data in one or more reference images of a reference color level. The one or more reference images correspond to the one or more source images and depict visual content that is the same as the visual content depicted by the one or more source images. The set of selected TPB prediction parameters is encoded in a video signal together with the source image data in the one or more source images as part of image metadata. The one or more mapped images are reconstructed and reproduced by a receiving device of the video signal.

[0024] The exemplary embodiments described herein relate to decoding TPB prediction parameters for image reconstruction and reproduction. One or more first images of a first color level are decoded from a video signal. Image metadata including a set of selected TPB prediction parameters for multiplication with a set of tensor product B smooth curve (TPB) basis functions are decoded from the video signal. The set of selected TPB prediction parameters is generated by an upstream video content processor. The set of selected TPB prediction parameters is used together with the set of TPB basis functions to generate predicted image data in one or more mapped images from first image data in one or more first images of the first color level. The upstream video content processor generates the set of selected TPB prediction parameters by minimizing the difference between the predicted image data in the one or more mapped images and the reference image data in one or more reference images of a reference color level. The one or more reference images correspond to the one or more first images and depict visual content that is the same as the visual content depicted by the one or more first images. The set of TPB prediction parameters is used together with the set of TPB basis functions to generate the one or more mapped images from the one or more first images. One or more display images derived from the one or more mapped images are reproduced by a display device.

[0025] Example Video Transmission Processing Pipeline

[0026] Figure 1A An example process of a video delivery pipeline (100) showing various stages from video capture / generation to HDR or SDR display is depicted. Example HDR displays may include, but are not limited to, image displays operated in conjunction with a TV, mobile device, home theater, etc. Example SDR displays may include, but are not limited to, SDR TVs, mobile devices, home theater displays, head mounted display devices, wearable display devices, etc.

[0027] The video frames (102) are captured or generated using an image generation block (105). The video frames (102) may be digitally captured (e.g., by a digital camera) or generated by a computer (e.g., using computer animation, etc.) to provide video data (107). Additionally, optionally, or alternatively, the video frames (102) may be captured on film by a film camera. The film is converted to a digital format to provide the video data (107). In some embodiments, the video data (107) may be edited or transformed into an image sequence (e.g., automatically without human input, manually, automatically with human input, etc.) before being passed to the next processing stage / phase of the video delivery pipeline (100).

[0028] The video data (107) is then provided to a processor for post-production editing (115). Post-production editing (115) may include adjusting or modifying the color or brightness of specific areas of the image to improve image quality or achieve a specific look for the image in accordance with the creative intent of the video creator. This is sometimes referred to as "color adjustment" or "color grading." Other edits (e.g., scene selection and sequencing, manual and / or automatic scene editing information generation, image cropping, computer-generated visual effects addition, etc.) may be performed at the post-production editing (115) to produce a revised version of the HDR image (117-1) or SDR (or relatively narrow dynamic range) image (117) (e.g., SDR, etc.).

[0029] In some embodiments, during post-production editing (115), the HDR image (117-1) is viewed on a reference HDR display that supports high dynamic range by a colorist performing post-production editing operations on the HDR image (117-1).

[0030] In some other embodiments, during post-production editing (115), the SDR image (117) is viewed on a reference display (125) that supports a standard dynamic range (or a relatively narrow dynamic range) by a colorist performing post-production editing operations on the SDR image (117).

[0031] In some embodiments, the encoding block (120) may implement, for example, Figure 2A or Figure 2B In an operating scenario where the encoding block (120) receives an HDR image (117-1) from a post-production editor (115), the HDR image (117-1) may be forward reshaped by the encoding block (120) into an SDR image (e.g., 117).

[0032] The SDR image (117) is compressed by an encoding block (120) into an encoded bitstream (122), for example, in a single layer. In some embodiments, the encoding block (120) may include an audio and video encoder for generating the encoded bitstream (122), such as those defined by ATSC, DVB, DVD, Blu-Ray, and other transport formats.

[0033] In some embodiments, the encoded bitstream (122) is encoded using an SDR image (117) that preserves the artistic intent of producing the SDR image (117) in post-production editing (115). Additionally, optionally, or alternatively, in some embodiments, the encoded bitstream (122) is encoded using an SDR image (117) that preserves the artistic intent of producing the HDR image (117-1) in post-production editing (115) that is forward reshaped into the SDR image (117).

[0034] The encoding block (120) may encode the SDR image (117) into video data in a video signal (e.g., an 8-bit SDR video signal, a 10-bit SDR video signal, etc.) that is backward compatible (or alternatively, non-backward compatible) with various SDR display devices (e.g., an SDR display, etc.). In a non-limiting example, the video signal encoded using the SDR image (117) may be a single-layer backward compatible (or alternatively, non-backward compatible) video signal.

[0035] In some embodiments, the encoded bitstream (122) is a video signal conforming to the same video signal format as the input SDR YCbCr video signal received by the encoding block (120). For example, in the case where the input SDR YCbCr video signal received by the encoding block (120) is an 8-bit SDR YCbCr video signal, the encoded bitstream (122) output by the encoding block (120) may represent an output 8-bit SDR YCbCr video signal with image metadata (which includes (but is not limited to) combiner metadata generated by the encoding block (120) and / or the post-production block (115). The combiner metadata (or inverse reconstruction mapping) can be used by a downstream decoder to perform inverse reconstruction (e.g., inverse tone mapping, etc.) on the SDR image (117) to produce an inverse reconstructed image that can be relatively accurately reproduced on (e.g., multiple target, etc.) HDR reference displays.

[0036] In some embodiments, one or more SDR to HDR conversion tools may be used to perform inverse tone mapping based at least in part on the combiner metadata to produce a reverse reshaped image from the SDR image (117) (or a decoded version thereof). As used herein, reverse reshaping refers to an image processing operation that converts the requantized image back to the original EOTF domain (e.g., gamma or PQ) or a different EOTF domain for further downstream processing (e.g., display management). Additionally, optionally, or alternatively, the reshaping (e.g., forward reshaping, reverse reshaping, etc.) described herein may refer to image processing operations that convert between different EOTFs, different color spaces, different dynamic ranges, etc.

[0037] The encoded bitstream (122) is further encoded using image metadata including, but not limited to, display management (DM) metadata that can be used by a downstream decoder to perform display management operations on an inverse reshaped image for an HDR reference display to produce a display image that is rendered on (e.g., multiple, etc.) device-specific HDR displays.

[0038] The encoded bitstream (122) is then transmitted downstream to a receiver, such as a decoding and playback device, a media source device, a media streaming client device, a television (such as a smart TV, etc.), a video converter, a movie theater, and the like. In the receiver (or downstream device), the encoded bitstream (122) is decoded by a decoding block (130) to produce a decoded image 182, which may be the same as the SDR image (117), subject to quantization errors generated in the compression performed by the encoding block (120) and the decompression performed by the decoding block (130).

[0039] Example SDR video content represented by the SDR image (117) or a decoded version thereof may be (but is not necessarily limited to being) SDR+ video content, an SDR image, an SDR movie version, an SDR+ image, an SDR media program, etc. As used herein, the term "SDR+" refers to a combination of SDR image data and metadata that, when combined together, allow the generation of corresponding high dynamic range (HDR) image data. The SDR+ image metadata may include combiner data for generating an inverse reshaping map (e.g., a TPB inverse reshaping map, etc.) that, when applied to an input SDR image, generates a corresponding HDR image. The SDR+ image allows for backward compatibility with legacy SDR displays that may ignore the SDR+ image metadata and only display the SDR image.

[0040] The image metadata transmitted to the receiving device along with the SDR video content may include combiner metadata generated according to the techniques described herein (e.g., automatically, in real time, in an offline process, etc.). In some embodiments, the video data (107) is provided to a processor for combiner metadata generation (115). The combiner metadata generation (115) may automatically generate the combiner metadata with little or no human interaction. The automatically generated combiner metadata may be used by the receiving device(s) to perform an inverse reshaping operation to generate a corresponding high dynamic range (HDR) image from the SDR image in the video data (107).

[0041] Combiner metadata generation (115) may be used to provide one or more valuable services of making video content available to a variety of display devices. A valuable service provided by combiner metadata generation (115) is the generation of HDR images from SDR images in operating scenarios where HDR images of video content depicted in the SDR images are not available, but SDR images depicting the video content are available, as mentioned above. Thus, the techniques described herein may be used to generate or combine HDR video content for HDR displays in operating scenarios where SDR images are available.

[0042] Another valuable service provided by the combiner metadata generation (115) is the ability to generate HDR video content for an HDR display (e.g., completely or partially, etc.) without relying on some or all of the manual work of a colorist (referred to as "color adjustment" or "color grading").

[0043] Additionally, optionally or alternatively, the DM metadata in the image metadata may be used by a downstream decoder to perform display management operations on the inversely reshaped image to produce a display image (e.g., an HDR display image, etc.) for reproduction on an HDR reference display device or other display device (e.g., a non-reference HDR display device, etc.).

[0044] In an operating scenario where the receiver operates with (or is attached to) an SDR display 140 that supports standard dynamic range or a relatively narrow dynamic range, the receiver may cause the decoded SDR image to be rendered directly or indirectly on the target display (140).

[0045] In an operating scenario where the receiver operates with (or is attached to) an HDR display 140-1 that supports a high dynamic range (e.g., 400 nits, 1000 nits, 4000 nits, 10000 nits or greater, etc.), the receiver may extract combiner metadata (e.g., TPB combiner metadata, etc.) from the encoded bitstream (122) (e.g., a metadata container in the encoded bitstream (122)), etc. and use the combiner metadata to combine an HDR image (132), which may be a reverse reshaped image generated from a reverse reshaped SDR image based on the combiner metadata. In addition, the receiver may extract DM metadata from the encoded bitstream (122) and apply a DM operation (135) to the HDR image (132) based on the DM metadata to generate a display image (137) for reproduction on an HDR (e.g., non-reference, etc.) display device (140-1) and cause the display image (137) to be reproduced on the HDR display device (140-1).

[0046] Codec Architecture

[0047] In some operational scenarios, SDR+ may be used to enhance SDR content for rendering on an HDR display device. Image metadata such as combiner metadata may be used to separately map the luma and chroma channels (or color space components) of an SDR image to generate (map) corresponding luma and chroma channels of an HDR image.

[0048] However, it should be noted that in various embodiments, the techniques described herein may be used with a single layer inverse display management (SLiDM) or non-SLiDM codec architecture. For example, the TPB metadata generation, transmission, and consumption described herein may be used in conjunction with a multi-layer video signal including two or more coded layers (including SDR or HDR content).

[0049] Additionally, optionally or alternatively, the TPB metadata generation, transmission and consumption described herein may be used in conjunction with non-SDR (or non-SDR+) video signals, such as HDR signals. For example, rather than including TPB reverse reconstruction metadata used by a receiving device to reversely reconstruct decoded SDR content to produce reconstructed HDR content, an HDR signal may include TPB forward reconstruction metadata used by a receiving device to forward reconstruct decoded HDR content to produce reconstructed SDR content.

[0050] Thus, although the techniques described herein may be described using an example codec architecture or video signals for illustration only, the techniques are not limited to the example codec architecture or video signals but may be implemented using other codec architectures or video signals.

[0051] Figure 2A to Figure 2C An exemplary codec architecture is described. More specifically, Figure 2A An example of a first encoder-side codec architecture that may be implemented using one or more computing processors in an upstream video encoder or the like is described. Figure 2B An example of a second encoder-side codec architecture that may be implemented using one or more computing processors in an upstream video encoder or the like is described. Figure 2C An example of a decoder-side codec architecture is described that may also be implemented using one or more computing processors in a downstream video decoder (eg, a receiver, etc.).

[0052] In the first framework, Figure 2A As described in , a backward-compatible SDR image (such as SDR image (117) or the like) is received as input on the encoder side of the codec architecture.

[0053] By way of illustration and not limitation, an inverse dynamic range mapping (DM) module 162 (which may represent an SDR to HDR conversion tool, etc.) is used to convert the SDR image (117) into an HDR image 148 for viewing on a reference HDR display. In some embodiments, the inverse DM module may also be referred to as an inverse tone mapping tool.

[0054] exist Figure 2B In the second illustrated architecture, an HDR image (148) for a reference HDR display or the like is received as input on the encoder side of the codec architecture. Here, an "HDR image for a reference HDR display" may refer to an HDR image that is color graded specifically for an HDR (reference) display.

[0055] By way of illustration and not limitation, forward reshaping module 164 (which may represent an HDR to SDR conversion tool, etc.) is used to convert the HDR image (148) into an SDR image (117) for viewing on an SDR display. In some embodiments, the forward reshaping module may also be referred to as a tone mapping tool.

[0056] In both the first encoder-side codec architecture and the second encoder-side codec architecture, an image metadata generator 150 (e.g., a portion of a coding block (120) etc.) receives both an SDR image (117) and an HDR image (148) as inputs, generates image metadata 152, such as TPB combiner metadata, DM metadata, etc. An inverse reshaped image (132) for an HDR (e.g., reference, target, etc.) display may be generated by inverse reshaping the SDR image (117) using an inverse reshaping function / curve specified in the TPB combiner metadata.

[0057] In some embodiments, the reverse reshaped image (132) represents a production quality or near production quality HDR image for an HDR display. The reverse reshaped image (132) can output the HDR video signal 160 to an HDR display device (e.g., via an HDMI interface, via a video link, etc.) and be reproduced on the HDR display device. In these embodiments, the receiver can retrieve the TPB combiner metadata and reconstruct and reproduce the HDR image reconstructed by reverse reshaping the SDR image (117) based on the TPB combiner metadata.

[0058] In both the first encoder-side architecture and the second encoder-side architecture, the compression block 142 (eg Figure 1A The SDR image (117) is compressed / encoded in a single layer 144 of a video signal (e.g., a portion of a coding block (120) of a video signal). An exemplary video signal may be (but is not necessarily limited to) Figure 1A The image metadata (152) (denoted as "rpu") generated by the image metadata generator (150) may be (e.g., by Figure 1A The coding blocks (120), etc.) are encoded into a video signal (eg, a coded bit stream, etc.).

[0059] In both the first encoder-side architecture and the second encoder-side architecture, the image metadata (152) may be carried in the video signal separately from the single layer in which the SDR image is encoded in the video signal. For example, the image metadata (152) may be encoded in a component stream of the encoded bitstream that may or may not be separate from the single layer (of the encoded bitstream) in which the SDR image (117) is encoded.

[0060] In both the first encoder-side architecture and the second encoder-side architecture, TPB combiner metadata in the image metadata (152) in the video signal may be used to enable a downstream receiver to inversely reshape an SDR image (117) (which may be encoded in the video signal) into a reconstructed image (or inversely reshaped image) for an HDR target display. An exemplary HDR target display may include (but is not necessarily limited to) any of the following: an HDR target display having display capabilities similar to those of an HDR reference display, an HDR target display having display capabilities different from those of the HDR reference display, an HDR target display using additional DM operations to map reconstructed video content of the HDR reference display to display video content of the HDR target display, etc.

[0061] In some operating schemes, in both the first encoder-side architecture and the second encoder-side architecture, the SDR content is encoded and transmitted in a single layer of a video signal (e.g., an encoded bitstream (122), etc.) by an upstream encoding device implementing the encoder-side codec architecture. The SDR content is received and decoded in a single layer of the video signal by a downstream decoding device (or receiver) implementing the decoder-side codec architecture. TPB combiner metadata is also encoded and transmitted in the video signal with the SDR content, so that the receiving device can reconstruct the HDR content based on the SDR content and the TPB combiner metadata.

[0062] In some embodiments, Figure 2C As illustrated in the drawings, a video signal encoded using an SDR image (117) in a single layer (144) and TPB inverse reconstruction metadata (152) as part of the overall image metadata are received as input on the decoder side of a first encoder side architecture and a second encoder side architecture.

[0063] Decompression block 154 (eg Figure 1A The decoded SDR image (182) may be a portion of a decoding block (130) of a video signal (e.g., a portion of a decoding block (130) of a video signal) that decompresses / decodes the compressed video data in the single layer (144) of the video signal into a decoded SDR image (182). The decoded SDR image (182) may be the same as the SDR image (117) subject to quantization errors in the compression block (142) and the decompression block (154). The decoded SDR image (182) may output an SDR video signal 156 (e.g., via an HDMI interface, via a video link, etc.) to and reproduced on an SDR display device.

[0064] In addition, the inverse reshaping block 158 extracts image metadata (152) (e.g., TPB combiner metadata (or inverse reshaping metadata)) from the input video signal, constructs an inverse reshaping function based on the extracted TPB combiner metadata in the image metadata, and performs an inverse reshaping operation on the decoded SDR image (117) based on the inverse reshaping function to generate an inverse reshaping image (132) (or a reconstructed HDR image) for a specific HDR target display.

[0065] In some embodiments, the reverse reshaped image represents a production quality or near production quality HDR image of an HDR reference display. The reverse reshaped image (132) can be output as an HDR video signal 160 (e.g., via an HDMI interface, via a video link, etc.) to and reproduced on an HDR display having display capabilities similar to those of the HDR reference display. Thus, in these embodiments, DM functionality may not be implemented by the receiver to simplify device operation and reduce device cost.

[0066] Additionally, optionally or alternatively, in some embodiments, the DM metadata may be transmitted to a receiver along with the TPB combiner metadata and the SDR image (117). Display management operations specific to an HDR target display having display capabilities different from the display capabilities of the HDR reference display may be performed on the inverse reshaped image (132) based at least in part on, for example, the DM metadata in the image metadata (152) to produce a display HDR image that is reproduced on the HDR target display.

[0067] Prediction based on B smooth curve

[0068] TPB (TPB-based) prediction can be used to derive selected operating parameters for mapping or transformation between or among different color grades that depict the same visual semantic content. As used herein, a color grade may refer to a version of a video image (e.g., revised, professionally color graded, color graded by a user or video professional, reconstructed, to be predicted, etc.).

[0069] The mapping or transformation between two color levels depicting the same visual semantic content may refer to a forward reshaping mapping or transformation, a reverse reshaping mapping or transformation, etc. Example reshaping operations are described in U.S. Provisional Patent Application No. 62 / 136,402 filed on March 20, 2015 (also published on January 18, 2018 as U.S. Patent Publication Application No. 2018 / 0020224), U.S. Provisional Patent Application No. 62 / 670,086 filed on May 11, 2018, the entire contents of which are incorporated herein by reference as if fully set forth.

[0070] In some operational aspects, forward reshaping refers to generating a video image of the same or lower dynamic range from a video image of a higher dynamic range (depicting the same visual semantic content).

[0071] In some operational scenarios, forward reshaping refers to producing a video image to be encoded of the same or different dynamic range (e.g., in an output video signal transmitted from an upstream video content processor to (several) downstream video content processors) from an input or source video image of the same dynamic range (depicting the same visual semantic content).

[0072] In some operational aspects, inverse reshaping refers to generating a higher dynamic range video image from a relatively lower dynamic range video image (depicting the same visual semantic content).

[0073] In some operational scenarios, inverse reshaping refers to generating video images of the same or different dynamic range from received video images of the same dynamic range (depicting the same visual semantic content) (e.g., encoded in a received video signal transmitted from an upstream video content processor to (several) downstream video content processors).

[0074] The TPB prediction techniques described herein may be used to generate image metadata or combiner metadata representing a forward reshaping mapping / transformation and / or a reverse reshaping mapping / transformation. B-smooth curve-based prediction may be performed for a single variable (e.g., a single color channel of a color space, a single color component of a color space, a channel in R, G, B channels, a channel in Y, Cb, Cr channels, etc.) and multiple variables (e.g., two or more color channels of a color space, two or more color components of a color space, two or more channels in R, G, B channels, two or more channels in Y, Cb, Cr channels, etc.) via a B-smooth curve of a tensor product. In some operating schemes, TPB prediction may be implemented in the SDR to HDR or HDR to SDR prediction process.

[0075] Table 1 below illustrates example polynomial smoothing curves.

[0076] Table 1

[0077]

[0078]

[0079] B-smooth curve basis functions can be constructed from piecewise polynomials that blend smoothly at knots (e.g., breakpoints connecting adjacent piecewise polynomials, etc.) to achieve a desired smoothness or continuity constraint. A B-smooth curve basis function consists of (n+1) polynomial segments of degree n (or order n), joined with knots of continuity or differentiability up to and including (n-1) order. Using the full set of B-smooth curve bases (or basis functions), a given function f(z) (e.g., a mapping, a curve, etc.) can be expressed by a linear combination of D=T+n-1 B-smooth curve basis functions (T represents the number of knots) as follows:

[0080]

[0081] It should be noted that the B-smooth curve basis function is only positive or supported on intervals based on two adjacent knots among the (eg, T, n+2, etc.) knots.

[0082] The 0th order (zeroth order) of the B smoothing curve basis function can be specified as follows:

[0083]

[0084] Higher order (nth order) basis functions for B smooth curves can be specified recursively as follows:

[0085]

[0086] Except for the internal nodes k0, k1, ..., k T-1 In addition, external junctions (eg, 2n external junctions) may also be placed outside [a, b] (which is the range in which the internal junctions are distributed).

[0087] Figures 3A to 3D Four example ensembles of 0th-order (zeroth-order) to 3rd-order B-smooth curve basis functions for a set of uniformly distributed knots (or knot points) are illustrated. Additional ensembles of B-smooth curve basis functions of different orders may be recursively constructed using expression (3) above.

[0088] In the operating scheme where the number of internal knots T = 8 knots and the full set of 2nd order B smooth curve basis functions is used, the total number of basis functions in the set D is 9. The corresponding set of 9 coefficients {m t A one-dimensional (1D) mapping or curve that can be used to predict or approximate a single-channel prediction of, for example, the luma channel of a color space used to represent video content.

[0089] For illustration only, a full set of 9 2nd order B-smooth curve basis functions are applied to predict or approximate a ID mapping or curve between HDR and SDR.

[0090] Assume that each of an SDR image (e.g., the jth SDR image, etc.) and a corresponding HDR image (e.g., the jth HDR image depicting the same visual semantic content as the SDR image but with a relatively higher dynamic range, etc.) comprises P pixels. and represents the normalized Y, C0 (e.g., Cb), and C1 (e.g., Cr) values ​​of the i-th pixel in the j-th SDR and HDR images, respectively. The single-channel B smoothing curve predictor can be used to obtain the normalized Y, C0 (e.g., Cb), and C1 (e.g., Cr) values ​​of the i-th pixel in the j-th SDR and HDR images, respectively. predict

[0091] A particular set of knots or nodes may be selected and used to construct a structure represented as D y The complete set of (=T+n-1) B smooth curve basis functions. A set of corresponding coefficients can be used To perform the HDR brightness codeword value Single channel prediction of SDR brightness codeword value as follows:

[0092]

[0093] Given P pixels in each of the SDR and HDR images, all P to-be-predicted (or target) SDR luma codeword values, corresponding coefficients and B smooth curve basis function Collected into matrix form to get the corresponding (input) HDR brightness codeword value Predicted (target) SDR brightness codeword value as follows:

[0094]

[0095] in

[0096]

[0097] Wherein the left hand side (LHS) in expression (8) represents the design matrix.

[0098] The ground truth vector including all P actual (eg, target, reference, etc.) SDR codeword values ​​is expressed as follows:

[0099]

[0100] The coefficients can be obtained by least squares solution in closed form The solution is as follows:

[0101]

[0102] To solve this least squares problem, special care can be trained to (for example) consider the B smooth curve matrix Relatively sparse scheme. In an operating scheme where the nodes can be preselected or fixed regardless of the type of signal or pixel data distribution, there may be empty pixels or no pixels in the knot intervals between adjacent knots. Because the B-smooth curve basis function is only positive or supported on zero in a relatively small interval and outside the interval, the empty intervals in which there may be no pixels can contribute to Some or all zero columns in (each of which includes all zeros). Such zero columns will make the matrix The calculation of is ambiguous or encounters singularities. In some operating schemes, to solve or improve this problem, Columns in the table that have all zeros or all relatively small values ​​(e.g., compared to a numerical threshold set by a program, by experience, or by a user, etc.), the corresponding coefficients Set to zero.

[0103] Will Each element in is represented by And the αth column is represented as An example process for reconstructing a design matrix excluding columns of all zero values ​​or all relatively small values ​​(eg, the sum of columns below a numerical threshold, every matrix element in a column below a numerical threshold, etc.) is illustrated in Table 2 below.

[0104] Table 2

[0105]

[0106]

[0107] In some operating schemes, instead of using the above expression (10), the above expression obtained in Table 2 may be used. To solve the least squares problem:

[0108]

[0109] Because the total number of columns Dy has been reduced to c (where c represents the new total number of columns other than the (Dy-c) columns each of which includes all zeros or a relatively small number of columns), the solution to expression (11) may lack coefficients or parameters corresponding to the excluded column(s). Using the example process illustrated in Table 3 below, a parameter / coefficient vector including parameters / coefficients for all columns may be constructed by filling the parameters / coefficients corresponding to the removed (Dy-c) columns with 0s (zeros) as follows:

[0110]

[0111] Table 3

[0112]

[0113] In some operational schemes, the positions of the knots or nodes are preselected or fixed. However, it should be noted that in various embodiments, the positions (and / or the total number) of the knots or nodes may or may not be preselected or fixed. For example, in some operational schemes, the positions (and / or the total number) of the knots or nodes {k} may be preselected or fixed because the positions (and / or the total number) of the knots or nodes may affect the optimality of the solution. t} and parameters / coefficients Adaptability is determined as part of the overall minimization (or optimization) problem or solution.

[0114] Although adaptively selecting the knots can further improve the performance and accuracy of mapping or transforming between or among different color levels of the video content, preset or preselected uniformly distributed knots bring exemplary advantages, including (but not limited to): (1) the knot positions of the B smooth curve basis functions in the image metadata do not need to be signaled or sent by the upstream video content processor to the downstream video content processor, which reduces the bitstream burden for carrying and encoding the image metadata; (2) the basis functions at the decoder side do not need to be recalculated in response to different knots that can otherwise be adaptively changed on a frame-by-frame basis; etc. In other words, the B smooth curve basis functions with preset or fixed knots can be hard-wired in logic or stored in data memory at the decoder side to reduce runtime computational load and / or hardware complexity.

[0115] TPB Cross-Color Channel Predictor

[0116] In some operating scenarios, there may be relatively large limitations in using a single channel predictor (e.g., a single channel (or 1D) B smooth curve predictor) to generate a mapping between HDR and SDR (or vice versa). A 1D B smooth curve predictor may provide prediction performance and accuracy comparable to other methods (e.g., cumulative density function or CDF-based methods, minimum mean square error or MMSE-based methods, etc.), possibly with additional computational complexity. The mapping problem of different color levels of video content exists in multi-dimensional color spaces (e.g., 3D color spaces (e.g., RGB, IPT, YDzDx, and YCbCr), three or more channels in a color space, etc.), and 1D functions or predictions may be limited in some operating scenarios. For example, a relatively accurate mapping (back and forth) between HDR and SDR may involve cross-color operations (e.g., cross-color operations similar to those manually performed by color grading experts, etc.), such as color transformations and saturation control. Therefore, using a 1-D predictor to solve this mapping problem in a multi-dimensional color space involving human perception of color, hue, and saturation at different brightness levels may not be optimal.

[0117] To improve prediction performance and accuracy, TPB prediction that is capable of exploiting local and / or global cross-color channel relationships between different color levels of video content may be employed according to the techniques described herein.

[0118] Compared to some approaches in which cross-channel prediction represents a global mapping operator, TPB cross-channel prediction can flexibly model mapping in each local partition in the color space or color gamut (e.g., applying region-specific mapping consistently to each of many local single-dimensional or multi-dimensional codeword regions instead of applying a global mapping to all codewords or colors, etc.) to outperform other approaches that do not implement the techniques described herein.

[0119] TPB cross-channel prediction can be applied to predict codewords in the luma or luminance channel from HDR to SDR (or vice versa). In the previous example of color space Y, C0 (or Cb), and C1 (or Cr), a full set of B smooth curve basis functions is provided independently in each color channel or dimension. Thus, there are three sets of B smooth curve basis functions for the Y, C0, and C1 color channels or dimensions, respectively. and Here, j represents the jth SDR and HDR image, t y The B-smooth curve basis function exponents representing the brightness or luminance Y dimension (given a set of knots along the Y dimension); t c0 represents the B-smooth curve basis function index for the chroma or hue C0 dimension (given a set of knots along the C0 dimension); t c1 represents the B-smooth curve basis function index of the chroma or chroma C1 dimension (given a set of nodes along the C1 dimension); i represents the pixel index of the P pixels in each of the SDR and HDR images.

[0120] B smooth curve basis function and The total number of B smooth curve basis functions in these three sets is expressed as and Given the corresponding set of nodes in each of the three dimensions, the above expression (3) can be used to construct the B-smooth curve basis function and Each of the three complete series.

[0121] The B smoothing curve basis functions can be obtained by obtaining all three color channels or dimensions and The TPB basis functions for predicting the codewords in the luminance and luma channels are constructed by multiplying the vectors of the three full sets of t. y ,t c0 and t c1 The TPB basis function of the (3D) TPB index of a unique combination can be given or calculated as follows:

[0122]

[0123] A set of corresponding coefficients can be used To perform the HDR brightness and chrominance codeword values and Cross-channel prediction of SDR brightness codeword values as follows:

[0124]

[0125] The 3D TPB index (t y ,t c0 and t c1 ) is vectorized into a 1D index (denoted as t) to simplify the expression here.

[0126] Previously with 3D TPB index (t y ,t c0 and t c1 ) can be rewritten as follows:

[0127]

[0128] make The TPB cross-channel prediction in the above expression (14) can be rewritten as follows:

[0129]

[0130] Given P pixels in each of the SDR and HDR images, all P to-be-predicted (or target) SDR luma codeword values, corresponding coefficients and cross channel TPB function Collected into matrix form to correspond to (input) HDR brightness and chrominance codeword values Predicted (target) SDR brightness codeword value as follows:

[0131]

[0132] in

[0133]

[0134]

[0135] Wherein the LHS of expression (18) represents the design matrix.

[0136] The coefficients can be obtained by least squares solution in closed form The solution is as follows:

[0137]

[0138] in represents the ground truth vector in the above expression (9).

[0139] To facilitate discussion, expression (20) can be rewritten using matrices and vectors as follows:

[0140]

[0141] therefore,

[0142]

[0143] Similarly, a TPB cross-channel predictor can be constructed for two chroma channels or dimensions. By way of illustration, the chroma channels C0 and C1 are denoted as C. Given three sets of known points in the Y, C0, and C1 dimensions, three full sets of B-smooth curve basis functions can be constructed for the Y, C0, and C1 dimensions: and TPB prediction for codewords in chroma channel C (where C can be C0 or C1). The total number of three complete sets of B smoothing curve basis functions can be respectively and Given a set of nodes in 3 dimensions, we have 3 individual basis function arrays and

[0144] Similar to the TPB cross-channel prediction of the codeword in the Y channel, for the chroma channel C, the exponential tensor elements can be used to construct the TPB cross-channel basis function as follows:

[0145]

[0146] Similarly, the 3D TPB index (t y ,t c0 and t c1 ) is vectorized into a 1D index (denoted as t) to simplify the expression here. The TPB cross-channel prediction of the codeword in the chroma channel C can be given as follows:

[0147]

[0148] Given P pixels in each of the SDR and HDR images, all P to-be-predicted (or target) SDR chroma codeword values, corresponding coefficients and cross channel TPB function Collected into matrix form to correspond to (input) HDR brightness and chrominance codeword values Predicted (target) SDR brightness codeword value as follows:

[0149]

[0150] in

[0151]

[0152] Wherein the LHS of expression (27) represents a TPB-based matrix.

[0153] The coefficients can be obtained by least squares solution in closed form The solution is as follows:

[0154]

[0155] To facilitate discussion, expression (29) can be rewritten using matrices and vectors as follows:

[0156]

[0157] therefore,

[0158]

[0159] In some operational schemes, the positions of the knots or nodes are preselected or fixed. However, it should be noted that in various embodiments, the positions (and / or the total number) of the knots or nodes may or may not be preselected or fixed. For example, in some operational schemes, the positions (and / or the total number) of the knots or nodes {k t} position (and / or total number) and parameters / coefficients Adaptability is determined as part of the overall minimization (or optimization) problem or TPB cross-channel prediction solution.

[0160] Although adaptive selection of nodes can further improve the performance and accuracy of mapping or transformation between or among different color levels of video content, preset or preselected nodes for uniform distribution in the multi-dimensional TPB tensor bring exemplary advantages, including (but not limited to): (1) no need to signal or send the node positions in the multi-dimensional TPB basis functions used in the image metadata by the upstream video content processor to the downstream video content processor, which reduces the bitstream burden for carrying and encoding the image metadata; (2) no need to recalculate the B-smooth curve or TPB basis function at the decoder side in response to different nodes that can otherwise be adaptively changed on a frame-by-frame basis; etc. In other words, TPB basis functions with preset or fixed nodes can be hard-wired in logic or stored in data memory at the decoder side to reduce runtime computational load and / or hardware complexity.

[0161] In an operating scheme where all three channels or dimensions each have the same TPB basis function, the S matrices for different channels are the same as follows:

[0162]

[0163] At the decoder side, the same calculation is performed Matrix, there is no need to calculate different S matrices for different channels or dimensions. Then, the cross-channel prediction codeword value for each channel can be obtained by multiplying the same S matrix with the corresponding prediction parameters / coefficients as follows:

[0164]

[0165] In another example, the two chroma channels use the same S matrix, while a different S matrix is ​​used for the luma channel. For example, the S matrix for the luma channel has a larger dimension than the S matrix for the chroma channels. In this case, the number of predictor coefficients for the luma channel will be larger than the number of predictor coefficients for each of the luma channels.

[0166] Efficient decoder architecture

[0167] When the knots or knots containing external knots are uniformly distributed, the B-smooth curve basis function can be represented by a linear combination of truncated polynomials. The n-th order truncated polynomial can be defined as follows:

[0168]

[0169] Represent the nodes as channels Y, C0, and C1 and When the nodes are evenly distributed and When the distance between each two consecutive (or adjacent) nodes can be expressed as h y 、h c0 and h c1 For simplicity, the nodes of each channel can be represented as {k t} and h.

[0170] The first order of the t-order B-smooth curve basis function can be constructed using a linear combination of truncated polynomials as follows:

[0171]

[0172] in Only positive or supported on [k t-1 k t+1] or on a 2h interval and zero (0) outside this range otherwise. As shown in expression (36), the 1st-order B-smooth curve basis function includes three terms, each of which includes a corresponding node multiplication factor (denoted as c) multiplied by the difference between an input value (e.g., an input or source codeword for predicting a target codeword that approximately represents a reference codeword) and a corresponding node. i , where i represents an integer between 0 and 2). For example, the first multiplication factor of the first term in expression (36) is The second multiplication factor of the second term in expression (36) is The third multiplication factor of the third term in expression (36) is

[0173] The 2nd order of the t-order B-smooth curve basis function can be constructed using a linear combination of truncated polynomials as follows:

[0174]

[0175] in Only positive or supported on [k t-1 k t+2 ] or on a 3h interval and otherwise zero (0) outside this range. As shown in expression (37), the 2nd order B smooth curve basis function includes four terms, each of which includes a corresponding node multiplication factor (denoted as c) multiplied by the difference between an input value (e.g., an input or source codeword for predicting a target codeword that approximately represents a reference codeword) and a corresponding node. i , where i represents an integer between 0 and 3). For example, the first multiplication factor of the first term in expression (37) is The second multiplication factor of the second term in expression (37) is The third multiplication factor of the third term in expression (37) is The fourth multiplication factor of the fourth term in expression (37) is

[0176] The 3rd order of the t-order B-smooth curve basis function can be constructed using a linear combination of truncated polynomials as follows:

[0177]

[0178] in Only positive or supported on [k t-2 k t+2] or on a 4h interval and otherwise zero (0) outside this range. As shown in expression (38), the 3rd order B smooth curve basis function includes five terms, each of which includes a corresponding node multiplication factor (denoted as c) multiplied by the difference between an input value (e.g., an input or source codeword for predicting a target codeword that approximately represents a reference codeword) and a corresponding node. i , where i represents an integer between 0 and 4). For example, the first multiplication factor of the first term in expression (38) is The second multiplication factor of the second term in expression (38) is The third multiplication factor of the third term in expression (38) is The fourth multiplication factor of the fourth term in expression (38) is The fifth multiplication factor of the fifth term in expression (38) is

[0179] These polynomial expressions with B-smooth curve basis functions allow video content processors such as encoders, decoders, codecs, etc. to use polynomials of B-smooth curve basis functions and avoid recursively obtaining B-smooth curve basis functions. This simplifies hardware implementation because implementing recursive B-smooth curve expressions can be relatively complex. Recursive expressions can also take a relatively long time and relatively large memory space to calculate and store. Truncated polynomial expressions can be used to save computation by trading off the flexibility of having uneven (e.g., adaptive, etc.) nodes.

[0180] Additionally, optionally or alternatively, the nth order B smooth curve basis function is positive due to the crossing of (n+1)h intervals. Given a point x, only (n+1) basis functions include support or possible positive values ​​within the range covering point x. Therefore, only these (n+1) basis functions are activated for the given point x. The remaining basis functions are zero (0) at the given point x because the given point x is outside the range in which the remaining basis functions are non-zero. To construct TPB basis functions for three color channels or dimensions, instead of activating all B-smooth curve basis functions, only the (n+1) basis functions are activated. 3 B smooth curve basis functions. This is from the (original) total number of B smooth curve basis functions Significantly reduces potentially many multiplications.

[0181] Decoder Implementation

[0182] Figure 1B Example processing blocks for TPB cross-channel prediction are described, which may be implemented in a video content processor (eg, decoder, encoder, codec, etc.) having one or more computer processors. Some or all of these processing blocks may be implemented in hardware, software, or a combination of hardware and software.

[0183] Block 192 includes a TPB basis generation operation that prepares the B smooth curve basis functions in each channel or dimension. Output D from each basis function in the Y channel or dimension y Different values, output D from each basis function in the Cb channel or dimension Cb Different values, output D from each basis function in the Cr channel or dimension Cr different values, etc. Since there are only (n+1) non-zero values ​​in each channel or dimension, if we only output non-zero signals or values, then there are only 3*(n+1) non-zero signals or values ​​from 3 channels or dimensions. As long as the nodes of the B-smooth curve basis functions are preconfigured, fixed, or otherwise available, the operations in this block can be preconfigured, fixed, and / or pre-performed without waiting to receive video content or image metadata.

[0184] Block 194 includes a TPB cross product operation that performs a cross product multiplication with a 1-dimensional B-smooth curve basis function to construct a higher dimensional TPB basis function as follows:

[0185] Brightness:

[0186] Chroma:

[0187] Since there are (n+1) non-zero values ​​in each channel or dimension, there are (n+1) 3 multiplications and (n+1) 3 The operations in this block can be fixed, but it is necessary to track the index used for TPB prediction.

[0188] Block 196 comprises a TPB coefficient multiplication operation which multiplies the non-zero value output from block 194 with the prediction parameters / coefficients read from the image metadata (e.g., combiner metadata) transmitted with the video content in the received video signal. Multiply. There are (n+1) for weighting each high-dimensional basis function 3 multiplications and an addition to sum all the values ​​to produce the final predicted value and as follows:

[0189] Brightness:

[0190] Chroma:

[0191] Where C represents C0 or C1.

[0192] The operation in block 196 may dynamically identify and / or assign base indices and parameter indices.

[0193] Figure 1Cand Figure 1D illustrate Figure 1B An exemplary implementation of the TPB base generation operation in block 192 of .

[0194] Figure 1C An example equation-based implementation illustrating the generation of a B-smooth curve basis function output, which can then be used to generate a tensor product representing the TPB basis function output. In some operational schemes, Figure 1C The equation-based implementation of can be based on any of the expressions (36) to (38) of the 1st to 3rd order B smooth curve basis functions. In some operational schemes, Figure 1C An equation-based implementation scheme may be based on truncated polynomial expressions similar to Expressions (36) to (38) of B-smooth curve basis functions of orders other than 1st to 3rd orders.

[0195] like Figure 1C As described in FIG. 1 , an input x (e.g., an input or source codeword based at least in part on which a target codeword approximately represents a reference codeword in a channel or dimension of a color space is predicted) and B smoothing curve basis function parameters (e.g., fixed, etc.) may be used to predict a target codeword approximately representing a reference codeword in a channel or dimension of a color space. t，i And the multiplication parameter c i ) is used to calculate the output value of the t-th order B smooth curve basis function (which can be used to form the TPB basis function having the above expression (24)). Then, the output value of the corresponding TPB basis function can be generated as the tensor product of the output value of the B smooth curve basis function.

[0196] In some operating scenarios, Figure 1C The equation-based implementation scheme of is instantiated into relatively many instances or threads running in parallel. In some operational schemes, Figure 1C The equation-based implementation of can be reused in (for example) a single thread or relatively few threads by loading different basis function parameters when the hardware (such as processing logic or processor implementing this equation-based method, etc.) operates at a relatively high frequency.

[0197] like Figure 1D As illustrated in , the output of each B smooth curve basis function can be stored in a local register. As mentioned before, there are only (n+1) non-zero outputs from a corresponding number of B smooth curve basis functions. Therefore, for a given input value x, only (n+1) basis function outputs need to be completed.

[0198] If the number of internal nodes is a power of 2 plus 1, the most significant bit or MSB can be used to quickly index the (n+1) non-zero outputs. Next, two (e.g., consecutive, adjacent, etc.) internal nodes {k t,i The interval between} is only a power of 2. Only the MSB bit can be used to identify the specific output and number of non-zero outputs.

[0199] Additionally, optionally, or alternatively, in addition to or in lieu of an equation-based implementation, the output of the B smooth curve basis function may be based on a lookup table (LUT). For example, in some operational implementations, a 1D-LUT may be constructed for each 1-DB smooth curve basis function. For example, several LUT table values ​​(e.g., a total of 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,41,47,48,49,51,40,41,42 If the cache or memory space is relatively large enough, all D y It should also be noted that, using uniformly distributed knots, the B-smooth curve basis function for each channel or dimension is simply a shifted version of one of the B-smooth curve basis functions, such as Figures 3A to 3D . This property can be used to achieve relatively efficient solution in both hardware and software design and memory space and operation. The output of the B smooth curve function can be generated by applying the shift calculation to the input based on multiple offsets. Therefore, the 1D-LUT only needs to cover (n+1)h intervals instead of the entire input value (such as signal, codeword, etc.) range to greatly reduce the number of required LUT table values. The only additional logic to be added or implemented may be the shift operation of the offset. If the hardware or processor can operate at a relatively high frequency, the outputs of different B smooth curve basis functions generated in the same channel or dimension can share (for example) the same processing logic with different offsets.

[0200] Figure 1E and Figure 1F illustrate Figure 1B An example implementation of the TPB cross-product operation in block 194 of .

[0201] Figure 1E An example TPB cross product operation in parallel mode is described. Figure 1E As illustrated in FIG. 1 , for each channel or dimension, (n+1) outputs are generated from the B smooth curve basis functions for each channel. For each channel or dimension, the TPB cross product operation in block 194 performs (n+1) on the corresponding (n+1) outputs. 3 multiplications and store the multiplication (or tensor product) result as a TPB cross-product output (such as shown in expression (24) etc.) in, for example, an output register. The TPB cross-product output is then provided to block 196 to be compared using expression (34) with, for example, prediction coefficients (such as those generated by an encoder, generated by an upstream video content processor, etc.) from image metadata received in or decoded from the video signal. etc.) are multiplied.

[0202] Figure 1F An exemplary TPB cross product operation in serial or sequential mode is described. Figure 1F As described in FIG. 1 , if the TPB cross product operation can be completed at a relatively high speed using relatively high frequency processing logic, the TPB cross product operation in block 194 and the TPB coefficient multiplication operation in block 196 can be combined together. Depending on the hardware capability and occupied area, (n+1) 3 The terms are divided into N groups, where N is a positive integer; each of the N groups has its own multiplier to be repeated (n+1) in a "for" loop, for example 3 / N times to complete the TPB cross product operation and the corresponding TPB coefficient multiplication. N represents the hardware design parameters used to achieve a specific design performance target (which includes (but is not limited to) speed, die size, clock frequency, etc.).

[0203] Figure 1G and Figure 1H An example flow chart illustrating application of TPB prediction to input or source color levels 158 of video content to generate target or mapped color levels that include TPB predicted (e.g., target, mapped, etc.) codewords that approximate reference codewords in reference code levels 160 of the video content. A method similar to Figure 1G and Figure 1H The present invention is a flowchart of a flowchart of a method for generating target or mapped color levels predicted from input or source color levels (158) that approximate reference code levels (160) representing video content. Some or all of these flowcharts may be implemented in software, hardware, a combination of software and hardware, etc. and may be executed by one or more computer processors.

[0204] By way of illustration only, the color levels (158) and (160) may be encoded in a sub-sampled format (e.g., 4:2:0 in a YCbCr color space). In a 4:2:0 sub-sampled format, chroma or chroma codewords (Cb and Cr) are 1 / 4 the size of luma or luminance codewords. To perform cross-color channel prediction (e.g., TPB cross-channel prediction), the sizes (or dimensions) of the codewords for all input or source color channels may be aligned (e.g., optionally up-sampled, optionally down-sampled, etc.).

[0205] like Figure 1G As illustrated in FIG. 1 , for TPB luma prediction, chroma or chroma codewords (Cb / Cr) of a source color level (158) may be upsampled by a processing block 164. The upsampled chroma codewords are output by the processing block (164) to a TPB luma prediction block 166. The upsampled chroma codewords in a combination of luma or luminance codewords (Y) of a source code level (158) of the same size (or dimension) may be used by the TPB luma prediction block (166) to predict a mapped or reshaped luma or luminance codeword that closely represents a luma or luminance codeword in a reference color level (160).

[0206] like Figure 1G As illustrated in FIG. 1 , for TPB chroma Cb prediction, the luma codeword (Y) of the source color level (158) may be downsampled by the processing block 162. The downsampled luma codeword is output by the processing block (162) to the TPB chroma Cb prediction block 168. The downsampled luma codeword in a combination of chroma codewords of the source code level (158) of the same size (or dimension) may be used by the TPB chroma Cb prediction block (168) to predict a mapped or reshaped chroma Cb codeword that closely represents the chroma Cb codeword in the reference color level (160).

[0207] like Figure 1G As illustrated in FIG. 1 , for TPB chroma Cr prediction, the downsampled luma codeword is output by the processing block (162) to the TPB chroma Cr prediction block 170. The downsampled luma codeword in the combination of chroma codewords of the source code level (158) of the same size (or dimension) can be used by the TPB chroma Cr prediction block (170) to predict a mapped or reshaped chroma Cr codeword that approximately represents the chroma Cr codeword in the reference color level (160).

[0208] In some operating scenarios, such as Figure 1H As illustrated in FIG, only the downsampled luma codeword derived from downsampling the luma codeword of the source color level (158) (e.g., using processing blocks 162-1 and 162-2, etc.) is used by the overall TPB prediction block 172 to obtain TPB predictions and TPB prediction parameters or coefficients for all channels. This significantly reduces memory consumption because only luma downsampling is used.

[0209] In some operational schemes, on the encoder side, TPB prediction parameters or coefficients may be generated by the TPB prediction blocks (e.g., 166, 168, 170, 172, etc.) as a solution that minimizes the difference between the luma / chroma codewords that are mapped or reshaped to align (e.g., optionally upsampled, optionally downsampled, etc.) the luma / chroma codewords with the reference color levels (160). The TPB prediction parameters or coefficients may be encoded in the video signal to a downstream video content processor as part of the image metadata (e.g., combiner metadata).

[0210] In some operational schemes, on the decoder side, the TPB prediction parameters or coefficients may be decoded by a downstream receiving device from a video signal as part of the image metadata. A decoded version of the source color levels (158) may also be decoded by the device from the video signal. The TPB prediction parameters or coefficients may be used by the device to generate a mapped or reshaped image representing a reconstructed color level that closely represents the reference color level (160). The mapped or reshaped image may be reproduced on a display device as a color level that is different from the decoded version of the source color level (158).

[0211] Additionally, optionally or alternatively, in some implementations, the total number of nodes and the order of the B-smoothing curves for both the lightness and chroma channels are the same, which can even further reduce computation. Figure 1B In the illustrated three-stage implementation, the two stages (i.e., TPB basis generation and TPB cross product) are identical for all three channels or dimensions. Thus, the only difference between channels is the different TPB prediction parameters / coefficients used for TPB coefficient multiplication in a single stage. It should be noted that in some other embodiments or implementations, the number of knots for different color channels may be different to provide additional flexibility or tradeoffs in achieving performance and / or accuracy.

[0212] TPB prediction based on 3DMT

[0213] In some operating schemes, TPB prediction may be performed using a codeword histogram or distribution generated based on a 3-D mapping table (3DMT) technique.

[0214] make Become a first 3D array comprising the first luma and chroma codewords of the i-th pixel of a first image (eg, input or source image, etc.) (eg, HDR image) at a first color level of the video content. becomes a second 3D array comprising second luma and chroma codewords for the i-th pixel of a second image (which corresponds to the first image or depicts the same visual content as the first image, such as a reference image, etc.) from a second color level of the video content (e.g., an SDR image).

[0215] The three channel luma and chroma codeword values ​​(Y, C0, and C1) for representing the image content of the first color level may be quantized or counted into a fixed number of 1D bins for each channel or component (e.g., a first fixed number of luma bins Q y , the second fixed number Q of chroma Cb bins C0 , the third fixed number Q of the chroma Cb bin C1 etc.). A 1D bin with a fixed number of The 3D histogram of the 3D bin is denoted as Ω Q，v (in In some operating schemes, the first three-channel luma and chroma codeword space, which includes all possible luma and chroma codeword values ​​for encoding the first color level, can be evenly divided into a fixed number of 3D bins, where each channel is evenly divided into a fixed number of 1D bins (e.g., the first fixed number Q y brightness bins, the second fixed number Q C0 chroma Cb bins, the third fixed number Q C1 chroma Cb bin, etc.).

[0216] Therefore, the 3D histogram Ω Q,v Contains a total of bins, so that each 3D bin can be identified by the corresponding bin index Specify; a bin represents or holds a count of the number of pixels in the first image (of a first color level) having three-channel quantized values ​​that fall within the boundaries of the 3D bin.

[0217] In addition, the sum of each color component in the second image (a second image of a second color level, such as a reference image approximately represented by a mapped image of the first image, etc.) may be maintained in the 3D histogram Ω Q,v In each 3D bin or for the 3D histogram Ω Q,v Each 3D bin of and are the sums of the (reference) luminance and chrominance codeword values ​​in the second image domain, respectively, so that each 3D bin contains the sum of the luminance and chrominance (C0 and C1) codeword values ​​of a pixel in the second image, where the pixel in the second image corresponds to the pixel of the first image whose count is stored in the same 3D bin.

[0218] Assume that each of the first image and the second image has P pixels. An example process for generating a 3D bin having a count of pixels of the first image at a first color level and a sum of codeword values ​​of pixels of the second image at a second color level (corresponding to pixels of the first image) is illustrated in Table 4 below.

[0219] Table 4

[0220]

[0221]

[0222] make Represents a 3D histogram Ω Q,v The center of the qth SDR bin of . These center values ​​for all images of the first color level are fixed and can be pre-computed. The corresponding reference or target HDR values ​​that approximately represent their center values ​​can be obtained using the example process illustrated in Table 5 below.

[0223] Table 5

[0224]

[0225] In some operating schemes, in the 3D histogram Ω Q,v , 3D bins each having a non-zero pixel count for pixels in the first image at the first color level are identified and retained, while all other 3D bins each having a zero pixel count (or a relatively small pixel count below a given pixel count threshold) for pixels in the first image at the first color level are discarded.k-1 There are k such bins, where The example process described in Table 6 below can be used to calculate and The average value of .

[0226] Table 6

[0227]

[0228] For a given valid bin index (e.g., a bin index with a non-zero pixel count, etc.), a mapping pair comprising to-be-mapped luma and chroma codeword values ​​of a first image at a first color level (as represented by the center value of the 3D bin with the bin index) and reference luma and chroma codeword values ​​of a second image at a second color level approximately represented by the mapped image (as represented by the average of the sum of the luma and chroma codeword values ​​of the corresponding pixels) may be obtained as follows:

[0229]

[0230] Based on the 3DMT (or the 3D bin generated therefrom), the cross-channel prediction mapping value TPB is obtained from the center value of the 3D bin represented in expression (41-1) and To approximate the reference value represented in expression (41-2) can be performed as follows:

[0231] Brightness:

[0232] Chroma:

[0233] The TPB predictor parameters or coefficients may be obtained via a process similar to that in the individual pixel based solution The design matrix and target vector may be constructed by taking input values ​​from a mapping table that includes all mapping pairs of all valid 3D bins (eg, Expression 41 above).

[0234] For TPB brightness prediction, the design matrix and target vector can be constructed as follows:

[0235]

[0236] For TPB chroma prediction, the design matrix and target vector can be constructed as follows:

[0237]

[0238] The solution of TPB prediction parameters or coefficients can be obtained through least squares solution as follows:

[0239]

[0240] The 3DMT based TPB prediction technique can be used to provide relatively fast computation speed. Instead of constructing the B and a matrices from all P individual pixels of each image pair, k table values ​​from a mapping table including 3D bin based mapping pairs can be used. In some operating schemes, k can be kept or constrained or selected to be in the range of thousands, much smaller than P which can be in the range of millions or even larger. The order of magnitude of computational savings can be 3.

[0241] Additionally, optionally or alternatively, 3DMT-based TPB prediction techniques may be used to mitigate or prevent majority and minority problems, which may promote or overweight relatively large image regions / areas in an image at the expense of relatively small image regions / areas in the same image. Having relatively fair weights in each color block represented by a 3D bin may help reduce color artifacts and improve color accuracy.

[0242] Scenario / segment / linear based TPB prediction

[0243] The TPB prediction described herein may be performed using scene-based, segment-based, and / or linear-based coding architectures.

[0244] Assuming there are F images / frames in a scene depicted in the video content of the video signal, using a scene-based architecture, all B and a from all frames within this scene can be summed up as follows:

[0245]

[0246] It should be noted that in various embodiments, B and a in expressions (49) or (50) may be constructed from pixel-based or 3DMT-based mapping data between different color levels of the video content.

[0247] When the B matrix may be in an ambiguous condition where rows (and columns) include all zeros (or relatively small values ​​that are individually or collectively below a threshold), these columns (and rows) may be removed from the B matrix, for example, using the example process illustrated in Table 7 below.

[0248] Table 7

[0249]

[0250]

[0251] Similarly, when matrix a may be in an ambiguous condition where columns correspond to columns (and rows) identified and excluded from matrix B, these elements in matrix a may be removed. An example process for identifying these table values ​​in matrix a is illustrated in Table 8 below.

[0252] Table 8

[0253]

[0254] The solution of scenario-based TPB prediction can be obtained as follows:

[0255]

[0256] An example process for generating all TPB prediction parameters or coefficients, including those corresponding to columns / rows identified and excluded from the B matrix, is illustrated in Table 9 below.

[0257] Table 9

[0258]

[0259]

[0260] In some embodiments, this scene-based TPB prediction method may be applied to a population of training image pairs to determine or derive a static TPB map.

[0261] Additionally, optionally or alternatively, segmentation-based and / or linear-based coding architectures may be used to perform TPB prediction. For example, in some operating schemes, a sliding window approach is used in a segmentation / linear-based architecture, where the techniques are similar to those used in the scene-based TPB prediction techniques described above. Scene-based TPB prediction techniques or their analogs may be applied to sliding windows by simply treating a "scene" as a sliding window (or vice versa).

[0262] Image metadata encoding / decoding syntax and semantics

[0263] Various syntaxes and semantics may be used to encode and / or decode image metadata including TPB prediction parameters or coefficients. Example syntax and semantics for encoding / decoding image metadata including TPB parameters or coefficients are illustrated in Table 10 below.

[0264] Table 10

[0265]

[0266]

[0267] In Table 10, "x" and "y" represent the two-dimensional indices of the image block(s) into which the image / frame is partitioned, and "cmp" represents the number of color space components or channels associated with the TPB parameters.

[0268] Some loop variables used in the encoding syntax in Table 10 are defined as follows:

[0269] tpb_num_basis[y][x][cmp][0]=(tpb_num_knot_minus1[y][x][cmp][0]+tpb_order_minus1[y][x][cmp][0]+1)

[0270] tpb_num_basis[y][x][cmp][1]=(tpb_num_knot_minus1[y][x][cmp][1]+tpb_order_minus1[y][x][cmp][1]+1)

[0271] tpb_num_basis[y][x][cmp][2]=(tpb_num_knot_minus1[y][x][cmp][2]+tpb_order_minus1[y][x][cmp][2]+1)

[0272] The semantics of some elements encoded / decoded using the coding syntax of Table 10 (e.g., represented as Exponential Golomb codes in differential coding, etc.) are defined as follows:

[0273] -tpb_num_knot_minus1[y][x][cmp][k] specifies the number of knots in the kth channel minus 1

[0274] -tpb_order_minus1[y][x][cmp][k] specifies TPB order minus 1

[0275] -tpb_zero_coef[y][x][cmp][i][j][k] specifies whether the coefficient is zero.

[0276] - When coefficient_data_type is equal to 0, tpb_int[y][x][cmp][i][j][k] specifies the integer part of fp_tpb_coef[y][x][cmp][i][j][k]. If coefficient_data_type is equal to 1, then tpb_int[y][x][cmp][i][j][k] does not exist. (For example, when coefficient_data_type is equal to 0, fp_tpb_coef[y][x][cmp][i][j][k] is used to derive the weighting coefficients (e.g., m) of the corresponding TPB basis function i associated with mapping_idc[y][x][cmp]. TPB，y，(opt) 、m TPB，C0，(opt) 、m TPB，C1，(opt) wait).

[0277] - When coefficient_data_type is equal to 0, tpb_coef[y][x][cmp][i] specifies the fractional part of fp_tpb_coef[y][x][cmp][i][j][k]. If coefficient_data_type is equal to 1, then tpb_coef[y][x][cmp][i][j][k] is used to derive the value of the weighted coefficient or gain coefficient associated with mapping_idc[y][x][cmp]. If coefficient_data_type is equal to 0, then the length of the tpb_coef[y][x][cmp][i][j][k] syntax element is coefficient_log2_denom bits. If coefficient_data_type is equal to 1, then the length of the tpb_coef[y][x][cmp][i][j][k] syntax element is 32 bits. The value of the weighted coefficient or gain coefficient in the normalization associated with mapping_idc[y][x][cmp] is derived as follows:

[0278] ο If coefficient_data_type is equal to 0, then the value of the weighted coefficient or gain coefficient is equal to fp_tpb_coef[y][x][cmp][i][j][k] = (tpb_int[y][x][cmp][i][j][k] << coefficient_log2_denom) + tpb_coef[y][x][cmp][i][j][k], where "<<" represents a shift operation.

[0279] ο If coefficient_data_type is equal to 1, then the value of the weighted coefficient or gain coefficient is equal to tpb_coef[y][x][cmp][i][j][k].

[0280] Exemplary process flow

[0281] Figure 4A Describe an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (such as an encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) may execute this process flow. In block 402, the image processing system determines a set of B-spline curve (TPB) basis functions of tensor products.

[0282] In block 404, the image processing system generates a set of selected TPB prediction parameters for use with the TPB basis function array to generate predicted image data in one or more mapped images from source image data in one or more source images at a source color level. The selected TPB prediction parameter array is generated by minimizing the difference between the predicted image data in the one or more mapped images and the reference image data in one or more reference images at a reference color level. The one or more reference images correspond to the one or more source images and depict the same visual content as depicted by the one or more source images.

[0283] In block 406 , the image processing system encodes the selected TPB prediction parameter array in a video signal as part of the image metadata along with source image data in one or more source images.

[0284] In block 408, the image processing system causes one or more mapped images to be reconstructed and reproduced by a receiving device of the video signal.

[0285] In an embodiment, at least one of the source image data or the reference image data is represented in a sub-sampled format of a color space.

[0286] In an embodiment, the one or more source images represent one of: images constituting a visual scene in a media program, images selected within a sliding window, images selected within a linear segment, and the like.

[0287] In an embodiment, the TPB basis function array is generated by the tensor product of one or more sets of B smooth curve basis functions; each set of the one or more sets of B smooth curve basis functions corresponds to a corresponding color channel in one or more color channels of the color space.

[0288] In an embodiment, at least one set of B-smooth curve basis functions among the one or more sets of B-smooth curve basis functions represents a full set of B-smooth curve basis functions of a particular order.

[0289] In an embodiment, the one or more sets of B-smooth curve basis functions include a set of B-smooth curve basis functions generated by a truncated polynomial and a set of uniformly distributed nodes.

[0290] In an embodiment, the combination of the selected TPB prediction parameter array and the TPB basis function array represents a cross-channel predictor for generating predicted picture data in one or more mapped pictures.

[0291] In an embodiment, a plurality of mapping pairs are used to generate TPB prediction parameter arrays, each of the plurality of mapping pairs comprising a first array of one or more source codewords generated from source image data and a second array of one or more reference codewords generated from reference image data.

[0292] In an embodiment, a plurality of mapping pairs are generated based on a three-dimensional mapping table (3DMT).

[0293] Figure 4B An example process flow according to an embodiment of the invention is described. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, prediction module and feature selection system, inverse mapping generation and application system, etc.) may perform this process flow. In block 452, a video decoding system decodes one or more first images at a first color level from a video signal.

[0294] In block 454, the video decoding system decodes from the video signal image metadata including a set of selected tensor product B smooth curve (TPB) prediction parameters for multiplication with a set of TPB basis functions.

[0295] The selected TPB prediction parameter array is generated by an upstream video content processor, wherein the selected TPB prediction parameter array is used together with the TPB basis function array to generate prediction image data in one or more mapped images from first image data in one or more first images at a first color level. The upstream video content processor generates the selected TPB prediction parameter array by minimizing the difference between the prediction image data in the one or more mapped images and the reference image data in one or more reference images at a reference color level. The one or more reference images correspond to the one or more first images and depict the same visual content as the visual content depicted by the one or more first images.

[0296] In block 456 , the video decoding system uses the TPB prediction parameter array together with the TPB basis function array to generate one or more mapped images from the one or more first images.

[0297] In block 458 , the video decoding system causes one or more display images derived from the one or more mapped images to be rendered by a display device.

[0298] In an embodiment, the video decoding system is further configured to perform: generating multiple B smooth curve basis function output values; applying a cross product operation to the multiple B smooth curve basis function output values ​​to generate multiple TPB basis function output values ​​to thereby generate a set of multiple TPB basis function output values; multiplying the TPB prediction parameter array decoded from the video signal with the multiple TPB basis function output value groups to generate a prediction codeword group.

[0299] In an embodiment, the selected TPB prediction parameter array is encoded as a plurality of weighting coefficients in a coding syntax that supports carrying respective ones of the plurality of weighting coefficients for corresponding TPB basis functions in a TPB basis function array.

[0300] In an embodiment, a computing device (e.g., a display device, a mobile device, a video converter, a multimedia device, etc.) is configured to perform any of the above methods. In an embodiment, an apparatus includes a processor and is configured to perform any of the above methods. In an embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, perform any of the above methods.

[0301] In one embodiment, a computing device includes one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, perform any of the above methods.

[0302] It should be noted that although separate embodiments are discussed herein, any combination of embodiments and / or portions of embodiments discussed herein may be combined to form further embodiments.

[0303] Example Computer System Implementation

[0304] Embodiments of the invention may be implemented using each of the following: a computer system, a system configured in an electronic circuit system and components, an integrated circuit (IC) device (e.g., a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD)), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may perform, control, or execute instructions related to adaptive perceptual quantization of images with enhanced dynamic range, such as the instructions described herein. The computer and / or IC may operate on any of the various parameters or values ​​associated with the adaptive perceptual quantization procedures described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0305] Specific embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present disclosure. For example, one or more processors in a display, an encoder, a video converter, a codec, or the like may implement methods related to the adaptive perceptual quantization of the above-mentioned HDR image by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention may also be provided in the form of a program product. The program product may include any non-transitory media that carries a set of computer-readable signals including instructions, which, when executed by a data processor, cause the data processor to perform the method of an embodiment of the present invention. The program product according to an embodiment of the present invention may be in any of various forms. The program product may include, for example, physical media, such as magnetic data storage media (including floppy disks, hard drives), optical data storage media (including CD ROMs, DVDs), electronic data storage media (including ROMs, flash memory RAMs), or the like. The computer-readable signals on the program product may be optionally compressed or encrypted.

[0306] When referring to the above-mentioned components (e.g., software modules, processors, assemblies, devices, circuits, etc.), unless otherwise indicated, references to the components (including references to "means") should be interpreted as including any components that perform the functions (e.g., functional equivalents) of the described components (including components that are structurally equivalent to the disclosed structures that perform the functions in the illustrative exemplary embodiments of the present invention) as equivalents of the components.

[0307] According to an embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing device may be hardwired to perform the techniques, or may include a digital electronic device (e.g., one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs)) that is permanently programmed to perform the techniques, or may include one or more general-purpose hardware processors that are programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs or FPGAs with custom programming to accomplish the techniques. The special-purpose computing device may be a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that incorporates hardwiring and / or program logic to implement the techniques.

[0308] For example, Figure 5 5 is a block diagram illustrating a computer system 500 upon which embodiments of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information and a hardware processor 504 coupled with bus 502 for processing information. Hardware processor 504 may be, for example, a general purpose microprocessor.

[0309] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage, coupled to bus 502 to store information and instructions executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, turn computer system 500 into a special-purpose machine customized to perform the operations specified in the instructions.

[0310] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 to store static information and instructions for processor 504. A storage device 510, such as a magnetic or optical disk, is provided and coupled to bus 502 to store information and instructions.

[0311] The computer system 500 may be coupled to a display 512, such as a liquid crystal display, via the bus 502 to display information to a computer user. An input device 514, which includes alphanumeric and other keys, is coupled to the bus 502 to communicate information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor arrow keys, for communicating directional information and command selections to the processor 504 and for controlling cursor movement on the display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y) to allow the device to specify a position within a plane.

[0312] The computer system 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic combined with the computer system to make or program the computer system 500 into a special purpose machine. According to an embodiment, the techniques described herein are performed by the computer system 500 in response to the processor 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium, such as the storage device 510. Execution of the sequences of instructions contained in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0313] The term "storage media" as used herein refers to any non-transitory media that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include volatile memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0314] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 502. Transmission media may also be in the form of acoustic or light waves, such as those generated during radio and infrared data communications.

[0315] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its volatile memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 may receive the data on the telephone line and use an infrared transmitter to convert the data into an infrared signal. An infrared detector may receive the data carried in the infrared signal and appropriate circuitry may place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may optionally be stored on the storage device 510 before or after execution by the processor 504.

[0316] The computer system 500 also includes a communication interface 518 coupled to the bus 502. The communication interface 518 provides two-way data communication coupled to a network link 520 connected to a local network 522. For example, the communication interface 518 may be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 may be a LAN card for providing a data communication connection to a compatible local area network (LAN). A wireless link may also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0317] The network link 520 typically provides data communication through one or more networks to other data devices. For example, the network link 520 may provide a connection through a local network 522 to a host computer 524 or a data device operated by an Internet Service Provider (ISP) 526. ISP 526, in turn, provides data communication services through a global packet data communication network, now commonly referred to as the "Internet" 528. Both the local network 522 and the Internet 528 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518, which carry the digital data to and from the computer system 500, are exemplary forms of transmission media.

[0318] Computer system 500 can send messages and receive data including program code through network(s), network link 520 and communication interface 518. In the Internet example, server 530 can transmit the request code of the application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0319] The received code may be executed by processor 504 as it is received and / or stored in storage device 510 or other non-volatile storage for later execution.

[0320] Equivalents, extensions, replacements and more

[0321] In the above description, embodiments of the present invention have been described with reference to many specific details that may vary with the implementation scheme. Therefore, the embodiments of the present invention claim and the applicant wishes to make it the only and exclusive indicator of the claimed embodiments of the present invention is the group of such claims issued from the present application in a specific manner (including any subsequent corrections) of claim issuance. Any definition explicitly set forth herein for the terms contained in such claims will control the meaning of such terms used in the claims. Therefore, the limitations, elements, properties, features, advantages or attributes that are not explicitly stated in the claims will never limit the scope of this claim in any way. Therefore, the specification and drawings should be regarded as intended to illustrate rather than limit.

[0322] Exemplary embodiments listed

[0323] The present invention may be embodied in any form described herein, including but not limited to the following enumerated exemplary embodiments (EEE) that describe the structure, features, and functions of some portions of embodiments of the present invention.

[0324] EEE1. A method comprising:

[0325] Determine a set of tensor product B smooth curve (TPB) basis functions;

[0326] generating a set of selected TPB prediction parameters for use with the set of TPB basis functions in generating predicted image data in one or more mapped images from source image data in one or more source images at a source color level, wherein the set of selected TPB prediction parameters is generated by minimizing a difference between the predicted image data in the one or more mapped images and reference image data in one or more reference images at a reference color level, wherein the one or more reference images correspond to the one or more source images and depict the same visual content as depicted by the one or more source images;

[0327] causing the set of selected TPB prediction parameters to be encoded in a video signal as part of picture metadata along with the source picture data in the one or more source pictures;

[0328] The one or more mapped images are caused to be reconstructed and reproduced by a receiving device of the video signal.

[0329] EEE2. The method according to EEE1, wherein at least one of the source image data or the reference image data is represented in a sub-sampled format of a color space. EEE3.

[0330] EEE3. A method according to EEE1 or 2, wherein the one or more source images represent one of the following: images constituting a visual scene in a media program, images selected within a sliding window, or images selected within a linear segment.

[0331] EEE4. A method according to any one of EEE1 to 3, wherein the group of TPB basis functions is generated by the tensor product of one or more groups of B smooth curve basis functions, and wherein each group of B smooth curve basis functions in the one or more groups of B smooth curve basis functions corresponds to a corresponding color channel in one or more color channels of a color space.

[0332] EEE5. A method according to EEE4, wherein at least one set of B smooth curve basis functions among the one or more sets of B smooth curve basis functions represents a complete set of B smooth curve basis functions of a specific order. EEE5.

[0333] EEE6. A method according to EEE4 or 5, wherein the one or more groups of B smooth curve basis functions include a group of B smooth curve basis functions generated by a truncated polynomial and a group of uniformly distributed nodes.

[0334] EEE7. A method according to any one of EEE1 to 6, wherein the combination of the set of selected TPB prediction parameters and the set of TPB basis functions represents a cross-channel predictor for generating the predicted image data in the one or more mapped images.

[0335] EEE8. A method according to any one of EEE1 to 7, wherein multiple mapping pairs are used to generate the set of TPB prediction parameters, each of the multiple mapping pairs comprising a first array of one or more source code words generated from the source image data and a second array of one or more reference code words generated from the reference image data.

[0336] EEE9. The method according to EEE8, wherein the multiple mapping pairs are generated based on a three-dimensional mapping table (3DMT).

[0337] EEE10. A method comprising:

[0338] decoding one or more first images at a first color level from a video signal;

[0339] decoding from the video signal image metadata including a set of selected TPB prediction parameters for multiplication with a set of tensor product TPB basis functions;

[0340] causing the set of TPB prediction parameters to be used together with the set of TPB basis functions to generate one or more mapped images from the one or more first images;

[0341] One or more display images derived from the one or more mapped images are caused to be reproduced by a display device.

[0342] EEE11. A method according to EEE10, wherein the set of selected TPB prediction parameters is generated by an upstream video content processor, wherein the set of selected TPB prediction parameters is used together with the set of TPB basis functions to generate predicted image data in one or more mapped images from first image data in one or more first images of a first color level, wherein the upstream video content processor generates the set of selected TPB prediction parameters by minimizing the difference between the predicted image data in the one or more mapped images and reference image data in one or more reference images of a reference color level, wherein the one or more reference images correspond to the one or more first images and depict the same visual content as the visual content depicted by the one or more first images.

[0343] EEE12. The method according to EEE10, further comprising:

[0344] Generate multiple B smooth curve basis function output values;

[0345] applying a cross product operation to the plurality of B smooth curve basis function output values ​​to generate a plurality of TPB basis function output values ​​to thereby generate a set of a plurality of TPB basis function output values;

[0346] The set of TPB prediction parameters decoded from the video signal is multiplied by the set of multiple TPB basis function output values ​​to generate a set of prediction codewords.

[0347] EEE13. The method according to EEE12, wherein generating a plurality of B smooth curve basis function output values ​​comprises:

[0348] For each B-smooth curve basis function of a plurality of B-smooth curve basis functions, determining a node and a corresponding multiplication factor used in a truncation polynomial to represent each B-smooth curve basis function;

[0349] The decoded codewords in the one or more first images are used as inputs to the truncated polynomial to generate output values ​​for each of the B-smooth curve basis functions.

[0350] EEE14. The method according to EEE12, further comprising:

[0351] storing the plurality of B smooth curve basis function output values ​​in a local register;

[0352] for each of the TPB basis functions, identifying (n+1) non-zero B smooth curve basis function output values ​​in each channel of a color space among the plurality of B smooth curve basis function output values, wherein n indicates an order of the plurality of B smooth curve basis functions;

[0353] The (n+1) non-zero B smooth curve basis function output values ​​are used as part of an input to generate an output value of each such TPB basis function from a decoded codeword in the one or more first images.

[0354] EEE15. The method according to EEE12, further comprising:

[0355] For each decoded codeword in the one or more first images, generating an output codeword in the one or more mapped images by performing the following operations:

[0356] Initializing the output codeword to zero;

[0357] For each of the TPB basis functions, execute:

[0358] generating (n+1) non-zero B smooth curve basis function output values ​​for each of three channels of the color space to thereby generate three sets of (n+1) non-zero B smooth curve basis function output values, wherein n indicates a first order of the plurality of B smooth curve basis functions;

[0359] applying a cross product operation to three sets of (n+1) non-zero B smooth curve basis function output values ​​to generate TPB basis function output values ​​from the decoded codewords in the one or more first images;

[0360] generating a product by multiplying the TPB basis function output value with a corresponding prediction parameter in the set of TPB prediction parameters;

[0361] The product is added to the output codeword.

[0362] EEE16. A method according to any one of EEE10 to 15, wherein the group of selected TPB prediction parameters is encoded as multiple weighting coefficients in a coding syntax, and the coding syntax supports carrying corresponding weighting coefficients of the multiple weighting coefficients for corresponding TPB basis functions in the group of TPB basis functions.

[0363] EEE17. A computer system configured to perform any of the methods according to EEE1 to EEE16.

[0364] EEE18. An apparatus comprising a processor and configured to perform any of the methods according to EEE1 to EEE16.

[0365] EEE19. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing a method according to any one of the methods described according to EEE1 to EEE16.

Claims

1. A method for image processing, comprising: generating a set of prediction parameters for generating predicted image data for at least one color channel of one or more mapped images from source image data in one or more source images of a source color level, wherein the one or more mapped images include M color channels, where M>1, Wherein generating the set of prediction parameters for the at least one color channel comprises: Determine a set of tensor products of B smooth curve TPB basis functions corresponding to the tensor product of M sets of B smooth curve basis functions; generating, as the set of prediction parameters for the at least one color channel, a set of selected TPB prediction parameters for use with the set of TPB basis functions for generating predicted image data for the at least one color channel of the one or more mapped images from the source image data in the one or more source images at the source color level, wherein the set of selected TPB prediction parameters is generated by minimizing a difference between the predicted image data for the at least one color channel in the one or more mapped images and reference image data in one or more reference images at a reference color level, wherein the one or more reference images correspond to the one or more source images and depict the same visual content as depicted by the one or more source images; causing the set of selected TPB prediction parameters to be encoded in a video signal as part of picture metadata together with the source picture data in the one or more source pictures so as to enable reconstruction and presentation of the one or more mapped pictures by a receiving device using the video signal; Wherein a plurality of mapping pairs are used to generate the set of TPB prediction parameters, each of the plurality of mapping pairs comprising a first array of one or more source codewords generated from the source image data and a second array of one or more reference codewords generated from the reference image data, and wherein the source codewords for pixels of the source image are divided into a fixed number of bins, and for each bin, an average value of the reference codewords for pixels of the reference image corresponding to the pixels of the source image in the corresponding bin is calculated, and the first array comprises center values ​​of the source codeword bins, and the second array comprises corresponding calculated averages of the reference codeword values.

2. The method of claim 1 , wherein determining the set of TPB basis functions comprises: determining a set of B smooth curve basis functions for each of the M color channels; and The set of TPB basis functions is determined as a tensor product of each of the M sets of B smooth curve basis functions.

3. A method according to claim 1 or claim 2, wherein at least one of the source image data or the reference image data is represented in a sub-sampled format of a color space.

4. The method of claim 1 or claim 2, wherein the one or more source images represent one of: images constituting a visual scene in a media program, images selected within a sliding window, or images selected within a linear segment. 5 . The method of claim 1 , wherein at least one of the M sets of B-smooth curve basis functions represents a full set of B-smooth curve basis functions of a particular order.

6. The method of claim 1 or claim 2, wherein each of the M sets of B smooth curve basis functions is generated using a set of uniformly distributed knots.

7. The method of claim 1 or claim 2, wherein each of the M sets of B smooth curve basis functions is generated using a truncated polynomial.

8. The method of claim 1 or claim 2, generating a set of selected TPB prediction parameters for each of the M color channels, wherein the set of selected TPB prediction parameters for at least two color channels are generated using the same set of TPB basis functions.

9. The method of claim 1 or claim 2, wherein the combination of the set of selected TPB prediction parameters and the set of TPB basis functions represents a cross-channel predictor for generating the predicted image data in the one or more mapped images.

10. The method according to claim 1 or claim 2, wherein the plurality of mapping pairs are generated based on a three-dimensional mapping table 3DMT.

11. A method for image processing, comprising: One or more first images of a first color level including M color channels are decoded from the video signal, wherein M>1； accessing a set of tensor products of B smooth curve TPB basis functions corresponding to the tensor products of M sets of B smooth curve basis functions; decoding from the video signal picture metadata including a set of selected TPB prediction parameters for multiplication with the set of TPB basis functions; causing the set of TPB prediction parameters to be used together with the set of TPB basis functions to generate one or more mapped images from the one or more first images; causing one or more display images derived from the one or more mapped images to be reproduced by a display device; wherein the set of TPB prediction parameters has been generated as a set of prediction parameters for at least one color channel of the one or more first images, the set of prediction parameters being used together with the set of TPB basis functions for generating predicted image data for the at least one color channel of the one or more first images from source image data in one or more source images at a source color level, wherein the set of TPB prediction parameters has been generated by minimizing a difference between the predicted image data for the at least one color channel of the one or more first images and reference image data in one or more reference images at a reference color level, wherein the one or more reference images correspond to the one or more source images and depict the same visual content as depicted by the one or more source images; wherein a plurality of mapping pairs have been used to generate the set of TPB prediction parameters, each of the plurality of mapping pairs comprising a first array of one or more source codewords generated from the source image data and a second array of one or more reference codewords generated from the reference image data, and wherein the source codewords for pixels of the source image have been divided into a fixed number of bins, and for each bin an average value of the reference codewords for pixels of the reference image corresponding to the pixels of the source image in the respective bin has been calculated, and the first array comprises the center values ​​of the source codeword bins, and the second array comprises the corresponding calculated average values ​​of the reference codeword values.

12. The method according to claim 11, further comprising: Generate multiple B smooth curve basis function output values; applying a cross product operation to the plurality of B smooth curve basis function output values ​​to generate a plurality of TPB basis function output values ​​to thereby generate a set of a plurality of TPB basis function output values; The set of TPB prediction parameters decoded from the video signal is multiplied by the set of multiple TPB basis function output values ​​to generate a set of prediction codewords.

13. The method according to claim 12, wherein generating a plurality of B-smooth curve basis function output values ​​comprises: For each B-smooth curve basis function of a plurality of B-smooth curve basis functions, determining a node and a corresponding multiplication factor used in a truncation polynomial to represent each B-smooth curve basis function; The decoded codewords in the one or more first images are used as inputs to the truncated polynomial to generate output values ​​for each of the B-smooth curve basis functions.

14. The method according to claim 12, further comprising: storing the plurality of B smooth curve basis function output values ​​in a local register; for each of the TPB basis functions, identifying n+1 non-zero B smooth curve basis function output values ​​in each channel of a color space among the plurality of B smooth curve basis function output values, wherein n indicates an order of the plurality of B smooth curve basis functions; The n+1 non-zero B smooth curve basis function output values ​​are used as part of an input to generate an output value of each such TPB basis function from a decoded codeword in the one or more first images.

15. The method according to claim 12, further comprising: For each decoded codeword in the one or more first images, generating an output codeword in the one or more mapped images by performing the following operations: Initializing the output codeword to zero; For each of the TPB basis functions, execute: generating n+1 non-zero B smooth curve basis function output values ​​for each of three channels of the color space to thereby generate three sets of n+1 non-zero B smooth curve basis function output values, wherein n indicates the order of the plurality of B smooth curve basis functions; applying a cross product operation to three sets of n+1 non-zero B smooth curve basis function output values ​​to produce TPB basis function output values ​​from the decoded codewords in the one or more first images; generating a product by multiplying the TPB basis function output value with a corresponding prediction parameter in the set of TPB prediction parameters; The product is added to the output codeword.

16. A method according to any one of claims 11 to 15, wherein the set of selected TPB prediction parameters is encoded as multiple weighting coefficients in a coding syntax, and the coding syntax supports carrying corresponding weighting coefficients of the multiple weighting coefficients for corresponding TPB basis functions in the set of TPB basis functions.

17. An apparatus comprising a processor and configured to perform the method of any one of claims 1 to 16.

18. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 16.

Images