Image optimization in mobile capture and editing applications

Abstract

HDR color patches are sampled across an HDR color space parameterized by the parameters. From the sampled HDR color patches, a reference SDR color patch, an input HDR color patch, and a reference HDR color patch are generated. An optimization algorithm is run to generate an optimized forward reshaping mapping and an optimized backward reshaping mapping. The optimized forward reshaping mapping is used to forward reshape the input HDR image into a forward reshaped SDR image, and the optimized backward reshaping mapping is used to backward reshape the front reshaped SDR image into a backward reshaped HDR image.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 321,390, filed March 18, 2022, and European Patent Application No. 22162983.5, filed March 18, 2022, each of which is incorporated by reference in its entirety.

[0002] technology More particularly, embodiments of the present disclosure relate to video codecs used to process images. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest black (shadows) to brightest white (highlights). In this sense, DR relates to "scene-referenced" intensities. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular width. In this sense, DR relates to "display-referenced" intensities. Unless a particular meaning is explicitly specified at any point in this description, the term should be presumed to be used in either sense, e.g., interchangeably.

[0004] As used herein, the term high dynamic range (HDR) refers to a DR width that spans approximately 14 to 15 orders of magnitude or more of the human visual system (HVS). In practice, the DR that humans can simultaneously perceive across a wide range of intensity ranges may be somewhat truncated compared to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may individually or interchangeably refer to the DR perceivable in a scene or image by the human visual system (HVS), including eye movements, taking into account any light adaptation changes across the scene or image. As used herein, EDR may refer to a DR that spans 5 to 6 orders of magnitude. Thus, while perhaps somewhat narrower compared to true scene-based HDR, EDR nevertheless represents a wide DR width and may also be referred to as HDR.

[0005] In practice, an image comprises one or more color components of a color space (e.g., luma Y, chroma Cb, and Cr), each represented with a precision of n bits per pixel (e.g., n=8). Using nonlinear luminance coding (e.g., gamma encoding), images with n≦8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images with n>8 are considered enhanced dynamic range images.

[0006] The reference electro-optical transfer function (EOTF) of a given display characterizes the relationship between the color values ​​(e.g., luminance) of the input video signal and the output screen color values ​​(e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT.1886, "Reference Electro-Optical Transfer Functions for Flat-Panel Displays Used in HDTV Studio Production" (March 2011), defines the reference EOTF for flat-panel displays. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. The term "metadata" here refers to any auxiliary information transmitted as part of the encoded bitstream to assist a decoder in rendering the decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein.

[0007] As used herein, the term "PQ" refers to perceptual luminance amplitude quantization. The human visual system responds in a highly nonlinear manner to increasing light levels. A person's ability to see a stimulus is affected by the stimulus's luminance, its size, the spatial frequencies that make up the stimulus, and the luminance level to which the eye is adapted at the particular moment the stimulus is viewed. In some embodiments, a perceptual quantizer function maps linear input gray levels to output gray levels that more closely match the contrast sensitivity thresholds of the human visual system. An exemplary PQ mapping function is described in SMPTE ST 2084:2014, "High Dynamic Range EOTF for Mastering Reference Displays" (hereinafter, "SMPTE"), which is incorporated herein by reference in its entirety. Here, given a fixed stimulus size, for each luminance level (e.g., stimulus level), the minimum visible contrast step at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

[0008] 200~1000cd / m 2 Displays supporting high luminance are representative of low dynamic range (LDR), also known as standard dynamic range (SDR), as compared to EDR (or HDR). EDR content may be displayed on EDR displays that support a higher dynamic range (e.g., 1,000 nits to 5,000 nits or more). Such displays may be defined using an alternative EOTF that supports high luminance capabilities (e.g., 0 to 10,000 nits or more). Examples of such EOTFs are defined in SMPTE 2084 and Rec. ITU-R BT.2100, "Picture Parameter Values ​​for High Dynamic Range Television for Use in Production and International Program Exchange" (06 / 2017). As recognized by the present inventors, improved techniques for structuring video content data that can be used to support the display capabilities of a wide variety of SDR and HDR display devices are desired.

[0009] Patent Literature 1 relates to image encoding and / or image decoding. As a non-limiting example, a technique is disclosed in which 3D mapping statistics are generated for a first image in a first dynamic range and a second image in a second dynamic range different from the first dynamic range. In the exemplary technique, multivariate multiple regression (MMR) coefficients are generated by solving an optimization problem formulated using an MMR matrix constructed with the 3D mapping statistics without a letterbox constraint, and are used to generate a chroma mapping for predicting chroma codeword values ​​for the second image. Furthermore, the exemplary technique determines whether a letterbox exists in the image, and if so, determines whether the chroma mapping accurately predicts the chroma codeword values ​​in the second image. According to a further non-limiting example, a reconstructed image generated by a recipient device by backward reshaping one of the images is rendered by a display device operating in conjunction with the recipient device. [Patent Document 1] WO2019 / 217751A1

[0010] Patent Document 2 relates to image encoding and / or image decoding. As a non-limiting example, a technique is disclosed in which a backward reshaping mapping table is initially generated as the inverse of a forward reshaping mapping table. In this exemplary technique, the backward reshaping mapping table is updated by replacing content-mapped luma codewords with forward reshaping luma codewords generated by applying a luma forward mapping to sampled luma codewords. Furthermore, according to this exemplary technique, a luma forward mapping is constructed from the forward reshaping mapping table, and the backward reshaping mapping table and the luma forward mapping are used to generate a backward reshaping mapping for creating a reconstructed image from the forward reshaping image. Furthermore, according to this exemplary technique, the forward reshaping image is encoded in the video signal along with image metadata specifying the backward reshaping mapping. According to a further non-limiting example, a receiver device of the video signal applies the backward reshaping mapping to the forward reshaping image to create a reconstructed image of a second dynamic range. [Patent Document 2] WO2021 / 076822A1

[0011] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise expressly stated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Likewise, it should not be assumed that problems identified with one or more approaches have been recognized in any prior art based on this section, unless otherwise expressly stated. [Brief explanation of the drawings]

[0012] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0013] Like reference numbers refer to like elements.

[0014] [Figure 1A] 1 illustrates an exemplary process flow for applying optimized reshaping operations. [Figure 1B] 1 illustrates an exemplary process flow for applying optimized reshaping operations. [Figure 1C] 1 illustrates an exemplary process flow for applying optimized reshaping operations. [Figure 1D] 1 illustrates an exemplary process flow for applying optimized reshaping operations. [Figure 1E] 1 illustrates an exemplary process flow for applying optimized reshaping operations. [Figure 2A] 1 illustrates example color distributions of irregular shapes associated with HDR and SDR color spaces, such as the R.2020, P3, and R.709 color spaces. [Figure 2B] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2C] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2D] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2E] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2F] 10 shows exemplary percentile Cb and Cr values ​​in forward reshaped SDR color spaces corresponding to different parameterized HDR color spaces. [Figure 2G] 10 shows exemplary percentile Cb and Cr values ​​in forward reshaped SDR color spaces corresponding to different parameterized HDR color spaces. [Figure 2H] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2I] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2J] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2K] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2L] 1 shows an exemplary color chart image. [Figure 2M] 1 shows an exemplary checkerboard image. [Figure 2N] 1 shows an exemplary checkerboard image. [Figure 2O] 1 shows an exemplary original color chart image displayed on a reference image display, an exemplary captured image from the color chart image, and a modified captured image generated from the captured image. [Figure 2P] 1 shows an exemplary distribution of RGB values. [Figure 2Q] 1 shows an exemplary alpha shape. [Figure 2R] 1 shows an exemplary alpha shape. [Figure 2S] 1 shows an exemplary alpha shape. [Figure 2T] 1 shows an exemplary alpha shape.

[0015] [Figure 3A] An exemplary process flow for finding an optimized color space for representing HDR colors is shown. [Figure 3B] An exemplary process flow for determining optimized values ​​for programmable parameters in an ISP pipeline is shown. [Figure 3C] An exemplary process flow for finding an optimized color space for representing HDR colors is shown. [Figure 3D] 1 illustrates an exemplary process flow for generating optimized reshaping operating parameters. [Figure 3E]An exemplary process flow for generating several different color charts is shown. [Figure 3F] An exemplary process flow for matching SDR and HDR colors between a captured image pair of an SDR image and an HDR image is shown. [Figure 3G] 1 illustrates an exemplary encoder-side image / video editing operation. [Figure 3H] 1 illustrates an exemplary decoder-side image / video editing operation. [Figure 3I] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3J] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3K] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3L] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3M] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3N] An exemplary process flow is shown.

[0016] [Figure 4A] An exemplary process flow is shown. [Figure 4B] An exemplary process flow is shown. [Figure 4C] An exemplary process flow is shown. [Figure 4D] An exemplary process flow is shown. [Figure 4E] An exemplary process flow is shown.

[0017] [Figure 5]1 illustrates a simplified block diagram of an exemplary hardware platform upon which a computer or computing device described herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0018] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obfuscating the present disclosure.

[0019] overview Image optimization, such as that associated with tensor-product B-Spline (TPB)-based image reshaping solutions, is described herein for video capture applications, including, but not limited to, mobile (video) capture applications. TPB-based solutions can provide or generate relatively accurate reconstructed images using robust predictors implemented in video codecs, such as backward-compatible video codecs. These predictors can operate with static mappings to gracefully handle or recover from transmission errors, such as frame drops, minimize power consumption, such as battery power consumption, reduce data overhead and / or processing / transmission latency, etc. In some operating scenarios, some or all of the TPB-based solutions described herein may be implemented in mobile applications that are subject to relatively strict battery or power constraints.

[0020] TPB-based image reshaping solutions can be adapted to operate in a variety of use cases. Some use cases allow the freedom to design and / or apply HDR-to-SDR mappings to generate SDR images to be encoded in (elementary layers of) a video signal, as described herein. Some use cases rely on mobile image signal processors (ISPs) with relatively limited programmable registers to generate SDR images to be encoded in (elementary layers of) a video signal. Various architectures can be used to implement the TPB-based solutions described herein. Additionally, optionally or alternatively, these architectures or TPB-based solutions can be used to provide or support backward compatibility.

[0021] Given a reference SDR image represented in an SDR color space or a color space within the SDR domain, a TPB-based solution can use a TPB optimization process to achieve or determine the largest or widest HDR color space or color space within the HDR domain to represent the corresponding reconstructed HDR image predicted from the SDR image. The wider the HDR color space, the greater the color deviation introduced into the SDR color space. The TPB optimization process can be performed to achieve or determine an optimized balance point or trade-off between achieving the largest or widest HDR color space, on the one hand, and introducing SDR color deviation, on the other hand. Additionally, optionally, or alternatively, the TPB optimization process described herein can be implemented to incorporate neutral color processing or perform constrained optimization to help preserve gray levels represented in the SDR image in the reconstructed HDR image predicted from the SDR image.

[0022] As more and more video is captured by computing devices and mobile devices equipped with various cameras, video editing is also becoming increasingly common. Users operating these devices may be allowed to manually adjust the appearance of the captured video or simply apply a default theme template to the captured video. Depending on the available computational resources and / or the preferences of the video editing tool designer / provider, video editing may occur in either a source domain, such as a source HDR domain where the source image is represented, or in a non-source domain, such as a generated SDR domain, where a pre-edited image may be converted from the source image in the source domain.

[0023] In an operating scenario where an SDR image is encoded in a video signal, video editing operations in the HDR domain or source domain do not adversely affect devices downstream of the video signal that reconstruct an HDR image from an SDR image decoded from the video signal at the decoder side, because these video editing operations do not interfere with the HDR-to-SDR mapping at the encoder side to generate an SDR image from an HDR image or a source image in the HDR domain, nor do they interfere with the SDR-to-HDR mapping at the decoder side to map back to an HDR image or reconstruct an approximation of an HDR image.

[0024] However, in these operating scenarios, video editing operations in the SDR domain are likely to adversely affect reversibility between the SDR and HDR domains. For example, an edited SDR image may not be able to be mapped back to the original or source HDR image. Furthermore, the color space representing the SDR image may be limited by a predefined range, such as the SMPTE range, that pixel values ​​or codewords cannot exceed. Color space conversions, such as YUV-RGB conversion, used for video editing operations may clip pixel values ​​or codewords, thus impairing or harming reversibility. The clipping techniques described herein may be used to reduce or prevent adverse editing effects for video editing operations, including, but not limited to, those performed on mobile devices. Two-level TPB boundary clipping may be implemented in the RGB domain to support or preserve the maximum edited colors in the SDR domain that can be propagated or mapped to a reconstructed image in the HDR domain.

[0025] Exemplary embodiments described herein relate to image generation. Sampled HDR color space points distributed throughout an HDR color space are constructed. The HDR color space is parameterized by primary color scaling parameters having candidate values ​​selected from among a plurality of candidate values. The primary color scaling parameters are used to calculate color space coordinates of at least one of a plurality of primary colors that describe the HDR color space. From the sampled HDR color space points in the HDR color space, reference SDR color space points represented in a reference SDR color space, input HDR color space points represented in an input HDR color space, and reference HDR color space points represented in the reference HDR color space are generated. A reshaping operation optimization algorithm is performed to generate a chain of optimized forward reshaping mappings and optimized backward reshaping mappings. The reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs. The optimized forward reshaping mapping is used to forward reshape an input HDR image in the input HDR color space into a forward reshaped SDR image in the forward reshaped SDR color space, while the optimized backward reshaping mapping is used to backward reshape a forward reshaped SDR image in the forward reshaped SDR color space into a backward reshaped HDR image.

[0026] Exemplary embodiments described herein relate to image generation. Sampled HDR color space points distributed throughout an HDR color space are constructed. The HDR color space is parameterized by primary color scaling parameters having candidate values ​​selected from among a plurality of candidate values. The primary color scaling parameters are used to calculate color space coordinates of at least one of a plurality of primary colors that describe the HDR color space. Input SDR color space points represented in an input SDR color space and reference HDR color space points represented in a reference HDR color space are generated from the sampled HDR color space points in the HDR color space. A reshaping operation optimization algorithm is performed to generate an optimized backward reshaping mapping. The reshaping operation optimization algorithm receives the input SDR color space points and the reference HDR color space points as inputs. The backward reshaping mapping is used to backward reshape an SDR image in the input SDR color space into a backward reshaped HDR image.

[0027] Exemplary embodiments described herein relate to image generation. A set of SDR image features is extracted from a training SDR image, while a set of HDR image features is extracted from a training HDR image. A subset of one or more SDR image features in the set of SDR image features is matched with a subset of one or more HDR image features in the set of HDR image features. The subset of one or more SDR image features and the subset of one or more HDR image features are used to generate a geometric transformation for spatially aligning a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image. After the training SDR and HDR images are spatially aligned by the geometric transformation, a set of SDR and HDR color patch pairs is determined from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image. An optimized SDR-HDR mapping is generated based at least in part on the set of SDR and HDR color patch pairs derived from the training SDR image and the training HDR image. The optimized SDR-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0028] Exemplary embodiments described herein relate to image generation. A respective camera distortion correction operation is performed on each training image in a pair of training standard dynamic range (SDR) images and training high dynamic range (HDR) images to generate a respective undistorted image in a pair of undistorted training SDR images and undistorted training HDR images. Each projective transform in the pair of SDR image projective transformation and HDR image projective transformation is generated using a corner pattern mark detected from each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images. Each projective transform in the pair of SDR image projective transformation and HDR image projective transformation is applied to each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images to generate a respective rectified image in the pair of rectified training SDR images and undistorted training HDR images. A set of SDR color patches is extracted from the rectified training SDR images, while a set of HDR color patches is extracted from the rectified training HDR images. The optimized SDR-HDR mapping is generated based at least in part on a set of SDR color patches and a set of HDR color patches derived from the training SDR image and the training HDR image, and is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0029] Exemplary embodiments described herein relate to a clipping operation on an edited image. Sampled HDR color space points distributed throughout an HDR color space used to represent a reconstructed HDR image are constructed. The sampled HDR color space points are converted to SDR color space points in a first SDR color space in which an SDR image to be edited by an editing device is represented. A bounding SDR color space rectangle is determined based on SDR codeword extrema of the SDR color space points in the first SDR color space. An irregular three-dimensional (3D) shape is determined from the distribution of the SDR color space points. Sampled SDR color space points distributed throughout the bounding SDR color space rectangle in the first SDR color space are constructed. The sampled SDR color space points and the irregular shape are used to generate a boundary clipping 3D lookup table (3D-LUT). The boundary clipping 3D-LUT uses the sampled SDR color space points as lookup keys. A clipping operation is performed on the edited SDR image in the first SDR color space based at least in part on the boundary clipping 3D-LUT to generate a boundary-clipped edited SDR image in the first SDR color space.

[0030] Optimizing reshaping in image / video capture applications A reshaping optimization process, such as a TPB and / or non-TPB optimization process, can be implemented or incorporated into a video capture application running on a computing device, such as a mobile device, in a variety of operating scenarios. The video capture application with the reshaping optimization process can be used to generate or output a video signal, such as a base layer or SDR image encoded therein. The reshaping optimization process can be implemented in a variety of solutions in a variety of operating scenarios to generate or optimize reshaping operating parameters, such as TPB and / or non-TPB coefficients, that are used with a base layer or SDR image to generate, construct, or reconstruct a non-base layer or HDR image having optimized image quality.

[0031] For illustrative purposes, the reshaping optimization process, or the solution implemented therein, can be classified into different types based on the particular SDR generation process or sub-process employed by the video capture application, as well as based on the particular reshaping path (forward (reshaping) path and / or backward (reshaping) path) that is subject to the reshaping optimization.

[0032] 1A shows a first exemplary reshaping optimization process that implements a white-box joint forward and backward TPB optimization design / solution (referred to as "WFB") using a (single) video capture device. The white-box joint forward and backward TPB optimization design / solution can be implemented for operating scenarios in which an SDR generation process or sub-process converts a reference HDR image to a reference SDR image using a white-box transformation operation, and both the forward and backward (reshaping) paths can undergo TPB optimization.

[0033] As used herein, a "white-box transformation" refers to a mapping or conversion from HDR to SDR that uses a clearly defined transformation or mapping function / formula that is specified or documented (e.g., publicly) in a standards-based or proprietary video coding specification. In contrast, a "black-box transformation" refers to a mapping or conversion operation from HDR to SDR that is not based on such a clearly defined transformation or mapping function / formula. For example, a black-box transformation may be implemented as an internal image signal process performed by an image signal processor without or with little to no clearly defined transformation or mapping function or formula that is specified or documented (e.g., publicly) in a standards-based or proprietary video coding specification.

[0034] 1A, an input video signal including an HDR image may be represented in an input color space, such as the full R.2020 color space. A generated SDR signal including a reference SDR image may be represented in a reference SDR color space. The reference SDR image may be an image generated from the HDR image using a known HDR-to-SDR mapping process, function, and / or formula.

[0035] 1A, in the forward reshaping path, a reference HDR image may be forward reshaped into a (forward) reshaped SDR image that approximates the reference SDR image based at least in part on optimized forward reshaping operating parameters (referred to as "forward TPB optimization") generated from the joint forward-backward TPB optimization solution. The reshaped SDR image may be represented in a reshaped SDR color space and generated by an upstream device to be included / encoded in a video signal output by the upstream device or its base layer (BL).

[0036] A downstream recipient device or video decoder of the video signal can decode the reshaped SDR image from the video signal. The decoded reshaped SDR image at the decoder side may be the same as the reshaped SDR image at the encoder side, but may be subject to errors introduced in the compression / decompression, encoding operations, and / or data transmission.

[0037] In a backward reshaping path implemented by a downstream device, the reshaped SDR image may be backward reshaped into a (backward) reshaped HDR image based at least in part on optimized backward reshaping operation parameters (referred to as "backward TPB optimization") generated from the same joint forward and backward TPB optimization solution. The reshaped HDR image—generated with the optimized backward reshaping operation parameters in the output HDR color space—represents an approximation or reconstructed version of the reference HDR image.

[0038] The joint forward and backward TPB optimization solution can generate optimized forward and backward reshaping operation parameters to cover the widest possible range in the output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0039] One or both of the optimized forward and backward reshaping operation parameters, such as the optimized forward and backward TPB coefficients, can be implemented or represented in a three-dimensional lookup table or 3D-LUT(s) to reduce processing time in the reshaping operation.

[0040] Because forward and backward reshaping operation parameters, such as forward and backward TPB coefficients, are jointly or concurrently designed or optimized in a joint forward and backward TPB optimization process, supported reshaped SDR and HDR color spaces used to represent reshaped SDR and HDR images can be jointly or concurrently designed or optimized in the same process.

[0041] In some operating scenarios, the optimized forward and backward reshaping motion parameters generated from the joint forward and backward TPB optimization process may be applied in a static single-layer backward compatible (SLBC) framework. Under this static framework, there is no need to obtain and optimize (e.g., dynamically) image-specific or image-dependent optimized forward and backward motion parameters, such as image-specific or image-dependent (or content-dependent) forward and backward TPB coefficients, on the fly, e.g., while images are being processed.

[0042] Rather, under a static SLBC framework, the same or static optimized forward and backward operating parameters, such as the same optimized forward and backward TPB coefficients, can be obtained or generated once—for example, offline or before performing a reshaping operation on either the (input) reference HDR image or the reshaped SDR image—for forward reshaping all (input) reference HDR images and backward reshaping all reshaped SDR images. In one example, a single set of static optimized forward and backward operating parameters can be generated offline by a system described herein and configured / deployed in or used by a capture device described herein to perform an image reshaping or reconstruction operation. In another example, multiple sets of static optimized forward and backward operating parameters can be generated offline by a system described herein and configured / deployed in or used by a capture device described herein to select a particular set of static optimized forward and backward operating parameters to perform an image reshaping or reconstruction operation.

[0043] The optimized static forward TPB coefficients can then be applied by an upstream device to forward reshape some or all of an (input) reference HDR image (e.g., a sequence of consecutive or sequential (input) reference HDR images) to generate a reshaped SDR image to be encoded in the (SLBC) video signal, while the optimized static backward TPB coefficients can be applied by a downstream recipient device of the video signal to some or all of a reshaped SDR image (e.g., a sequence of consecutive or sequential reshaped SDR images) decoded from the video signal to generate or reconstruct a reshaped HDR image.

[0044] 1B shows a second exemplary reshaping optimization process using a (single) video capture device to implement a white-box rear-only TPB optimization design / solution (referred to as "WB"). The white-box rear-only TPB optimization design / solution can be implemented for operating scenarios in which an SDR generation process or sub-process converts a reference HDR image to a reference SDR image using a white-box transformation operation, and only the rear (reshaping) path undergoes TPB optimization.

[0045] 1B, an input video signal including an HDR image may be represented in an input color space, such as the P3 color space. A generated SDR signal including a reference SDR image may be represented in a reference SDR color space. The reference SDR image may be an image generated from the HDR image using a known HDR-to-SDR mapping process, function, and / or formula.

[0046] 1B, the reference HDR image may be processed by a programmable image signal processor or ISP (e.g., using a given image signal processing function / formula / operation, etc.) based at least in part on optimized ISP operating parameters (referred to as "ISP parameter optimization") generated from the ISP optimization solution into an ISP SDR image that approximates the reference SDR image. The ISP SDR image may be expressed in an ISP SDR color space and generated by an upstream device to be included / encoded in a video signal or its base layer (BL) output by the upstream device.

[0047] A downstream recipient device or video decoder of the video signal can decode the ISP SDR image from the video signal. The decoded ISP SDR image at the decoder side may be the same as the ISP SDR image at the encoder side, but may be subject to errors introduced in the compression / decompression, encoding operations, and / or data transmission.

[0048] In the backward reshaping path as implemented by a downstream device, the ISP SDR image may be backward reshaped into a (backward) reshaped HDR image based at least in part on optimized backward reshaping operation parameters (referred to as "backward TPB optimization") generated from a backward-only TPB optimization solution. The reshaped HDR image—generated with the optimized backward reshaping operation parameters in the output HDR color space—represents an approximation or reconstructed version of the reference HDR image.

[0049] The backward-only TPB optimization solution can generate optimized backward reshaping operation parameters that cover the widest possible range in the output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0050] The optimized backward reshaping operation parameters, such as the optimized backward TPB coefficients, can be implemented or represented in a three-dimensional look-up table or 3D-LUT to reduce the processing time in the reshaping operation.

[0051] In some operating scenarios, the backward reshaping operating parameters generated from the backward-only TPB optimization process can be applied in a static single-layer inverse display mapping (SLiDM) framework. Under this static framework, there is no need to (e.g., dynamically) obtain or optimize image-specific or image-dependent (or content-dependent) optimized backward operating parameters, such as image-specific or image-dependent (or content-dependent) backward TPB coefficients.

[0052] Rather, under the static SLiDM framework, the same or static optimized backward operating parameters, such as the same optimized backward TPB coefficients, can be obtained or generated once—e.g., offline or before performing a reshaping operation on any of the reshaped SDR images—to backward reshape ISP SDR images. In one example, a single set of static optimized backward operating parameters can be generated offline by a system described herein and configured / deployed in or used by a capture device described herein to perform an image reshaping or reconstruction operation. In another example, multiple sets of static optimized backward operating parameters can be generated offline by a system described herein and configured / deployed in or used by a capture device described herein to select a particular set of static optimized backward operating parameters to perform an image reshaping or reconstruction operation.

[0053] The optimized static backward TPB coefficients can then be applied by a downstream recipient device of the video signal to some or all of the ISP SDR images decoded from the video signal (e.g., a sequence of consecutive or sequential ISP SDR images) to generate or reconstruct a reshaped HDR image.

[0054] The ISP SDR image encoded in the video signal may or may not be identical to the desired SDR appearance as represented by the reference image. The operating parameters or their settings in the programmable ISP can be optimized to approximate the reference image. Thus, in the "WB" operating scenario of FIG. 1B, the SDR image for backward reshaping into a reshaped HDR image is provided as the ISP SDR image or is fixed. For comparison, in the WFB operating scenario of FIG. 1A, the SDR image for backward reshaping into a reshaped HDR image is a forward reshaped SDR image, which can be optimized by applying optimized forward reshaping operating parameters generated in a joint forward and backward TPB optimization process that generates optimized forward and backward reshaping parameters. In other words, TPB optimization is not used in the forward path to generate the ISP SDR image with the aim of improving or enhancing both the desired appearance of the SDR image encoded in the video signal and the desired appearance of the reshaped HDR image. In these operating scenarios, relatively large deviations from the desired appearance can occur.

[0055] 1C shows a third exemplary reshaping optimization process implementing a black-box rear-only TPB optimization design / solution (referred to as "BB1") using a (single) video capture device. The black-box rear-only TPB optimization design / solution may be implemented for operating scenarios in which an SDR image is not generated from an HDR image using a white-box transformation operation, and only the rear (reshaping) path undergoes TPB optimization.

[0056] The reshaping operational parameters for these operational scenarios can be generated using training SDR images and training HDR images generated or captured by the same video capture device, such as the same mobile device, forming a plurality of SDR and HDR image pairs, each including a training SDR image and a corresponding training HDR image.

[0057] The training SDR image and the training HDR image in the same SDR and HDR image pair may be acquired at different time instances / points (e.g., a few milliseconds apart, a fraction of a second apart, a few seconds apart, etc.) using the same capture device with the same ISP. Because it is difficult, if not impossible, to maintain the same shooting position and process the training SDR and HDR images in the same manner at different time instances / points, the training SDR and HDR images may not be strictly spatially or temporally aligned with each other. For example, the training SDR image may be locally tone mapped or locally enhanced. Meanwhile, the training HDR image may be generated or acquired from multiple camera exposures. Thus, in the "BB1" operating scenario, the relationship between pixel values ​​or codeword values ​​in the training SDR image and the corresponding pixel values ​​or codeword values ​​in the training HDR image in the same image pair may be treated or assumed to be a black box.

[0058] The training SDR image and HDR image in each image pair may first be spatially aligned. The spatially aligned training SDR image and HDR image in the image pair may be used to determine or find matching color pairs. These matching color pairs can then be used to generate optimized reshaping operation parameters, such as optimized TPB and / or non-TPB coefficients, for reshaping or mapping the (e.g., non-training, reference, etc.) SDR image to a backward-reshaped or reconstructed HDR image that approximates the (e.g., non-training, reference, etc.) HDR image.

[0059] 1C, an SDR image, such as a reference SDR image, may be generated by an upstream device, such as a capture device, for inclusion / encoding in a video signal or base layer (BL) thereof output by the upstream device. Exemplary reference SDR images described herein may include, but are not necessarily limited to, an SDR image generated from a programmable image signal processor of the upstream device or a capture device operating in conjunction with the upstream device.

[0060] A downstream recipient device or video decoder of the video signal can decode the reference SDR image from the video signal. The decoded reference SDR image at the decoder side may be the same as the reference SDR image at the encoder side, but may be subject to errors introduced in the compression / decompression, encoding operations, and / or data transmission.

[0061] In the backward reshaping path as implemented by a downstream device, the reference SDR image may be backward reshaped into a (backward) reshaped HDR image based at least in part on optimized backward reshaping operating parameters (referred to as "backward TPB optimization") generated from a black-box backward-only TPB optimization design / solution. The reshaped HDR image—generated with the optimized backward reshaping operating parameters in the output HDR color space—represents an approximate or reconstructed version of the reference HDR image that can or could be generated by the same capture device.

[0062] The black-box backward-only TPB optimization design / solution can generate optimized backward reshaping operation parameters that cover the widest possible range in the output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0063] The optimized backward reshaping operation parameters, such as the optimized backward TPB coefficients, can be implemented or represented in a three-dimensional look-up table or 3D-LUT to reduce the processing time in the reshaping operation.

[0064] In some operating scenarios, the backward reshaping operating parameters generated from the black-box backward-only TPB optimization design / solution may be applied in the SLiDM framework. Under this static framework, there is no need to obtain and optimize (e.g., dynamically) image-specific or image-dependent optimized backward operating parameters, such as image-specific or image-dependent (or content-dependent) backward TPB coefficients.

[0065] Rather, under a static SLiDM framework, the same or statically optimized backward operating parameters, such as the same optimized backward TPB coefficients, can be obtained or generated once—for example, offline or before performing a reshaping operation on any of the reshaped SDR images—to backward reshape spatially aligned training SDR images generated by an upstream capture device into reshaped or reconstructed HDR images that approximate the spatially aligned training HDR images generated by the same capture device.

[0066] The optimized static backward TPB coefficients can then be applied by a downstream recipient device of the video signal to some or all of a (e.g., non-training, etc.) reference SDR image decoded from the video signal (e.g., a sequence of consecutive or sequential reference SDR images, etc.) to generate or reconstruct a reshaped HDR image that approximates a reference HDR image that can or could be generated from the same upstream capture device that generates or captures the reference SDR image.

[0067] Because TPB optimization is used only in the backward path, the resulting appearance of the reshaped HDR image generated from backward reshaping the reference SDR image may have a relatively large deviation from the desired appearance of the reference HDR image in the "BB1" operating scenario.

[0068] In some "BB1" operating scenarios, as shown in Figure 1D, a camera raw image generated from a camera ISP can be directly encoded in the video signal output by an upstream (capture) device instead of a reference SDR image generated from a programmable image signal processor. During the training phase, the training camera raw image, which may or may not be an SDR image, can be spatially and / or temporally aligned with a training HDR image within the same image pair of the camera raw image and HDR image. Matching color pairs from the spatially aligned camera raw image and HDR image image pair can be used to optimize backward reshaping operation parameters, such as the backward TPB coefficients. During the non-training or deployment phase, the optimized backward reshaping operation parameters can be used by a downstream device of the video signal encoded with the non-training camera raw image to backward reshape the decoded non-training camera raw image from the video signal into a backward reshaped or reconstructed (non-training) HDR image that approximates the reference HDR image that can or should be generated by the same upstream (capture) device.

[0069] 1E illustrates a fourth exemplary reshaping optimization process using a different video capture device to implement a black-box rear-only reshaping optimization design / solution (referred to as "BB2"). The "BB2" reshaping optimization design / solution may be implemented for an operating scenario in which an SDR image is generated by a first capture device, an HDR image generated from the SDR image is generated (e.g., intended to be generated) by a second, different capture device, and only the rear (reshaping) path undergoes reshaping optimization. Furthermore, the reshaping optimization in the "BB2" operating scenario may or may not be a TPB optimization.

[0070] The reshaping operating parameters in the "BB2" operating scenario can be generated using training SDR images and training HDR images generated or captured by two video capture devices, such as two mobile devices, of different makes and / or models, respectively, forming a plurality of SDR and HDR image pairs, each including a training SDR image and a corresponding training HDR image.

[0071] The training SDR image and the training HDR image in the same SDR image and HDR image pair may be acquired using the same image, for example, the same checkerboard image rendered on the same or similar image display(s). In the "BB1" operating scenario, the relationship between pixel values ​​or codeword values ​​in the training SDR image and corresponding pixel values ​​or codeword values ​​in the training HDR image in the same image pair may be treated as or assumed to be a black box.

[0072] The training SDR and HDR images in each image pair may first be spatially aligned. The spatially aligned training SDR and HDR images in the image pair may be used to determine or find matching color patches rendered with a test image, such as a checkerboard image, on the same image display or display type. These matching color patches may then be used to construct a three-dimensional lookup table (3D-LUT) for mapping SDR pixel values ​​or codeword values ​​of color patches in the test image to corresponding HDR pixel values ​​or codeword values ​​of the same color patches.

[0073] The 3D-LUT derived based in part or in whole on the test image can be used to generate optimized static or dynamic reshaping operating parameters for reshaping or mapping a (e.g., non-training, reference, etc.) SDR image to a backward reshaped or reconstructed HDR image that approximates the (e.g., non-training, reference, etc.) HDR image.

[0074] 1E, an SDR image, such as a reference SDR image, may be generated by an upstream device, such as a first capture device, to be included / encoded in a video signal or its base layer (BL) output by the upstream device. Exemplary reference SDR images described herein may include, but are not necessarily limited to, an SDR image generated from a programmable image signal processor of the first capture device or the upstream device.

[0075] A downstream recipient device or video decoder of the video signal can decode the reference SDR image acquired by the first capture device from the video signal. The decoded reference SDR image at the decoder side may be the same as the reference SDR image at the encoder side, but may be subject to errors introduced in the compression, decompression, encoding operations, and / or data transmission.

[0076] In the backward reshaping path performed by the downstream device, the reference SDR image may be backward reshaped into a (backward) reshaped HDR image based at least in part on optimized backward reshaping operating parameters (referred to as "dynamic backward function optimization") generated from the "BB2" reshaping optimization design / solution. The reshaped HDR image—generated with the optimized backward reshaping operating parameters in the output HDR color space—represents an approximate or reconstructed version of the reference HDR image that can or could be generated by a second, different capture device.

[0077] The "BB2" reshaping optimization design / solution can generate optimized backward reshaping operation parameters to cover as wide a range as possible in the output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0078] In some operating scenarios, the rear reshaping operating parameters generated from the "BB2" reshaping optimization design / solution may be applied in a static or dynamic SLiDM framework.

[0079] Because the "BB2" reshaping optimization is used only in the backward path, the resulting appearance of a reshaped HDR image generated from backward reshaping a reference SDR image may have a relatively large deviation from the desired appearance of the reference HDR image. The first and second capture devices may be mobile devices that can operate differently in their respective video capture applications, for example, using different exposure times, different video processing operations, different mapping functions / relationships, etc.

[0080] Dynamic mapping using image-dependent or image-specific reshaping operating parameters can be used to back-reshape an SDR image of a first capture device into an HDR image that can or should be produced by a second capture device. This is particularly true in operational scenarios where the first and second capture devices operate with relatively large differences between the respective video / image capture applications running on the first and second capture devices and / or the respective ISPs used in the first and second capture devices. For example, during a training phase, multiple sets of training SDR and HDR images or image pairs can be used to derive multiple sets of optimized reshaping operating parameters for each of the multiple sets of training SDR and HDR images or image pairs. During a deployment or application phase, specific image characteristics, such as the overall brightness of some or all regions of a particular image, can be dynamically evaluated as the particular image is being processed. A specific set of optimized reshaping operating parameters can be adaptively and / or dynamically selected for the particular image from among the multiple sets of optimized reshaping operating parameters. The selection is based on particular image characteristics of that particular image in relation to or in comparison with image characteristics of each of a plurality of sets of training SDR and HDR images or image pairs.

[0081] White-box joint TPB forward and backward optimization (WFB) Many design factors can be considered for reshaping optimization to help expand the coverage of supported color spaces in which reshaped images are represented. First, a tensor-product bi-spline (TPB) predictor with optimized TPB coefficients generated from the reshaping optimization can be used to obtain or achieve relatively higher prediction accuracy than other types of predictors, including but not limited to MMR predictors. The multi-knot and continuity properties of a tensor-product bi-spline predictor or prediction function can be exploited or used to cover a relatively wide color range or portion of the color space with relatively high accuracy. An exemplary TPB predictor can be found in U.S. Provisional Patent Application No. 62 / 908,770, "Tensor-product B-spline predictor," filed October 1, 2019, by Guan-Ming Su, Harshad Kadu, Qing Song, and Neeraj J. Gadgil, the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0082] In comparison, while a multi-piece MMR predictor or prediction function may be superior to a single-piece MMR predictor or prediction function, the multi-piece MMR predictor or prediction function is likely to introduce discontinuities between different MMR pieces, which may adversely affect or even prohibit the use of the multi-piece MMR predictor or prediction function in many operating scenarios. Exemplary MMR-based operations are described in U.S. Patent No. 8,811,490, which is incorporated herein by reference in its entirety.

[0083] A TPB predictor can be advantageously used in an SLBC-based video codec. More specifically, a forward TPB predictor can be used to generate a forward reshaped SDR image that approximates a reference SDR image, and a backward TPB predictor can be used to generate a backward reshaped or reconstructed HDR image that approximates a reference HDR image. An exemplary use of a TPB predictor with an SLBC-based video codec can be found in U.S. Provisional Patent Application No. 63 / 255,057, entitled "Tensor-product B-spline prediction for HDR video in mobile applications," by H. Kadu et al., filed October 13, 2021, the entire contents of which are incorporated herein by reference as if fully set forth herein. Furthermore, a BESA (Backward Error Subtraction for signal Adjustment) algorithm / method with modifications to support neutral color preservation can be used in a pipeline of chained reshaping functions to jointly optimize the forward and backward TPB predictors to achieve relatively high reversibility or a relatively accurate approximation to the reference HDR image by the backward reshaped or reconstructed HDR image. Exemplary BESA algorithms / methods can be found in U.S. Provisional Patent Application No. 63 / 013,063, "Reshaping functions for HDR imaging with continuity and reversibility constraints," by GM Su, filed April 21, 2020; U.S. Provisional Patent Application No. 63 / 013,807, "Iterative optimization of reshaping functions in single-layer HDR image codec," by GM Su and H. Kadu, filed April 22, 2020; and PCT Application No. PCT / US2021 / 028475, filed April 21, 2021, the entire contents of which are incorporated by reference herein as if fully set forth herein.

[0084] Although a TPB predictor has better prediction accuracy than other types of predictors, it can incur a relatively large signal or bitrate overhead for transmitting the TPB coefficients in a video signal, and it can also incur a relatively large computational cost for constructing the TPB equations or basis functions used in the TPB predictor.

[0085] In some operating scenarios, to avoid or reduce signal or bitrate overhead and computational costs, a built-in or static TPB predictor may be used to reshape some or all images in an entire video sequence without changing the TPB coefficients used in the static TPB predictor during playback of the video sequence. More specifically, some or all of the (built-in or static) TPB coefficients may be cached or stored within a video application, such as a video capture / editing application, without the need to explicitly transmit these TPB coefficients through or within the encoded video signal or bitstream for the images to be reshaped by the built-in or static TPB predictor. In one example, some or all of the TPB coefficients may be pre-loaded or pre-configured in a downstream recipient device before the video signal or bitstream is received and processed by the downstream recipient device. Additionally, optionally, or alternatively, multiple sets of TPB coefficients may be pre-loaded or pre-configured in a downstream recipient device before the video signal or bitstream is received and processed by the downstream recipient device. A video application may simply pick or select a particular set of TPB coefficients for use with a static TPB predictor from among multiple sets of TPB coefficients based, for example, on a simple indicator (e.g., a binary indicator, a multi-bit indicator, etc.) signaled or transmitted in a video signal or bitstream. Exemplary static TPB predictors or prediction functions can be found in the aforementioned U.S. Provisional Patent Application No. 63 / 255,057.

[0086] In some operating scenarios, a mobile device may be used to host and execute video capture and / or editing applications. Users of the mobile device may edit captured images in these video capture and / or editing applications. An edited image, such as an edited HDR image, which may be intended to be displayed or viewed, for example, on an image display of a non-mobile device having higher or greater display capabilities than the image display of the mobile device, may have a higher, larger, broader, and / or wider luminance range and / or color range or color distribution than an (original) captured image, such as an HDR image originally captured from the mobile device's camera sensor. For example, a captured HDR image is often limited by the mobile device's camera sensor and ISP output, such that it is represented in a relatively narrow color space or gamut, such as P3, whereas an edited HDR image may be represented in a relatively wide color space or gamut, such as the entire R.2020 color space.

[0087] To design a static mapping for a reshaping operation, it is highly desirable to optimize the forward / backward TPB coefficients to cover the highest possible dynamic range and the widest possible color space or gamut, while also achieving the highest possible bitrate and computational efficiency.

[0088] Full HDR reversibility between a reconstructed HDR image and a reference HDR image in which the reconstructed HDR is identical requires that colors in the original or reconstructed HDR domain (or color space) be identified or made identifiable in the reshaped SDR domain (or color space) so that identified colors in the SDR domain can be mapped back to distinguishable colors in the HDR domain. Thus, full reversibility likely requires a one-to-one mapping from HDR to SDR and a mapping from SDR back to HDR. The wider the HDR domain or color space to be supported, the more codewords the SDR domain or color space must have. Given that the total number of available SDR pixel or codeword values ​​in the SDR domain (e.g., corresponding to a lower bit depth than the HDR domain) is typically smaller than the total number of required HDR pixel or codeword values ​​in the HDR domain, full HDR reversibility may not be possible in some operating scenarios. In fact, some captured or edited images may contain combinations of diffuse and / or specular colors that form color distributions such as those shown as irregular shapes in FIG. 2A that exceed the R.2020 color space (“BT.2020”), let alone smaller or SDR color spaces such as the R.709 color space (“BT.709”) or the (DCI)P3 color space, and therefore are not fully represented by them.

[0089] In many operating scenarios, a nonlinear reshaping mapping or function can be used to relatively efficiently distribute the available pixel or codeword values ​​to generate a (forward) reshaped SDR that approximates the reference SDR image as closely as possible and helps the reconstructed HDR image approximate the reference HDR image as closely as possible. However, the reshaped SDR generated using the nonlinear reshaping mapping or function may not be identical to the reference SDR image, but rather may contain some deviations from the reference SDR image.

[0090] Additionally, optionally, or alternatively, in some operating scenarios, in order to maintain the same or similar appearance of the reference SDR and HDR images in the reshaped SDR and HDR images, the reshaping optimization described herein can be performed using a neutral color constraint that preserves neutral colors in the color mapping performed in the reshaping operation.

[0091] Given its ability to approximate functions with relatively high nonlinearity, the TPB predictor or function can be incorporated into the reshaping operation to support or achieve a relatively wide HDR color space for representing the backward reshaped or reconstructed HDR image.

[0092] A potential downside is that the TPB predictor may require a relatively large number of TPB coefficients to represent or approximate the nonlinear SDR-HDR and / or HDR-SDR mapping function. To prevent ill-defined conditions, numerical instability, slow convergence issues, etc. that may arise when solving the optimization problem for the TPB coefficients, a static TPB predictor may be pre-generated and deployed with video codecs such as those used by upstream and / or downstream devices (before these video codecs are used to process video sequences, such as captured and / or edited video sequences).

[0093] For illustrative purposes only, in the operating scenario shown in FIG. 1A, the (forward reshaped) SDR domain or color space may be that of R.709, while the (original or backward reshaped) HDR domain or color space may be that of R.2020. As shown in FIG. 2A, the R.2020 color space is much larger than the R.709 color space. Thus, although it is difficult to map the entire R.2020 to R.709 and then map R.709 back to the original R.2020, the reshaping optimization techniques described herein can be used to optimize the coverage of the HDR color space for representing the backward reshaped or reconstructed HDR image while maintaining the same or similar appearance of the reference SDR and HDR images in the reshaped SDR and HDR images. These techniques can be used to concurrently optimize reshaping operation parameters, such as TPB coefficients, for both forward and backward reshaping operations.

[0094] While it may not be possible to cover the entire R.2020 color space in all scenarios, a subset or subspace in the R.2020 color space (e.g., a particular color gamut depicted as a triangle formed by the primary colors in a color coordinate system) can be selected or chosen by a reshaping optimization operation to maintain HDR reversibility to the reference HDR image and acceptable SDR approximation of the reference SDR image in the reshaped HDR and SDR images.

[0095] By way of example and not limitation, a subset or subspace in the R.2020 color space (or an HDR color space for representing a reshaped HDR image) can be defined or characterized by a specific white point and three specific primary colors (red, green, and blue). The specific white point may be selected or fixed to be the D65 white point. For a subset or subspace in R.2020 supported by a reshaping operation, there may be significant design freedom to select three specific primary colors from many possible primary color combinations. The reshaping optimization techniques described herein can be used to determine or select specific primary colors for a subset or subspace in R.2020 to be optimized primaries for achieving or reaching maximum perceptual quality and / or color coding efficiency and / or reversibility between SDR and HDR images.

[0096] The Macadam ellipse represents the most perceptible color difference, in that the HVS may not be able to distinguish color differences within the same ellipse. The Pointer's gamut can represent all diffuse colors that can be perceived by the HVS. In the Macadam ellipse and Pointer's gamut, green is less important than red and blue and is less distinguishable / perceived by the HVS. Thus, in some operating scenarios, specific optimized primaries for a subset or subspace within R.2020 to represent a reshaped or reconstructed HDR image can be selected to cover more red and blue than green, especially when a one-to-one mapping relationship cannot be supported by the SDR-HDR and HDR-SDR mapping in the reshaping operation.

[0097] As used herein, primaries, sometimes referred to as primary colors, may be used to define the vertices of a polygon, such as a triangle, that represents a particular color space or gamut, such as a color space specified by a standard, a color space supported by a display, or a color space supported by a video signal. For example, a standard color space with a standard-specified white point may be represented in the CIExy color space coordinate system or CIExy chromaticity diagram by a triangle whose vertices are specified by the three standard-specified primaries. The CIExy coordinates of the primaries (red or R, green or G, blue or B) and white points that define the R.709, P3, and R.2020 color spaces, respectively, are specified in Table 1 below. [Table 1]

[0098] The CIExy coordinates of the primary colors and corresponding white points of each standard color space in Table 1 are (R x (c) ,R y (c) ),(G x (c) ,G y (c) ),(B x (c) ,B y (c) ) and (W x (c) ,W y (c) ) and (c) is the standard color space.

[0099] Thus, the P3 color space may be characterized by the CIExy coordinates of the P3 primaries and the P3 white point as follows: x (P3) ,R y (P3) ),(G x (P3) ,G y (P3) ),(B x (P3) ,B y (P3) ),(W x (P3) ,Wy (P3) Similarly, the R.2020 color space may be characterized by the CIExy coordinates of the R.2020 primaries and the R.2020 white point as follows: x (R2020) ,R y (R2020) ),(G x (R2020) ,G y (R2020) ),(B x (R2020) ,B y (R2020) ),(W x (R2020) ,W y (R2020) ).

[0100] The backward reshaped HDR color space for representing the backward reshaped or reconstructed HDR image is represented as (a) color space. Thus, the CIExy coordinates of the primary colors and white point that define the (a) color space are (R x (a) ,R y (a) ),(G x (a) ,G y (a) ),(B x (a) ,B y (a) ) and (W x (a) ,W y (a) ) may also be written.

[0101] As mentioned above, green is less important or less distinguishable / perceived by the HVS than red and blue. In some operating scenarios, the red and blue primaries of the (a) color space may be chosen to match the primaries of the R.2020 color space, as follows: R x (a) =R x (R2020) (1-1) R y (a) =R y (R2020) (1-2) B x (a) =B x (R2020) (2-1) B y (a) =B y (R2020) (2-2)

[0102] Furthermore, like all R.709, P3, and R.2020 color spaces that are specified using the D65 white point, the (a) color space can also be specified with the D65 white point as follows: (W x (a) ,W y (a) )=(0.3127 0.3290) (3)

[0103] The green primary color of this (a) color space is the green primary color (G x (P3) ,G y (P3) ) and the green primary (G x (R2020) ,G y (R2020) ) can be selected along the line between the two green primaries in the P3 and R.2020 color spaces. Any point along the line between the two green primaries in the P3 and R.2020 color spaces can be expressed as a linear combination of these two green primaries with a weighting factor a as follows: G x (a) =aG x (P3) +(1-a)G x (R2020) (4-1) G y (a) =aG y (P3) +(1-a)G y (R2020) (4-2)

[0104] Thus, the optimization problem of finding maximum support from the (a) color space for the R.2020 color space can be simplified to the problem of selecting a weighting factor a. When a = 0, the (a) color space is the entire R.2020 color space. When a = 1, as shown in FIG. 2B (where the (a) color space is denoted as "TPB" or "Current TPB Cover Colors"), the red and blue vertices or red and blue primaries of the (a) color space are the same as the red and blue vertices or red and blue primaries of the R.2020 color space, while the green vertex or green primary of the (a) color space is the same as the green vertex or green primary of the P3 color space. Figures 2C through 2E show three exemplary (a) color spaces for a = 0.9, 0.5, and 0.25, respectively.

[0105] Congruent color spaces and TPB optimization Because the range or coverage by the backward-reshaped (a) color space on the R.2020 color space can be controlled by the parameter a, the overall TPB (based reshaping) optimization problem becomes how to optimize the TPB coefficients in both the forward and backward paths so that (i) the forward-reshaped SDR domain (or color space) for representing the reshaped SDR image is close to the reference SDR domain (or color space) for representing the reference SDR image (especially in the neutral color or color space portions that are more sensitive to HVS than the non-neutral color or color space portions), and (ii) the backward-reshaped or reconstructed HDR domain (or color space) for representing the backward-reshaped or reconstructed HDR image is as closely identical as possible to the reference HDR domain (or color space) for representing the reference HDR image. Ideally, if perfect reconstruction or perfect reversibility can be achieved, the backward-reshaped or reconstructed HDR image is identical to the reference HDR image.

[0106] As shown in Figures 2B to 2E, in order to cover as much of the R.2020 color space as possible with the backward-reshaped or reconstructed HDR domain or color space, the TPB optimization problem reduces to finding or searching for the smallest possible value for the parameter a such that the above SDR and HDR quality conditions are met.

[0107] 3A illustrates an exemplary process flow for finding the smallest possible value of parameter a to maximize SDR and HDR quality in reshaped SDR and HDR images. The process flow of FIG. 3A can iterate through multiple candidate values ​​for parameter a in an iterative order, which can be sequential or non-sequential.

[0108] Block 302 involves selecting a current (e.g., to be iterated) value for parameter a as the next candidate value (e.g., initially the first candidate value) in a plurality of candidate values ​​for parameter a. Given the current value of parameter a along with the candidate backward-reshaped HDR color space, optimized reshaping operational parameters, such as optimized TPB coefficients, are obtained in one or more subsequent process flow blocks of FIG. 3A.

[0109] Block 304 includes constructing sample points in a candidate backward reshaping HDR color space or preparing two sampled data sets. By way of example and not limitation, the candidate backward reshaping HDR color space may be, but is not limited to, a Hybrid Log Gamma (HLG) RGB color space (referred to as "(a) RGB color space HLG").

[0110] The first of the two sampled data sets is a uniformly sampled data set of color patches. Each color patch in the uniformly sampled data set of color patches includes three dimensional directions, denoted as R-axis, G-axis, and B-axis for the R, G, and B component colors, respectively. (a) Each RGB color uniformly sampled from the RGB color space HLG

number

[0111] Each uniformly sampled data point or RGB color in a uniformly sampled dataset

number

number

[0112] For simplicity, (i,j,k) in the above equation (5) can be vectorized or simply denoted as p. Correspondingly, the uniformly sampled data points or RGB colors

number

number

number

[0113] The second of the two sampled data sets prepared or constructed in block 304 is a neutral color data set. This second data set includes a plurality of neutral colors or neutral color patches (also called gray colors or gray color patches).

[0114] The second data set may be used to store input gray color patches in the input domain as output gray color patches in the output domain when the input gray color patches in the input domain are mapped or reshaped to output gray color patches in a reshaping operation as described herein. The input gray color patches in the input domain (or input color space) may be given increased weighting in the optimization problem compared to other color patches to reduce the likelihood that the reshaping operation will map these input gray color patches to non-gray color patches in the output domain (or output color space).

[0115] The second data set (gray color data set or gray color data set) may be prepared or constructed by uniformly sampling R, G, B values ​​along a line connecting the first gray color (0,0,0) and the second gray color (1,1,1) in the RGB domain (e.g., (a) RGB color space HLG, etc.), as follows: n Create nodes or gray patches.

number

[0116] All N in the second data set n The nodes can be grouped or organized into a neutral color vector / matrix as follows:

number

[0117] The neutral color vector / matrix in equation (8) above is N t(a positive integer greater than or equal to 1) times, and n N t Neutral color patches can be generated.

number

[0118] The repetition of neutral colors in the second data set increases the weighting of neutral or gray colors relative to other colors, so that neutral colors can be preserved more in the optimization problem than other colors.

[0119] The first data set of (all sampled) colors and the second (repeated) data set of neutral colors in equations (6) and (9) can be grouped or arranged together into a single combined vector / matrix as follows:

number

[0120] The total number of vector / matrix elements (repeated and non-repeated color patches) in the combined vector / matrix (Equation (10)) is N = N n N t +N u Each vector / matrix element or color patch (row) in the above equation (10) may be expressed as:

number

[0121] Block 306 involves converting the color values ​​of the color patches (rows) represented in the vector / matrix elements of the combined vector / matrix (Equation (10)) in (a) RGB color space (or (a) RGB color space HLG) to corresponding color values ​​in the standard-based R.2020 color space or R.2020 RGB color space HLG.

[0122] The three red, green, and blue primary colors in (a) RGB color space HLG, which correspond to the three vertices or points of the triangle defining (a) RGB color space HLG in the CIExy chromaticity diagram, can be converted from CIExy values ​​to CIE XYZ values ​​by the following formulas:

number

[0123] as red, green, and blue primaries in (x,y) or CIExy values ​​(R x (a) ,R y (a) ),(G x (a) ,G y (a) ),(B x (a) ,B y (a) ), then (R X (a) ,R Y (a) ,R Z (a) ),(G X (a) ,G Y (a) ,G Z (a) ),(B X (a) ,B Y (a) ,B Z (a) ) can be obtained using the conversion formula (12) above. Similarly, (W x (a) ,W y (a) ) as the white point in (x,y) or CIE xy values, then (W X (a) ,W Y (a) ,W Z (a)) can be found using the same transformation formula in equation (12) above.

[0124] To convert the color values ​​of the color patches (rows) represented by the vector / matrix elements of the combined vector / matrix (Equation (10)) in (x,y) or CIE xy values ​​to the corresponding (X,Y,Z) or CIE XYZ values, P (a)→XYZ We can construct a 3x3 transformation matrix denoted as

number

number

[0125] a In order to convert the color values ​​of the color patches (rows) represented by the vector / matrix elements of the combined vector / matrix (Equation (10)) in the RGB color space HLG into the corresponding color values ​​in the R.2020 RGB color space HLG, P (a)→R2020 We can construct a 3x3 transformation matrix denoted as

number

[0126] Thus, in block 306, the color values ​​of the color patches (rows) represented by the vector / matrix elements of the combined vector / matrix (Equation (10)) in a RGB color space (or (a) RGB color space HLG) can be converted to corresponding color values ​​in the standard-based R.2020 color space or R.2020 RGB color space HLG as follows:

number

[0127] Block 308 converts the color values ​​of the color patches (rows) represented by the vector / matrix elements of the combined vector / matrix (Equation (18)) in the R.2020 RGB color space HLG into corresponding color values ​​in the R.2020 YCbCr color space HLG as follows:

number

number

[0128]

number

number

[0129] Block 310 converts the color values ​​of the color patches (rows) represented in the vector / matrix elements of the vector / matrix (Equation (18)) in the R.2020 RGB color space HLG into corresponding color values ​​in the R.709 SDR YCbCr color space.

number

number

number

[0130]

number

[0131] Block 312 converts the color values ​​of the color patches (rows) represented in the vector / matrix elements of the vector / matrix (Equation (18)) in the R.2020 RGB color space HLG into corresponding color values ​​in the R.2020 YCbCr color space PQ as follows:

number

number

number

[0132]

number

[0133] Block 314 is

number

[0134] The BESA algorithm is a sequential iterative algorithm in which each current iteration is able to modify the reference SDR signal according to the backward prediction error measured or determined in the previous iteration.

[0135] The modified BESA algorithm as described herein can implement neutral color preservation and avoid modifying neutral color patches. To do this, a set of neutral color patch indices corresponding to the neutral color patches may be generated as follows:

number

[0136] Let ch denote channels (e.g., three channels) in the forward-reshaped SDR domain or color space. Let F denote forward reshaping or forward path. Let B denote backward reshaping or backward path.

[0137] S F ch The forward reshaped SDR color patches characterized by the corresponding forward reshaped SDR codewords can be predicted from the input HDR color patches characterized by the corresponding input HDR codewords derived using equation (19) using a per-channel forward generator matrix (also called a design matrix) denoted as

number

number

number

number

[0138] The forward reshaped SDR color patches can be predicted by multiplying the forward generator matrix of equation (28) by the forward TPB coefficients. Examples of exemplary predicted reshaped codewords (equivalent or similar to color patches herein) using TPB coefficients are described in the aforementioned U.S. Provisional Patent Application No. 62 / 908,770.

[0139] Forward-reshaped SDR color patches or codewords can be backward-reshaped into backward-reshaped HDR color patches or codewords, for example, through a TPB-based backward-reshaping operation.

[0140] In the backward pass, at each iteration of the BESA algorithm (e.g., the kth iteration), S B ch,(k) Using a per-channel backward generator matrix denoted as , a backward reshaped HDR color patch characterized by a corresponding backward reshaped HDR codeword can be predicted from a forward reshaped SDR color patch characterized by a corresponding forward reshaped SDR codeword derived in the forward pass. The backward generator matrix for each iteration is calculated using the cross-channel forward TPB basis functions

number

number

[0141] The backward reshaped HDR color patches can be predicted by multiplying the backward generator matrix in equation (29) by the backward TPB coefficients.

[0142] The backward prediction error for each iteration in the BESA algorithm can be determined by comparing the backward reshaped HDR color patches or codewords collected in a vector / matrix with the reference HDR color patches or codewords in the per-channel backward look vector / matrix derived from Equation (25). The per-channel backward look vector / matrix can be generated or pre-computed from Equation (25), stored or cached in computer memory, and fixed for all iterations, as follows:

number

[0143] A reference SDR signal or a reference SDR color patch or codeword may be used as a prediction target for a forward reshaping operation to generate a forward reshaped SDR color patch or codeword to approximate the reference SDR color patch or codeword. In the BESA algorithm, the reference SDR color patch or codeword

number

[0144] Using the reference SDR color patch or codeword for each channel at iteration step k, we calculate the vector / matrix r as follows: F ch,(k) can be constructed.

number

[0145] Before the first iteration step, the vector in equation (31) above is set to the original reference SDR signal represented by equations (22) and (23) as follows:

number

[0146] At iteration step k, the forward TPB coefficients (m F ch,(k) ) can be generated via a least-squares solution to an optimization problem that minimizes the difference between the forward reshaped SDR color patch or codeword and the reference SDR color patch or codeword determined for that iteration, as follows:

number

[0147] The predicted SDR color patch or codeword in iteration step k for channel ch can be calculated from the optimized values ​​for the forward TPB coefficients (e.g., per channel) as follows:

number

[0148] At iteration step k, the backward TPB coefficients (m B ch,(k)) can be generated via a least-squares solution to an optimization problem that minimizes the difference between the backward-reshaped HDR color patch or codeword and the reference HDR color patch or codeword (the latter is fixed for all iterations in the BESA algorithm).

number

[0149] The predicted HDR color patch or codeword in iteration step k for channel ch can be calculated from the optimized values ​​for the backward TPB coefficients (e.g., per channel) as follows:

number

[0150] Calculate the backward prediction error and back-propagate the error to the reference non-neutral SDR signal.

[0151] As mentioned above, the backward prediction error for each iteration in the BESA algorithm can be determined by comparing the backward reshaped HDR color patch or codeword with the reference HDR color patch or codeword derived from equation (25) (the latter is fixed for all iterations in the BESA algorithm).

[0152] Among the backward prediction errors, the backward prediction errors for non-neutral colors (or non-gray colors) can be backpropagated to update or modify the reference SDR color patches or code words for these non-neutral colors (or non-gray colors). The modified SDR color patches or code words for non-neutral colors (or non-gray colors) can be combined with the (unmodified) reference SDR color patches or code words for neutral colors (or gray colors) to serve as prediction targets for the forward pass in the next or upcoming iteration in the BESA algorithm.

[0153] In some operating scenarios, the backward prediction error can be calculated as the difference between the original or reference HDR signal (or reference HDR color patches / codewords therein) and the predicted HDR signal (or backward reshaped HDR color patches / codewords therein) at iteration step k, as follows:

number

[0154] In response to determining that color patch i is within the neutral color set Φ, the reference SDR codeword for the color patch in the Cb and Cr channels of the reference SDR signal for the next iteration step (k+1) can be set to a gray color value such as 0.5, as follows:

number

[0155] Instead, in response to determining that color patch i is not within the neutral color set Φ, the reference SDR codeword for the color patch in the Cb and Cr channels of the reference SDR signal for the next iteration step (k+1) can be set to an updated or modified value according to the backward prediction error, as follows:

number

[0156] The neutral color preservation implemented using equations (38) through (40) can be used to prevent neutral colors in the reference SDR signal from deviating to or toward non-neutral colors for every iteration in the BESA algorithm, resulting in improved perceptual quality related to gray levels in the reshaped image.

[0157] The BESA algorithm may be iterated sequentially up to a total number of iterations. In some operating scenarios, the total number of iterations for the BESA algorithm may be specifically selected to balance color deviations in the reshaped SDR and / or HDR images. For example, different total numbers of iterations in the BESA algorithm may generate forward and backward TPB coefficients that produce reshaped SDR images with different SDR appearances and reshaped HDR images with different HDR appearances. A total number of iterations, such as 10, 15, etc., that produces reshaped SDR and HDR images with relatively high-quality appearances may be selected for the BESA algorithm.

[0158] A BESA algorithm with neutral color preservation may be performed for each candidate value for parameter a. For example, in block 314, a BESA algorithm with neutral color preservation may be performed for the current candidate value of parameter a.

[0159] Block 316 involves determining whether the current candidate value for parameter a is the last candidate value of multiple candidate values ​​for parameter a. If so, process flow proceeds to block 318. If not, process flow returns to block 302.

[0160] Block 318 involves selecting an optimal or optimized value for parameter a and calculating (or simply selecting those already calculated) optimized forward and backward TPB coefficients for (a) RGB color space that correspond to the optimized value for parameter a.

[0161] The optimal value selection for the parameter a is {a,m F,a ch ,m B,a ch This can be formulated as an optimization problem to find (a particular set of values ​​for)}. The optimization is as follows: (1) (a) the RGB color space represents the largest HDR color space (e.g., covers the largest portion of the R.2020 RGB color space);

number

number

number

[0162] This optimization problem involves converting the results from each a RGB color space into its corresponding optimal or optimized TPB coefficients m F,a ch and m B,a ch Among all possible or candidate a values, the smallest a (corresponding to the largest color space coverage in the R.2020 color space) that achieves minimized HDR error with a relatively small SDR error.

number

number

[0163] 2F and 2G show example percentile Cb and Cr values ​​in the forward reshaped SDR domain or color space for different values ​​of parameter a. Different values ​​of parameter a determine different color space coverage of the backward reshaped HDR domain or color space in the R.2020 domain or color space, and thus affect different reconstructions or generation of the backward reshaped HDR image in the R.2020 domain or color space, resulting in different distortions of the backward reshaped HDR image compared to the reference HDR image to which the backward reshaped HDR image approximates—e.g., distortion metrics

number

[0164] The vertical axis of FIG. 2F represents the 3 percentile values ​​in the Cb and Cr channels in the forward-reshaped SDR domain for different values ​​for the parameter a value represented by the horizontal axis of FIG. 2F (e.g., obtained using the BESA algorithm with neutral color preservation for 10 successive iterations). The smaller the value for parameter a, the greater the range covered by the a color space in the R.2020 color space. The 3 percentile value may be used explicitly or implicitly to represent the minimum value in the Cb and Cr channels of the forward-reshaped SDR (e.g., YCbCr, etc.) domain or color space. The smaller the 3 percentile value, the smaller the minimum value and, therefore, the more extended the codeword value range in the forward-reshaped SDR (YCbCr, etc.) domain or color space. As can be seen from FIG. 2F, the lower the value for parameter a, the lower the 3 percentile value in the Cb and Cr channels of the forward-reshaped SDR (e.g., YCbCr, etc.) domain or color space.

[0165] The vertical axis of FIG. 2G represents the 97th percentile values ​​in the Cb and Cr channels in the forward-reshaped SDR domain for different values ​​for the parameter a represented by the horizontal axis of FIG. 2G (e.g., obtained using the BESA algorithm with neutral color preservation for 10 successive iterations). The 97th percentile values ​​may be used explicitly or implicitly to represent the maximum values ​​in the Cb and Cr channels of the forward-reshaped SDR (e.g., YCbCr) domain or color space. The larger the 97th percentile value, the larger the maximum value and thus the more extended the codeword value range in the forward-reshaped SDR (YCbCr) domain or color space. As can be seen from FIG. 2G, the 97th percentile values ​​in both the Cb and Cr channels of the forward-reshaped SDR domain or color space remain approximately constant.

[0166] In some operating scenarios, the image dataset used in block 318 to determine distortion in the reshaped SDR and HDR images and to select an optimized value for parameter a based in part or in whole on the distortion may include a video bitstream acquired using one or more specific video capture devices, such as a mobile phone (e.g., supporting the SMPTE ST 2094 video coding profile, supporting Dolby Vision Profile 8.4, etc.). A non-limiting example of an optimized value for parameter a may be, but is not necessarily limited to, 0.5. This determines the maximum color space coverage in the R.2020 color space for the backward-reshaped HDR domain or color space. If a wider backward-reshaped color space (e.g., corresponding to a value smaller than the optimized value for parameter a) is used or supported, the reshaped SDR colors and codewords may begin to deviate from the reference SDR colors and codewords with relatively large distortion. This is because increasing the backward-reshaped HDR domain or color space means packing distinct forward-reshaped SDR colors (or codewords) more tightly into the forward-reshaped SDR domain or color space, leading to the forward-reshaped SDR colors being displaced from their original 3D positions represented in the reference SDR colors (or codewords) that are approximated by the forward-reshaped SDR colors.

[0167] As mentioned above, forward and backward TPB coefficients for a TPB-based reshaping operation can be determined for a given value, such as an optimized value for parameter a. These TPB coefficients can be multiplied in a formula with a forward or backward generator matrix constructed with TPB basis functions using the input codeword as an input parameter to generate a forward or reshaped codeword, which can involve performing numerous calculations involved in calculating TPB basis function values ​​for each pixel of a large number of pixels of an image.

[0168] In some operating scenarios, to speed up or reduce computations, forward and backward 3D-LUTs may be constructed from pre-computed TPB basis function values ​​from sampled values ​​and optimized forward and backward TPB coefficients generated for an optimized value for parameter a. The forward and backward 3D-LUTs, or lookup nodes / entries therein, may be pre-constructed before being deployed at runtime to process an input image, and applied at runtime with relatively simple lookup operations in the forward and backward paths, or corresponding forward and backward reshaping operations performed therein, relative to the input image.

[0169] The optimized value for parameter a is opt The corresponding optimal forward and backward TPB coefficients are denoted as m F ch,opt and m B ch,opt It is written as follows.

[0170] The forward 3D-LUT can be used to forward reshape input HDR colors (or codewords) in an input HDR domain or color space, such as the R.2020 domain or color space (which includes the a color space), to forward reshaped SDR colors (or codewords) in a forward reshaped SDR domain or color space.

[0171] A color space identified within an input HDR domain or color space, such as the R2020 (container) domain or color space, may be used to clip out HDR colors or codewords represented outside the color space. Forward TPB coefficients can be applied to input HDR colors or codewords in a color space identified within the input HDR domain or color space, such as the R2020 (container) domain or color space, to generate a predicted or forward-reshaped SDR color or codeword for each lookup node or entry in the forward 3D-LUT. As a result, the forward 3D-LUT contains multiple lookup nodes or entries, each of which maps or forward-reshapes a respective input (cross-channel or three-channel) HDR color or codeword to a corresponding predicted or forward-reshaped (cross-channel or three-channel) SDR color or codeword.

[0172] In some operating scenarios, the forward 3D-LUT construction process involves a first step in which a 3D uniform sampling grid is prepared in the input HDR domain or color space, such as the R2020 (container) domain or color space.

[0173] By way of example and not limitation, the input HDR domain or color space is the R.2020 YCbCr color space HLG, which includes three dimensions or channels: Y axis, Cb axis, and Cr axis. The input HDR values ​​consist of normalized values ​​in the value range [0,1] for each axis.

number

number

[0174] Although the valid values ​​in the entire R.2020 YCbCr color space may be limited (so as not to cover the entire 3D cube of YCbCr values), the entire sampling value grid may be used to cover the entire 3D cube of YCbCr values ​​in order to reduce the chance of codeword deviation caused by the compression operation.

[0175] For simplicity, (i,j,k) is vectorized or denoted as p. Hence, the sampled values ​​in R.2020 YCbCr are

number

number

number

[0176] The forward 3D-LUT construction process includes a second step in which the sampled values ​​in the R.2020 YCbCr color space HLG can be converted to corresponding values ​​(or colors) in the R.2020 RGB color space HLG as follows:

number

[0177] The forward 3D-LUT construction process is performed by converting the converted values ​​in the R.2020 RGB color space HLG to the optimized values ​​for the parameter a (a optA third step involves converting a corresponding value (or color) in the RGB color space HLG (denoted as a RGB color space).

number

[0178] The forward 3D-LUT construction process includes a fourth step in which (a) the converted values ​​in the RGB color space HLG may be clipped as follows:

number

[0179] The forward 3D-LUT construction process includes a fifth step in which the clipped values ​​in a RGB color space HLG can be converted to corresponding values ​​(or colors) in the R.2020 RGB color space HLG as follows:

number

[0180] The forward 3D-LUT construction process includes a sixth step in which values ​​in the R.2020 RGB color space HLG derived using equation (47) above may be converted to corresponding values ​​(or colors) in the R.2020 YCbCr color space HLG as follows:

number

[0181] The forward 3D-LUT construction process includes a seventh step, where for each lookup node / entry in the forward 3D-LUT, the optimized forward TPB coefficients are added to the forward generator matrix S F chThe forward generator matrix is ​​constructed using the input HDR colors or codewords (or codeword values) in the R.2020 YCbCr color space HLG derived using equation (48) above as input parameters to the forward TPB basis functions. The mapped or forward reshaped SDR colors or codewords (or codeword values) are obtained as follows:

number

[0182] In the above equation (49), the forward generator matrix S F ch is the input HDR color or codeword (or codeword value) V YCbCr (FL),(R2020) as input to the forward TPB basis functions.

number

[0183] The mapped or forward reshaped SDR color or codeword (or codeword value) (Equation (49)) may be used as the lookup value of the lookup node / entry of the forward 3D-LUT. Meanwhile, the input HDR color or codeword (or codeword value) used as the input parameter to the forward TPB basis function may be used as the lookup key of the lookup node / entry of the forward 3D-LUT. At runtime, the mapped or forward reshaped SDR color or codeword can simply be looked up in the forward 3D-LUT using the lookup key as the input HDR color or codeword to be forward reshaped to the mapped or forward reshaped SDR color or codeword.

[0184] A backward 3D-LUT as described above can be used to backward reshape reshaped SDR colors (or codewords) in a forward reshaped SDR domain or color space to backward reshaped HDR colors (or codewords) in a backward reshaped HDR domain or color space, such as the R.2020 domain or color space (which includes the a color space).

[0185] In some operating scenarios, a backward 3D-LUT construction process, which may be simpler than the forward 3D-LUT construction process discussed previously, may be implemented or executed to build or construct the backward 3D-LUT. In some operating scenarios, to speed up or reduce computations, the backward 3D-LUT may be constructed from pre-calculated TPB basis function values ​​from sampled values ​​and optimized backward TPB coefficients generated for an optimized value for parameter a. The backward 3D-LUT or lookup nodes / entries therein may be pre-constructed before being deployed at runtime to process an input image, such as a forward-reshaped SDR image, and at runtime, a relatively simple lookup operation in the backward path or a corresponding backward reshaping operation performed thereon may be applied to the input image.

[0186] The pre-constructed backward 3D-LUT can be deployed at a decoder side, where a receiving downstream device at the decoder side can receive and decode a video signal encoded with a forwardly reshaped SDR image in a forwardly reshaped SDR domain or color space, and can apply backward reshaping to the forwardly reshaped SDR image using the 3D-LUT to generate a backwardly reshaped HDR image in a backwardly reshaped HDR domain or color space, such as a color space included in the R.2020 domain or color space.

[0187] While the shape formed or described by the complete boundary of the forward-reshaped SDR domain or color space mapped in the TPB-based forward reshaping operation may not be a simple 3D cube, the forward-reshaped SDR values ​​in the forward-reshaped SDR domain or color space may be clipped to the tightest or smallest 3D cube (e.g., a 3D rectangle, a rescaled 3D cube from a 3D rectangle, etc.) that contains or supports all the forward-reshaped SDR values ​​represented in all lookup nodes / entries of the forward 3D-LUT, for example, without clipping these SDR values ​​represented in the forward 3D-LUT.

[0188] In some operating scenarios, the backward 3D-LUT construction process includes a first step in which a complete sampling value grid can be prepared in a forward reshaped SDR domain or color space, such as the entire R.709 SDR YCbCr color space.

[0189] The backward 3D-LUT construction process includes a second step in which minimum and maximum values ​​for each dimension or (color) channel in the forward reshaped SDR domain or color space are determined among the forward reshaped SDR values ​​in the forward 3D-LUT and may be used to restrict the (input) data range of the forward reshaped SDR (e.g., YCbCr, etc.) colors or codewords that serve as input for the backward path.

[0190] The backward 3D-LUT construction process involves a third step, where for each lookup node / entry in the backward 3D-LUT, the optimized backward TPB coefficients are added to the backward generator matrix S B ch(e.g., as shown in equation (29) above) which is constructed using forward reshaped SDR colors or codewords (or codeword values) within the clipped or limited (input) data range derived in the second step of the backward 3D-LUT construction process as input parameters to the backward TPB basis functions to obtain mapped or backward reshaped HDR colors or codewords (or codeword values).

[0191] The mapped or backward-reshaped HDR colors or codewords (or codeword values) may be used as lookup values ​​of the lookup nodes / entries of the backward 3D-LUT, while the input or forward-reshaped SDR colors or codewords (or codeword values) used as input parameters to the backward TPB basis functions may be used as lookup keys of the lookup nodes / entries of the backward 3D-LUT. At runtime, the mapped or backward-reshaped HDR colors or codewords can simply be looked up in the backward 3D-LUT using the lookup keys as the input or forward-reshaped SDR colors or codewords that are backward-reshaped to the mapped or backward-reshaped HDR colors or codewords.

[0192] White-box backward optimization (WB) Forward TPB-based reshaping or a corresponding forward 3D-LUT may not be used in all video capture and / or editing devices due to the cost and / or computational overhead involved in implementing or operating with forward TPB-based reshaping or a forward 3D-LUT in video capture and / or editing applications.

[0193] In some operating scenarios, a hardware-based solution, such as one implemented with an available ISP, may be used in a video capture and / or editing device to perform HDR-to-SDR conversion, such as HDR HLG to SDR image generation. An existing ISP pipeline deployed with the device may operate with one or more programmable parameters to generate, convert, and / or output an SDR image based, at least in part or in whole, on a corresponding HDR (e.g., HLG) image captured by the device.

[0194] In the forward path, the programmable parameters for the ISP pipeline can be specifically set or configured to cause the ISP pipeline to output an SDR image that approximates as closely as possible a reference SDR image generated using a white-box HDR-to-SDR conversion function.

[0195] In the backward path, a backward-reshaped HDR image can be generated from the SDR image output from the ISP pipeline using backward TPB-based reshaping or a corresponding backward 3D-LUT. The backward TPB coefficients used in the backward path may, for example, be optimized to cover as much of the backward-reshaped HDR domain or color space, e.g., the R.2020 color space, as possible.

[0196] Figure 3B shows an exemplary process flow for determining or generating optimized values ​​for one or more programmable parameters in an ISP pipeline used to generate or output an SDR image from an HDR image (ISP).

[0197] In some operating scenarios, the ISP pipeline may be a relatively fixed, programmable module implemented in ISP (hardware) to approximate a white-box HDR-to-SDR conversion function, such as a white-box (e.g., known, well-defined, etc.) conversion function for converting an HDR HLG image to a corresponding SDR image in block 330.

[0198] The ISP pipeline may include or implement (1) a first set of three one-dimensional lookup tables (1 D-LUTs) for HLG RGB to linear RGB conversion, (2) a subsequent 3x3 matrix for conversion from an HDR image in an HDR domain or color space, such as the R.2020 color space, to an SDR domain or color space, such as the R.709 color space, and (3) a second set of three 1D-LUTs implementing a subsequent BT.1886 linear to nonlinear (gamma) conversion using a (e.g., standards-based) HLG optical-to-optical transfer function (OOTF).

[0199] For illustrative purposes only, the HDR image may be an HDR HLG image retrieved from an image dataset or database, as shown in Figure 3B. The one or more programmable parameters for the ISP pipeline may be a design parameter (γ BT1886 (denoted as "(

[0200] The process flow of FIG. 3B involves determining the design parameter γ so that the ISP SDR image best approximates the reference SDR image generated in block 330. BT1866 The process flow can be used to find the optimized value for the design parameter γ BT1886 may iterate through multiple candidate values ​​for , in an iteration order that may be sequential or non-sequential.

[0201] Block 322 determines the design parameter γ BT1866The current (e.g., to be iterated) value of the design parameter γ BT1866 selecting a next candidate value (e.g., initially a first candidate value, etc.) in the plurality of candidate values ​​for the design parameter γ BT1866 Given the current values ​​for , the ISP SDR image can be compared to a reference SDR image in one or more subsequent process flow blocks of Figure 3B.

[0202] Block 324 involves applying the first set of 1D-LUTs to the HDR HLG(RGB) image retrieved from the database to generate a corresponding HDR linear RGB image, as follows:

number

[0203] Block 326 includes applying the 3x3 matrix to the corresponding HDR linear RGB image to generate a corresponding SDR linear RGB image. The 3x3 matrix may be given as: P R2020→R709 =P R2020→XYZ) P XYZ→R709 (52)

[0204] The SDR linear RGB codeword in the corresponding SDR linear RGB image generated using the 3 × 3 matrix in equation (52) above is l ch (R709) It may be expressed as:

[0205] Block 328 maps a second set of 1D-LUTs to the SDR linear RGB codewords l in the corresponding SDR linear RGB image. ch (R709) , and the ISP SDR codeword (s ch (R709) This involves generating a

[0206] In some operating scenarios, as previously described, the second set of 1D-LUTs merges or combines the linear to non-linear SDR conversion with the HLG OOTF, as follows:

number

number

[0207] The intermediate SDR codeword (dotted l) in the above equation (53) ch (R709) ) to ISP SDR codeword s ch (R709) The linear to nonlinear SDR transformation used to generate γ may be the linear to γ ​​transformation defined in BT.1886 as follows:

number

[0208] Blocks 322-328 of FIG. 3B—or the combination of the first set of 1D-LUTs represented by equation (51) above, the 3×3 matrix represented by equation (52) above, and the second set of 1D-LUTs represented by equations (53)-(56) above—are implemented using an ISP pipeline to collectively determine the design parameter γ BT1886 The HDR HLG to SDR conversion function (f HLG→ISPSDR Implement the

[0209] Block 332 calculates the HDR HLG to SDR conversion function f HLG→ISPSDR and a reference SDR image generated from a white-box HDR-to-SDR conversion function in block 330. A quality assessment function such as MSE, RMSE, SAD, PSNR, SSIM, etc. may be used to calculate the difference.

[0210] Block 334 determines the design parameter γ BT1886 The current candidate value for the design parameter γ BT1886 If so, process flow proceeds to block 336. If not, process flow returns to block 322.

[0211] Block 336 determines the design parameter γ BT1886 The design parameter γ BT1886 The selection of an optimized value for the design parameter γ among multiple candidate values ​​is performed such that the difference between the ISP SDR image and a reference SDR image calculated using a relatively large image dataset or database is minimized, as described below. BT1886 A specific value of (γ BT1886,opt This can be formulated as an optimization problem to find (denoted as

[0212]

number

[0213] where D() represents the quality assessment function used in block 332.

[0214] gamma BT1886,opt An example value of may be, but is not limited to, 2.115.

[0215] SDR to PQ TPB optimization As described above, in a "WFB" use case or operating scenario, the BESA algorithm may be used under the SLBC framework to generate optimized reshaping operation parameters used by the forward and backward reshaping operations to generate a forwardly reshaped SDR image and a backwardly reshaped HDR image, whereas in a "WB" use case or operating scenario, only backward reshaping may be performed under the SLiDM framework on a non-forwardly reshaped SDR image encoded in an output video signal (e.g., an ISP SDR image generated using an ISP pipeline implemented with a video capture and / or editing device) to generate a corresponding backwardly reshaped or reconstructed HDR image.

[0216] In "WB" use cases or operating scenarios, forward TPB reshaping from an input HDR domain or color space, such as the R.2020 HDR color space HLG, to a forward-reshaped SDR domain or color space, such as the R.709 SDR YCbCr color space, could not be used to help utilize the full range of codewords supported by the R.709 SDR YCbCr color space. ISP SDR codewords in ISP SDR images encoded in an output video signal are often hard-limited in the R.709 color space portion generated or supported by the video capture and / or editing device or the ISP pipeline implemented therein. It may be difficult for ISP SDR codewords in the hard-limited R.709 color space portion to be further expanded or mapped by backward reshaping, such as backward TPB-based reshaping, to a backward-reshaped or reconstructed HDR domain or color space.

[0217] However, as in the "WFB" use cases or operating scenarios, in "WB" use cases or operating scenarios, optimized reshaping operating parameters, such as TPB coefficients, may help to expand the coverage of supported color spaces in which the rear-reshaped HDR images produced in the reshaping optimization are represented. In many "WB" operating scenarios, the rear-reshaped or reconstructed HDR domain or color space achieved with rear-TPB-based reshaping can be only slightly larger than the R.709 color space compared to the largest (a) color space achievable in "WFB" use cases or operating scenarios.

[0218] For illustrative purposes only, in an operating scenario such as that shown in FIG. 1B, the ISP SDR domain or color space may be that of R.709 or a hard-limited portion thereof in the ISP pipeline, while the (original or backward-reshaped) HDR domain or color space may be that of R.2020.

[0219] By way of example and not limitation, a subset or subspace in the R.2020 color space (or an HDR color space for representing a reshaped HDR image) can be defined or characterized by a particular white point and three particular primary colors (red, green, and blue). The particular white point may be selected or fixed to be the D65 white point.

[0220] The backward reshaped HDR color space for representing the backward reshaped or reconstructed HDR image is denoted as (b) color space. Therefore, the CIExy coordinates of the primary colors and white point that define the (b) color space are (R x (b) ,R y (b) ),(G x (b) ,G y (ab) ),(B x (b) ,B y (b) ) and (W x (b) ,W y (b) (b) The color space can be specified as a D65 white point.

[0221] (b) Each of the primary colors in the color space is a corresponding primary color in the P3 color space (G x (P3) ,G y (P3) ) and each primary color (G x (R2020) ,G y (R2020) ) Any point along the line between two respective primaries in the P3 color space and the R.2020 color space can be expressed as a linear combination of these two respective primaries with a weighting factor b as follows:

number

[0222] Thus, the optimization problem of finding the maximum support from the (b) color space for the R.2020 color space can be simplified as a problem of selecting a weighting factor b. When b = 0, the (b) color space is the entire R.2020 color space. When b = 1, the (b) color space is the P3 color space, as shown in Figure 2K (where the (b) color space is denoted as "TPB" or "colors covered by TPB (b color space)"). Figures 2H through 2J show three exemplary (b) color spaces for b = 0.25, 0.50, and 0.75, respectively.

[0223] As shown in Figures 2H to 2K, the TPB optimization problem reduces to finding or searching for the smallest possible value for the parameter b in order to cover as much of the R.2020 color space as possible with the backward-reshaped or reconstructed HDR domain or color space.

[0224] Figure 3C shows an exemplary process flow for finding the minimum possible value for parameter b. The process flow of Figure 3C can iterate through multiple candidate values ​​for parameter b in an iterative order that can be sequential or non-sequential.

[0225] Block 342 includes selecting the current (e.g., iterated, etc.) value for parameter b as the next candidate value (e.g., initially the first candidate value, etc.) among multiple candidate values ​​for parameter b. Given a candidate backward-reshaped HDR color space with the current value for parameter b, optimized reshaping operational parameters, such as optimized TPB coefficients, can be obtained in one or more subsequent process flow blocks of FIG. 3C.

[0226] Block 344 includes constructing sample points or preparing two sampled data sets in a candidate backward-reshaped HDR color space. By way of example and not limitation, the candidate backward-reshaped HDR color space may be, but is not limited to, a hybrid log-gamma (HLG) RGB color space (referred to as "(b) RGB color space HLG").

[0227] Similar to block 304 of Figure 3A, the first of the two sampled data sets is a uniformly sampled data set of color patches. Each color patch in the uniformly sampled data set of color patches includes three dimensions, labeled R, G, and B, for the R, G, and B component colors, respectively. (b) Each RGB color uniformly sampled from the RGB color space HLG.

number

number

[0228] For simplicity, (i,j,k) can be vectorized or simply denoted as p. Correspondingly, the uniformly sampled data points or RGB colors (dotted v ijk (u) ) is simply

number

number

[0229] The second of the two sampled data sets prepared or constructed in block 344 is a neutral color data set. This second data set includes a plurality of neutral colors or neutral color patches (also called gray colors or gray color patches).

[0230] The second data set may be used to preserve input gray color patches in the input domain as output gray color patches in the output domain when the input gray color patches in the input domain are mapped or reshaped to output gray color patches in a reshaping operation as described herein. The input gray color patches in the input domain (or input color space) may be given increased weighting in the optimization problem compared to other color patches to reduce the likelihood that the reshaping operation will map these input gray color patches to non-gray color patches in the output domain (or output color space).

[0231] A second data set—a gray color data set or gray color data set—is prepared or constructed by uniformly sampling R, G, and B values ​​along a line connecting a first gray color (0,0,0) and a second gray color (1,1,1) in the RGB domain (e.g., (b) RGB color space HLG, etc.), as follows: n Produces nodes or gray patches:

number

[0232] N in the second data set n All the nodes can be grouped or organized into a neutral color vector / matrix as follows:

number

[0233] The neutral color vector / matrix in equation (8) above is N t (a positive integer greater than or equal to 1) times, and n N t Neutral color patches can be generated.

number

[0234] The repetition of neutral colors in the second data set increases the weighting of neutral or gray colors relative to other colors, so that neutral colors can be preserved more in the optimization problem than other colors.

[0235] The first data set of (all sampled) colors and the second (repeated) data set of neutral colors in equations (61) and (64) can be collapsed or put together into a single combined vector / matrix as follows:

number

[0236] Combined Vectors / Matrices (V with tilde) RGB (b) The total number of vector / matrix elements (repeated and non-repeated color patches) in n N t +N u The above equation (10) (V RGB (b)Each vector / matrix element or color patch (row) in ( ) may be written as:

number

[0237] Block 346 is a combined vector / matrix (V with tilde) in (b) RGB color space (or (b) RGB color space HLG). RGB (b) ) to corresponding color values ​​in the standard-based R.2020 color space or the R.2020 RGB color space HLG as follows:

number

number

[0238] Block 348 converts vectors / matrices (V with tildes) in the R.2020 RGB color space HLG as follows: RGB (R2020) ) to convert the color values ​​of the color patches (rows) represented in the vector / matrix elements into the corresponding ISP SDR color values ​​in the R.709 SDR RGB color space.

number

number

number

[0239] S with tilde YCbCr (R709) Each color patch (row) in can be represented as follows:

number

[0240] Block 350 is a vector / matrix (V with tilde) in the R.2020 RGB color space HLG as follows: RGB The color values ​​of the color patches (rows) represented in the vector / matrix elements of (R2020) are converted to the corresponding color values ​​in the R.2020 YCbCr color space PQ.

number

number

[0241] R with a tilde YCbCr (R2020) Each color patch (row) in can be represented as follows:

number

[0242] Block 352 is S with a tildeYCbCr (R709) and R with a tilde YCbCr (R2020) as input to a backward TPB optimization algorithm to generate optimized backward TPB coefficients for TPB-based reshaping for the (b) color space that corresponds to the current value for parameter b.

[0243] To do this, we use the TPB basis functions and the S YCbCr (R709) From the SDR codewords represented in {right arrow over (x)}, a backward generator matrix may be constructed as follows:

number

[0244] The backward prediction error can be determined by comparing the backward reshaped HDR color patches or codewords organized in a vector / matrix with the reference HDR color patches or codewords in the per-channel backward look vector / matrix derived and minimized when solving the TPB optimization problem. The per-channel backward look vector / matrix can be generated or pre-computed from equation (71) above, stored or cached in computer memory, and fixed for all iterations, as follows:

number

[0245] Posterior TPB coefficients (e.g., per channel) (m B ch ) can be generated via a least-squares solution to an optimization problem that minimizes the difference between the backward-reshaped HDR color patches or codewords and the reference HDR color patches or codewords, as follows:

number

[0246] The per-channel predicted (or backward-reshaped or reconstructed) HDR codeword for each channel ch can be calculated as follows:

number

[0247] Block 354 involves determining whether the current candidate value for parameter b is the last candidate value among multiple candidate values ​​for parameter b. If so, process flow proceeds to block 356. If not, process flow returns to block 342.

[0248] Block 356 involves selecting an optimal or optimized value for parameter b and calculating (or simply selecting those already calculated) optimized forward and backward TPB coefficients for the (b) RGB color space that correspond to the optimized value for parameter b.

[0249] Similar to the "WFB" use case or operating scenario, in the "WB" use case or operating scenario, the (b) color space is also part of the optimization. The (b) color space and the backward TPB coefficients can be jointly optimized. Thus, the optimization problem is to find the optimal color space that covers the largest HDR color space or color space portion while still minimizing the HDR (r B ch ,^r B ch ) so that the (b) color space is generated to achieve minimized HDR prediction error, denoted as {b,m B,b ch} can be formulated to find (a particular value for).

number

[0250] An exemplary optimized value for the parameter b may be, but is not limited to, 1, which corresponds to the P3 color space.

[0251] Black-box TPB backward optimization on a single device (BB1) Some (dual-mode) video capture devices or mobile devices support both SDR and HDR capture modes and can therefore output either SDR or HDR images. Some (mono-mode) video capture devices or mobile devices support only SDR capture mode. Under the techniques described herein, SDR-HDR mappings can be modeled or generated using a dual-mode video capture device. Some or all of these SDR-HDR mappings can be applied, for example, in a downloaded and / or installed video capture application, to SDR images captured by either a dual-mode video capture device or a mono-mode video capture device to "upconvert" SDR images acquired in SDR capture mode to corresponding HDR images, regardless of whether the video capture device supports HDR capture mode.

[0252] In some operating scenarios, the computational environment or processing power of a mobile device may be limited, so a static (backward) 3D-LUT derived at least in part from the TPB basis functions and optimized TPB coefficients may be used, which represents a static SDR-HDR mapping that maps or backward reshapes SDR images, e.g., all SDR images in an SDR video sequence, to generate corresponding HDR images, e.g., all HDR images in a corresponding HDR video sequence.

[0253] The static SDR-HDR mapping may be modeled at least in part based on multiple image pairs of HDR and SDR images captured by a particular camera of a dual-mode video capture device. Each image pair in the HDR and SDR image pairs includes an SDR image and an HDR image corresponding to the SDR image. The SDR and HDR images depict the same visual scene in the real world and may be subject to spatial alignment errors caused by spatial translation that may occur between a first time point when the SDR image is captured by a particular camera operating in SDR capture mode and a second time point when the HDR image is captured by the same camera operating in HDR capture mode. For example, for each visual scene in the physical world, the video capture device may operate in HDR mode to capture the HDR image in an image pair and the SDR image in the same image pair using SDR mode. This may be repeated to generate multiple image pairs of HDR and SDR images to generate a relatively large number of different color patches (or a relatively large number of different colors or different code words) used to generate the static SDR-HDR mapping.

[0254] Modeling static SDR-HDR mapping using captured SDR and HDR images poses several challenges. First, while both SDR and HDR capture modes can use the same ISP within the same video capture device, the video capture device may apply different image capture settings (e.g., exposure settings for a particular camera) for the same scene to obtain optimized image quality in the captured SDR and HDR images. As a result, static SDR-HDR mapping may be able to model or approximate to some extent the HDR image capture settings actually implemented by the video capture device. Second, the spatial alignment between the captured SDR and HDR images used to derive the SDR and HDR image pairs may not be strict. For example, selection of a specific capture mode between the SDR and HDR capture modes may be performed by touching the screen of the video capture device, which may cause the video capture device to lose or move from its previous spatial position and / or orientation. Furthermore, temporal alignment between a captured SDR video / image sequence and a captured HDR video / image sequence can easily occur because these SDR and HDR sequences are captured at different time instances or durations. Some depicted visual objects in these sequences may move in and out of the camera field of view. Some local areas within the camera field of view may have occlusions or de-occlusions from time to time.

[0255] In some operating scenarios, to resolve spatial and / or temporal alignment issues between a captured SDR image and a corresponding captured HDR image, a registration operation may be performed to generate an aligned SDR image and a corresponding aligned HDR image, which may be included in an SDR-HDR image image pair. The image pair may then be used to determine or establish SDR color patches or colors in the aligned SDR image and corresponding HDR color patches or colors in the corresponding aligned HDR image. These SDR and HDR color patches or colors may be used to derive at least some color patches in a set of corresponding SDR and HDR color patches or colors for the purpose of deriving or generating a static SDR-HDR mapping.

[0256] The SDR and HDR video capture process may be performed to provide comprehensive coverage of different scenes and / or exposures at different times of day, such as early morning to late night, for both indoor and outdoor scenes. For each scene, the same video capture device may be used in both SDR and HDR modes to capture SDR and HDR video sequences. In some operating scenarios, specific frames / images, such as the first frame / image of each of the video sequences, may be selected or extracted for the purpose of constructing or deriving multiple image pairs of SDR and HDR images to be included in a training image dataset or database.

[0257] FIG. 3D illustrates an exemplary process flow for generating optimized reshaping operational parameters, such as optimized or optimal TPB coefficients, using corresponding (or matched) SDR and HDR color patches or colors in a captured SDR image and a captured HDR image (corresponding to the captured SDR image), captured by a video capture device operating in SDR and HDR capture modes, respectively.

[0258] Block 362 includes receiving a captured SDR image and locating or extracting a set of SDR image feature points within the captured SDR image. Block 364 includes receiving a captured HDR image and locating or extracting a set of HDR image feature points within the captured HDR image. The set of SDR image feature points and the set of HDR image feature points may be sets of the same feature point type.

[0259] The set of feature point types may include feature point types across a wide variety of different feature point types, including, but not necessarily limited to, any, some, or all of the following: Binary-Robust-Invariant-Scalable-Keypoints or BRISK features detected with the BRISK algorithm; Features from Accelerated-Segment-Test or corners detected using the FAST algorithm; features detected from the KAZE algorithm; corners detected using the Minimum Eigenvalue algorithm; features generated with the Maximally-Stable-Extremal-Regions or MSER algorithm; keypoints detected with the Oriented-FAST-and-Rotated or ORB algorithm; features extracted from the scale-invariant-feature-transform or SIFT algorithm; Speeded-Up-Robust-Features or features extracted from the SURF algorithm, etc.

[0260] Some or all of the feature points in the set of SDR image feature points and the set of HDR image feature points may each be represented by a feature vector or descriptor, such as an array of feature values ​​(e.g., numerical values).

[0261] Block 366 includes matching some or all of the feature points in the set of SDR image feature points with some or all of the feature points in the set of HDR image feature points. For each feature point of a particular type in the SDR image in the set of SDR image feature points, a matching metric can be calculated between that feature point in the SDR image and each feature point of the same type in the HDR image in the set of HDR image feature points. In a non-limiting example, the matching metric can be calculated as the sum of absolute differences (SAD)—or another metric function used to measure the difference between two feature points—between a first feature vector representing the feature point in the SDR image and a second feature vector representing the feature point in the HDR image. A particular feature point with the lowest SAD or matching metric can be selected from among the feature points of the same type in the HDR image as a possible match for that feature point in the SDR image. In response to determining that the lowest SAD or matching metric is lower than the matching difference threshold, the particular feature point in the HDR image may be identified or determined as a match for the feature point in the SDR image, and the feature point in the SDR image and the particular feature point in the HDR image form a matched SDR and HDR feature point pair. Otherwise, the particular feature point may not be identified as such a match. This matching operation may be performed iteratively for all feature points in the set of SDR image feature points, thereby resulting in a set of matched SDR and HDR feature point pairs.

[0262] Block 368 includes calculating a geometric transformation, such as a 3x3 2D affine transformation, between the SDR image and the HDR image based partially or wholly on the set of matched SDR and HDR feature point pairs. The geometric transformation can be calculated or derived using coordinates (e.g., pixel rows and pixel columns) of the matched feature points in the SDR and HDR images from the set of matched SDR and HDR feature point pairs.

[0263] For each pair (e.g., kth) of matched SDR and HDR features, the 2D coordinates of the HDR feature in that pair are (τ ix ,τ iy ) and the first vector is denoted as τ i =[τ ix τ iy 1], while the 2D coordinates of the SDR feature points can be included as (η ix ,η iy ) and the second vector η i =[η ix η iy 1] may be included.

[0264] The total number of matched SDR and HDR feature point pairs in the set of matched SDR and HDR feature point pairs is denoted as N. The vectors generated from the 2D coordinates of the feature points in all pairs of matched SDR and HDR feature points in the set of matched SDR and HDR feature point pairs can be put together as follows:

number

[0265] As mentioned above, the geometric transformation may be expressed as a 3x3 matrix as follows:

number

[0266] Mathematically, a first vector and a second vector representing the SDR feature points and the HDR feature points, respectively, in each pair of matched SDR and HDR feature points in the set of matched SDR and HDR feature point pairs can be related to each other through a 3×3 matrix representing the geometric transformation as follows:

number

[0267] Alternatively, for all N pairs of matched feature points in the set of matched SDR and HDR feature point pairs, the vectors representing the SDR and HDR feature points can be related through a 3x3 matrix representing the geometric transformation as follows:

number

[0268] The values ​​of the matrix elements in the 3×3 matrix representing the geometric transformation may be generated or obtained as a solution (e.g., a least-squares solution) of an optimization problem that minimizes a transformation or alignment error. More specifically, the optimization problem may be formulated as follows:

number

[0269] The optimized or optimum value for the 3x3 matrix representing the geometric transformation can be obtained via a least squares solution to the optimization problem of equation () above as follows:

number

[0270] Block 370 includes applying the geometric transformation to one of the SDR image and the HDR image. In the present example, the transformation is applied to the HDR image, thereby shifting HDR pixel positions in the HDR image to match SDR pixel positions in the SDR image for each of the three channels, e.g., Y, Cb, and Cr, in which the HDR image is represented. As a result, most HDR pixels in the HDR image are spatially aligned with corresponding SDR pixels in the SDR image. The remaining HDR pixels and the remaining SDR pixels with which no HDR pixels are spatially aligned may be excluded (e.g., assigned out-of-range pixel values) from being used as (valid) color patches or colors for purposes of generating static SDR-HDR mapping or TPB coefficients or static backward 3D-LUTs.

[0271] Block 372 includes finding corresponding valid color patches or colors in the SDR and HDR images. The valid color patches or colors can be obtained from the codeword values ​​of spatially aligned pixels in the SDR and HDR images. As described above, the spatially aligned pixels can be generated by applying a geometric transformation generated from the matched SDR and HDR feature points.

[0272] In some operating scenarios, to increase the spatial alignment accuracy or reliability of the geometric transformation, after (initial) matched SDR and HDR feature points having a matching metric lower than a minimum matching difference threshold are identified, a further matching threshold may be applied to select or distinguish a subset of final matched SDR and HDR feature points from or among the (initial) matched SDR and HDR feature points.

[0273] For each spatially aligned pixel in the SDR image, an SDR code word, such as an SDR Y / Cb / Cr value, may be determined for the spatially aligned pixel in the SDR image. For co-located or spatially aligned pixels (corresponding to the spatially aligned pixels in the SDR image) in the spatially transformed HDR image generated by the geometric transformation, a corresponding HDR code word, such as a corresponding HDR Y / Cb / Cr value, may be determined for the spatially aligned pixel in the HDR image. If a transformed HDR pixel is not available (e.g., has a value of 0), the HDR pixel may be discarded or may be prevented from being considered as part of the matching color patch or code word.

[0274] For each pair (e.g., the i-th, etc.) of matched SDR and HDR color patches or colors generated from a pair of spatially aligned SDR and HDR pixels in the SDR and HDR images, the matched SDR and HDR color patches or colors (or codeword values), respectively, may be given as follows:

number

[0275] The matched SDR and HDR color patches or colors (or codeword values) in all pairs of matched SDR and HDR color patches, as generated from all pairs of spatially aligned SDR and HDR pixels in the SDR and HDR images of all image pairs of SDR and HDR image pairs, can be collected together from all images and merged into two matrices as follows:

number

[0276] Block 374 calculates the matrix (S with tildes) in equations (88) and (89) above. YCbCr (R709)and R with a tilde YCbCr (R2020) ] as input to solve or generate optimized or optimal TPB coefficients for backward reshaping or static SDR-HDR mapping.

[0277] The posterior generator matrix for each channel is the posterior TPP basis function and the matrix [S YCbCr (R709) ] may be generated as follows:

number

[0278] The per-channel observation matrix for SDR-HDR mapping is the matrix [R YCbCr (R2020) ] may be constructed as follows:

number

[0279] The optimized or optimum backward TPB coefficients for a channel ch can be solved by a least squares solution as follows:

number

[0280] The predicted or backward reshaped HDR value for channel ch can be calculated as follows:

number

[0281] Optimizing Black Box Backward Reshaping on Two Devices (BB2) In a "BB2" operating scenario, (backward) reshaping mapping can be used to map an SDR image (ISP-captured) captured by a first video capture device in an SDR capture mode to generate an (ISP-mapped) HDR image that simulates the HDR appearance of an HDR image (ISP-captured) captured by a second, different video capture device in an HDR capture mode. In some operating scenarios, the first video capture device may be a relatively low-end mobile phone capable of capturing only SDR images or pictures, and the second video capture device may be a relatively high-end phone capable of capturing HDR images or pictures. Although the first and second video capture devices may operate with different hardware configurations and capabilities, reshaping mapping is used to reshape the SDR image (ISP-captured) captured by the first device into an (ISP-mapped) HDR image that approximates the HDR image (ISP-captured) captured by the second device.

[0282] The reshaping (SDR-HDR) mapping may be modeled at least in part based on a plurality of image pairs formed by (training) SDR images captured by a first device and (training) HDR images captured by a second device, where each image pair in the HDR-SDR image pair includes an SDR image and an HDR image corresponding to the SDR image.

[0283] In some "BB2" operating scenarios, some or all of the image pairs may include captured SDR and HDR images depicting visual scenes, such as real-world natural indoor / outdoor scenes, that are subject to spatial and / or temporal alignment errors associated with the first and second video capture devices (e.g., initially, before the image alignment operation). Similar to the "BB1" use cases or operating scenarios, in these "BB2" use cases or operating scenarios, the image alignment operation between corresponding SDR and HDR images in the image pairs may be performed using, for example, a geometric transformation constructed using selected feature points extracted from the SDR and HDR images. The SDR and HDR color patches or colors determined from the aligned SDR and HDR images in the image pair(s) may then be used to generate a reshaped (SDR-HDR) mapping as described herein.

[0284] In some "BB2" operating scenarios, instead of or in addition to acquiring images from natural scenes, some or all of the multiple image pairs may include captured SDR and HDR images (e.g., initially, before image alignment operations, etc.) depicting color charts displayed on one or more reference image displays of the same type, e.g., in a laboratory environment. For example, the color charts may be generated as 16-bit full HD RGB (color chart) TIFF images. These TIFF images including the color charts may be displayed as perceptually quantized (PQ) video signals on a reference image display, such as a PRM TV. The color charts displayed on the reference image displays may be captured by first and second video capture devices, respectively.

[0285] 2N shows an exemplary TIFF image including a color chart. As shown, the color chart may be a central square in the TIFF image, which includes multiple color blocks having distinct sets of colors to be captured and matched between a first device and a second device. Each color in the distinct color set may be different from all other colors in the distinct color set. The distinct color set may be displayed by a reference image display(s) having distinct sets of pixel / codeword values. Thus, each color in the distinct color set may correspond to a respective pixel value or codeword value in the distinct pixel / codeword value set.

[0286] The different TIFF images may include different color charts having different colors or different sets of distinct colors, and each color chart in each of the different TIFF images may correspond to a respective set of distinct colors from the different colors or sets of distinct colors.

[0287] Each of the four corner rectangles in the TIFF image of Figure 2N contains a checkerboard pattern. The same corner rectangle or checkerboard pattern may be included in different TIFF images containing different color charts. The checkerboard patterns at the four corners of a TIFF image can be used as a spatial key or reference mark for estimating a projective transformation, as described in more detail below. Additionally, optionally, or alternatively, the TIFF image may contain one or more numbers, binary coding, a QR code, or the like as a unique identifier (ID) assigned or used to identify the TIFF image, the color chart therein, or a set or plurality of colors therein.

[0288] To find as many color correspondences / mapping relationships as possible between the first and second video capture devices, the color chart in the TIFF image described herein may include as many different colors (different sets of colors or multiple different colors) as possible corresponding to as many different pixel / codeword values ​​as possible, subject to the display capability of the reference image display to distinguish between different colors. Furthermore, the color chart may be displayed with different overall intensities or illuminations to allow the first and second video capture devices to have different exposure settings under different lighting conditions. Thus, the same colors displayed in the color chart in the TIFF image can be displayed with different intensities or illuminations on the reference image display(s).

[0289] FIG. 3E shows an exemplary process flow for generating multiple distinct color charts to be included in multiple different TIFF images. To generate each color chart, the mean and variance of the colors (e.g., RGB) in the color chart may be determined and used to randomly generate colors using a statistical distribution. In some operating scenarios, the statistical distribution may represent a beta distribution or distribution type. Different combinations of mean and variance values ​​for the statistical distribution or distribution type may be used to generate different color charts with different distinct colors or sets of distinct colors.

[0290] Block 382 determines a set of possible mean values ​​in the PQ domain or color space (M PQ Define or determine the

[0291] In an operating scenario where the reference image display used to display the TIFF image including the color chart is a PRM TV, the maximum and minimum luminance values ​​supported by the reference image display may range between 1000 nits and 0 nits, or even a larger dynamic range and contrast ratio. By way of example and not limitation, the minimum and maximum PQ values ​​that can be displayed by the reference image display without clipping may be given as P0 = L2PQ(0) and P1 = L2PQ(1000), respectively, where L2PQ(·) denotes the mapping function from (linear or non-PQ) luminance to (non-linear or PQ) luma codewords in the PQ domain or color space.

[0292] In a non-limiting example, the set M PQ is L on a logarithmic scale lower From L upper The lower bound L can be defined as the set of distinct PQ luma codewords corresponding to (linear) luminance values ​​that are evenly distributed within the (linear) luminance value range of lower is 10 -4 may be set to a value such as L upper is 10 2.99 may be set to a value such as |M| (e.g., a fractional exponent value selected for numerical stability), which depends in part or in whole on the image capture capabilities of the first and second devices (e.g., ability to capture the darkest and brightest luminances). A logarithmic scale is used because the luma codewords in captured SDR and HDR images may be approximately linear with respect to the logarithm of luminance within the luminance value range. The total number of possible mean values ​​in the set of possible mean values ​​- magnitude |M PQ |――is,|M PQ It may be set as |=2048, but is not necessarily limited to this.

[0293] Block 384 involves defining or determining a set of possible shape coefficients to be used to generate possible variance values ​​for a statistical distribution or distribution type (eg, a beta distribution, etc.).

[0294] In some operating scenarios, the beta distribution used to randomly select colors or codeword values ​​is defined or has support on a scaled or closed value interval [0,1]. Given PQ values ​​in the PQ codeword value range [P0,P1], the scaled mean μ (where 0<μ<1, or within the closed value interval of the beta distribution) may be calculated as follows:

number

[0295] The scaled variance (σ) for the beta distribution in the scaled or closed value interval [0,1] 2 ) may be set to be proportional to μ(1−μ) as follows to avoid over / underexposure caused by large variations in color blocks (e.g., luminance, etc.) during capture of SDR and HDR images by the first and second devices: σ 2 =μ(1-μ) / θ (95) where θ represents a shape factor that influences or determines the shape (e.g., more compressed, more expanded, etc.) of the Beta distribution. A value for the shape factor can be selected from a set of possible shape factors Θ to allow the generated color chart to have a relatively high diversity in codeword values ​​and / or colors resulting therefrom. A non-limiting example for the set of possible shape factors may be Θ={3, 6, 9, 12}.

[0296] Block 386 involves generating a plurality of all unique combinations of scaled means and shape factors using a set of possible means and sets of shape factors for different instances of a statistical distribution (e.g., a beta distribution, etc.). In the present example, the total number of unique combinations is |M PQΘ|=2048×4=8192, where |Θ| represents the total number of elements in the set Θ or the size of the set Θ. In some operating scenarios, a different TIFF image containing a different color chart is generated for each unique combination of scaled mean values ​​and shape factors among all the multiple unique combinations of scaled mean values ​​and shape factors, thereby reducing the total number of different color charts, or corresponding different TIFF images, to |M PQ It is given as |×|Θ|.

[0297] Block 388 includes selecting the current combination of scaled mean and shape factor values ​​as the next combination from among all the multiple unique combinations of scaled mean and shape factor values, wherein a set or plurality of different pixel or code word values ​​that respectively specify or define a set or plurality of different colors for the current color chart corresponding to the current combination of scaled mean and shape factor values ​​is determined by the scaled mean μ in the current combination and the current scaled variance σ in the current combination. 2 can be generated from the beta distribution given

[0298] Block 390 involves calculating or defining the beta distribution as follows: f(x;α,β)=x α-1 (1-x) β-1 / B(α,β) x∈[0,1] (96) where α and β are the beta distribution parameters; B(α,β) represents a normalization constant. The beta distribution parameters α and β can be derived from the mean and variance of the beta distribution as follows: α=μν, β=(1-μ)ν where ν=μ(1-μ) / σ 2 -1 (97)

[0299] Block 392 involves generating a set of different colors (or pixel / codeword values) (e.g., 144 or so) for the current color chart or the current color chart image based on a set of random numbers (e.g., 144 or so) generated from a beta distribution. Each random number, denoted x (where 0≦x≦1), can be scaled back or converted to a pixel value or codeword value (referred to as a “PQ value”) for the corresponding channel within the PQ value range [P0,P1]. x PQ =x(P1-P0)+P0(98)

[0300] A set of PQ values, generated from scaling or converting the set of random numbers back to the PQ value range, can then be generated from the set of random numbers and used as the per-channel code word values ​​or set of code words for the set of colors in the current color chart. In some operating scenarios, for each color chart, the same beta distribution (with the same mean and shape coefficient) is used with, for example, three different sets of random numbers to generate per-channel code words for each of multiple channels (e.g., RGB) of the color space in which the display image is rendered by the reference image display. Thus, the average color generated from averaging the colors represented in all color blocks in the color chart will be close to a neutral or gray color.

[0301] Block 394 involves generating a central block of a current color chart or corresponding TIFF image. The current color chart may include a set of color blocks (e.g., 2D squares), each of which is a single color given by a specific (cross-channel or composite) pixel or codeword value with three per-channel pixel or codeword values. Three sets of per-channel codewords generated from the same beta distribution, but using three different sets of random numbers, may be used by the reference image representation to drive the rendering of the red, blue, and green channels of the color blocks in the current color chart or the set of color blocks in the current TIFF image, respectively.

[0302] Additionally, optionally, or alternatively, the RGB values ​​used to specify or generate a background color for the background of the color chart image may be calculated based on said mean PQ value μ from which the scaled mean of the beta distribution was derived. PQ may be set to

[0303] Block 396 involves determining whether the current combination of scaled mean values ​​and shape factor values ​​is the last combination in the plurality of all unique combinations of scaled mean values ​​and shape factors. If so, the process flow for generating a plurality of different color charts or different color chart images ends. If not, the process flow returns to block 388.

[0304] By way of example and not limitation, n c A total number of different colors or color blocks, e.g., 144, may be generated for each color chart. The total number of colors or color blocks generated from multiple different color charts and color chart images is |M PQ |×|Θ|×n c= 2048 × 4 × 144 = 1179648 colors or color blocks, which may be used to determine correspondence / match relationships between SDR color patches or codewords captured in SDR capture mode with a first video capture device and corresponding HDR color patches or codewords captured in HDR capture mode with a second video capture device.

[0305] In some operating scenarios, multiple TIFF images each containing multiple color charts can be displayed or rendered sequentially on a reference image display, such as a PRM TV, at a constant playback frame rate and captured in SDR and HDR images by first and second video capture devices, respectively. Because the illumination of the reference image display can vary with viewing angle, the SDR and HDR images can be captured separately by first and second video capture devices (e.g., two phones) positioned at the same position and orientation relative to or with respect to the reference image display. Thus, SDR and HDR video signals or bitstreams containing the captured SDR and HDR images of the color charts or TIFF images rendered on the reference image display can be generated by the first and second devices or their cameras, respectively. Because the playback frame rate of the reference image display can be kept constant, the frame numbers of the color chart can be relatively easily determined, and an image pair can be established in which an SDR image captured by a first video capture device and an HDR image captured by a second video capture device represent the same color chart among the multiple color charts. The captured SDR and HDR images may be represented in SDR and HDR YCbCr (or YUV) color spaces, for example, in a YUV image / video file.

[0306] 3F shows an exemplary process flow for matching SDR and HDR colors between captured SDR and HDR image pairs for each color chart in a plurality of color charts. To extract the colors of each individual color block from the captured SDR and HDR images of a color chart image containing a color chart, the captured SDR and HDR images can be transformed into the same layout as the (original) color chart image displayed on the reference image display.

[0307] Block 3002 includes receiving a captured (SDR or HDR) checkerboard image, which may be taken by a camera of one of the first and second video capture devices from a displayed checkerboard image displayed on a reference image display. Blocks 3002-3006 of the same process flow of FIG. 3F may be performed with respect to a captured checkerboard image taken by a camera of the other of the first and second video capture devices from the same displayed checkerboard image displayed on a reference image display.

[0308] Block 3004 includes detecting checkerboard corners from a checkerboard image captured by a camera of the device.

[0309] Block 3006 includes calculating or calibrating camera distortion coefficients for the camera using the checkerboard corners detected from the captured checkerboard image. The 3D (reference) coordinates of the checkerboard image displayed on the reference image display may first be determined in a stationary 3D coordinate system relative to the reference image display. Camera parameters, such as distortion coefficients, and other intrinsic parameters of the camera used by the device to generate the captured checkerboard image may be calculated as optimized values ​​that produce the best mapping (e.g., minimum error or minimum discrepancy) between the 3D reference coordinates of the checkerboard corners in the captured checkerboard image and the 2D (image) coordinates of the checkerboard corners. This calibration process may be performed in either the YUV or RGB color space to which the displayed or captured checkerboard image may be converted or represented.

[0310] 2M and 2N show two exemplary checkerboard images detected from captured (HDR and SDR, respectively) images of the checkerboard image. To achieve relatively high accuracy (e.g., within half a pixel of camera reprojection error) when projecting the captured images by each camera of the multiple (e.g., 100) captured checkerboard images onto the display image on the reference image display, distortion coefficients or other intrinsic parameters of each camera used in the first and second video capture devices can be determined, calculated, or calibrated.

[0311] The (camera-specific) distortion coefficients and intrinsic parameters of the cameras of the first and second video capture devices obtained in processing blocks 3002-3006 can be used for analysis or correlation between captured SDR and HDR images from TIFF images containing respective color charts as follows:

[0312] Block 3008 includes receiving a captured (SDR or HDR) image of the TIFF image to be displayed on the reference image display, including a color chart and checkerboard corners (or a checkerboard pattern at the corners). The captured (SDR or HDR) image may be captured by a camera for which distortion coefficients have been previously obtained using a previously captured checkerboard image (which may not include a color chart) in block 3006.

[0313] Block 3010 involves correcting or de-distorting the captured image of the displayed TIFF image to compensate for camera lens distortion of the camera, using distortion coefficients obtained in a camera calibration process, such as block 3006.

[0314] Block 3012 includes detecting checkerboard corners in the captured image, where the checkerboard corners are captured from checkerboard corners (e.g., four corners) in the displayed TIFF image on a reference image display, which may be a PPM TV.

[0315] Block 3014 involves estimating a projective transformation between the image coordinates of the captured image and the image coordinates of the original TIFF image displayed on the reference image display and captured within the received captured image.

[0316] Block 3016 includes using the estimated projective transformation to rectify the captured image to the same layout as the original TIFF image. Using the estimated projective transformation, the captured image can be rectified to the same spatial layout as the original TIFF image.

[0317] FIG. 2O shows (a) a captured image from the (displayed) TIFF image, (b) a modified captured image generated by correcting the captured image with camera distortion correction and projective transformation, and (c) the original TIFF color chart image displayed on a reference image display. Because the modified captured image in FIG. 2O(b) is generated from a distortion removal or correction operation performed on the captured image in FIG. 2O(a), some pixels in the modified image in FIG. 2O(b) may be undefined.

[0318] Block 3018 involves locating and extracting (a set of) individual color blocks from the color chart within the modified captured image. As used herein, a color block is designated by a single corresponding (e.g., RGB, YCbCr, composite, etc.) pixel or codeword value. Corresponding individual codewords that designate each individual color block in the captured image can be determined based on the pixel values ​​or codeword values ​​of the pixels within these individual color blocks in the modified captured image. Furthermore, individual original codewords that designate each individual original color block in the original TIFF image (corresponding to each individual color block in the modified captured image) can be determined from the RGB or YUV file for the original TIFF image.

[0319] The correspondence / mapping relationship between the SDR color blocks or code words extracted from the (modified) captured SDR image of the original color chart image and the HDR color blocks or code words extracted from the (modified) captured HDR image of the same original color chart image can be established based in part or in whole on the individual original color blocks or code words in the original TIFF image from which both the (modified) SDR and HDR images are derived.

[0320] Using a plurality of TIFF images, a plurality of SDR images captured in an SDR capture mode by a first video capture device from the displayed TIFF images, and a plurality of HDR images captured in an HDR capture mode by a second video capture device from the same displayed TIFF images, a plurality of correspondences or mapping relationships can be established between sets of SDR colors and sets of HDR images.

[0321] Several approaches can be used to map an SDR image to an HDR image.

[0322] In a first approach, in some "BB2" operating scenarios, as in some "BB1" operating scenarios, the mapped SDR and HDR color patches or colors can be used to generate (e.g., static) SDR-to-HDR reshaping operating parameters, such as TPB coefficients. These TPB coefficients can be combined with TPB basis functions using SDR codewords of the SDR image as input parameters to predict corresponding HDR codewords of the reconstructed HDR image. A static 3D-LUT may be pre-constructed and deployed on a video capture device (e.g., the first video capture device) to map SDR images captured by a first video capture device to reconstructed HDR images that simulate the HDR appearance of a second video capture device. Because the HDR color blocks or code words used to generate the reshaping operation parameters are extracted from a captured (training) HDR image of the second video capture device, the predicted HDR code words generated using the reshaping operation parameters are likely to provide a mapped HDR appearance in the reconstructed HDR image that is similar to the actual HDR appearance of the actual HDR image captured by the second video capture device.

[0323] In the second approach, in some "BB2" operating scenarios, non-TPB optimization may be used to generate non-TPB reshaping operation parameters in a non-TPB reshaping operation for mapping a captured SDR image taken by a first video capture device to a reconstructed HDR image that simulates the HDR appearance of a second video capture device. The 3D-LUT for the non-TPB reshaping operation may be constructed directly without the relatively high continuity and smoothness supported by the TPB reshaping operation. This non-TPB second approach offers relatively high design freedom and can be relatively flexible, even if different colors represented in nearby 3D-LUT entries / nodes may or may not have relatively high continuity and smoothness. In some operating scenarios, with its relatively high design freedom, the non-TPB second approach may be more suitable than the first approach for two-device or "BB2" operating scenarios in which an SDR image from one device is mapped to an HDR image that simulates the HDR appearance of another device.

[0324] A second non-TPB approach can utilize a 3D mapping table (3DMT) to construct a backward reshaping mapping, or a backward lookup table (BLUT) representing the backward reshaping mapping. The BLUT may be used to map SDR (e.g., cross-channel, 3-channel, etc.) codewords of an SDR image to HDR (e.g., chroma channel, per chroma channel, etc.) codewords of a reconstructed HDR image. Exemplary operations for constructing a BLUT-like backward reshaping mapping from a 3DMT can be found in U.S. Patent Application No. 17 / 054,495, filed May 9, 2019, the contents of which are incorporated herein by reference in their entirety as if fully set forth herein.

[0325] For illustrative purposes only, an SDR codeword may be represented in three dimensions or channels Y, Cb, and Cr in the SDR YCbCr color space. The 3D mapping table may be generated from a 3D histogram having multiple 3D histogram bins. The multiple 3D histogram bins are represented by a set of three positive integers, Q. y ,Q Cb ,Q Cr may correspond to multiple color space partitions generated by dividing each dimension or channel by a respective positive integer among y ×Q Cb ×Q Cr ) 3D histogram bins are generated.

[0326] Multiple 3D histogram bins (Ω Q,s Each 3D histogram bin (denoted as q) in the SDR (YCbCr) color space has a respective bin index or three respective quantized channel values ​​q = (q y ,q Cb ,q Cr ), which stores the pixel counts for all SDR color patches or codewords in the SDR color space partition represented by the 3D histogram bin. All bin indices (or quantized channel values) for multiple 3D histogram bins can be grouped into a set of bin index values ​​denoted Q, where Q = [Q y ,Q C0 ,Q Cr ].

[0327] Additionally, the sum of the HDR codewords mapped to the SDR codewords in each 3D histogram in the multiple 3D histograms may be calculated and stored for the 3D histogram bin. y Q,v , Ψ Cb Q,v and Ψ Cr Q,vLet denote the sum of the HDR codewords (also called "mapped HDR luma and chroma values").

[0328] An example procedure for generating SDR pixel counts and mapped HDR luma and chroma values ​​for the 3D histogram bins of the histogram is shown in Table 2 below. [Table 2]

[0329] (s q y,(B) , s q Cb,(B) , s q Cr,(B) Let q denote the (representative) SDR codeword at the center of the q-th 3D histogram bin. The representative SDR codewords for all 3D histogram bins can be fixed for all SDR images / frames and can be pre-computed with the exemplary procedure shown in Table 3 below. [Table 3]

[0330] Next, 3D histogram bins with a non-zero (SDR) pixel count may be identified and retained from the plurality of 3D histogram bins, and all other 3D histogram bins with a zero (SDR) pixel count may be discarded or removed from the plurality of 3D histogram bins.

[0331] q0, q1, … q k-1 Ω, which has a non-zero (SDR) number of pixels, q Q,s Let ≠ 0 denote all k 3D histogram bins for which ≠ 0, where k is a positive integer. The mean values ​​of the mapped HDR luma and chroma values

number

[0332] A 3D-LUT can be constructed from a 3DMT represented by a 3D histogram that stores the number of SDR pixels in each 3D histogram bin and their mapped HDR codeword values. Each node / entry in the 3D-LUT or 3DMT stores the specific SDR codeword (q) represented in each of the k 3D histogram bins. y ,q Cb ,q Cr ) and calculated for the 3D histogram bins in Table 4 above.

number

[0333] As mentioned above, exemplary operations for constructing a 3D-LUT from a 3DMT to function as a non-TPB backward reshaping mapping or BLUT (e.g., operations for predicting HDR chroma codewords per chroma channel from cross-channel SDR codewords) are described in the aforementioned U.S. patent application Ser. No. 17 / 054,495.

[0334] In some "BB2" operating scenarios, a luma backward reshaping mapping may be used to backward reshape the SDR luma codewords of an SDR image into predicted, or backward reshaped, HDR luma codewords of the corresponding HDR image using a GPR-based model. Exemplary generation of luma reshaping mapping using a GPR-based model can be found in U.S. Provisional Patent Application No. 62 / 887,123, "Efficient User-Defined SDR-HDR Conversion Using Model Templates," by Guan-Ming Su and Harshad, filed August 15, 2019, and PCT Application No. PCT / US2020 / 046032, filed August 12, 2020, the contents of which are incorporated by reference in their entireties as if fully set forth herein. For example, a CDF matching curve may be generated from each training SDR-HDR image pair in a plurality of training SDR-HDR image pairs. Using a set of SDR points (e.g., 15 points) uniformly sampled across the SDR codeword range (e.g., the entire SDR codeword range), a corresponding mapped HDR codeword can be found in each of the CDF matching curves. For each sample SDR point in the set of uniformly sampled SDR points, a GPR model can be constructed based on a histogram having multiple (e.g., 128) luma-bin histograms and used to generate a corresponding luma reshaping mapping.

[0335] TPB optimization during editing Image / video editing is a common application in video capture devices, such as mobile devices, to allow users to adjust color, contrast, brightness, or other user preferences. Image / video editing can be done or performed on the encoder side before the captured images are compressed into a (compressed) video signal or bitstream. Additionally, optionally, or alternatively, image / video editing can be done or performed on the decoder side after the video signal bitstream is decoded or decompressed.

[0336] The image / video editing operations described herein can be performed in either or both the HDR domain and the SDR domain. Image / video editing operations in the HDR domain are relatively easy to perform. For example, after HDR content or images are edited, the edited HDR content or images can be passed as an input or reference HDR image to a video pipeline that generates corresponding reshaped or ISP SDR content or images to be encoded in a video signal or bitstream. In comparison, image / video editing operations in the SDR domain can be relatively difficult. While HDR-to-SDR (HDR to SDR) and / or SDR-to-HDR (SDR to HDR) mappings can be designed or generated to ensure or enhance reversibility between the SDR and HDR domains, image / video editing operations performed on SDR images to be encoded into a video signal or bitstream likely pose difficulties or challenges in inverting or backward reshaping the edited SDR image to generate a reconstructed HDR image that approximates the reference HDR image.

[0337] FIG. 3G illustrates exemplary image / video editing operations at the encoder side performed in an upstream device, such as a video capture device, or a video encoder. As shown, an input HDR image may be represented in an input HDR domain or color space (denoted "HLG YCbCr"). A TPB-based forward reshaping operation (denoted "Forward TPB") may be performed to forward reshape the input HDR image into a forward reshaped SDR image represented in a forward reshaped SDR domain or color space (denoted "SDR YCbCr"). An image / video editing operation (denoted "RGB domain editing") may be performed to edit the forward reshaped SDR image to generate an edited SDR image represented in an edited forward reshaped SDR domain or color space (denoted "Edited SDR YCbCr"). The edited SDR image represented in the edited forward reshaped SDR domain or color space may be compressed or encoded into a video signal or bitstream using one or more video codecs of the upstream device.

[0338] FIG. 3H illustrates exemplary decoder-side image / video editing operations performed in a downstream receiving device or video decoder. As illustrated, a video signal or bitstream is decoded by one or more video codecs in the downstream device into an SDR image represented in the SDR domain or color space (denoted "SDR YCbCr"). An image / video editing operation (denoted "RGB domain editing") can be performed to edit the SDR image and generate an edited SDR image represented in an edited forward-reshaped SDR domain or color space (denoted "edited SDR YCbCr"). A TPB-based backward reshaping operation (denoted "backward TPB") can be performed on the edited SDR image represented in the edited forward-reshaped SDR domain or color space to generate a backward-reshaped or reconstructed HDR image in a reconstructed HDR domain or color space (denoted "PQ YCbCr").

[0339] As shown in Figures 3G and 3H, an SDR image encoded into a video signal at the encoder side or decoded from a video signal may be represented in an SDR domain or color space (e.g., forward reshaped), such as an SDR YCbCr domain or color space. Using this, a video codec can be programmed or developed to perform image processing operations relatively efficiently. By way of example and not limitation, image / video editing operations can be performed in an SDR RGB domain or color space, as shown in Figures 3G and 3H. RGB-to-YCbCr and YCbCr-to-RGB conversions may be performed through conversions (e.g., standard-defined conversions) such as SMPTE gamut conversions.

[0340] As more colors are pushed into the SDR domain or color space, codewords in the forward-reshaped domain or color space, such as the SDR YCbCr color space, often exceed a codeword value range, such as the SMPTE range, in the SDR RGB color space in which image / video editing operations are performed. When applying a YCbCr-to-RGB conversion (which converts YCbCr codewords in the YCbCr codeword value range (normalized to a value range of [0,1]) of the SDR YCbCr color space to RGB codewords in the SDR RGB color space), some of the RGB codewords may exceed or fall outside the SMPTE range (normalized to a value range of [0,1]) of the SDR RGB color space. These out-of-range codeword values ​​can be clipped, but the clipped codeword values ​​may not be able to be restored to the original, unclipped HDR codewords in a backward-reshaping operation (e.g., TPB-based, etc.). Additional operations described herein may be used to improve reversibility and reduce or avoid visual artifacts in image / video editing applications.

[0341] 3I to 3M show some example solutions for clipping out-of-range codewords in image / video editing applications.

[0342] In some operating scenarios, as shown in Figure 3I, out-of-range SDR RGB codewords resulting from a YCbCr-to-RGB conversion (denoted "YCbCr-to-RGB clipping conversion") performed on an SDR YCbCr image at the encoder or decoder side can be clipped to a valid or specified codeword value range, such as [0,1]—e.g., all values ​​greater than 1 are hard clipped to 1, and all values ​​less than zero are hard clipped to 0. Image / video editing operations ("editing in the RGB domain") can then be performed using SDR RGB codewords within the valid or specified codeword value range in the SDR RGB color space. The edited SDR RGB codewords in the SDR RGB color space can be converted and clipped to transformed edited SDR YCbCr codewords within a valid or specified codeword value range, such as [0,1], in the transformed edited SDR YCbCr color space (denoted "edited SDR YCbCr") by an RGB-to-YCbCr conversion (denoted "RGB-to-YCbCr clipping conversion").

[0343] In some operating scenarios, as shown in FIG. 3J, out-of-range SDR RGB codewords resulting from a YCbCr-to-RGB conversion (denoted "YCbCr-to-RGB unclipping conversion") performed on an SDR YCbCr image at the encoder or decoder side are not clipped to a valid or specified codeword value range, such as [0, 1]. An image / video editing operation ("editing in the RGB domain") can be performed using unclipped (input) SDR RGB codewords within a valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr-to-RGB conversion with unclipping operation allows the original information in the (input) SDR YCbCr codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation takes all possible real codeword values ​​within the input SDR RGB codeword (e.g., including values ​​less than 0 or greater than 1 within a limited but wider value range than

[0001] ). As a result, some information in the input SDR YCbCr codeword can be preserved in the input SDR RGB codeword as well as in the edited SDR RGB codeword. An SDR RGB boundary operation (denoted "RGB boundary

[0001] clipping")—for example, optionally based on a user preference of a user running a video editing application—can be performed to scale down or push edited SDR RGB codewords that are outside the valid or specified range of [0,1] into the valid or specified range of [0,1], thereby generating scaled edited SDR RGB codewords within the valid or specified range of [0,1].The scaled edited SDR RGB codewords in the SDR RGB color space can be converted and clipped to transformed scaled edited SDR YCbCr codewords within a valid or specified codeword value range, such as [0,1], of the transformed scaled edited SDR YCbCr color space (denoted "edited SDR YCbCr") by an RGB-to-YCbCr conversion (denoted "RGB-to-YCbCr clipping conversion").

[0344] In some operating scenarios, as shown in FIG. 3K, out-of-range SDR RGB codewords resulting from a YCbCr-to-RGB conversion (denoted "YCbCr-to-RGB non-clipping conversion") performed on an SDR YCbCr image at the encoder or decoder side are not clipped to a valid or specified codeword value range, such as [0, 1]. An image / video editing operation ("editing in the RGB domain") can be performed using unclipped (input) SDR RGB codewords within a valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr-to-RGB conversion with non-clipping operation allows the original information in the (input) SDR YCbCr codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation takes all possible real codeword values ​​within the input SDR RGB codeword (e.g., including values ​​less than 0 or greater than 1 within a limited but broader value range than

[0001] ). As a result, some information in the input SDR YCbCr codeword can be preserved in the input SDR RGB codeword as well as in the edited SDR RGB codeword. Unlike that shown in Figure 3J, in the operating scenario shown in Figure 3K, an SDR RGB boundary operation may not be performed. The edited SDR RGB codeword in the SDR RGB color space can be converted and clipped to a converted edited SDR YCbCr codeword within a valid or specified codeword value range (such as [0, 1]) of the converted edited SDR YCbCr color space (denoted "edited SDR YCbCr") by an RGB-to-YCbCr conversion (denoted "RGB-to-YCbCr clipping conversion").

[0345] In some operating scenarios, as shown in FIG. 3L, out-of-range SDR RGB codewords resulting from a YCbCr-to-RGB conversion (denoted "YCbCr-to-RGB non-clipping conversion") performed on an SDR YCbCr image at the encoder or decoder side are not clipped to a valid or specified codeword value range, such as [0, 1]. Image / video editing operations ("editing in the RGB domain") can be performed using unclipped (input) SDR RGB codewords within a valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr-to-RGB conversion with non-clipping operation allows the original information in the (input) SDR YCbCr codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation takes all possible real codeword values ​​within the input SDR RGB codeword (e.g., including values ​​less than 0 or greater than 1 within a limited but broader value range than

[0001] ). As a result, some information in the input SDR YCbCr codeword can be preserved in the input SDR RGB codeword as well as in the edited SDR RGB codeword. An SDR RGB boundary operation (denoted "RGB boundary clipping 3D-LUT") can be performed—for example, optionally based on user preferences of a user operating a video editing application—to scale down or push edited SDR RGB codewords outside the (3D) boundary supported by the TPB-based reshaping operation into the boundary, thereby generating scaled edited SDR RGB codewords that are supported by the TPB-based reshaping operation or within a boundary that is well-defined in the TPB-based reshaping operation. The boundary supported by the TPB-based reshaping operation need not be a regular shape or simply a cube defined by a fixed value range of [0,1].The scaled edited SDR RGB codewords in the SDR RGB color space can be converted and clipped to transformed scaled edited SDR YCbCr codewords within a valid or specified codeword value range, such as [0,1], of the transformed scaled edited SDR YCbCr color space (denoted "edited SDR YCbCr") by an RGB-to-YCbCr conversion (denoted "RGB-to-YCbCr clipping conversion").

[0346] The (input) SDR YCbCr image can be derived from the original HDR image through forward reshaping or ISP processing using known (e.g., white box, ISP, etc.) HDR-SDR forward reshaping or mapping. The boundaries can be determined partially or fully based on the HDR-SDR forward reshaping or mapping and / or any applicable color space transformation matrix(ies) and expressed as a (TPB boundary clipping) 3D-LUT. For example, boundary pixel or codeword values ​​in the forward reshaped or ISP SDR color space can be determined using full grid sampling data covering the entire HDR domain or color space in which the original HDR image is represented.

[0347] After an image / video editing operation ("editing in the RGB domain"), the edited SDR RGB codewords can be scaled, squeezed, or irregularly clipped to a 3D shape described or enclosed by a boundary. In some operating scenarios, the scaled edited SDR YCbCr codewords in the SDR RGB color space can be converted to transformed scaled edited SDR YCbCr codewords by an RGB-to-YCbCr conversion ("RGB-to-YCbCr clipping conversion") without further clipping by an RGB-to-YCbCr conversion ("RGB-to-YCbCr clipping conversion"), because the transformed scaled edited SDR YCbCr codewords are already positioned within corresponding boundaries in the SDR YCbCr color space supported by the TPB-based backward reshaping operation. In these operating scenarios, as shown in FIG. 3J, a maximized number of colors can be preserved in the transformed scaled edited SDR YCbCr codeword, as well as in the backward-reshaped HDR codeword generated from the transformed scaled edited SDR YCbCr codeword by the TPB-based backward-reshaping operation, while avoiding the generation of color artifacts.

[0348] In some operating scenarios, as shown in FIG. 3M, out-of-range SDR RGB codewords resulting from a YCbCr-to-RGB conversion (denoted "YCbCr-to-RGB non-clipping conversion") performed on an SDR YCbCr image at the encoder or decoder side are not clipped to a valid or specified codeword value range, such as [0, 1]. An image / video editing operation ("editing in the RGB domain") can be performed using unclipped (input) SDR RGB codewords within a valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr-to-RGB conversion with non-clipping operation allows the original information in the (input) SDR YCbCr codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation takes all possible real codeword values ​​within the input SDR RGB codeword (e.g., including values ​​less than 0 or greater than 1 within a limited but broader value range than

[0001] ). As a result, some information in the input SDR YCbCr codeword can be preserved in the input SDR RGB codeword as well as in the edited SDR RGB codeword. The edited SDR RGB codeword in the SDR RGB color space can be converted and clipped to a transformed edited SDR YCbCr codeword within a valid or specified codeword value range, such as [0,1], of the transformed edited SDR YCbCr color space (denoted "edited SDR YCbCr") by an RGB-to-YCbCr conversion (denoted "RGB-to-YCbCr clipping conversion").The transformed edited SDR YCbCr codewords in the transformed edited SDR YCbCr color space (“edited SDR YCbCr”) can be clipped by an SDR YCbCr boundary operation (denoted “YCbCr boundary clipping 3D-LUT”)—e.g., optionally based on user preferences of a user operating the video editing application—that can be performed to scale down or push the transformed edited SDR YCbCr codewords outside the (3D) boundary supported by the TPB-based reshaping operation into the boundary, thereby generating scaled transformed edited SDR YCbCr codewords in the scaled transformed edited SDR YCbCr color space (denoted “edited SDR YCbCr”) that are supported by the TPB-based reshaping operation or within a well-defined boundary in the TPB-based reshaping operation. The boundaries supported by TPB-based reshaping operations may not simply be regular shapes such as cubes with sides defined as the valid range or specified (normalized) range [0,1].

[0349] The (input) SDR YCbCr image can be derived from the original HDR image through forward reshaping or ISP processing using a known HDR-to-SDR forward transform or mapping (e.g., white box, ISP, etc.). The boundaries can be determined based on the HDR-to-SDR forward transform or mapping and / or any applicable color space transformation matrix(ies) and represented using a (TPB boundary clipping) 3D-LUT. For example, boundary pixel or codeword values ​​in the forward reshaped or ISP SDR color space can be determined using full grid sampling data covering the entire HDR domain or color space in which the original HDR image is represented.

[0350] Building a boundary clipping 3D-LUT and clipping The forward reshaping or mapping from HDR to SDR, such as TPB-based forward reshaping, may be a nonlinear function. While the input HDR codewords in the input HDR domain or color space may be well-ordered within a simple 3D cube, the boundaries of the mapped SDR codewords produced by applying the (nonlinear or TPB) HDR-to-SDR mapping may be relatively irregular, unlike the 3D cube.

[0351] To perform TPB boundary clipping for relatively irregular boundaries, a TPB boundary clipping 3D-LUT may be constructed. The 3D-LUT can be searched using an SDR codeword as an input (or lookup key), and in response to determining that the SDR codeword is within the boundary, the 3D-LUT returns the SDR codeword as a value. Otherwise, in response to determining that the SDR codeword is outside the boundary, the 3D-LUT can return a clipped SDR codeword (within the boundary) that is different from the original SDR codeword.

[0352] In some operating scenarios, boundary clipping can be implemented at two levels: at the first level, regular clipping is performed using a range defined by minimum and maximum values ​​(or lower and upper bounds) that define the (3D) codeword range; at the second level, irregular clipping is performed using a (TPB) boundary clipping 3D-LUT.

[0353] FIG. 3N illustrates an exemplary process flow for constructing a boundary clipping 3D-LUT (e.g., TPB). The left side of the process flow illustrated in FIG. 3N can be implemented or performed to construct a 3D boundary using the alphaShape technique. The right side of the process flow illustrated in FIG. 3N can be implemented or performed to generate a 3D-LUT for irregular clipping using the constructed 3D boundary.

[0354] Block 3022 creates a 3D uniform sampling grid or set of sampled values.

number

[0355] The sampled values ​​in the R.2020 YCbCr color space HLG may be converted to corresponding values ​​in the R.2020 RGB color space HLG, as shown in equation (44) above. The converted values ​​in the R.2020 RGB color space HLG may then be further converted to an optimized value for the parameter a (a ), as shown in equation (45) above. opt ) may be converted to a corresponding value in the RGB color space HLG. (a) The converted value in the RGB color space HLG may be clipped as shown in equation (46) above and converted to a corresponding clipped value in the R.2020 RGB color space HLG as shown in equation (47) above. The clipped value in the R.2020 RGB color space HLG derived using equation (47) above may be converted to a corresponding clipped value V in the R.2020 YCbCr color space HLG as shown in equation (48) above. YCbCr (FL),(R2020) may be converted to

[0356] Block 3024 receives the clipped or constrained (HDR YCbC HLG) value V yCbCr (FL),(R2020) (referred to as "Constraint Input" in FIG. 3N) to apply TPB-based forward reshaping (referred to as "Forward TPB" in FIG. 3N). More specifically, optimized forward TPB coefficients for TPB-based forward reshaping are used, or the forward generator matrix S F ch (in equation (50) above). The generator matrix is ​​multiplied by the clipped HDR values ​​V in the R.2020 YCbCr color space HLG.yCbCr (FL),(R2020) as input parameters to the forward TPB basis functions. The mapped or forward reshaped clipped SDR or R.709 YCbCr values ​​are obtained as follows:

number

[0357] Block 3046 is

number

number

[0358] R.709 RGB values ​​converted from R.709 YCbCr values ​​may contain values ​​outside a valid or specified range, such as the [0,1] value range.

[0359] For each channel, the minimum and maximum values ​​in the R.709 RGB values ​​given in equation (100) can be measured or determined as follows:

number

[0360] These extreme values ​​can be used as lower and upper bounds in block 3030 to construct or prepare a uniformly sampled 3D grid or set of sampling values ​​in the unclipped RGB domain or color space.

[0361] FIG. 2P shows an example distribution of R.709 RGB values ​​(in an SDR RGB image) converted from R.709 YCbCr values. As shown in the figure, the distribution of R.709 RGB values ​​in an SDR RGB image is an irregular shape other than a 3D cube. The irregular shape with a 3D boundary represents the space of the largest supported SDR RGB colors that can be mapped back, or back-reshaped, to reconstructed HDR colors in the reconstructed HDR domain or color space without losing information. Any SDR RGB colors outside this irregular shape or range may experience information loss in the back-reshaping operation.

[0362] After an image / video editing operation is performed on an SDR RGB image, the resulting (3D) codeword range or distribution of the edited codewords or colors can be wider (e.g., much wider) than the irregular shape in the SDR RGB color space, resulting in many SDR RGB codewords that are not defined or supported by the backward reshaping operation.

[0363] Furthermore, it may be difficult to characterize, represent, or approximate the actual 3D boundary of an irregular shape using analytical formulas or multiple 2D planes that serve to cut a 3D cube in RGB color space into said irregular shape.

[0364] As mentioned above, a two-level clipping solution can be used to clip the edited codewords back to the maximum supported RGB color space represented by the irregular shapes. More specifically, at the first level:

number

number

[0365] Block 3028 of FIG. 3N is a set of SDR RGB codewords.

number

number

[0366] The alphaShape object created to represent the alphaShape is a set of SDR RGB codewords.

number

number

[0367] The function f that constructs the alphaShape αS (Φ,r α ) where r α represents the radius parameter.

[0368] A set of SDR RGB codewords

number

number

[0369] Figures 2Q to 2T show different r α A set of SDR RGB points or codewords in

number

[0370] alphaShape provides a bounding polyhedron as a clipping boundary for an irregular shape, allowing to determine whether a 3D point represented by an SDR RGB codeword is inside or outside the irregular shape.

[0371] I αS Let (αS,x) denote the binary function used to determine whether a 3D point or SDR RGB codeword (denoted as x) is inside an irregular shape. Let a returned binary value of "1" indicate it is inside the irregular shape, and a returned binary value of "0" indicate it is outside.

[0372] NN αS Let (αS, x) denote an index function that takes a given query 3D point or SDR RGB codeword, such as x, as a second input parameter and returns the nearest neighbor point in αS for the query 3D point or SDR RGB codeword x.

[0373] As mentioned above, a boundary clipping 3D-LUT can be constructed within the SDR RGB domain or color space to clip any SDR RGB codewords outside an irregular shape representing the maximum supported color space by (TPB-based forward and backward) reshaping operations.

[0374] Block 3030 includes constructing the boundary clipping 3D-LUT as a full-grid 3D-LUT with multiple nodes / entries that contain the query SDR codeword as a lookup key and the returned SDR codeword for the query SDR codeword as a value.

[0375] The extrema determined for each color channel (at block 3026 in FIG. 3N)

number

number

number

[0376] Therefore, the total number of query SDR codewords represented in the 3D-LUT is N u =N R N G N B It may be given as:

[0377] Block 3032 includes beginning execution of a node processing loop for each node / entry in the 3D-LUT by selecting a current node / entry from among multiple nodes / entries in the 3D-LUT (e.g., in a sequential or non-sequential loop / iterative order, etc.).

[0378] For simplicity, (i,j,k) in the above equation (103) can be vectorized as p. The current node / entry may be the p-th node / entry among multiple nodes / entries in the 3D-LUT. The lookup key for the p-th node / entry is the u on the left side of the above equation (103). p The query SDR RGB codeword may be represented by the pth represented query SDR RGB codeword denoted as p Let denote the current output clipped SDR RGB codeword or value returned by the p-th node / entry of the 3D-LUT when the p-th representation of the query SDR RGB codeword is given as the lookup key.

[0379] Block 3034 finds the current, or pth, represented query SDR RGB codeword u for the current node / entry. p is the alphaShape αS generated in block 3028. (R709) (or a shape constructed using the alpha shape construction function as shown in equation (102) above). The current or pth represented query SDR RGB codeword u for the current node / entry is p In response to determining that is within the alphaShape, process flow continues to block 3036. Otherwise, process flow continues to block 3040.

[0380] The nodes / entries in the 3D-LUT contain a subset of nodes or entries, each of which has an alphaShape αS (R709) Each lookup key is specified by a query SDR RGB codeword expressed in . Thus, a subset of nodes or entries is represented by alphaShape αS (R709) Contains the nodes or entries that are considered to be within

[0381] Block 3040 calculates the alphaShape αS for the current node / entry. (R709) The closest node / entry is the closest represented query SDR RGB codeword (u with bar) p ) as a lookup key, which finds the closest represented query SDR RGB codeword (u p ] is alphaShape αS (R709) Compare the current or p-th represented query SDR RGB codeword u with all other represented query SDR RGB codewords for all other nodes / entries in the subset of nodes or entries in p has a minimum distance (measured as Euclidean or non-Euclidean distance in SDR RGB color space) to

[0382] In some operating scenarios, the closest represented query SDR RGB codeword (u with a bar) for the current or p-th represented query SDR RGB codeword up p ] is the index function NN αS It may be given as follows using (αS,x):

number

[0383] The closest represented query SDR RGB codeword (with a bar) is the current or pth represented query SDR RGB codeword u p This boundary clipping value (u with a bar) represents the boundary clipping value for p ] is set to the return value for the current or pth node entry in the 3D-LUT, while the current or pth represented query SDR RGB codeword up is set to the lookup key for the current or pth node entry in the 3D-LUT.

[0384] Block 3036 calculates the current or pth represented query SDR RGB codeword u p as a return value for the current, or pth, node entry in the 3D-LUT (in addition to being the lookup key), and determining whether the current, or pth node / entry is the last node or entry among the multiple nodes / entries in the 3D-LUT. In response to determining that the current, or pth node / entry is the last node or entry among the multiple nodes / entries in the 3D-LUT, process flow proceeds to block 3038. Otherwise, process flow returns to block 3032.

[0385] Block 3038 calculates the final boundary clipping 3D-LUT (f 3DLUT αS This involves outputting the 3D-LUT as a 3D-LUT (denoted as ()).

[0386] Final boundary clipping f 3DLUT αS An exemplary procedure for generating () is shown in Table 5 below. [Table 5]

[0387] (Final) boundary clipping 3D-LUT f 3DLUT αSGiven (), boundary clipping can be performed relatively efficiently on the (e.g., edited, etc.) SDR image using the two-level solution described above. As described above, regular clipping can be performed to ensure that all (e.g., edited, etc.) SDR RGB codewords in the SDR image are within the extreme values ​​or upper / lower limits for the SDR RGB codewords in the SDR RGB color space in which the image / video editing operation on the SDR image was performed. Regular clipping is followed by irregular clipping, which uses the (regularly clipped, if applicable) SDR codewords as lookup keys to find the (final) boundary clipping 3D-LUT f 3DLUT αS (), boundary clipping, and the (final) boundary clipping 3D-LUT f 3DLUT αS By using the returned value from () as the output (further irregularly clipped, if applicable) SDR codeword in the output (e.g., clipped, edited, etc.) SDR image.

[0388] In various operating scenarios, including but not limited to those shown in Figures 3I to 3M, regular and / or irregular clipping operations as described herein can be applied in video capture and / or editing applications to ensure maximized support for reconstructing HDR images from SDR images and to prevent / reduce visual artifacts in the reconstructed HDR images.

[0389] Process Flow Example FIG. 4A illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., an encoding device / module, a transcoding device / module, a decoding device / module, an inverse tone mapping device / module, a tone mapping device / module, a media device / module, an inverse mapping generation and application system, etc.) may perform this process flow. In block 402, the system described herein constructs sampled high dynamic range (HDR) color space points distributed throughout the HDR color space. The HDR color space is parameterized by primary color scaling parameters having candidate values ​​selected from among multiple candidate values. The primary color scaling parameters are used to calculate the color space coordinates of at least one of the multiple primary colors that describe the HDR color space.

[0390] In block 404, the system generates from the sampled HDR color space points in the HDR color space: (a) reference standard dynamic range (SDR) color space points represented in the reference SDR color space, (b) input HDR color space points represented in the input HDR color space, and (c) reference HDR color space points represented in the reference HDR color space.

[0391] In block 406, the system executes a reshaping operation optimization algorithm to generate a chain of optimized forward reshaping mappings and optimized backward reshaping mappings. The reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs.

[0392] In an embodiment, an optimized forward reshaping mapping is used to forward reshape an input HDR image in an input HDR color space into a forward reshaped SDR image in a forward reshaped SDR color space, and an optimized backward reshaping mapping is used to backward reshape the forward reshaped SDR image in the forward reshaped SDR color space into a backward reshaped HDR image.

[0393] In one embodiment, the sampled HDR color space points are constructed in the HDR color space without using any images.

[0394] In one embodiment, a plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for a plurality of candidate values ​​for the primary color scaling parameters; each chain in the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings includes a respective optimized forward reshaping mapping and a respective optimized backward reshaping mapping.

[0395] In an embodiment, the sampled HDR color space points are mapped to reference SDR color space points based, at least in part, on a predefined HDR-to-SDR mapping.

[0396] In one embodiment, multiple sets of prediction errors are calculated for multiple chains of optimized forward reshaping mappings and optimized backward reshaping mappings; each set of prediction errors in the multiple sets of prediction errors is calculated for a respective chain in the multiple chains of optimized forward reshaping mappings and optimized backward reshaping mappings; and the multiple sets of prediction errors are used to select a particular candidate value for the primary color scaling parameter from multiple candidate values.

[0397] In one embodiment, specific candidate values ​​for the primary color scaling parameters are used to generate a specific chain of specific optimized forward reshaping mappings and specific optimized backward reshaping mappings.

[0398] In one embodiment, the particular optimized forward reshaping mapping is represented in a forward reshaping 3D lookup table.

[0399] In one embodiment, the particular optimized posterior reshaping mapping is represented in a posterior reshaping three-dimensional lookup table.

[0400] In one embodiment, a video encoder applies the optimized forward reshaping mapping to a sequence of input HDR images to generate a sequence of forward reshaped SDR images, and encodes the sequence of forward reshaped SDR images into a video signal.

[0401] In one embodiment, a video decoder decodes a sequence of forward reshaped SDR images from a video signal and applies an optimized backward reshaped mapping to the sequence of forward reshaped SDR images to generate a sequence of backward reshaped HDR images.

[0402] In one embodiment, a sequence of display images derived from the sequence of backward reshaped HDR images is rendered on an image display operating in conjunction with a video decoder.

[0403] In one embodiment, the HDR color space and the input HDR color space share a common white point.

[0404] In one embodiment, the reshaping motion optimization algorithm represents a Backward Error Subtraction for Signal Conditioning (BESA) algorithm.

[0405] FIG. 4B illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., an encoding device / module, a transcoding device / module, a decoding device / module, an inverse tone mapping device / module, a tone mapping device / module, a media device / module, an inverse mapping generation and application system, etc.) may perform this process flow. In block 422, the system described herein constructs sampled HDR color space points distributed throughout the HDR color space. The HDR color space is parameterized by primary color scaling parameters having candidate values ​​selected from among multiple candidate values. The primary color scaling parameters are used to calculate the color space coordinates of at least one of the multiple primary colors that describe the HDR color space.

[0406] In block 424, the system generates from the sampled HDR color space points in the HDR color space: (a) input SDR color space points represented in the input SDR color space, and (b) reference HDR color space points represented in the reference HDR color space.

[0407] In block 426, the system executes a reshaping motion optimization algorithm to generate an optimized backward reshaping mapping. The reshaping motion optimization algorithm receives as input the input SDR color space points and the reference HDR color space points.

[0408] In one embodiment, the backward reshaping mapping is used to backward reshape an SDR image in an input SDR color space into a backward reshaped HDR image.

[0409] In one embodiment, the sampled HDR color space points are constructed in the HDR color space without any images.

[0410] In one embodiment, a plurality of optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for a plurality of candidate values ​​of the primary color scaling parameters; each optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings comprises a respective optimized backward reshaping mapping.

[0411] In one embodiment, multiple sets of prediction errors are calculated for multiple optimized backward reshaping mappings; each set of prediction errors in the multiple sets of prediction errors is calculated for a respective optimized backward reshaping mapping in the multiple optimized backward reshaping mappings; and the multiple sets of prediction errors are used to select a particular candidate value for the primary color scaling parameter from among multiple candidate values.

[0412] In an embodiment, the sampled HDR color space points are processed by a programmable ISP pipeline into input SDR color space points based at least in part on optimized values ​​for programmable configuration parameters of the programmable ISP pipeline.

[0413] In one embodiment, optimized values ​​for the programmable configuration parameters of the programmable ISP pipeline are determined by minimizing the approximation error between an ISP SDR image generated by the programmable ISP pipeline from an HDR image and a reference SDR image generated by applying a predefined HDR-to-SDR mapping to the same HDR image.

[0414] 4C illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., an encoding device / module, a transcoding device / module, a decoding device / module, an inverse tone mapping device / module, a tone mapping device / module, a media device / module, an inverse mapping generation and application system, etc.) may perform this process flow. In block 442, the system described herein extracts a set of SDR image features from the training SDR image and a set of HDR image features from the training HDR image.

[0415] At block 444, the system matches a subset of one or more SDR image features in the set of SDR image features with a subset of one or more HDR image features in the set of HDR image features.

[0416] At block 446, the system generates a geometric transformation to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image using a subset of one or more SDR image feature points and a subset of one or more HDR image feature points.

[0417] In block 448, the system determines a set of SDR and HDR color patch pairs from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and HDR image have been spatially aligned by a geometric transformation.

[0418] In block 450, the system generates an optimized SDR-HDR mapping based at least in part on a set of SDR and HDR color patch pairs derived from the training SDR and HDR images.

[0419] In block 452, the system applies the optimized SDR-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0420] In an embodiment, training SDR images and training HDR images are captured from a three-dimensional (3D) visual scene by a capture device operating in SDR and HDR capture modes, respectively.

[0421] In an embodiment, the training SDR image and the training HDR image form training SDR image and HDR image pairs in a plurality of pairs of training SDR image and HDR image; and the optimized SDR-HDR mapping is generated based at least in part on a plurality of sets of SDR and HDR color patch pairs derived from the plurality of pairs of training SDR image and HDR image.

[0422] In one embodiment, each SDR image feature point in the subset of one or more SDR image features is matched with a respective HDR image feature point in the subset of one or more HDR image features; the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm.

[0423] In one embodiment, the common feature point extraction algorithm represents one of a binary robust invariant scalable keypoint algorithm, an accelerated feature from segment test algorithm, a KAZE algorithm, a minimum eigenvalue algorithm, a maximum stable extremum region algorithm, a directed FAST and rotation algorithm, a scale invariant feature transformation algorithm, a speeded up robust feature algorithm, etc.

[0424] 4D illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., an encoding device / module, a transcoding device / module, a decoding device / module, an inverse tone mapping device / module, a tone mapping device / module, a media device / module, an inverse mapping generation and application system, etc.) may perform this process flow. At block 462, the system described herein performs a respective camera distortion correction operation on each training image of the training SDR image and training HDR image pair to generate a respective undistorted image in the undistorted training SDR image and undistorted training HDR image pair.

[0425] In block 464, the system generates a respective projective transformation for each pair of SDR and HDR image projective transformations using the corner pattern marks detected from each undistorted image in the pair of undistorted training SDR and HDR images.

[0426] In block 466, the system applies each projective transformation in the pair of SDR image projective transformation and HDR image projective transformation to a respective undistorted image in the pair of undistorted training SDR image and undistorted training HDR image to generate a respective rectified image in the pair of rectified training SDR image and rectified training HDR image.

[0427] In block 468, the system extracts a set of SDR color patches from the modified training SDR image and a set of HDR color patches from the modified training HDR image.

[0428] In block 470, the system generates an optimized SDR-HDR mapping based at least in part on the set of SDR and HDR color patches derived from the training SDR and HDR images.

[0429] At block 472, the system applies the optimized SDR-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0430] In an embodiment, the training SDR image and the training HDR image are captured from a common color target image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively.

[0431] In one embodiment, the common color chart image is selected from a plurality of color chart images, each of which includes a distinct distribution of color patches arranged on a two-dimensional color chart.

[0432] In one embodiment, distinct distributions of color patches are generated using random colors randomly selected from a common statistical distribution having a particular combination of statistical mean and variance.

[0433] In one embodiment, the common color chart image is rendered on the screen of a common reference image display, from which it is captured by the first and second capture devices.

[0434] In one embodiment, each camera distortion correction operation is based, at least in part, on camera-specific distortion coefficients generated from a camera calibration process performed on the camera used to acquire the training images.

[0435] In one embodiment, the set of SDR color patches and the set of HDR color patches are used to derive a three-dimensional mapping table (3DMT), and an optimized SDR-HDR mapping is generated based at least in part on the 3DMT.

[0436] In an embodiment, the optimized SDR-HDR mapping represents one of a TPB-based mapping or a non-TPB-based mapping.

[0437] In one embodiment, the optimized SDR-HDR mapping is one of a static mapping applied to all non-training SDR images represented in the video signal, or a dynamic mapping generated based at least in part on a particular value distribution of the SDR codewords of one of the non-training SDR images represented in the video signal.

[0438] 4E illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) may perform this process flow. In block 482, the system described herein constructs sampled HDR color space points distributed throughout the HDR color space used to represent the reconstructed HDR image.

[0439] At block 484, the system converts the sampled HDR color space points to SDR color space points in a first SDR color space in which the SDR image to be edited by the editing device is represented.

[0440] In block 486, the system determines a bounding SDR color space rectangle based on the SDR codeword extrema of the SDR color space points in the first SDR color space and determines an irregular 3D shape from the distribution of the SDR color space points.

[0441] In block 488, the system constructs sampled SDR color space points distributed throughout a bounding SDR color space rectangle in the first SDR color space.

[0442] In block 490, the system uses the sampled SDR color space points and the irregular shape to generate a boundary clipping 3D-LUT, which uses the sampled SDR color space points as lookup keys.

[0443] In block 492, the system performs a clipping operation on the edited SDR image in the first SDR color space based at least in part on the boundary clipping 3D-LUT to generate a boundary-clipped edited SDR image in the first SDR color space.

[0444] In one embodiment, the clipping operation includes first performing regular clipping on the edited SDR image using a bounding SDR color space rectangle to generate a normally clipped edited SDR image, and then performing irregular clipping on the normally clipped edited SDR image using a 3D-LUT to generate a bounded clipped edited SDR image.

[0445] In one embodiment, a set of one or more SDR pixels in an SDR image to be edited are edited from one or more first luma values ​​to one or more second luma values ​​in the edited image, where the one or more second luma values ​​are different from the one or more first luma values.

[0446] In one embodiment, a set of one or more SDR pixels in an SDR image to be edited are edited from one or more first chrominance values ​​to one or more second chrominance values ​​in the edited image; the one or more second chrominance values ​​are different from the one or more first chrominance values.

[0447] In one embodiment, image details that were depicted in the SDR image being edited are removed in the edited SDR image.

[0448] In one embodiment, image details not depicted in the SDR image being edited are added in the edited SDR image.

[0449] In one embodiment, the 3D-LUT includes one or more nodes, each including a lookup key and a lookup value; the lookup key is equal to the lookup value, and the lookup key is within the irregular shape.

[0450] In one embodiment, the 3D-LUT includes one or more nodes, each including a lookup key and a lookup value; the lookup key is outside the irregular shape and the lookup value is inside the irregular shape.

[0451] In one embodiment, the lookup value is determined based on an index function that takes the irregular shape and the lookup key as input and returns the nearest neighbor of the lookup key as output.

[0452] In some embodiments, a computing device, such as a display device, a mobile device, a set-top box, or a multimedia device, is configured to perform any of the aforementioned methods. In some embodiments, the device comprises a processor and is configured to perform any of the aforementioned methods. In some embodiments, a non-transitory computer-readable storage medium storing software instructions that, when executed by one or more processors, result in the performance of any of the aforementioned methods.

[0453] In one embodiment, a computing device comprising one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the aforementioned methods.

[0454] It should be noted that although separate embodiments are discussed herein, any combination of the embodiments and / or sub-embodiments discussed herein may be combined to form further embodiments.

[0455] Exemplary Computer System Implementation Embodiments of the present invention may be implemented using computer systems, systems configured in electronic circuits and components, integrated circuit (IC) devices such as microcontrollers, field programmable gate arrays (FPGAs), or other configurable or programmable logic devices (PLDs), discrete-time or digital signal processors (DSPs), application-specific ICs (ASICs), and / or apparatuses including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or execute instructions related to adaptive perceptual quantization of images with enhanced dynamic range, such as those described herein. The computer and / or IC may calculate any of a variety of parameters or values ​​associated with the adaptive perceptual quantization process described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0456] Certain implementations of the present invention include computer processors that execute software instructions that cause the processor to perform the methods of the present disclosure. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. can implement methods related to adaptive perceptual quantization of HDR images, as described above, by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention may be provided in the form of a program product. A program product may include any non-transitory medium that carries a set of computer-readable signals containing instructions that, when executed by a data processor, cause the data processor to perform the methods of embodiments of the present invention. Program products according to embodiments of the present invention may be in any of a wide variety of forms. Program products may include physical media, such as magnetic data storage media including floppy diskettes and hard disk drives, optical data storage media including CD-ROMs and DVDs, ROMs, and electronic data storage media including flash RAM. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0457] Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise specified, the reference to that component (including the reference to "means") should be interpreted as including any component that performs the function of (e.g., is functionally equivalent to) the described component as an equivalent of that component, including components that are not structurally equivalent to the disclosed structure that performs the function in exemplary embodiments of the invention.

[0458] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) permanently programmed to perform the techniques, or may include one or more general-purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination thereof. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to implement the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices incorporating hard-wired logic and / or program logic to implement the techniques.

[0459] 5 is a block diagram illustrating a computer system 500 in which an embodiment of the present invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0460] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions executed by processor 504. Main memory 506 may be used to store temporary variables or other intermediate information during execution of instructions executed by processor 504. Such instructions, when stored on a non-transitory storage medium accessible to processor 504, make computer system 500 a special-purpose machine customized to perform the operations specified in the instructions.

[0461] Computer system 500 further includes a read-only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0462] Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to bus 502 for communicating information and command selections to processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 504 and for controlling cursor movement on display 512. This input device typically has two degrees of freedom in two axes, i.e., a first axis (e.g., x) and a second axis (e.g., y), allowing the device to specify a position in a plane.

[0463] Computer system 500 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, when combined with the computer system, renders or programs computer system 500 a special-purpose machine. According to one embodiment, the techniques described herein are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0464] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific fashion. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, and EPROM, FLASH-EPROM, NVRAM, or any other memory chip or cartridge.

[0465] Storage media is distinct from, but may be used in the context of, transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0466] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 500 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on bus 502. Bus 502 carries the data to main memory 506, from which processor 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by processor 504.

[0467] Computer system 500 also includes a communication interface 518 coupled to bus 502. The communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, communication interface 518 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 518 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 518 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0468] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526, in turn, provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 528. Local network 522 and Internet 528 both use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks, the signals on network link 520, and the signals through communication interface 518, which carry the digital data to and from computer system 500, are exemplary forms of transmission media.

[0469] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0470] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0471] Equivalents, Extensions, Substitutes, and Others In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary depending on the implementation. Thus, the sole and exclusive indication of what is, and is intended by applicant to be, a claimed embodiment of the present invention is the set of claims allowed on this application, in the specific form in which such claims are allowed, including any subsequent amendments thereto. Definitions expressly set forth in this specification for terms contained in such claims shall govern the meaning of those terms used in such claims. Thus, no limitations, elements, properties, features, advantages, or attributes not expressly recited in a claim should in any way limit the scope of such claims. Accordingly, the specification and drawings are to be regarded in an illustrative, and not a restrictive, sense.

[0472] Itemized Exemplary Embodiments The present invention may be embodied in any of the forms described herein, including, but not limited to, the following enumerated example embodiments (EEE), which describe the structure, features, and functionality of some portions of embodiments of the present invention.

[0473] [EEE1] constructing sampled high dynamic range (HDR) color space points distributed throughout an HDR color space, the HDR color space being parameterized by primary color scaling parameters having candidate values ​​selected from a plurality of candidate values, the primary color scaling parameters being used to calculate color space coordinates of at least one primary color among a plurality of primary colors defining the HDR color space; generating, from the sampled HDR color space points in the HDR color space, (a) reference standard dynamic range (SDR) color space points represented in a reference SDR color space, (b) input HDR color space points represented in an input HDR color space, and (c) reference HDR color space points represented in a reference HDR color space; running a reshaping operation optimization algorithm to generate a chain of optimized forward reshaping mappings and optimized backward reshaping mappings, the reshaping operation optimization algorithm using the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs; the optimized forward reshaping mapping is used to forward reshape an input HDR image in the input HDR color space into a forward reshaped SDR image in a forward reshaped SDR color space, and the optimized backward reshaping mapping is used to backward reshape the front reshaped SDR image in the front reshaped SDR color space into a backward reshaped HDR image. method. [EEE2] The method of EEE1, wherein the sampled HDR color space points are constructed in the HDR color space without using any images. [EEE3] The method of any one of EEE1 and EEE2, wherein a plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for the plurality of candidate values ​​for the primary color scaling parameters; and each chain in the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings includes a respective optimized forward reshaping mapping and a respective optimized backward reshaping mapping. [EEE4] 4. The method of any one of EEE1 to EEE3, wherein the sampled HDR color space points are mapped to the reference SDR color space points based, at least in part, on a predefined HDR-SDR mapping. [EEE5] 5. The method of any one of EEE1 to EEE4, wherein a plurality of sets of prediction errors are calculated for the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings; each set of prediction errors in the plurality of sets of prediction errors is calculated for a respective chain in the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings; and the plurality of sets of prediction errors are used to select a particular candidate value for the primary color scaling parameter from the plurality of candidate values. [EEE6] The method according to EEE5, wherein the particular candidate values ​​for the primary color scaling parameters are used to generate a particular chain of a particular optimized forward reshaping mapping and a particular optimized backward reshaping mapping. [EEE7] The method of EEE6, wherein the particular optimized forward reshaping mapping is represented in a forward reshaping 3D lookup table. [EEE8] 8. The method of claim 6 or 7, wherein the particular optimized posterior reshaping mapping is represented in a posterior reshaping three-dimensional lookup table. [EEE9] 9. The method of any one of EEE6 to 8, wherein a video encoder applies the optimized forward reshaping mapping to a sequence of input HDR images to generate a sequence of forward reshaped SDR images, and encodes the sequence of forward reshaped SDR images into a video signal. [EEE10] 10. The method of any one of EEE6 to 9, wherein a video decoder decodes a sequence of forward reshaped SDR images from a video signal and applies the optimized backward reshaping mapping to the sequence of forward reshaped SDR images to generate a sequence of backward reshaped HDR images. [EEE11] The method of claim 8, wherein a sequence of display images derived from the sequence of backward reshaped HDR images is rendered on an image display operating in conjunction with the video decoder. [EEE12] 11. The method of any one of EEE1 to 10, wherein the HDR color space and the input HDR color space share a common white point. [EEE13] The method of any one of EEE1 to EEE10, wherein said reshaping operation optimization algorithm represents a Backward Error Subtraction for Signal Conditioning with Neutral Color Preservation (BESA) algorithm. [EEE14] constructing sampled high dynamic range (HDR) color space points distributed throughout an HDR color space, the HDR color space being parameterized by primary color scaling parameters having candidate values ​​selected from a plurality of candidate values, the primary color scaling parameters being used to calculate color space coordinates of at least one primary color among a plurality of primary colors defining the HDR color space; generating, from the sampled HDR color space points in the HDR color space, (a) input standard dynamic range (SDR) color space points represented in an input SDR color space, and (b) reference HDR color space points represented in a reference HDR color space; running a reshaping operation optimization algorithm to generate an optimized backward reshaping mapping, the reshaping operation optimization algorithm receiving the input SDR color space points and the reference HDR color space points as inputs; the optimized backward reshaping mapping is used to backward reshape an SDR image in the input SDR color space into a backward reshaped HDR image; method. [EEE15] The method of claim 8, wherein the sampled HDR color space points are constructed in the HDR color space without any images. [EEE16] 16. The method of claim 14 or 15, wherein a plurality of optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for the plurality of candidate values ​​for the primary color scaling parameters; and each optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings comprises a respective optimized backward reshaping mapping. [EEE17] 17. The method of any one of EEE14 to 16, wherein a plurality of sets of prediction errors are calculated for the plurality of optimized backward reshaping mappings; each set of prediction errors in the plurality of sets of prediction errors is calculated for a respective optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings; and the plurality of sets of prediction errors are used to select a particular candidate value for the primary color scaling parameter from the plurality of candidate values. [EEE18] 18. The method of any one of EEE14 to 17, wherein the sampled HDR color space points are processed by a programmable image signal processor (ISP) pipeline into the input SDR color space points based at least in part on optimized values ​​for programmable configuration parameters of the programmable image signal processor (ISP) pipeline. [EEE19] 19. The method of any one of EEE14 to 18, wherein the optimized values ​​for the programmable configuration parameters of the programmable ISP pipeline are determined by minimizing an approximation error between an ISP SDR image generated by the programmable ISP pipeline from an HDR image and a reference SDR image generated by applying a predefined HDR-SDR mapping to the same HDR image. [EEE20] 20. The method of any one of EEE1 to 19, wherein the distribution of sampled HDR color space points follows a predetermined pattern of the sample HDR color space points in the HDR color space. [EEE21] 21. The method of any one of EEE1 to EEE20, wherein at least one primary color of the plurality of primaries defining the HDR color space corresponds to a linear combination of this primary color in (i) an SDR color space and (ii) an HDR color space, and at least one weighting factor of the linear combination is based on the primary color scaling parameter. [EEE22] 22. The method of any one of EEE1 to EEE21, wherein the constructing, generating and executing operations are performed for each candidate value of the primary color scaling parameters from the plurality of candidate values. [EEE23] The method of claim EEE22, wherein a particular candidate value from the plurality of candidate values ​​for the primary color scaling parameters is selected based on a comparison of results of running the reshaping operation optimization algorithm for each candidate value from the plurality of candidate values. [EEE24] 24. The method of any one of EEE1 to 23, wherein the reshaping operation optimization algorithm strikes a balance between achieving a wide HDR color space of the backward reshaped HDR image and achieving low SDR color deviation. [EEE25] The method of any one of EEE5 and EEE17, wherein the particular candidate value from among the plurality of candidate values ​​for the color primary scaling parameter is selected to achieve a balance between minimized prediction error and minimized SDR color deviation. [EEE26] The method of any one of EEE5 and EEE17, wherein the particular candidate value from among the plurality of candidate values ​​for the color primary scaling parameters is selected to achieve a minimized prediction error with SDR color deviation below a predetermined threshold. [EEE27] The method of any one of EEE1 to EEE26, wherein the optimized backward reshaping mapping is a tensor product B-spline (TPB) based mapping, and the reshaping operation optimization algorithm generates at least one optimized TPB coefficient for this TPB mapping. [EEE28] Extracting a set of standard dynamic range (SDR) image feature points from the training SDR images and extracting a set of high dynamic range (HDR) image feature points from the training HDR images; matching a subset of one or more SDR image features in the set of SDR image features with a subset of one or more HDR image features in the set of HDR image features; generating a geometric transformation using the subset of one or more SDR image feature points and the subset of one or more HDR image feature points to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image; determining a set of SDR and HDR color patch pairs from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and HDR image have been spatially aligned by the geometric transformation; generating an optimized SDR-HDR mapping based at least in part on the set of SDR and HDR color patch pairs derived from the training SDR image and the training HDR image; applying the optimized SDR-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images. method. [EEE29] The method of EEE28, wherein the training SDR images and the training HDR images are captured from a three-dimensional (3D) visual scene by a capture device operating in SDR and HDR capture modes, respectively. [EEE30] 30. The method of claim 38, wherein the training SDR image and the training HDR image form training SDR and HDR image pairs in a plurality of pairs of training SDR and HDR images; and the optimized SDR-HDR mapping is generated based at least in part on a plurality of sets of SDR and HDR color patch pairs derived from the plurality of pairs of training SDR and HDR images. [EEE31] 31. The method of any one of EEE28 to EEE30, wherein each SDR image feature point in the subset of one or more SDR image feature points is matched with a respective HDR image feature point in the subset of one or more HDR image feature points; and the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm. [EEE32] The method according to EEE31, wherein the common feature point extraction algorithm represents one of a binary robust invariant scalable keypoint algorithm, an accelerated feature from segment test algorithm, a KAZE algorithm, a minimum eigenvalue algorithm, a maximum stable extremum region algorithm, a directed FAST and rotation algorithm, a scale invariant feature transformation algorithm, or a speeded up robust feature algorithm, etc. [EEE33] performing a respective camera distortion correction operation on each training image in the pair of training standard dynamic range (SDR) image and training high dynamic range (HDR) image to generate a respective undistorted image in the pair of undistorted training SDR image and undistorted training HDR image; generating each projective transformation in a pair of SDR and HDR image projective transformations using corner pattern marks detected from each undistorted image in the pair of undistorted training SDR and HDR images; applying each projective transformation in the pair of SDR image projective transformation and HDR image projective transformation to a respective undistorted image in the pair of undistorted training SDR image and undistorted training HDR image to generate a respective rectified image in the pair of rectified training SDR image and rectified training HDR image; extracting a set of SDR color patches from the modified training SDR image and extracting a set of HDR color patches from the modified training HDR image; generating an optimized SDR-HDR mapping based at least in part on the set of SDR color patches and the set of HDR color patches derived from the training SDR image and the training HDR image; applying the optimized SDR-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images. method. [EEE34] The method of EEE33, wherein the training SDR images and the training HDR images are captured from a common color target image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively. [EEE35] The method of EEE34, wherein the common color chart image is selected from a plurality of color chart images, each comprising a distinct distribution of color patches arranged on a two-dimensional color chart. [EEE36] The method according to EEE35, wherein said distinct distributions of color patches are generated using random colors randomly selected from a common statistical distribution having a particular combination of statistical mean and variance. [EEE37] 37. The method of any one of EEE33 to 36, wherein the common color chart image is rendered on a screen of a common reference image display from which it is captured by the first capture device and the second capture device. [EEE38] 38. The method of any one of EEE33 to 37, wherein the respective camera distortion correction operations are based, at least in part, on camera-specific distortion coefficients generated from a camera calibration process performed on the camera used to acquire the training images. [EEE39] 39. The method of any one of EEE33 to 38, wherein the set of SDR color patches and the set of HDR color patches are used to derive a three-dimensional mapping table (3DMT), and the optimized SDR-HDR mapping is generated based at least in part on the 3DMT. [EEE40] 40. The method of any one of EEE33 to 39, wherein the optimized SDR-HDR mapping represents one of a Tensor Product B-Spline (TPB) based mapping or a non-TPB based mapping. [EEE41] 41. The method of any one of EEE33 to 40, wherein the optimized SDR-HDR mapping is one of a static mapping applied to all non-training SDR images represented in the video signal, or a dynamic mapping generated based at least in part on a particular value distribution of SDR codewords of certain non-training SDR images represented in the video signal. [EEE42] constructing sampled high dynamic range (HDR) color space points distributed throughout the HDR color space used to represent the reconstructed HDR image; converting the sampled HDR color space points to standard dynamic range (SDR) color space points in a first SDR color space in which an SDR image to be edited by an editing device is represented; determining a bounding SDR color space rectangle based on extrema of SDR codeword values ​​of the SDR color space points in the first SDR color space, and determining an irregular three-dimensional (3D) shape from the distribution of the SDR color space points; constructing sampled SDR color space points distributed throughout the bounding SDR color space rectangle in the first SDR color space; generating a boundary clipping 3D lookup table (3D-LUT) using the sampled SDR color space points and the irregular shape, the boundary clipping 3D-LUT using the sampled SDR color space points as lookup keys; performing a clipping operation on the edited SDR image in the first SDR color space based at least in part on the boundary clipping 3D-LUT to generate a boundary-clipped edited SDR image in the first SDR color space. method. [EEE43] 8. The method of claim 6, wherein the clipping operation comprises first performing regular clipping on the edited SDR image using the bounding SDR color space rectangle to generate a regular clipped edited SDR image, and then performing irregular clipping on the regular clipped edited SDR image using the 3D-LUT to generate the boundary clipped edited SDR image. [EEE44] 8. The method of claim 42 or 43, wherein a set of one or more SDR pixels in the SDR image to be edited are edited from one or more first luminance values ​​to one or more second luminance values ​​in the edited image, the one or more second luminance values ​​being different from the one or more first luminance values. [EEE45] 45. The method of any one of EEE42 to 44, wherein a set of one or more SDR pixels in the SDR image to be edited are edited from one or more first chrominance values ​​to one or more second chrominance values ​​in the edited image; and the one or more second chrominance values ​​are different from the one or more first chrominance values. [EEE46] 46. ​​The method of any one of EEE42 to 45, wherein image details that were depicted in the SDR image to be edited are removed in the edited SDR image. [EEE47] 47. The method of any one of EEE42 to 46, wherein image details not depicted in the SDR image to be edited are added in the edited SDR image. [EEE48] The method of any one of EEE42 to 47, wherein the 3D-LUT comprises one or more nodes each comprising a lookup key and a lookup value; the lookup key is equal to the lookup value, and the lookup key is within the range of the irregular shape. [EEE49] 49. The method of any one of EEE42 to 48, wherein the 3D-LUT comprises one or more nodes, each node comprising a lookup key and a lookup value; the lookup key being outside the irregular shape and the lookup value being within the irregular shape. [EEE50] The method of claim 39, wherein the lookup value is determined based on an index function that takes the irregular shape and the lookup key as input and returns as output a nearest neighbor of the lookup key. [EEE51] 8. An apparatus having a processor and configured to perform a method according to any one of EEE1 to EEE50. [EEE52] A non-transitory computer-readable storage medium storing computer-executable instructions for carrying out a method on one or more processors according to the method of any one of EEE1 to EEE50. [EEE53] A computer system configured to carry out the method of any one of EEE1 to EEE50.

Claims

1. 1. A method for tensor product B-spline (TPB) based image reshaping of high dynamic range (HDR) input images, the method comprising: constructing sampled HDR color space points distributed throughout an HDR color space, wherein sampling the HDR color space points includes sampling each primary color of the color space by a predetermined number of units to obtain a sampled dataset of color patches representing a subset of the HDR color space, the subset of the HDR color space being defined as a triangle formed by the primary colors in a color coordinate system and parameterized by primary color scaling parameters having candidate values ​​iteratively selected from a plurality of candidate values, the primary color scaling parameters serving as input parameters for implementing a reshaping operation optimization algorithm for TPB-based image reshaping, the primary color scaling parameters indicating a size of the subset of the HDR color space; generating, from the sampled HDR color space points in the HDR color space, (a) reference standard dynamic range (SDR) color space points represented in a reference SDR color space, (b) input HDR color space points represented in an input HDR color space, and (c) reference HDR color space points represented in a reference HDR color space; executing the reshaping operation optimization algorithm to optimize forward and backward TPB predictors in a pipeline of chained reshaping functions including a forward reshaping mapping and a corresponding backward reshaping mapping, wherein the reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs to generate an optimized forward reshaping mapping and a corresponding optimized backward reshaping mapping based on candidate values ​​of the primary color scaling parameters corresponding to a largest subset of the HDR color space that can minimize a weighted sum of HDR error and SDR error; the optimized forward reshaping mapping is used to forward reshape an input HDR image in the input HDR color space into a forward reshaped SDR image in a forward reshaped SDR color space, and the optimized backward reshaping mapping is used to backward reshape the front reshaped SDR image in the front reshaped SDR color space into a backward reshaped HDR image. method.

2. 2. The method of claim 1 , wherein the execution of the reshaping operation optimization algorithm is processed iteratively to generate a plurality of forward reshaping mappings and corresponding backward reshaping mappings, each forward reshaping mapping and corresponding backward reshaping mapping corresponding to a respective one of the plurality of candidate values ​​for the primary color scaling parameters.

3. The method of claim 1 , wherein the sampled HDR color space points are mapped to the reference SDR color space points based, at least in part, on a predefined HDR-to-SDR mapping.

4. 2. The method of claim 1 , wherein a set of prediction errors is calculated for each forward reshaping mapping and a corresponding backward reshaping mapping; and the set of prediction errors is used to select a particular candidate value from among the plurality of candidate values ​​for the primary color scaling parameters.

5. The method of claim 1 , wherein the HDR color space and the input HDR color space share a common white point.

6. The method of claim 1 , wherein the reshaping motion optimization algorithm represents a Backward Error Subtraction for Signal Conditioning with Neutral Color Preservation (BESA) algorithm.

7. 1. A method for tensor product B-spline (TPB) based image reshaping of standard dynamic range (SDR) input images, the method comprising: constructing sampled HDR color space points distributed throughout an HDR color space, wherein sampling the HDR color space points includes sampling each primary color of the color space by a predetermined number of units to obtain a sampled dataset of color patches representing a subset of the HDR color space, the subset of the HDR color space being defined as a triangle formed by the primary colors in a color coordinate system and parameterized by primary color scaling parameters having candidate values ​​iteratively selected from a plurality of candidate values, the primary color scaling parameters serving as input parameters for implementing a reshaping operation optimization algorithm for TPB-based image reshaping, the primary color scaling parameters indicating a size of the subset of the HDR color space; generating, from the sampled HDR color space points in the HDR color space, (a) input standard dynamic range (SDR) color space points represented in an input SDR color space, and (b) reference HDR color space points represented in a reference HDR color space; executing the reshaping operation optimization algorithm to optimize a backward reshaping mapping, the reshaping operation optimization algorithm receiving as input the input SDR color space points and the reference HDR color space points, and generating an optimized backward reshaping mapping based on candidate values ​​of the primary color scaling parameters corresponding to a largest subset of the HDR color space that can achieve a minimized HDR error; the optimized backward reshaping mapping is used to backward reshape an SDR image in the input SDR color space into a backward reshaped HDR image; method.

8. 8. The method of claim 7, wherein the execution of the reshaping operation optimization algorithm is processed iteratively to generate a plurality of backward reshaping mappings, each backward reshaping mapping corresponding to a respective one of the plurality of candidate values ​​for the primary color scaling parameters.

9. 8. The method of claim 7, wherein a respective set of prediction errors is calculated for each backward reshaping mapping; and the set of prediction errors is used to select a particular candidate value from among the plurality of candidate values ​​for the primary color scaling parameter.

10. 8. The method of claim 7, wherein the sampled HDR color space points are processed by a programmable image signal processor (ISP) pipeline into the input SDR color space points based at least in part on optimized values ​​for programmable configuration parameters of the programmable image signal processor (ISP) pipeline.

11. 11. The method of claim 10, wherein the optimized values ​​for the programmable configuration parameters of the programmable ISP pipeline are determined by minimizing an approximation error between an ISP SDR image generated by the programmable ISP pipeline from an HDR image and a reference SDR image generated by applying a predefined HDR-SDR mapping to the same HDR image.

12. 12. Apparatus comprising a processor and configured to perform the method of any one of claims 1 to 11.

13. A non-transitory computer-readable storage medium storing computer-executable instructions for carrying out the method on one or more processors according to the method of any one of claims 1 to 11.

Images