Image optimization for mobile capture and editing applications

Abstract

HDR color patches are sampled across an HDR color space parameterized by the parameters. From the sampled HDR color patches, a reference SDR color patch, an input HDR color patch, and a reference HDR color patch are generated. An optimization algorithm is run to generate an optimized forward reshaping mapping and an optimized backward reshaping mapping. The optimized forward reshaping mapping is used to forward reshape the input HDR image into a forward reshaped SDR image, and the optimized backward reshaping mapping is used to backward reshape the front reshaped SDR image into a backward reshaped HDR image.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 321,390, filed March 18, 2022, and European Patent Application No. 22162983.5, filed March 18, 2022, each of which is incorporated by reference in its entirety.

[0002] technology The present disclosure relates generally to images. More particularly, embodiments of the present disclosure relate to video codecs used to process images. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest black (shadows) to brightest white (highlights). In this sense, DR relates to "scene-based" intensities. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular width. In this sense, DR relates to "display-based" intensities. Unless a particular meaning is expressly designated at any point in the description herein as having a particular significance, it should be presumed that the term may be used in either sense, e.g., interchangeably.

[0004] As used herein, the term high dynamic range (HDR) refers to a DR width that spans about 14-15 orders of magnitude or more of the human visual system (HVS). In practice, the DR that allows humans to simultaneously perceive a wide range in intensity range may be somewhat truncated compared to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may individually or interchangeably relate to the DR perceivable in a scene or image by the human visual system (HVS), including eye movements, allowing for some light adaptation changes across the scene or image. As used herein, EDR may relate to a DR that spans 5-6 orders of magnitude. Thus, although perhaps somewhat narrower in comparison to true scene-based HDR, EDR nevertheless represents a wide DR width and may also be referred to as HDR.

[0005] In practice, an image comprises one or more color components of a color space (e.g., luma Y, chroma Cb, and Cr), each of which is represented with a precision of n bits per pixel (e.g., n=8). Using non-linear luminance coding (e.g., gamma encoding), images with n≦8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images with n>8 are considered enhanced dynamic range images.

[0006] The reference electro-optical transfer function (EOTF) of a given display characterizes the relationship between the color values ​​(e.g., luminance) of the input video signal and the output screen color values ​​(e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT.1886, “Reference electro-optical transfer function for flat-panel displays used in HDTV studio production” (March 2011), defines a reference EOTF for flat-panel displays. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. The term “metadata” here relates to any auxiliary information transmitted as part of the encoded bitstream to assist a decoder in rendering the decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters as described herein.

[0007] The term "PQ" as used herein refers to perceptual luminance amplitude quantization. The human visual system responds in a highly nonlinear manner to increasing levels of light. A human's ability to see a stimulus is affected by the luminance of that stimulus, the magnitude of the stimulus, the spatial frequencies that make up the stimulus, and the luminance level to which the eye has adapted at the particular moment the stimulus is being viewed. In some embodiments, a perceptual quantizer function maps linear input gray levels to output gray levels that better match contrast sensitivity thresholds in the human visual system. Exemplary PQ mapping functions are described in SMPTE ST 2084:2014 "High Dynamic Range EOTF for Mastering Reference Displays" (hereinafter "SMPTE"), which is incorporated herein by reference in its entirety. Here, given a fixed stimulus size, for each luminance level (e.g., a stimulus level), the minimum visible contrast step at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

[0008] 200~1000cd / m 2 Displays supporting high or low luminance are representative of low dynamic range (LDR), also known as standard dynamic range (SDR), as compared to EDR (or HDR). EDR content may be displayed on EDR displays supporting higher dynamic range (e.g., 1,000 nits to 5,000 nits or more). Such displays may be defined using an alternative EOTF supporting high luminance capabilities (e.g., 0 to 10,000 nits or more). Examples of such EOTFs are defined in SMPTE 2084 and Rec. ITU-R BT.2100 "Picture Parameter Values ​​for High Dynamic Range Television for Use in Production and International Program Exchange" (06 / 2017). As appreciated by the inventors herein, improved techniques for structuring video content data that can be used to support the display capabilities of a wide variety of SDR and HDR display devices are desired.

[0009] Patent document 1 relates to image encoding and / or image decoding. As a non-limiting example, a technique is disclosed in which 3D mapping statistics are generated for a first image in a first dynamic range and a second image in a second dynamic range different from the first dynamic range. In an exemplary technique, multivariate multiple regression (MMR) coefficients are generated by solving an optimization problem formulated using an MMR matrix constructed with the 3D mapping statistics without letterbox constraints and are used to generate a chroma mapping for predicting chroma codeword values ​​of the second image. Further, according to the exemplary technique, it is determined whether a letterbox is present in the image, and if so, whether the chroma mapping accurately predicts the chroma codeword values ​​in the second image. According to a further non-limiting example, a reconstructed image generated by a recipient device by backward reshaping one of the images is rendered by a display device operating in association with the recipient device. [Patent Document 1] WO2019 / 217751A1

[0010] Patent document 2 relates to image encoding and / or image decoding. As a non-limiting example, a technique is disclosed in which a backward reshaping mapping table is initially generated as the inverse of the forward reshaping mapping table. In this exemplary technique, the backward reshaping mapping table is updated by replacing the content-mapped luminance codewords with forward reshaping luminance codewords generated by applying a luminance forward mapping to the sampled luminance codewords. Further, according to this exemplary technique, a luminance forward mapping is constructed from the forward reshaping mapping table, and the backward reshaping mapping table and the luminance forward mapping are used to generate a backward reshaping mapping for creating a reconstructed image from the forward reshaping image. Further, according to this exemplary technique, the forward reshaping image is encoded in the video signal together with image metadata that specifies the backward reshaping mapping. According to a further non-limiting example, a receiver device of the video signal applies a backward reshaping mapping to the forward reshaping image to create a reconstructed image of a second dynamic range. [Patent Document 2] WO2021 / 076822A1

[0011] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise expressly stated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, it should not be assumed that problems identified with one or more approaches have been recognized in any prior art based on this section, unless otherwise expressly stated. [Brief description of the drawings]

[0012] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0013] The same reference numbers refer to similar elements.

[0014] [Figure 1A] 1 illustrates an exemplary process flow for applying an optimized reshaping operation. [Figure 1B] 1 illustrates an exemplary process flow for applying an optimized reshaping operation. [Figure 1C] 1 illustrates an exemplary process flow for applying an optimized reshaping operation. [Figure 1D] 1 illustrates an exemplary process flow for applying an optimized reshaping operation. [Figure 1E] 1 illustrates an exemplary process flow for applying an optimized reshaping operation. [Figure 2A] 1 illustrates example color distributions of irregular shapes associated with HDR and SDR color spaces, such as the R.2020, P3, and R.709 color spaces. [Figure 2B] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2C] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2D] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2E] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2F] 13 shows example percentile Cb and Cr values ​​in a forward reshaped SDR color space corresponding to different parameterized HDR color spaces. [Figure 2G] 13 shows example percentile Cb and Cr values ​​in a forward reshaped SDR color space corresponding to different parameterized HDR color spaces. [Figure 2H] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2I] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2J] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2K] 1 illustrates an example parameterized color space in the R.2020 color space. [Figure 2L] 1 shows an exemplary color chart image. [Figure 2M] 1 illustrates an exemplary checkerboard image. [Figure 2N] 1 illustrates an exemplary checkerboard image. [Figure 2O] 1 shows an exemplary original color chart image displayed on a reference image display, an exemplary captured image from the color chart image, and a modified captured image generated from the captured image. [Figure 2P] 1 shows an exemplary distribution of RGB values. [Figure 2Q] 1 illustrates an example alpha shape. [Figure 2R] 1 illustrates an example alpha shape. [Figure 2S] 1 illustrates an example alpha shape. [Figure 2T] 1 illustrates an example alpha shape.

[0015] [Figure 3A] An exemplary process flow for finding an optimized color space for representing HDR colors is shown. [Figure 3B] An exemplary process flow for determining optimized values ​​for programmable parameters in an ISP pipeline is shown. [Figure 3C] An exemplary process flow for finding an optimized color space for representing HDR colors is shown. [Figure 3D] 1 illustrates an exemplary process flow for generating optimized reshaping operating parameters. [Figure 3E]1 shows an exemplary process flow for producing several different color charts. [Figure 3F] 1 illustrates an exemplary process flow for matching SDR and HDR colors between a captured image pair of an SDR image and an HDR image. [Figure 3G] 1 illustrates an exemplary encoder-side image / video editing operation. [Figure 3H] 1 illustrates an exemplary decoder-side image / video editing operation. [Figure 3I] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3J] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3K] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3L] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3M] 1 illustrates an exemplary solution for clipping out-of-range codewords in image / video editing applications. [Figure 3N] An exemplary process flow is shown.

[0016] [Figure 4A] An exemplary process flow is shown. [Figure 4B] An exemplary process flow is shown. [Figure 4C] An exemplary process flow is shown. [Figure 4D] An exemplary process flow is shown. [Figure 4E] An exemplary process flow is shown.

[0017] [Diagram 5]FIG. 1 illustrates a simplified block diagram of an exemplary hardware platform upon which a computer or computing device described herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obfuscating the present disclosure.

[0019] overview Image optimization, such as those associated with tensor-product B-Spline (TPB)-based image reshaping solutions, are described herein for video capture applications, including but not limited to mobile (video) capture applications. TPB-based solutions can provide or generate relatively accurate reconstructed images using robust predictors implemented in a video codec, such as a backward-compatible video codec. These predictors can operate with static mappings to gracefully handle or recover from transmission errors, such as frame dropping, minimize power consumption, such as battery power consumption, reduce data overhead and / or processing / transmission latency, etc. In some operating scenarios, some or all of the TPB-based solutions described herein may be implemented in mobile applications that are subject to relatively strict battery or power constraints.

[0020] The TPB-based image reshaping solution can be adapted to work in a variety of use cases. Some of the use cases allow the freedom to design and / or apply HDR to SDR mappings to generate SDR images to be encoded in the (base layers of) a video signal, as described herein. Some of the use cases rely on mobile image signal processors (ISPs) with relatively limited programmable registers to generate SDR images to be encoded in the (base layers of) a video signal. Various architectures can be used to implement the TPB-based solutions described herein. Additionally, optionally or alternatively, these architectures or TPB-based solutions can be used to provide or support backward compatibility.

[0021] Given a reference SDR image represented in an SDR color space or a color space within the SDR domain, the TPB-based solution can use a TPB optimization process to achieve or determine the maximum or widest HDR color space or color space within the HDR domain to represent the corresponding reconstructed HDR image predicted from the SDR image. The wider the HDR color space, the greater the color deviation introduced into the SDR color space. The TPB optimization process can be performed to achieve or determine an optimized balance point or trade-off between achieving the maximum or widest HDR color space on the one hand and introducing SDR color deviations on the other hand. Additionally, optionally, or alternatively, the TPB optimization process described herein can be implemented to incorporate neutral color processing or perform constraint optimization to help preserve the gray levels represented in the SDR image in the reconstructed HDR image predicted from the SDR image.

[0022] As more and more video is captured by computing devices and mobile devices with various cameras in practice, video editing is also becoming more and more common. Users operating these devices may be allowed to manually adjust the appearance of the captured video or simply apply a default theme template to the captured video. Depending on the available computational resources and / or the preferences of the video editing tool designer / provider, video editing may be performed either in a source domain, such as a source HDR domain where the source image is represented, or in a non-source domain, such as a generated SDR domain, which may be a destination where a pre-edited image is transformed from the source image in the source domain.

[0023] In an operating scenario where an SDR image is encoded in the video signal, video editing operations in the HDR domain or source domain do not adversely affect devices downstream of the video signal that reconstruct an HDR image from an SDR image decoded from the video signal at the decoder side, because these video editing operations do not interfere with the HDR-SDR mapping at the encoder side to generate an SDR image from an HDR image or a source image in the HDR domain, and do not interfere with the SDR-HDR mapping at the decoder side to map back to an HDR image or reconstruct an approximation of an HDR image.

[0024] However, in these operating scenarios, video editing operations in the SDR domain are likely to adversely affect reversibility between the SDR and HDR domains. For example, an edited SDR image may not be able to be mapped back to the original or source HDR image. Furthermore, the color space representing the SDR image may be restricted with a predefined range, such as the SMPTE range, that pixel values ​​or codewords cannot exceed. Color space conversions, such as YUV-RGB conversions, used for video editing operations may result in pixel values ​​or codewords being clipped, thus impairing or harming reversibility. The clipping techniques described herein may be used to reduce or prevent deleterious editing effects on video editing operations, including but not limited to those performed on mobile devices. Two-level TPB boundary clipping may be implemented in the RGB domain to support or preserve the maximum edited colors in the SDR domain that can be used to propagate or map to a reconstructed image in the HDR domain.

[0025] Exemplary embodiments described herein relate to image generation. Sampled HDR color space points distributed throughout an HDR color space are constructed. The HDR color space is parameterized by primary color scaling parameters having candidate values ​​selected from among a plurality of candidate values. The primary color scaling parameters are used to calculate color space coordinates of at least one of a plurality of primary colors describing the HDR color space. From the sampled HDR color space points in the HDR color space, reference SDR color space points represented in a reference SDR color space, input HDR color space points represented in an input HDR color space, and reference HDR color space points represented in a reference HDR color space are generated. A reshaping operation optimization algorithm is performed to generate a chain of optimized forward reshaping mappings and optimized backward reshaping mappings. The reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs. The optimized forward reshaping mapping is used to forward reshape an input HDR image in the input HDR color space to a forward reshaped SDR image in the forward reshaped SDR color space, while the optimized backward reshape mapping is used to backward reshape a forward reshaped SDR image in the forward reshaped SDR color space to a backward reshaped HDR image.

[0026] Exemplary embodiments described herein relate to image generation. Sampled HDR color space points are constructed that are distributed throughout an HDR color space. The HDR color space is parameterized by primary color scaling parameters with candidate values ​​selected from among a plurality of candidate values. The primary color scaling parameters are used to calculate color space coordinates of at least one of a plurality of primary colors that describe the HDR color space. From the sampled HDR color space points in the HDR color space, input SDR color space points represented in an input SDR color space and reference HDR color space points represented in a reference HDR color space are generated. A reshaping operation optimization algorithm is performed to generate an optimized backward reshaping mapping. The reshaping operation optimization algorithm receives as input the input SDR color space points and the reference HDR color space points. The backward reshaping mapping is used to backward reshape an SDR image in the input SDR color space into a backward reshaped HDR image.

[0027] Exemplary embodiments described herein relate to image generation. A set of SDR image feature points is extracted from a training SDR image, while a set of HDR image feature points is extracted from a training HDR image. A subset of one or more SDR image feature points in the set of SDR image feature points is matched with a subset of one or more HDR image feature points in the set of HDR image feature points. The subset of one or more SDR image feature points and the subset of one or more HDR image feature points are used to generate a geometric transformation for spatially aligning a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image. After the training SDR and HDR images are spatially aligned by the geometric transformation, a set of SDR and HDR color patch pairs is determined from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image. An optimized SDR-HDR mapping is generated based at least in part on the set of SDR and HDR color patch pairs derived from the training SDR image and the training HDR image. The optimized SDR-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0028] Exemplary embodiments described herein relate to image generation. A respective camera distortion correction operation is performed on each training image in a training standard dynamic range (SDR) image and training high dynamic range (HDR) image pair to generate a respective undistorted image in a pair of undistorted training SDR images and undistorted training HDR images. Each projective transform in the pair of SDR image projective transformation and HDR image projective transformation is generated using a corner pattern mark detected from each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images. Each projective transform in the pair of SDR image projective transformation and HDR image projective transformation is applied to each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images to generate a respective rectified image in the pair of rectified training SDR images and undistorted training HDR images. A set of SDR color patches is extracted from the rectified training SDR image, while a set of HDR color patches is extracted from the rectified training HDR image. The optimized SDR-HDR mapping is generated based at least in part on the set of SDR color patches and the set of HDR color patches derived from the training SDR image and the training HDR image. The optimized SDR-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0029] Exemplary embodiments described herein relate to a clipping operation on an edited image. Sampled HDR color space points distributed throughout an HDR color space used to represent a reconstructed HDR image are constructed. The sampled HDR color space points are transformed to SDR color space points in a first SDR color space in which an SDR image edited by an editing device is represented. A bounding SDR color space rectangle is determined based on SDR codeword extrema of the SDR color space points in the first SDR color space. An irregular three-dimensional (3D) shape is determined from the distribution of the SDR color space points. Sampled SDR color space points distributed throughout the bounding SDR color space rectangle in the first SDR color space are constructed. The sampled SDR color space points and the irregular shape are used to generate a boundary clipping 3D lookup table (3D-LUT). The boundary clipping 3D-LUT uses the sampled SDR color space points as lookup keys. A clipping operation is performed on the edited SDR image in the first SDR color space based at least in part on the boundary clipping 3D-LUT to generate a boundary clipped edited SDR image in the first SDR color space.

[0030] Optimizing reshaping in image / video capture applications A reshape optimization process, such as a TPB and / or non-TPB optimization process, can be implemented or incorporated into a video capture application executing on a computing device, such as a mobile device, in a variety of operating scenarios. A video capture application having a reshape optimization process can be used to generate or output a video signal, such as a base layer or SDR image encoded therein. The reshape optimization process can be implemented in a variety of solutions in a variety of operating scenarios to generate or optimize reshape operating parameters, such as TPB and / or non-TPB coefficients, that are used in conjunction with a base layer or SDR image to generate, construct or reconstruct a non-base layer or HDR image having optimized image quality.

[0031] For illustrative purposes, the reshaping optimization process, or the solutions implemented therein, can be categorized into different types based on the particular SDR generation process or sub-process employed by the video capture application, as well as based on the particular reshaping path (forward (reshaping) path and / or backward (reshaping) path) that is subject to the reshaping optimization.

[0032] 1A illustrates a first exemplary reshaping optimization process that implements a white-box joint forward and backward TPB optimization design / solution (referred to as "WFB") using a (single) video capture device. The white-box joint forward and backward TPB optimization design / solution may be implemented for operational scenarios in which an SDR generation process or sub-process converts a reference HDR image to a reference SDR image using a white-box transformation operation, and both the forward and backward (reshaping) paths can undergo TPB optimization.

[0033] As used herein, a "white-box transformation" refers to a mapping or transformation from HDR to SDR that uses a clearly defined transformation or mapping function / formula that is specified or documented (e.g., publicly) in a standard-based or proprietary video encoding specification. In contrast, a "black-box transformation" refers to a mapping or transformation operation from HDR to SDR that is not based on such a clearly defined transformation or mapping function / formula. For example, a black-box transformation may be implemented as an internal image signal process performed by an image signal processor without or with little to no clearly defined transformation or mapping function or formula that is specified or documented (e.g., publicly) in a standard-based or proprietary video encoding specification.

[0034] 1A, an input video signal including an HDR image may be represented in an input color space, such as the full R.2020 color space. A generated SDR signal including a reference SDR image may be represented in a reference SDR color space. The reference SDR image may be an image generated from the HDR image using a known HDR-to-SDR mapping process, function, and / or formula.

[0035] 1A, in the forward reshaping path, a reference HDR image may be forward reshaped into a (forward) reshaped SDR image that approximates the reference SDR image based at least in part on optimized forward reshaping operation parameters (referred to as "forward TPB optimization") generated from the joint forward-backward TPB optimization solution. The reshaped SDR image may be represented in a reshaped SDR color space and generated by an upstream device to be included / encoded in a video signal output by the upstream device or in its base layer (BL).

[0036] A downstream recipient device or video decoder of the video signal can decode the reshaped SDR image from the video signal. The decoded reshaped SDR image at the decoder side may be the same as the reshaped SDR image at the encoder side, but may be subject to errors introduced in the compression / decompression, encoding operations, and / or data transmission.

[0037] In the backward reshaping path implemented by the downstream device, the reshaped SDR image may be backward reshaped into a (backward) reshaped HDR image based at least in part on optimized backward reshaping operation parameters (referred to as "backward TPB optimization") generated from the same joint forward and backward TPB optimization solution. The reshaped HDR image -- generated with the optimized backward reshaping operation parameters in the output HDR color space -- represents an approximation or reconstructed version of the reference HDR image.

[0038] The joint forward and backward TPB optimization solution can generate optimized forward and backward reshaping operation parameters to cover the widest possible range in the output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0039] One or both of the optimized forward and backward reshaping operation parameters, such as the optimized forward and backward TPB coefficients, may be implemented or represented in a three-dimensional lookup table or 3D-LUT(s) to reduce processing time in the reshaping operation.

[0040] Because forward and backward reshaping operation parameters, such as forward and backward TPB coefficients, are jointly or concurrently designed or optimized in a joint forward and backward TPB optimization process, the supported reshaped SDR and HDR color spaces used to represent the reshaped SDR and HDR images can be jointly or concurrently designed or optimized in the same process.

[0041] In some operating scenarios, the optimized forward and backward reshaping motion parameters generated from the joint forward and backward TPB optimization process may be applied in a static single-layer backward compatible (SLBC) framework. Under this static framework, there is no need to obtain and optimize (e.g., dynamically, etc.) image-specific or image-dependent optimized forward and backward motion parameters, such as image-specific or image-dependent (or content-dependent) forward and backward TPB coefficients, on the fly, e.g., while images are being processed.

[0042] Rather, under the static SLBC framework, the same or static optimized forward and backward motion parameters, such as the same optimized forward and backward TPB coefficients, can be obtained or generated once, for example, offline or before performing a reshaping operation on either the (input) reference HDR image or the reshaped SDR image, for forward reshaping all (input) reference HDR images and backward reshaping the reshaped SDR images. In one example, a single set of static optimized forward and backward motion parameters can be generated offline by the system described herein and configured / deployed in or used by the capture device described herein to perform an image reshaping or reconstruction operation. In another example, multiple sets of static optimized forward and backward motion parameters can be generated offline by the system described herein and configured / deployed in or used by the capture device described herein to select a particular set of static optimized forward and backward motion parameters to perform an image reshaping or reconstruction operation.

[0043] The optimized static forward TPB coefficients may then be applied by an upstream device to forward reshape some or all of an (input) reference HDR image (e.g., a sequence of consecutive or sequential (input) reference HDR images) to generate a reshaped SDR image that is encoded in the (SLBC) video signal, while the optimized static backward TPB coefficients may be applied by a downstream recipient device of the video signal to some or all of a reshaped SDR image (e.g., a sequence of consecutive or sequential reshaped SDR images) decoded from the video signal to generate or reconstruct a reshaped HDR image.

[0044] 1B illustrates a second exemplary reshaping optimization process implementing a white-box rear-only TPB optimization design / solution (referred to as "WB") using a (single) video capture device. The white-box rear-only TPB optimization design / solution may be implemented for operational scenarios in which an SDR generation process or sub-process converts a reference HDR image to a reference SDR image using a white-box conversion operation, and only the rear (reshaping) path undergoes TPB optimization.

[0045] 1B, the input video signal including the HDR image may be represented in an input color space, such as the P3 color space. The generated SDR signal including the reference SDR image may be represented in a reference SDR color space. The reference SDR image may be an image generated from the HDR image using a known HDR-to-SDR mapping process, function, and / or formula.

[0046] 1B, the reference HDR image may be processed by a programmable image signal processor or ISP (e.g., using a given image signal processing function / formula / operation, etc.) based at least in part on optimized ISP operating parameters (referred to as "ISP parameter optimization") generated from the ISP optimization solution into an ISP SDR image that approximates the reference SDR image. The ISP SDR image may be generated by an upstream device to be represented in an ISP SDR color space and included / encoded in a video signal output by the upstream device or in its base layer (BL).

[0047] A downstream recipient device or a video decoder of the video signal can decode the ISP SDR image from the video signal. The decoded ISP SDR image at the decoder side may be the same as the ISP SDR image at the encoder side, but may be subject to errors introduced in the compression / decompression, encoding operations, and / or data transmission.

[0048] In the rear reshaping path as implemented by a downstream device, the ISP SDR image may be rear reshaped into a (rear) reshaped HDR image based at least in part on optimized rear reshaping operation parameters (referred to as "rear TPB optimization") generated from a rear-only TPB optimization solution. The reshaped HDR image -- generated with the optimized rear reshaping operation parameters in the output HDR color space -- represents an approximation or reconstructed version of the reference HDR image.

[0049] The backward-only TPB optimization solution can generate optimized backward reshaping operation parameters that cover the widest possible range in the output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0050] The optimized backward reshaping operation parameters, such as the optimized backward TPB coefficients, may be implemented or represented in a three-dimensional look-up table or 3D-LUT to reduce processing time in the reshaping operation.

[0051] In some operating scenarios, the backward reshaping motion parameters generated from the backward-only TPB optimization process may be applied in a static single-layer inverse display mapping (SLiDM) framework. Under this static framework, there is no need to obtain or optimize (e.g., dynamically) image-specific or image-dependent optimized backward motion parameters, such as image-specific or image-dependent (or content-dependent) backward TPB coefficients.

[0052] Rather, under the static SLiDM framework, the same or static optimized backward operation parameters, such as the same optimized backward TPB coefficients, can be obtained or generated once, e.g., offline or before performing a reshaping operation on any of the reshaped SDR images, to backward reshape an ISP SDR image. In one example, a single set of static optimized backward operation parameters can be generated offline by a system described herein and configured / deployed in or used by a capture device described herein to perform an image reshaping or reconstruction operation. In another example, multiple sets of static optimized backward operation parameters can be generated offline by a system described herein and configured / deployed in or used by a capture device described herein to select a particular set of static optimized backward operation parameters to perform an image reshaping or reconstruction operation.

[0053] The optimized static backward TPB coefficients may then be applied by a downstream recipient device of the video signal to some or all of an ISP SDR image (e.g., a sequence of consecutive or sequential ISP SDR images) decoded from the video signal to generate or reconstruct a reshaped HDR image.

[0054] The ISP SDR image encoded in the video signal may or may not be identical to the desired SDR appearance as represented by the reference image. The operating parameters in the programmable ISP or their settings can be optimized to approximate the reference image. Thus, in the "WB" operating scenario of FIG. 1B, the SDR image for backward reshaping to the reshaped HDR image is given as the ISP SDR image or is fixed. For comparison, in the WFB operating scenario of FIG. 1A, the SDR image for backward reshaping to the reshaped HDR image is a forward reshaped SDR image, which can be optimized by applying the optimized forward reshaping operating parameters generated in a joint forward and backward TPB optimization process that generates both optimized forward reshaping parameters and backward reshaping parameters. In other words, TPB optimization is not used in the forward path to generate the ISP SDR image with the aim of improving or enhancing both the desired appearance of the SDR image encoded in the video signal and the desired appearance of the reshaped HDR image. In these operating scenarios, relatively large deviations from the desired appearance may occur.

[0055] 1C illustrates a third exemplary reshaping optimization process implementing a black-box rear-only TPB optimization design / solution (referred to as "BB1") with a (single) video capture device. The black-box rear-only TPB optimization design / solution may be implemented for operating scenarios where an SDR image is not generated from an HDR image using a white-box conversion operation and only the rear (reshaping) path undergoes TPB optimization.

[0056] The reshaping operational parameters in these operational scenarios can be generated using training SDR images and training HDR images generated or captured by the same video capture device, such as the same mobile device, forming a number of SDR and HDR image pairs, each of which includes a training SDR image and a training HDR image corresponding to the training SDR image.

[0057] The training SDR and training HDR images in the same SDR and HDR image pair may be acquired at different time instances / times (e.g., milliseconds apart, fractions of a second apart, seconds apart, etc.) using the same capture device using the same ISP. The training SDR and HDR images may not be strictly spatially or temporally aligned with each other because it is difficult, if not impossible, to maintain the same shooting position and process the training SDR and HDR images in an identical manner at different time instances / times. For example, the training SDR image may be locally tone mapped or locally enhanced. Meanwhile, the training HDR image may be generated or acquired from multiple camera exposures. Thus, in the "BB1" operating scenario, the relationship between pixel values ​​or codeword values ​​in the training SDR image and the corresponding pixel values ​​or codeword values ​​in the training HDR image in the same image pair may be treated or assumed to be a black box.

[0058] The training SDR image and the HDR image in each of the image pairs may first be spatially aligned. The spatially aligned training SDR image and the HDR image in the image pair may be used to determine or find matching color pairs. These matching color pairs may then be used to generate optimized reshaping operation parameters, such as optimized TPB and / or non-TPB coefficients, for reshaping or mapping the (e.g., non-training, reference, etc.) SDR image to a backward reshaped or reconstructed HDR image that approximates the (e.g., non-training, reference, etc.) HDR image.

[0059] 1C, an SDR image, such as a reference SDR image, may be generated by an upstream device, such as a capture device, to be included / encoded in a video signal or its base layer (BL) output by the upstream device. Exemplary reference SDR images described herein may include, but are not necessarily limited to, an SDR image generated from a programmable image signal processor of the upstream device or a capture device operating in conjunction with the upstream device.

[0060] A downstream recipient device or video decoder of the video signal can decode the reference SDR image from the video signal. The decoded reference SDR image at the decoder side may be the same as the reference SDR image at the encoder side, but may be subject to errors introduced in the compression / decompression, encoding operations, and / or data transmission.

[0061] In the backward reshaping path as implemented by a downstream device, the reference SDR image may be backward reshaped into a (backward) reshaped HDR image based at least in part on optimized backward reshaping operation parameters (referred to as "backward TPB optimization") generated from a black-box backward-only TPB optimization design / solution. The reshaped HDR image--generated with the optimized backward reshaping operation parameters in the output HDR color space--represents an approximation or reconstructed version of the reference HDR image that can or could be generated by the same capture device.

[0062] The black-box rear-only TPB optimization design / solution can generate optimized rear reshaping operation parameters that cover the widest possible range in the output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0063] The optimized backward reshaping operation parameters, such as the optimized backward TPB coefficients, may be implemented or represented in a three-dimensional look-up table or 3D-LUT to reduce processing time in the reshaping operation.

[0064] In some operating scenarios, the backward reshaping operating parameters generated from the black-box backward-only TPB optimized design / solution may be applied in the SLiDM framework. Under this static framework, there is no need to obtain and optimize (e.g., dynamically) image-specific or image-dependent optimized backward operating parameters, such as image-specific or image-dependent (or content-dependent) backward TPB coefficients.

[0065] Rather, under a static SLiDM framework, the same or statically optimized backward operation parameters, such as the same optimized backward TPB coefficients, can be obtained or generated once - for example, offline or before performing a reshaping operation on any of the reshaped SDR images - to backward reshape a spatially aligned training SDR image generated by an upstream capture device into a reshaped or reconstructed HDR image that approximates the spatially aligned training HDR image generated by the same capture device.

[0066] The optimized static backward TPB coefficients can then be applied by a downstream recipient device of the video signal to some or all of a (e.g., non-training, etc.) reference SDR image (e.g., a sequence of consecutive or sequential reference SDR images, etc.) decoded from the video signal to generate or reconstruct a reshaped HDR image that approximates a reference HDR image that can or could be generated from the same upstream capture device that generates or captures the reference SDR image.

[0067] Because TPB optimization is used only in the backward path, the resulting appearance of a reshaped HDR image generated from backward reshaping a reference SDR image may have a relatively large deviation from the desired appearance of the reference HDR image in the "BB1" operating scenario.

[0068] In some "BB1" operating scenarios, as shown in FIG. 1D, a camera raw image generated from a camera ISP can be directly encoded in the video signal output by an upstream (capture) device instead of a reference SDR image generated from a programmable image signal processor. In the training phase, the training camera raw image, which may or may not be an SDR image, can be spatially and / or temporally aligned with a training HDR image in the same image pair of camera raw image and HDR image. Matching color pairs from the spatially aligned image pair of camera raw image and HDR image can be used to optimize backward reshaping operation parameters such as backward TPB coefficients. In the non-training or unfolding phase, the optimized backward reshaping operation parameters can be used by a downstream device of the video signal encoded with the non-training camera raw image to backward reshape the decoded non-training camera raw image from the video signal into a backward reshaped or reconstructed (non-training) HDR image that approximates a reference HDR image that can or should be generated by the same upstream (capture) device.

[0069] 1E illustrates a fourth exemplary reshape optimization process implementing a black-box rear-only reshape optimization design / solution (referred to as "BB2") using a different video capture device. The "BB2" reshape optimization design / solution may be implemented for an operating scenario in which an SDR image is generated by a first capture device and an HDR image generated from the SDR image is generated (e.g., intended to be generated) by a second, different capture device, and only the rear (reshape) path undergoes reshape optimization. Furthermore, the reshape optimization in the "BB2" operating scenario may or may not be a TPB optimization.

[0070] The reshaping operational parameters in the "BB2" operational scenario can be generated using training SDR and training HDR images generated or captured by two video capture devices, such as two mobile devices, of different makes and / or models, respectively, that form a plurality of SDR and HDR image pairs, each including a training SDR image and a training HDR image corresponding to the training SDR image.

[0071] The training SDR image and the training HDR image in the same SDR image and HDR image pair may be acquired using the same image, for example, the same checkerboard image rendered on the same or similar image display(s). In the "BB1" operating scenario, the relationship between pixel values ​​or codeword values ​​in the training SDR image and corresponding pixel values ​​or codeword values ​​in the training HDR image in the same image pair may be treated as or assumed to be a black box.

[0072] The training SDR and HDR images in each of the image pairs may first be spatially aligned. The spatially aligned training SDR and HDR images in the image pairs may be used to determine or find matching color patches rendered in a test image, such as a checkerboard image on the same image display or same image display type. These matching color patches may then be used to build a three-dimensional lookup table (3D-LUT) for mapping SDR pixel values ​​or codeword values ​​of color patches in the test image to corresponding HDR pixel values ​​or codeword values ​​of the same color patches.

[0073] The 3D-LUT derived based in part or in whole on the test images can be used to generate optimized static or dynamic reshaping operating parameters for reshaping or mapping the (e.g., non-training, reference, etc.) SDR image to a backward reshaped or reconstructed HDR image that approximates the (e.g., non-training, reference, etc.) HDR image.

[0074] 1E, an SDR image, such as a reference SDR image, may be generated by an upstream device, such as a first capture device, to be included / encoded in a video signal or its base layer (BL) output by the upstream device. Exemplary reference SDR images described herein may include, but are not necessarily limited to, an SDR image generated from a programmable image signal processor of the first capture device or the upstream device.

[0075] A downstream recipient device or video decoder of the video signal can decode the reference SDR image acquired by the first capture device from the video signal. The decoded reference SDR image at the decoder side may be the same as the reference SDR image at the encoder side, but may be subject to errors introduced in the compression, decompression, encoding operations and / or data transmission.

[0076] In the backward reshaping path performed by the downstream device, the reference SDR image may be backward reshaped into a (backward) reshaped HDR image based at least in part on optimized backward reshaping operation parameters (referred to as "dynamic backward function optimization") generated from the "BB2" reshaping optimization design / solution. The reshaped HDR image--generated with the optimized backward reshaping operation parameters in the output HDR color space--represents an approximate or reconstructed version of the reference HDR image that can or could be generated by a second, different capture device.

[0077] The "BB2" reshaping optimization design / solution can generate optimized backward reshaping operation parameters to cover the widest possible range in the output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0078] In some operating scenarios, the rear reshaping operating parameters generated from the "BB2" reshaping optimization design / solution may be applied in a static or dynamic SLiDM framework.

[0079] Because the "BB2" reshaping optimization is used only in the backward path, the resulting appearance of a reshaped HDR image generated from backward reshaping a reference SDR image may have a relatively large deviation from the desired appearance of the reference HDR image. The first and second capture devices may be mobile devices that can operate differently in their respective video capture applications, for example, using different exposure times, different video processing operations, different mapping functions / relationships, etc.

[0080] Dynamic SLiDM frames, in which dynamic mapping with image-dependent or image-specific reshaping operating parameters may be used to back-reshape an SDR image of a first capture device into an HDR image that can or should be produced by a second capture device, particularly in operational scenarios in which the first and second capture devices operate with relatively large differences between the respective video / image capture applications executed on the first and second capture devices and / or between the respective ISPs used in the first and second capture devices. For example, in a training phase, multiple sets of optimized reshaping operating parameters may be derived for the multiple sets of training SDR and HDR images or image pairs using the multiple sets of training SDR and HDR images or image pairs, respectively. In a deployment or application phase, a particular image characteristic, such as the overall brightness of some or all areas of a particular image, may be dynamically evaluated as that particular image is being processed. A particular set of optimized reshaping operating parameters may be adaptively and / or dynamically selected from the multiple sets of optimized reshaping operating parameters for that particular image. The selection is based on particular image characteristics of that particular image in relation to or in comparison with image characteristics of each of a plurality of sets of training SDR and HDR images or image pairs.

[0081] White-box joint TPB forward and backward optimization (WFB) Many design factors can be considered for reshaping optimization to help expand the coverage of supported color spaces in which reshaped images are represented. First, a tensor-product bi-spline (TPB) predictor with optimized TPB coefficients generated from the reshaping optimization can be used to obtain or achieve relatively higher prediction accuracy than other types of predictors, including but not limited to MMR predictors. The multi-knot and continuity properties of the tensor-product bi-spline predictor or prediction function can be exploited or used to cover a relatively wide color range or color space portion with relatively high accuracy. An exemplary TPB predictor can be found in U.S. Provisional Patent Application No. 62 / 908,770, filed October 1, 2019, by Guan-Ming Su, Harshad Kadu, Qing Song, and Neeraj J. Gadgil, entitled "Tensor-product B-spline predictor," the contents of which are incorporated herein by reference in their entirety as if fully set forth herein.

[0082] In comparison, while a multi-piece MMR predictor or prediction function may be superior to a single-piece MMR predictor or prediction function, the multi-piece MMR predictor or prediction function is likely to introduce discontinuities between different MMR pieces that may adversely affect or even prohibit the use of a multi-piece MMR predictor or prediction function in many operational scenarios. Exemplary MMR-based operations are described in U.S. Patent No. 8,811,490, which is incorporated herein by reference in its entirety.

[0083] A TPB predictor can be advantageously used in an SLBC-based video codec. More specifically, a forward TPB predictor can be used to generate a forward reshaped SDR image that approximates a reference SDR image, and a backward TPB predictor can be used to generate a backward reshaped or reconstructed HDR image that approximates a reference HDR image. An exemplary use of a TPB predictor with an SLBC-based video codec can be found in U.S. Provisional Patent Application No. 63 / 255,057, filed October 13, 2021, by H. Kadu et al., entitled "Tensor-product B-spline prediction for HDR video in mobile applications," the contents of which are incorporated herein by reference in their entirety as if fully set forth herein. Furthermore, a BESA (Backward Error Subtraction for signal Adjustment) algorithm / method with modifications to support neutral color preservation can be used in a pipeline of chained reshaping functions to jointly optimize the forward and backward TPB predictors to achieve relatively high reversibility or a relatively accurate approximation to the reference HDR image by the backward reshaped or reconstructed HDR image. Exemplary BESA algorithms / methods can be found in U.S. Provisional Patent Application No. 63 / 013,063, "Reshaping functions for HDR imaging with continuity and reversibility constraints," by GM Su, filed on April 21, 2020, U.S. Provisional Patent Application No. 63 / 013,807, "Iterative optimization of reshaping functions in single-layer HDR image codec," by GM su, H. Kadu, filed on April 22, 2020, and PCT Application No. PCT / US2021 / 028475, filed on April 21, 2021, the contents of which are incorporated by reference in their entireties as if fully set forth herein.

[0084] Although the TPB predictor has better prediction accuracy than other types of predictors, a relatively large signal or bitrate overhead may be incurred to transmit the TPB coefficients in the video signal, and a relatively large computational cost may be incurred to construct the TPB equations or basis functions used in the TPB predictor.

[0085] In some operating scenarios, to avoid or reduce signal or bitrate overhead and computational costs, a built-in or static TPB predictor may be used to reshape some or all images in an entire video sequence without changing the TPB coefficients used in the static TPB predictor during playback of the video sequence. More specifically, some or all of the (built-in or static) TPB coefficients may be cached or stored within a video application, such as a video capture / editing application, without the need to explicitly transmit these TPB coefficients through or within the encoded video signal or bitstream for the images to be reshaped by the built-in or static TPB predictor. In one example, some or all of the TPB coefficients may be pre-loaded or pre-configured in a downstream recipient device before the video signal or bitstream is received and processed by the downstream recipient device. Additionally, optionally, or alternatively, multiple sets of TPB coefficients may be pre-loaded or pre-configured in a downstream recipient device before the video signal or bitstream is received and processed by the downstream recipient device. A video application may simply pick or select a particular set of TPB coefficients for use with a static TPB predictor from among multiple sets of TPB coefficients based on, for example, a simple indicator (e.g., a binary indicator, a multi-bit indicator, etc.) that is signaled or transmitted in a video signal or bitstream. Exemplary static TPB predictors or prediction functions may be found in the aforementioned U.S. Provisional Patent Application No. 63 / 255,057.

[0086] In some operating scenarios, a mobile device may be used to host and execute video capture and / or editing applications. A user of the mobile device may edit the captured images in these video capture and / or editing applications. An edited image, such as an edited HDR image, which may be intended to be displayed or viewed, for example, on an image display of a non-mobile device having a higher or larger display capability than the display capabilities of the image display of the mobile device, may have a higher, larger, broader, and / or wider luminance range and / or color range or color distribution than an (original) captured image, such as an HDR image originally captured from a camera sensor of the mobile device. For example, a captured HDR image is often limited by the mobile device's camera sensor and ISP output to be represented in a relatively narrow color space or gamut, such as P3, whereas an edited HDR image may be represented in a relatively wide color space or gamut, such as the entire R.2020 color space.

[0087] To design a static mapping for a reshaping operation, it is highly desirable to optimize the forward / backward TPB coefficients to cover the highest possible dynamic range and the widest possible color space or gamut, while also achieving the highest possible bitrate and computational efficiency.

[0088] Full HDR reversibility between a reconstructed HDR image and a reference HDR image in which the reconstructed HDR is identical requires that colors in the original or reconstructed HDR domain (or color space) be identified or made identifiable in the reshaped SDR domain (or color space) to allow identified colors in the SDR domain to be mapped back to distinguishable colors in the HDR domain. Thus, full reversibility likely requires a one-to-one mapping from HDR to SDR and from SDR back to HDR. The larger the HDR domain or color space to be supported, the more codewords the SDR domain or color space must have. Given that the total number of available SDR pixel or codeword values ​​in the SDR domain (e.g., corresponding to a lower bit depth than the HDR domain) is typically less than the total number of required HDR pixel or codeword values ​​in the HDR domain, full HDR reversibility may not be possible in some operating scenarios. In fact, some captured or edited images may contain combinations of diffuse and / or specular colors that form color distributions, such as those shown as irregular shapes in FIG. 2A, that exceed the R.2020 color space ("BT.2020"), let alone smaller or SDR color spaces such as the R.709 color space ("BT.709") or the (DCI) P3 color space, and therefore are not fully represented by it.

[0089] In many operating scenarios, a nonlinear reshaping mapping or function can be used to generate a (forward) reshaped SDR that distributes the available pixel or codeword values ​​relatively efficiently, approximates the reference SDR image as closely as possible, and helps the reconstructed HDR image to approximate the reference HDR image as closely as possible, although the reshaped SDR generated using the nonlinear reshaping mapping or function may not be identical to the reference SDR image, but rather may contain some deviations from the reference SDR image.

[0090] Additionally, optionally, or alternatively, in some operating scenarios, in order to maintain the same or similar appearance of the reference SDR and HDR images in the reshaped SDR and HDR images, the reshaping optimizations described herein can be performed using a neutral color constraint that preserves neutral colors in the color mapping performed in the reshaping operation.

[0091] Given its ability to approximate functions with relatively high nonlinearity, the TPB predictor or function can be incorporated into the reshaping operation to support or achieve a relatively wide HDR color space for representing the backward reshaped or reconstructed HDR image.

[0092] A potential downside is that the TPB predictor may require a relatively large number of TPB coefficients to represent or approximate a nonlinear SDR-HDR and / or HDR-SDR mapping function. To prevent ill-defined conditions, numerical instabilities, slow convergence issues, etc. that may arise when solving an optimization problem for the TPB coefficients, a static TPB predictor may be pre-generated and deployed with video codecs as used by upstream and / or downstream devices (before those video codecs are used to process video sequences, such as captured and / or edited video sequences).

[0093] For illustrative purposes only, in the operation scenario shown in FIG. 1A, the (forward reshaped) SDR domain or color space may be that of R.709, while the (original or backward reshaped) HDR domain or color space may be that of R.2020. As shown in FIG. 2A, the R.2020 color space is much larger than the R.709 color space. Thus, although it is difficult to map the entire R.2020 to R.709 and then map R.709 back to the original R.2020, the reshaping optimization techniques described herein can be used to optimize the coverage of the HDR color space for representing the backward reshaped or reconstructed HDR image while maintaining the same or similar appearance of the reference SDR and HDR images in the reshaped SDR and HDR images. These techniques can be used to simultaneously optimize reshaping operation parameters, such as TPB coefficients, for both forward and backward reshaping operations.

[0094] While it may not be possible to cover the entire R.2020 color space in all scenarios, a subset or subspace in the R.2020 color space (e.g., a particular color gamut depicted as a triangle formed by the primary colors in a color coordinate system) can be selected or chosen by a reshaping optimization operation to maintain HDR invertibility to the reference HDR image and an acceptable SDR approximation of the reference SDR image in the reshaped HDR image and SDR image.

[0095] By way of example and not limitation, a subset or subspace in the R.2020 color space (or an HDR color space for representing a reshaped HDR image) can be defined or characterized by a specific white point and three specific primary colors (red, green, blue). The specific white point may be selected or fixed to be the D65 white point. There may be significant design freedom to select the three specific primary colors from many possible combinations of primary colors for a subset or subspace in R.2020 that is supported by the reshaping operation. The reshaping optimization techniques described herein can be used to determine or select the specific primary colors for a subset or subspace in R.2020 to be the optimized primaries for achieving or reaching maximum perceptual quality and / or color coding efficiency and / or reversibility between SDR and HDR images.

[0096] The Macadam ellipse represents the color difference that is just perceptible in that the HVS may not be able to distinguish color differences within the same ellipse. The Pointer's gamut can represent all diffuse colors that can be perceived by the HVS. In the Macadam ellipse and Pointer's gamut, green is less important and less distinguished / perceived by the HVS than red and blue. Thus, in some operating scenarios, specific optimized primaries for a subset or subspace in R.2020 for representing a reshaped or reconstructed HDR image can be selected to cover more red and blue colors than green, especially when a one-to-one mapping relationship cannot be supported by SDR-HDR and HDR-SDR mapping in the reshaping operation.

[0097] As used herein, the primaries, sometimes referred to as primary colors, may be used to define the vertices of a polygon, such as a triangle, that represents a particular color space or gamut, such as a color space specified by a standard, a color space supported by a display, or a color space supported by a video signal. For example, a standard color space with a standard-specified white point may be represented in the CIExy color space coordinate system or CIExy chromaticity diagram by a triangle whose vertices are specified by the three standard-specified primaries. The CIExy coordinates of the primaries (red or R, green or G, blue or B) and white points that define the R.709, P3 and R.2020 color spaces, respectively, are specified in Table 1 below. [Table 1]

[0098] The CIExy coordinates of the primary colors and the corresponding white points of each standard color space in Table 1 are (R x (c) ,R y (c) ),(G x (c) ,G y (c) ),(B x (c) ,B y (c) ) and (W x (c) ,W y (c) ), and (c) is the standard color space.

[0099] Thus, the P3 color space may be characterized in terms of the CIExy coordinates of the P3 primaries and the P3 white point as follows: x (P3) ,R y (P3) ),(G x (P3) ,G y (P3) ),(B x (P3) ,B y (P3) ),(W x (P3) ,Wy (P3) Similarly, the R.2020 color space may be characterized in terms of the CIExy coordinates of the R.2020 primaries and the R.2020 white point as follows: x (R2020) ,R y (R2020) ),(G x (R2020) ,G y (R2020) ),(B x (R2020) ,B y (R2020) ),(W x (R2020) ,W y (R2020) ).

[0100] Let us denote the backward reshaped HDR color space for representing the backward reshaped or reconstructed HDR image as (a) color space. Thus, the CIExy coordinates of the primaries and white point that define (a) color space are (R x (a) ,R y (a) ),(G x (a) ,G y (a) ),(B x (a) ,B y (a) ) and (W x (a) ,W y (a) ) may also be written.

[0101] As mentioned above, green is less important or less distinguishable / perceived by the HVS than red and blue. In some operating scenarios, the red and blue primaries of the (a) color space may be selected to match the primaries of the R.2020 color space, as follows: R x (a) =R x (R2020) (1-1) R y (a) =R y (R2020) (1-2) B x (a) =B x (R2020) (2-1) B y (a) =B y (R2020) (2-2)

[0102] In addition, like all R.709, P3, and R.2020 color spaces that are specified with a D65 white point, the (a) color space can also be specified with a D65 white point as follows: (W x (a) ,W y (a) )=(0.3127 0.3290) (3)

[0103] The green primary color of this (a) color space is the green primary color (G x (P3) ,G y (P3) ) and the green primary (G x (R2020) ,G y (R2020) ) any point along the line between the two green primaries in the P3 and R.2020 color spaces can be expressed as a linear combination of these two green primaries with a weighting factor a as follows: G x (a) =aG x (P3) +(1-a)G x (R2020) (4-1) G y (a) =aG y (P3) +(1-a)G y (R2020) (4-2)

[0104] Thus, the optimization problem of finding the maximum support from the (a) color space for the R.2020 color space can be simplified as a problem of selecting a weighting factor a. For a=0, the (a) color space is the entire R.2020 color space. For a=1, as shown in FIG. 2B (where the (a) color space is denoted as "TPB" or "current TPB cover colors"), the red and blue vertices or red and blue primaries of the (a) color space are the same as the red and blue vertices or red and blue primaries of the R.2020 color space, while the green vertex or green primary of the (a) color space is the same as the green vertex or green primary of the P3 color space. Figures 2C to 2E show three exemplary (a) color spaces for a=0.9, 0.5, and 0.25, respectively.

[0105] Congruent color space and TPB optimization Since the range or coverage by the backward reshaped (a) color space on the R.2020 color space can be controlled by the parameter a, the overall TPB (based reshaping) optimization problem becomes how to optimize the TPB coefficients in both the forward and backward paths such that (i) the forward reshaped SDR domain (or color space) for representing the reshaped SDR image is close to the reference SDR domain (or color space) for representing the reference SDR image (especially in the neutral color or color space parts that are more sensitive to HVS than the non-neutral color or color space parts), and (ii) the backward reshaped or reconstructed HDR domain (or color space) for representing the backward reshaped or reconstructed HDR image is as closely identical as possible to the reference HDR domain (or color space) for representing the reference HDR image. Ideally, if perfect reconstruction or perfect reversibility can be achieved, the backward reshaped or reconstructed HDR image is identical to the reference HDR image.

[0106] As shown in Figures 2B to 2E, in order to cover as much of the R.2020 color space as possible by the backward reshaped or reconstructed HDR domain or color space, the TPB optimization problem reduces to finding or searching the minimum possible value for the parameter a such that the above SDR and HDR quality conditions are met.

[0107] 3A illustrates an exemplary process flow for finding the minimum possible value of parameter a to maximize SDR and HDR quality in reshaped SDR and HDR images. The process flow of FIG. 3A may iterate through multiple candidate values ​​for parameter a in an iterative order, which may be sequential or non-sequential.

[0108] Block 302 includes selecting a current (e.g., to be iterated, etc.) value for parameter a to be the next candidate value (e.g., initially the first candidate value, etc.) in a plurality of candidate values ​​for parameter a. Given the backward reshaped HDR color space candidates along with the current value of parameter a, optimized reshaping operational parameters, such as optimized TPB coefficients, are obtained in one or more subsequent process flow blocks of FIG. 3A.

[0109] Block 304 includes constructing sample points or preparing two sampled data sets in a candidate backward reshaping HDR color space. By way of example and not limitation, the candidate backward reshaping HDR color space may be, but is not limited to, a Hybrid Log Gamma (HLG) RGB color space (referred to as "(a) RGB color space HLG").

[0110] The first of the two sampled data sets is a uniformly sampled data set of color patches. Each color patch in the uniformly sampled data set of color patches includes three dimensional directions, denoted as R-axis, G-axis, and B-axis for the R, G, and B component colors, respectively. (a) Each RGB color uniformly sampled from the RGB color space HLG

number

[0111] Each uniformly sampled data point or RGB color in a uniformly sampled data set

number

number

[0112] For simplicity, (i,j,k) in the above equation (5) may be vectorized or simply denoted as p. Correspondingly, the uniformly sampled data points or RGB colors

number

number

number

[0113] The second of the two sampled data sets prepared or constructed in block 304 is a neutral color data set. This second data set includes a plurality of neutral colors or neutral color patches (also called gray colors or gray color patches).

[0114] The second data set may be used to preserve input gray color patches in the input domain as output gray color patches in the output domain when the input gray color patches in the input domain are mapped or reshaped to output gray color patches in a reshaping operation as described herein. The input gray color patches in the input domain (or input color space) may be given increased weighting in the optimization problem compared to other color patches to reduce the likelihood that the reshaping operation will map these input gray color patches to non-gray color patches in the output domain (or output color space).

[0115] The second data set (gray color data set or gray color data set) may be prepared or constructed by uniformly sampling R,G,B values ​​along a line connecting the first gray color (0,0,0) and the second gray color (1,1,1) in the RGB domain (e.g., (a) RGB color space HLG, etc.), as follows: n Create nodes or gray patches.

number

[0116] All N in the second data set n The nodes can be grouped or organized into a neutral color vector / matrix as follows:

number

[0117] The neutral color vector / matrix in equation (8) above is N t(a positive integer greater than or equal to 1) times, so that the N n N t Neutral color patches can be generated.

number

[0118] The repetition of neutral colors in the second data set increases the weighting of neutral or gray colors relative to other colors, so that neutral colors can be preserved more in the optimization problem than other colors.

[0119] The first data set of (all sampled) colors and the second (repeated) data set of neutral colors in equations (6) and (9) can be collapsed or placed together into a single combined vector / matrix as follows:

number

[0120] The total number of vector / matrix elements (repeated and non-repeated color patches) in the combined vector / matrix (Equation (10)) is N = N n N t +N u Each vector / matrix element or color patch (row) in the above equation (10) may be expressed as:

number

[0121] Block 306 includes converting color values ​​of the color patches (rows) represented in the vector / matrix elements of the combined vector / matrix (Equation (10)) in (a) RGB color space (or (a) RGB color space HLG) to corresponding color values ​​in the standard-based R.2020 color space or R.2020 RGB color space HLG.

[0122] The three red, green, and blue primary colors in (a) RGB color space HLG, which correspond to the three vertices or points of the triangle defining (a) RGB color space HLG in the CIExy chromaticity diagram, can be converted from CIExy values ​​to CIE XYZ values ​​by the following formulas:

number

[0123] As red, green, and blue primaries in (x,y) or CIExy values ​​(R x (a) ,R y (a) ),(G x (a) ,G y (a) ),(B x (a) ,B y (a) ), then the (R X (a) ,R Y (a) ,R Z (a) ),(G X (a) ,G Y (a) ,G Z (a) ),(B X (a) ,B Y (a) ,B Z (a) ) can be obtained by using the conversion formula (12) above. Similarly, (W x (a) ,W y (a) ) as the white point in (x,y) or CIE xy values, then (W X (a) ,W Y (a) ,W Z (a)) can be obtained using the same transformation formula (12) above.

[0124] To convert the color values ​​of the color patches (rows) represented by the vector / matrix elements of the combined vector / matrix (Equation (10)) in (x,y) or CIE xy values ​​to the corresponding (X,Y,Z) or CIE XYZ values, P (a)→XYZ We can construct a 3×3 transformation matrix, denoted as

number

number

[0125] In order to convert the color values ​​of the color patches (rows) represented by the vector / matrix elements of the combined vector / matrix (Equation (10)) in the a RGB color space HLG to the corresponding color values ​​in the R.2020 RGB color space HLG, P (a)→R2020 We can construct a 3×3 transformation matrix, denoted as

number

[0126] Thus, in block 306, the color values ​​of the color patches (rows) represented by the vector / matrix elements of the combined vector / matrix (Equation (10)) in a RGB color space (or (a)RGB color space HLG) can be converted to corresponding color values ​​in the standard-based R.2020 color space or R.2020 RGB color space HLG as follows:

number

[0127] Block 308 converts the color values ​​of the color patches (rows) represented by the vector / matrix elements of the combined vector / matrix (Equation (18)) in the R.2020 RGB color space HLG into corresponding color values ​​in the R.2020 YCbCr color space HLG as follows:

number

number

[0128]

number

number

[0129] Block 310 converts the color values ​​of the color patches (rows) represented in the vector / matrix elements of the vector / matrix (Equation (18)) in the R.2020 RGB color space HLG into corresponding color values ​​in the R.709 SDR YCbCr color space.

number

number

number

[0130]

number

[0131] Block 312 converts the color values ​​of the color patches (rows) represented in the vector / matrix elements of the vector / matrix (Equation (18)) in the R.2020 RGB color space HLG into corresponding color values ​​in the R.2020 YCbCr color space PQ as follows:

number

number

number

[0132]

number

[0133] Block 314 is

number

[0134] The BESA algorithm is a sequential iterative algorithm in which each current iteration is able to modify the reference SDR signal according to the backward prediction error measured or determined in the previous iteration.

[0135] The modified BESA algorithm as described herein can implement neutral color preservation and avoid modifying neutral color patches. To do this, a set of neutral color patch indices that identify or correspond to neutral color patches may be generated as follows:

number

[0136] Let ch denote the channels (e.g., 3 channels) in the forward reshaping SDR domain or color space. Let F denote forward reshaping or forward path. Let B denote backward reshaping or backward path.

[0137] S F ch A forward reshaped SDR color patch characterized by a corresponding forward reshaped SDR codeword can be predicted from an input HDR color patch characterized by a corresponding input HDR codeword derived using equation (19) using a per-channel forward generator matrix (also called a design matrix) denoted by

number

number

number

number

[0138] The forward reshaped SDR color patches may be predicted by multiplying the forward generator matrix of equation (28) by the forward TPB coefficients. Examples of exemplary predicted reshaped codewords (equivalent or similar to color patches herein) using TPB coefficients are described in the aforementioned U.S. Provisional Patent Application No. 62 / 908,770.

[0139] Forward reshaped SDR color patches or codewords can be backward reshaped into backward reshaped HDR color patches or codewords, for example through a TPB-based backward reshaping operation.

[0140] In the backward pass, at each iteration of the BESA algorithm (e.g., the kth iteration), S B ch,(k) Using the per-channel backward generator matrix denoted by , a backward reshaped HDR color patch characterized by a corresponding backward reshaped HDR codeword can be predicted from a forward reshaped SDR color patch characterized by a corresponding forward reshaped SDR codeword derived in the forward pass. The backward generator matrix for each iteration step is calculated by using the cross-channel forward TPB basis functions

number

number

[0141] The backward reshaped HDR color patches can be predicted by multiplying the backward generator matrix in equation (29) with the backward TPB coefficients.

[0142] The backward prediction error for each iteration in the BESA algorithm can be determined by comparing the backward reshaped HDR color patches or codewords collected in a vector / matrix with the reference HDR color patches or codewords in the per-channel backward look vector / matrix derived from equation (25). The per-channel backward look vector / matrix can be generated or pre-computed from equation (25) and stored or cached in computer memory and fixed for every iteration, as follows:

number

[0143] A reference SDR signal or a reference SDR color patch or codeword may be used as a prediction target for a forward reshaping operation to generate a forward reshaped SDR color patch or codeword to approximate the reference SDR color patch or codeword. In the BESA algorithm, the reference SDR color patch or codeword

number

[0144] Using the reference SDR color patch or codeword for each channel at iteration step k, we calculate the vector / matrix r as follows: F ch,(k) can be constructed.

number

[0145] Before the first iteration step, the vector in equation (31) above is set to the original reference SDR signal represented by equations (22) and (23) as follows:

number

[0146] At iteration step k, the forward TPB coefficients (m F ch,(k) , can be generated via a least-squares solution to an optimization problem that minimizes the difference between the forward reshaped SDR color patches or codewords and the reference SDR color patches or codewords determined for that iteration, as follows:

number

[0147] The predicted SDR color patch or codeword in iteration step k for channel ch can be calculated from the optimized values ​​for the forward TPB coefficients (e.g., per channel) as follows:

number

[0148] At iteration step k, the posterior TPB coefficients (m B ch,(k)The backward-reshaped HDR color patch or codeword (denoted as ) can be generated via a least-squares solution to an optimization problem that minimizes the difference between the backward-reshaped HDR color patch or codeword and the reference HDR color patch or codeword (the latter is fixed for all iterations in the BESA algorithm).

number

[0149] The predicted HDR color patch or codeword in iteration step k for channel ch can be calculated from the optimized values ​​for the backward TPB coefficients (e.g., per channel) as follows:

number

[0150] Calculate the backward prediction error and back-propagate the error to the reference non-neutral SDR signal.

[0151] As mentioned above, the backward prediction error for each iteration in the BESA algorithm can be determined by comparing the backward reshaped HDR color patch or codeword with a reference HDR color patch or codeword derived from equation (25) (the latter is fixed for all iterations in the BESA algorithm).

[0152] Among the backward prediction errors, the backward prediction errors for non-neutral colors (or non-gray colors) can be back-propagated to update or modify the reference SDR color patches or code words for these non-neutral colors (or non-gray colors). The modified SDR color patches or code words for non-neutral colors (or non-gray colors) can be combined with the (unmodified) reference SDR color patches or code words for neutral colors (or gray colors) to serve as prediction targets for the forward pass in the next or upcoming iteration step in the BESA algorithm.

[0153] In some operating scenarios, the backward prediction error can be calculated as the difference between the original or reference HDR signal (or reference HDR color patches / codewords therein) and the predicted HDR signal (or backward reshaped HDR color patches / codewords therein) at iteration step k, as follows:

number

[0154] In response to determining that color patch i is within the neutral color set Φ, the reference SDR codeword for the color patch in the Cb and Cr channels of the reference SDR signal for the next iteration step (k+1) may be set to a gray color value, such as 0.5, as follows:

number

[0155] Instead, in response to determining that color patch i is not within the neutral color set Φ, the reference SDR codeword for the color patch in the Cb and Cr channels of the reference SDR signal for the next iteration step (k+1) may be set to an updated or modified value according to the backward prediction error, as follows:

number

[0156] The neutral color preservation implemented using equations (38) to (40) can be used to ensure that neutral colors in the reference SDR signal do not deviate to or towards non-neutral colors for every iteration in the BESA algorithm, resulting in improved perceptual quality related to gray levels in the reshaped image.

[0157] The BESA algorithm may be iteratively iterated up to a total number of iterations. In some operating scenarios, the total number of iterations for the BESA algorithm may be specifically selected to balance color deviations in the reshaped SDR and / or HDR images. For example, different total numbers of iterations in the BESA algorithm may generate forward and backward TPB coefficients that generate reshaped SDR images with different SDR appearances and reshaped HDR images with different HDR appearances. A total number of iterations, such as 10, 15, etc., that generates reshaped SDR and HDR images with relatively high quality appearances may be selected for the BESA algorithm.

[0158] A BESA algorithm with neutral color preservation may be performed for each candidate value for parameter a. For example, in block 314, a BESA algorithm with neutral color preservation may be performed for the current candidate value of parameter a.

[0159] Block 316 involves determining whether the current candidate value for parameter a is the last candidate value of multiple candidate values ​​for parameter a. If so, process flow proceeds to block 318. If not, process flow returns to block 302.

[0160] Block 318 involves selecting an optimal or optimized value for parameter a, and calculating (or simply selecting those already calculated) optimized forward and backward TPB coefficients for (a) RGB color space that correspond to the optimized value for parameter a.

[0161] The optimal choice of value for parameter a is {a,m F,a ch ,m B,a ch The optimization can be formulated as an optimization problem to find (a particular set of values ​​for) {\displaystyle \mathbb {R}\ ...

number

number

number

[0162] This optimization problem involves converting the results from each a RGB color space into their corresponding optimal or optimized TPB coefficients m F,a ch and m B,a ch Among all possible or candidate a values, the smallest a (corresponding to the maximum color space coverage in the R.2020 color space) that can achieve minimized HDR error with a relatively small SDR error. HDR prediction error

number

number

[0163] 2F and 2G show example percentile Cb and Cr values ​​in the forward reshaped SDR domain or color space for different values ​​of parameter a. Different values ​​of parameter a determine different color space coverage of the backward reshaped HDR domain or color space in the R.2020 domain or color space, and thus affect different reconstructions or generation of the backward reshaped HDR image in the R.2020 domain or color space, and different distortions of the backward reshaped HDR image compared to the reference HDR image to which the backward reshaped HDR image approximates, e.g., distortion metrics

number

[0164] The vertical axis of FIG. 2F represents the 3 percentile values ​​in the Cb and Cr channels in the forward reshaped SDR domain for different values ​​for the parameter a value represented by the horizontal axis of FIG. 2F (e.g., obtained using the BESA algorithm with neutral color preservation for 10 successive iterations). The smaller the value for the parameter a, the greater the range that the a color space covers in the R.2020 color space. The 3 percentile value may be used explicitly or implicitly to represent the minimum value in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space. The smaller the 3 percentile value, the smaller the minimum value and thus the more extended the codeword value range in the forward reshaped SDR (YCbCr) domain or color space. As can be seen from FIG. 2F, the lower the value for the parameter a, the lower the 3 percentile value in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space.

[0165] The vertical axis of FIG. 2G represents the 97th percentile values ​​in the Cb and Cr channels in the forward reshaped SDR domain for different values ​​for the parameter a value represented by the horizontal axis of FIG. 2G (e.g., obtained using the BESA algorithm with neutral color preservation for 10 successive iterations). The 97th percentile values ​​may be used explicitly or implicitly to represent the maximum values ​​in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr) domain or color space. The larger the 97th percentile value, the larger the maximum value and thus the more extended the codeword value range in the forward reshaped SDR (YCbCr) domain or color space. As can be seen from FIG. 2G, the 97th percentile values ​​in both the Cb and Cr channels of the forward reshaped SDR domain or color space maintain approximately constant values.

[0166] In some operating scenarios, the image data set used in block 318 to determine distortion in the reshaped SDR and HDR images and to select an optimized value for parameter a based in part or in whole on the distortion may include a video bitstream acquired using one or more specific video capture devices such as a mobile phone (e.g., supporting SMPTE ST 2094 video coding profile, supporting Dolby Vision profile 8.4, etc.). A non-limiting example of an optimized value for parameter a may be, but is not necessarily limited to, 0.5. This determines the maximum color space coverage in the R.2020 color space for the backward reshaped HDR domain or color space. When a wider backward reshaped color space (e.g., corresponding to a smaller value than the optimized value for parameter a) is used or supported, the reshaped SDR colors and codewords may begin to deviate from the reference SDR colors and codewords with a relatively large distortion. This is because increasing the backward reshaped HDR domain or color space means packing distinct forward reshaped SDR colors (or codewords) more tightly into the forward reshaped SDR domain or color space, leading to the forward reshaped SDR colors moving from their original 3D positions represented in the reference SDR colors (or codewords) that are approximated by the forward reshaped SDR colors.

[0167] As mentioned above, forward and backward TPB coefficients for a TPB-based reshaping operation can be determined for a given value, such as an optimized value for parameter a. These TPB coefficients can be multiplied in a formula with a forward or backward generator matrix constructed with TPB basis functions using the input codeword as an input parameter to generate a forward or reshaped codeword, which may involve performing numerous calculations involved in calculating TPB basis function values ​​for each pixel of a large number of pixels of an image.

[0168] In some operating scenarios, to speed up or reduce computations, forward and backward 3D-LUTs may be constructed from pre-computed TPB basis function values ​​from sampled values ​​and optimized forward and backward TPB coefficients generated for an optimized value for parameter a. The forward and backward 3D-LUTs or lookup nodes / entries therein may be pre-constructed before being deployed at run-time to process an input image, and applied at run-time with respect to the input image with relatively simple lookup operations in the forward and backward paths, or with corresponding forward and backward reshaping operations performed therein.

[0169] The optimized value for parameter a is opt The corresponding optimal forward and backward TPB coefficients are denoted as m F ch,opt and m B ch,opt It is written as follows.

[0170] The forward 3D-LUT can be used to forward reshape input HDR colors (or codewords) in an input HDR domain or color space, such as the R.2020 domain or color space (which the a color space is included in), to forward reshaped SDR colors (or codewords) in a forward reshaped SDR domain or color space.

[0171] A color space identified within an input HDR domain or color space, such as the R2020 (container) domain or color space, may be used to clip out HDR colors or codewords that are represented outside the a color space. The forward TPB coefficients may be applied to input HDR colors or codewords in a color space identified within an input HDR domain or color space, such as the R2020 (container) domain or color space, to generate, for each lookup node or entry in the forward 3D-LUT, a predicted or forward reshaped SDR color or codeword. As a result, the forward 3D-LUT includes multiple lookup nodes or entries, each of which maps or forward reshaped a respective input (cross-channel or three-channel) HDR color or codeword to a corresponding predicted or forward reshaped (cross-channel or three-channel) SDR color or codeword.

[0172] In some operating scenarios, the forward 3D-LUT construction process involves a first step where a 3D uniform sampling grid is prepared in the input HDR domain or color space, such as the R2020 (container) domain or color space.

[0173] By way of example and not limitation, the input HDR domain or color space is the R.2020 YCbCr color space HLG, which includes three dimensions or channels: Y axis, Cb axis, and Cr axis. The input HDR values ​​consist of normalized values ​​in the value range of [0,1] on each axis.

number

number

[0174] Although the valid values ​​in the entire R.2020 YCbCr color space may be limited (so as not to cover the entire 3D cube of YCbCr values), the entire sampling value grid may be used to cover the entire 3D cube of YCbCr values, to reduce the chance of codeword deviations caused by the compression operation.

[0175] For simplicity, (i,j,k) is vectorized or denoted as p. Thus, the sampled value in R.2020 YCbCr is

number

number

number

[0176] The forward 3D-LUT construction process includes a second step in which the sampled values ​​in the R.2020 YCbCr color space HLG may be converted to corresponding values ​​(or colors) in the R.2020 RGB color space HLG as follows:

number

[0177] The forward 3D-LUT construction process is such that the converted values ​​in the R.2020 RGB color space HLG are converted to the optimized values ​​for the parameter a (a optThe third step involves converting the corresponding values ​​(or colors) in the RGB color space HLG (denoted as a

number

[0178] The forward 3D-LUT construction process includes a fourth step in which (a) the converted values ​​in the RGB color space HLG may be clipped as follows:

number

[0179] The forward 3D-LUT construction process includes a fifth step in which the clipped values ​​in the a RGB color space HLG may be converted to corresponding values ​​(or colors) in the R.2020 RGB color space HLG as follows:

number

[0180] The forward 3D-LUT construction process includes a sixth step in which values ​​in the R.2020 RGB color space HLG derived using equation (47) above may be converted to corresponding values ​​(or colors) in the R.2020 YCbCr color space HLG as follows:

number

[0181] The forward 3D-LUT construction process includes a seventh step, where for each lookup node / entry in the forward 3D-LUT, the optimized forward TPB coefficients are added to the forward generator matrix S F chThe forward generator matrix is ​​constructed with the input HDR colors or codewords (or codeword values) in the R.2020 YCbCr color space HLG derived using equation (48) above as input parameters to the forward TPB basis functions. The mapped or forward reshaped SDR colors or codewords (or codeword values) are obtained as follows:

number

[0182] In the above equation (49), the forward generator matrix S F ch is the input HDR color or codeword (or codeword value) V YCbCr (FL),(R2020) can be constructed or constructed using as input to the forward TPB basis functions.

number

[0183] The mapped or forward reshaped SDR color or codeword (or codeword value) (Eq. (49)) may be used as the lookup value of the lookup node / entry of the forward 3D-LUT, while the input HDR color or codeword (or codeword value) used as the input parameter to the forward TPB basis function may be used as the lookup key of the lookup node / entry of the forward 3D-LUT. At runtime, the mapped or forward reshaped SDR color or codeword may simply be looked up in the forward 3D-LUT with the lookup key as the input HDR color or codeword to be forward reshaped to the mapped or forward reshaped SDR color or codeword.

[0184] A backward 3D-LUT as described above can be used to backward reshape reshaped SDR colors (or codewords) in a forward reshaped SDR domain or color space to backward reshaped HDR colors (or codewords) in a backward reshaped HDR domain or color space, such as the R.2020 domain or color space (which includes the a color space).

[0185] In some operating scenarios, a backward 3D-LUT construction process, which may be simpler than the forward 3D-LUT construction process discussed previously, may be implemented or executed to build or construct the backward 3D-LUT. In some operating scenarios, to speed up or reduce computations, the backward 3D-LUT may be constructed from pre-calculated TPB basis function values ​​from sampled values ​​and optimized backward TPB coefficients generated for an optimized value for parameter a. The backward 3D-LUT or lookup nodes / entries therein may be pre-constructed before being deployed at run-time to process an input image, such as a forward reshaped SDR image, and at run-time, a relatively simple lookup operation in the backward path or a corresponding backward reshaping operation performed therein may be applied on the input image.

[0186] The pre-constructed backward 3D-LUT can be deployed at a decoder side, where a receiving downstream device at the decoder side can receive and decode a video signal encoded with a forward reshaped SDR image in a forward reshaped SDR domain or color space, and can apply backward reshaping to the forward reshaped SDR image using the 3D-LUT to generate a backward reshaped HDR image in a backward reshaped HDR domain or color space, such as a color space included in the R.2020 domain or color space.

[0187] While the shape formed or described by the complete boundary of the forward reshaped SDR domain or color space mapped in the TPB-based forward reshaping operation may not be a simple 3D cube, the forward reshaped SDR values ​​in the forward reshaped SDR domain or color space may be clipped to the tightest or smallest 3D cube (e.g., a 3D rectangle, a rescaled 3D cube from a 3D rectangle, etc.) that contains or supports all the forward reshaped SDR values ​​represented in all lookup nodes / entries of the forward 3D-LUT, without, for example, clipping these SDR values ​​represented in the forward 3D-LUT.

[0188] In some operating scenarios, the backward 3D-LUT construction process includes a first step in which a complete sampling value grid may be prepared in a forward reshaped SDR domain or color space, such as the entire R.709 SDR YCbCr color space.

[0189] The backward 3D-LUT construction process includes a second step in which minimum and maximum values ​​for each dimension or (color) channel in the forward reshaped SDR domain or color space are determined among the forward reshaped SDR values ​​in the forward 3D-LUT and may be used to restrict the (input) data range of the forward reshaped SDR (e.g., YCbCr, etc.) colors or codewords that serve as input for the backward path.

[0190] The backward 3D-LUT construction process includes a third step, where for each lookup node / entry in the backward 3D-LUT, the optimized backward TPB coefficients are added to the backward generator matrix S B ch(e.g., as shown in equation (29) above) which is constructed using the forward reshaped SDR colors or codewords (or codeword values) within the clipped or limited (input) data range derived in the second step of the backward 3D-LUT construction process as input parameters to the backward TPB basis functions to obtain the mapped or backward reshaped HDR colors or codewords (or codeword values).

[0191] The mapped or backward reshaped HDR colors or codewords (or codeword values) may be used as lookup values ​​of the lookup nodes / entries of the backward 3D-LUT, while the input or forward reshaped SDR colors or codewords (or codeword values) used as input parameters to the backward TPB basis functions may be used as lookup keys of the lookup nodes / entries of the backward 3D-LUT. At runtime, the mapped or backward reshaped HDR colors or codewords can simply be looked up in the backward 3D-LUT with the lookup keys as the input or forward reshaped SDR colors or codewords that are backward reshaped to the mapped or backward reshaped HDR colors or codewords.

[0192] White Box Backward Optimization (WB) Forward TPB-based reshaping or a corresponding forward 3D-LUT may not be used in all video capture and / or editing devices due to the cost and / or computational overhead involved in implementing or operating with forward TPB-based reshaping or a forward 3D-LUT in a video capture and / or editing application.

[0193] In some operating scenarios, a hardware-based solution, such as implemented with an available ISP, may be used in a video capture and / or editing device to perform HDR-to-SDR conversion, such as HDR HLG to SDR image generation, and an existing ISP pipeline deployed with the device may operate with one or more programmable parameters to generate, convert, and / or output an SDR image based, at least in part or in whole, on a corresponding HDR (e.g., HLG, etc.) image captured by the device.

[0194] In the forward path, the programmable parameters for the ISP pipeline can be specifically set or configured to cause the ISP pipeline to output an SDR image that approximates as closely as possible a reference SDR image generated using a white-box HDR-SDR conversion function.

[0195] In the backward path, a backward reshaped HDR image can be generated from the SDR image output from the ISP pipeline using the backward TPB-based reshaping or a corresponding backward 3D-LUT. The backward TPB coefficients used in the backward path may be optimized, for example, to cover as much of the backward reshaped HDR domain or color space, e.g., the R.2020 color space, as possible.

[0196] FIG. 3B illustrates an exemplary process flow for determining or generating optimized values ​​for one or more programmable parameters in an ISP pipeline used to generate or output an SDR image from an HDR image (ISP).

[0197] In some operating scenarios, the ISP pipeline may be a relatively fixed, programmable module implemented in ISP (hardware) to approximate a white-box HDR-SDR conversion function, such as a white-box (e.g., known, well-defined, etc.) conversion function for converting an HDR HLG image to a corresponding SDR image in block 330.

[0198] The ISP pipeline may include or implement: (1) a first set of three one-dimensional lookup tables (1 D-LUTs) for HLG RGB to linear RGB conversion, (2) a 3×3 matrix for conversion from an HDR image in an HDR domain or color space, such as the R.2020 color space, to an SDR domain or color space, such as the R.709 color space, followed by (3) a second set of three 1D-LUTs implementing a BT.1886 linear to nonlinear (gamma) conversion using the (e.g., standard-based) HLG optical-to-optical transfer function (OOTF).

[0199] For illustrative purposes only, the HDR image may be an HDR HLG image retrieved from an image dataset or database, as shown in Figure 3B. The one or more programmable parameters for the ISP pipeline may be a design parameter (γ BT1886 ) may be used.

[0200] The process flow of FIG. 3B is based on a design parameter γ that determines how the ISP SDR image best approximates the reference SDR image generated in block 330. BT1866 The process flow can be used to search for an optimized value for the design parameter γ BT1886 The method may iterate through multiple candidate values ​​for , in an iteration order that may be sequential or non-sequential.

[0201] Block 322 determines the design parameter γ BT1866Let the current (e.g., to be iterated) value for the design parameter γ BT1866 selecting to a next candidate value (e.g., initially to a first candidate value, etc.) in the plurality of candidate values ​​for the design parameter γ BT1866 Given the current value for , the ISP SDR image can be compared to a reference SDR image in one or more subsequent process flow blocks of FIG. 3B.

[0202] Block 324 involves applying the first set of 1D-LUTs to the HDR HLG(RGB) image retrieved from the database to generate a corresponding HDR linear RGB image, as follows:

number

[0203] Block 326 includes applying the 3×3 matrix to a corresponding HDR linear RGB image to generate a corresponding SDR linear RGB image. The 3×3 matrix may be given as: P R2020→R709 =P R2020→XYZ) P XYZ→R709 (52)

[0204] The SDR linear RGB codeword in the corresponding SDR linear RGB image generated using the 3 × 3 matrix in equation (52) above is l ch (R709) It may be expressed as:

[0205] Block 328 maps the second set of 1D-LUTs to the SDR linear RGB codewords l in the corresponding SDR linear RGB image. ch (R709) , and the ISP SDR codeword (s ch (R709) The method includes generating a

[0206] In some operating scenarios, as described above, the second set of 1D-LUTs merges or combines the linear to non-linear SDR conversion with the HLG OOTF, as follows:

number

number

[0207] The intermediate SDR codeword (dotted l) in the above equation (53) ch (R709) ) to ISP SDR codeword s ch (R709) The linear to nonlinear SDR transformation used to generate may be the linear to gamma transformation defined in BT.1886 as follows:

number

[0208] Blocks 322-328 of FIG. 3B - or the combination of the first set of 1D-LUTs represented by equation (51) above, the 3×3 matrix represented by equation (52) above, and the second set of 1D-LUTs represented by equations (53)-(56) above - are implemented using an ISP pipeline to collectively determine the design parameter γ BT1886 The HDR HLG to SDR conversion function (f HLG→ISPSDR Implement the following:

[0209] Block 332 converts the HDR HLG to SDR conversion function f HLG→ISPSDR and a reference SDR image generated from a white-box HDR-SDR conversion function in block 330. A quality assessment function such as MSE, RMSE, SAD, PSNR, SSIM, etc. may be used to calculate the difference.

[0210] Block 334 is a step for determining the design parameter γ BT1886 The current candidate value for the design parameter γ BT1886 If so, process flow proceeds to block 336. If not, process flow returns to block 322.

[0211] Block 336 determines the design parameter γ BT1886 The method includes selecting an optimal or optimized value for the design parameter γ BT1886 The selection of an optimized value for γ is determined by selecting the design parameter γ from among multiple candidate values ​​such that the difference between the ISP SDR image and a reference SDR image calculated using a relatively large image data set or database is minimized, as described below. BT1886 A specific value for (γ BT1886,opt This can be formulated as an optimization problem to find

[0212]

number

[0213] Here, D() represents the quality assessment function used in block 332.

[0214] Gamma BT1886,opt An example value of may be, but is not limited to, 2.115.

[0215] SDR to PQ TPB optimization As described above, in a "WFB" use case or operating scenario, the BESA algorithm may be used under the SLBC framework to generate optimized reshaping operation parameters used by the forward and backward reshaping operations to generate a forward reshaped SDR image and a backward reshaped HDR image, whereas in a "WB" use case or operating scenario, only backward reshaping may be performed under the SLiDM framework on a non-forward reshaped SDR image encoded in an output video signal, such as an ISP SDR image generated using an ISP pipeline implemented with a video capture and / or editing device, to generate a corresponding backward reshaped or reconstructed HDR image.

[0216] In a "WB" use case or operating scenario, forward TPB reshaping from an input HDR domain or color space, such as the R.2020 HDR color space HLG, to a forward reshaped SDR domain or color space, such as the R.709 SDR YCbCr color space, could not be used to help exploit the full range of codewords supported by the R.709 SDR YCbCr color space. The ISP SDR codewords in the ISP SDR image encoded in the output video signal are often hard-limited in the R.709 color space portion generated or supported by the video capture and / or editing device or the ISP pipeline implemented therein. It may be difficult for the ISP SDR codewords in the hard-limited R.709 color space portion to be further expanded or mapped by backward reshaping, such as backward TPB-based reshaping, to a backward reshaped or reconstructed HDR domain or color space.

[0217] However, as in the "WFB" use cases or operating scenarios, in "WB" use cases or operating scenarios, optimized reshaping operating parameters such as TPB coefficients may help to expand the coverage of supported color spaces in which the backward reshaped HDR image produced in the reshaping optimization is represented. In many "WB" operating scenarios, the backward reshaped or reconstructed HDR domain or color space achieved with backward TPB-based reshaping can be only slightly larger than the R.709 color space compared to the maximum (a) color space achievable in the "WFB" use cases or operating scenarios.

[0218] For illustrative purposes only, in an operating scenario such as that shown in FIG. 1B, the ISP SDR domain or color space may be that of R.709 or a hard-limited portion thereof in the ISP pipeline, while the (original or backward reshaped) HDR domain or color space may be that of R.2020.

[0219] By way of example and not limitation, a subset or subspace in the R.2020 color space (or an HDR color space for representing a reshaped HDR image) can be defined or characterized by a particular white point and three particular primary colors (red, green, blue). The particular white point may be selected or fixed to be the D65 white point.

[0220] The backward reshaped HDR color space for representing the backward reshaped or reconstructed HDR image is denoted as (b) color space. Therefore, the CIExy coordinates of the primary colors and white point that define the (b) color space are (R x (b) ,R y (b) ),(G x (b) ,G y (ab) ),(B x (b) ,B y (b) ) and (W x (b) ,W y (b) (b) The color space can be specified as a D65 white point.

[0221] (b) Each of the primary colors in the color space is a corresponding primary color in the P3 color space (G x (P3) ,G y (P3) ) and each primary color (G x (R2020) ,G y (R2020) ) Any point along the line between two respective primaries in the P3 color space and the R.2020 color space can be expressed as a linear combination of these two respective primaries with a weighting factor b as follows:

number

[0222] Thus, the optimization problem of finding the maximum support from the (b) color space for the R.2020 color space can be simplified as a problem of selecting a weighting factor b. For b=0, the (b) color space is the entire R.2020 color space. For b=1, the (b) color space is the P3 color space, as shown in FIG. 2K (where the (b) color space is denoted as "TPB" or "colors covered by TPB (b color space)"). Figures 2H to 2J show three exemplary (b) color spaces for b=0.25, 0.50, and 0.75, respectively.

[0223] As shown in Figures 2H to 2K, the TPB optimization problem reduces to finding or searching for the minimum possible value for the parameter b in order to cover as much of the R.2020 color space as possible by the backward reshaped or reconstructed HDR domain or color space.

[0224] 3C illustrates an exemplary process flow for finding the minimum possible value for parameter b. The process flow of FIG. 3C may iterate through multiple candidate values ​​for parameter b in an iterative order that may be sequential or non-sequential.

[0225] Block 342 includes selecting the current (e.g., iterated, etc.) value for parameter b to be the next candidate value (e.g., initially the first candidate value, etc.) among multiple candidate values ​​for parameter b. Given a candidate backward reshaped HDR color space with a current value for parameter b, optimized reshaping operation parameters, such as optimized TPB coefficients, may be obtained in one or more subsequent process flow blocks of FIG. 3C.

[0226] Block 344 includes constructing sample points or preparing two sampled data sets in a candidate backward reshaped HDR color space. By way of example and not limitation, the candidate backward reshaped HDR color space may be, but is not limited to, a Hybrid Log-Gamma (HLG) RGB color space (referred to as "(b) RGB color space HLG").

[0227] Similar to block 304 of FIG. 3A, the first of the two sampled data sets is a uniformly sampled data set of color patches. Each color patch in the uniformly sampled data set of color patches includes three dimensions, denoted as R axis, G axis, and B axis for the R, G, and B component colors, respectively. (b) Each RGB color uniformly sampled from the RGB color space HLG.

number

number

[0228] For simplicity, (i,j,k) can be vectorized or simply denoted as p. Correspondingly, we can use the uniformly sampled data points or RGB colors (dotted v ijk (u) ) is simply

number

number

[0229] The second of the two sampled data sets prepared or constructed in block 344 is a neutral color data set. This second data set includes a plurality of neutral colors or neutral color patches (also called gray colors or gray color patches).

[0230] The second data set may be used to preserve input gray color patches in the input domain as output gray color patches in the output domain when the input gray color patches in the input domain are mapped or reshaped to output gray color patches in a reshaping operation as described herein. The input gray color patches in the input domain (or input color space) may be given increased weighting in the optimization problem compared to other color patches to reduce the likelihood that the reshaping operation will map these input gray color patches to non-gray color patches in the output domain (or output color space).

[0231] A second data set, a gray color data set or gray color data set, is prepared or constructed by uniformly sampling R,G,B values ​​along a line connecting the first gray color (0,0,0) and the second gray color (1,1,1) in the RGB domain (e.g., (b) RGB color space HLG, etc.), as follows: n This produces nodes or gray patches:

number

[0232] N in the second data set n All these nodes can be grouped or summarised into a neutral colour vector / matrix as follows:

number

[0233] The neutral color vector / matrix in equation (8) above is N t (a positive integer greater than or equal to 1) times, so that the N n N t Neutral color patches can be generated.

number

[0234] The repetition of neutral colors in the second data set increases the weighting of neutral or gray colors relative to other colors, so that neutral colors can be preserved more in the optimization problem than other colors.

[0235] The first data set of (all sampled) colors and the second (repeated) data set of neutral colors in equations (61) and (64) can be collapsed or put together into a single combined vector / matrix as follows:

number

[0236] Combined Vectors / Matrices (V with tilde RGB (b) The total number of vector / matrix elements (repeated and non-repeated color patches) in n N t +N u The above formula (10) (with a tilde V RGB (b)Each vector / matrix element or color patch (row) in may be written as:

number

[0237] Block 346 calculates the combined vector / matrix (V with tilde) in (b) RGB color space (or (b) RGB color space HLG). RGB (b) ) to corresponding color values ​​in the standard-based R.2020 color space or the R.2020 RGB color space HLG as follows:

number

number

[0238] Block 348 converts vectors / matrices (V with tilde) in the R.2020 RGB color space HLG as follows: RGB (R2020) ) to convert the color values ​​of the color patches (rows) represented in the vector / matrix elements into the corresponding ISP SDR color values ​​in the R.709 SDR RGB color space.

number

number

number

[0239] S with tilde YCbCr (R709) Each color patch (row) in can be represented as:

number

[0240] Block 350 converts vectors / matrices (V with tilde) in the R.2020 RGB color space HLG as follows: RGB The color values ​​of the color patches (rows) represented in the vector / matrix elements of (R2020) are converted to the corresponding color values ​​in the R.2020 YCbCr color space PQ.

number

number

[0241] R with tilde YCbCr (R2020) Each color patch (row) in can be represented as:

number

[0242] Block 352 is S with a tildeYCbCr (R709) and R with a tilde YCbCr (R2020) as input to a backward TPB optimization algorithm to generate optimized backward TPB coefficients for TPB-based reshaping for the (b) color space that corresponds to the current value for parameter b.

[0243] To do this, we use the TPB basis set and the S YCbCr (R709) From the SDR codewords represented in , a backward generator matrix may be constructed as follows:

number

[0244] The backward prediction error can be determined by comparing the backward reshaped HDR color patches or codewords organized in a vector / matrix with the reference HDR color patches or codewords in the per-channel backward look vector / matrix derived and minimized in solving the TPB optimization problem. The per-channel backward look vector / matrix can be generated or pre-computed from equation (71) above and stored or cached in computer memory and fixed for all iterations as follows:

number

[0245] The backward TPB coefficients (e.g., per channel) (m B ch Optimized values ​​for (denoted as ) can be generated via a least-squares solution to an optimization problem that minimizes the difference between the backward-reshaped HDR color patches or codewords and the reference HDR color patches or codewords, as follows:

number

[0246] The per-channel predicted (or backward reshaped or reconstructed) HDR codeword for each channel ch can be calculated as follows:

number

[0247] Block 354 involves determining whether the current candidate value for parameter b is the last candidate value among multiple candidate values ​​for parameter b. If so, process flow proceeds to block 356. If not, process flow returns to block 342.

[0248] Block 356 involves selecting an optimal or optimized value for parameter b, and calculating (or simply selecting those already calculated) optimized forward and backward TPB coefficients for the (b) RGB color space that correspond to the optimized value for parameter b.

[0249] Similar to the "WFB" use case or operating scenario, in the "WB" use case or operating scenario, the (b) color space is also part of the optimization. The (b) color space and the backward TPB coefficients can be jointly optimized. Thus, the optimization problem is to find a color space that covers the maximum HDR color space or color space portion, while still being able to optimize the DHDR (r B ch ,^r B ch ) so that (b) a color space is generated that achieves a minimized HDR prediction error, denoted as {b,m B,b ch} can be formulated as finding (a particular value for).

number

[0250] An exemplary optimized value for the parameter b may be, but is not limited to, 1, which corresponds to the P3 color space.

[0251] Black-box TPB backward optimization on a single device (BB1) Some (dual mode) video capture devices or mobile devices support both SDR and HDR capture modes and can therefore output either SDR or HDR images. Some (mono mode) video capture devices or mobile devices support only SDR capture mode. Under the techniques described herein, SDR-HDR mappings can be modeled or generated using a dual mode video capture device. Some or all of these SDR-HDR mappings can be applied, e.g., in a downloaded and / or installed video capture application, to SDR images captured by either a dual mode video capture device or a mono mode video capture device to "upconvert" SDR images acquired in SDR capture mode to corresponding HDR images, regardless of whether these video capture devices support HDR capture mode.

[0252] In some operating scenarios, since the computational environment or processing power of a mobile device may be limited, a static (backward) 3D-LUT derived at least in part from the TPB basis functions and optimized TPB coefficients may be used, which represents a static SDR-HDR mapping that maps or backward reshapes SDR images, e.g., all SDR images in an SDR video sequence, to generate corresponding HDR images, e.g., all HDR images in a corresponding HDR video sequence.

[0253] The static SDR-HDR mapping may be modeled at least in part based on a plurality of image pairs of HDR and SDR images captured by a particular camera of a dual mode video capture device. Each image pair in the HDR and SDR image image pairs includes an SDR image and an HDR image corresponding to the SDR image. The SDR and HDR images depict the same visual scene in the real world and may be subject to spatial alignment errors caused by spatial translation that may occur between a first time when the SDR image is captured by a particular camera operating in an SDR capture mode and a second time when the HDR image is captured by the same camera operating in an HDR capture mode. For example, for each visual scene in the physical world, the video capture device may operate in an HDR mode to capture an HDR image in an image pair and use an SDR mode to capture an SDR image in the same image pair. This may be repeated to generate a plurality of image pairs of HDR and SDR images to generate a relatively large number of different color patches (or a relatively large number of different colors or different code words) that are used to generate the static SDR-HDR mapping.

[0254] There are several challenges in modeling a static SDR-HDR mapping using captured SDR and HDR images. First, although both SDR and HDR capture modes can use the same ISP within the same video capture device, the video capture device can apply different image capture settings (e.g., exposure settings for a particular camera, etc.) for the same scene to obtain the designed optimized image quality in the captured SDR and HDR images. As a result, the static SDR-HDR mapping may be able to model or approximate to some extent the HDR image capture settings actually implemented by the video capture device. Second, the spatial alignment between the captured SDR and HDR images used to derive the image pairs of the SDR and HDR images may not be strict. For example, the selection of a particular capture mode between the SDR and HDR capture modes may be performed by touching the screen of the video capture device, which may cause the video capture device to lose or move from its previous spatial position and / or orientation. Furthermore, temporal alignment between captured SDR video / image sequences and captured HDR video / image sequences can easily occur because these SDR and HDR sequences are captured at different time instances or durations. Some depicted visual objects in these sequences may enter or leave the camera field of view. Some local regions within the camera field of view may have occlusions or de-occlusions from time to time.

[0255] In some operating scenarios, to solve spatial and / or temporal alignment problems between a captured SDR image and a corresponding captured HDR image, a registration operation is performed to generate an aligned SDR image and a corresponding aligned HDR image, and the aligned SDR image and the corresponding aligned HDR image can be included in an image pair of an SDR image and an HDR image. The image pair can then be used to determine or establish SDR color patches or colors in the aligned SDR image and corresponding HDR color patches or colors in the corresponding aligned HDR image. These SDR and HDR color patches or colors can be used to derive at least some color patches in a set of corresponding SDR and HDR color patches or colors for the purpose of deriving or generating a static SDR-HDR mapping.

[0256] The SDR and HDR video capture process may be performed to provide comprehensive coverage of different scenes and / or exposures at different times of day, such as from early morning to late night, for both indoor and outdoor scenes. For each scene, the same video capture device may be used in both SDR and HDR modes to capture SDR and HDR video sequences. In some operational scenarios, specific frames / images, such as the first frame / image of each of the video sequences, may be selected or extracted for the purpose of constructing or deriving multiple image pairs of SDR and HDR images to be included in a training image dataset or database.

[0257] FIG. 3D illustrates an exemplary process flow for generating optimized reshaping operational parameters, such as optimized or optimal TPB coefficients, using corresponding (or matching) SDR and HDR color patches or colors in a captured SDR image and a captured HDR image (corresponding to the captured SDR image), captured by a video capture device operating in SDR and HDR capture modes, respectively.

[0258] Block 362 includes receiving a captured SDR image and locating or extracting a set of SDR image feature points in the captured SDR image. Block 364 includes receiving a captured HDR image and locating or extracting a set of HDR image feature points in the captured HDR image. The set of SDR image feature points and the set of HDR image feature points may be sets of the same feature point type.

[0259] The set of feature point types may include feature point types in a wide variety of different feature point types, including, but not necessarily limited to, any, some, or all of the following: Binary-Robust-Invariant-Scalable-Keypoints or BRISK features detected with the BRISK algorithm; Features from Accelerated-Segment-Test or corners detected using the FAST algorithm; features detected from the KAZE algorithm; corners detected using the Minimum Eigenvalue algorithm; maximally-stable-extremal-regions or features generated with the MSER algorithm; keypoints detected with the Oriented-FAST-and-Rotated or ORB algorithm; features extracted from the scale-invariant-feature-transform or SIFT algorithm; features extracted from the Speeded-Up-Robust-Features or SURF algorithm, etc.

[0260] Some or all of the feature points in the set of SDR image feature points and the set of HDR image feature points may each be represented by a feature vector or a descriptor, such as an array of feature values ​​(e.g., numerical values).

[0261] Block 366 includes matching some or all of the feature points in the set of SDR image feature points with some or all of the feature points in the set of HDR image feature points. For each feature point of a particular type in the SDR image in the set of SDR image feature points, a matching metric can be calculated between that feature point in the SDR image and each feature point of the same type in the HDR image in the set of HDR image feature points. In a non-limiting example, the matching metric can be calculated as the sum of absolute differences, or SAD, between a first feature vector representing the feature point in the SDR image and a second feature vector representing the feature point in the HDR image - or another metric function used to measure the difference between two feature points. A particular feature point having the lowest SAD or matching metric can be selected from among the feature points of the same type in the HDR image as a possible match for that feature point in the SDR image. In response to determining that the lowest SAD or matching metric is lower than the matching difference threshold, the particular feature point in the HDR image may be identified or determined as a match for the feature point in the SDR image, and the feature point in the SDR image and the particular feature point in the HDR image form a matched SDR and HDR feature point pair. Otherwise, the particular feature point may not be identified as such a match. This matching operation may be performed iteratively for all feature points in the set of SDR image feature points, thereby resulting in a set of matched SDR and HDR feature point pairs.

[0262] Block 368 includes computing a geometric transformation, such as a 3×3 2D affine transformation, between the SDR image and the HDR image based in part or in whole on the set of matched SDR and HDR feature point pairs. The geometric transformation can be computed or derived using coordinates (e.g., pixel rows and pixel columns) of matched feature points in the SDR and HDR images from the set of matched SDR and HDR feature point pairs.

[0263] For each matched pair of SDR and HDR features (e.g., k-th pair), the 2D coordinates of the HDR feature in that pair are (τ ix , τ iy ) and the first vector is denoted as τ i = [τ ix τ iy 1], while the 2D coordinates of the SDR feature points can be included as (η ix ,η iy ) and the second vector η i = [η ix η iy 1] may be included.

[0264] Denote the total number of matched SDR and HDR feature point pairs in the set of matched SDR and HDR feature point pairs as N. The vectors generated from the 2D coordinates of the feature points in all pairs of matched SDR and HDR feature points in the set of matched SDR and HDR feature point pairs can be put together as follows:

number

[0265] As mentioned above, the geometric transformation may be expressed as a 3×3 matrix, as follows:

number

[0266] Mathematically, a first vector and a second vector representing the SDR feature points and the HDR feature points, respectively, in each pair of matched SDR feature points and HDR feature points in the set of matched SDR feature point and HDR feature point pairs can be related to each other through a 3×3 matrix representing the geometric transformation as follows:

number

[0267] Alternatively, for every N pairs of matched feature points in the set of matched SDR and HDR feature point pairs, the vectors representing the SDR and HDR feature points can be related through a 3×3 matrix representing the geometric transformation as follows:

number

[0268] The values ​​of the matrix elements in the 3×3 matrix representing the geometric transformation may be generated or obtained as a solution (e.g., a least-squares solution, etc.) of an optimization problem that minimizes a transformation or alignment error. More specifically, the optimization problem may be formulated as follows:

number

[0269] The optimized or optimum values ​​for the 3×3 matrix representing the geometric transformation can be obtained via a least squares solution to the optimization problem in equation () above as follows:

number

[0270] Block 370 includes applying the geometric transformation to one of the SDR image and the HDR image. In the present example, the transformation is applied to the HDR image, such that HDR pixel positions in the HDR image are shifted to match SDR pixel positions in the SDR image for each of the three channels, e.g., Y, Cb, Cr, in which the HDR image is represented. As a result, most HDR pixels in the HDR image are spatially aligned with corresponding SDR pixels in the SDR image. The remaining HDR pixels and the remaining SDR pixels with which no HDR pixels are spatially aligned may be excluded (e.g., assigned out-of-range pixel values) from being used as (valid) color patches or colors for the purpose of generating static SDR-HDR mapping or TPB coefficients or static backward 3D-LUT.

[0271] Block 372 includes finding corresponding valid color patches or colors in the SDR and HDR images. The valid color patches or colors can be obtained from the codeword values ​​of the spatially aligned pixels in the SDR and HDR images. As described above, the spatially aligned pixels can be generated by applying a geometric transformation generated from the matched SDR and HDR feature points.

[0272] In some operational scenarios, after the (initial) matched SDR and HDR feature points having a matching metric lower than a minimum matching difference threshold are identified, a further matching threshold may be applied to select or distinguish a subset of final matched SDR and HDR feature points from or among the (initial) matched SDR and HDR feature points in order to increase the spatial alignment accuracy or reliability of the geometric transformation.

[0273] For each spatially aligned pixel in the SDR image, an SDR code word, such as an SDR Y / Cb / Cr value, may be determined for the spatially aligned pixel in the SDR image. For co-located or spatially aligned pixels (corresponding to the spatially aligned pixels in the SDR image) in the spatially transformed HDR image generated by the geometric transformation, a corresponding HDR code word, such as a corresponding HDR Y / Cb / Cr value, may be determined for the spatially aligned pixel in the HDR image. If a transformed HDR pixel is not available (e.g., has a value of 0), the HDR pixel may be discarded or may be prevented from being considered as part of the matching color patch or code word.

[0274] For each pair (e.g., the i-th, etc.) of matched SDR and HDR color patches or colors generated from a pair of spatially aligned SDR and HDR pixels in the SDR and HDR images, the matched SDR and HDR color patches or colors (or codeword values), respectively, may be given as follows:

number

[0275] The matched SDR and HDR color patches or colors (or codeword values) in every pair of matched SDR and HDR color patches, as generated from every pair of spatially aligned SDR and HDR pixels in the SDR and HDR images of every image pair of SDR and HDR image pairs, can be collected together from all images and merged into two matrices as follows:

number

[0276] Block 374 calculates the matrix S in equations (88) and (89) above. YCbCr (R709)and R with a tilde YCbCr (R2020) ] as input to solve or generate optimized or optimal TPB coefficients for backward reshaping or static SDR-HDR mapping.

[0277] The backward generator matrix for each channel is the backward TPP basis function and the matrix [S YCbCr (R709) ] may be generated as follows:

number

[0278] The per-channel observation matrix for SDR-HDR mapping is the matrix (R YCbCr (R2020) ] may be constructed as follows:

number

[0279] The optimized, or optimal, backward TPB coefficients for a channel ch can be solved by a least squares solution as follows:

number

[0280] The predicted, or backward reshaped, HDR value for channel ch can be calculated as follows:

number

[0281] Optimizing Black Box Backward Reshaping on Two Devices (BB2) In a "BB2" operating scenario, (backward) reshape mapping may be used to map an SDR image captured by a first video capture device in an SDR capture mode (ISP captured) to generate an (ISP mapped) HDR image that simulates the HDR appearance of an HDR image captured by a second, different video capture device in an HDR capture mode (ISP captured). In some operating scenarios, the first video capture device may be a relatively low-end mobile phone capable of capturing only SDR images or pictures, and the second video capture device may be a relatively high-end phone capable of capturing HDR images or pictures. Although the first and second video capture devices may operate with different hardware configurations and capabilities, the reshape mapping is used to reshape the SDR image captured by the first device (ISP captured) into an (ISP mapped) HDR image that approximates the HDR image captured by the second device (ISP captured).

[0282] The reshaping (SDR-HDR) mapping may be modeled based at least in part on a plurality of image pairs formed by a (training) SDR image captured by a first device and a (training) HDR image captured by a second device, where each image pair in the HDR and SDR image image pairs includes an SDR image and an HDR image corresponding to the SDR image.

[0283] In some "BB2" operating scenarios, some or all of the image pairs of the plurality of image pairs may include captured SDR and HDR images depicting visual scenes, such as real-world natural indoor / outdoor scenes, that are subject to spatial and / or temporal alignment errors associated with the first and second video capture devices (e.g., initially, prior to the image alignment operation). Similar to the "BB1" use cases or operating scenarios, in these "BB2" use cases or operating scenarios, the image alignment operation between corresponding SDR and HDR images in the image pairs may be performed using, for example, a geometric transformation constructed using selected feature points extracted from the SDR and HDR images. The SDR and HDR color patches or colors determined from the aligned SDR and HDR images in the image pair(s) may then be used to generate a reshaped (SDR-HDR) mapping as described herein.

[0284] In some "BB2" operating scenarios, instead of or in addition to acquiring images from natural scenes, some or all of the image pairs of the plurality of image pairs may include captured SDR and HDR images (e.g., initially, before the image alignment operation, etc.) depicting color charts displayed on one or more reference image displays of the same type, e.g., in a laboratory environment. For example, the color charts can be generated as 16-bit full HD RGB (color chart) TIFF images. These TIFF images including the color charts can be displayed as perceptually quantized (PQ) video signals on a reference image display, such as a PRM TV. The color charts displayed on the reference image displays can be captured by first and second video capture devices, respectively.

[0285] FIG. 2N illustrates an exemplary TIFF image including a color chart. As illustrated, the color chart may be a central square in the TIFF image, which includes multiple color blocks having a set of distinct colors to be captured and matched between a first device and a second device. Each color in the set of distinct colors may be different from all other colors in the set of distinct colors. The set of distinct colors may be displayed by a reference image display(s) having a set of distinct pixel / codeword values. Thus, each color in the set of distinct colors may correspond to a respective pixel value or codeword value in the set of distinct pixel / codeword values.

[0286] The different TIFF images may include different color charts having different colors or different sets of distinct colors, and each color chart in each of the different TIFF images may correspond to a respective set of distinct colors of the different colors or sets of distinct colors.

[0287] Each of the four corner rectangles in the TIFF image of FIG. 2N includes a checkerboard pattern. The same corner rectangle or the same checkerboard pattern may be included in different TIFF images that include different color charts. The checkerboard patterns at the four corners of the TIFF image may be used as spatial keys or reference marks for the estimation of the projective transformation, as described in more detail below. Additionally, optionally, or alternatively, the TIFF image may include one or more numbers, binary coding, QR codes, etc., as a unique identifier (ID) assigned or used to identify the TIFF image or a color chart therein, or a set of colors or colors therein, etc.

[0288] In order to find as many color correspondences / mapping relationships as possible between the first and second video capture devices, the color chart in the TIFF image described herein may include as many different colors (different sets of colors or different colors) as possible corresponding to as many different pixel / codeword values ​​as possible, subject to the display capability of the reference image display to distinguish different colors. Furthermore, the color chart may be displayed with different overall intensities or illuminations to allow the first and second video capture devices to have different exposure settings under different lighting conditions. Thus, the same color displayed in the color chart in the TIFF image can be displayed with different intensities or illuminations on the reference image display(s).

[0289] FIG. 3E illustrates an exemplary process flow for generating multiple different color charts to be included in multiple different TIFF images. To generate each color chart, the mean and variance of the colors (e.g., RGB) in the color chart may be determined and used to randomly generate colors using a statistical distribution (of some kind). In some operating scenarios, the statistical distribution represents a beta distribution or distribution type. Different combinations of mean and variance values ​​of the statistical distribution or distribution type can be used to generate different color charts with different multiple distinct colors or different sets of distinct colors.

[0290] Block 382 determines a set of possible average values ​​in the PQ domain or color space (M PQ Define or determine the

[0291] In an operating scenario where the reference image display used to display the TIFF image containing the color chart is a PRM TV, the maximum and minimum luminance values ​​supported by the reference image display may range between 1000 nits and 0 nits, or even a larger dynamic range and contrast ratio. By way of example and not limitation, the minimum and maximum PQ values ​​that can be displayed by the reference image display without clipping are P 0 = L2PQ(0) and P 1 = L2PQ(1000), where L2PQ(·) denotes a mapping function from (linear or non-PQ) luminance to (non-linear or PQ) luma codewords in the PQ domain or color space.

[0292] In a non-limiting example, set M PQ is L on a logarithmic scale. lower From L upper The lower bound L can be defined as the set of distinct PQ luma codewords corresponding to (linear) luminance values ​​that are evenly distributed within the (linear) luminance value range of lower is 10 -4 may be set to a value such as L upper is 10 2.99 may be set to a value such as |M| (e.g., a fractional exponent value selected for numerical stability), which depends in part or in whole on the image capture capabilities of the first and second devices (e.g., ability to capture the dimmest and brightest luminances). A logarithmic scale is used because the luma codewords in captured SDR and HDR images may be approximately linear with respect to the logarithm of luminance within the luminance value range. The total number of possible averages in the set of possible averages - the magnitude |M PQ |――is,|M PQ It may be set as |=2048, but is not necessarily limited to this.

[0293] Block 384 involves defining or determining a set of possible shape coefficients to be used to generate possible variance values ​​for a statistical distribution or distribution type (eg, a beta distribution, etc.).

[0294] In some operating scenarios, the beta distribution used to randomly select colors or codeword values ​​is defined, or has support, on a scaled, or closed, value interval [0,1]. 0 ,P 1 Given a PQ value of , the scaled mean μ (where 0<μ<1, or within the closed value interval of the beta distribution) may be calculated as follows:

number

[0295] The scaled variance (σ) for the beta distribution in the scaled, or closed, interval [0,1] 2 ) may be set to be proportional to μ(1−μ) as follows to avoid over / underexposure caused by large variations in color blocks (e.g., luminance, etc.) during capture of SDR and HDR images by the first and second devices: σ 2 =μ(1-μ) / θ (95) Here, θ represents a shape factor that influences or determines the shape of the beta distribution (e.g., more compressed, more expanded, etc.). A value for the shape factor can be selected from a set of possible shape factors Θ to allow the generated color chart to have a relatively high diversity in the codeword values ​​and / or colors resulting therefrom. A non-limiting example for the set of possible shape factors may be Θ={3, 6, 9, 12}.

[0296] Block 386 involves generating a plurality of all unique combinations of scaled means and shape coefficients using a set of possible means and a set of shape coefficients for different instances of a statistical distribution (e.g., a beta distribution, etc.). In the present example, the total number of unique combinations is |M PQ The number of elements in the set Θ or the size of the set Θ is given as |M × |Θ| = 2048 × 4 = 8192, where |Θ| represents the total number of elements in the set Θ or the size of the set Θ. In some operating scenarios, a different TIFF image containing a different color chart is generated for each unique combination of the scaled mean and shape factor among all the multiple unique combinations of the scaled mean and shape factor, thereby denoting the total number of different color charts, or corresponding different TIFF images, as |M PQ Give it as |×|Θ|.

[0297] Block 388 includes selecting the current combination of scaled mean and shape factor values ​​as the next combination from among all the multiple unique combinations of scaled mean and shape factor values, wherein a set or a plurality of different pixel or code word values ​​that respectively specify or define a set or a plurality of different colors for the current color chart corresponding to the current combination of scaled mean and shape factor values ​​is determined based on the scaled mean μ in the current combination and the current scaled variance σ in the current combination. 2 can be generated from the beta distribution, given

[0298] Block 390 involves calculating or defining the beta distribution as follows: f(x;α,β)=x α-1 (1-x) β-1 / B(α,β) x∈[0,1] (96) where α and β are the beta distribution parameters; B(α,β) represents a normalization constant. The beta distribution parameters α and β can be derived from the mean and variance of the beta distribution as follows: α=μν, β=(1-μ)ν where ν=μ(1-μ) / σ 2 -1 (97)

[0299] Block 392 involves generating a set of different (e.g., 144, etc.) colors (or pixel / codeword values) for the current color chart or the current color chart image based on a set of (e.g., 144, etc.) random numbers generated from a beta distribution. Each random number, denoted x (where 0≦x≦1), is a number in the PQ value range [P 0 ,P 1 ] can be scaled back or converted to the corresponding per-channel pixel values ​​or codeword values ​​(called "PQ values") in x PQ =x(P 1 -P 0 )+P 0 (98)

[0300] A set of PQ values ​​generated from scaling or converting the set of random numbers back to the PQ value range can then be generated from the set of random numbers and used as the code word values ​​or code words per channel for the set of colors in the current color chart. In some operating scenarios, for each color chart, the same beta distribution (with the same mean and shape factor) is used with, for example, three different sets of random numbers to generate code words per channel for each of the multiple (e.g., RGB) channels of the color space in which the display image is rendered by the reference image display. Thus, the average color generated from averaging the colors represented in all color blocks in the color chart will be close to a neutral or gray color.

[0301] Block 394 includes generating a central block of the current color chart or corresponding TIFF image. The current color chart may include a set of color blocks (e.g., 2D squares, etc.), each of which is a single color given by a particular (cross-channel or composite) pixel or codeword value with three per-channel pixel or codeword values. Three sets of per-channel codewords generated from the same beta distribution, but with three different sets of random numbers, may be used by the reference image representation to drive the rendering of the red, blue, and green channels of the color blocks in the current color chart or the set of color blocks in the current TIFF image, respectively.

[0302] Additionally, optionally, or alternatively, RGB values ​​used to specify or generate a background color for the background of the color chart image may be calculated based on said average PQ value μ from which the scaled mean of the beta distribution is derived. PQ may be set to

[0303] Block 396 includes determining whether the current combination of scaled averages and shape factor values ​​is the last combination in the plurality of all unique combinations of scaled averages and shape factors. If so, the process flow for generating a plurality of distinct color charts or distinct color chart images ends. If not, the process flow returns to block 388.

[0304] By way of example and not limitation, n c For example, 144 different colors or color blocks may be generated for each color chart. The total number of colors or color blocks generated from the multiple different color charts and color chart images is |M PQ |×|Θ|×n c= 2048 × 4 × 144 = 1179648 colors or color blocks, which may be used to determine a correspondence / match relationship between SDR color patches or codewords captured in an SDR capture mode with a first video capture device and corresponding HDR color patches or codewords captured in an HDR capture mode with a second video capture device.

[0305] In some operating scenarios, multiple TIFF images each including multiple color charts can be displayed or rendered iteratively on a reference image display such as a PRM TV at a constant playback frame rate, and can be captured in SDR and HDR images by a first and a second video capture device, respectively. Because the illumination of the reference image display can vary with viewing angle, the SDR and HDR images can be captured separately by a first video capture device and a second video capture device (e.g., two phones, etc.) that are placed at the same position and orientation with respect to or with respect to the reference image display. Thus, SDR and HDR video signals or bitstreams including the captured SDR and HDR images of the color charts, or the TIFF images rendered on the reference image display, can be generated by the first and second devices or their cameras, respectively. Since the playback frame rate of the reference image display can be kept constant, the frame numbers of the color chart can be relatively easily determined, and an image pair can be established in which an SDR image captured by a first video capture device and an HDR image captured by a second video capture device represent the same color chart among the multiple color charts. The captured SDR and HDR images may be represented in SDR and HDR YCbCr (or YUV) color spaces, for example in a YUV image / video file.

[0306] 3F shows an exemplary process flow for matching SDR and HDR colors between captured SDR and HDR image pairs for each color chart in a plurality of color charts. To extract the colors of each of the individual color blocks from the captured SDR and HDR images of a color chart image including a color chart, the captured SDR and HDR images can be transformed into the same layout as the (original) color chart image displayed on the reference image display.

[0307] Block 3002 includes receiving a captured (SDR or HDR) checkerboard image, which may be taken by a camera of one of the first and second video capture devices from a displayed checkerboard image displayed on a reference image display. Blocks 3002-3006 of the same process flow of FIG. 3F may be performed with respect to a captured checkerboard image taken by the other camera of the first and second video capture devices from the same displayed checkerboard image displayed on the reference image display.

[0308] Block 3004 involves detecting checkerboard corners from a checkerboard image captured by a camera of the device.

[0309] Block 3006 includes calculating or calibrating camera distortion coefficients for the camera using the detected checkerboard corners from the captured checkerboard image. The 3D (reference) coordinates of the checkerboard image displayed on the reference image display may first be determined in a stationary 3D coordinate system relative to the reference image display. Camera parameters such as distortion coefficients and other intrinsic parameters of the camera used by the device to generate the captured checkerboard image may be calculated as optimized values ​​that produce the best mapping (e.g., minimal error or minimal discrepancy) between the 3D reference coordinates of the checkerboard corners and the 2D (image) coordinates of the checkerboard corners in the captured checkerboard image. This calibration process may be performed in either the YUV or RGB color space to which the displayed or captured checkerboard image may be transformed or represented in.

[0310] 2M and 2N show two exemplary checkerboard images detected from captured (HDR and SDR, respectively) images of the checkerboard image. In order to achieve a relatively high degree of accuracy (e.g., within half a pixel for camera reprojection error, etc.) when projecting the captured images by each camera of the multiple (e.g., 100, etc.) captured checkerboard images onto the display image on the reference image display, the distortion coefficients or other intrinsic parameters of each camera used in the first and second video capture devices can be determined, calculated, or calibrated.

[0311] The (camera-specific) distortion coefficients and intrinsic parameters of the cameras of the first and second video capture devices obtained in processing blocks 3002-3006 can be used for analysis or correlation between captured SDR and HDR images from TIFF images containing respective color charts as follows:

[0312] Block 3008 includes receiving a captured (SDR or HDR) image of the TIFF image to be displayed on the reference image display, including a color chart and checkerboard corners (or a checkerboard pattern at the corners). The captured (SDR or HDR) image may be captured by a camera for which distortion coefficients have been previously obtained in block 3006 using a previously captured checkerboard image (which may not include a color chart).

[0313] Block 3010 involves correcting or de-distorting the captured image of the displayed TIFF image using distortion coefficients obtained in a camera calibration process, such as block 3006, to compensate for camera lens distortion of the camera.

[0314] Block 3012 includes detecting checkerboard corners in the captured image, where the checkerboard corners are captured from checkerboard corners (e.g., four corners, etc.) in the displayed TIFF image on a reference image display, which may be a PPM TV.

[0315] Block 3014 involves estimating a projective transformation between the image coordinates of the captured image and the image coordinates of the original TIFF image captured within the received captured image and displayed on the reference image display.

[0316] Block 3016 includes rectifying the captured image to the same layout as the original TIFF image using the estimated projective transformation. Using the estimated projective transformation, the captured image can be rectified to the same spatial layout as the original TIFF image.

[0317] FIG. 2O shows (a) a captured image from the (displayed) TIFF image, (b) a modified captured image generated by modifying the captured image with camera distortion correction and projective transformation, and (c) the original TIFF color chart image displayed on a reference image display. Because the modified captured image in FIG. 2O(b) is generated from a distortion removal or correction operation performed on the captured image in FIG. 2O(a), some pixels in the modified image in FIG. 2O(b) may be undefined.

[0318] Block 3018 includes locating and extracting (a set of) individual color blocks from the color chart in the modified captured image. As used herein, a color block is designated by a single corresponding (e.g., RGB, YCbCr, composite, etc.) pixel or codeword value. Corresponding individual codewords designating each individual color block in the captured image can be determined based on pixel values ​​or codeword values ​​of pixels in these individual color blocks in the modified captured image. Furthermore, individual original codewords designating each individual original color block in the original TIFF image (corresponding to the individual color blocks of the modified captured image) can be determined from an RGB or YUV file for the original TIFF image.

[0319] The correspondence / mapping relationship between the SDR color blocks or codewords extracted from the (modified) captured SDR image of the original color chart image and the HDR color blocks or codewords extracted from the (modified) captured HDR image of the same original color chart image can be established based in part or in whole on the individual original color blocks or codewords in the original TIFF image from which both the (modified) SDR and HDR images are derived.

[0320] Multiple correspondences or mapping relationships can be established between a set of SDR colors and a set of HDR images using a plurality of TIFF images, a plurality of SDR images captured in an SDR capture mode by a first video capture device from the displayed TIFF images, and a plurality of HDR images captured in an HDR capture mode by a second video capture device from the same displayed TIFF images.

[0321] To map an SDR image to an HDR image, several approaches can be used.

[0322] In the first approach, like in some "BB1" operating scenarios, in some "BB2" operating scenarios, the mapped SDR and HDR color patches or colors can be used to generate (e.g., static) SDR to HDR reshaping operating parameters such as TPB coefficients. These TPB coefficients can be combined with TPB basis functions using SDR codewords of the SDR image as input parameters to predict corresponding HDR codewords of the reconstructed HDR image. A static 3D-LUT may be pre-constructed and deployed on a video capture device (e.g., the first video capture device) to map the SDR image captured by the first video capture device to a reconstructed HDR image that simulates the HDR appearance of the second video capture device. Because the HDR color blocks or code words used to generate the reshaping operation parameters are extracted from a captured (training) HDR image of the second video capture device, the predicted HDR code words generated using the reshaping operation parameters are likely to provide a mapped HDR appearance in the reconstructed HDR image that is similar to the actual HDR appearance of the actual HDR image captured by the second video capture device.

[0323] In the second approach, in some "BB2" operating scenarios, non-TPB optimization may be used to generate non-TPB reshaping operation parameters in a non-TPB reshaping operation for mapping a captured SDR image taken by a first video capture device to a reconstructed HDR image that simulates the HDR appearance of a second video capture device. The 3D-LUT for the non-TPB reshaping operation may be directly constructed without the relatively high continuity and smoothness supported by the TPB reshaping operation. This non-TPB second approach offers a relatively high degree of design freedom and may be relatively flexible even if different colors represented in nearby 3D-LUT entries / nodes may or may not have relatively high continuity and smoothness. In some operating scenarios, with the relatively high degree of design freedom, the non-TPB second approach may be more suitable than the first approach for two-device or "BB2" operating scenarios where an SDR image from one device is mapped to an HDR image that simulates the HDR appearance of another device.

[0324] A second non-TPB approach may utilize a 3D mapping table (3DMT) to construct a backward reshaping mapping, or a backward lookup table (BLUT) that represents the backward reshaping mapping. The BLUT may be used to map SDR (e.g., cross-channel, 3-channel, etc.) codewords of an SDR image to HDR (e.g., chroma channel, per chroma channel, etc.) codewords of a reconstructed HDR image. Exemplary operations for constructing a BLUT-like backward reshaping mapping from a 3DMT can be found in U.S. Patent Application No. 17 / 054,495, filed May 9, 2019, the contents of which are hereby incorporated by reference in their entirety as if fully set forth herein.

[0325] For purposes of illustration only, an SDR codeword may be represented in three dimensions or channels Y, Cb, and Cr in the SDR YCbCr color space. The 3D mapping table may be generated from a 3D histogram having multiple 3D histogram bins. The multiple 3D histogram bins may be a set of three positive integers, Q y ,Q Cb ,Q Cr , which may correspond to multiple color space partitions generated by dividing each dimension or channel by a respective positive integer among y ×Q Cb ×Q Cr ) 3D histogram bins are generated.

[0326] Multiple 3D histogram bins (Ω Q,s Each 3D histogram bin (denoted as q) in the SDR (YCbCr) color space has a respective bin index or three respective quantized channel values ​​q=(q y ,q Cb ,q Cr ), which stores the pixel counts for all SDR color patches or codewords in the SDR color space partition represented by the 3D histogram bin. All bin indices (or quantized channel values) for multiple 3D histogram bins can be collected into a set of bin index values, denoted Q, where Q = [Q y ,Q C0 ,Q Cr ].

[0327] Additionally, a sum of the HDR codewords mapped to the SDR codewords in each 3D histogram in the multiple 3D histograms may be calculated and stored for the 3D histogram bin. y Q,v , Ψ Cb Q,v and Ψ Cr Q,vLet denote the sum of HDR codewords (also called "mapped HDR luma and chroma values").

[0328] An example procedure for generating SDR pixel counts and mapped HDR luma and chroma values ​​for the 3D histogram bins of the histogram is shown in Table 2 below. [Table 2]

[0329] (s q y,(B) , s q Cb,(B) , s q Cr,(B) Let q denote the (representative) SDR codeword at the center of the q-th 3D histogram bin. The representative SDR codewords for all 3D histogram bins can be fixed for all SDR images / frames and can be precomputed with the exemplary procedure shown in Table 3 below. [Table 3]

[0330] Next, 3D histogram bins having a non-zero (SDR) pixel count may be identified and retained from the plurality of 3D histogram bins, and all other 3D histogram bins having a zero (SDR) pixel count may be discarded or removed from the plurality of 3D histogram bins.

[0331] q 0 , q 1 , …q k-1 Ω, which has a non-zero (SDR) number of pixels, or q Q,s Let denote all k 3D histogram bins for which ≠ 0, where k is a positive integer. The average of the mapped HDR luma and chroma values.

number

[0332] A 3D-LUT can be constructed from a 3DMT represented by a 3D histogram that stores the number of SDR pixels in each 3D histogram bin and their respective mapped HDR codeword values. Each node / entry in the 3D-LUT or 3DMT stores the specific SDR codeword (q) represented in each 3D histogram bin of k 3D histogram bins. y ,q Cb ,q Cr ) and the mapping key calculated for the 3D histogram bins in Table 4 above.

number

[0333] As mentioned above, exemplary operations for constructing a 3D-LUT from a 3DMT to function as a non-TPB backward reshaping mapping or BLUT (e.g., operations for predicting HDR chroma codewords per chroma channel from cross-channel SDR codewords) are described in the aforementioned U.S. patent application Ser. No. 17 / 054,495.

[0334] In some "BB2" operating scenarios, luma backward reshaping mapping may be used to backward reshape the SDR luma codeword of an SDR image to a predicted or backward reshaped HDR luma codeword of a corresponding HDR image using a GPR-based model. Exemplary generation of luma reshaping mapping using a GPR-based model may be found in U.S. Provisional Patent Application No. 62 / 887,123, filed August 15, 2019, by Guan-Ming Su and Harshad, entitled "Efficient User-Defined SDR-HDR Conversion Using Model Templates," and PCT Application No. PCT / US2020 / 046032, filed August 12, 2020, the contents of which are incorporated herein by reference in their entirety as if fully set forth herein. For example, a CDF matching curve may be generated from each training SDR-HDR image pair in a plurality of training SDR-HDR image pairs. A set of SDR points uniformly sampled (e.g., 15, etc.) across the SDR codeword range (e.g., the entire SDR codeword range) can be used to find a corresponding mapped HDR codeword in each of the CDF matching curves. For each sample SDR point in the set of uniformly sampled SDR points, a GPR model can be constructed based on a histogram having multiple (e.g., 128, etc.) luma bin histograms and used to generate a corresponding luma reshaping mapping.

[0335] Edit-time TPB optimization Image / video editing is a common application in video capture devices, such as mobile devices, to allow a user to adjust color, contrast, brightness, or other user preferences. Image / video editing may be done or performed at the encoder side before the captured images are compressed into a (compressed) video signal or bitstream. Additionally, optionally, or alternatively, image / video editing may be done or performed at the decoder side after the video signal bitstream is decoded or decompressed.

[0336] The image / video editing operations described herein can be performed in either or both the HDR domain and the SDR domain. Image / video editing operations in the HDR domain can be performed relatively easily. For example, after an HDR content or image is edited, the edited HDR content or image can be passed as an input or reference HDR image to a video pipeline that generates a corresponding reshaped or ISP SDR content or image to be encoded in a video signal or bitstream. In comparison, image / video editing operations in the SDR domain can be relatively difficult. Although HDR-SDR and / or SDR-HDR mappings can be designed or generated to ensure or enhance reversibility between the SDR and HDR domains, image / video editing operations performed on an SDR image to be encoded into a video signal or bitstream are likely to cause difficulties or challenges in inverting or backward reshaping the edited SDR image to generate a reconstructed HDR image that approximates the reference HDR image.

[0337] FIG. 3G illustrates exemplary image / video editing operations at the encoder side performed at an upstream device such as a video capture device or video encoder. As illustrated, an input HDR image may be represented in an input HDR domain or color space (denoted as “HLG YCbCr”). A TPB-based forward reshaping operation (denoted as “Forward TPB”) may be performed to forward reshape the input HDR image into a forward reshaped SDR image represented in a forward reshaped SDR domain or color space (denoted as “SDR YCbCr”). An image / video editing operation (denoted as “RGB domain editing”) may be performed to edit the forward reshaped SDR image to generate an edited SDR image represented in an edited forward reshaped SDR domain or color space (denoted as “Edited SDR YCbCr”). The edited SDR image represented in the edited forward reshaped SDR domain or color space may be compressed or encoded into a video signal or bitstream using one or more video codecs of the upstream device.

[0338] FIG. 3H illustrates an exemplary image / video editing operation at the decoder side performed at a downstream receiving device or video decoder. As illustrated, a video signal or bitstream is decoded by one or more video codecs of the downstream device into an SDR image represented in an SDR domain or color space (denoted as "SDR YCbCr"). An image / video editing operation (denoted as "RGB domain editing") may be performed to edit the SDR image to generate an edited SDR image represented in an edited forward reshaped SDR domain or color space (denoted as "edited SDR YCbCr"). A TPB-based backward reshaping operation (denoted as "backward TPB") may be performed on the edited SDR image represented in the edited forward reshaped SDR domain or color space to generate a backward reshaped or reconstructed HDR image in a reconstructed HDR domain or color space (denoted as "PQ YCbCr").

[0339] As shown in Figures 3G and 3H, an SDR image encoded into a video signal at the encoder side or decoded from a video signal may be represented in an SDR domain or color space (e.g., forward reshaped, etc.), such as an SDR YCbCr domain or color space, using which a video codec can be programmed or developed to perform image processing operations relatively efficiently. By way of example and not limitation, image / video editing operations may be performed in an SDR RGB domain or color space, as shown in Figures 3G and 3H. RGB-YCbCr and YCbCr-RGB conversions may be performed through conversions (e.g., standard-defined, etc.), such as SMPTE range conversions.

[0340] As more colors are pushed into the SDR domain or color space, codewords in the forward reshaped domain or color space, such as the SDR YCbCr color space, often exceed a codeword value range, such as the SMPTE range, in the SDR RGB color space in which image / video editing operations are performed. When applying a YCbCr to RGB conversion (which converts YCbCr codewords in the YCbCr codeword value range (normalized to a value range of [0,1]) of the SDR YCbCr color space to RGB codewords in the SDR RGB color space), some of the RGB codewords may exceed or be outside the SMPTE range (normalized to a value range of [0,1]) of the SDR RGB color space. These out-of-range codeword values ​​may be clipped, but the clipped codeword values ​​may not be able to be restored to the original unclipped HDR codewords in a backward reshaping operation (e.g., TPB-based, etc.). Additional operations described herein may be used to improve reversibility and reduce or avoid visual artifacts in image / video editing applications.

[0341] 3I to 3M show some example solutions for clipping out-of-range codewords in image / video editing applications.

[0342] In some operating scenarios, as shown in Figure 3I, out-of-range SDR RGB codewords resulting from a YCbCr to RGB conversion (denoted as "YCbCr to RGB clipping conversion") performed on an SDR YCbCr image at the encoder or decoder side can be clipped to a valid or specified codeword value range such as [0,1]--e.g., all values ​​greater than 1 are hard clipped to 1 and all values ​​less than zero are hard clipped to 0. Image / video editing operations ("editing in RGB domain") can then be performed using SDR RGB codewords within the valid or specified codeword value range in the SDR RGB color space. The edited SDR RGB codewords in the SDR RGB color space can be transformed and clipped to transformed edited SDR YCbCr codewords within a valid or specified codeword value range, such as [0,1], in the transformed edited SDR YCbCr color space (denoted "edited SDR YCbCr") by an RGB-to-YCbCr transform (denoted "RGB-to-YCbCr clipping transform").

[0343] In some operation scenarios, as shown in FIG. 3J, out-of-range SDR RGB codewords resulting from a YCbCr-RGB conversion (denoted as “YCbCr-RGB unclipped conversion”) performed on an SDR YCbCr image at the encoder or decoder side are not clipped to a valid or specified codeword value range, such as [0,1]. An image / video editing operation (“editing in RGB domain”) can be performed with an unclipped (input) SDR RGB codeword within a valid or specified codeword value range in the SDR RGB color space to generate an edited SDR codeword. The YCbCr-RGB conversion with unclipping operation allows the original information in the (input) SDR YCbCr codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation takes all possible real codeword values ​​(e.g., including values ​​less than 0 or greater than 1 within a limited but wider value range than

[0001] ) in the input SDR RGB codeword. As a result, some information in the input SDR YCbCr codeword can be preserved in the input SDR RGB codeword as well as in the edited SDR RGB codeword. An SDR RGB boundary operation (denoted as "RGB boundary

[0001] clipping"), e.g., optionally based on a user preference of a user running a video editing application, can be performed to scale down or push edited SDR RGB codewords that are outside the valid or specified range of [0,1] into the valid or specified range of [0,1], thereby generating scaled edited SDR RGB codewords within the valid or specified range of [0,1].The scaled edited SDR RGB codewords in the SDR RGB color space can be converted and clipped to transformed scaled edited SDR YCbCr codewords within a valid or specified codeword value range, such as [0,1], in the transformed scaled edited SDR YCbCr color space (denoted "edited SDR YCbCr") by an RGB-to-YCbCr conversion (denoted "RGB-to-YCbCr clipping conversion").

[0344] In some operation scenarios, as shown in FIG. 3K, out-of-range SDR RGB codewords resulting from a YCbCr-RGB conversion (denoted as “YCbCr-RGB non-clipping conversion”) performed on an SDR YCbCr image at the encoder or decoder side are not clipped to a valid or specified codeword value range, such as [0,1]. An image / video editing operation (“editing in RGB domain”) can be performed with an unclipped (input) SDR RGB codeword within a valid or specified codeword value range in the SDR RGB color space to generate an edited SDR codeword. The YCbCr-to-RGB conversion with non-clipping operation allows the original information in the (input) SDR YCbCr codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation takes all possible real codeword values ​​(e.g., including values ​​less than 0 or greater than 1 within a limited but wider value range than

[0001] ) in the input SDR RGB codeword. As a result, some information in the input SDR YCbCr codeword can be preserved in the input SDR RGB codeword as well as in the edited SDR RGB codeword. Unlike that shown in Fig. 3J, in the operation scenario shown in Fig. 3K, the SDR RGB boundary operation may not be performed. The edited SDR RGB codeword in the SDR RGB color space can be converted and clipped to a converted edited SDR YCbCr codeword within a valid or specified codeword value range (such as [0,1]) of the converted edited SDR YCbCr color space (denoted as "edited SDR YCbCr") by an RGB-YCbCr conversion (denoted as "RGB-YCbCr clipping conversion").

[0345] In some operation scenarios, as shown in FIG. 3L, out-of-range SDR RGB codewords resulting from a YCbCr-RGB conversion (denoted as “YCbCr-RGB non-clipping conversion”) performed on an SDR YCbCr image at the encoder or decoder side are not clipped to a valid or specified codeword value range, such as [0,1]. An image / video editing operation (“editing in the RGB domain”) can be performed with an unclipped (input) SDR RGB codeword within a valid or specified codeword value range in the SDR RGB color space to generate an edited SDR codeword. The YCbCr-to-RGB conversion with non-clipping operation allows the original information in the (input) SDR YCbCr codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation takes all possible real codeword values ​​(e.g., including values ​​less than 0 or greater than 1 within a limited but wider value range than

[0001] ) in the input SDR RGB codeword. As a result, some information in the input SDR YCbCr codeword can be preserved in the input SDR RGB codeword as well as in the edited SDR RGB codeword. An SDR RGB boundary operation (denoted as "RGB boundary clipping 3D-LUT") can be performed--for example, optionally based on user preferences of a user operating a video editing application--to scale down or push edited SDR RGB codewords outside the (3D) boundary supported by the TPB-based reshaping operation into the boundary, thereby generating scaled edited SDR RGB codewords supported by the TPB-based reshaping operation or within a boundary well defined in the TPB-based reshaping operation. The boundary supported by the TPB-based reshaping operation does not have to be a regular shape, or simply a cube defined by a fixed value range of [0,1].The scaled edited SDR RGB codewords in the SDR RGB color space can be converted and clipped to transformed scaled edited SDR YCbCr codewords within a valid or specified codeword value range, such as [0,1], in the transformed scaled edited SDR YCbCr color space (denoted "edited SDR YCbCr") by an RGB-to-YCbCr conversion (denoted "RGB-to-YCbCr clipping conversion").

[0346] The (input) SDR YCbCr image can be derived from the original HDR image through a forward reshaping or ISP process using a known (e.g., white box, ISP, etc.) HDR-SDR forward reshaping or mapping. The boundaries can be determined partially or fully based on the HDR-SDR forward reshaping or mapping and / or any applicable color space transformation matrix(es) and expressed as a (TPB boundary clipping) 3D-LUT. For example, the boundary pixel or codeword values ​​in the forward reshaping or ISP SDR color space can be determined using a complete grid sampling data covering the entire HDR domain or color space in which the original HDR image is represented.

[0347] After an image / video editing operation ("editing in the RGB domain"), the edited SDR RGB codeword can be scaled, or pushed, or irregularly clipped to a 3D shape depicted or enclosed by a boundary. In some operating scenarios, the scaled edited SDR YCbCr codeword in the SDR RGB color space can be converted to a transformed scaled edited SDR YCbCr codeword by an RGB-to-YCbCr conversion ("RGB-to-YCbCr clipping conversion") without further clipping by an RGB-to-YCbCr conversion ("RGB-to-YCbCr clipping conversion"), since the transformed scaled edited SDR YCbCr codeword is already positioned within a corresponding boundary in the SDR YCbCr color space supported by the TPB-based backward reshaping operation. In these operating scenarios as shown in FIG. 3J, a maximized number of colors can be preserved in the transformed scaled edited SDR YCbCr codeword, as well as in the backward-reshaped HDR codeword generated from the transformed scaled edited SDR YCbCr codeword by the TPB-based backward-reshaping operation, while avoiding the generation of color artifacts.

[0348] In some operation scenarios, as shown in FIG. 3M, out-of-range SDR RGB codewords resulting from a YCbCr-RGB conversion (denoted as “YCbCr-RGB non-clipping conversion”) performed on an SDR YCbCr image at the encoder or decoder side are not clipped to a valid or specified codeword value range, such as [0,1]. An image / video editing operation (“editing in RGB domain”) can be performed with an unclipped (input) SDR RGB codeword within a valid or specified codeword value range in the SDR RGB color space to generate an edited SDR codeword. The YCbCr-to-RGB conversion with non-clipping operation allows the original information in the (input) SDR YCbCr codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation takes all possible real codeword values ​​(e.g., including values ​​less than 0 or greater than 1 within a limited but wider value range than

[0001] ) in the input SDR RGB codeword. As a result, some information in the input SDR YCbCr codeword can be preserved in the input SDR RGB codeword as well as in the edited SDR RGB codeword. The edited SDR RGB codeword in the SDR RGB color space can be transformed and clipped to a transformed edited SDR YCbCr codeword within a valid or specified codeword value range, such as [0,1], of the transformed edited SDR YCbCr color space (denoted as "edited SDR YCbCr") by an RGB-to-YCbCr transformation (denoted as "RGB-to-YCbCr clipping transformation").The transformed edited SDR YCbCr codewords in the transformed edited SDR YCbCr color space ("edited SDR YCbCr") may be clipped by an SDR YCbCr boundary operation (denoted "YCbCr boundary clipping 3D-LUT") - e.g., optionally based on user preferences of a user operating the video editing application - which may be performed to scale down or push the transformed edited SDR YCbCr codewords outside a (3D) boundary supported by the TPB-based reshaping operation to inside the boundary, thereby generating scaled transformed edited SDR YCbCr codewords in a scaled transformed edited SDR YCbCr color space (denoted "edited SDR YCbCr") that are supported by the TPB-based reshaping operation or within a well-defined boundary in the TPB-based reshaping operation. The boundaries supported by a TPB-based reshape operation may not simply be regular shapes such as a cube with sides defined as the valid range or a specified (normalized) range [0,1].

[0349] The (input) SDR YCbCr image can be derived from the original HDR image through a forward reshaping or ISP process using a known (e.g., white-box, ISP, etc.) HDR-SDR forward transform or mapping. The boundaries can be determined and represented using a (TPB boundary clipping) 3D-LUT based on the HDR-SDR forward transform or mapping and / or any applicable color space transformation matrix(es). For example, boundary pixel or codeword values ​​in the forward reshape or ISP SDR color space can be determined using full grid sampling data covering the entire HDR domain or color space in which the original HDR image is represented.

[0350] Constructing and clipping boundary clipping 3D-LUT The forward reshaping or mapping from HDR to SDR, such as TPB-based forward reshaping, may be a non-linear function. While the input HDR codewords in the input HDR domain or color space may be well-ordered within a simple 3D cube, the boundaries of the mapped SDR codewords generated by applying the (non-linear or TPB) HDR-SDR mapping may be relatively irregular, unlike the 3D cube.

[0351] To perform (TPB) boundary clipping for relatively irregular boundaries, a (TPB) boundary clipping 3D-LUT may be constructed. The 3D-LUT may be searched using an SDR codeword as an input (or lookup key) and, in response to determining that the SDR codeword is within the boundary, may return the SDR codeword as a value. Otherwise, in response to determining that the SDR codeword is outside the boundary, the 3D-LUT may return a clipped SDR codeword (that is within the boundary) that is different from the original SDR codeword.

[0352] In some operating scenarios, boundary clipping can be implemented at two levels: at the first level, regular clipping is performed using a range defined by minimum and maximum values ​​(or lower and upper limits) that define the (3D) codeword range; at the second level, irregular clipping is performed using a (TPB) boundary clipping 3D-LUT.

[0353] FIG. 3N illustrates an exemplary process flow for constructing a boundary clipping 3D-LUT (e.g., TPB). The left side of the process flow illustrated in FIG. 3N can be implemented or executed to construct a 3D boundary using the alphaShape technique. The right side of the process flow illustrated in FIG. 3N can be implemented or executed to generate a 3D-LUT for irregular clipping using the constructed 3D boundary.

[0354] Block 3022 determines a 3D uniform sampling grid or set of sampled values.

number

[0355] The sampled values ​​in the R.2020 YCbCr color space HLG may be converted to corresponding values ​​in the R.2020 RGB color space HLG as shown in equation (44) above. The converted values ​​in the R.2020 RGB color space HLG may then be further converted to the optimized value for parameter a (a opt ) may be converted to a corresponding value in a RGB color space HLG corresponding to the (a) RGB color space HLG. The converted value in (a) RGB color space HLG may be clipped as shown in equation (46) above and converted to a corresponding clipped value in R.2020 RGB color space HLG as shown in equation (47) above. The clipped value in the R.2020 RGB color space HLG derived using equation (47) above may be converted to a corresponding clipped value V in the R.2020 YCbCr color space HLG as shown in equation (48) above. YCbCr (FL),(R2020) may be converted to

[0356] Block 3024 outputs the clipped or constrained (HDR YCbC HLG) value V yCbCr (FL),(R2020) (referred to as “Constraint Input” in FIG. 3N ) to apply TPB-based forward reshaping (referred to as “Forward TPB” in FIG. 3N ). More specifically, the optimized forward TPB coefficients for the TPB-based forward reshaping are used, or the forward generator matrix S F ch (in equation (50) above). The generator matrix is ​​multiplied by the clipped HDR value VyCbCr (FL),(R2020) as input parameters to the forward TPB basis functions. The mapped or forward reshaped clipped SDR or R.709 YCbCr values ​​are obtained as follows:

number

[0357] Block 3046 is

number

number

[0358] R.709 RGB values ​​converted from R.709 YCbCr values ​​may contain values ​​outside a valid or specified range, such as the value range of [0,1].

[0359] For each channel, the minimum and maximum values ​​in the R.709 RGB values ​​given in equation (100) can be measured or determined as follows:

number

[0360] These extreme values ​​can be used as lower and upper bounds in block 3030 to construct or prepare a uniformly sampled 3D grid or set of sampling values ​​in the unclipped RGB domain or color space.

[0361] FIG. 2P illustrates an example distribution of R.709 RGB values ​​(in an SDR RGB image) converted from R.709 YCbCr values. As shown in the figure, the distribution of R.709 RGB values ​​in the SDR RGB image is an irregular shape other than a 3D cube. The irregular shape with a 3D boundary represents the space of the largest supported SDR RGB colors that can be mapped back, or back-reshaped, to a reconstructed HDR color in the reconstructed HDR domain or color space without losing information. Any SDR RGB color outside of this irregular shape or range may experience information loss in the back-reshaping operation.

[0362] After an image / video editing operation is performed on an SDR RGB image, the resulting (3D) codeword range or distribution of edited codewords or colors can be wider (e.g., much wider) than the irregular shapes in the SDR RGB color space, resulting in many SDR RGB codewords that are not defined or supported by the backward reshaping operation.

[0363] Furthermore, it may be difficult to characterize, represent, or approximate the actual 3D boundary of an irregular shape using analytical formulas or multiple 2D planes that serve to cut a 3D cube in RGB color space into said irregular shape.

[0364] As mentioned above, a two-level clipping solution can be used to clip the edited codewords back to the maximum supported RGB color space represented by the irregular shapes. More specifically, at the first level:

number

number

[0365] Block 3028 of FIG. 3N is a set of SDR RGB codewords.

number

number

[0366] The alphaShape object created to represent the alphaShape is a set of SDR RGB codewords.

number

number

[0367] A function f to construct the alphaShape αS (Φ,r α ), where r α represents the radius parameter.

[0368] A set of SDR RGB codewords

number

number

[0369] Figures 2Q to 2T show different r α A set of SDR RGB points or codewords in

number

[0370] The alphaShape provides a bounding polyhedron as a clipping boundary for the irregular shape, allowing to determine whether a 3D point represented by an SDR RGB codeword is inside or outside the irregular shape.

[0371] I αS Let (αS,x) denote the binary function used to determine whether a 3D point or SDR RGB codeword (denoted as x) is inside the irregular shape. Let a returned binary value of "1" indicate inside the irregular shape and a returned binary value of "0" indicate outside.

[0372] NN αS Let (αS,x) denote an index function that takes a given query 3D point or SDR RGB codeword, such as x, as a second input parameter and returns the nearest neighbor point in αS for the query 3D point or SDR RGB codeword x.

[0373] As described above, a boundary clipping 3D-LUT can be constructed within the SDR RGB domain or color space to clip any SDR RGB codewords outside an irregular shape representing the maximum supported color space by (TPB-based forward and backward) reshaping operations.

[0374] Block 3030 includes constructing the boundary clipping 3D-LUT as a full grid 3D-LUT with multiple nodes / entries that include the query SDR codeword as a lookup key and the returned SDR codeword for the query SDR codeword as a value.

[0375] The extreme values ​​determined (at block 3026 in FIG. 3N) for each color channel

number

number

number

[0376] Therefore, the total number of query SDR codewords represented in the 3D-LUT is N u =N R N G N B It may be given as:

[0377] Block 3032 includes beginning execution of a node processing loop for each node / entry in the 3D-LUT by selecting a current node / entry from among multiple nodes / entries in the 3D-LUT (e.g., in a sequential or non-sequential loop / sequential iteration order, etc.).

[0378] For simplicity, (i,j,k) in the above equation (103) can be vectorized as p. The current node / entry may be the p-th node / entry among multiple nodes / entries in the 3D-LUT. The lookup key for the p-th node / entry is the u p The query SDR RGB codeword may be represented by the p-th represented query SDR RGB codeword denoted as p Let denote the current output clipped SDR RGB codeword or value returned by the p-th node / entry of the 3D-LUT given the p-th representation of the query SDR RGB codeword as lookup key.

[0379] Block 3034 calculates the current or p-th represented query SDR RGB codeword u for the current node / entry. p is the alphaShape αS generated in block 3028. (R709) (or a shape constructed using the alpha shape construction function as shown in equation (102) above). p In response to determining that is within the alphaShape, process flow proceeds to block 3036. Otherwise, process flow proceeds to block 3040.

[0380] The nodes / entries in the 3D-LUT contain a subset of nodes or entries, each of which has an alphaShape αS (R709) Each lookup key is specified by a query SDR RGB codeword represented in alphaShape αS (R709) Contains the nodes or entries that are considered to be within

[0381] Block 3040 calculates the alphaShape αS for the current node / entry. (R709) The nearest node / entry is found by finding the closest represented query SDR RGB codeword (u with bar). p ) as a lookup key, which finds the closest represented query SDR RGB codeword (u p ] is alphaShape αS (R709) The current or p-th represented query SDR RGB codeword u is compared to all other represented query SDR RGB codewords for all other nodes / entries in the subset of nodes or entries in p has a minimum distance (measured as Euclidean or non-Euclidean distance in SDR RGB color space) to

[0382] In some operating scenarios, the closest represented query SDR RGB codeword (u with a bar) for the current or p-th represented query SDR RGB codeword up p ] is the index function NN αS It may be given by using (αS,x) as follows:

number

[0383] The closest represented query SDR RGB codeword (with a bar) is the current or pth represented query SDR RGB codeword u p This boundary clipping value (with a bar) represents the boundary clipping value for p ] is set to the return value for the current or pth node entry in the 3D-LUT, while the current or pth represented query SDR RGB codeword up is set to the lookup key for the current or pth node entry in the 3D-LUT.

[0384] Block 3036 selects the current or p-th represented query SDR RGB codeword u p as a return value for the current or pth node entry in the 3D-LUT (in addition to being a lookup key), and determining whether the current or pth node / entry is the last node or entry among the multiple nodes / entries in the 3D-LUT. In response to determining that the current or pth node / entry is the last node or entry among the multiple nodes / entries in the 3D-LUT, process flow proceeds to block 3038. If not, process flow returns to block 3032.

[0385] Block 3038 calculates the final boundary clipping 3D-LUT (f 3DLUT αS This involves outputting the 3D-LUT as a 3D-LUT (denoted as ()).

[0386] Final boundary clipping f 3DLUT αS An exemplary procedure for generating () is shown in Table 5 below. [Table 5]

[0387] (Final) Boundary Clipping 3D-LUT f 3DLUT αSGiven (), boundary clipping can be performed relatively efficiently on the (e.g., edited, etc.) SDR image using the two-level solution described above. As described above, regular clipping can be performed to ensure that all (e.g., edited, etc.) SDR RGB codewords in the SDR image are within the extreme values ​​or upper / lower limits for SDR RGB codewords in the SDR RGB color space in which the image / video editing operation on the SDR image was performed. Regular clipping is followed by irregular clipping, which uses the (regularly clipped, if applicable) SDR codewords as lookup keys to find the (final) boundary clipping 3D-LUT f 3DLUT αS (), clip the boundary, and then use the (final) boundary-clipping 3D-LUT f 3DLUT αS By using the return value from () as the output (further irregularly clipped, if applicable) SDR codeword in the output (e.g. clipped, edited, etc.) SDR image.

[0388] In various operating scenarios, including but not limited to those shown in Figures 3I to 3M, regular and / or irregular clipping operations as described herein can be applied in video capture and / or editing applications with the purpose of ensuring maximized support for reconstructing an HDR image from an SDR image and preventing / reducing visual artifacts in the reconstructed HDR image.

[0389] Process Flow Example FIG. 4A illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) may perform this process flow. In block 402, the system described herein constructs sampled high dynamic range (HDR) color space points distributed throughout the HDR color space. The HDR color space is parameterized by primary color scaling parameters having candidate values ​​selected from among a plurality of candidate values. The primary color scaling parameters are used to calculate color space coordinates of at least one of a plurality of primaries that describe the HDR color space.

[0390] In block 404, the system generates from the sampled HDR color space points in the HDR color space: (a) reference standard dynamic range (SDR) color space points represented in the reference SDR color space, (b) input HDR color space points represented in the input HDR color space, and (c) reference HDR color space points represented in the reference HDR color space.

[0391] In block 406, the system executes a reshaping motion optimization algorithm to generate a chain of optimized forward reshaping mappings and optimized backward reshaping mappings. The reshaping motion optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs.

[0392] In an embodiment, an optimized forward reshaping mapping is used to forward reshape an input HDR image in an input HDR color space to a forward reshaped SDR image in a forward reshaped SDR color space, and an optimized backward reshaping mapping is used to backward reshape the forward reshaped SDR image in the forward reshaped SDR color space to a backward reshaped HDR image.

[0393] In one embodiment, the sampled HDR color space points are constructed in the HDR color space without using any images.

[0394] In one embodiment, multiple chains of optimized front reshaping mappings and optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for multiple candidate values ​​for the primary color scaling parameters; each chain in the multiple chains of optimized front reshaping mappings and optimized backward reshaping mappings includes a respective optimized front reshaping mapping and a respective optimized backward reshaping mapping.

[0395] In an embodiment, the sampled HDR color space points are mapped to reference SDR color space points based, at least in part, on a predefined HDR-SDR mapping.

[0396] In one embodiment, multiple sets of prediction errors are calculated for multiple chains of optimized forward reshaping mappings and optimized backward reshaping mappings; each set of prediction errors in the multiple sets of prediction errors is calculated for a respective chain in the multiple chains of optimized forward reshaping mappings and optimized backward reshaping mappings; and the multiple sets of prediction errors are used to select a particular candidate value from among multiple candidate values ​​for the primary color scaling parameters.

[0397] In one embodiment, particular candidate values ​​for the primary color scaling parameters are used to generate a particular chain of particular optimized forward reshaping mappings and particular optimized backward reshaping mappings.

[0398] In one embodiment, the particular optimized frontal reshape mapping is represented in a frontal reshape 3D lookup table.

[0399] In one embodiment, the particular optimized posterior reshape mapping is represented in a posterior reshape three-dimensional lookup table.

[0400] In one embodiment, a video encoder applies the optimized forward reshaping mapping to a sequence of input HDR images to generate a sequence of forward reshaped SDR images, and encodes the sequence of forward reshaped SDR images into a video signal.

[0401] In one embodiment, a video decoder decodes a sequence of forward reshaped SDR images from a video signal and applies an optimized backward reshaped mapping to the sequence of forward reshaped SDR images to generate a sequence of backward reshaped HDR images.

[0402] In one embodiment, a sequence of display images derived from the sequence of backward reshaped HDR images is rendered on an image display operating in conjunction with a video decoder.

[0403] In one embodiment, the HDR color space and the input HDR color space share a common white point.

[0404] In one embodiment, the reshape motion optimization algorithm represents a Backward Error Subtraction for Signal Conditioning (BESA) algorithm.

[0405] FIG. 4B illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) may perform this process flow. In block 422, the system described herein constructs sampled HDR color space points distributed throughout the HDR color space. The HDR color space is parameterized by primary color scaling parameters having candidate values ​​selected from among a plurality of candidate values. The primary color scaling parameters are used to calculate color space coordinates of at least one of a plurality of primaries that describe the HDR color space.

[0406] In block 424, the system generates from the sampled HDR color space points in the HDR color space: (a) input SDR color space points represented in the input SDR color space, and (b) reference HDR color space points represented in the reference HDR color space.

[0407] In block 426, the system executes a reshaping motion optimization algorithm to generate an optimized backward reshaping mapping. The reshaping motion optimization algorithm receives as input the input SDR color space points and the reference HDR color space points.

[0408] In one embodiment, the backward reshaping mapping is used to backward reshape an SDR image in an input SDR color space into a backward reshaped HDR image.

[0409] In one embodiment, the sampled HDR color space points are constructed in the HDR color space without any images.

[0410] In one embodiment, a plurality of optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for a plurality of candidate values ​​of the primary color scaling parameters; each optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings comprises a respective optimized backward reshaping mapping.

[0411] In one embodiment, multiple sets of prediction errors are calculated for a plurality of optimized backward reshaping mappings; each set of prediction errors in the multiple sets of prediction errors is calculated for a respective optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings; and the multiple sets of prediction errors are used to select a particular candidate value from among a plurality of candidate values ​​for the primary color scaling parameters.

[0412] In an embodiment, the sampled HDR color space points are processed by a programmable ISP pipeline into input SDR color space points based at least in part on optimized values ​​for programmable configuration parameters of the programmable ISP pipeline.

[0413] In an embodiment, optimized values ​​for the programmable configuration parameters of the programmable ISP pipeline are determined by minimizing an approximation error between an ISP SDR image generated by the programmable ISP pipeline from an HDR image and a reference SDR image generated by applying a predefined HDR-SDR mapping to the same HDR image.

[0414] 4C illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., an encoding device / module, a transcoding device / module, a decoding device / module, an inverse tone mapping device / module, a tone mapping device / module, a media device / module, an inverse mapping generation and application system, etc.) may perform this process flow. In block 442, the system described herein extracts a set of SDR image feature points from the training SDR image and extracts a set of HDR image feature points from the training HDR image.

[0415] At block 444, the system matches a subset of one or more SDR image feature points in the set of SDR image feature points with a subset of one or more HDR image feature points in the set of HDR image feature points.

[0416] At block 446, the system generates a geometric transformation to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image using the subset of one or more SDR image feature points and the subset of one or more HDR image feature points.

[0417] In block 448, the system determines a set of SDR and HDR color patch pairs from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and the HDR image have been spatially aligned by a geometric transformation.

[0418] In block 450, the system generates an optimized SDR-HDR mapping based at least in part on a set of SDR and HDR color patch pairs derived from the training SDR image and the training HDR image.

[0419] In block 452, the system applies the optimized SDR-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0420] In an embodiment, training SDR images and training HDR images are captured from a three-dimensional (3D) visual scene by a capture device operating in SDR and HDR capture modes, respectively.

[0421] In one embodiment, the training SDR image and the training HDR image form training SDR and HDR image pairs in a plurality of pairs of training SDR and HDR images; and the optimized SDR-HDR mapping is generated based at least in part on a plurality of sets of SDR and HDR color patch pairs derived from the plurality of pairs of training SDR and HDR images.

[0422] In one embodiment, each SDR image feature point in the subset of one or more SDR image features is matched with a respective HDR image feature point in the subset of one or more HDR image features; the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm.

[0423] In one embodiment, the common feature point extraction algorithm represents one of a binary robust invariant scalable keypoint algorithm, an accelerated features from segment testing algorithm, a KAZE algorithm, a minimum eigenvalue algorithm, a maximum stable extremum region algorithm, a directed FAST and rotation algorithm, a scale invariant feature transformation algorithm, a speeded up robust features algorithm, and the like.

[0424] 4D illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) may perform this process flow. At block 462, the system described herein performs a respective camera distortion correction operation on each training image of the training SDR image and training HDR image pair to generate a respective undistorted image in the undistorted training SDR image and undistorted training HDR image pair.

[0425] In block 464, the system generates a respective projective transformation for the pair of SDR image projective transformation and HDR image projective transformation using the corner pattern marks detected from each undistorted image in the pair of undistorted training SDR image and undistorted training HDR image.

[0426] In block 466, the system applies each projective transformation in the pair of SDR image projective transformation and HDR image projective transformation to a respective undistorted image in the pair of undistorted training SDR image and undistorted training HDR image to generate a respective rectified image in the pair of rectified training SDR image and rectified training HDR image.

[0427] In block 468, the system extracts a set of SDR color patches from the modified training SDR image and extracts a set of HDR color patches from the modified training HDR image.

[0428] In block 470, the system generates an optimized SDR-HDR mapping based at least in part on the set of SDR and HDR color patches derived from the training SDR and training HDR images.

[0429] At block 472, the system applies the optimized SDR-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0430] In an embodiment, the training SDR image and the training HDR image are captured from a common color target image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively.

[0431] In one embodiment, the common color chart image is selected from a plurality of color chart images, each of which includes a distinct distribution of color patches arranged on a two-dimensional color chart.

[0432] In one embodiment, distinct distributions of color patches are generated using random colors randomly selected from a common statistical distribution having a particular combination of statistical mean and variance.

[0433] In one embodiment, the common color chart image is rendered on a screen of a common reference image display, from which it is captured by the first capture device and the second capture device.

[0434] In one embodiment, each camera distortion correction operation is based, at least in part, on camera-specific distortion coefficients that are generated from a camera calibration process performed on the camera used to acquire the training images.

[0435] In an embodiment, the set of SDR color patches and the set of HDR color patches are used to derive a three-dimensional mapping table (3DMT), and an optimized SDR-HDR mapping is generated based at least in part on the 3DMT.

[0436] In an embodiment, the optimized SDR-HDR mapping represents one of a TPB-based mapping or a non-TPB-based mapping.

[0437] In one embodiment, the optimized SDR-HDR mapping is one of a static mapping applied to all non-training SDR images represented in the video signal, or a dynamic mapping generated based at least in part on a particular value distribution of SDR codewords of certain non-training SDR images represented in the video signal.

[0438] 4E illustrates an exemplary process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) may perform this process flow. In block 482, the system described herein constructs sampled HDR color space points that are distributed throughout the HDR color space used to represent the reconstructed HDR image.

[0439] At block 484, the system converts the sampled HDR color space points to SDR color space points in a first SDR color space in which the SDR image to be edited by the editing device is represented.

[0440] In block 486, the system determines a bounding SDR color space rectangle based on the SDR codeword extrema of the SDR color space points in the first SDR color space and determines the irregular 3D shape from the distribution of the SDR color space points.

[0441] In block 488, the system constructs sampled SDR color space points distributed throughout a bounding SDR color space rectangle in the first SDR color space.

[0442] In block 490, the system uses the sampled SDR color space points and the irregular shape to generate a boundary clipping 3D-LUT, which uses the sampled SDR color space points as lookup keys.

[0443] In block 492, the system performs a clipping operation on the edited SDR image in the first SDR color space based at least in part on the boundary clipping 3D-LUT to generate a boundary clipped edited SDR image in the first SDR color space.

[0444] In one embodiment, the clipping operation includes first performing regular clipping on the edited SDR image using a bounding SDR color space rectangle to generate a regular clipped edited SDR image, and then performing irregular clipping on the regular clipped edited SDR image using a 3D-LUT to generate a bounded clipped edited SDR image.

[0445] In one embodiment, a set of one or more SDR pixels in an SDR image to be edited are edited from one or more first luma values ​​to one or more second luma values ​​in the edited image, where the one or more second luma values ​​are different from the one or more first luma values.

[0446] In one embodiment, a set of one or more SDR pixels in an SDR image to be edited are edited from one or more first chrominance values ​​to one or more second chrominance values ​​in the edited image; the one or more second chrominance values ​​are different from the one or more first chrominance values.

[0447] In one embodiment, image details that were depicted in the SDR image to be edited are removed in the edited SDR image.

[0448] In one embodiment, image details not depicted in the SDR image being edited are added in the edited SDR image.

[0449] In one embodiment, the 3D-LUT includes one or more nodes, each of which includes a lookup key and a lookup value; the lookup key is equal to the lookup value, and the lookup key is within the irregular shape.

[0450] In one embodiment, the 3D-LUT includes one or more nodes, each of which includes a lookup key and a lookup value; the lookup key is outside the irregular shape and the lookup value is within the irregular shape.

[0451] In one embodiment, the lookup value is determined based on an index function that takes the irregular shape and the lookup key as input and returns the nearest neighbors of the lookup key as output.

[0452] In some embodiments, a computing device, such as a display device, a mobile device, a set-top box, a multimedia device, etc., is configured to perform any of the aforementioned methods. In some embodiments, a device comprises a processor and is configured to perform any of the aforementioned methods. In some embodiments, a non-transitory computer-readable storage medium storing software instructions that, when executed by one or more processors, result in the performance of any of the aforementioned methods.

[0453] In one embodiment, a computing device comprising one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, cause the device to perform any of the aforementioned methods.

[0454] It should be noted that although separate embodiments are discussed herein, any combination of the embodiments and / or sub-embodiments discussed herein may be combined to form further embodiments.

[0455] Exemplary Computer System Implementation The embodiments of the present invention may be implemented using a computer system, a system configured in electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or execute instructions related to adaptive perceptual quantization of images with enhanced dynamic range, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​associated with the adaptive perceptual quantization process described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0456] Certain implementations of the present invention include a computer processor executing software instructions that cause the processor to perform the methods of the present disclosure. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc., may perform methods related to adaptive perceptual quantization of HDR images as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention may be provided in the form of a program product. The program product may include any non-transitory medium carrying a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the methods of the embodiments of the present invention. A program product according to an embodiment of the present invention may be in any of a wide variety of forms. The program product may include a physical medium, such as, for example, a magnetic data storage medium including a floppy diskette, a hard disk drive, an optical data storage medium including a CD-ROM, a DVD, a ROM, an electronic data storage medium including a flash RAM, etc. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0457] When a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise noted, the reference to that component (including the reference to "means") should be interpreted as including any component that performs the function of the described component (e.g., is functionally equivalent) as an equivalent of that component, including components that are not structurally equivalent to the disclosed structures that perform the function in exemplary embodiments of the invention.

[0458] According to an embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) permanently programmed to perform the techniques, or may include one or more general-purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination thereof. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to realize the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices incorporating hard-wired logic and / or program logic to implement the techniques.

[0459] For example, Figure 5 is a block diagram illustrating a computer system 500 in which an embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0460] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions executed by processor 504. Main memory 506 may be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, make computer system 500 a special-purpose machine customized to perform operations specified in the instructions.

[0461] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0462] The computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 504 and for controlling cursor movement on the display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), allowing the device to specify a position in a plane.

[0463] Computer system 500 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and / or program logic that combine with the computer system to make or program computer system 500 into a special purpose machine. According to an embodiment, the techniques described herein are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0464] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0465] Storage media is distinct from, but may be used in the context of, transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0466] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus 502. The bus 502 carries the data to main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by the processor 504.

[0467] Computer system 500 also includes a communication interface 518 coupled to bus 502. The communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, the communication interface 518 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 may be a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0468] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526, in turn, provides data communication services through the world-wide packet data communication network now commonly referred to as the "Internet" 528. Local network 522 and Internet 528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks, the signals on network link 520 and the signals through communication interface 518, which carry the digital data to and from computer system 500, are exemplary forms of transmission media.

[0469] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0470] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0471] Equivalents, Extensions, Substitutes, and Others In the above specification, embodiments of the invention have been described with reference to numerous specific details that may vary depending on the implementation. Thus, the sole and exclusive indication of what is and is intended by the applicant to be a claimed embodiment of the invention is the set of claims allowed for this application, in the particular form in which such claims are allowed, including any subsequent amendments. Any definitions expressly set forth in this specification for terms contained in such claims shall govern the meaning of the terms used in such claims. Thus, any limitations, elements, properties, features, advantages or attributes not expressly recited in a claim should not limit the scope of the claim in any manner. Thus, the specification and drawings are to be regarded in an illustrative, and not restrictive, sense.

[0472] LISTED ILLUSTRATIVE EMBODIMENTS The present invention may be embodied in any of the forms described herein, including, but not limited to, the following enumerated example embodiments (EEE) that describe the structure, features, and functions of some portions of the embodiments of the present invention.

[0473] [EEE1] constructing sampled high dynamic range (HDR) color space points distributed throughout an HDR color space, the HDR color space being parameterized by primary color scaling parameters having candidate values ​​selected from a plurality of candidate values, the primary color scaling parameters being used to calculate color space coordinates of at least one primary color of a plurality of primaries defining the HDR color space; generating from the sampled HDR color space points in the HDR color space: (a) reference standard dynamic range (SDR) color space points represented in a reference SDR color space, (b) input HDR color space points represented in an input HDR color space, and (c) reference HDR color space points represented in a reference HDR color space; running a reshaping operation optimization algorithm to generate a chain of optimized forward reshaping mappings and optimized backward reshaping mappings, the reshaping operation optimization algorithm using the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs; the optimized forward reshaping mapping is used for forward reshaping an input HDR image in the input HDR color space to a forward reshaped SDR image in a forward reshaped SDR color space, and the optimized backward reshaping mapping is used for backward reshaping the front reshaped SDR image in the front reshaped SDR color space to a backward reshaped HDR image. method. [EEE2] The method of claim 1, wherein the sampled HDR color space points are constructed in the HDR color space without using any images. [EEE3] The method of any one of EEE1 to EEE2, wherein a plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for the plurality of candidate values ​​for the primary color scaling parameters; each chain in the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings includes a respective optimized forward reshaping mapping and a respective optimized backward reshaping mapping. [EEE4] 4. The method of any one of EEE1 to EEE3, wherein the sampled HDR color space points are mapped to the reference SDR color space points based at least in part on a predefined HDR-SDR mapping. [EEE5] 5. The method of any one of EEE1 to EEE4, wherein a plurality of sets of prediction errors are calculated for the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings; each set of prediction errors in the plurality of sets of prediction errors is calculated for a respective chain in the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings; and the plurality of sets of prediction errors are used to select a particular candidate value from the plurality of candidate values ​​for the primary color scaling parameters. [EEE6] The method according to EEE5, wherein the particular candidate values ​​for the primary color scaling parameters are used to generate a particular chain of particular optimized forward reshaping mappings and particular optimized backward reshaping mappings. [EEE7] The method of EEE6, wherein the particular optimized forward reshaping mapping is represented in a forward reshaping 3D lookup table. [EEE8] 8. The method of claim 6 or 7, wherein the particular optimized posterior reshaping mapping is represented in a posterior reshaping 3D lookup table. [EEE9] 9. The method of any one of claims 6 to 8, wherein a video encoder applies the optimized forward reshaping mapping to a sequence of input HDR images to generate a sequence of forward reshaped SDR images and encodes the sequence of forward reshaped SDR images into a video signal. [EEE10] 10. The method of any one of EEE6 to 9, wherein a video decoder decodes a sequence of forward reshaped SDR images from a video signal and applies the optimized backward reshaping mapping to the sequence of forward reshaped SDR images to generate a sequence of backward reshaped HDR images. [EEE11] The method of claim 8, wherein a sequence of display images derived from the sequence of backward reshaped HDR images is rendered on an image display operating in conjunction with the video decoder. [EEE12] 11. The method of any one of EEE1 to 10, wherein the HDR color space and the input HDR color space share a common white point. [EEE13] The method of any one of EEE1 to EEE10, wherein the reshaping motion optimization algorithm represents a Backward Error Subtraction for Signal Conditioning with Neutral Color Preservation (BESA) algorithm. [EEE14] constructing sampled high dynamic range (HDR) color space points distributed throughout an HDR color space, the HDR color space being parameterized by primary color scaling parameters having candidate values ​​selected from a plurality of candidate values, the primary color scaling parameters being used to calculate color space coordinates of at least one primary color of a plurality of primaries defining the HDR color space; generating, from the sampled HDR color space points in the HDR color space, (a) input standard dynamic range (SDR) color space points represented in an input SDR color space, and (b) reference HDR color space points represented in a reference HDR color space; running a reshaping motion optimization algorithm to generate an optimized backward reshaping mapping, the reshaping motion optimization algorithm receiving as input the input SDR color space points and the reference HDR color space points; the optimized backward reshaping mapping is used to backward reshape an SDR image in the input SDR color space into a backward reshaped HDR image; method. [EEE15] The method according to claim 8, wherein the sampled HDR color space points are constructed in the HDR color space without any images. [EEE16] The method of any one of claims 1 to 5, wherein a plurality of optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for the plurality of candidate values ​​for the primary color scaling parameters; each optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings comprises a respective optimized backward reshaping mapping. [EEE17] 17. The method of any one of EEE14 to 16, wherein a plurality of sets of prediction errors are calculated for the plurality of optimized backward reshaping mappings; each set of prediction errors in the plurality of sets of prediction errors is calculated for a respective optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings; and the plurality of sets of prediction errors are used to select a particular candidate value from among the plurality of candidate values ​​for the primary color scaling parameters. [EEE18] 18. The method of any one of claims 81 to 85, wherein the sampled HDR color space points are processed by a programmable image signal processor (ISP) pipeline into the input SDR color space points based at least in part on optimized values ​​for programmable configuration parameters of the ISP pipeline. [EEE19] 19. The method according to any one of EEE14 to 18, wherein the optimized values ​​for the programmable configuration parameters of the programmable ISP pipeline are determined by minimizing an approximation error between an ISP SDR image generated by the programmable ISP pipeline from an HDR image and a reference SDR image generated by applying a predefined HDR-SDR mapping to the same HDR image. [EEE20] 20. The method of any one of EEE1 to 19, wherein the distribution of sampled HDR color space points follows a predetermined pattern of the sample HDR color space points in the HDR color space. [EEE21] 21. The method of any one of EEE1 to 20, wherein at least one primary color of the plurality of primaries defining the HDR color space corresponds to a linear combination of this primary color in (i) an SDR color space and (ii) an HDR color space, and at least one weighting factor of the linear combination is based on the primary color scaling parameter. [EEE22] The method of any one of EEE1 to EEE21, wherein the constructing, generating and executing operations are performed for each candidate value of the primary color scaling parameters from the plurality of candidate values. [EEE23] The method of claim EEE22, wherein a particular candidate value from among the plurality of candidate values ​​for the primary color scaling parameters is selected based on a comparison of results of executing the reshaping operation optimization algorithm for each candidate value from the plurality of candidate values. [EEE24] The method of any one of EEE1 to 23, wherein the reshaping operation optimization algorithm strikes a balance between achieving a wide HDR color space of the backward reshaped HDR image and achieving low SDR color deviation. [EEE25] The method according to any one of claims 8 to 11, wherein the particular candidate value from among the plurality of candidate values ​​for the color primary scaling parameters is selected to achieve a balance between minimized prediction error and minimized SDR color deviation. [EEE26] The method of any one of EEE5 and EEE17, wherein the particular candidate value from among the plurality of candidate values ​​for the color primary scaling parameters is selected to achieve a minimized prediction error with SDR color deviation below a predetermined threshold. [EEE27] The method of any one of EEE1 to EEE26, wherein the optimized backward reshaping mapping is a tensor product B-spline (TPB) based mapping, and the reshaping operation optimization algorithm generates at least one optimized TPB coefficient for this TPB mapping. [EEE28] extracting a set of standard dynamic range (SDR) image feature points from the training SDR image and extracting a set of high dynamic range (HDR) image feature points from the training HDR image; matching a subset of one or more SDR image feature points in the set of SDR image feature points with a subset of one or more HDR image feature points in the set of HDR image feature points; generating a geometric transformation using the subset of one or more SDR image feature points and the subset of one or more HDR image feature points to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image; determining a set of SDR and HDR color patch pairs from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and HDR image are spatially aligned by the geometric transformation; generating an optimized SDR-HDR mapping based at least in part on the set of SDR and HDR color patch pairs derived from the training SDR image and the training HDR image; and applying the optimized SDR-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images. method. [EEE29] The method of EEE28, wherein the training SDR images and the training HDR images are captured from a three-dimensional (3D) visual scene by a capture device operating in SDR and HDR capture modes, respectively. [EEE30] The method of any one of claims 28 to 29, wherein the training SDR image and the training HDR image form training SDR and HDR image pairs in a plurality of pairs of training SDR and HDR images; and the optimized SDR-HDR mapping is generated based at least in part on a plurality of sets of SDR and HDR color patch pairs derived from the plurality of pairs of training SDR and HDR images. [EEE31] 31. The method of any one of EEE28 to 30, wherein each SDR image feature point in the subset of one or more SDR image feature points is matched with a respective HDR image feature point in the subset of one or more HDR image feature points; and the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm. [EEE32] The method according to EEE31, wherein the common feature point extraction algorithm represents one of a binary robust invariant scalable keypoint algorithm, an accelerated segment testing feature algorithm, a KAZE algorithm, a minimum eigenvalue algorithm, a maximum stable extremum region algorithm, a directed FAST and rotate algorithm, a scale invariant feature transformation algorithm, or a speeded up robust feature algorithm, etc. [EEE33] performing a respective camera distortion correction operation on each training image in the pair of training standard dynamic range (SDR) image and training high dynamic range (HDR) image to generate a respective undistorted image in the pair of undistorted training SDR image and undistorted training HDR image; generating each projective transformation in a pair of SDR and HDR image projective transformations using the corner pattern marks detected from each undistorted image in the pair of undistorted training SDR and undistorted training HDR images; applying each projective transformation in the pair of SDR image projective transformation and HDR image projective transformation to a respective undistorted image in the pair of undistorted training SDR image and undistorted training HDR image to generate a respective rectified image in the pair of rectified training SDR image and rectified training HDR image; extracting a set of SDR color patches from the modified training SDR image and extracting a set of HDR color patches from the modified training HDR image; generating an optimized SDR-HDR mapping based at least in part on the set of SDR color patches and the set of HDR color patches derived from the training SDR image and the training HDR image; and applying the optimized SDR-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images. method. [EEE34] The method of EEE33, wherein the training SDR images and the training HDR images are captured from a common color chart image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively. [EEE35] The method according to EEE34, wherein the common color chart image is selected from a plurality of color chart images, each comprising a separate distribution of color patches arranged on a two-dimensional color chart. [EEE36] The method according to EEE35, wherein said distinct distributions of color patches are generated using random colors randomly selected from a common statistical distribution having a particular combination of statistical mean and variance. [EEE37] The method of any one of EEE33 to 36, wherein the common color chart image is rendered on a screen of a common reference image display from which it is captured by the first capture device and the second capture device. [EEE38] The method of any one of EEE33 to 37, wherein the respective camera distortion correction operation is based, at least in part, on camera-specific distortion coefficients generated from a camera calibration process performed on the camera used to acquire the training images. [EEE39] 39. The method of any one of EEE33 to 38, wherein the set of SDR color patches and the set of HDR color patches are used to derive a three-dimensional mapping table (3DMT), and the optimized SDR-HDR mapping is generated based at least in part on the 3DMT. [EEE40] 40. The method of any one of EEE33 to 39, wherein the optimized SDR-HDR mapping represents one of a Tensor Product B-Spline (TPB) based mapping or a non-TPB based mapping. [EEE41] 41. The method according to any one of claims 33 to 40, wherein the optimized SDR-HDR mapping is one of a static mapping applied to all non-training SDR images represented in a video signal, or a dynamic mapping generated based at least in part on a particular value distribution of SDR codewords of certain non-training SDR images represented in the video signal. [EEE42] constructing sampled high dynamic range (HDR) color space points distributed throughout the HDR color space used to represent the reconstructed HDR image; converting the sampled HDR color space points to standard dynamic range (SDR) color space points in a first SDR color space in which an SDR image to be edited by an editing device is represented; determining a bounding SDR color space rectangle based on extrema of SDR codeword values ​​of the SDR color space points in the first SDR color space and determining an irregular three-dimensional (3D) shape from a distribution of the SDR color space points; constructing sampled SDR color space points distributed throughout the bounding SDR color space rectangle in the first SDR color space; generating a boundary clipping 3D lookup table (3D-LUT) using the sampled SDR color space points and the irregular shape, the boundary clipping 3D-LUT using the sampled SDR color space points as lookup keys; performing a clipping operation on the edited SDR image in the first SDR color space based at least in part on the boundary clipping 3D-LUT to generate a boundary clipped edited SDR image in the first SDR color space. method. [EEE43] The method of claim EEE42, wherein the clipping operation includes first performing regular clipping on the edited SDR image using the bounding SDR color space rectangle to generate a regular clipped edited SDR image, and then performing irregular clipping on the regular clipped edited SDR image using the 3D-LUT to generate the boundary clipped edited SDR image. [EEE44] The method of any one of claims 42 to 43, wherein a set of one or more SDR pixels in the edited SDR image are edited from one or more first luminance values ​​to one or more second luminance values ​​in the edited image, the one or more second luminance values ​​being different from the one or more first luminance values. [EEE45] A method according to any one of EEE42 to 44, wherein a set of one or more SDR pixels in the edited SDR image are edited from one or more first chrominance values ​​to one or more second chrominance values ​​in the edited image; the one or more second chrominance values ​​being different from the one or more first chrominance values. [EEE46] The method of any one of EEE42 to 45, wherein image details depicted in the SDR image to be edited are removed in the edited SDR image. [EEE47] 47. The method of any one of EEE42 to 46, wherein image details not depicted in the SDR image to be edited are added in the edited SDR image. [EEE48] The method of any one of EEE42 to 47, wherein the 3D-LUT comprises one or more nodes, each node comprising a lookup key and a lookup value; the lookup key being equal to the lookup value, the lookup key being within the range of the irregular shape. [EEE49] The method of any one of EEE42 to 48, wherein the 3D-LUT comprises one or more nodes, each node comprising a lookup key and a lookup value; the lookup key being outside the irregular shape and the lookup value being within the irregular shape. [EEE50] The method of claim 8, wherein the lookup value is determined based on an index function that takes the irregular shape and the lookup key as input and returns as output a nearest neighbor of the lookup key. [EEE51] An apparatus having a processor and configured to carry out the method of any one of EEE1 to EEE50. [EEE52] A non-transitory computer-readable storage medium storing computer-executable instructions for executing a method on one or more processors according to the method of any one of EEE1 to EEE50. [EEE53] A computer system configured to carry out the method according to any one of claims 1 to 50.

Claims

1. constructing sampled high dynamic range (HDR) color space points distributed throughout an HDR color space, the HDR color space being parameterized by primary color scaling parameters having candidate values ​​selected from a plurality of candidate values, the primary color scaling parameters being used to calculate color space coordinates of at least one primary color of a plurality of primaries defining the HDR color space; generating from the sampled HDR color space points in the HDR color space: (a) reference standard dynamic range (SDR) color space points represented in a reference SDR color space, (b) input HDR color space points represented in an input HDR color space, and (c) reference HDR color space points represented in a reference HDR color space; running a reshaping operation optimization algorithm to generate a chain of optimized forward reshaping mappings and optimized backward reshaping mappings, the reshaping operation optimization algorithm using the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs; the optimized forward reshaping mapping is used for forward reshaping an input HDR image in the input HDR color space to a forward reshaped SDR image in a forward reshaped SDR color space, and the optimized backward reshaping mapping is used for backward reshaping the front reshaped SDR image in the front reshaped SDR color space to a backward reshaped HDR image. method.

2. The method of claim 1 , wherein the sampled HDR color space points are constructed in the HDR color space without using any images.

3. 3. The method of claim 1 or 2, wherein a plurality of chains of optimized front reshaping mappings and optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for the plurality of candidate values ​​for the primary color scaling parameters; and each chain in the plurality of chains of optimized front reshaping mappings and optimized backward reshaping mappings includes a respective optimized front reshaping mapping and a respective optimized backward reshaping mapping.

4. 4. The method of claim 1, wherein the sampled HDR color space points are mapped to the reference SDR color space points based at least in part on a predefined HDR-SDR mapping.

5. 5. The method of claim 1, wherein a plurality of sets of prediction errors are calculated for the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings; each set of prediction errors in the plurality of sets of prediction errors is calculated for a respective chain in the plurality of chains of optimized forward reshaping mappings and optimized backward reshaping mappings; and the plurality of sets of prediction errors are used to select a particular candidate value from the plurality of candidate values ​​for the primary color scaling parameters.

6. The method of claim 1 , wherein the HDR color space and the input HDR color space share a common white point.

7. 7. The method according to claim 1, wherein the reshaping motion optimization algorithm represents a Backward Error Subtraction for Signal Conditioning with Neutral Color Preservation (BESA) algorithm.

8. constructing sampled high dynamic range (HDR) color space points distributed throughout an HDR color space, the HDR color space being parameterized by primary color scaling parameters having candidate values ​​selected from a plurality of candidate values, the primary color scaling parameters being used to calculate color space coordinates of at least one primary color of a plurality of primaries defining the HDR color space; generating, from the sampled HDR color space points in the HDR color space, (a) input standard dynamic range (SDR) color space points represented in an input SDR color space, and (b) reference HDR color space points represented in a reference HDR color space; running a reshaping motion optimization algorithm to generate an optimized backward reshaping mapping, the reshaping motion optimization algorithm receiving as input the input SDR color space points and the reference HDR color space points; the optimized backward reshaping mapping is used to backward reshape an SDR image in the input SDR color space into a backward reshaped HDR image; method.

9. The method of claim 8 , wherein the sampled HDR color space points are constructed in the HDR color space without any images.

10. 10. The method of claim 8 or 9, wherein a plurality of optimized backward reshaping mappings are generated by the reshaping operation optimization algorithm for the plurality of candidate values ​​for the primary color scaling parameters; and each optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings comprises a respective optimized backward reshaping mapping.

11. 11. The method of claim 8, wherein a plurality of sets of prediction errors are calculated for the plurality of optimized backward reshaping mappings; each set of prediction errors in the plurality of sets of prediction errors is calculated for a respective optimized backward reshaping mapping in the plurality of optimized backward reshaping mappings; and the plurality of sets of prediction errors are used to select a particular candidate value from among the plurality of candidate values ​​for the primary color scaling parameters.

12. 12. The method of claim 8, wherein the sampled HDR color space points are processed by a programmable image signal processor (ISP) pipeline into the input SDR color space points based at least in part on optimized values ​​for programmable configuration parameters of the programmable image signal processor (ISP) pipeline.

13. 13. The method of claim 8, wherein the optimized values ​​for the programmable configuration parameters of the programmable ISP pipeline are determined by minimizing an approximation error between an ISP SDR image generated by the programmable ISP pipeline from an HDR image and a reference SDR image generated by applying a predefined HDR-SDR mapping to the same HDR image.

14. Apparatus comprising a processor and configured to carry out the method of any one of claims 1 to 13.

15. A non-transitory computer readable storage medium storing computer executable instructions for carrying out, on one or more processors, a method according to any one of the preceding claims.

Images