Image optimization in mobile capture and editing applications

Through image reshaping technology based on tensor product B-splines, the efficiency and quality problems of HDR image to SDR image mapping in mobile devices are solved, and efficient and reversible image processing is achieved, suitable for video editing and transmission of mobile devices.

CN118985127APending Publication Date: 2024-11-19DOLBY LABORATORIES LICENSING CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380033052.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-18
Filing Date
2023-03-17
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently process and map high dynamic range (HDR) images to standard dynamic range (SDR) images in mobile devices, especially during video editing and transmission, resulting in image quality degradation and irreversibility problems.

Method used

Using a tensor product B-spline (TPB)-based image remodeling solution, generate and process HDR and SDR images through optimized forward and reverse remodeling mapping, combined with neutral color processing and constraint optimization to ensure image quality and reversibility.

Benefits of technology

It realizes efficient processing of HDR images on mobile devices, generates high-quality SDR images, and maintains the reversibility of HDR images, reducing data overhead and processing delays, and adapts to different use cases and device constraints.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0005074771080000191
    Figure BDA0005074771080000191
  • Figure BDA0005074771080000197
    Figure BDA0005074771080000197
  • Figure BDA0005074771080000198
    Figure BDA0005074771080000198
Patent Text Reader

Abstract

An HDR color patch is sampled in an HDR color space parameterized by a parameter. And generating a reference SDR color block from the sampling HDR color block, an input HDR color block and a reference HDR color block. An optimization algorithm is executed to generate an optimized forward remodeling map and an optimized reverse remodeling map. The optimized forward remodeling map is used to forward remodel an input HDR image to a forward remodeled SDR image, and the optimized reverse remodeling map is used to reverse remodel the forward remodeled SDR image to a reverse remodeled HDR image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 321,390, filed on March 18, 2022, and European Patent Application 22 162 983.5, filed on March 18, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to images and more particularly to video codecs for processing images. Background Art

[0004] As used herein, the term "dynamic range (DR)" may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., brightness, luminance) in an image, e.g., from the darkest black (dark colors) to the brightest white (highlights). In this sense, DR is related to "scene-referred" intensities. DR may also relate to the ability of a display device to adequately or approximately render a range of intensities of a particular breadth. In this sense, DR is related to "display-referred" intensities. Unless a particular sense is expressly specified at any point in the description herein to have a particular meaning, it should be inferred that the terms may be used in either sense, e.g., interchangeably.

[0005] As used herein, the term "high dynamic range (HDR)" refers to a DR width that spans approximately 14 to 15 or more orders of magnitude across the human visual system (HVS). In practice, the DR, over a wide range of intensity ranges that humans can simultaneously perceive, may be slightly truncated relative to HDR. As used herein, the terms "enhanced dynamic range (EDR) or visual dynamic range (VDR)" may be individually or interchangeably related to such a DR that can be perceived within a scene or image by the human visual system (HVS) including eye movement, allowing for some light adaptation changes across the scene or image. As used herein, EDR may refer to a DR that spans 5 to 6 orders of magnitude. Therefore, while HDR may be slightly narrower relative to a real scene reference, EDR may represent a wide DR width and may also be referred to as HDR.

[0006] In practice, an image includes one or more color components of a color space (e.g., luminance Y and chrominance Cb and Cr), where each color component is represented by n bits of precision per pixel (e.g., n=8). Using nonlinear light intensity coding (e.g., gamma coding), images where n≤8 (e.g., color 24-bit JPEG images) are considered as images of standard dynamic range, while images where n>8 can be considered as images of enhanced dynamic range.

[0007] The reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between the color values ​​(e.g., luminance) of an input video signal and the output screen color values ​​(e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT.1886, "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011), defines a reference EOTF for flat panel displays, which is incorporated herein by reference in its entirety. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. The term "metadata" herein refers to any auxiliary information that is transmitted as part of an encoded bitstream and that assists a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters as described herein.

[0008] As used herein, the term "PQ" refers to the quantization of the perceived brightness amplitude. The human visual system responds to the increased light level in a very nonlinear manner. The ability of humans to observe stimuli is affected by the following factors: the brightness of the stimuli, the size of the stimuli, the spatial frequency that constitutes the stimuli, and the brightness level that the eyes adapt to at the specific moment of viewing the stimuli. In some embodiments, the perception quantizer function maps the linear input grayscale to the output grayscale that better matches the contrast sensitivity threshold in the human visual system. In SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays" (hereinafter referred to as "SMPTE"), an example PQ mapping function is described, which is incorporated herein by reference in its entirety, wherein, given a fixed stimulus size, for each brightness level (e.g., stimulus level, etc.), the minimum visible contrast step at the brightness level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

[0009] Supports 200 to 1,000 cd / m 2 A display with a brightness of 1,000 nits or less represents a lower dynamic range (LDR) associated with EDR (or HDR), also known as standard dynamic range (SDR). EDR content can be displayed on an EDR display that supports a higher dynamic range (e.g., from 1,000 nits to 5,000 nits or more). Such a display can be defined using an alternative EOTF that supports high brightness capabilities (e.g., 0 to 10,000 nits or more). Examples of such EOTFs are defined in SMPTE 2084 and Rec.ITU-R BT.2100, “Image parameter values ​​for high Dynamic range television for use in production and international programme exchange” (06 / 2017). As the inventors understand herein, it is expected that improved techniques for synthesizing video content data can be used to support the display capabilities of various SDR and HDR display devices.

[0010] WO 2019 / 217751 A1 relates to image encoding and / or image decoding. As a non-limiting example, a technique is disclosed according to which 3D mapping statistics are generated for a first image of a first dynamic range and a second image of a second dynamic range different from the first dynamic range. In this example technique, multivariate multiple regression (MMR) coefficients are generated by solving an optimization problem formulated using an MMR matrix constructed using the 3D mapping statistics without black edge constraints, and are used to generate a chromaticity map for predicting chromaticity codeword values ​​of the second image. Further according to the example technique, it is determined whether there are black edges in the image, and if so, whether the chromaticity map accurately predicts the chromaticity codeword values ​​in the second image. According to a further non-limiting example, a reconstructed image generated by a receiving device by inversely reshaping one of the images is rendered by a display device operating with the receiving device.

[0011] WO 2021 / 076822 A1 relates to image encoding and / or image decoding. As a non-limiting example, a technique is disclosed according to which a reverse reshaping map is initially generated as an inverse map of a forward reshaping map. In this example technique, the reverse reshaping map is updated by replacing the luminance codewords of the content map with forward reshaped luminance codewords, which are generated by applying the luminance forward map to the sampled luminance codewords. Further according to the example technique, a luminance forward map is constructed from the forward reshaping map, and a reverse reshaping map is generated using the reverse reshaping map and the luminance forward map for creating a reconstructed image from the forward reshaped image. Further according to the example technique, the forward reshaped image is encoded in a video signal together with image metadata specifying the reverse reshaping map. According to a further non-limiting example, a receiving device of the video signal applies the reverse reshaping map to the forward reshaped image to create a reconstructed image of a second dynamic range.

[0012] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0014] Figures 1A to 1E illustrates an example process flow for applying an optimized reshape operation; Figure 2A illustrates example color distributions of irregular shapes associated with HDR and SDR color spaces such as R.2020, P3, and R.709 color spaces; FIG. 2B to FIG. 2E and Figure 2H to Figure 2K An example parameterized color space in the R.2020 color space is illustrated; Figure 2F and Figure 2G illustrates example percentile Cb and Cr values ​​in a forward reshaped SDR color space corresponding to different parameterized HDR color spaces; Figure 2L An example color table image is illustrated; Figure 2M and Figure 2N An example chessboard image is illustrated; Fig.2O illustrates an example original color table image displayed on a reference image display, an example captured image from the color table image, and a corrected captured image generated from the captured image; Figure 2P An example RGB value distribution is illustrated; Figure 2Q to Figure 2T illustrates an example alphaShape;

[0015] Figure 3A and Figure 3C illustrates an example process flow for finding an optimized color space to represent HDR colors; Figure 3B An example process flow for determining optimized values ​​of programmable parameters in an ISP pipeline is illustrated; Figure 3D illustrates an example process flow for generating optimized reshaping operation parameters; Figure 3E illustrates an example process flow for generating a plurality of different color tables; Figure 3F An example process flow for matching SDR and HDR colors between an image pair of captured SDR and HDR images is illustrated; Figure 3G and Figure 3H illustrates example encoder-side and decoder-side image / video editing operations; FIG. 3I to FIG. 3M An example solution for clipping out-of-range codewords in image / video editing applications is illustrated;

[0016] FIG. 4A to FIG. 4E illustrates an example process flow; and

[0017] Figure 5 Illustrated is a simplified block diagram of an example hardware platform upon which a computer or computing device as described herein may be implemented. DETAILED DESCRIPTION

[0018] In the following description, for the purpose of illustration, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is apparent that the present disclosure can be practiced without these specific details. In other cases, in order to avoid unnecessarily obscuring, obscuring or confusing the present disclosure, well-known structures and devices are not described in detail.

[0019] Overview

[0020] Image optimizations are described herein for video capture applications including but not limited to mobile (video) capture applications, such as those related to image reconstruction solutions based on tensor product B-splines (TPB). Solutions based on TPB can use robust predictors implemented with video codecs such as backward compatible video codecs to provide or generate relatively accurate reconstructed images. These predictors can operate with static mappings to calmly handle or recover from transmission errors such as lost frames, minimize power consumption such as battery power consumption, reduce data overhead and / or processing / transmission delays, etc. In some operating scenarios, some or all of the solutions based on TPB as described herein can be implemented with mobile applications that are subject to relatively stringent battery or power constraints.

[0021] The TPB-based image reshaping solution can be adapted to operate in different use cases. Some use cases allow the freedom to design and / or apply HDR to SDR mapping to generate SDR images to be encoded in (the base layer of) the video signal, as described herein. Some use cases rely on a mobile image signal processor (ISP) with relatively limited programmable registers to generate SDR images to be encoded in (the base layer of) the video signal. Different architectures can be used to implement the TPB-based solution as described herein. Additionally, optionally or alternatively, these architectures or TPB-based solutions can be used to provide or support backward compatibility.

[0022] Given a reference SDR image represented in an SDR color space or a color space in an SDR domain, a TPB-based solution may use a TPB optimization process to achieve or determine the maximum or widest HDR color space or a color space in an HDR domain to represent a corresponding reconstructed HDR image predicted from the SDR image. The wider the HDR color space, the more color deviations are introduced into the SDR color space. The TPB optimization process may be implemented to achieve or determine an optimized balance point or trade-off between achieving the maximum or widest HDR color space on the one hand and introducing SDR color deviations on the other hand. Additionally, optionally or alternatively, a TPB optimization process as described herein may be implemented to combine neutral color processing or perform constrained optimization to help preserve the grayscale represented in the SDR image in the reconstructed HDR image predicted from the SDR image.

[0023] As various camera-equipped computing or mobile devices capture more and more videos in practice, video editing has become more and more popular. Users operating these devices may be allowed to manually adjust the appearance of the captured video or simply apply a default theme template to the captured video. Depending on the available computing resources and / or the preferences of the video editing tool designer / provider, video editing can be done in a source domain such as the source HDR domain (in which the source image is represented) or a non-source domain such as the generated SDR domain (the pre-edited image can be converted from the source image in the source domain to the non-source domain).

[0024] In an operational scenario where an SDR image is to be encoded in a video signal, video editing operations in the HDR domain or source domain do not adversely affect the reconstruction of an HDR image from an SDR image decoded from the video signal at the decoder side by a device downstream of the video signal. This is because these video editing operations do not interfere with the HDR to SDR mapping that generates an SDR image from an HDR image or source image in the HDR domain at the encoder side, and do not interfere with the approximate SDR to HDR mapping that maps back to an HDR image or reconstructs an HDR image at the decoder side.

[0025] However, in these operating scenarios, video editing operations in the SDR domain will likely adversely affect the reversibility between the SDR domain and the HDR domain. For example, an edited SDR image may not be able to be mapped back to the original or source HDR image. In addition, the color space used to represent the SDR image may be limited, and its pixel values ​​or codewords cannot exceed a predefined range such as the SMPTE range. Color space conversions such as YUV to RGB conversion for video editing operations may cause pixel values ​​or codewords to be sheared and thus damage or harm reversibility. The shearing techniques described herein can be used to reduce or prevent adverse editing effects on video editing operations (including but not limited to those performed with mobile devices). Two-level TPB boundary shearing can be implemented in the RGB domain to support or retain the maximum edited colors in the SDR domain, which can be used to propagate or map back to the reconstructed images in the HDR domain.

[0026] The example embodiments described herein relate to image generation. Sampled HDR color space points distributed in an HDR color space are constructed. The HDR color space is parameterized by a color source scaling parameter having a candidate value selected from a plurality of candidate values. The color source scaling parameter is used to calculate the color space coordinates of at least one color source of a plurality of color sources outlining the HDR color space. Reference SDR color space points represented in a reference SDR color space, input HDR color space points represented in an input HDR color space, and reference HDR color space points represented in a reference HDR color space are generated from the sampled HDR color space points in the HDR color space. A reshaping operation optimization algorithm is performed to generate an optimized forward reshaping mapping and an optimized reverse reshaping mapping chain. The reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as input. The optimized forward reshaping mapping is used to forward reshape the input HDR images in the input HDR color space into forward reshaped SDR images in the forward reshaped SDR color space, and the optimized reverse reshaping mapping is used to reverse reshape the forward reshaped SDR images in the forward reshaped SDR color space into reverse reshaped HDR images.

[0027] The example embodiments described herein relate to image generation. Sampled HDR color space points distributed in an HDR color space are constructed. The HDR color space is parameterized by a chromatic amplification parameter having a candidate value selected from a plurality of candidate values. The chromatic amplification parameter is used to calculate the color space coordinates of at least one chromatic amplification of a plurality of chromatic amplifications outlining the HDR color space. Input SDR color space points represented in an input SDR color space and reference HDR color space points represented in a reference HDR color space are generated from the sampled HDR color space points in the HDR color space. A reshaping operation optimization algorithm is performed to generate an optimized reverse reshaping mapping. The reshaping operation optimization algorithm receives the input SDR color space points and the reference HDR color space points as input. The reverse reshaping mapping is used to reversely reshape the SDR image in the input SDR color space into a reversely reshaped HDR image.

[0028] The example embodiments described herein relate to image generation. A set of SDR image feature points are extracted from a training SDR image, and a set of HDR image feature points are extracted from a training HDR image. A subset of one or more SDR image feature points in the set of SDR image feature points is matched with a subset of one or more HDR image feature points in the set of HDR image feature points. A geometric transformation is generated using the subset of the one or more SDR image feature points and the subset of the one or more HDR image feature points to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image. A set of paired SDR color blocks and HDR color blocks are determined from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image has been spatially aligned with the training HDR image by the geometric transformation. An optimized SDR to HDR mapping is generated based at least in part on the set of paired SDR color blocks and HDR color blocks obtained from the training SDR image and the training HDR image. The optimized SDR to HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0029] Example embodiments described herein relate to image generation. A corresponding camera distortion correction operation is performed on each training image in a pair of training standard dynamic range (SDR) images and training high dynamic range (HDR) images to generate a corresponding undistorted image in a pair of undistorted training SDR images and undistorted training HDR images. A corresponding projection transformation in a pair of SDR image projection transformations and HDR image projection transformations is generated using corner pattern markers detected from each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images. Each projection transformation in the pair of SDR image projection transformations and HDR image projection transformations is applied to the corresponding undistorted image in the pair of undistorted training SDR images and undistorted training HDR images to generate a corresponding corrected image in a pair of corrected training SDR images and corrected training HDR images. A set of SDR color patches is extracted from the corrected training SDR image, and a set of HDR color patches is extracted from the corrected training HDR image. An optimized SDR to HDR mapping is generated based at least in part on the set of SDR color patches and the set of HDR color patches obtained from the training SDR image and the training HDR image. The optimized SDR-to-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0030] The example embodiments described herein relate to a shearing operation on an edited image. Sampled HDR color space points distributed in an HDR color space for representing a reconstructed HDR image are constructed. The sampled HDR color space points are converted to SDR color space points in a first SDR color space, in which an SDR image to be edited by an editing device is represented. A delimiting SDR color space rectangle is determined based on extreme SDR codeword values ​​of the SDR color space points in the first SDR color space. An irregular three-dimensional (3D) shape is determined from the distribution of the SDR color space points. Sampled SDR color space points distributed in the delimiting SDR color space rectangle in the first SDR color space are constructed. A boundary shearing 3D lookup table (3D-LUT) is generated using the sampled SDR color space points and the irregular shape. The boundary shearing 3D-LUT uses the sampled SDR color space points as lookup keys. A shearing operation is performed on the edited SDR image in the first SDR color space based at least in part on the boundary shearing 3D-LUT to generate a boundary sheared edited SDR image in the first SDR color space.

[0031] Reshaping optimization in image / video capture applications

[0032] Reshaping optimization processes such as TPB and / or non-TPB optimization processes may be implemented or incorporated into video capture applications that are run on computing devices such as mobile devices in various operating scenarios. Video capture applications with reshaping optimization processes may be used to generate or output video signals such as base layers or SDR images encoded therein. The reshaping optimization process may implement different solutions in different operating scenarios to generate or optimize reshaping operation parameters such as TPB and / or non-TPB coefficients for use with base layers or SDR images to generate, construct, or reconstruct non-base layers or HDR images with optimized picture quality.

[0033] For illustrative purposes, the reshaping optimization processes or solutions implemented therein can be classified into different types based on the specific SDR generation process or sub-process adopted by the video capture application and based on the specific reshaping path (forward (reshaping) path and / or reverse (reshaping) path) for performing the reshaping optimization.

[0034] Figure 1A A first example reshaping optimization process implementing a white-box joint forward and reverse TPB optimization design / solution (referred to as "WFB") with a (single) video capture device is illustrated. The white-box joint forward and reverse TPB optimization design / solution can be implemented for the following operation scenarios: an SDR generation process or sub-process converts a reference HDR image to a reference SDR image using a white-box conversion operation, and both the forward and reverse (reshaping) paths can be subjected to TPB optimization.

[0035] As used herein, "white-box conversion" refers to an HDR to SDR mapping or conversion performed using well-defined conversion or mapping functions / equations, such as those specified or documented in a standard-based or proprietary video coding specification (e.g., publicly, etc.). In contrast, "black-box conversion" refers to an HDR to SDR mapping or conversion performed using a conversion or mapping operation that is not based on a well-defined conversion or mapping function / equation. For example, a black-box conversion may be implemented as internal image signal processing performed by an image signal processor that does not rely or relies little on any well-defined conversion or mapping function or equation specified or documented in a standard-based or proprietary video coding specification (e.g., publicly, etc.).

[0036] By way of example and not limitation, Figure 1A In the present invention, an input video signal including an HDR image may be represented in an input color space, such as a full R.2020 color space. A generated SDR signal including a reference SDR image may be represented in a reference SDR color space. The reference SDR image may be an image generated from the HDR image using a known HDR to SDR mapping process, function, and / or equation.

[0037] like Figure 1A As shown in , in the forward reshaping path, the reference HDR image may be forward reshaped into a (forward) reshaped SDR image that is approximate to the reference SDR image based at least in part on optimized forward reshaping operation parameters generated from the joint forward and backward TPB optimization solution (denoted as "forward TPB optimization"). The reshaped SDR image may be represented in a reshaped SDR color space and generated by an upstream device to be included / encoded in a video signal or its base layer (BL) output by the upstream device.

[0038] A downstream receiving device or video decoder of the video signal may decode the reshaped SDR image from the video signal. The decoded reshaped SDR image at the decoder side may be identical to the reshaped SDR image at the encoder side, but errors may be introduced in compression / decompression, encoding operations, and / or data transmission.

[0039] In a reverse reshaping path as implemented by a downstream device, the reshaped SDR image may be reversely reshaped into a (reverse) reshaped HDR image based at least in part on optimized reverse reshaping operation parameters generated from the same joint forward and reverse TPB optimization solution (denoted as "Reverse TPB Optimization"). The reshaped HDR image generated with the optimized reverse reshaping operation parameters in the output HDR color space represents an approximate or reconstructed version of the reference HDR image.

[0040] The joint forward and backward TPB optimization solution can generate optimized forward and backward reshaping operation parameters to cover as wide as possible output HDR color space in which the reshaped or reconstructed HDR images generated at the decoder side are represented.

[0041] One or both of the optimized forward and reverse reshaping operation parameters, such as optimized forward and reverse TPB coefficients, may be implemented or represented in a three-dimensional lookup table or 3D-LUT to reduce processing time in the reshaping operation.

[0042] Since the forward and reverse reshaping operation parameters such as forward and reverse TPB coefficients are jointly or simultaneously designed or optimized in the joint forward and backward TPB optimization process, the supported reshaping SDR and HDR color spaces for representing the reshaping SDR and HDR images can be jointly or simultaneously designed or optimized in the same process.

[0043] In some operational scenarios, the optimized forward and reverse reshaping operation parameters generated from the joint forward and reverse TPB optimization process can be applied in a static single layer backward compatible (SLBC) framework. Under this static framework, it is not necessary to obtain (e.g., dynamically, etc.) optimized image-specific or image-dependent optimized forward and reverse operation parameters, such as image-specific or image-dependent (or content-dependent) forward and reverse TPB coefficients, such as obtained on the fly when processing the image.

[0044] Instead, under the static SLBC framework, the same or statically optimized forward and reverse operation parameters, such as the same optimized forward and reverse TPB coefficients, can be obtained or generated at one time (e.g., offline or before performing a reshaping operation on any of the (input) reference HDR images or the reshaped SDR images) for forward reshaping all (input) reference HDR images and reverse reshaping the reshaped SDR images. In an example, a single set of statically optimized forward and reverse operation parameters can be generated offline by a system as described herein, and configured / deployed in or used by a capture device as described herein to perform an image reshaping or reconstruction operation. In another example, multiple sets of statically optimized forward and reverse operation parameters can be generated offline by a system as described herein, and configured / deployed in or used by a capture device as described herein to select a specific set of statically optimized forward and reverse operation parameters to perform an image reshaping or reconstruction operation.

[0045] Thereafter, the upstream device may apply the optimized static forward TPB coefficients to forward reshape some or all (input) reference HDR images (e.g., a continuous or sequential (input) reference HDR image sequence, etc.) to generate a reshaped SDR image to be encoded in a (SLBC) video signal, while the downstream receiving device of the video signal may apply the optimized static reverse TPB coefficients to some or all reshaped SDR images decoded from the video signal (e.g., a continuous or sequential reshaped SDR image sequence, etc.) to generate or reconstruct a reshaped HDR image.

[0046] Figure 1B A second example reshaping optimization process implementing a white-box reverse-only TPB optimization design / solution (referred to as "WB") with a (single) video capture device is illustrated. The white-box reverse-only TPB optimization design / solution can be implemented for the following operation scenarios: the SDR generation process or sub-process converts a reference HDR image to a reference SDR image using a white-box conversion operation, and only the reverse (reshaping) path is subjected to TPB optimization.

[0047] By way of example and not limitation, Figure 1BIn the present invention, an input video signal including an HDR image may be represented in an input color space such as a P3 color space. A generated SDR signal including a reference SDR image may be represented in a reference SDR color space. The reference SDR image may be an image generated from the HDR image using a known HDR to SDR mapping process, function, and / or equation.

[0048] like Figure 1B As shown in , a programmable image signal processor or ISP can process a reference HDR image into an ISP SDR image that approximates a reference SDR image (e.g., using a given image signal processing function / equation / operation, etc.) based at least in part on optimized ISP operating parameters generated from an ISP optimization solution (denoted as "ISP parameter optimization"). The ISP SDR image can be represented in an ISP SDR color space and generated by an upstream device to be included / encoded in a video signal or its base layer (BL) output by the upstream device.

[0049] A downstream receiving device or video decoder of the video signal can decode the ISP SDR image from the video signal. The decoded ISP SDR image on the decoder side may be the same as the ISP SDR image on the encoder side, but errors may be introduced in compression / decompression, encoding operations and / or data transmission.

[0050] In a reverse reshaping path as implemented by a downstream device, the ISP SDR image may be reverse reshaped into a (reverse) reshaped HDR image based at least in part on optimized reverse reshaping operation parameters generated from a reverse-only TPB optimization solution (denoted as "Reverse TPB Optimization"). The reshaped HDR image generated with the optimized reverse reshaping operation parameters in the output HDR color space represents an approximate or reconstructed version of the reference HDR image.

[0051] Only the inverse TPB optimization solution can generate optimized inverse reshaping operation parameters to cover as wide as possible an output HDR color space in which the reshaped or reconstructed HDR images generated at the decoder side are represented.

[0052] The optimized inverse reshaping operation parameters, such as optimized inverse TPB coefficients, may be implemented or represented in a three-dimensional lookup table or 3D-LUT to reduce processing time in the reshaping operation.

[0053] In some operational scenarios, the inverse reshaping operational parameters generated from the inverse TPB-only optimization process may be applied in a static single-layer inverse display mapping (SLiDM) framework. Under this static framework, there is no need to obtain (e.g., dynamically, etc.) inverse operational parameters that optimize image-specific or image-dependent optimizations, such as image-specific or image-dependent (or content-dependent) inverse TPB coefficients.

[0054] Instead, under the static SLiDM framework, the same or statically optimized reverse operation parameters, such as the same optimized reverse TPB coefficients, may be obtained or generated once (e.g., offline or before performing a reshaping operation on any of the reshaped SDR images) for reverse reshaping the ISP SDR image. In an example, a single set of statically optimized reverse operation parameters may be generated offline by a system as described herein, and configured / deployed in or used by a capture device as described herein to perform an image reshaping or reconstruction operation. In another example, multiple sets of statically optimized reverse operation parameters may be generated offline by a system as described herein, and configured / deployed in or used by a capture device as described herein to select a particular set of statically optimized reverse operation parameters to perform an image reshaping or reconstruction operation.

[0055] Thereafter, a downstream receiving device of the video signal may apply the optimized static inverse TPB coefficients to some or all ISP SDR images decoded from the video signal (e.g., a continuous or sequential ISP SDR image sequence, etc.) to generate or reconstruct a reshaped HDR image.

[0056] The ISP SDR image encoded in the video signal may or may not be identical to the desired SDR appearance as represented by the reference image. The operating parameters or settings in the programmable ISP can be optimized to approximate the reference image. Figure 1B In the "WB" operation scenario of , the SDR image used for reverse reshaping into the reshaped HDR image is given or fixed as the ISP SDR image. By comparison, in Figure 1A In the WFB operation scenarios of FIG. 1 , the SDR images used for reverse reshaping into reshaped HDR images are forward reshaped SDR images that can be optimized by applying the optimized forward reshaping operation parameters generated in the joint forward and reverse TPB optimization process that generates both optimized forward and reverse reshaping parameters. In other words, TPB optimization is not used in the forward path to generate ISP SDR images for improving or enhancing both the desired appearance of the SDR images encoded in the video signal and the desired appearance of the reshaped HDR images. Relatively large deviations from the desired appearance may occur in these operation scenarios.

[0057] Figure 1C A third example reshaping optimization process implementing a black-box reverse-only TPB optimization design / solution (referred to as "BB1") with a (single) video capture device is illustrated. The black-box reverse-only TPB optimization design / solution can be implemented for the following operation scenarios: generating SDR images from HDR images without using white-box conversion operations, and only the reverse (reshaping) path is subjected to TPB optimization.

[0058] The reshaping operation parameters in these operation scenarios can be generated using training SDR images and training HDR images generated or acquired by the same video capture device, such as the same mobile device, etc. These training SDR images and training HDR images form a plurality of SDR and HDR image pairs, each of which includes a training SDR image and a training HDR image corresponding to the training SDR image.

[0059] The training SDR image and the training HDR image in the same SDR and HDR image pair may be acquired at different moments / time points (e.g., a few milliseconds apart, a few fractions of a second apart, a few seconds apart, etc.) using the same capture device using the same ISP. The training SDR image and the training HDR image may not be precisely spatially aligned or temporally aligned with each other because it is difficult, if not impossible, to maintain the same shooting position and process the training SDR image and the training HDR image in the same manner at different moments / time points. For example, the training SDR image may be locally tone mapped or locally enhanced, while the training HDR image may be generated or obtained from multiple camera exposures. Therefore, in the "BB1" operating scenario, the relationship between the pixel or codeword value in the training SDR image and the corresponding pixel or codeword value in the training HDR image in the same image pair may be considered or assumed to be a black box.

[0060] The training SDR image and the training HDR image in each of the image pairs may first be spatially aligned. The spatially aligned training SDR image and the training HDR image in the image pair may be used to determine or find matching color pairs. These matching color pairs may then be used to generate optimized reshaping operation parameters such as optimized TPB and / or non-TPB coefficients for reshaping or mapping back the (e.g., non-training, reference, etc.) SDR image to an inversely reshaped or reconstructed HDR image that approximates the (e.g., non-training, reference, etc.) HDR image.

[0061] like Figure 1C As shown in , an upstream device such as a capture device may generate an SDR image such as a reference SDR image to be included / encoded in a video signal or a base layer (BL) thereof output by the upstream device. Example reference SDR images as described herein may include, but are not necessarily limited to, SDR images generated by a programmable image signal processor of an upstream device or a capture device operating in conjunction with the upstream device.

[0062] A downstream receiving device or video decoder of the video signal can decode the reference SDR image from the video signal. The decoded reference SDR image at the decoder side may be the same as the reference SDR image at the encoder side, but errors may be introduced in compression / decompression, encoding operations and / or data transmission.

[0063] In a reverse reshaping path as implemented by a downstream device, the reference SDR image may be reverse reshaped into a (reverse) reshaped HDR image based at least in part on optimized reverse reshaping operation parameters generated from a black-box-only reverse TPB optimization solution (denoted as "Reverse TPB Optimization"). The reshaped HDR image generated with the optimized reverse reshaping operation parameters in the output HDR color space represents an approximate or reconstructed version of the reference HDR image that may or may be generated by the same capture device.

[0064] The black-box inverse-only TPB optimization design / solution can generate optimized inverse reshaping operation parameters to cover as wide as possible output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0065] The optimized inverse reshaping operation parameters, such as optimized inverse TPB coefficients, may be implemented or represented in a three-dimensional lookup table or 3D-LUT to reduce processing time in the reshaping operation.

[0066] In some operational scenarios, the inverse reshaping operational parameters generated from the black-box inverse-only TPB optimization design / solution can be applied in the SLiDM framework. Under this static framework, there is no need to obtain (e.g., dynamically, etc.) inverse operational parameters that optimize image-specific or image-dependent optimizations, such as image-specific or image-dependent (or content-dependent) inverse TPB coefficients.

[0067] Instead, under the static SLiDM framework, the same or statically optimized inverse operation parameters, such as the same optimized inverse TPB coefficients, can be obtained or generated once (e.g., offline or before performing a reshaping operation on any of the reshaped SDR images) for inversely reshaping spatially aligned training SDR images generated by an upstream capture device into reshaped or reconstructed HDR images that are approximately the same as the spatially aligned training HDR images generated by the same capture device.

[0068] Thereafter, a downstream receiving device of the video signal may apply the optimized static inverse TPB coefficients to some or all (e.g., non-training, etc.) reference SDR images decoded from the video signal (e.g., a continuous or sequential reference SDR image sequence, etc.) to generate or reconstruct a reshaped HDR image that is approximate to a reference HDR image that may or may be generated from the same upstream capture device that generated or captured the reference SDR images.

[0069] Since TPB optimization is used only in the reverse path, in the "BB1" operation scenario, the resulting appearance of the reshaped HDR image generated from the reverse reshaping of the reference SDR image may have a relatively large deviation from the desired appearance of the reference HDR image.

[0070] In some "BB1" operation scenarios, such as Figure 1D As illustrated in , a camera raw image generated from a camera ISP may be encoded directly in a video signal output by an upstream (capture) device, rather than a reference SDR image generated from a programmable image signal processor. During a training phase, a training camera raw image, which may or may not be an SDR image, may be spatially and / or temporally aligned with a training HDR image in the same image pair of a camera raw and HDR image. Inverse reshaping operation parameters, such as inverse TPB coefficients, may be optimized using matching color pairs from the spatially aligned image pairs of camera raw and HDR images. During a non-training or deployment phase, a downstream device encoding a video signal with a non-training camera raw image may use the optimized inverse reshaping operation parameters to inversely reshape a decoded non-training camera raw image from a video signal into an inversely reshaped or reconstructed (non-training) HDR image that is approximate to a reference HDR image that may or may be generated by the same upstream (capture) device.

[0071] Figure 1E A fourth example reshaping optimization process implementing a black-box reverse-only reshaping optimization design / solution (referred to as "BB2") with a different video capture device is illustrated. The "BB2" reshaping optimization design / solution may be implemented for an operating scenario in which an SDR image is generated by a first capture device, and an HDR image to be generated from the SDR image is generated by a second, different capture device (e.g., intended to be, etc.), and only the reverse (reshaping) path is subjected to reshaping optimization. Additionally, the reshaping optimization in the "BB2" operating scenario may or may not be a TPB optimization.

[0072] The reshaping operation parameters in the "BB2" operation scenario can be generated using training SDR images and training HDR images generated or acquired by two video capture devices (such as two mobile devices of different brands and / or models). These training SDR images and training HDR images form a plurality of SDR and HDR image pairs, each of which includes a training SDR image and a training HDR image corresponding to the training SDR image.

[0073] The training SDR images and training HDR images in the same SDR and HDR image pair may be acquired using the same image, such as the same checkerboard image, rendered on the same or similar image display. In the "BB1" operating scenario, the relationship between the pixel or codeword values ​​in the training SDR image and the corresponding pixel or codeword values ​​in the training HDR image in the same image pair may be considered or assumed to be a black box.

[0074] The training SDR image and the training HDR image in each of the image pairs may first be spatially aligned. The spatially aligned training SDR image and the training HDR image in the image pair may be used to determine or find matching color patches that are rendered on the same image display or the same image display type together with a test image such as a checkerboard image. These matching color patches may then be used to construct a three-dimensional lookup table (3D-LUT) to map the SDR pixel or codeword values ​​of the color patches in the test image to the corresponding HDR pixel or codeword values ​​of the same color patches.

[0075] A 3D-LUT derived partially or entirely based on a test image may be used to generate optimized static or dynamic reshaping operation parameters for reshaping or mapping back a (e.g., non-training, reference, etc.) SDR image to an inversely reshaped or reconstructed HDR image that approximates a (e.g., non-training, reference, etc.) HDR image.

[0076] like Figure 1E As shown in , an upstream device such as a first capture device may generate an SDR image such as a reference SDR image to be included / encoded in a video signal or a base layer (BL) thereof output by the upstream device. Example reference SDR images as described herein may include, but are not necessarily limited to, an SDR image generated from a programmable image signal processor of a first capture device or an upstream device.

[0077] A downstream receiving device or video decoder of the video signal may decode the reference SDR image acquired with the first capture device from the video signal. The decoded reference SDR image at the decoder side may be identical to the reference SDR image at the encoder side, but errors may be introduced during compression, decompression, encoding operations and / or data transmission.

[0078] In the reverse reshaping path as implemented by a downstream device, the reference SDR image may be reverse reshaped into a (reverse) reshaped HDR image based at least in part on optimized reverse reshaping operation parameters generated from the "BB2" reshaping optimization design / solution (denoted as "dynamic reverse function optimization"). The reshaped HDR image generated with the optimized reverse reshaping operation parameters in the output HDR color space represents an approximate or reconstructed version of the reference HDR image that may or may be generated by a second, different capture device.

[0079] The "BB2" reshaping optimization design / solution can generate optimized inverse reshaping operation parameters to cover as wide as possible output HDR color space, in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

[0080] In some operational scenarios, the inverse remodeling operational parameters generated from the “BB2” remodeling optimization design / solution can be applied in a static or dynamic SLiDM framework.

[0081] Since only the "BB2" reshaping optimization is used in the reverse path, the resulting appearance of the reshaped HDR image generated from the reverse reshaping of the reference SDR image may have a relatively large deviation from the desired appearance of the reference HDR image. The first capture device and the second capture device may be mobile devices that may operate in different ways (e.g., with different exposure times, different video processing operations, different mapping functions / relationships, etc.) in their respective video capture applications.

[0082] In a dynamic SLiDM frame, a dynamic mapping using image-dependent or image-specific reshaping operation parameters can be used to reversely reshape an SDR image of a first capture device into an HDR image that can or can be generated by a second capture device, particularly in an operating scenario where the first capture device and the second capture device operate with relatively large differences between the corresponding video / image capture applications running on the first capture device and the second capture device and / or between the corresponding ISPs used in the first capture device and the second capture device. For example, in a training phase, multiple sets of training SDR images and training HDR images or image pairs can be used to obtain multiple sets of optimized reshaping operation parameters for multiple sets of training SDR images and training HDR images or image pairs, respectively. In a deployment or application phase, specific image characteristics, such as the overall brightness of some or all regions of a specific image, can be dynamically evaluated when processing a specific image. Based on specific image characteristics of a specific image that are related to or compared to corresponding image characteristics of multiple sets of training SDR images and training HDR images or image pairs, a specific set of optimized reshaping operation parameters can be adaptively and / or dynamically selected for a specific image from multiple sets of optimized reshaping operation parameters.

[0083] White-box joint TPB forward and backward optimization (WFB)

[0084] Multiple design factors of reshaping optimization can be considered to help increase the coverage of the supported color space in which the reshaped image is to be represented. First, a tensor product bi-spline (TPB) predictor with optimized TPB coefficients generated from the reshaping optimization can be used to obtain or achieve relatively high prediction accuracy compared to other types of predictors including but not limited to MMR predictors. The multi-knot and continuity characteristics of the tensor product bi-spline predictor or prediction function can be utilized or used to cover a relatively wide color range or color space portion with relatively high accuracy. An example TPB predictor can be found in U.S. Provisional Patent Application Serial No. 62 / 908,770 filed by Guan-Ming Su, Harshad Kadu, Qing Song, Qing Song and Neeraj J. Gadgil on October 1, 2019, "Tensor-product B-spline predictor", the contents of which are incorporated herein by reference as fully set forth herein.

[0085] In contrast, although a multi-segment MMR predictor or prediction function may be better than a single-segment MMR predictor or prediction function, discontinuities between different MMR segments may be introduced into the multi-segment MMR predictor or prediction function, thereby adversely affecting and even prohibiting the use of the multi-segment MMR predictor or prediction function in many operating scenarios. Example MMR-based operations are described in U.S. Patent 8,811,490, which is incorporated herein by reference in its entirety as if fully set forth herein.

[0086] The TPB predictor can be advantageously used in an SLBC-based video codec. More specifically, the forward TPB predictor can be used to generate a forward reshaped SDR image that is approximate to a reference SDR image, while the reverse TPB predictor can be used to generate a reverse reshaped or reconstructed HDR image that is approximate to a reference HDR image. An example use of the TPB predictor with an SLBC-based video codec can be found in U.S. Provisional Patent Application Serial No. 63 / 255,057, filed by H. Kadu et al. on October 13, 2021, "Tensor-product B-spline prediction for HDR video in mobile applications," the contents of which are incorporated herein by reference as fully set forth herein. In addition, a BESA (Backward Error Subtraction for Signal Adjustment) algorithm / method modified to support neutral color preservation can be used in a chained reshaping function pipeline to optimize the forward and reverse TPB predictors together to achieve relatively high reversibility or relatively accurate approximation of a reference HDR image through a reverse reshaped or reconstructed HDR image. Example BESA algorithms / methods can be found in U.S. Provisional Patent Application Serial No. 63 / 013,063, filed on April 21, 2020 by GM.Su, “Reshaping functions for HDR imaging with continuity and reversibility constraints,” U.S. Provisional Patent Application Serial No. 63 / 013,807, filed on April 22, 2020 by GM.Su and H.Kadu, “Iterative optimization of reshaping functions in single-layer HDR image codec,” and PCT Application Serial No. PCT / US2021 / 028475, filed on April 21, 2021, the contents of which are incorporated herein by reference in their entirety as if fully set forth herein.

[0087] Although the TPB predictor has better prediction accuracy than other types of predictors, transmitting TPB coefficients in a video signal may incur relatively large signal or bit rate overhead. In addition, constructing the TPB equations or basis functions used in the TPB predictor may incur relatively large computational costs.

[0088] In some operating scenarios, in order to avoid or reduce signal or bit rate overhead and computational cost, some or all images in the whole video sequence can be reshaped using a built-in or static TPB predictor without changing the TPB coefficients used in the static TPB predictor during the playback of the video sequence. More specifically, some or all (built-in or static) TPB coefficients can be cached or stored in video applications such as video capture / editing applications, without the need to explicitly transmit these TPB coefficients by encoding a video signal or bit stream with an image to be reshaped by a built-in or static TPB predictor or in the video signal or bit stream. In an example, some or all TPB coefficients can be preloaded or preconfigured in a downstream receiving device before a downstream receiving device receives and processes a video signal or bit stream. Additionally, optionally or alternatively, multiple groups of TPB coefficients can be preloaded or preconfigured in a downstream receiving device before a downstream receiving device receives and processes a video signal or bit stream. For example, based on a simple indicator (e.g., a binary indicator, a multi-bit indicator, etc.) signaled or transmitted in a video signal or bitstream, a video application can simply pick or select a particular set of TPB coefficients from multiple sets of TPB coefficients to use for a static TPB predictor. An example static TPB predictor or prediction function can be found in the aforementioned U.S. Provisional Patent Application Serial No. 63 / 255,057.

[0089] In some operating scenarios, a mobile device can be used to host or run video capture and / or editing applications. Users of mobile devices can edit captured images in these video capture and / or editing applications. For example, edited images such as edited HDR images that may be intended to be displayed or viewed on an image display of a non-mobile device with a display capability higher than or larger than the image display of the mobile device may have a brightness range and / or color range or color distribution that is higher, larger, wider and / or wider than the (original) captured images such as HDR images originally captured from the camera sensor of the mobile device. For example, the captured HDR image may typically be limited by the camera sensor and ISP output of the mobile device to be represented in a relatively narrow color space or color gamut such as P3, while the edited HDR image can be represented in a relatively wide color space or color gamut such as the entire R.2020 color space.

[0090] In order to design a static mapping for a reshaping operation, it is highly desirable to optimize the forward / backward TPB coefficients to cover as high a dynamic range as possible and as wide a color space or gamut as possible while achieving as much bitrate and computational efficiency as possible.

[0091] Full HDR reversibility between a reconstructed HDR image and a reference HDR image that is identical to the reconstructed HDR will require distinguishing colors in the original or reconstructed HDR domain (or color space) that are to be distinguished or distinguishable in the reshaped SDR domain (or color space) so that the distinguished colors in the SDR domain can be mapped back to distinguishable colors in the HDR domain. Therefore, full reversibility will likely require a one-to-one mapping from HDR to SDR and from SDR back to HDR. The wider the supported HDR domain or color space, the more codewords the SDR domain or color space needs to have. Given that the total number of available SDR pixel or codeword values ​​in the SDR domain (e.g., corresponding to a lower bit depth than the HDR domain, etc.) is typically less than the total number of required HDR pixel or codeword values ​​in the HDR domain, full HDR reversibility may not be achieved in some operating scenarios. In fact, some captured or edited images may contain codewords formed as in Figure 2A , which are shown as diffuse and / or specular color combinations of irregularly shaped color distributions that exceed the R.2020 color space (“BT.2020”) and are therefore not fully representable by that color space, let alone smaller color spaces or SDR color spaces such as the R.709 color space (“BT.709”) or the (DCI) P3 color space.

[0092] In many operating scenarios, a nonlinear reshaping mapping or function can be used to relatively efficiently distribute the available pixel or codeword values ​​and generate a (forward) reshaped SDR that approximates the reference SDR image as closely as possible and helps the reconstructed HDR image approximate the reference HDR image as closely as possible, although the reshaped SDR generated by the nonlinear reshaping mapping or function may be different from the reference SDR image, but may contain some deviations from the reference SDR image.

[0093] Additionally, optionally, or alternatively, in some operating scenarios, in order to maintain the same or similar appearance of the reference SDR and HDR images in the reshaped SDR and HDR images, reshaping optimizations as described herein may be performed under neutral color constraints that preserve neutral colors in the color mapping performed with the reshaping operations.

[0094] Given that a TPB predictor or function has the ability to approximate functions with relatively high nonlinearity, the TPB predictor or function may be incorporated into the reshaping operation to support or enable a relatively wide HDR color space for representing the inversely reshaped or reconstructed HDR image.

[0095] A potential disadvantage is that the TPB predictor may require a relatively large number of TPB coefficients to represent or approximate the non-linear SDR to HDR and / or HDR to SDR mapping functions. To prevent possible ambiguous conditions, numerical instabilities, slow convergence issues, etc. that may arise when solving the optimization problem of the TPB coefficients, static TPB predictors may be pre-generated and deployed with video codecs (such as those used by upstream devices and / or downstream devices) and then used to process video sequences such as captured and / or edited video sequences.

[0096] For illustrative purposes only, Figure 1A In the operating scenario shown in , the (forward reshaped) SDR domain or color space may be that of R.709, and the (original or reverse reshaped) HDR domain or color space may be that of R.2020. Figure 2A As illustrated in , the R.2020 color space is much larger than the R.709 color space. Therefore, while it may be difficult to map the entire R.2020 to R.709 and then map R.709 back to R.2020, reshaping optimization techniques as described herein can be used to optimize the coverage of the HDR color space used to represent the reverse reshaped or reconstructed HDR image and maintain the same or similar appearance of the reference SDR and HDR images in the reshaped SDR and HDR images. These techniques can be used to optimize reshaping operation parameters such as TPB coefficients for both forward and reverse reshaping operations.

[0097] While it may not be possible to cover the entire R.2020 color space in all scenarios, a subset or subspace in the R.2020 color space (e.g., a specific color gamut outlined as a triangle formed by color primaries in a color coordinate system, etc.) can be selected or picked through a reshaping optimization operation to maintain HDR reversibility of the reference HDR image and an acceptable SDR approximation of the reference SDR image in the reshaped HDR and SDR images.

[0098] By way of illustration and not limitation, a subset or subspace in an R.2020 color space (or an HDR color space used to represent a reshaped HDR image) may be defined or characterized by a specific white point and three specific chromatic algebras (red, green, blue). The specific white point may be selected or fixed to be a D65 white point. There may be a great deal of design freedom to select three specific chromatic algebras from many possible chromatic algebra combinations for a subset or subspace in R.2020 to be supported by the reshaping operation. The reshaping optimization techniques as described herein may be used to determine or select specific chromatic algebras for a subset or subspace in R.2020 as optimized chromatic algebras for achieving or attaining maximum perceived quality and / or color coding efficiency and / or reversibility between SDR images and HDR images.

[0099] The MacAdam ellipses represent only noticeable color differences, since the HVS may not be able to distinguish color differences within the same ellipse. The color gamut of the pointer may represent all diffuse colors that the HVS can perceive. In the MacAdam ellipses and the color gamut of the pointer, green is less important or less distinguishable / perceivable by the HVS than red and blue. Therefore, in some operating scenarios, a specific optimized color source for a subset or subspace in R.2020 used to represent the reshaped or reconstructed HDR image may be selected to cover more red and blue than green, especially when the SDR to HDR and HDR to SDR mappings cannot support a one-to-one mapping relationship in the reshaping operation.

[0100] As used herein, a color source may also be referred to as a primary color and may be used to define the angles of a polygon such as a triangle that represents a specific color space or color gamut (such as a color space specified by a standard, a color space supported by a display, a color space supported by a video signal, etc.). For example, a standard color space with a standard specified white point may be represented by a triangle whose angles are specified by three standard specified color sources in a CIExy color space coordinate system or a CIExy chromaticity diagram. The CIExy coordinates of the color sources (red or R, green or G, blue or B) and white points that define R.709, P3, and R.2020 color spaces, respectively, are specified in Table 1 below.

[0101] Table 1

[0102]

[0103] Each color source and the CIExy coordinates of the corresponding white point in the standard color space in Table 1 can be expressed as and Among them, (c) is the standard color space.

[0104] Therefore, the P3 color space can be characterized by the following CIExy coordinates of the P3 color primaries and the P3 white point: Similarly, the R.2020 color space can be characterized by the following CIExy coordinates of the R.2020 color primaries and the R.2020 white point:

[0106] The inversely reshaped HDR color space used to represent the inversely reshaped or reconstructed HDR image is denoted as the (a) color space. Therefore, the CIExy coordinates of the color primaries and white point defining the (a) color space can be expressed as and

[0107] As mentioned above, green is less important or less distinguishable / perceivable by the HVS than red and blue. In some operating scenarios, the red and blue chromatic algebra of the (a) color space may be selected to match the red and blue chromatic algebra of the R.2020 color space as follows:

[0108]

[0109] Additionally, like all R.709, P3, and R.2020 color spaces that are specified with a D65 white point, the (a) color space can also be specified with a D65 white point, as follows:

[0110]

[0111] (a) The green color source of the color space can be the green color source along the P3 color space in the CIExy coordinate system or the chromaticity diagram Green color source with R.2020 color space Any point along the line between the two green primaries in the P3 color space and the R.2020 color space can be represented as a linear combination of the two green primaries with a weight factor a, as follows:

[0112]

[0113] Therefore, the optimization problem of finding the maximum support for the R.2020 color space from the (a) color space can be simplified to the problem of selecting the weight factor a. When a = 0, the (a) color space becomes the entire R.2020 color space. When a = 1, as Figure 2B As illustrated in (where (a) the color space is represented as "TPB" or "colors currently covered by TPB"), the red and blue corners or red and blue color primaries of the (a) color space are the same as the red and blue corners or red and blue color primaries of the R.2020 color space, and the green corner or green color primaries of the (a) color space are the same as the green corner or green color primaries in the P3 color space. FIG. 2C to FIG. 2E Three example (a) color spaces are illustrated, where a=0.9, 0.5, and 0.25, respectively.

[0114] Joint color space and TPB optimization

[0115] Since the range or coverage of the inverse reshaped (a) color space on the R.2020 color space can be controlled by parameter a, the overall TPB (reshaping-based) optimization problem becomes how to optimize the TPB coefficients in both the forward path and the backward path so that (i) the forward reshaped SDR domain (or color space) used to represent the reshaped SDR image is close to the reference SDR domain (or color space) used to represent the reference SDR image, especially in the neutral colors or color space parts that are more sensitive to HVS than the non-neutral colors or color space parts, and (ii) the inverse reshaped or reconstructed HDR domain (or color space) used to represent the inverse reshaped or reconstructed HDR image is as closely identical as possible to the reference HDR domain (or color space) used to represent the reference HDR image. Ideally, the inverse reshaped or reconstructed HDR image is identical to the reference HDR image if perfect reconstruction or complete reversibility can be achieved.

[0116] like FIG. 2B to FIG. 2E As illustrated in , in order to make the inversely reshaped or reconstructed HDR domain or color space cover as large an R.2020 color space as possible, the TPB optimization problem boils down to finding or searching for the minimum possible value of parameter a so that the above SDR and HDR quality conditions are met.

[0117] Figure 3A An example process flow for finding the minimum possible value of parameter a for maximizing SDR and HDR quality in reshaped SDR and HDR images is illustrated. Figure 3A The process flow of may iterate through multiple candidate values ​​of parameter a in an iteration order that may be a sequential order or a non-sequential order.

[0118] Block 302 includes selecting a current (e.g., to be iterated, etc.) value of parameter a to be the next candidate value (e.g., initially the first candidate value, etc.) among a plurality of candidate values ​​for parameter a. Given a candidate inversely reshaped HDR color space having a current value of parameter a, it is possible to Figure 3A Optimized reshaping operation parameters such as optimized TPB coefficient are obtained in one or more subsequent process flow boxes.

[0119] Block 304 includes constructing sample points or preparing two sample data sets in a candidate inversely reshaped HDR color space. By way of example and not limitation, the candidate inversely reshaped HDR color space may be a hybrid log-gamma (HLG) RGB color space (referred to as "(a) RGB color space HLG").

[0120] The first of the two sampling datasets is a uniformly sampled color patch dataset. Each color patch in the uniformly sampled color patch dataset consists of a corresponding RGB color uniformly sampled from (a) the RGB color space HLG (denoted as ) characterizes or represents that the (a) RGB color space HLG includes three dimensions represented by the R axis, the G axis, and the B axis, which are respectively represented as R, G, and B component colors. Each axis or dimension of the (a) RGB color space HLG is normalized to a value range of [0,1] and is respectively represented by N R 、N G and N B Here, N R 、N G and N B Each of represents a positive integer greater than one (1). Therefore, the total number of color patches or sampled data points in the uniformly sampled color patch data set is given by:

[0121] N u =N R N G N B (4)

[0122] Each uniformly sampled data point or RGB color in the uniformly sampled color patch dataset is given as follows

[0123] where i∈[0,...,N R -1], j∈[0,...,N G -1],k∈[0,...,N B -1]

[0124] For simplicity, (i, j, k) in the above expression (5) can be vectorized or simply represented as p. Accordingly, the uniformly sampled data points or RGB colors It can be simply expressed as All N u The nodes (each of which represents a unique combination of a specific R-axis cell / partition, a specific B-axis cell / partition, and a specific G-axis cell / partition) can be grouped or collected into a vector / matrix as follows:

[0125]

[0126] The second of the two sample data sets prepared or constructed in block 304 is a neutral color data set. The second data set includes a plurality of neutral colors or neutral color patches (also referred to as gray or gray color patches).

[0127] The second data set can be used to retain input gray color patches in the input domain as output gray color patches in the output domain when the input gray color patches in the input domain are mapped to or reshaped into output gray color patches in a reshaping operation as described herein. Increased weighting can be given to input gray color patches in the input domain (or input color space) in the optimization problem compared to other color patches to reduce the likelihood that these input gray color patches are mapped to non-gray color patches in the output domain (or output color space) by the reshaping operation.

[0128] N can be generated by uniformly sampling the R, G, and B values ​​along a line connecting a first gray (0,0,0) and a second gray (1,1,1) in an RGB domain (eg, (a) RGB color space HLG, etc.). n Nodes or gray blocks are used to prepare or construct a second data set (gray data set or gray data set), as follows:

[0129]

[0130] where i∈[0,...,N n -1]

[0131] All N in the second dataset n The nodes can be grouped or collected into a neutral color vector / matrix as follows:

[0132]

[0133] The neutral color vector / matrix in the above expression (8) can be repeated N times. t (a positive integer not less than (1)) times to generate the N in the (now repeated) second data set n N t A neutral color block, as follows:

[0134]

[0135] Repeating the neutral colors in the repeated second neutral color data set increases the weight of the neutral colors or grays compared to other colors. Therefore, the neutral colors can be retained more in the optimization problem than other colors.

[0136] The first (all sampled) color data set and the second neutral color (repeated) data set in expressions (6) and (9) can be collected or placed together in a single combined vector / matrix as follows:

[0137]

[0138] Combine vectors / matrices The total number of vector / matrix elements (repeating and non-repeating patches) in is N=N n N t +N u In the above expression (10), Each vector / matrix element or color patch (row) in can be represented as follows:

[0139]

[0140] Block 306 includes converting the combined vector / matrix in (a) RGB color space (or (a) RGB color space HLG) The color values ​​of the color blocks (rows) represented in the vector / matrix elements are converted to the corresponding color values ​​in the standard R.2020 color space or R.2020RGB color space HLG.

[0141] The three primary colors of red, green and blue in the (a) RGB color space HLG corresponding to the three corners or points of the triangle defining the (a) RGB color space HLG in the CIExy chromaticity diagram can be converted from CIE xy values ​​to CIE XYZ values ​​via the following equations:

[0142] Y=1 (12-1)

[0143]

[0144] In the given In the case of red, green, and blue primaries as (x, y) or CIE xy values, the same red, green, and blue primaries as (X, Y, Z) or CIEXYZ values ​​can be obtained using the conversion equations in the above expression (12), which are respectively expressed as Similarly, given In the case of a white point as (x,y) or CIE xy values, the same conversion equation in equation (12) above can be used to obtain the same white point as (X,Y,Z) or CIE XYZ values, expressed as

[0145] Represented as P (a)→XYZ The 3x3 transformation matrix can be constructed as the combined vector / matrix in The color values ​​of the color blocks (rows) represented by the vector / matrix elements of the (x,y) or CIE xy values ​​are converted to the corresponding (X,Y,Z) or CIE XYZ values ​​as follows

[0146]

[0147] in, represents the element-wise multiplication between the two matrices on the right side (RHS) in Expression (13); and the two matrices on the RHS in Expression (13) can be constructed as follows:

[0148]

[0149] in,

[0150]

[0151] Represented as P (a)→R2020 The overall 3×3 transformation matrix can be constructed as a combined vector / matrix that puts HLG in aRGB color space The color values ​​of the color blocks (rows) represented in the vector / matrix elements are converted to the corresponding color values ​​in the R.2020RGB color space HLG as follows:

[0152] P (a)→R2020 =P (a)→XYZ P XYZ→R2020 (17)

[0153] Among them, P XYZ→R2020 Represents a 3×3 conversion matrix that converts XYZ color values ​​to corresponding values ​​in the R.2020 RGB color space HLG.

[0154] Thus, in block 306, the combined vector / matrix in aRGB color space (or (a)RGB color space HLG) may be The color values ​​of the color blocks (rows) represented in the vector / matrix elements are converted to the corresponding color values ​​in the standard R.2020 color space or R.2020RGB color space HLG as follows:

[0155]

[0156] Block 308 includes converting the vector / matrix into the R.2020RGB color space HLG The color values ​​of the color blocks (rows) represented in the vector / matrix elements of are converted to the corresponding color values ​​in the R.2020YCbCr color space HLG (expressed as ),as follows:

[0157]

[0158] in, Represents a conversion function or matrix that converts a (R.2020RGB) color value in the R.2020RGB color space HLG to a corresponding (HDR R.2020YCbCr) color value in the R.2020YCbCr color space HLG.

[0159] Each color block (row) in can be represented as follows:

[0160]

[0161] Block 310 includes converting the vector / matrix into the R.2020RGB color space HLG The color values ​​of the color blocks (rows) represented in the vector / matrix elements are converted or mapped to the corresponding color values ​​in the R.709SDR YCbCr color space (expressed as ),as follows:

[0162]

[0163]

[0164] Among them, f HLG→W_SDR () represents the HDR R.2020RGB color value in the R.2020RGB color space HLG Convert to the corresponding R.709SDR RGB color value in the R.709SDR RGB color space (expressed as )’s white-box HDR to SDR mapping function; and Represents then converting the R.709SDR RGB color value in the R.709SDR RGB color space Convert to the corresponding color value in R.709SDR YCbCr color space A conversion function or matrix. Non-limiting examples of white-box HDR to SDR mapping functions are described in BT.2390-10, “High dynamic range television for production and international programme exchange” (November 2021), which is incorporated herein by reference in its entirety. Other examples of white-box HDR to SDR mapping functions may include, but are not necessarily limited to, any of the following: a standard-based HDR to SDR conversion function, a proprietary HDR to SDR conversion function, an HDR to SDR conversion function with a linear and / or non-linear domain, an HDR to SDR mapping function using a gamma function, a power function, etc.

[0165] Each color block (row) in can be represented as follows:

[0166]

[0167] Box 312 includes converting the vector / matrix in the R.2020RGB color space HLG The color value of the color block (row) represented by the vector / matrix element is converted or mapped to the corresponding color value in the R.2020YCbCr color space PQ (expressed as ),as follows:

[0168]

[0169]

[0170] Among them, f HLG→PQ () indicates that the HDR R.2020RGB HLG color values ​​in the R.2020RGB color space HLG are converted, for example, using the transfer functions described in the aforementioned SMPTE 2084 and Rec.ITU-R BT.2100. Converted to the corresponding HDR R.2020RGB PQ color value in the R.2020HDR RGB color space PQ (expressed as )’s HLG to PQ conversion function; and The HDR R.2020 RGB PQ color values ​​in the R.2020 HDR RGB color space PQ are then converted, for example, using the transfer functions described in the aforementioned SMPTE 2084 and Rec.ITU-R BT.2100 Convert to the corresponding color value in R.2020YCbCr color space PQ The transfer function or matrix of .

[0171] Each color block (row) in can be represented as follows:

[0172]

[0173] Block 314 includes using and Forward and reverse reshaping operation parameters, such as forward and reverse TPB coefficients, may be obtained or generated as input to a (TPB) BESA algorithm that may be enhanced or modified with neutral color preservation.

[0174] The BESA algorithm is an iterative algorithm, where each current iteration may modify the reference SDR signal based on the inverse prediction error measured or determined in the previous iteration.

[0175] The modified BESA algorithm as described herein can implement neutral color preservation and avoid modifying neutral color patches. To this end, a set of neutral color patch indices that identify or correspond to neutral color patches can be generated as follows:

[0176]

[0177] Wherein, δ represents a neutral color value threshold value that determines or specifies a maximum color deviation range in absolute value, within which the color value meets the condition of neutral color. Example values ​​of the neutral color value threshold value δ may include, but are not necessarily limited to, any of the following items: 1 / 2048, 1 / 1024, etc.

[0178] Let ch denote the channel in the forward reshaped SDR domain or color space (e.g., three channels of , etc.). Let F denote the forward reshaping or forward path. Let B denote the reverse reshaping or reverse path.

[0179] Expressed as The per-channel forward generation matrix (also called the design matrix) can be used to generate the input HDR color patch from the corresponding input HDR codeword represented by expression (19) Predict the forward reshaped SDR color patch as represented by the corresponding forward reshaped SDR codeword. The forward generator matrix can be derived from the cross-channel forward TPB basis function to is generated or precomputed, stored or cached in computer memory, and fixed in all iterations for the forward path as follows:

[0180]

[0181] Among them, the cross-channel forward TPB basis function to Adopting or accepting the above expression (20) The color blocks (or rows) in are taken as input parameters.

[0182] The forward reshaped SDR color block can be predicted by multiplying the forward generation matrix in expression (28) with the forward TPB coefficients. Example predicted reshaped codewords with TPB coefficients (which are equivalent or similar to the color blocks herein) are described in the aforementioned U.S. Provisional Patent Application Serial No. 62 / 908,770.

[0183] The forward reshaped SDR color blocks or codewords may be reversely reshaped into reverse reshaped HDR color blocks or codewords, for example, by a TPB-based reverse reshaping operation.

[0184] In the reverse path, at each iteration (e.g., kth iteration, etc.) in the BESA algorithm, it is expressed as The per-channel reverse generation matrix of can be used to predict the reverse reshaped HDR color patches as represented by the corresponding reverse reshaped HDR codewords from the forward reshaped SDR color patches as represented by the corresponding forward reshaped SDR codewords obtained in the forward path. The reverse generation matrix for each iteration can be obtained from the cross-channel forward TPB basis function to is generated, and not fixed across all iterations for the reverse path, as follows:

[0185]

[0186] in, to Indicates the forward reshaped SDR color block or codeword.

[0187] The inverse reshaped HDR color block can be predicted by multiplying the inverse generation matrix in expression (29) with the inverse TPB coefficients.

[0188] The reverse prediction error for each iteration in the BESA algorithm can be determined by comparing the reverse reshaped HDR patches or codewords collected in the vector / matrix with the reference HDR patches or codewords in the per-channel reverse observation vector / matrix obtained from expression (25). The per-channel reverse observation vector / matrix can be generated or pre-computed from expression (25), stored or cached in computer memory, and fixed in all iterations as follows:

[0189]

[0190] The reference SDR signal or the reference SDR color patch or codeword can be used as a prediction target for the forward reconstruction operation to generate the forward reshaped SDR color patch or codeword into an approximate reference SDR color patch or codeword. (where k represents an iteration index starting from zero (0) as the first iteration in the BESA algorithm) may be initialized to the SDR color patch or codeword obtained using the above expression (22) for the first iteration (k=0), and thereafter may be modified at the end of each iteration in part or in whole based on the prediction error determined for that iteration. The modified reference SDR color patch or codeword may be used as a prediction target for a forward reshaping operation in the next iteration in the BESA algorithm to generate the forward reshaped SDR color patch or codeword as an approximate modified reference SDR color patch or codeword.

[0191] The per-channel reference SDR color patch or codeword can be used to construct the vector / matrix as follows at the kth iteration

[0192]

[0193] Before the first iteration, the vector in the above expression (31) is set to the original reference SDR signal represented by expressions (22) and (23), as follows:

[0194]

[0195] At the kth iteration, the forward TPB coefficients (expressed as ) that minimizes the difference between the forward reshaped SDR patch or codeword and the reference SDR patch or codeword determined for that iteration, as follows:

[0196]

[0197] The predicted SDR color patch or codeword for channel ch at the kth iteration may be calculated from the optimized values ​​of the (e.g., per-channel, etc.) forward TPB coefficients as follows:

[0198]

[0199] At the kth iteration, the inverse TPB coefficients (expressed as) can be generated (eg, per channel, etc.) via the least squares solution of the optimization problem. ) that minimizes the difference between the inversely reshaped HDR patch or codeword and the reference HDR patch or codeword (which is fixed for all iterations in the BESA algorithm), as follows:

[0200]

[0201] The predicted HDR color block or codeword for channel ch at the kth iteration may be calculated from the optimized values ​​of the (e.g., per-channel, etc.) inverse TPB coefficients as follows:

[0202]

[0203] The inverse prediction error is calculated and propagated back to the reference non-neutral SDR signal.

[0204] As described above, the reverse prediction error for each iteration in the BESA algorithm can be determined by comparing the reverse reshaped HDR color block or codeword with the reference HDR color block or codeword obtained from expression (25) (which is fixed for all iterations in the BESA algorithm).

[0205] Among these backward prediction errors, the backward prediction errors of non-neutral colors (or non-gray) can be propagated back to update or modify the reference SDR color patches or codewords of these non-neutral colors (or non-gray). The modified SDR color patches or codewords of non-neutral colors (or non-gray) can be combined with the (unmodified) reference SDR color patches or codewords of neutral colors (or gray) to be used as the prediction target of the forward path in the next or upcoming iteration in the BESA algorithm.

[0206] In some operational scenarios, the reverse prediction error may be calculated as the difference between the original or reference HDR signal (or a reference HDR color block / codeword therein) and the predicted HDR signal (or a reverse reshaped HDR color block / codeword therein) at the kth iteration as follows:

[0207]

[0208] In response to determining that the color block i is in the neutral color set Φ, the reference SDR codewords of the color block in the Cb and Cr channels of the reference SDR signal may be set to a gray value, such as 0.5, for the upcoming (k+1)th iteration as follows:

[0209]

[0210] Otherwise, in response to determining that the color block i is not in the neutral color set Φ, the reference SDR codewords of the color blocks in the Cb and Cr channels of the reference SDR signal may be set to updated or modified values ​​according to the reverse prediction error for the upcoming (k+1)th iteration as follows:

[0211]

[0212] Where ε represents the reverse prediction error threshold, above which the reverse prediction error can be propagated to update the color blocks of non-neutral colors; α ch represents a fixed or configurable learning rate for scaling the backward prediction error; λ represents the upper limit or maximum allowed value of the reference codeword for modifying non-neutral color patches. An example value of ε can be, but is not necessarily limited to, 0.000005. α ch An example value of may be, but is not necessarily limited to, 1. An example value of λ may be, but is not necessarily limited to, 2. (This implies no limitation, as the SDR and / or HDR codewords herein may be normalized to a normalized domain or range of 0 to 1).

[0213] Neutral color preservation as performed by expressions (38) to (40) can be used to ensure that neutral colors in the reference SDR signal do not deviate to or from non-neutral colors in all iterations of the BESA algorithm, thereby improving the perceived quality of grayscale related in the reshaped image.

[0214] The BESA algorithm may be iterated up to a total number of iterations. In some operating scenarios, the total number of iterations of the BESA algorithm may be specifically selected to balance color bias in the reshaped SDR and / or HDR images. For example, different total numbers of iterations in the BESA algorithm may generate forward and reverse TPB coefficients that produce reshaped SDR images of different SDR appearances and reshaped HDR images of different HDR appearances. A total number of iterations may be selected for the BESA algorithm to generate reshaped SDR and HDR images of relatively high quality appearance, such as ten times, fifteen times, etc.

[0215] The BESA algorithm with neutral color preservation may be performed for each candidate value of parameter a.For example, in block 314, the BESA algorithm with neutral color preservation may be performed for the current candidate value of parameter a.

[0216] Block 316 includes determining whether the current candidate value for parameter a is the last candidate value among the plurality of candidate values ​​for parameter a. If so, process flow proceeds to block 318. Otherwise, process flow returns to block 302.

[0217] Block 318 includes selecting an optimal or optimized value for parameter a, and calculating (or simply selecting already calculated) optimized forward and reverse TPB coefficients corresponding to the optimized value for parameter a for the (a) RGB color space.

[0218] Choosing the optimal value of parameter a can be formulated as finding The optimization problem of (a specific set of values ​​of) is such that (1) (a) the RGB color space represents the maximum HDR color space (e.g., for covering the largest part of the R.2020RGB color space, etc.) to achieve The HDR prediction error is minimized, and (2) the forward reshaped SDR color block or codeword has the expression The relatively small or acceptable SDR color deviations are as follows:

[0219]

[0220] Here, w is a weighting factor used to balance between HDR error and SDR error.

[0221] This optimization problem can be solved by using the corresponding optimal or optimized TPB coefficients for each aRGB color space. and The results from this color space are checked to solve. Among all possible or candidate a values, the smallest a value (corresponding to the maximum color space coverage in the R.2020 color space) can achieve the minimum HDR error and a relatively small SDR error. HDR prediction error and SDR color deviation It can be a subjective (e.g., with input from a human observer, etc.) or an objective distortion function (e.g., with no or little input from a human observer, etc.), such as those measured with an image dataset from a database. Example distortion functions may include, but are not necessarily limited to, any of the following: mean square error (or deviation) or MSE, root mean square error (or deviation) or RMSE, sum of absolute differences or SAD, peak signal-to-noise ratio or PSNR, structural similarity index (SSIM), etc.

[0222] Figure 2F and Figure 2G Example percentile Cb and Cr values ​​for different values ​​of parameter a in the forward reshaped SDR domain or color space are illustrated. Different values ​​of parameter a determine different color space coverage of the reverse reshaped HDR domain or color space in the R.2020 domain or color space, and thus affect the reconstruction or generation of the reverse reshaped HDR image in the R.2020 domain or color space in different ways, and result in different distortions of the reverse reshaped HDR image compared to a reference HDR image to which the reverse reshaped HDR image is to be approximated (e.g., as measured by a distortion metric expressed or measured).

[0223] Figure 2F The vertical axis is for Figure 2F Different values ​​of the parameter a value represented by the horizontal axis (e.g., obtained with the BESA algorithm with neutral color preservation in ten iterations, etc.) represent the 3 percentile values ​​in the Cb and Cr channels in the forward reshaped SDR domain. The lower the value of parameter a, the greater the coverage of the a color space in the R.2020 color space. The 3 percentile value can be used explicitly or implicitly to represent the minimum value in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space. The smaller the 3 percentile value, the smaller the minimum value, and therefore the larger the range of codeword values ​​in the forward reshaped SDR (YCbCr) domain or color space extends. As Figure 2F It can be shown in , that the lower the value of parameter a, the lower the 3rd percentile values ​​in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space.

[0224] Figure 2G The vertical axis is for Figure 2GDifferent values ​​of the parameter a value represented by the horizontal axis (e.g., obtained with the BESA algorithm with neutral color preservation in ten iterations, etc.) represent the 97th percentile values ​​in the Cb and Cr channels in the forward reshaped SDR domain. The 97th percentile value can be used explicitly or implicitly to represent the maximum value in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space. The larger the 97th percentile value, the larger the maximum value, and therefore the larger the range of codeword values ​​in the forward reshaped SDR (YCbCr) domain or color space. As Figure 2G It can be shown in that the 97th percentile values ​​in both the Cb and Cr channels of the forward reshaped SDR domain or color space remain almost constant values.

[0225] In some operational scenarios, in block 318, the image data set used to determine distortions in the reshaped SDR and HDR images and select an optimized value for parameter a based in part or in whole on these distortions may include a video bitstream acquired with one or more specific video capture devices such as mobile phones (e.g., supporting a video coding profile in SMPTE ST 2094, supporting Dolby Vision Profile 8.4, etc.). A non-limiting example of an optimized value for parameter a may be, but is not necessarily limited to, 0.5, which determines the maximum color space coverage of the inverse reshaped HDR domain or color space in the R.2020 color space. As wider inverse reshaped color spaces are used or supported (e.g., corresponding to values ​​less than the optimized value of parameter a, etc.), the reshaped SDR colors and codewords may begin to deviate from the reference SDR colors and codewords with relatively larger distortions. This is because increasing the reverse reshaped HDR domain or color space means squeezing the distinguished forward reshaped SDR colors (or codewords) more tightly into the forward reshaped SDR domain or color space, causing the forward reshaped SDR colors to move from the original 3D positions represented in the reference SDR colors (or codewords) to be approximated by the forward reshaped SDR colors.

[0226] As described above, forward and reverse TPB coefficients for a TPB-based reshaping operation may be determined for a given value of parameter a, such as an optimized value. These TPB coefficients may be multiplied in a mathematical equation with a forward or reverse generation matrix constructed with TPB basis functions using an input codeword as an input parameter to generate a forward or reshaped codeword, which may require performing numerous calculations involving calculating TPB basis function values ​​for each of numerous pixels of the image.

[0227] In some operational scenarios, in order to speed up or reduce computation, forward and reverse 3D-LUTs may be constructed from pre-computed TPB basis function values ​​for sampled values ​​and optimized forward and reverse TPB coefficients generated for optimized values ​​of parameter a. The forward and reverse 3D-LUTs or the lookup nodes / entries therein may be pre-built before being deployed at runtime to process input images and applied to relatively simple lookup operations in the forward and reverse paths or corresponding forward and reverse reshaping operations performed therein on input images at runtime.

[0228] Denote the optimized value of parameter a as a opt The corresponding optimal forward and reverse TPB coefficients are expressed as and

[0229] The forward 3D-LUT can be used to forward reshape the input HDR colors (or codewords) in an input HDR domain or color space, such as the R.2020 domain or color space (including the a color space), into forward reshaped SDR colors (or codewords) in a forward reshaped SDR domain or color space.

[0230] The alpha color space identified inside an input HDR domain or color space, such as an R 2020 (container) domain or color space, can be used to cut out HDR colors or codewords represented outside the alpha color space. The forward TPB coefficients can be applied to the input HDR colors or codewords in the alpha color space identified inside the input HDR domain or color space, such as an R 2020 (container) domain or color space, to generate predicted or forward reshaped SDR colors or codewords for each lookup node or entry in the forward 3D-LUT. Therefore, the forward 3D-LUT includes a plurality of lookup nodes or entries, each of which maps or forward reshapes a corresponding input (cross-channel or three-channel) HDR color or codeword to a corresponding predicted or forward reshaped (cross-channel or three-channel) SDR color or codeword.

[0231] In some operating scenarios, the forward 3D-LUT construction process includes a first step in which a 3D uniform sampling grid is prepared in an input HDR domain or color space, such as the R 2020 (container) domain or color space.

[0232] By way of illustration and not limitation, the input HDR domain or color space is an R.2020YCbCr color space HLG including three dimensions or channels (i.e., Y axis, Cb axis, Cr axis). The input HDR values ​​(denoted as ) can be sampled uniformly in each dimension or channel as follows:

[0233]

[0234] where i∈[0,...,N Y -1], j∈[0,...,N Cb -1],k∈[0,...,N Cr -1]

[0235] Among them, the total number of sample values ​​along each axis of the YCbCr axis is represented as N Y 、N Cb and N Cr Therefore, the total number of combinations of sample values ​​on all these axes is N u =N Y N Cb N Cr .

[0236] Although the valid values ​​in the entire R.2020 YCbCr color space may be limited (not covering the entire 3D cube of YCbCr values), the entire sample value grid can be used to cover the entire 3D cube of YCbCr values ​​to reduce the possibility of codeword bias caused by the compression operation.

[0237] For simplicity, (i,j,k) can be vectorized or represented as p. Therefore, the sample value in R.2020YCbCr It can be expressed as The vector / matrix can be obtained by u The nodes are collected together to construct, as follows:

[0238]

[0239] The forward 3D-LUT construction process includes a second step, in which the sampled values ​​in the R.2020YCbCr color space HLG can be converted to corresponding values ​​(or colors) in the R.2020RGB color space HLG as follows:

[0240]

[0241] The forward 3D-LUT construction process includes a third step, in which the converted value in the R.2020RGB color space HLG can be converted into an optimized value (denoted as a) with parameter a in the a RGB color space HLG. opt ) corresponding values ​​(or colors) are as follows:

[0242]

[0243] The forward 3D-LUT construction process includes a fourth step, in which the converted values ​​in (a) RGB color space HLG may be clipped as follows:

[0244]

[0245] The forward 3D-LUT construction process includes a fifth step, in which the clipped values ​​in the aRGB color space HLG can be converted to corresponding values ​​(or colors) in the R.2020RGB color space HLG as follows:

[0246]

[0247] The forward 3D-LUT construction process includes a sixth step, in which the value obtained by the above expression (47) in the R.2020RGB color space HLG can be converted to the corresponding value (or color) in the R.2020YCbCr color space HLG as follows:

[0248]

[0249] The forward 3D-LUT construction process includes a seventh step, in which, for each lookup node / entry in the forward 3D-LUT, the optimized forward TPB coefficients are combined with the input HDR color or codeword (or codeword value) obtained using the above expression (48) in the R.2020YCbCr color space HLG as the input parameter of the forward TPB basis function to construct a forward generation matrix Used together to obtain mapped or forward reshaped SDR colors or codewords (or codeword values) as follows:

[0250]

[0251] In the above expression (49), the input HDR color or codeword (or codeword value) can be used As input to the forward TPB basis function to construct or construct the forward generator matrix as follows:

[0252]

[0253] Mapped or forward reshaped SDR color or codeword (or codeword value) The input HDR colors or codewords (or codeword values) used as the input parameters of the forward TPB basis functions can be used as the lookup keys of the lookup nodes / entries of the forward 3D-LUT. At runtime, these mapped or forward reshaped SDR colors or codewords can be simply looked up in the forward 3D-LUT with the lookup keys as the input HDR colors or codewords to be forward reshaped into the mapped or forward reshaped SDR colors or codewords.

[0254] As mentioned earlier, the reverse 3D-LUT can be used to reversely reshape the reshaped SDR colors (or codewords) in the forward reshaped SDR domain or color space into reverse reshaped HDR colors (or codewords) in a reverse reshaped HDR domain or color space such as the R.2020 domain or color space (including the a color space).

[0255] In some operating scenarios, a reverse 3D-LUT construction process that may be simpler than the forward 3D-LUT construction process discussed above may be implemented or executed to construct or construct a reverse 3D-LUT. In some operating scenarios, in order to speed up or reduce calculations, a reverse 3D-LUT may be constructed from pre-computed TPB basis function values ​​of sample values ​​and optimized reverse TPB coefficients generated for optimized values ​​of parameter a. The reverse 3D-LUT or the lookup nodes / entries therein may be pre-built prior to being deployed at runtime to process input images such as forward reshaped SDR images, and applied to relatively simple lookup operations in the reverse path or corresponding reverse reshaping operations performed therein on the input images at runtime.

[0256] The pre-built reverse 3D-LUT can be deployed on the decoder side. A receiving downstream device on the decoder side can receive and decode a video signal encoded with a forward reshaped SDR image in a forward reshaped SDR domain or color space, and apply reverse reshaping to the forward reshaped SDR image using the 3D-LUT to generate a reverse reshaped HDR image in a reverse reshaped HDR domain or color space (such as a color space contained in the R.2020 domain or color space).

[0257] Although the shape formed or outlined by the complete boundaries of the forward reshaped SDR domain or color space mapped by utilizing the TPB-based forward reshaping operation may not be a simple 3D cube, the forward reshaped SDR values ​​in the forward reshaped SDR domain or color space can be clipped to the tightest or smallest 3D cube (e.g., a 3D rectangle, a 3D cube rescaled from a 3D rectangle, etc.) that contains or supports all the forward reshaped SDR values ​​represented in all lookup nodes / entries of the forward 3D-LUT, for example without clipping these SDR values ​​represented in the forward 3D-LUT.

[0258] In some operational scenarios, the inverse 3D-LUT construction process includes a first step in which a complete grid of sample values ​​may be prepared in a forward reshaped SDR domain or color space, such as the full R.709 SDR YCbCr color space.

[0259] The reverse 3D-LUT construction process includes a second step, in which the minimum and maximum values ​​of each dimension or (color) channel in the forward reshaped SDR domain or color space can be determined among the forward reshaped SDR values ​​in the forward 3D-LUT and used to limit the (input) data range of the forward reshaped SDR (e.g., YCbCr, etc.) colors or codewords used as input to the reverse path.

[0260] The inverse 3D-LUT construction process includes a third step in which, for each lookup node / entry in the inverse 3D-LUT, the optimized inverse TPB coefficients are combined with an inverse generation matrix constructed using the forward reshaped SDR colors or codewords (or codeword values) in the clipped or limited (input) data range obtained in the second step of the inverse 3D-LUT construction process as input parameters of the inverse TPB basis function. (e.g., as illustrated in expression (29) above, etc.) to obtain a mapped or inversely reshaped HDR color or codeword (or codeword value).

[0261] The mapped or reverse reshaped HDR colors or codewords (or codeword values) can be used as lookup values ​​for lookup nodes / entries of the reverse 3D-LUT, while the input or forward reshaped SDR colors or codewords (or codeword values) used as input parameters of the reverse TPB basis functions can be used as lookup keys for lookup nodes / entries of the reverse 3D-LUT. At runtime, these mapped or reverse reshaped HDR colors or codewords can be simply looked up in the reverse 3D-LUT with the lookup keys as inputs to be reverse reshaped into mapped or reverse reshaped HDR colors or codewords or forward reshaped SDR colors or codewords.

[0262] White Box Backward Optimization (WB)

[0263] Due to the cost and / or computational overhead involved in implementing or operating forward TPB-based reshaping or forward 3D-LUT in video capture and / or editing applications, forward TPB-based reshaping or corresponding forward 3D-LUT may not be used in all video capture and / or editing devices.

[0264] In some operational scenarios, HDR to SDR conversion, such as HDR HLG to SDR image generation, may be performed in a video capture and / or editing device using a hardware-based solution such as implemented with an available ISP. An existing ISP pipeline deployed with the device may operate with one or more programmable parameters to generate, convert, and / or output an SDR image based in part or in whole on a corresponding HDR (e.g., HLG, etc.) image captured by the device.

[0265] In the forward path, programmable parameters for the ISP pipeline may be specifically set or configured to cause the ISP pipeline to output an SDR image that approximates as closely as possible a possible reference SDR image generated with a white-box HDR to SDR conversion function.

[0266] In the reverse path, the reverse TPB-based reshaping or the corresponding reverse 3D-LUT can be used to generate a reverse reshaped HDR image from the SDR image output from the ISP pipeline. The reverse TPB coefficients used in the reverse path can be optimized to, for example, cover as much of the reverse reshaped HDR domain or color space as possible, such as the R.2020 color space.

[0267] Figure 3B An example process flow for determining or generating optimized values ​​for one or more programmable parameters in an ISP pipeline for generating or outputting (ISP) SDR images from HDR images is illustrated.

[0268] In some operational scenarios, the ISP pipeline may be a relatively limited programmable module that is implemented with the ISP (hardware) to approximate a white-box HDR to SDR conversion function such as a white-box (e.g., known, well-defined, etc.) conversion function for converting an HDR HLG image to a corresponding SDR image in block 330.

[0269] The ISP pipeline may include or implement: (1) a first set of three one-dimensional lookup tables (1D-LUTs) for converting from HLG RGB to linear RGB, (2) followed by a 3×3 matrix for converting from an HDR image in an HDR domain or color space, such as the R.2020 color space, to an SDR domain or color space, such as the R.709 color space, and (3) followed by a second set of three 1D-LUTs implementing BT.1886 linear to non-linear (gamma) conversion using an (e.g., standards-based, etc.) HLG optical-to-optical transfer function (OOTF).

[0270] For illustration purposes only, an HDR image may be obtained from Figure 3B The one or more programmable parameters for the ISP pipeline may be the design parameters specified in the BT.1886 standard (denoted as γ BT1886 ).

[0271] Figure 3B The process flow can be used to search for design parameters of ISP SDR imagesγ BT1886 The optimized value of γ is used to best approximate the reference SDR image generated in block 330. The process flow can be performed by designing the parameters γ in an iterative order that can be sequential or non-sequential. BT1886 Iterate over multiple candidate values.

[0272] Block 322 includes selecting a design parameter γ BT1886 The current (i.e., to be iterated, etc.) value of the design parameter γ BT1886 The next candidate value among multiple candidate values ​​of (for example, initially the first candidate value, etc.). Given the design parameter γ BT1886 In the case of the current value of Figure 3B The ISP SDR image is compared with the reference SDR image in one or more subsequent process flow boxes.

[0273] Block 324 includes applying a first set of 1D-LUTs to the HDR HLG (RGB) image retrieved from the database to generate a corresponding HDR linear RGB image, as follows:

[0274]

[0275] Among them, ch represents the red, green or blue channel; Represents the HDRHLG codeword in the (input) HDR HLG image; represents the HDR linear codeword in the corresponding HDR linear image generated from the first set of 1D-LUT as expressed in Expression (51); parameters (a, b, c) can be set to (0.17883277, 0.28466892, 0.55991073).

[0276] Block 326 includes applying a 3×3 matrix to the corresponding HDR linear RGB image to generate a corresponding SDR linear RGB image. The 3×3 matrix may be given as follows:

[0277] P R2020→R709 =P R2020-XYZ P XYZ→R709 (52)

[0278] The SDR linear RGB codeword in the corresponding SDR linear RGB image generated by the 3×3 matrix in the above expression (52) can be expressed as

[0279] Block 328 includes applying the second set of 1D-LUTs to the SDR linear RGB codewords in the corresponding SDR linear RGB image. To generate the corresponding ISP SDR codeword in the corresponding ISP SDR image (expressed as ).

[0280] In some operating scenarios, as described above, the second set of 1D-LUTs merges or combines the linear to non-linear SDR conversion with the HLG OOTF given as follows:

[0281]

[0282] in, Represents the SDR linear RGB codeword from the corresponding SDR linear RGB image Generated intermediate SDR codeword; L w =100;L b = 0; and

[0283] For the intermediate SDR codeword from the above expression (53) Generate ISP SDR codewords The linear-to-non-linear SDR conversion may be a linear-to-gamma conversion as defined in BT.1886, as follows:

[0284]

[0285] Among them, γ BT1886 represents the design parameters mentioned above.

[0286] As implemented by ISP pipeline Figure 3B The blocks 322 to 328 of (or a combination of the first set of 1D-LUTs represented in the above expression (51), the 3×3 matrix represented in the above expression (52), and the second set of 1D-LUTs represented in the above expressions (53) to (56)) are modified with the design parameter γ BT1886 The specific values ​​of (e.g., the current candidate value, etc.) jointly implement the HDR HLG to SDR conversion function (denoted as f HLG→ISPSDR ).

[0287] Block 332 includes determining the conversion function f from HDR HLG to SDR in blocks 322 to 328 HLG→ISPSDR The difference (e.g., quality, etc.) between the generated ISPSDR image and the reference SDR image generated from the white-box HDR to SDR conversion function in block 330. Quality assessment functions such as MSE, RMSE, SAD, PSNR, SSIM, etc. may be used to calculate these differences.

[0288] Block 334 includes determining the design parameter γ BT1886 Is the current candidate value of the design parameter γ BT1886 If yes, the process flow proceeds to block 336. Otherwise, the process flow returns to block 322.

[0289] Block 336 includes selecting a design parameter γ BT1886 The optimal or optimized value of . Select the design parameter γ BT1886 The optimal value of can be expressed as finding the design parameter γ from multiple candidate values. BT1886 A specific value of (denoted as γBT1886,opt ) such that the difference between the ISP SDR image as computed using a relatively large image dataset or database and the reference SDR image is minimized as follows:

[0290] γ BT1886,opt =argmin∑ t D(f HLG→ISPSDR (V t ,γ BT1886 )-f HLG→W_SDR (V t )) (57)

[0291] Wherein, D() represents the quality assessment function used in block 332 .

[0292] γ BT1886,opt An example value of may be, but is not necessarily limited to, 2.115.

[0293] SDR to PQ TPB Optimization

[0294] As described above, in a "WFB" use case or operating scenario, the BESA algorithm may be used under the SLBC framework to generate optimized reshaping operation parameters that are used by forward and reverse reshaping operations to generate forward reshaped SDR images and reverse reshaped HDR images. In contrast, in a "WB" use case or operating scenario, only reverse reshaping may be performed under the SLiDM framework on non-forward reshaped SDR images encoded in the output video signal (e.g., ISP SDR images generated by an ISP pipeline implemented with a video capture and / or editing device) to generate corresponding reverse reshaped or reconstructed HDR images.

[0295] In the "WB" use case or operating scenario, forward TPB reshaping from an input HDR domain or color space such as R.2020 HDR color space HLG to a forward reshaped SDR domain or color space such as R.709 SDR YCbCr color space cannot be used to help utilize the full codeword range supported by the R.709 SDR YCbCr color space. The ISP SDR codewords encoded in the ISP SDR images in the output video signal are typically hard-constrained in time in the R.709 color space portion generated or supported by the video capture and / or editing device or the ISP pipeline implemented therein. The ISP SDR codewords in the hard-constrained R.709 color space portion may be difficult to further extend or map back to the reverse reshaped or reconstructed HDR domain or color space by reverse reshaping such as reverse TPB-based reshaping.

[0296] However, as in the "WFB" use case or operating scenario, in the "WB" use case or operating scenario, optimized reshaping operation parameters such as TPB coefficients may be generated in the reshaping optimization to help increase the coverage of supported color spaces in which the inverse reshaped HDR images are to be represented. In many "WB" operating scenarios, the inverse reshaped or reconstructed HDR domain or color space achieved with inverse TPB-based reshaping may be only slightly larger than the R.709 color space, compared to the maximum (a) color space achievable in the "WFB" use case or operating scenario.

[0297] For illustrative purposes only, Figure 1B In the operating scenario shown in , the ISP SDR domain or color space can be the domain or color space of R.709 or its hard limit part in the ISP pipeline, and the (original or reverse-reshaped) HDR domain or color space can be the domain or color space of R.2020.

[0298] By way of illustration and not limitation, a subset or subspace in the R.2020 color space (or HDR color space used to represent the reshaped HDR image) may be defined or characterized by a specific white point and three specific color primaries (red, green, blue). The specific white point may be selected or fixed to be the D65 white point.

[0299] The inversely reshaped HDR color space used to represent the inversely reshaped or reconstructed HDR image is denoted as the (b) color space. Therefore, the CIExy coordinates of the color primaries and white point defining the (b) color space can be expressed as and (b) White point of color space Can be specified as D65 white point.

[0300] (b) Each of the color primaries of the color space may be a corresponding color primaries along the P3 color space in the CIExy coordinate system or chromaticity diagram Corresponding color primaries to the R.2020 color space Any point along the line between two corresponding chromatic algebras in the P3 color space and the R.2020 color space can be represented as a linear combination of the two corresponding chromatic algebras with a weight factor b, as follows:

[0301]

[0302]

[0303] Therefore, the optimization problem of finding the maximum support for the R.2020 color space from the (b) color space can be simplified to the problem of selecting the weight factor b. When b = 0, the (b) color space becomes the entire R.2020 color space. When b = 1, as Figure 2K As shown in the figure (wherein the (b) color space is represented as "TPB" or "TPB-covered color (b color space)"), the (b) color space becomes the P3 color space. Figures 2H to 2J Three example (b) color spaces are illustrated, where b=0.25, 0.50 and 0.75, respectively.

[0304] like Figure 2H to Figure 2K As illustrated in , in order to make the inversely reshaped or reconstructed HDR domain or color space cover as large an R.2020 color space as possible, the TPB optimization problem boils down to finding or searching for the minimum possible value of the parameter b.

[0305] Figure 3C An example process flow for finding the minimum possible value of parameter b is illustrated. Figure 3C The process flow of may iterate through multiple candidate values ​​of parameter b in an iteration order that may be a sequential order or a non-sequential order.

[0306] Block 342 includes selecting a current (e.g., to be iterated, etc.) value of parameter b to be the next candidate value (e.g., initially the first candidate value, etc.) of a plurality of candidate values ​​for parameter b. Given a candidate inversely reshaped HDR color space having a current value of parameter b, the inversely reshaped HDR color space may be selected in Figure 3C Optimized reshaping operation parameters such as optimized TPB coefficient are obtained in one or more subsequent process flow boxes.

[0307] Block 344 includes constructing sample points or preparing two sample data sets in a candidate inversely reshaped HDR color space. By way of example and not limitation, the candidate inversely reshaped HDR color space may be a hybrid log-gamma (HLG) RGB color space (referred to as "(b)RGB color space HLG").

[0308] and Figure 3A Similar to block 304 of FIG. 1 , the first of the two sampled data sets is a uniformly sampled color patch data set. Each color patch in the uniformly sampled color patch data set consists of a corresponding RGB color uniformly sampled from (b) the RGB color space HLG (expressed as ) characterizes or represents that the (b) RGB color space HLG includes three dimensions of R axis, G axis, and B axis, which are respectively represented as R, G, and B component colors. Each axis or dimension of the (b) RGB color space HLG is normalized to a value range of [0,1] and is respectively represented by N R 、N G and N BHere, N R 、N G and N B Each of represents a positive integer greater than one (1). Therefore, the total number of color patches or sampled data points in the uniformly sampled color patch data set is given as N u =N R N G N B . Each uniformly sampled data point or RGB color in the uniformly sampled color patch dataset Given as where i∈[0,...,N R -1],j∈[0,...,N G -1],k∈[0,...,N B -1].

[0309] For simplicity, (i,j,k) can be vectorized or simply represented as p. Correspondingly, the uniformly sampled data points or RGB colors It can be simply expressed as All N u The nodes (each of which represents a unique combination of a specific R-axis cell / partition, a specific B-axis cell / partition, and a specific G-axis cell / partition) can be grouped or collected into a vector / matrix as follows:

[0310]

[0311] The second of the two sample data sets prepared or constructed in block 344 is a neutral color data set. The second data set includes a plurality of neutral colors or neutral color patches (also referred to as gray or gray color patches).

[0312] The second data set can be used to retain input gray color patches in the input domain as output gray color patches in the output domain when the input gray color patches in the input domain are mapped to or reshaped into output gray color patches in a reshaping operation as described herein. Increased weighting can be given to input gray color patches in the input domain (or input color space) in the optimization problem compared to other color patches to reduce the likelihood that these input gray color patches are mapped to non-gray color patches in the output domain (or output color space) by the reshaping operation.

[0313] N can be generated by uniformly sampling the R, G, and B values ​​along a line connecting a first gray (0,0,0) and a second gray (1,1,1) in an RGB domain (eg, (b) RGB color space HLG, etc.). n Nodes or gray blocks are used to prepare or construct a second data set (gray data set or gray data set), as follows:

[0314]

[0315] where i∈[0,...,N n -1]

[0316] All N in the second dataset n The nodes can be grouped or collected into a neutral color vector / matrix as follows:

[0317]

[0318] The neutral color vector / matrix in the above expression (8) can be repeated N times. t (a positive integer not less than (1)) times to generate the N in the (now repeated) second data set n N t A neutral color block, as follows:

[0319]

[0320] Repeating the neutral colors in the repeated second neutral color data set increases the weight of the neutral colors or grays compared to other colors. Therefore, the neutral colors can be retained more in the optimization problem than other colors.

[0321] The first (all sampled) color data set and the second neutral color (repeated) data set in expressions (61) and (64) can be collected or placed together in a single combined vector / matrix as follows:

[0322]

[0323] Combine vectors / matrices The total number of vector / matrix elements (repeating and non-repeating patches) in is N=N n N t +N u In the above expression (10), Each vector / matrix element or color patch (row) in can be represented as follows:

[0324]

[0325] Block 346 includes converting the combined vector / matrix in (b) RGB color space (or (b) RGB color space HLG) The color values ​​of the color blocks (rows) represented in the vector / matrix elements are converted to the corresponding color values ​​in the standard R.2020 color space or R.2020RGB color space HLG as follows:

[0326]

[0327] Among them, P(b)→R2020 Represents the conversion matrix from (b) color space to R.2020RGB color space HLG, which can be used with Figure 3A Construct P in the process flow (a)→R2020 is constructed in a similar way, as follows:

[0328] P (b)→XYZ (68-1)

[0329] P (b)→R2020 =P (b)→XYZ P XYZ→R2020 (68-2)

[0330] Among them, P (b)→XYZ represents the transformation matrix from (b) color space to XYZ color space, which can be used with Figure 3A Construct P in the process flow (a)→XYZ is constructed in a similar manner.

[0331] Box 348 includes converting the vector / matrix in the R.2020RGB color space HLG The color values ​​of the color blocks (rows) represented in the vector / matrix elements are converted or mapped to the corresponding ISP SDR color values ​​in the R.709SDR RGB color space (expressed as ) and then further to the corresponding color value in the R.709SDR YCbCr color space (expressed as ),as follows:

[0332]

[0333] Among them, f HLG→ISPSDR () represents the ISP equation, such as through the ISP pipeline using Figure 3B The process flow for generating the design parameters γ BT1886 Optimized values ​​for implementing the HDR HLG to SDR conversion function.

[0334] Each color block (row) in can be represented as follows:

[0335]

[0336] Box 350 includes converting the vector / matrix in the R.2020RGB color space HLG The color value of the color block (row) represented by the vector / matrix element is converted or mapped to the corresponding color value in the R.2020YCbCr color space PQ (expressed as ),as follows:

[0337]

[0338] Each color block (row) in can be represented as follows:

[0339]

[0340] Block 352 includes using and As input to the inverse TPB optimization algorithm, inverse TPB coefficients corresponding to the current value of parameter b for the (b) color space are generated for TPB-based reshaping optimization.

[0341] To this end, it is possible to The TPB basis functions and SDR codewords represented in the above construct the inverse generation matrix as follows:

[0342]

[0343] The reverse prediction error can be determined by comparing the reverse reshaped HDR patches or codewords collected in the vector / matrix with the reference HDR patches or codewords in the per-channel reverse observation vector / matrix obtained and minimized when solving the TPB optimization problem. The per-channel reverse observation vector / matrix can be generated or pre-computed from the above expression (71), stored or cached in computer memory, and fixed in all iterations as follows:

[0344]

[0345] The inverse TPB coefficients (expressed as ), which minimizes the difference between the inversely reshaped HDR color block or codeword and the reference HDR color block or codeword, as follows:

[0346]

[0347] The per-channel predicted (or inversely reshaped or reconstructed) HDR codeword for each channel ch may be calculated as follows:

[0348]

[0349] Block 354 includes determining whether the current candidate value for parameter b is the last candidate value among the plurality of candidate values ​​for parameter b. If so, process flow proceeds to block 356. Otherwise, process flow returns to block 342.

[0350] Block 356 includes selecting an optimal or optimized value for parameter b, and calculating (or simply selecting already calculated) optimized forward and reverse TPB coefficients corresponding to the optimized value for parameter b for the (b)RGB color space.

[0351] Similar to the "WFB" use case or operation scenario, in the "WB" use case or operation scenario, (b) color space is also part of the optimization. The (b) color space and the inverse TPB coefficients can be optimized jointly. Therefore, the optimization problem can be formulated as finding (a specific value of ) such that (b) the color space is generated to cover the largest HDR color space or color space portion while achieving the expression The minimized HDR prediction error is as follows:

[0352]

[0353] An example optimized value for parameter b may be, but is not necessarily limited to, 1, which corresponds to the P3 color space.

[0354] Black-box TPB reverse optimization in a single device (BB1)

[0355] Some (dual mode) video capture devices or mobile devices support both SDR and HDR capture modes and can therefore output SDR or HDR images. Some (single mode) video capture devices or mobile devices only support SDR capture mode. Under the technology as described herein, SDR to HDR mappings can be modeled or generated with dual mode video capture devices. Some or all of these SDR to HDR mappings can then be applied to SDR images captured by dual mode video capture devices or single mode video capture devices, for example in a downloaded and / or installed video capture application, to "up-convert" SDR images acquired with the SDR capture mode to corresponding HDR images, regardless of whether these video capture devices support HDR capture mode.

[0356] In some operating scenarios, since the computing environment in the mobile device or the processing power of the mobile device may be limited, a static (inverse) 3D-LUT and optimized TPB coefficients obtained at least in part from the TPB basis function can be used to represent a static SDR to HDR mapping, which maps or inversely reshapes SDR images such as all SDR images in an SDR video sequence to generate corresponding HDR images such as all HDR images in a corresponding HDR video sequence.

[0357] Static SDR to HDR mapping can be modeled based at least in part on multiple image pairs of HDR and SDR images acquired with a specific camera of a dual-mode video capture device. Each image pair of the HDR and SDR images includes an SDR image and an HDR image corresponding to the SDR image. The SDR image and the HDR image depict the same visual scene in the real world as having spatial alignment errors caused by spatial movement, which may occur between a first time point when a specific camera operating in the SDR capture mode captures the SDR image and a second time point when the same camera operating in the HDR capture mode captures the HDR image. For example, for each visual scene in the physical world, the video capture device can operate in HDR mode to capture the HDR image in the image pair and use the SDR mode to capture the SDR image in the same image pair. This operation can be repeated to generate multiple image pairs of HDR and SDR images to generate a relatively large number of different color blocks (or a relatively large number of different colors or different codewords) for generating a static SDR to HDR mapping.

[0358] There are several challenges in modeling static SDR to HDR mapping using captured SDR and HDR images. First, although both SDR and HDR capture modes can use the same ISP inside the same video capture device, the video capture device can apply different image capture settings (e.g., exposure settings of a specific camera, etc.) for the same scene to obtain the designed optimized picture quality in the captured SDR and HDR images. Therefore, static SDR to HDR mapping may be able to model or approximate the HDR image capture settings actually implemented by the video capture device to some extent. Secondly, the spatial alignment between the captured SDR and HDR images of the image pair to be used to obtain the SDR and HDR images may not be accurate. For example, the selection of a specific capture mode between the SDR capture mode and the HDR capture mode can be performed via the screen of the touching video capture device, which may cause the video capture device to lose or move away from its previous spatial position and / or orientation. In addition, the temporal alignment between the captured SDR video / image sequence and the captured HDR video / image sequence may easily occur because these SDR and HDR sequences are captured at different times or durations. Some of the visual objects depicted in these sequences can move in or out of the camera field of view. Some local areas in the camera's field of view may be occluded or unoccluded from time to time.

[0359] In some operational scenarios, a registration operation may be performed to resolve spatial and / or temporal alignment issues between a captured SDR image and a corresponding captured HDR image to generate an aligned SDR image and a corresponding aligned HDR image and include the aligned SDR image and the corresponding aligned HDR image in an image pair of SDR and HDR images. The image pair may then be used to determine or establish SDR patches or colors in the aligned SDR image and corresponding HDR patches or colors in the corresponding aligned HDR image. These SDR and HDR patches or colors may be used to derive at least some of a set of corresponding SDR and HDR patches or colors for deriving or generating a static SDR to HDR mapping.

[0360] The SDR and HDR video capture processes may be performed to provide comprehensive coverage of different scenes and / or exposures at different times of day (e.g., from early morning to late evening) for both indoor and outdoor scenes. For each scene, the same video capture device may be used in both SDR mode and HDR mode to capture SDR and HDR video sequences. In some operational scenarios, specific frames / images, such as the first frame / image, of each of the video sequences may be selected or extracted for use in constructing or obtaining multiple image pairs of SDR and HDR images to be included in a training image dataset or database.

[0361] Figure 3D An example process flow is illustrated for generating optimized reshaping operation parameters such as optimized or optimal TPB coefficients using corresponding (or matched) SDR and HDR color blocks or colors in a captured SDR image and a captured HDR image (corresponding to the captured SDR image) captured by a video capture device operating in SDR and HDR capture modes, respectively.

[0362] Block 362 includes receiving a captured SDR image and finding or extracting a set of SDR image feature points in the captured SDR image. Block 364 includes receiving a captured HDR image and finding or extracting a set of HDR image feature points in the captured HDR image. The set of SDR image feature points and the set of HDR image feature points may be the same set of feature point types.

[0363] The set of feature point types may include feature point types among various different feature point types, including but not necessarily limited to any, some or all of the following: BRISK features detected using Binary Robust Invariant Scalable Keypoints or BRISK algorithm; corners detected using Features-from-Accelerated-Segment-Test or FAST algorithm; features detected from KAZE algorithm; corners detected using minimum eigenvalue algorithm; features generated using Maximum Stable Extreme Region or MSER algorithm; key points detected using oriented FAST and rotation or ORB algorithm; features extracted from scale-invariant feature transform or SIFT algorithm; features extracted from accelerated robust features or SURF algorithm; and the like.

[0364] Each of some or all of the set of SDR image feature points and the set of HDR image feature points may be represented by a feature vector or descriptor such as an array of (eg, numerical values, etc.) feature values.

[0365] Block 366 includes matching some or all of the feature points in the set of SDR image feature points with some or all of the feature points in the set of HDR image feature points. For each feature point of a particular type in the set of SDR image feature points in the SDR image, a matching metric between the feature point in the SDR image and each of the feature points of the same type in the set of HDR image feature points in the HDR image can be calculated. In a non-limiting example, the matching metric can be calculated as the sum of absolute differences or SAD (or another metric function for measuring the difference between two feature points) between a first feature vector representing a feature point in the SDR image and a second feature vector representing a feature point in the HDR image. The particular feature point with the lowest SAD or matching metric can be selected from the feature points of the same type in the HDR image as a possible match for the feature point in the SDR image. In response to determining that the lowest SAD or matching metric is below a matching difference threshold, the particular feature point in the HDR image can be identified or determined as a match for the feature point in the SDR image; the feature point in the SDR image and the particular feature point in the HDR image form a pair of matched SDR and HDR feature points. Otherwise, the particular feature point may not be identified as such a match. The matching operation may be repeatedly performed for all feature points in the set of SDR image feature points, thereby producing a set of paired matched SDR and HDR feature points.

[0366] Block 368 includes computing a geometric transformation, such as a 3×3 2D affine transformation, between the SDR image and the HDR image based in part or in whole on the set of paired matched SDR and HDR feature points. The geometric transformation may be computed or derived using coordinates (e.g., pixel rows and pixel columns, etc.) of matched feature points from the set of paired matched SDR and HDR feature points in the SDR and HDR images.

[0367] For each (e.g., kth, etc.) pair of matched SDR and HDR feature points, the 2D coordinates of the HDR feature points in the pair can be expressed as (τ ix , τ iy ) and included in the i =[τ ix τ iy 1], and the 2D coordinates of the SDR feature point can be expressed as (η ix , η iy ) and included in the second vector η i =[η ix η iy 1].

[0368] The total number of paired matched SDR and HDR feature points in the set of paired matched SDR and HDR feature points is denoted as N. Vectors generated from the 2D coordinates of feature points in all paired matched SDR and HDR feature points in the set of paired matched SDR and HDR feature points may be collected together as follows:

[0369]

[0370] As mentioned above, the geometric transformation can be represented as a 3×3 matrix as follows:

[0371]

[0372] Mathematically, the first vector and the second vector respectively representing the SDR and HDR feature points in each pair of the set of matched SDR and HDR feature points may be related to each other by a 3×3 matrix representing a geometric transformation as follows:

[0373]

[0374] Alternatively, for all N pairs of matched feature points in the set of pairs of matched SDR and HDR feature points, the vectors representing the SDR and HDR feature points may be related by a 3×3 matrix representing the geometric transformation as follows:

[0375]

[0376] The values ​​of the matrix elements in the 3×3 matrix representing the geometric transformation can be generated or obtained as a solution (e.g., a least square solution, etc.) to an optimization problem that minimizes the transformation or alignment error. More specifically, the optimization problem can be expressed as follows:

[0377]

[0378] The optimized or optimal value of the 3×3 matrix representing the geometric transformation can be obtained via the least squares solution of the optimization problem in the above expression (), as follows:

[0379] F opt =(T T T) -1 (T T H) (84)

[0380] Block 370 includes applying a geometric transformation to one of the SDR and HDR images. In the present example, for example, for each of the three channels such as Y, Cb and Cr in which the HDR image is represented, a transformation is applied to the HDR image so that the HDR pixel positions in the HDR image are shifted to coincide with the SDR pixel positions in the SDR image. Thus, most of the HDR pixels in the HDR image are spatially aligned with the corresponding SDR pixels in the SDR image. The remaining HDR pixels and the remaining SDR pixels (with which no HDR pixels are spatially aligned) can be excluded (e.g., assigned out-of-range pixel values, etc.) for use as (valid) color blocks or colors for generating a static SDR to HDR mapping or TPB coefficients or a static inverse 3D-LUT.

[0381] Block 372 includes finding corresponding valid patches or colors in the SDR and HDR images. The valid patches or colors may be obtained from the codeword values ​​of the spatially aligned pixels in the SDR and HDR images. As described above, the spatially aligned pixels may be generated by applying a geometric transformation, which in turn is generated from the matched SDR and HDR feature points.

[0382] In some operational scenarios, in order to enhance the spatial alignment accuracy or reliability of the geometric transformation, after identifying the (initial) matched SDR and HDR feature points with a matching metric below a minimum matching difference threshold, another matching threshold can be applied to select or distinguish a subset of final matched SDR and HDR feature points from the (initial) matched SDR and HDR feature points.

[0383] For each spatially aligned pixel in the SDR image, an SDR codeword, such as an SDR Y / Cb / Cr value, may be determined for the spatially aligned pixel in the SDR image. For co-located or spatially aligned pixels in a spatially transformed HDR image generated using a geometric transformation (corresponding to the spatially aligned pixels in the SDR image), a corresponding HDR codeword, such as a corresponding HDR Y / Cb / Cr value, may be determined for the spatially aligned pixel in the HDR image. If the transformed HDR pixel is unusable (e.g., has a value of 0, etc.), the HDR pixel may be discarded or prevented from being considered as part of a matched color block or codeword.

[0384] In each pair (e.g., the i-th pair, etc.) of matched SDR and HDR patches or colors as generated from a pair of spatially aligned SDR and HDR pixels in the SDR and HDR images, the matched SDR and HDR patches or colors (or codeword values), respectively, may be given as follows:

[0385]

[0386] All pairs of matched SDR and HDR color patches or colors (or codeword values) in all pairs of spatially aligned SDR and HDR pixels in the SDR and HDR images in all image pairs of SDR and HDR image pairs may be collected together from all images and merged into two matrices as follows:

[0387] From SDR images

[0388] From HDR images

[0389] Block 374 includes using the matrices in expressions (88) and (89) above and As input to solve or generate optimized or optimal TPB coefficients for inverse reshaping or static SDR to HDR mapping.

[0390] From the matrix The inverse TPP basis functions and codewords in generate the inverse generation matrix for each channel as follows:

[0391]

[0392] From the matrix Construct a per-channel observation matrix for SDR to HDR mapping as follows:

[0393]

[0394] The optimized or optimal inverse TPB coefficients for channel ch can be solved via a least squares solution as follows:

[0395]

[0396] The predicted or inversely reshaped HDR value for channel ch may be calculated as follows:

[0397]

[0398] Black-box reverse reshaping optimization in two devices (BB2)

[0399] In a "BB2" operating scenario, a (reverse) reshaping mapping may be used to map an (ISP captured) SDR image captured by a first video capture device in an SDR capture mode to generate an (ISP mapped) HDR image that simulates the HDR appearance of an (ISP captured) HDR image captured by a second, different video capture device operating in an HDR capture mode. In some operating scenarios, the first video capture device may be a relatively low-end mobile phone that may only capture SDR images or pictures, while the second video capture device may be a relatively high-end phone that may capture HDR images or pictures. Although the first and second video capture devices may operate with different hardware configurations and capabilities, the reshaping mapping is used to reshape the (ISP captured) SDR image captured by the first device into an (ISP mapped) HDR image that approximates the (ISP captured) HDR image captured by the second device.

[0400] The reshaping (SDR to HDR) mapping may be modeled based at least in part on a plurality of image pairs formed by (training) SDR images captured by a first device and (training) HDR images captured by a second device. Each of the image pairs of HDR and SDR images includes an SDR image and an HDR image corresponding to the SDR image.

[0401] In some "BB2" operational scenarios, some or all of the plurality of image pairs may include (e.g., initially, prior to an image alignment operation, etc.) captured SDR and HDR images that depict real-world visual scenes, such as natural indoor / outdoor scenes, as having spatial and / or temporal alignment errors associated with the first and second video capture devices. As in the "BB1" use cases or operational scenarios, in these "BB2" use cases or operational scenarios, image alignment operations between corresponding SDR and HDR images in the image pairs may be performed, for example, using geometric transformations constructed using selected feature points extracted from the SDR and HDR images. Subsequently, SDR and HDR color patches or colors determined from the aligned SDR and HDR images in the image pairs may be used to generate a reshape (SDR to HDR) mapping as described herein.

[0402] In some "BB2" operating scenarios, instead of or in addition to using images from natural scenes, some or all of the plurality of image pairs may include captured SDR and HDR images (e.g., initially, before image alignment operations, etc.), for example, in a laboratory environment, that depict color tables displayed on one or more reference image displays of the same type. For example, the color tables may be generated as 16-bit full HD RGB (color table) TIFF images. These TIFF images containing the color tables may be displayed as perceptually quantized (PQ) video signals on a reference image display such as a PRM TV. The color tables displayed on the reference image display may then be captured by the first and second video capture devices, respectively.

[0403] Figure 2N An example TIFF image including a color table is illustrated. As shown, the color table can be a central square in the TIFF image that includes a plurality of color blocks with a set of distinct colors to be captured and matched between a first device and a second device. Each color in the set of distinct colors can be different from all other colors in the set of distinct colors. The set of distinct colors can be displayed by a reference image display with a set of distinct pixel / codeword values. Therefore, each color in the set of distinct colors can correspond to a corresponding pixel or codeword value in the set of distinct pixel / codeword values.

[0404] Different TIFF images may include different color tables with different pluralities of colors or different sets of distinct colors. Each color table in a corresponding one of the different TIFF images may correspond to a corresponding one of the different pluralities or sets of distinct colors.

[0405] Figure 2N Each of the four corner rectangles in the TIFF image includes a checkerboard pattern. The same corner rectangle or the same checkerboard pattern may be included in different TIFF images containing different color tables. The checkerboard patterns at the four corners of the TIFF image can be used as a spatial key or fiducial marker for estimating a projective transformation, as will be discussed in detail later. Additionally, optionally or alternatively, the TIFF image may include one or more numbers, binary codes, QR codes, etc., as a unique identifier (ID) assigned to or used to identify the TIFF image or the color table therein or a group or more colors therein, etc.

[0406] In order to find the correspondence / mapping relationship between as many colors as possible between the first video capture device and the second video capture device, the color table in the TIFF image as described herein can include as many different (groups or multiple) colors as possible (corresponding to as many different pixel / codeword values ​​as possible), depending on the display capability of the reference image display to distinguish different colors. In addition, the color table can be displayed with different overall intensities or illuminations to enable the first and second video capture devices to perform different exposure settings under different lighting conditions. Therefore, the same color in the color table displayed in the TIFF image can be displayed with different intensities or illuminations on the reference image display.

[0407] Figure 3E An example process flow for generating a plurality of different color tables to be included in a plurality of different TIFF images is illustrated. To generate each color table, mean and variance values ​​of the colors in the color table (e.g., RGB, etc.) may be determined and used to randomly generate colors using a (type of) statistical distribution. In some operational scenarios, the statistical distribution represents a Beta distribution or a distribution type. Different combinations of mean and variance values ​​of the statistical distribution or distribution type may be used to generate different color tables having different pluralities or groups of distinct colors therein.

[0408] Block 382 includes defining or determining a set of possible means in the PQ domain or color space (denoted as ).

[0409] In an operating scenario where the reference image display used to display the TIFF image containing the color table is a PRM TV, the maximum and minimum luminance values ​​supported by the reference image display may be in the range between 1000 nits and 0 nits, or even larger dynamic range and contrast. By way of example and not limitation, the minimum and maximum PQ values ​​that the reference image display can display without clipping may be given by: P0=L2PQ(0) and P1=L2PQ(1000), respectively, where L2PQ(·) represents a mapping function from (linear or non-PQ) luminance to (non-linear or PQ) luminance codewords in the PQ domain or color space.

[0410] In a non-limiting example, the set can be defined as being uniformly distributed on a logarithmic scale from L 下限 To L 上限 A set of different PQ brightness code words corresponding to the (linear) brightness values ​​in the (linear) brightness value range. 下限 Can be set to 10 -4 equal value, and the upper limit L 上限 Can be set to 10 2.99=The number of possible means (or magnitudes) in the set of possible means may be determined in part or in whole by the first device and the second device, depending on the image capture capabilities of the first device and the second device (e.g., capturing the dimmest and brightest light levels, etc.). The logarithmic scale used as the luma codeword in the captured SDR and HDR images may be approximately linear with the logarithm of the luma in the range of luma values. ) can be, but is not necessarily limited to, the following sets:

[0411] Block 384 includes defining or determining a set of possible shape coefficients for generating possible variance values ​​for a statistical distribution or type of distribution (eg, a Beta distribution, etc.).

[0412] In some operating scenarios, the Beta distribution used to randomly select a color or codeword value is defined or has support in a scaled or closed value interval [0,1]. Given a PQ value in the PQ codeword value range [P0,P1], the scaled mean μ (where 0 < μ < 1 or in a closed value interval of the Beta distribution) can be calculated as follows:

[0413] μ=(μ PQ -P0) / (P1-P0), where

[0414] The scaled variance of the Beta distribution (denoted by σ) in the scaled or closed value interval [0,1] 2 ) can be adaptively set to be proportional to μ(1-μ) to avoid over / under exposure caused by large (e.g., brightness, etc.) differences in color patches during capture of SDR and HDR images by the first device and the second device, as follows:

[0415] σ 2 =μ(1-μ) / θ (95)

[0416] Wherein θ represents a shape coefficient that affects or determines the shape of the Beta distribution (e.g., more compressed, more expanded, etc.). The value of the shape coefficient may be selected from a set of possible shape coefficients θ to allow the generated color table to have a relatively high diversity of codeword values ​​and / or colors generated therefrom. A non-limiting example of the set of possible shape coefficients may be: θ={3, 6, 9, 12}.

[0417] Block 386 includes using the set of possible means and the set of shape coefficients to generate a plurality of all unique combinations of scaled means and shape coefficients for different instances of a statistical (e.g., Beta, etc.) distribution. In the present example, the total number of unique combinations is given as follows: Where |Θ| represents the total number or magnitude of elements in the set Θ. In some operational scenarios, a different TIFF image with a different color table may be generated for each unique combination of the scaling mean and the shape coefficient in a plurality of all unique combinations of the scaling mean and the shape coefficient, so that the total number of different color tables or corresponding different TIFF images is given as

[0418] Block 388 includes selecting the current combination of the scaled mean and the shape coefficient as the next combination from the plurality of all unique combinations of the scaled mean and the shape coefficient. A set of one or more different pixel or codeword values ​​that respectively specify or define a set of one or more different colors of the current color table corresponding to the current combination of the scaled mean and the shape coefficient may be generated from a Beta distribution, taking into account the scaled mean μ in the current combination and the current scaled variance σ in the current combination. 2 .

[0419] Block 390 includes calculating or defining a Beta distribution as follows:

[0420] f(x;α,β)=x α-1 (1-x) β-1 / B(α,β)where, x∈[0,1] (96)

[0421] Where α and β are Beta distribution parameters; B(α, β) represents the normalization constant. The Beta distribution parameters α and β can be obtained from the mean and variance of the Beta distribution as follows:

[0422] α=μv and β=(1-μ)ν, where ν=μ(1-μ) / σ 2 -1 (97)

[0423] Block 392 includes generating a set of different (e.g., 144, etc.) colors (or pixel / codeword values) for a current color table or current color table image based on a set (e.g., 144, etc.) of random numbers generated from a Beta distribution. Each random number, denoted x (where 0≤x≤1), may be scaled back or converted to a corresponding per-channel pixel or codeword value (referred to as a “PQ value”) within a PQ value range [P0, P1] as follows:

[0424] x PQ =x(P1-P0)+P0 (98)

[0425] A set of PQ values ​​generated from scaling or converting the set of random numbers back into the PQ value range can then be generated from the set of random numbers and used as a set of per-channel codeword values ​​or codewords for the set of colors in the current color table. In some operating scenarios, for each color table, the same Beta distribution (with the same mean and shape coefficient), for example, with three different sets of random numbers, is used to generate per-channel codewords for each of a plurality of (e.g., RGB, etc.) channels of a color space in which the display image is to be rendered by the reference image display. Thus, the average color generated from the averaged colors represented in all color patches in the color table is close to a neutral color or gray.

[0426] Block 394 includes generating a current color table or central block of the corresponding TIFF image. The current color table may include a set (e.g., 2D squares, etc.) of color blocks, each of which is a single color given by a specific (cross-channel or composite) pixel or codeword value having three per-channel pixel or codeword values. Three sets of per-channel codewords generated from the same Beta distribution (but with three different sets of random numbers) may be used by the reference image display to drive rendering of the red, blue, and green color block channels in the set of color blocks in the current color table or current TIFF image, respectively.

[0427] Additionally, optionally or alternatively, the RGB value of the background color of the background used to specify or generate the color table image may be set to the PQ mean μ PQ , from which the scaled mean of the Beta distribution is obtained.

[0428] Block 396 includes determining whether the current combination of the scaling mean and the shape coefficient is the last combination of all unique combinations of the scaling mean and the shape coefficient. If so, the process flow ends to generate a plurality of different color tables or different color table images. Otherwise, the process flow returns to block 388.

[0429] By way of illustration and not limitation, different colors or color blocks may be generated for each color table as n c The total number of colors or color patches generated from multiple different color tables and color table images can be given as: Colors or color blocks that can be used to determine the correspondence / matching relationship between SDR color blocks or codewords captured by a first video capture device in an SDR capture mode and corresponding HDR color blocks or codewords captured by a second video capture device in an HDR capture mode.

[0430] In some operational scenarios, multiple TIFF images each containing multiple color tables may be iteratively displayed or rendered on a reference image display such as a PRM TV at a constant playback frame rate and may be captured in SDR and HDR images, respectively, by a first and second video capture device. Since the illumination of the reference image display may vary with viewing angle, the SDR and HDR images may be captured separately by a first and second video capture device (e.g., two phones, etc.) placed at the same position and orientation relative to or with reference to the reference image display. Thus, SDR and HDR video signals or bitstreams containing the captured SDR and HDR images of the color tables or TIFF images as rendered on the reference image display may be generated by the first and second devices or cameras, respectively. Since the playback frame rate of the reference image display may remain constant, it may be relatively easy to determine the number of frames of the color table to establish an image pair in which the SDR image captured by the first video capture device and the HDR image captured by the second video capture device depict the same color table of the multiple color tables. The captured SDR and HDR images may be represented in SDR and HDR YCbCr (or YUV) color spaces, such as in YUV image / video files.

[0431] Figure 3F An example process flow for matching SDR and HDR colors between an image pair of captured SDR and HDR images for each color table in a plurality of color tables is illustrated. To extract corresponding colors of individual color patches from captured SDR and HDR images of color table images containing color tables, the captured SDR and HDR images may be converted to the same layout as the (original) color table images displayed on a reference image display.

[0432] Block 3002 includes receiving captured (SDR or HDR) checkerboard images that may be obtained by a camera of one of the first and second video capture devices from a displayed checkerboard image displayed on a reference image display. Figure 3F 3002 to 3006 of the same process flow of FIG. 3004 , these captured checkerboard patterns are obtained by a camera of the other of the first and second video capture devices from the same displayed checkerboard image displayed on the reference image display.

[0433] Block 3004 includes detecting checkerboard corners from a checkerboard image captured by a camera of the device.

[0434] Box 3006 includes calculating or calibrating the camera distortion coefficients of the camera using the chessboard angles detected from the captured chessboard image. The 3D (reference) coordinates of the chessboard image displayed on the reference image display may first be determined in a 3D coordinate system fixed to the reference image display. Camera parameters such as the distortion coefficients and other intrinsic parameters of the camera used by the device to generate the captured chessboard image may be calculated as optimized values ​​that generate the best mapping (e.g., minimum error or minimum mismatch, etc.) between the 3D reference coordinates of the chessboard angles in the captured chessboard image and the 2D (image) coordinates of the chessboard angles. The calibration process may be performed in a YUV or RGB color space, and the displayed chessboard image or the captured chessboard image may be converted to or represented in the color space.

[0435] Figure 2M and Figure 2N Two example checkerboard images detected from captured (HDR and SDR, respectively) images of the checkerboard image are illustrated. Multiple (e.g., 100, etc.) captured checkerboard images can be used to determine, calculate, or calibrate distortion coefficients or other intrinsic parameters of respective cameras used in first and second video capture devices to achieve relatively high accuracy (e.g., within half a pixel for camera reprojection error, etc.) in projecting images captured by respective cameras of a displayed image on a reference image display onto a displayed image on a reference image display.

[0436] The (camera-specific) distortion coefficients and intrinsic parameters of the cameras of the first and second video capture devices obtained in processing blocks 3002 to 3006 may then be used to analyze or correlate the captured SDR and HDR images from TIFF images containing corresponding color tables as follows.

[0437] Block 3008 includes receiving a captured (SDR or HDR) image containing a color table and a TIFF image of a checkerboard corner (or a checkerboard pattern in a corner) (displayed on a reference image display). The captured (SDR or HDR) image may be captured by a camera, having previously captured a checkerboard image (which may not contain a color table) in block 3006 to pre-obtain distortion coefficients for these images.

[0438] Block 3010 includes correcting or de-distorting the captured image of the displayed TIFF image, for example using distortion coefficients obtained during a camera calibration process such as block 3006, to correct for camera lens distortion of the camera.

[0439] Block 3012 includes detecting checkerboard corners in the captured image. The checkerboard corners are captured from checkerboard corners (eg, four corners thereof, etc.) in a displayed TIFF image on a reference image display, which may be a PPM TV.

[0440] Block 3014 includes estimating a projective transformation between image coordinates of the captured image and image coordinates of an original TIFF image displayed on a reference image display and captured in the received captured image.

[0441] Block 3016 includes using the estimated projective transformation to correct the captured image to the same layout as the original TIFF image.With the estimated projective transformation, the captured image can be corrected to the same spatial layout as the original TIFF image.

[0442] Fig.2O The diagram shows (a) a captured image from a (displayed) TIFF image, (b) a corrected captured image generated by correcting the captured image using camera distortion correction and a projective transformation, and (c) an original TIFF color map image displayed on a reference image display. Fig.2O (a) The de-distortion or correction operation performed on the captured image generates Fig.2O (b) is the corrected captured image, so Fig.2O Some pixels in the rectified image of (b) may be undefined.

[0443] Block 3018 includes locating and extracting (a set of) individual color blocks from the color table in the corrected captured image. As used herein, a color block having a single corresponding (e.g., RGB, YCbCr, composite, etc.) pixel or codeword value is specified. Corresponding individual codewords that respectively specify individual color blocks in the captured image can be determined based on the pixel or codeword values ​​of the pixels in these individual color blocks in the corrected captured image. In addition, individual original codewords that respectively specify individual original color blocks in the original TIFF image (corresponding to individual color blocks of the corrected captured image) can be determined from the RGB or YUV file of the original TIFF image.

[0444] The correspondence / mapping relationship between SDR color blocks or codewords extracted from the (corrected) captured SDR image of the original color table image and HDR color blocks or codewords extracted from the (corrected) captured HDR image of the same original color table image can be established partially or entirely based on separate original color blocks or codewords in the original TIFF image (from which both the (corrected) SDR and HDR images are obtained).

[0445] Using multiple TIFF images, multiple SDR images captured from displayed TIFF images by a first video capture device in SDR capture mode, and multiple HDR images captured from the same displayed TIFF images by a second video capture device in HDR capture mode, multiple correspondences or mapping relationships between a set of SDR colors and a set of HDR images can be established.

[0446] Several methods can be used to map an SDR image to an HDR image.

[0447] In the first method, as in some "BB1" operating scenarios, in some "BB2" operating scenarios, mapped SDR and HDR color patches or colors can be used to generate (e.g., static, etc.) SDR to HDR reshaping operation parameters such as TPB coefficients. These TPB coefficients can be combined with TPB basis functions using the SDR codewords of the SDR image as input parameters to predict the corresponding HDR codewords of the reconstructed HDR image. The static 3D-LUT can be pre-built and deployed to a video capture device (e.g., a first video capture device, etc.) for mapping the SDR image captured by the first video capture device to a reconstructed HDR image that simulates the HDR appearance of the second video capture device. Since the HDR color patches or codewords used to generate the reshaping operation parameters are extracted from the captured (training) HDR image of the second video capture device, the predicted HDR codewords generated with the reshaping operation parameters are likely to provide a mapped HDR appearance similar to the actual HDR appearance of the actual HDR image captured by the second video capture device in the reconstructed HDR image.

[0448] In a second method, in some "BB2" operating scenarios, non-TPB optimization can be used to generate non-TPB reshaping operation parameters, which are used in the non-TPB reshaping operation to map the captured SDR image obtained by the first video capture device to a reconstructed HDR image that simulates the HDR appearance of the second video capture device. The 3D-LUT for the non-TPB reshaping operation can be directly constructed without the relatively high continuity and smoothness supported by the TPB reshaping operation. The non-TPB second method provides a relatively high degree of design freedom and can be relatively flexible, even if different colors represented in nearby 3D-LUT entries / nodes may or may not have relatively high continuity and smoothness. In some operating scenarios, with relatively high degrees of design freedom, the non-TPB second method can be more suitable than the first method for dual-device or "BB2" operating scenarios, in which the SDR image from one device is to be mapped to an HDR image that simulates the HDR appearance of another device.

[0449] A non-TPB second method may utilize a 3D mapping table (3DMT) to construct a reverse reshaping map or a reverse lookup table (BLUT) representing the reverse reshaping map. The BLUT may be used to map SDR (e.g., across channels, three channels, etc.) codewords of an SDR image to HDR (e.g., chroma channels, per chroma channels, etc.) codewords of a reconstructed HDR image. Example operations for constructing a reverse reshaping map such as a BLUT from a 3DMT may be found in U.S. Patent Application Serial No. 17 / 054,495 filed on May 9, 2019, the contents of which are incorporated herein by reference in their entirety as if fully set forth herein.

[0450] For illustration purposes only, the SDR codewords may be represented in three dimensions or channels Y, Cb, Cr in the SDR YCbCr color space. A 3D mapping table may be generated from a 3D histogram having a plurality of 3D histogram bins. The plurality of 3D histogram bins may correspond to bins that are spaced by three positive integers Q. y , Q Cb , Q Cr The corresponding positive integers in the color space divide each dimension or channel into multiple color space partitions, resulting in a total of (Q y ×Q Cb ×Q Cr ) 3D histogram bins.

[0451] Multiple 3D histogram bins (denoted as Ω) of a 3D histogram Q,s Each 3D histogram bin (denoted as q) in the SDR (YCbCr) color space may be assigned a corresponding bin index or three corresponding quantized channel values ​​q=(q y ,q Cb ,q Cr ), and stores the pixel counts of all SDR color blocks or codewords within the SDR color space partitions represented by the 3D histogram bins. All bin indices (or quantized channel values) for multiple 3D histogram bins may be collected into a set of bin index values ​​denoted as Q, where

[0452] Additionally, the sum of the HDR codewords mapped to the SDR codewords in each of the plurality of 3D histograms may be calculated and stored for the 3D histogram bin. and Represents the sum of HDR codewords (also referred to as “mapped HDR luma and chroma values”).

[0453] An example procedure for generating SDR pixel counts and mapped HDR luminance and chrominance values ​​for 3D histogram bins of a histogram is illustrated in Table 2 below.

[0454] Table 2

[0455]

[0456] let represents the (representative) SDR codeword at the center of the qth 3D histogram bin. The representative SDR codewords for all 3D histogram bins may be fixed for all SDR images / frames and pre-computed with an example procedure as shown in Table 3 below.

[0457] Table 3

[0458]

[0459] Next, 3D histogram bins having non-zero (SDR) pixel counts may be identified and retained among the plurality of 3D histogram bins, wherein all other 3D histogram bins having zero (SDR) pixel counts are discarded or removed from the plurality of 3D histogram bins.

[0460] Let q0,q1,…q k-1 represents all k 3D histogram bins, each with non-zero (SDR) pixel counts or Where k is a positive integer. For example, using the procedure shown in Table 4 below, the mapped HDR luminance and chrominance values ​​may be calculated for each of the k 3D histogram bins with non-zero (SDR) pixel counts by dividing the sum of the HDR luminance and chrominance values ​​mapped by the SDR codeword for the SDR pixel by the number of SDR pixels in the 3D histogram bin and the pixel count. and The average value of .

[0461] Table 4

[0462]

[0463] A 3D-LUT may be constructed from a 3DMT represented by a 3D histogram storing corresponding SDR pixel counts and corresponding mapped HDR codeword values ​​in 3D histogram bins. Each node / entry in a 3D-LUT or 3DMT may be constructed with a specific SDR codeword (q) represented in each of the k 3D histogram bins. y ,q Cb ,q Cr ) and is calculated for the 3D histogram bins as in Table 4 above. Additionally, optionally or alternatively, for those 3D histogram bins with zero SDR pixel counts, the nearest or neighboring 3D bin with the closest bin index may be used to fill in the missing mapped HDR values, for example by trilinear interpolation.

[0464] As described above, example operations for constructing a 3D-LUT from a 3DMT for use as a non-TPB inverse reshaping map or BLUT (e.g., for predicting HDR chroma codewords per chroma channel from cross-channel SDR codewords) are described in the aforementioned U.S. patent application serial number 17 / 054,495.

[0465] In some "BB2" operating scenarios, a luma reverse reshaping map may be used to reverse reshape the SDR luma codewords of an SDR image into predicted or reverse reshaped HDR luma codewords of a corresponding HDR image using a GPR-based model. Example generation of luma reshaping maps using a GPR-based model may be found in U.S. Provisional Patent Application Serial No. 62 / 887,123, "Efficient user-defined SDR-to-HDR conversion with model templates," filed by Guan-Ming Su and Harshad Kadu on August 15, 2019, and PCT Application Serial No. PCT / US2020 / 046032, filed on August 12, 2020, the contents of which are incorporated herein by reference in their entirety as fully set forth herein. For example, a CDF matching curve may be generated from each of a plurality of training SDR-HDR image pairs. Using a set (e.g., 15, etc.) of uniformly sampled SDR points within an SDR codeword range (e.g., the entire SDR codeword range, etc.), a corresponding mapped HDR codeword can be found in each of the CDF matching curves. For each sample SDR point in the set of uniformly sampled SDR points, a GPR model can be constructed based on a histogram with multiple (e.g., 128, etc.) luma bin histograms and used to generate a corresponding luma reshaping map.

[0466] TPB optimization in editing

[0467] Image / video editing is a common application in video capture devices such as mobile devices to allow users to adjust color, contrast, brightness or other user preferences. Image / video editing can be performed or executed on the encoder side before the captured images are compressed into a (compressed) video signal or bitstream. Additionally, optionally or alternatively, image / video editing can be performed or executed on the decoder side after the video signal bitstream is decoded or decompressed.

[0468] Image / video editing operations as described herein may be performed in either or both of the HDR domain and the SDR domain. Image / video editing operations in the HDR domain may be performed relatively easily. For example, after editing HDR content or images, the edited HDR content or images may be passed as input or reference HDR images to a video pipeline that generates corresponding reshaped or ISP SDR content or images to be encoded in a video signal or bitstream. In contrast, image / video editing operations in the SDR domain may be relatively challenging. Although HDR to SDR and / or SDR to HDR mappings may be designed or generated to ensure or enhance reversibility between the SDR domain and the HDR domain, image / video editing operations performed with SDR images to be encoded in a video signal or bitstream may result in difficulty or challenging in reversing or reverse reshaping the edited SDR images to generate a reconstructed HDR image that is similar to the reference HDR image.

[0469] Figure 3G An example image / video editing operation performed on the encoder side with an upstream device such as a video capture device or a video encoder is illustrated. As shown, an input HDR image may be represented in an input HDR domain or color space (denoted as "HLG YCbCr"). A TPB-based forward reshaping operation (denoted as "Forward TPB") may be performed to forward reshape the input HDR image into a forward reshaped SDR image represented in a forward reshaped SDR domain or color space (denoted as "SDR YCbCr"). An image / video editing operation (denoted as "RGB domain editing") may be performed to edit the forward reshaped SDR image to generate an edited SDR image represented in a forward reshaped SDR domain or color space (denoted as "Edited SDR YCbCr") after editing. The edited SDR image represented in the forward reshaped SDR domain or color space after editing may be compressed or encoded into a video signal or bitstream using one or more video codecs of an upstream device.

[0470] Figure 3HAn example image / video editing operation performed on the decoder side with a downstream receiving device or video decoder is illustrated. As shown, a video signal or bitstream is decoded by one or more video codecs of a downstream device into an SDR image represented in an SDR domain or color space (denoted as "SDR YCbCr"). An image / video editing operation (denoted as "RGB domain editing") may be performed to edit the SDR image so as to generate an edited SDR image represented in a forward reshaped SDR domain or color space after editing (denoted as "edited SDR YCbCr"). A TPB-based reverse reshaping operation (denoted as "reverse TPB") may be performed on the edited SDR image represented in the forward reshaped SDR domain or color space after editing to generate a reverse reshaped or reconstructed HDR image in a reconstructed HDR domain or color space ("PQ YCbCr").

[0471] like Figure 3G and Figure 3H As shown in , SDR images encoded into a video signal at the encoder side or decoded from a video signal can be represented in an SDR domain or color space such as an SDR YCbCr domain or color space (e.g., after forward reshaping, etc.), with which the video codec can be programmed or developed to perform image processing operations relatively efficiently. By way of example and not limitation, image / video editing operations can occur in the SDR RGB domain or color space, such as Figure 3G and Figure 3H RGB to YCbCr conversion or YCbCr to RGB conversion may be performed by conversion such as SMPTE range conversion (eg, as specified by a standard, etc.).

[0472] As more colors are squeezed into the SDR domain or color space, codewords in a forward reshaped domain or color space, such as an SDR YCbCr color space, can often exceed a codeword value range, such as a SMPTE range, in an SDR RGB color space in which image / video editing operations are performed. Applying a YCbCr to RGB conversion, which converts YCbCr codewords in the YCbCr codeword value range (normalized within the value range [0,1]) of the SDR YCbCr color space to RGB codewords of the SDR RGB color space, can cause some RGB codewords to exceed or exceed the SMPTE range (normalized within the value range [0,1]) of the SDR RGB color space. Although these out-of-range codeword values ​​can be clipped, the clipped codeword values ​​may not be able to be restored to the non-clipped original HDR codeword in a reverse reshaping operation (e.g., based on TPB, etc.). Additional operations as described herein can be used to enhance reversibility and reduce or avoid visual artifacts in image / video editing applications.

[0473] FIG. 3I to FIG. 3M Several example solutions for clipping out-of-range codewords in image / video editing applications are illustrated.

[0474] In some operational scenarios, such as Fig. 3I As illustrated in , out-of-range SDR RGB codewords generated from a YCbCr to RGB conversion (denoted as "YCbCr to RGB clipping conversion") performed on an SDR YCbCr image at the encoder or decoder side may be clipped (e.g., all codewords greater than one (1) are hard clipped to 1, all codewords less than zero (0) are hard clipped to 0, etc.) to a valid or specified codeword value range such as [0,1]. Image / video editing operations ("editing in the RGB domain") may then be performed with the SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space. The edited SDR RGB codewords in the SDR RGB color space may be converted and clipped to converted edited SDR YCbCr codewords in a valid or specified codeword value range such as [0,1] of a converted edited SDR YCbCr color space (denoted as "edited SDR YCbCr") by an RGB to YCbCr conversion (denoted as "RGB to YCbCr clipping conversion").

[0475] In some operational scenarios, such as Figure 3JAs illustrated in , the out-of-range SDRRGB codewords generated from the YCbCr to RGB conversion performed on the SDR YCbCr image at the encoder or decoder side (denoted as "YCbCr to RGB unclipping conversion") are not clipped to a valid or specified codeword value range such as [0,1]. The image / video editing operation ("editing in the RGB domain") can then be performed with the unclipped (input) SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr to RGB conversion performed with the unclipping operation allows the original information in the (input) SDR RGB codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation accepts all possible real codeword values ​​in the input SDR RGB codeword (including less than 0 or greater than 1, for example in a limited but wider range of values ​​than [0 1]). Therefore, some information in the input SDR YCbCr codeword can be retained in the input SDR RGB codeword and the edited SDR RGB codeword. An SDR RGB boundary operation (denoted as "RGB boundary [0 1] shearing") can be performed (e.g., optionally based on a user preference of a user operating a video editing application) to shrink or squeeze the edited SDR RGB codeword outside the valid or specified range [0,1] back to the inside of the valid or specified range [0,1], thereby generating a scaled edited SDR RGB codeword within the valid or specified range [0,1]. The scaled edited SDR RGB codeword in the SDR RGB color space can be converted and sheared into a converted scaled edited SDR YCbCr color space (denoted as "edited SDR YCbCr") in a valid or specified codeword value range such as [0,1] by an RGB to YCbCr conversion (denoted as "RGB to YCbCr shear conversion").

[0476] In some operational scenarios, such as Figure 3KAs illustrated in , the out-of-range SDRRGB codewords generated from the YCbCr to RGB conversion performed on the SDR YCbCr image at the encoder or decoder side (denoted as "YCbCr to RGB unclipping conversion") are not clipped to a valid or specified codeword value range such as [0,1]. The image / video editing operation ("editing in the RGB domain") can then be performed with the unclipped (input) SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr to RGB conversion performed with the unclipping operation allows the original information in the (input) SDR RGB codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation accepts all possible real codeword values ​​in the input SDR RGB codeword (including less than 0 or greater than 1, for example in a limited but wider range of values ​​than [0 1]). Therefore, some information in the input SDR YCbCr codeword can be retained in the input SDR RGB codeword as well as in the edited SDR RGB codeword. Figure 3J The differences shown in the figure are Figure 3K In the operation scenario illustrated in , SDR RGB boundary operations may not be performed. The edited SDR RGB codewords in the SDR RGB color space may be converted and clipped to converted edited SDR YCbCr codewords in a converted edited SDR YCbCr color space (denoted as "Edited SDR YCbCr") in a valid or specified codeword value range such as [0,1] by an RGB to YCbCr conversion (denoted as "RGB to YCbCr clipping conversion").

[0477] In some operational scenarios, such as Figure 3LAs illustrated in , the out-of-range SDRRGB codewords generated from the YCbCr to RGB conversion performed on the SDR YCbCr image at the encoder or decoder side (denoted as "YCbCr to RGB unclipping conversion") are not clipped to a valid or specified codeword value range such as [0,1]. The image / video editing operation ("editing in the RGB domain") can then be performed with the unclipped (input) SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr to RGB conversion performed with the unclipping operation allows the original information in the (input) SDR RGB codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation accepts all possible real codeword values ​​in the input SDR RGB codeword (including less than 0 or greater than 1, for example in a limited but wider range of values ​​than [0 1]). Thus, some information in the input SDR YCbCr codewords may be retained in the input SDR RGB codewords as well as in the edited SDR RGB codewords. An SDR RGB boundary operation (denoted as "RGB Boundary Clipping 3D-LUT") may be performed, for example, optionally based on user preferences of a user operating a video editing application to shrink or squeeze edited SDR RGB codewords outside the (3D) boundary supported by the TPB-based reshaping operations back inside the boundary, thereby generating scaled edited SDR RGB codewords within the boundary supported by the TPB-based reshaping operations or well-defined in these reshaping operations. The boundaries supported by the TPB-based reshaping operations may not be regular shapes or cubes that are simply defined with a fixed value range [0,1]. The scaled edited SDR RGB codewords in the SDR RGB color space can be converted and sheared into converted scaled edited SDR YCbCr codewords in a valid or specified codeword value range such as [0,1] in the SDR RGB color space (denoted as "Edited SDR YCbCr") through an RGB to YCbCr conversion (denoted as "RGB to YCbCr shear conversion").

[0478] The (input) SDR YCbCr image may be obtained from the original HDR image by forward reshaping or ISP processing with a known (e.g., white box, ISP, etc.) HDR to SDR forward transform or mapping, and the boundaries may be determined and represented as a (TPB boundary clipping) 3D-LUT based in part or in whole on the HDR to SDR forward or mapping and / or any applicable color space conversion matrix. For example, using full grid sampling data covering the entire HDR domain or color space in which the original HDR image is represented, the boundary pixel or codeword values ​​in the forward reshaping or ISP SDR color space may be determined.

[0479] After an image / video editing operation ("editing in the RGB domain"), the edited SDR RGB codewords may be scaled or squeezed (or irregularly sheared) into a 3D shape outlined or enclosed by a boundary. In some operational scenarios, since the converted scaled edited SDR YCbCr codewords are already placed within corresponding boundaries in the SDR YCbCr color space supported by the TPB-based inverse reshaping operation, the scaled edited SDR RGB codewords in the SDRRGB color space may be converted into converted scaled edited SDR YCbCr codewords by an RGB to YCbCr conversion ("RGB to YCbCr shearing conversion") without further shearing by an RGB to YCbCr conversion ("RGB to YCbCr shearing conversion"). In such scenarios, Figure 3J In these operation scenarios illustrated in , a maximized number of colors can be retained in the converted, scaled, edited SDR YCbCr codewords and in the inverse reshaped HDR codewords generated from the converted, scaled, edited SDR YCbCr codewords by the TPB-based inverse reshaping operation while avoiding the generation of color artifacts.

[0480] In some operational scenarios, such as Figure 3MAs illustrated in , the out-of-range SDRRGB codewords generated from the YCbCr to RGB conversion performed on the SDR YCbCr image at the encoder or decoder side (denoted as "YCbCr to RGB unclipping conversion") are not clipped to a valid or specified codeword value range such as [0,1]. The image / video editing operation ("editing in the RGB domain") can then be performed with the unclipped (input) SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr to RGB conversion performed with the unclipping operation allows the original information in the (input) SDR RGB codeword to be preserved in the (input) SDR RGB codeword received by the image / video editing operation. The image / video editing operation accepts all possible real codeword values ​​in the input SDR RGB codeword (including less than 0 or greater than 1, for example in a limited but wider range of values ​​than [0 1]). Therefore, some information in the input SDR YCbCr codeword can be retained in the input SDR RGB codeword and the edited SDR RGB codeword. The edited SDR RGB codeword in the SDR RGB color space can be converted and sheared into a converted edited SDR YCbCr codeword in a valid or specified codeword value range such as [0,1] in a converted edited SDR YCbCr color space (denoted as "Edited SDR YCbCr") by an RGB to YCbCr conversion (denoted as "RGB to YCbCr shear conversion"). The converted edited SDR YCbCr codewords in the converted edited SDRYCbCr color space ("edited SDR YCbCr") may be cropped, for example, optionally based on user preferences of a user operating a video editing application by SDR YCbCr boundary operations (denoted as "YCbCr Boundary Cropping 3D-LUT"), which may be performed to shrink or squeeze the converted edited SDR YCbCr codewords outside the (3D) boundary supported by the TPB-based reshaping operations back into the boundary, thereby generating scaled converted edited SDR YCbCr codewords in the scaled converted edited SDR YCbCr color space (denoted as "edited SDRYCbCr") within the boundary supported by the TPB-based reshaping operations or well-defined in these reshaping operations. The boundaries supported by the TPB-based reshaping operations may not be regular shapes, such as cubes, whose sides are simply defined as a valid or specified (normalized) range [0,1].

[0481] The (input) SDR YCbCr image may be obtained from the original HDR image by forward reshaping or ISP processing with a known (e.g., white box, ISP, etc.) HDR to SDR forward transform or mapping, and the boundaries may be determined based on the HDR to SDR forward or mapping and / or any applicable color space conversion matrix and represented with a (TPB boundary clipping) 3D-LUT. For example, using full grid sampling data covering the entire HDR domain or color space in which the original HDR image is represented, the boundary pixel or codeword values ​​in the forward reshaping or ISP SDR color space may be determined.

[0482] Boundary Clipping 3D-LUT Construction and Clipping

[0483] HDR to SDR forward reshaping or mapping such as TPB-based forward reshaping may be a non-linear function. Although the input HDR codewords in the input HDR domain or color space may be well organized in a simple 3D cube, unlike a 3D cube, the boundaries of the mapped SDR codewords generated from applying a (non-linear or TPB) HDR to SDR mapping may be relatively irregular.

[0484] In order to perform (TPB) boundary clipping for relatively irregular boundaries, a (TPB) boundary clipping 3D-LUT may be constructed. The 3D-LUT may be searched with an SDR codeword as input (or a lookup key) and the SDR codeword may be returned as a value in response to determining that the SDR codeword is within the boundary. Otherwise, in response to determining that the SDR codeword is outside the boundary, the 3D-LUT may return a clipped SDR codeword different from the original SDR codeword, the clipped SDR codeword being within the boundary.

[0485] In some operating scenarios, boundary clipping can be implemented in two levels. In the first level, regular clipping is performed with a range defined by the minimum and maximum values ​​(or lower and upper limits) that define the (3D) codeword range. In the second level, irregular clipping is performed with a (TPB) boundary clipping 3D-LUT.

[0486] Figure 3N An example process flow for constructing a boundary clipping 3D-LUT (eg, TPB, etc.) is shown. Figure 3N The process flow shown in the left side is to construct a 3D boundary using the alphaShape technique. Figure 3N The process flow shown on the right side is to use the constructed 3D boundary to generate a 3D-LUT for irregular clipping.

[0487] Block 3022 includes preparing a 3D uniform sampling grid or set of sample values ​​in an input HDR domain or color space, such as an R.2020 (container) domain or color space. (eg, as given in equations (42) and (43) above, etc.) By way of illustration and not limitation, the HDR domain or color space represents the HDR YCbCr color space HLG.

[0488] The sampled values ​​in the R.2020YCbCr color space HLG may be converted to corresponding values ​​in the R.2020RGB color space HLG, as indicated in the above expression (44). The converted values ​​in the R.2020RGB color space HLG may be further converted to an optimized value (a) of the parameter a in the a RGB color space HLG. opt ) corresponding values, as indicated in the above expression (45). The converted values ​​in (a) RGB color space HLG can be clipped, as indicated in the above expression (46), and these converted values ​​can be converted to corresponding clipped values ​​in R.2020RGB color space HLG, as indicated in the above expression (47). The clipped values ​​in R.2020RGB color space HLG obtained by the above expression (47) can be converted to corresponding clipped values ​​in R.2020YCbCr color space HLG As indicated in the above expression (48).

[0489] Box 3024 includes the clipped or constrained (HDR YCbC HLG) values (Refers to Figure 3N The "constrained input" in Figure 3N More specifically, the optimized forward TPB coefficients for TPB-based forward reshaping are related to the forward generation matrix (in the above expression (50)) together or multiplied, the forward generation matrix is ​​the clipped HDR value in the R.2020YCbCr color space HLG is constructed as the input parameter of the forward TPB basis function to obtain The SDR or R.709YCbCr values ​​after mapping or forward reshaping are as follows:

[0490]

[0491] Block 3046 includes converting the representation to The R.709YCbCr values ​​to generate SDR or R.709RGB values ​​in the unclipped RGB domain or color space are as follows:

[0492]

[0493] R.709RGB values, such as converted from R.709YCbCr values, may include values ​​outside a valid or specified range, such as the value range [0, 1].

[0494] For each channel, the minimum and maximum values ​​among the R.709RGB values ​​given in expression (100) may be measured or determined as follows:

[0495]

[0496] These extreme values ​​may be used as lower and upper bounds in block 3030 to construct or prepare a uniformly sampled 3D grid or set of sample values ​​in the unclipped RGB domain or color space.

[0497] Figure 2P An example distribution of R.709RGB values ​​(in an SDR RGB image) as converted from R.709YCbCr values ​​is illustrated. As shown, the distribution of R.709RGB values ​​in the SDR RGB image has an irregular shape instead of a 3D cube. The irregular shape with a 3D boundary represents the maximum supported SDR RGB color space that can be mapped back to or inversely reshaped into a reconstructed HDR color in a reconstructed HDR domain or color space without losing information. Any SDR RGB color outside of this irregular shape or range will suffer information loss in the inverse reshaping operation.

[0498] After performing an image / video editing operation on an SDR RGB image, the resulting (3D) codeword range or distribution of the edited codewords or colors may be wider (e.g., much wider, etc.) than the irregular shape in the SDR RGB color space, resulting in many SDR RGB codewords that are not defined or supported by the inverse reshaping operation.

[0499] Additionally, it may be difficult to characterize, represent, or approximate the actual 3D boundaries of irregular shapes using analytical equations or multiple 2D planes used to cut a 3D cube in RGB color space into the irregular shapes.

[0500] As described above, a two-level clipping solution can be used to clip the edited codeword back into the maximum supported RGB color space as represented in the irregular shape. More specifically, at the first level, by Regular clipping can be applied by clipping the edited codeword within a fixed 3D range. Regular clipping can utilize a relatively simple clipping function based on if-else or min() / max() operations to reduce the (input) edited codeword range to a range consisting of A 3D cube (including a rectangle) defined or bounded. At the second level, irregular shearing is applied by shearing those codewords (among the codewords after simple shearing edit) that are still outside the irregular shape into the maximum supported SDR RGB color space outlined or specified by the irregular shape. Since the relatively irregular shearing boundaries of the maximum supported SDR RGB color space are not suitable for relatively simple shearing operations with analytical equations, simple codes or operations, the boundary shearing 3D-LUT can be used to perform irregular shearing at the second level of the two-level shearing solution.

[0501] Figure 3N Block 3028 includes: Given a set of SDR RGB codewords In the case of , an alphaShape can be constructed using an available alphaShape construction tool such as the alphaShape() MATLAB function. As used herein, an alphaShape is a shape that encompasses a set of 2-D or 3-D points (such as the set of SDR RGB codewords in the current example). ) is a delimited area or volume.

[0502] The alphaShape object created to represent the alphaShape can be manipulated to summarize the set of SDR RGB codewords The bounding polygon of the SDR RGB codeword is tightened or relaxed The surrounding alphaShape is for example a fit to non-convex areas. Additional points may be added or removed, for example to suppress holes or areas or simply to suppress the alphaShape.

[0503] Let f be the function used to construct alphaShape. αS (Φ, r α ), where r α Represents the radius parameter.

[0504] By using this set of SDR RGB codewords Passed as the first input parameter to the function, the alphaShape can be constructed by the following alphaShape constructor: (R709) The bounding polyhedron of :

[0505]

[0506] Figure 2Q to Figure 2T The figure shows different r α The set of SDR RGB points or codewords Example alphaShape. As shown, the radius parameter r αLarger values ​​of r produce or construct larger polyhedra, which can result in coarser quantization if nearest neighbors outside the SDR RGB points or codewords are searched and used to fill in the clipped values ​​of these outer SDR RGB points. α Smaller polyhedra are generated or constructed, which results in a higher probability of forming holes and has less effective boundaries. The radius parameter r can be adjusted for different video editing applications. α Adjust to different values ​​(e.g., 0.1, 0.5, etc.) to maximize the effectiveness or coverage of supported SDR colors and avoid introducing visual artifacts.

[0507] alphaShape provides a bounding polygon as a clipping boundary for the irregular shape and allows determining whether a 3D point represented by an SDR RGB codeword is inside or outside the irregular shape.

[0508] Let I αS (αS, x) represents a binary function for determining whether a 3D point or SDR RGB codeword (denoted as x) is inside an irregular shape. Assume that a returned binary value of "1" indicates inside the irregular shape and a returned binary value of "0" indicates outside.

[0509] Let NN αS (αS, x) denotes an indexing function that receives a given query 3D point or SDR RGB codeword such as x as a second input parameter and returns the nearest neighbor of the query 3D point or SDR RGB codeword x in αS.

[0510] As described above, a boundary clipping 3D-LUT can be constructed in the SDR RGB domain or color space to clip any SDR RGB codewords outside the irregular shape representing the maximum supported color space through (TPB-based forward and reverse) reshaping operations.

[0511] Block 3030 includes constructing a boundary clipping 3D-LUT as a full-grid 3D-LUT having a plurality of nodes / entries. These nodes / entries in the 3D-LUT include a query SDR codeword as a lookup key and include a return SDR codeword for the query SDR codeword as a value.

[0512] For each color channel and OK (in Figure 3NThe extreme values ​​of the query SDR codewords represented or included in the 3D-LUT may form a full grid including 3D points sampled uniformly within the range of the 3D codewords defined by the extreme values ​​or lower / upper limits. For example, for each dimension or color channel represented by a corresponding axis in the R axis, G axis, B axis, the query SDR codewords in that dimension or color channel may be used as lower or upper limits. The codeword value range between can be uniformly sampled into three total partitions N corresponding to the three dimensions or color channels R 、N G and N B The corresponding total number of partitions is as follows:

[0513]

[0514] where i∈[0,...,N R -1], j∈[0,...,N G -1],k∈[0,...,N B -1]

[0515] Therefore, the total number of query SDR codewords represented in the 3D-LUT can be given as N u =N R N G N B .

[0516] Block 3032 includes beginning to execute a node processing loop for each node / entry in the 3D-LUT by selecting a current node / entry among multiple nodes / entries in the 3D-LUT (eg, in a sequential or non-sequential loop / iteration order, etc.).

[0517] For simplicity, (i, j, k) in the above expression (103) can be vectorized to p. The current node / entry can be the p-th node / entry among multiple nodes / entries in the 3D-LUT. The lookup key of the p-th node / entry can be represented by u on the LHS of the above expression (103). p Given the p-th query SDR RGB codeword represented by as the search key, let Represents the current output clipped SDR RGB codeword or value returned by the pth node / entry of the 3D-LUT.

[0518] Block 3034 includes checking or determining the query SDR RGB codeword u represented by the current or pth query SDR RGB codeword u of the current node / entry. p Whether the alphaShapeαS generated in box 3028 (R709)(or a shape constructed using the alphaShape constructor as shown in expression (102) above). In response to determining the query SDR RGB codeword u represented by the current or pth node / entry p In alphaShape αS (R709) Internally, process flow proceeds to block 3036. Otherwise, process flow proceeds to block 3040.

[0519] The plurality of nodes / entries in the 3D-LUT comprises a subset of nodes or entries, each of which has an alphaShape located at αS (R709) The query SDR RGB codewords represented within the query specify the corresponding lookup key. Therefore, the node or entry subset including the alphaShapeαS (R709) A node or entry within a .

[0520] Box 3040 includes alphaShapeαS (R709) Find the closest node / entry for the current node / entry among the subset of nodes / entries in . The closest node / entry has the closest query SDR RGB codeword represented As the lookup key, make the same (R709) The nearest represented query SDR RGB codeword is compared to all other represented query SDR RGB codewords of all other nodes / entries within the node or entry subset Having the query SDR RGB codeword u represented by the current or pth p The minimum distance between two pixels, as measured by Euclidean or non-Euclidean distance in SDR RGB color space.

[0521] In some operational scenarios, the index function NN mentioned above can be used αS (αS, x) gives the current or pth query SDR RGB codeword u represented by p The nearest queried SDR RGB codeword represented by as follows:

[0522]

[0523] The nearest query SDR RGB codeword represented Indicates the current or pth query SDR RGB codeword u p The boundary clipping value of is set to the return value of the current or pth node entry in the 3D-LUT, and the current or pth query SDR RGB codeword u represented by pis set as the lookup key for the current or p-th node entry in the 3D-LUT.

[0524] Block 3036 includes converting the current or pth represented query SDR RGB codeword u p Set to the return value of the current or p-th node entry in the 3D-LUT (in addition to being a lookup key) and determine whether the current or p-th node / entry is the last node or entry of the multiple nodes / entries in the 3D-LUT. In response to determining that the current or p-th node / entry is the last node or entry of the multiple nodes / entries in the 3D-LUT, process flow proceeds to block 3038. Otherwise, process flow returns to block 3032.

[0525] Block 3038 includes outputting the 3D-LUT as a final boundary clipping 3D-LUT in the SDR RGB color space (denoted as ).

[0526] The following table 5 illustrates the methods used to generate the final boundary cut The sample program.

[0527] Table 5

[0528]

[0529] Given the (final) boundary clipping In the case of , the aforementioned two-stage solution can be used to relatively efficiently perform boundary clipping on the (e.g., edited, etc.) SDR image. As described above, regular clipping can be performed to ensure that all (e.g., edited, etc.) SDR RGB codewords in the SDR image are within the extreme values ​​or upper / lower limits of the SDR RGB codewords in the SDR RGB color space in which the image / video editing operation has been performed on the SDR image. Regular clipping is performed and then the (if applicable, regularly clipped) SDR codeword is passed as a lookup key to the (final) boundary clipping. to perform irregular shearing, perform boundary shearing, and use the values ​​from the (final) boundary shearing The return value is used as the output (or other irregularly cropped, if applicable) SDR codeword in the output (e.g., after cropping and editing, etc.) SDR image.

[0530] In various operating scenarios (including but not limited to FIG. 3I to FIG. 3M ), regular and / or irregular cropping operations as described herein may be applied to video capture and / or editing applications to ensure maximum support for reconstructing HDR images from SDR images and preventing / reducing visual artifacts in the reconstructed HDR images.

[0531] Example process flow

[0532] Figure 4A An example process flow according to an embodiment of the present invention is illustrated. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, reverse mapping generation and application system, etc.) can perform this process flow. In block 402, a system as described herein constructs sampled HDR color space points distributed in a high dynamic range (HDR) color space. The HDR color space is parameterized by a color source scaling parameter having a candidate value selected from a plurality of candidate values. The color source scaling parameter is used to calculate the color space coordinates of at least one color source of a plurality of color sources that delineate the HDR color space.

[0533] In box 404, the system generates from these sampled HDR color space points in the HDR color space: (a) a reference standard dynamic range (SDR) color space point represented in a reference SDR color space, (b) an input HDR color space point represented in an input HDR color space, and (c) a reference HDR color space point represented in a reference HDR color space.

[0534] In block 406, the system executes a reshaping operation optimization algorithm to generate an optimized forward reshaping mapping and an optimized reverse reshaping mapping chain. The reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs.

[0535] In an embodiment, the optimized forward reshaping mapping is used to forward reshape the input HDR image in the input HDR color space into a forward reshaped SDR image in the forward reshaped SDR color space; and the optimized reverse reshaping mapping is used to reverse reshape these forward reshaped SDR images in the forward reshaped SDR color space into a reverse reshaped HDR image.

[0536] In an embodiment, the sampled HDR color space points are constructed in the HDR color space without using any image.

[0537] In an embodiment, a plurality of chains of optimized forward reshaping mappings and optimized reverse reshaping mappings are generated for a plurality of candidate values ​​of the chromatic aboriginal scaling parameter by the reshaping operation optimization algorithm; each of the plurality of chains of optimized forward reshaping mappings and optimized reverse reshaping mappings includes a corresponding optimized forward reshaping mapping and a corresponding optimized reverse reshaping mapping.

[0538] In an embodiment, the sampled HDR color space points are mapped to the reference SDR color space points based at least in part on a predefined HDR to SDR mapping.

[0539] In an embodiment, multiple sets of prediction errors are calculated for the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; each set of prediction errors in the multiple sets of prediction errors is calculated for corresponding chains in the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; and the multiple sets of prediction errors are used to select specific candidate values ​​from multiple candidate values ​​of the chromatic aboriginal scaling parameter.

[0540] In an embodiment, a particular candidate value of the chroma scaling parameter is used to generate a particular chain of a particular optimized forward reshaping map and a particular optimized reverse reshaping map.

[0541] In an embodiment, the specifically optimized forward reshaping map is represented in a forward reshaping three-dimensional lookup table.

[0542] In an embodiment, the specifically optimized inverse reshaping mapping is represented in an inverse reshaping three-dimensional lookup table.

[0543] In an embodiment, a video encoder applies the optimized forward reshaping mapping to an input HDR image sequence to generate a forward reshaping SDR image sequence and encodes the forward reshaping SDR image sequence into a video signal.

[0544] In an embodiment, a video decoder decodes a forward reshaped SDR image sequence from a video signal and applies the optimized inverse reshaping mapping to the forward reshaped SDR image sequence to generate a inverse reshaped HDR image sequence.

[0545] In an embodiment, a display image sequence derived from the inverse reshaped HDR image sequence is rendered on an image display operative with the video decoder.

[0546] In an embodiment, the HDR color space and the input HDR color space share a common white point.

[0547] In an embodiment, the reshaping operation optimization algorithm represents a Backward Error Subtraction for Signal Adjustment (BESA) algorithm.

[0548] Figure 4BAn example process flow according to an embodiment of the present invention is illustrated. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, reverse mapping generation and application system, etc.) can perform this process flow. In block 422, a system as described herein constructs sampled HDR color space points distributed in an HDR color space. The HDR color space is parameterized by a chromatic aboriginal scaling parameter having a candidate value selected from a plurality of candidate values. The chromatic aboriginal scaling parameter is used to calculate the color space coordinates of at least one chromatic aboriginal of a plurality of chromatic aboriginals that delineate the HDR color space.

[0549] In block 424, the system generates from the sampled HDR color space points in the HDR color space: (a) input SDR color space points represented in the input SDR color space and (b) reference HDR color space points represented in the reference HDR color space.

[0550] In block 426, the system executes a reshaping operation optimization algorithm to generate an optimized inverse reshaping map. The reshaping operation optimization algorithm receives as input the input SDR color space points and the reference HDR color space points.

[0551] In an embodiment, the inverse reshaping mapping is used to inversely reshape the SDR image in the input SDR color space into a inversely reshaped HDR image.

[0552] In an embodiment, the sampled HDR color space points are constructed in the HDR color space without using any image.

[0553] In an embodiment, a plurality of optimized inverse reshaping mappings are generated for a plurality of candidate values ​​of the chroma scaling parameter by the reshaping operation optimization algorithm; each optimized inverse reshaping mapping of the plurality of optimized inverse reshaping mappings comprises a corresponding optimized inverse reshaping mapping.

[0554] In an embodiment, multiple sets of prediction errors are calculated for the multiple optimized inverse reshaping mappings; each set of prediction errors in the multiple sets of prediction errors is calculated for a corresponding optimized inverse reshaping mapping in the multiple optimized inverse reshaping mappings; and the multiple sets of prediction errors are used to select a specific candidate value from a plurality of candidate values ​​of the chromatic aboriginal scaling parameter.

[0555] In an embodiment, the programmable ISP pipeline processes the sampled HDR color space points into the input SDR color space points based at least in part on optimized values ​​of programmable configuration parameters of the programmable ISP pipeline.

[0556] In an embodiment, optimized values ​​of programmable configuration parameters of the programmable ISP pipeline are determined by minimizing an approximation error between ISP SDR images generated from HDR images by the programmable ISP pipeline and reference SDR images generated by applying a predefined HDR to SDR mapping to these same HDR images.

[0557] Figure 4C An example process flow according to an embodiment of the present invention is illustrated. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, reverse mapping generation and application system, etc.) can perform this process flow. In block 442, a system as described herein extracts a set of SDR image feature points from a training SDR image and extracts a set of HDR image feature points from a training HDR image.

[0558] In block 444, the system matches a subset of one or more SDR image feature points in the set of SDR image feature points with a subset of one or more HDR image feature points in the set of HDR image feature points.

[0559] In box 446, the system uses the subset of the one or more SDR image feature points and the subset of the one or more HDR image feature points to generate a geometric transformation to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image.

[0560] In box 448, the system determines a set of pairs of SDR color patches and HDR color patches from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and the training HDR image have been spatially aligned by the geometric transformation.

[0561] In block 450, the system generates an optimized SDR-to-HDR mapping based at least in part on the set of pairs of SDR color patches and HDR color patches derived from the training SDR image and the training HDR image.

[0562] In block 452, the system applies the optimized SDR-to-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0563] In an embodiment, the training SDR image and the training HDR image are captured from a three-dimensional (3D) visual scene by capture devices operating in SDR and HDR capture modes, respectively.

[0564] In an embodiment, the training SDR image and the training HDR image form a pair of training SDR images and training HDR images among multiple pairs of training SDR images and training HDR images; and the optimized SDR to HDR mapping is generated at least in part based on multiple groups of paired SDR color blocks and HDR color blocks obtained from the multiple pairs of training SDR images and training HDR images.

[0565] In an embodiment, each SDR image feature point in the subset of the one or more SDR image feature points is matched with a corresponding HDR image feature point in the subset of the one or more HDR image feature points; and the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm.

[0566] In an embodiment, the common feature point extraction algorithm represents one of the following: binary robust invariant scalable key point algorithm; accelerated segmented test feature algorithm; KAZE algorithm; minimum eigenvalue algorithm; maximum stable extreme area algorithm; directional FAST and rotation algorithm; scale-invariant feature transformation algorithm; accelerated robust feature algorithm; and the like.

[0567] Figure 4D An example process flow according to an embodiment of the present invention is illustrated. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, reverse mapping generation and application system, etc.) can perform this process flow. In block 462, a system as described herein performs a corresponding camera distortion correction operation on each training image in a pair of training SDR images and training HDR images to generate a corresponding undistorted image in a pair of undistorted training SDR images and undistorted training HDR images.

[0568] In block 464, the system generates a corresponding projective transform in a pair of SDR image projective transforms and HDR image projective transforms using the corner pattern signatures detected from each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images.

[0569] In box 466, the system applies each of the projective transforms in the pair of SDR image projective transforms and HDR image projective transforms to a corresponding undistorted image in the pair of undistorted training SDR images and undistorted training HDR images to generate a corresponding rectified image in a pair of rectified training SDR images and rectified training HDR images.

[0570] In block 468, the system extracts a set of SDR color patches from the corrected training SDR image and a set of HDR color patches from the corrected training HDR image.

[0571] In block 470, the system generates an optimized SDR-to-HDR mapping based at least in part on the set of SDR color patches and the set of HDR color patches derived from the training SDR image and the training HDR image.

[0572] In block 472, the system applies the optimized SDR-to-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0573] In an embodiment, the training SDR image and the training HDR image are captured from a common color table image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively.

[0574] In an embodiment, the common color table image is selected from a plurality of color table images, each of the plurality of color table images comprising a different color block distribution arranged into a two-dimensional color table.

[0575] In an embodiment, the different color patch distributions are generated with random colors randomly selected from a common statistical distribution having a specific statistical mean and variance value combination.

[0576] In an embodiment, the common color table image is rendered on a screen of a common reference image display and is captured from the screen by the first capture device and the second capture device.

[0577] In an embodiment, the respective camera distortion correction operations are based at least in part on camera specific distortion coefficients generated from a camera calibration process performed with the camera used to acquire the training images.

[0578] In an embodiment, a three-dimensional mapping table (3DMT) is obtained using the set of SDR color patches and the set of HDR color patches; and the optimized SDR to HDR mapping is generated based at least in part on the 3DMT.

[0579] In an embodiment, the optimized SDR to HDR mapping represents one of the following: a TPB-based mapping or a non-TPB-based mapping.

[0580] In an embodiment, the optimized SDR to HDR mapping is one of: a static mapping applied to all non-training SDR images represented in the video signal, or a dynamic mapping generated based at least in part on a particular SDR codeword value distribution for one of the non-training SDR images represented in the video signal.

[0581] Figure 4EAn example process flow according to an embodiment of the present invention is illustrated. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, reverse mapping generation and application system, etc.) can perform this process flow. In block 482, a system as described herein constructs sampled HDR color space points distributed in an HDR color space for representing a reconstructed HDR image.

[0582] In box 484, the system converts the sampled HDR color space points to SDR color space points in a first SDR color space in which the SDR image to be edited by the editing device is represented.

[0583] In block 486, the system determines a bounding SDR color space rectangle based on extreme SDR codeword values ​​of the SDR color space points in the first SDR color space and determines an irregular 3D shape from the distribution of the SDR color space points.

[0584] In block 488, the system constructs sampled SDR color space points distributed in the bounding SDR color space rectangle in the first SDR color space.

[0585] The system generates a boundary clipping 3D-LUT using the sampled SDR color space points and the irregular shape in block 490. The boundary clipping 3D-LUT uses the sampled SDR color space points as lookup keys.

[0586] In block 492, the system performs a shearing operation on the edited SDR image in the first SDR color space based at least in part on the boundary-cropped 3D-LUT to generate a boundary-cropped edited SDR image in the first SDR color space.

[0587] In an embodiment, these cropping operations include first performing regular cropping on the edited SDR image using the delimiting SDR color space rectangle to generate a regularly cropped edited SDR image, and then performing irregular cropping on the regularly cropped edited SDR image using the 3D-LUT to generate the boundary cropped edited SDR image.

[0588] In an embodiment, a group of one or more SDR pixels in the SDR image to be edited is edited from one or more first brightness values ​​to one or more second brightness values ​​in the edited image; the one or more second brightness values ​​are different from the one or more first brightness values.

[0589] In an embodiment, a group of one or more SDR pixels in the SDR image to be edited is edited from one or more first chromaticity values ​​to one or more second chromaticity values ​​in the edited image; the one or more second chromaticity values ​​are different from the one or more first chromaticity values.

[0590] In an embodiment, image details depicted in the SDR image to be edited are removed in the edited SDR image.

[0591] In an embodiment, image details not depicted in the SDR image to be edited are added to the edited SDR image.

[0592] In an embodiment, the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; the lookup key is equal to the lookup value; the lookup key is within the irregular shape.

[0593] In an embodiment, the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; the lookup key is outside the irregular shape, and the lookup value is inside the irregular shape.

[0594] In an embodiment, the lookup value is determined based on an index function that takes the irregular shape and the lookup key as input and returns the nearest neighbor of the lookup key as output.

[0595] In an embodiment, a computing device such as a display device, a mobile device, a set-top box, a multimedia device, etc. is configured to perform any of the aforementioned methods. In an embodiment, an apparatus includes a processor and is configured to perform any of the aforementioned methods. In an embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause any of the aforementioned methods to be performed.

[0596] In an embodiment, a computing device includes one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, causes performance of any of the foregoing methods.

[0597] Note that although separate embodiments are discussed herein, any combination of embodiments and / or portions of embodiments discussed herein may be combined to form further embodiments.

[0598] Example Computer System Implementation

[0599] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or implement instructions related to adaptive perceptual quantization of images with enhanced dynamic range, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to the adaptive perceptual quantization process described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0600] Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present disclosure. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement methods related to adaptive perceptual quantization of HDR images as described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention can also be provided in the form of a program product. The program product may include any non-transient medium carrying a set of computer-readable signals, the set of computer-readable signals including instructions that, when executed by a data processor, cause the data processor to perform the method of an embodiment of the present invention. The program product according to an embodiment of the present invention may take any of a variety of forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

[0601] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents (e.g., functionally equivalent) to the components, including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

[0602] According to one embodiment, the technology described herein is implemented by one or more special computing devices. Special computing devices can be hard-wired, for performing these technologies, or can include digital electronic devices that are permanently programmed to perform these technologies, such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), or can include one or more general-purpose hardware processors that are programmed to perform these technologies according to program instructions in firmware, memory, other storage devices, or combinations. This special computing device can also combine customized hard-wired logic, ASIC or FPGA with custom programming to realize these technologies. Special computing devices can be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other equipment that incorporates hard wiring and / or program logic to implement technology.

[0603] For example, Figure 5 is a block diagram illustrating a computer system 500 upon which embodiments of the present invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general purpose microprocessor.

[0604] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, cause computer system 500 to become a special-purpose machine customized to perform the operations specified in the instructions.

[0605] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0606] The computer system 500 may be coupled to a display 512, such as a liquid crystal display, via the bus 502 for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 504 and for controlling cursor movement on the display 512. Typically, this input device has two degrees of freedom in two axes, a first axis (e.g., an x-axis) and a second axis (e.g., a y-axis), allowing the device to specify a position in a plane.

[0607] The computer system 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, enables the computer system 500 to become or be programmed as a special purpose machine. According to one embodiment, the computer system 500 performs the techniques described herein in response to the processor 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium, such as the storage device 510. Execution of the sequences of instructions contained in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hardwired circuits may be used in place of or in combination with software instructions.

[0608] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, diskettes, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs and EPROMs, flash EPROMs, NVRAMs, any other memory chips or storage boxes.

[0609] Storage media are distinct from but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0610] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may be initially carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 may receive the data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may optionally be stored on the storage device 510 before or after execution by the processor 504.

[0611] Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides bidirectional data communication coupled to network link 520, which is connected to local network 522. For example, communication interface 518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection to a telephone line of a corresponding type. As another example, communication interface 518 can be a local area network (LAN) card for providing a data communication connection with a compatible LAN. A wireless link can also be implemented. In any such implementation, communication interface 518 sends and receives electrical signals, electromagnetic signals, or optical signals carrying digital data streams representing various types of information.

[0612] The network link 520 typically provides data communication to other data devices through one or more networks. For example, the network link 520 can provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. The ISP 526 in turn provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 528. Both the local network 522 and the Internet 528 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518 (which carry the digital data to and from the computer system 500) are example forms of transmission media.

[0613] Computer system 500 can send messages and receive data, including program code, through the network, network link 520 and communication interface 518. In the Internet example, server 530 can transmit the requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0614] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510 or other non-volatile storage for later execution.

[0615] Equivalents, extensions, alternatives and miscellaneous

[0616] In the foregoing description, embodiments of the present invention have been described with reference to many specific details, which may vary depending on the implementation. Therefore, the only and exclusive indication of the claimed embodiments of the present invention and the claimed embodiments of the present invention considered by the applicant is the set of claims issued in a specific form according to the present application, wherein such claim issuance includes any subsequent corrections. Any definition explicitly set forth herein for the terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes not explicitly cited in the claims should not limit the scope of such claims in any way. Therefore, this specification and the drawings should be viewed in an illustrative rather than a restrictive sense.

[0617] Example embodiments of the enumeration

[0618] The present invention may be embodied in any form described herein, including but not limited to the following enumerated example embodiments (EEE) which describe the structure, features, and functionality of some portions of embodiments of the present invention.

[0619] EEE1. A method comprising:

[0620] constructing sampled high dynamic range (HDR) color space points distributed in a HDR color space, wherein the HDR color space is parameterized by a chromatic ablation parameter having a candidate value selected from a plurality of candidate values, wherein the chromatic ablation parameter is used to calculate color space coordinates of at least one chromatic ablation of a plurality of chromatic ablation delineating the HDR color space;

[0621] generating from the sampled HDR color space points in the HDR color space: (a) a reference standard dynamic range (SDR) color space point represented in a reference SDR color space, (b) an input HDR color space point represented in an input HDR color space, and (c) a reference HDR color space point represented in a reference HDR color space;

[0622] executing a reshaping operation optimization algorithm to generate an optimized forward reshaping mapping and an optimized chain of reverse reshaping mappings, wherein the reshaping operation optimization algorithm uses as input the reference SDR color space point, the input HDR color space point, and the reference HDR color space point;

[0623] wherein the optimized forward reshaping mapping is used to forward reshape the input HDR image in the input HDR color space into a forward reshaped SDR image in the forward reshaped SDR color space; wherein the optimized reverse reshaping mapping is used to reverse reshape the forward reshaped SDR image in the forward reshaped SDR color space into a reverse reshaped HDR image.

[0624] EEE2. A method as described in EEE1, wherein the sampled HDR color space points are constructed in the HDR color space without using any image.

[0625] EEE3. A method as described in EEE1 or EEE2, wherein a plurality of optimized forward reshaping mappings and optimized reverse reshaping mapping chains are generated for a plurality of candidate values ​​of the color source scaling parameter through the reshaping operation optimization algorithm; wherein each of the plurality of optimized forward reshaping mappings and optimized reverse reshaping mapping chains includes a corresponding optimized forward reshaping mapping and a corresponding optimized reverse reshaping mapping.

[0626] EEE4. A method as described in any one of EEE1 to EEE3, wherein the sampled HDR color space points are mapped to the reference SDR color space points based at least in part on a predefined HDR to SDR mapping.

[0627] EEE5. A method as described in any one of EEE1 to EEE4, wherein multiple groups of prediction errors are calculated for the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; wherein each group of prediction errors in the multiple groups of prediction errors is calculated for a corresponding chain in the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; wherein the multiple groups of prediction errors are used to select a specific candidate value from multiple candidate values ​​of the chromatic algebra scaling parameter.

[0628] EEE6. A method as described in EEE5, wherein a specific candidate value of the color source scaling parameter is used to generate a specific chain of a specific optimized forward reshaping mapping and a specific optimized reverse reshaping mapping.

[0629] EEE7. A method as described in EEE6, wherein the specifically optimized forward reshaping mapping is represented in a forward reshaping three-dimensional lookup table.

[0630] EEE8. A method as described in EEE6 or EEE7, wherein the specifically optimized inverse reshaping mapping is represented in an inverse reshaping three-dimensional lookup table.

[0631] EEE9. A method as described in any one of EEE6 to EEE8, wherein the video encoder applies the optimized forward reshaping mapping to an input HDR image sequence to generate a forward reshaped SDR image sequence and encodes the forward reshaped SDR image sequence into a video signal.

[0632] EEE10. A method as described in any one of EEE6 to EEE9, wherein the video decoder decodes a forward reshaped SDR image sequence from a video signal and applies the optimized reverse reshaping mapping to the forward reshaped SDR image sequence to generate a reverse reshaped HDR image sequence.

[0633] EEE11. A method as described in EEE10, wherein a display image sequence obtained from the inversely reshaped HDR image sequence is rendered on an image display operating with the video decoder.

[0634] EEE12. A method as described in any one of EEE1 to EEE10, wherein the HDR color space and the input HDR color space share a common white point.

[0635] EEE13. A method as described in any one of EEE1 to EEE10, wherein the reshaping operation optimization algorithm represents a Backward Error Subtraction with Neutral Color Preservation for Signal Adjustment (BESA) algorithm.

[0636] EEE14. A method comprising:

[0637] constructing sampled high dynamic range (HDR) color space points distributed in a HDR color space, wherein the HDR color space is parameterized by a chromatic ablation parameter having a candidate value selected from a plurality of candidate values, wherein the chromatic ablation parameter is used to calculate color space coordinates of at least one chromatic ablation of a plurality of chromatic ablation delineating the HDR color space;

[0638] generating from the sampled HDR color space points in the HDR color space: (a) an input standard dynamic range (SDR) color space point represented in an input SDR color space and (b) a reference HDR color space point represented in a reference HDR color space;

[0639] executing a reshaping operation optimization algorithm to generate an optimized inverse reshaping mapping, wherein the reshaping operation optimization algorithm receives as input the input SDR color space point and the reference HDR color space point;

[0640] The optimized inverse reshaping mapping is used to inversely reshape the SDR image in the input SDR color space into a inversely reshaped HDR image.

[0641] EEE15. A method as described in EEE14, wherein the sampled HDR color space points are constructed in the HDR color space without using any image.

[0642] EEE16. A method as described in EEE14 or EEE15, wherein multiple optimized inverse reshaping mappings are generated for multiple candidate values ​​of the color source scaling parameter through the reshaping operation optimization algorithm; wherein each of the multiple optimized inverse reshaping mappings includes a corresponding optimized inverse reshaping mapping.

[0643] EEE17. A method as described in any one of EEE14 to EEE16, wherein multiple groups of prediction errors are calculated for the multiple optimized inverse reshaping mappings; wherein each group of prediction errors in the multiple groups of prediction errors is calculated for a corresponding optimized inverse reshaping mapping in the multiple optimized inverse reshaping mappings; wherein the multiple groups of prediction errors are used to select a specific candidate value from multiple candidate values ​​of the chromatic algebra scaling parameter.

[0644] EEE18. A method as described in any one of EEE14 to EEE17, wherein a programmable image signal processor (ISP) pipeline processes the sampled HDR color space points into the input SDR color space points at least in part based on optimized values ​​of programmable configuration parameters of the programmable ISP pipeline.

[0645] EEE19. A method as described in any one of EEE14 to EEE18, wherein the optimized values ​​of the programmable configuration parameters of the programmable ISP pipeline are determined by minimizing the approximate error between an ISP SDR image generated from an HDR image by the programmable ISP pipeline and a reference SDR image generated by applying a predefined HDR to SDR mapping to the same HDR image.

[0646] EEE20. A method as described in any one of EEE1 to 19, wherein the distribution of sampled HDR color space points conforms to a predetermined pattern of sampled HDR color space points in the HDR color space.

[0647] EEE21. A method as described in any one of EEE1 to 20, wherein at least one color primary of the multiple color primary that outlines the HDR color space corresponds to a linear combination of the color primary in (i) SDR color space and (ii) HDR color space, wherein at least one weight factor of the linear combination is based on the color primary scaling parameter.

[0648] EEE22. A method as described in any one of EEE1 to 21, wherein construction, generation and execution operations are performed for each candidate value of the color original scaling parameter from the multiple candidate values.

[0649] EEE23. A method as described in EEE22, wherein a specific candidate value is selected from the multiple candidate values ​​based on a comparison of the results of executing the reshaping operation optimization algorithm for each candidate value from the multiple candidate values ​​of the color original scaling parameter.

[0650] EEE24. A method as described in any one of EEE1 to 23, wherein the reshaping operation optimization algorithm balances achieving a wide HDR color space of the inversely reshaped HDR image on the one hand and achieving low SDR color deviation on the other hand.

[0651] EEE25. A method as described in EEE5 or EEE17, wherein the specific candidate value is selected from multiple candidate values ​​of the color source scaling parameter to achieve a balance between minimizing prediction error and minimizing SDR color deviation.

[0652] EEE26. A method as described in EEE5 or EEE17, wherein the specific candidate value is selected from multiple candidate values ​​of the color source scaling parameter to minimize the prediction error when the SDR color deviation is lower than a predetermined threshold.

[0653] EEE27. A method as described in any one of EEE1 to 26, wherein the optimized inverse reshaping mapping is a mapping based on a tensor product B-spline (TPB), and wherein the reshaping operation optimization algorithm generates at least one optimized TPB coefficient for the TPB mapping.

[0654] EEE28. A method comprising:

[0655] Extracting a set of SDR image feature points from the training standard dynamic range (SDR) images and extracting a set of HDR image feature points from the training high dynamic range (HDR) images;

[0656] matching a subset of one or more SDR image feature points in the set of SDR image feature points with a subset of one or more HDR image feature points in the set of HDR image feature points;

[0657] generating a geometric transformation using the subset of the one or more SDR image feature points and the subset of the one or more HDR image feature points to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image;

[0658] determining a set of pairs of SDR color patches and HDR color patches from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and the training HDR image have been spatially aligned by the geometric transformation;

[0659] generating an optimized SDR-to-HDR mapping based at least in part on the set of pairs of SDR color patches and HDR color patches derived from the training SDR image and the training HDR image;

[0660] The optimized SDR-to-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0661] EEE29. A method as described in EEE28, wherein the training SDR image and the training HDR image are captured from a three-dimensional (3D) visual scene by capture devices operating in SDR and HDR capture modes, respectively.

[0662] EEE30. A method as described in EEE28 or EEE29, wherein the training SDR image and the training HDR image form a pair of training SDR images and training HDR images among multiple pairs of training SDR images and training HDR images; wherein the optimized SDR to HDR mapping is generated at least in part based on multiple groups of paired SDR color blocks and HDR color blocks obtained from the multiple pairs of training SDR images and training HDR images.

[0663] EEE31. A method as described in any one of EEE28 to EEE30, wherein each SDR image feature point in the subset of the one or more SDR image feature points is matched with a corresponding HDR image feature point in the subset of the one or more HDR image feature points; wherein the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm.

[0664] EEE32. A method as described in EEE31, wherein the common feature point extraction algorithm represents one of the following items: binary robust invariant scalable key point algorithm; accelerated segmented test feature algorithm; KAZE algorithm; minimum eigenvalue algorithm; maximum stable extreme area algorithm; directional FAST and rotation algorithm; scale-invariant feature transformation algorithm; or accelerated robust feature algorithm.

[0665] EEE33. A method comprising:

[0666] performing a corresponding camera distortion correction operation on each training image in a pair of training standard dynamic range (SDR) images and training high dynamic range (HDR) images to generate a corresponding undistorted image in a pair of undistorted training SDR images and undistorted training HDR images;

[0667] generating a corresponding projective transform in a pair of SDR image projective transforms and HDR image projective transforms using corner pattern markers detected from each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images;

[0668] Applying each of the pair of SDR image projective transforms and HDR image projective transforms to a corresponding undistorted image in the pair of undistorted training SDR images and undistorted training HDR images to generate a corresponding rectified image in a pair of rectified training SDR images and rectified training HDR images;

[0669] Extracting a set of SDR color patches from the corrected training SDR image and a set of HDR color patches from the corrected training HDR image;

[0670] generating an optimized SDR to HDR mapping based at least in part on the set of SDR color patches and the set of HDR color patches derived from the training SDR image and the training HDR image;

[0671] The optimized SDR-to-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0672] EEE34. A method as described in EEE33, wherein the training SDR image and the training HDR image are captured from a common color table image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively.

[0673] EEE35. A method as described in EEE34, wherein the common color table image is selected from a plurality of color table images, each of the plurality of color table images comprising a different color block distribution arranged into a two-dimensional color table.

[0674] EEE36. A method as described in EEE35, wherein the different color block distributions are generated using random colors randomly selected from a common statistical distribution having a specific statistical mean and variance value combination.

[0675] EEE37. A method as described in any one of EEE33 to EEE36, wherein the common color table image is rendered on a screen of a common reference image display and is captured from the screen by the first capture device and the second capture device.

[0676] EEE38. A method as described in any one of EEE33 to EEE37, wherein the corresponding camera distortion correction operation is at least partially based on camera-specific distortion coefficients generated from a camera calibration process performed with a camera used to acquire the training image.

[0677] EEE39. A method as described in any one of EEE33 to EEE38, wherein the set of SDR color blocks and the set of HDR color blocks are used to obtain a three-dimensional mapping table (3DMT); wherein the optimized SDR to HDR mapping is generated at least partially based on the 3DMT.

[0678] EEE40. A method as described in any one of EEE33 to EEE39, wherein the optimized SDR to HDR mapping represents one of the following items: a tensor product B-spline (TPB) based mapping or a non-TPB based mapping.

[0679] EEE41. A method as described in any one of EEE33 to EEE40, wherein the optimized SDR to HDR mapping is one of: a static mapping applied to all non-training SDR images represented in the video signal, or a dynamic mapping generated at least in part based on a particular SDR codeword value distribution of one of the non-training SDR images represented in the video signal.

[0680] EEE42. A method comprising:

[0681] Constructing sampled high dynamic range (HDR) color space points distributed in a HDR color space for representing a reconstructed HDR image;

[0682] converting the sampled HDR color space points into SDR color space points in a first standard dynamic range (SDR) color space, the SDR image to be edited by the editing device being represented in the first SDR color space;

[0683] determining a bounding SDR color space rectangle based on extreme SDR codeword values ​​of the SDR color space points in the first SDR color space and determining an irregular three-dimensional (3D) shape from a distribution of the SDR color space points;

[0684] constructing sampled SDR color space points distributed in the delimited SDR color space rectangle in the first SDR color space;

[0685] generating a boundary clipping 3D lookup table (3D-LUT) using the sampled SDR color space point and the irregular shape, wherein the boundary clipping 3D-LUT uses the sampled SDR color space point as a lookup key;

[0686] A shearing operation is performed on the edited SDR image in the first SDR color space based at least in part on the boundary-cropped 3D-LUT to generate a boundary-cropped edited SDR image in the first SDR color space.

[0687] EEE43. As described in the method of EEE42, the cropping operation includes first using the delimiting SDR color space rectangle to perform regular cropping on the edited SDR image to generate a regularly cropped edited SDR image, and then using the 3D-LUT to perform irregular cropping on the regularly cropped edited SDR image to generate the boundary cropped edited SDR image.

[0688] EEE44. A method as described in EEE42 or EEE43, wherein a group of one or more SDR pixels in the SDR image to be edited are edited from one or more first brightness values ​​to one or more second brightness values ​​in the edited image; wherein the one or more second brightness values ​​are different from the one or more first brightness values.

[0689] EEE45. A method as described in any one of EEE42 to EEE44, wherein a group of one or more SDR pixels in the SDR image to be edited are edited from one or more first chromaticity values ​​to one or more second chromaticity values ​​in the edited image; wherein the one or more second chromaticity values ​​are different from the one or more first chromaticity values.

[0690] EEE46. A method as described in any one of EEE42 to EEE45, wherein image details depicted in the SDR image to be edited are removed in the edited SDR image.

[0691] EEE47. A method as described in any one of EEE42 to EEE46, wherein image details not depicted in the SDR image to be edited are added to the edited SDR image.

[0692] EEE48. A method as described in any one of EEE42 to EEE47, wherein the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; wherein the lookup key is equal to the lookup value; wherein the lookup key is within the irregular shape.

[0693] EEE49. A method as described in any one of EEE42 to EEE48, wherein the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; wherein the lookup key is outside the irregular shape and the lookup value is within the irregular shape.

[0694] EEE50. A method as described in EEE49, wherein the lookup value is determined based on an index function, which takes the irregular shape and the lookup key as input and returns the nearest neighbor of the lookup key as output.

[0695] EEE51. A device comprising a processor and configured to perform any of the methods described in EEE1 to 50.

[0696] EEE52. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for performing a method using one or more processors according to any of the methods described in EEE1 to 50.

[0697] EEE53. A computer system configured to perform any of the methods described in EEE1 to 50.

Claims

1. A method comprising: constructing sampled high dynamic range (HDR) color space points distributed in a HDR color space, wherein the HDR color space is parameterized by a chromatic ablation parameter having a candidate value selected from a plurality of candidate values, wherein the chromatic ablation parameter is used to calculate color space coordinates of at least one chromatic ablation of a plurality of chromatic ablation delineating the HDR color space; generating from the sampled HDR color space points in the HDR color space: (a) a reference standard dynamic range (SDR) color space point represented in a reference SDR color space, (b) an input HDR color space point represented in an input HDR color space, and (c) a reference HDR color space point represented in a reference HDR color space; executing a reshaping operation optimization algorithm to generate an optimized forward reshaping mapping and an optimized chain of reverse reshaping mappings, wherein the reshaping operation optimization algorithm uses as input the reference SDR color space point, the input HDR color space point, and the reference HDR color space point; wherein the optimized forward reshaping mapping is used to forward reshape the input HDR image in the input HDR color space into a forward reshaped SDR image in the forward reshaped SDR color space; wherein the optimized reverse reshaping mapping is used to reverse reshape the forward reshaped SDR image in the forward reshaped SDR color space into a reverse reshaped HDR image.

2. The method of claim 1, wherein: The sampled HDR color space points are constructed in the HDR color space without using any image.

3. The method according to claim 1 or 2, wherein: A plurality of chains of optimized forward reshaping mappings and optimized reverse reshaping mappings are generated for a plurality of candidate values ​​of the chromatic amplification parameter by the reshaping operation optimization algorithm; wherein each of the plurality of chains of optimized forward reshaping mappings and optimized reverse reshaping mappings comprises a corresponding optimized forward reshaping mapping and a corresponding optimized reverse reshaping mapping.

4. The method according to any one of claims 1 to 3, wherein: The sampled HDR color space points are mapped to the reference SDR color space points based at least in part on a predefined HDR to SDR mapping.

5. The method according to any one of claims 1 to 4, wherein: Multiple sets of prediction errors are calculated for the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; wherein each set of prediction errors in the multiple sets of prediction errors is calculated for a corresponding chain in the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; wherein the multiple sets of prediction errors are used to select a specific candidate value from a plurality of candidate values ​​of the chromatic aboriginal scaling parameter.

6. The method according to any one of claims 1 to 5, wherein: The HDR color space and the input HDR color space share a common white point.

7. The method according to any one of claims 1 to 6, wherein: The reshaping operation optimization algorithm represents a Backward Error Subtraction with Neutral Color Preservation for Signal Adjustment (BESA) algorithm.

8. A method comprising: constructing sampled high dynamic range (HDR) color space points distributed in a HDR color space, wherein the HDR color space is parameterized by a chromatic ablation parameter having a candidate value selected from a plurality of candidate values, wherein the chromatic ablation parameter is used to calculate color space coordinates of at least one chromatic ablation of a plurality of chromatic ablation delineating the HDR color space; generating from the sampled HDR color space points in the HDR color space: (a) an input standard dynamic range (SDR) color space point represented in an input SDR color space and (b) a reference HDR color space point represented in a reference HDR color space; executing a reshaping operation optimization algorithm to generate an optimized inverse reshaping mapping, wherein the reshaping operation optimization algorithm receives as input the input SDR color space point and the reference HDR color space point; The optimized inverse reshaping mapping is used to inversely reshape the SDR image in the input SDR color space into a inversely reshaped HDR image.

9. The method of claim 8, wherein: The sampled HDR color space points are constructed in the HDR color space without using any image.

10. The method according to claim 8 or 9, wherein: A plurality of optimized inverse reshaping maps are generated for a plurality of candidate values ​​of the chroma scaling parameter by the reshaping operation optimization algorithm; wherein each optimized inverse reshaping map of the plurality of optimized inverse reshaping maps comprises a corresponding optimized inverse reshaping map.

11. The method according to any one of claims 8 to 10, wherein: Multiple sets of prediction errors are calculated for the multiple optimized inverse reshaping mappings; wherein each set of prediction errors in the multiple sets of prediction errors is calculated for a corresponding optimized inverse reshaping mapping in the multiple optimized inverse reshaping mappings; wherein the multiple sets of prediction errors are used to select a specific candidate value from a plurality of candidate values ​​of the chromatic aboriginal scaling parameter.

12. The method according to any one of claims 8 to 11, wherein: A programmable image signal processor (ISP) pipeline processes the sampled HDR color space points into the input SDR color space points based at least in part on optimized values ​​of programmable configuration parameters of the programmable ISP pipeline.

13. The method according to any one of claims 8 to 12, wherein: Optimized values ​​of programmable configuration parameters of the programmable ISP pipeline are determined by minimizing an approximation error between an ISP SDR image generated from an HDR image by the programmable ISP pipeline and a reference SDR image generated by applying a predefined HDR to SDR mapping to the same HDR image.

14. An apparatus comprising a processor and configured to perform any of the methods of claims 1 to 13.

15. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing, with one or more processors, any of the methods of claims 1 to 13.