Iterative optimization of the integer function in a single-layer HDR image codec

Abstract

A method for generating (a) a forward shaping function for compressing an input high dynamic range (HDR) image into a shaped standard dynamic range (SDR) image and (b) a backward shaping function for decompressing the shaped SDR image into a reconstructed HDR image, the method comprising: (i) optimizing the forward shaping function to minimize the deviation between the shaped SDR image and the input SDR image corresponding to the input HDR image; (ii) optimizing the backward shaping function to minimize the deviation between the reconstructed HDR image and the input HDR image; and (iii) applying correction to the input SDR image until a termination condition is met, and repeating the steps of optimizing the forward shaping function and the backward shaping function based on the corrected input SDR image.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 013,807 and European Patent Application No. 20170837.7, both filed on April 22, 2020, which are incorporated herein by reference in their entirety. Technical Field

[0003] This application relates to high dynamic range images, and more specifically, to encoding high dynamic range images in a manner that enables subsequent reconstruction of the high dynamic range image from standard dynamic range data. Background Technology

[0004] Historically, most digital images (such as digital cinema) have been created with an 8-bit bit depth. However, this bit depth presents a challenging trade-off between contrast and resolution because it is less than that of the human visual system. If the high dynamic range between the brightest and darkest pixels is encoded with an 8-bit bit depth, viewers may perceive the deviation length between adjacent pixel values ​​at brightness levels. For example, stripes may appear in parts of a scene with only very gradual changes in brightness. On the other hand, if the deviation length between pixel values ​​is set to a value imperceptible to humans, the dynamic range will be limited to a value smaller than that of the human visual system. To overcome these problems and further enhance the viewing experience, digital images are now typically created with larger bit depths (such as 10-bit or 12-bit) to produce High Dynamic Range (HDR) content. To ensure compatibility with various viewing devices, digital images are typically offered with both the standard 8-bit bit depth and higher bit depths (such as 10 or 12 bits).

[0005] Relaying digital images to a viewing system typically involves a codec. First, the image is encoded into data in the desired data format. Encoded data can include both image data and metadata. After the data is transmitted to the viewing system via a communication link, the viewing system decodes the data to reconstruct and display the image. Even when relaying HDR content, the communication link is typically limited to a standard bit depth of 8 bits. Therefore, some HDR codecs are configured to (a) compress the HDR image to an 8-bit bit depth during encoding, and (b) decompress the compressed data during decoding to at least approximately reconstruct the HDR image. Furthermore, since not all viewing systems are capable of decoding and displaying HDR images, a two-layer codec can be used to provide both standard dynamic range (SDR) and HDR images. However, in some scenarios, a single-layer codec is preferred, where a single image dataset is transmitted to the viewing system's decoder, and then, depending on the characteristics of the viewing system, the image dataset can be decoded to generate either an SDR or HDR image.

[0006] In a single-layer codec, encoding utilizes a forward shaping function to generate shaped SDR image data from input HDR image data, and decoding utilizes a backward shaping function to at least approximately reconstruct the input HDR image data from the shaped SDR image data. The backward shaping function is relayed from the encoder to the decoder as metadata accompanying the shaped SDR image data. If the system is not configured to process HDR image data, the decoder can forgo the backward shaping function and directly use the shaped SDR image data to generate the SDR image. The quality of the reconstructed HDR image data is determined at least in part by the quality of both the forward and backward shaping functions.

[0007] Traditionally, for an input SDR and HDR image pair, the forward shaping function is optimized to minimize the difference between the shaped SDR image and the input SDR image, and the backward shaping function is optimized to minimize the difference between the reconstructed HDR image and the input HDR image. The only adjustable optimization parameters are the characteristics of the forward and backward shaping functions. Summary of the Invention

[0008] This paper discloses a method, software product, and apparatus for determining the forward and backward shaping functions in a single-layer high dynamic range codec using an iterative optimization process. Traditionally, for a pair of input standard dynamic range (SDR) and high dynamic range (HDR) images, the forward shaping function is optimized to minimize the difference between the shaped SDR image and the input SDR image, and the backward shaping function is optimized to minimize the difference between the reconstructed HDR image and the input HDR image, where the only adjustable optimization parameters are the characteristics of the forward and backward shaping functions. Optimizing these conventionally determined characteristics of the forward and backward shaping functions to the required high-quality processing can be computationally intensive, which is infeasible in some potential application scenarios. This iterative optimization process does not simply adjust the characteristics of the shaping functions; instead, iteratively evaluates the optimization error and feeds this error back to the input SDR image as a correction. This iterative process, employing backward error propagation to the input SDR image, simplifies the optimization of the shaping functions, thus improving the performance of the shaping functions and / or making the associated codec suitable for a wider range of applications.

[0009] In one embodiment, a method is provided for generating (a) a forward shaping function for compressing an input high dynamic range (HDR) image into a shaped standard dynamic range (SDR) image and (b) a backward shaping function for decompressing the shaped SDR image into a reconstructed HDR image, the method comprising: (i) optimizing the forward shaping function to minimize the deviation between the shaped SDR image and the input SDR image corresponding to the input HDR image; (ii) optimizing the backward shaping function to minimize the deviation between the reconstructed HDR image and the input HDR image; and (iii) applying correction to the input SDR image until a termination condition is met, and repeating the steps of optimizing the forward shaping function and the backward shaping function based on the corrected input SDR image.

[0010] In one embodiment, a software product is provided for generating (a) a forward shaping function for compressing an input high dynamic range (HDR) image into a shaped standard dynamic range (SDR) image and (b) a backward shaping function for decompressing the shaped SDR image into a reconstructed HDR image. The software product includes a non-transitory computer-readable medium storing machine-readable instructions. These instructions include (i) a forward optimization instruction, which, when executed by the processor, causes the processor to optimize the forward shaping function to minimize the deviation between the shaped SDR image and the input SDR image corresponding to the input HDR image; (ii) a backward optimization instruction, which, when executed by the processor, causes the processor to optimize the backward shaping function to minimize the deviation between the reconstructed HDR image and the input HDR image; (iii) an SDR image correction instruction, which, when executed by the processor, causes the processor to apply correction to the input SDR image; and (iv) an iterative control instruction, which, when executed by the processor, causes the processor to evaluate whether a termination condition is met, and if the termination condition is not met, executes the SDR image correction instruction and re-executes the forward optimization instruction and the backward optimization instruction based on the corrected input SDR image. Attached Figure Description

[0011] Figure 1 The illustration shows an example of a single-layer high dynamic range (HDR) image codec in an example use case.

[0012] Figure 2 The illustration shows a method according to an embodiment for generating (a) a forward shaping function for compressing an input HDR image into a shaped standard dynamic range (SDR) image and (b) a backward shaping function for decompressing the shaped SDR image into a reconstructed HDR image.

[0013] Figure 3 The illustration shows a method for optimizing a forward shaping function and a backward shaping function according to an embodiment, which can be implemented by using backward error propagation of the input SDR image in each iteration of the iterative optimization of the shaping function.

[0014] Figure 4 An encoder for encoding input SDR and HDR image pairs in a single-layer HDR image codec is illustrated according to an embodiment.

[0015] Figure 5A and Figure 5B The illustration shows a method according to an embodiment for generating a forward and backward shaping function for a color image using backward error propagation only in the chroma channel, wherein each optimization iteration in the chroma channel utilizes multivariate multiple regression.

[0016] Figure 6A and Figure 6B The diagram shows... Figure 5A and Figure 5B An embodiment of the method based on pixel-by-pixel processing.

[0017] Figures 7A to 7C The diagram shows... Figure 5A and Figure 5B An embodiment of the method for processing quantized pixel values ​​based on 3D binning.

[0018] Figure 8 The illustration depicts a pixel-based method for generating forward and backward shaping functions for a color image according to an embodiment, wherein optimization based on iterative tensor product b-spline approximation and backward error propagation are employed in each color component channel.

[0019] Figures 9 to 16 The illustration is applied to the example image. Figure 2 A drawing of an embodiment of the method. Detailed Implementation

[0020] Figure 1 The illustration depicts a single-layer high dynamic range (HDR) image codec 100 in an example use case. In this use case, a standard dynamic range (SDR) version and an HDR version of the same image are relayed from a production system 110 to an HDR-enabled viewing system 130 via a communication link 120. The production system 110 includes an encoder 112. The viewing system 130 includes a decoder 132 and a display 136. The codec 100 includes an iterative shaping function optimizer 114 and an HDR reconstructor 134. The display 136 is not part of the codec 100 but depends on it. The iterative shaping function optimizer 114 is implemented in the encoder 112, and the HDR reconstructor 134 is implemented in the decoder 132.

[0021] In operation, production system 110 generates or receives input SDR image 180 and input HDR image 182, which represent two different versions of the same image. In another scenario, input SDR image 180 represents the original image content, and input HDR image 182 is generated based on input SDR image 180, as a compatible and potentially enhanced version of input SDR image 180. Encoder 112 runs an iterative shaping function optimizer 114 to process input SDR image 180 and input HDR image 182 to optimize forward and backward shaping functions. Based on these optimized shaping functions, encoder 112 generates a shaped SDR image 184 and metadata 190. Metadata 190 encodes at least the specification of the optimized backward shaping function that decoder 132's HDR reconstructor 134 will use.

[0022] Production system 110 relays the shaped SDR image 184 and metadata 190 to viewing system 130 via communication link 120. Communication link 120 may be limited to a bit depth that does not allow HDR images to be transmitted in the direction of their native bit depth. In one example, communication link 120 includes one or more interfaces and / or storage media limited to 8 bits or 10 bits. Communication link 120 may include one or more wireless or wired transmission paths, such as those used in movie streaming. Alternatively, communication link 120 may utilize physical storage media, such as DVDs or Blu-ray discs. Viewing system 130 may be a television with a built-in decoder 132, a television (or other display) coupled to a DVD / Blu-ray player having a decoder 132 integrated into the DVD / Blu-ray player, or a computer (e.g., desktop, laptop, smartphone, or tablet) with a built-in decoder 132.

[0023] After receiving the shaped SDR image 184 and metadata 190, decoder 132 runs HDR reconstructor 134 to generate a reconstructed HDR image 186 by applying a backward shaping function specified by metadata 190 to the shaped SDR image 184. Viewing system 130 can then display the reconstructed HDR image 186 on display 136.

[0024] In alternative use cases, if a viewing system that does not support HDR receives the shaped SDR image 184 and metadata 190, the viewing system that does not support HDR can ignore the backward shaping function specification included in the metadata 190 and instead display the shaped SDR image 184 as is, or display the SDR image generated based on the shaped SDR image 184.

[0025] The iterative shaping function optimizer 114 is configured to perform an iterative process to determine the forward shaping function and the backward shaping function. In each iteration, the iterative shaping function optimizer 114 optimizes (a) the forward shaping function to minimize the deviation between the shaped SDR image 184 and the input SDR image 180, and (b) the backward shaping function to minimize the deviation between the reconstructed HDR image 186 and the input HDR image 182. While the optimization process of a conventional single-layer HDR codec relies solely on adjusting the characteristics of the forward and backward shaping functions, the iterative shaping function optimizer 114 utilizes the backward error propagation of the input SDR image 180. More specifically, the iterative shaping function optimizer 114 continuously corrects the input SDR image 180 and iteratively optimizes the forward and backward shaping functions based on the corrected version of the input SDR image 180 until a termination condition is met. This iterative process, which employs backpropagation of the error to the input SDR image 180, reduces the complexity of a single iteration in optimizing the shaping function. It not only improves the performance of the resulting shaping function but also makes the codec 100 suitable for a wider range of applications than traditional single-layer HDR codecs.

[0026] The encoder 112 may include a processor and non-transitory memory, wherein the iterative shaping function optimizer 114 is implemented as software stored in the non-transitory memory. This software may be provided as a standalone product, implemented on a third-party computer, for example, as part of a production system, along with a third-party processor.

[0027] Figure 2 A method 200 is illustrated for generating (a) a forward shaping function for compressing an input HDR image into a shaped SDR image and (b) a backward shaping function for decompressing the shaped SDR image into a reconstructed HDR image. Method 200 takes an input SDR image 180 and an input HDR image 182 as inputs. Method 200 employs iterative optimization of the shaping function using backward error propagation to the input SDR image 180 and is configured for a single-layer HDR image codec. An iterative shaping function optimizer 114 can implement method 200 to process the input SDR image 180 and the input HDR image 182, thereby generating a shaped SDR image 184 and a portion of metadata 190 specifying the backward shaping function. Method 200 includes several iterations of steps 210, 220, 230, and 240.

[0028] In each iteration, step 210 optimizes the forward shaping function to minimize the deviation between the shaped SDR image 184 and the input SDR image 180, and step 220 optimizes the backward shaping function to minimize the deviation between the reconstructed HDR image 186 and the input HDR image 182. Each of steps 210 and 220 can utilize various optimization methods, such as multivariate multiple regression (MMR), tensor product b-spline (TPBS) approximation, or other optimization methods known in the art. Step 220 can implement step 222, applying the backward shaping function to the shaped SDR image 184 generated by the forward shaping function optimized in step 210 in the same iteration. In one example of step 210, the iterative shaping function optimizer 114 receives the input SDR image 180 and the input HDR image 182, and optimizes the forward shaping function to minimize (a) the deviation between the input SDR image 180 and (b) the shaped SDR image 184 generated by applying the forward shaping function to the input HDR image 182. In a related example of step 220, the iterative shaping function optimizer 114 processes the input HDR image 182 and the shaped SDR image 184 (generated by applying an optimized forward shaping function to the input SDR image 180). In this process, the iterative shaping function optimizer 114 optimizes the backward shaping function to minimize the deviation between (a) the input HDR image 182 and (b) the reconstructed HDR image 186 generated by applying the backward shaping function to the shaped SDR image 184.

[0029] Step 230 is a decision step performed after steps 210 and 220, which evaluates whether a termination condition is met. If the termination condition is not met, method 200 proceeds to step 240. If the termination condition is met, method 200 may continue to execute one or both of steps 250 and 252. Step 230 may be performed by iterative shaping function optimizer 114. In one embodiment, the termination condition is that the deviation metric between the reconstructed HDR image 186 and the input HDR image 182 (hereinafter referred to as the single HDR (HDR-only) deviation metric) is not greater than a threshold amount. In another embodiment, the termination condition is that a composite deviation metric is met, wherein the composite deviation metric is a weighted combination of (i) the deviation between the reconstructed HDR image 186 and the input HDR image 182 and (ii) the deviation between the shaped SDR image 184 and the input SDR image 180. Either of these two embodiments can be extended to further limit the number of iterations such that a termination condition is met when either of the following occurs: (1) the deviation metric (single HDR or composite) is not greater than the associated threshold amount, and (2) a predefined maximum number of iterations has been performed.

[0030] The input SDR image 180 and the input HDR image 182 can be color images with multiple color component channels. In one embodiment relating to the color image, the deviation metric (e.g., either a single HDR deviation metric or a composite deviation metric) is based on all the multiple color component channels. In another embodiment relating to the color image, the deviation metric is based only on a subset of the multiple color component channels. For example, when the color component channels are a luma channel, a first chroma channel, and a second chroma channel, the deviation metric can be based on the single luma channel, such that the luma channel is used to guide the iterative process; the deviation metric can be based only on the chroma channel, such that the chroma channel is used to guide the iterative process. In another embodiment of method 200, relating to a color image with a luma channel and two chroma channels, iterative optimization is applied to the chroma channel, while the forward and backward shaping functions for the luma channel are optimized only in a single iteration of steps 210 and 220.

[0031] Step 240 applies correction to the input SDR image 180. In one example of step 240, the iterative shaping function optimizer 114 applies correction to the input SDR image 180 before feeding the corrected input SDR image 180 to the next iteration of steps 210, 220, and 240. In an embodiment, step 230 evaluates a bias metric (such as a single HDR bias metric), and step 240 corrects the input SDR image 180 by an amount that is a function of the bias metric. For example, step 240 may correct the input SDR image 180 by an amount proportional to the single HDR bias metric, optionally limited to a predefined maximum correction amount to prevent excessive alteration to the input SDR image. In an alternative example, step 240 uses a more complex function to associate the correction amount with the bias metric to guide iterative optimization in a desired manner. Step 240 may correct the input SDR image 180 pixel-by-pixel, where each pixel of the input SDR image 180 is corrected by an amount that depends on the bias metric associated with that pixel.

[0032] Step 250 outputs the shaped SDR image generated using the forward shaping function optimized in this iteration during the last iteration of step 210. In one example of step 250, the iterative shaping function optimizer 114 outputs the shaped SDR image 184. Step 252 outputs one or more parameters characterizing the backward shaping function optimized in the last iteration of step 220. In one example of step 252, the iterative shaping function optimizer 114 outputs these one or more parameters as part of metadata 190.

[0033] although Figure 2As not shown, method 200 may include maintaining an uncorrected version of the input SDR image 180. In such an embodiment, all corrections to the input SDR image 180 in the iteration of step 240 may be performed on a local copy of the input SDR image 180.

[0034] When using method 200 to process video, method 200 can be repeated for each frame of the video. Alternatively, method 200 can re-optimize the forward and backward shaping functions less frequently, and for example, use the same forward and backward shaping functions for several consecutive frames.

[0035] Mathematical description of an embodiment of method 200:

[0036] Here, notation is introduced to discuss an embodiment of method 200, which is applied to an input signal having t frames, where t is greater than or equal to one, and each frame is represented by an input SDR image 180 and an input HDR image 182. Let... Let represent the i-th pixel at frame t in the ch channel of the input HDR signal. Let represent the i-th pixel at frame t in the ch channel of the input SDR signal. For the k-th iteration, let the k-th iteration of the forward shaping function be expressed as... make The k-th iteration of the parameter set used is denoted as Let the k-th iteration of the backward shaping function be expressed as make The k-th iteration of the parameter set used is denoted as make Let the ith pixel at frame t in the ch channel of the shaped SDR signal be... The ch channel of the reconstructed HDR signal represents the i-th pixel at frame t, and lets This represents the i-th pixel at frame t in the ch channel of the corrected input SDR signal.

[0037] In the first iteration (k=0), this embodiment of method 200 sets the corrected input SDR signal as the original input SDR signal, that is,

[0038]

[0039] In this embodiment of method 200, in each iteration k, step 210 finds the parameters of the forward shaping function. These parameters minimize the difference between the shaped SDR signal and the corrected input SDR signal:

[0040]

[0041] in,

[0042]

[0043] In each iteration k of this embodiment of method 200, step 210 finds the parameters of the backward shaping function, i.e. These parameters minimize the difference between the reconstructed HDR signal and the input HDR signal:

[0044]

[0045] in,

[0046]

[0047] At the end of each iteration k, based on the reconstructed HDR signal and the input HDR signal, step 240 in this embodiment of method 200 determines the corrected input SDR signal to be used in iteration k+1. Quantity This amount is and Functions:

[0048]

[0049] The g function can be a simple subtraction function.

[0050]

[0051] Optionally, scaling can be performed by correcting the scaling factor and / or further incorporating additional consideration of the gradient of the backward shaping function. Then, step 240 sets the corrected input SDR signal for the next iteration as follows:

[0052]

[0053] This embodiment of method 200 defines the deviation measures in HDR and SDR as follows:

[0054]

[0055] and

[0056]

[0057] In this embodiment, step 230 continues the iteration of method 200 until either the preset maximum number of iterations or the composite deviation metric is not greater than a threshold.

[0058]

[0059] Among them, w v and w sIt is a combination weight. In one implementation, w v It equals one, and w s The value equals zero, causing the composite bias metric to collapse into a single HDR bias metric. In another implementation, the combined weight w v and w s It is set to define the desired trade-off between HDR distortion and SDR distortion, as discussed in further detail below in "Example 1: Polynomial Integer Function," which shows the results obtained using method 200 in an example using a 4th-order polynomial integer function.

[0060] Figure 3 A method 300 for optimizing a forward shaping function and a backward shaping function is illustrated, which can be implemented by employing backward error propagation to the input SDR image in each iteration of the iterative optimization of the shaping function. Method 300 is an embodiment of steps 210 and 220 of method 200 and can be executed by an iterative shaping function optimizer 114. Method 300 includes steps 310 and 320, which are embodiments of steps 210 and 220, respectively. Step 310 optimizes one or more parameters 390 of the forward shaping function (e.g., FP) using a forward prediction model. t ch,(k) This minimizes the deviation between (a) the input SDR image 180 and (b) the shaped SDR image 184 generated from the input HDR image 182 using the forward shaping function. Step 320 optimizes one or more parameters 392 of the backward shaping function using a backward prediction model, for example, To minimize the deviation between (a) the input HDR image 182 and (b) the reconstructed HDR image 186 generated from the shaped SDR image 184 by the backward shaping function.

[0061] In each of steps 310 and 320, the prediction model may incorporate either the MMR or TPBS approximation.

[0062] When method 300 is implemented within method 200, step 252 of method 200 (if included) may output parameters(s) 392. In one embodiment, the parameters(s) 390 generated in each iteration of step 310 are not used in any subsequent steps. In another embodiment, the parameters(s) 390 generated in the iteration of step 310 may be used as the starting point for subsequent iterations of step 310. The reconstructed HDR image 186 generated in each iteration of step 320 may be used in (a) step 230 to evaluate whether a termination condition is met, and (b) step 240 to determine the amount by which the input SDR image 180 has been corrected.

[0063] Figure 4An encoder 400 is illustrated for encoding input SDR and HDR image pairs in a single-layer HDR image codec. Encoder 400 is an embodiment of encoder 112 configured to perform method 200. Encoder 400 includes a processor 410, an interface 412, and a non-transitory machine-readable memory 420. Memory 420 stores machine-readable instructions 430, including forward optimization instructions 440, backward optimization instructions 450, SDR image correction instructions 460, and iterative control instructions 470. Memory 420 may further include data storage 480 configured to store images, parameters, and / or other data used by encoder 400.

[0064] Encoder 400 receives input SDR image 180 and input HDR image 182 via interface 412. Encoder 400 can store the input SDR image 180 and input HDR image 182 in data storage 480. Once encoder 400 has executed method 200, encoder 400 can output shaped SDR image 184 and metadata 190 via interface 412, wherein metadata 190 includes the specification of the backward shaping function optimized by encoder 400. Interface 412 may include both an input interface and an output interface.

[0065] Forward optimization instruction 440 is configured to, when executed by processor 410, control processor 410 to execute step 210 of method 200. To execute step 210, processor 410 may retrieve the input SDR image 180 and the input HDR image 182 from data storage 480, and store or update the resulting shaped SDR image 184 in data storage 480. In step 210, during the optimization of forward shaping parameters 390, processor 410 may further use data storage 480 to temporarily store these forward shaping parameters.

[0066] The backward optimization instruction 450 is configured to, when executed by the processor 410, control the processor 410 to execute step 220 of method 200. In order to execute step 220, the processor 410 may obtain the shaped SDR image 184 and the input HDR image 182 from the data storage 480, and store or update the obtained backward shaping parameters 392 and the reconstructed HDR image 186 in the data storage 480.

[0067] The iteration control instruction 470 is configured to, when executed by the processor 410, control the processor 410 to execute step 230 of method 200. The iteration control instruction 470 may include a termination condition 472. When the iteration control instruction 470 is executed, the processor 410 may retrieve one or more of the input HDR image 182, the reconstructed HDR image 186, the input SDR image 180, and the shaped SDR image 184 from the data storage 480 to determine whether the termination condition 472 is met. The termination condition 472 may include a deviation threshold 474 (e.g., δ). The termination condition 472 may also include a maximum number of iterations 476.

[0068] SDR image correction instruction 460 is configured to, when executed by processor 410, control processor 410 to execute step 240 of method 200. Processor 410 may obtain input HDR image 182 and reconstructed HDR image 186 from data storage 480 to determine the amount of correction to be made to input SDR image 180, and then update input SDR image 180 in data storage 480.

[0069] Instruction 430 may further include one or both of MMR instruction 442 and TPBS instruction 444. MMR instruction 442 may be called by forward optimization instruction 440 and / or backward optimization instruction 450 to perform associated optimization based on MMR. TPBS instruction 444 may be called by forward optimization instruction 440 and / or backward optimization instruction 450 to perform associated optimization based on TPBS approximation.

[0070] Figure 5A and Figure 5B The illustration depicts a method 500 for generating forward and backward shaping functions for a color image using backpropagation only in the chroma channels, wherein each optimization iteration in the chroma channels utilizes MMR. Method 500 is an embodiment of method 200 and method 300 can also be implemented. Method 500 can be executed by an iterative shaping function optimizer 114 (e.g., implemented in encoder 400). Figure 5A This is a flowchart of the forward and backward shaping functions used to generate the luminance components of the input SDR and HDR image pairs. Figure 5B This is a flowchart of the forward and backward shaping functions used to generate each chroma component of the input SDR and HDR image pairs.

[0071] For the luminance component of the color versions of the input SDR image 180 and the input HDR image 182, method 500 includes steps 510L and 520L, which are luminance channel-specific embodiments of steps 210 and 220, respectively. For the luminance channel, method 500 takes the input SDR image 580 and the input HDR image 582 as inputs. The input SDR image 580 and the input HDR image 582 are color images. The input SDR image 580 has a luminance component (referred to as input SDR image luminance component 580L) and two chrominance components (referred to as input SDR image chrominance components 580C0 and 580C1). The input HDR image 582 has a luminance component (referred to as input HDR image luminance component 582L) and two chrominance components (referred to as input HDR image chrominance components 582C0 and 582C1). Step 510L processes the input SDR image luminance component 580L and the input HDR image luminance component 582L. Step 510L optimizes the forward shaping function (characterized by the forward shaping parameter 590L of the luminance component) to minimize the deviation between the luminance component 584L of the shaped SDR image and the luminance component 580L of the input SDR image. Step 520L optimizes the backward shaping function (characterized by the backward shaping parameter 592L of the luminance component) to minimize the deviation between the luminance component 586L of the reconstructed HDR image and the luminance component 582L of the input HDR image.

[0072] Once step 510L is completed, method 500 can proceed to step 550L, which outputs the luminance component 584L of the shaped SDR image. Once step 520L is completed, method 500 can proceed to step 552L, which outputs the backward shaping parameters 592L of the luminance component.

[0073] For the associated chroma channel, method 500 includes steps 510C, 520C, 530C, and 540C, which are chroma-specific embodiments of steps 210, 220, 230, and 240, respectively.

[0074] Step 510C utilizes MMR to optimize the forward shaping function (characterized by the forward shaping parameter 590C0 of the first chroma component and 590C1 of the second chroma component) to minimize (a) the deviation between the chroma component 584C0 of the shaped SDR image and the chroma component 580C0 of the input SDR image, and (b) the deviation between the chroma component 584C1 of the shaped SDR image and the chroma component 580C1 of the input SDR image. The MMR process in step 510C takes all color components of the input HDR image 582 as input to (a) simultaneously predict the shaped SDR chroma components 584C0 and 584C1 and (b) optimize the forward shaping function of the chroma channel characterized by their respective forward shaping parameters 590C0 and 590C1.

[0075] Next, step 520C utilizes MMR to optimize the backward shaping function (characterized by the backward shaping parameters 592C0 of the first chroma component and 592C1 of the second chroma component) to minimize the deviation between (a) the reconstructed HDR image chroma component 586C0 and the input HDR image chroma component 582C0, and (b) the reconstructed HDR image chroma component 586C1 and the input HDR image chroma component 582C1. As input to the predicted reconstructed HDR chroma components and the backward shaping function that generates the two chroma channels, the MMR process in step 520 uses the shaped SDR chroma components 584C0 and 584C1 (generated in step 510C in the same iteration) and the shaped SDR luminance component 584L (generated in step 510L).

[0076] Step 530C is a decision step to evaluate whether the termination condition for the chroma components is met. If the termination condition is not met, method 500 proceeds to the next iteration of steps 540C, 510C, 520C, and 530C. Step 540C applies correction to one or both of the input SDR image chroma components 580C0 and 580C1. Steps 530C and 540C are chroma-specific embodiments of steps 230 and 240, respectively. If step 530C finds that the termination condition is met, method 500 can execute steps 550C and 552C. Step 550C outputs the shaped SDR image chroma components 584C0 and 584C1 generated in the last iteration of step 510C. Step 552C outputs the backward shaping parameters 592C0 and 590C1 of the chroma components optimized in the last iteration of step 520C.

[0077] In one type of embodiment, the optimization in steps 510C, 520C, 510L, and 520L is based on individual pixels of the image in question. (See below for reference.) Figure 6A and Figure 6B One such embodiment is discussed. In another type of embodiment, the optimization in steps 510C, 520C, 510L, and 520L is based on three-dimensional (3D) bins of the quantized values ​​of the image involved. In these embodiments, at least during some or all of the processing of method 500, the shaped SDR image chroma component 584C, the shaped SDR image luminance component 584L, the reconstructed HDR image chroma component 586C, and the reconstructed HDR image luminance component 586L can be represented by their 3D bins rather than by the more conventional pixel representation. Similarly, the shaping parameters 590C, 592C, 590L, and 592L can be related to 3D bins rather than individual pixels. (Refer to below...) Figure 7A and Figure 7B An embodiment of method 500 based on a 3D box is discussed below. References also include... Figure 7Aand Figure 7B The 3D box was discussed in more detail.

[0078] Figure 6A and Figure 6B A pixel-based method 600 is illustrated for generating forward and backward shaping functions for a color image using only backward error propagation in the chroma channels, wherein each optimization iteration in the chroma channels is based on MMR. Method 600 is an embodiment of method 500. Method 600 can be executed by an iterative shaping function optimizer 114 (e.g., implemented in encoder 400). Figure 6A This is a flowchart of the forward and backward shaping functions used to generate the luminance components of the input SDR and HDR image pairs. Figure 6B This is a flowchart of the forward and backward shaping functions used to generate each chroma component of the input SDR and HDR image pairs.

[0079] For the luminance component, method 600 includes steps 610L and 620L, which are embodiments of steps 510L and 520L, respectively.

[0080] Step 610L optimizes the forward shaping function for the luminance component and includes steps 612L and 614L. Step 612L processes individual pixels of the luminance component 684L of the shaped SDR image and the luminance component 580L of the input SDR image to determine a first total (across all pixels) deviation between the shaped SDR image luminance component 684L and the input SDR image luminance component 580L. Step 614L optimizes the pixel-based parameters 690L of the forward shaping function for the luminance channel to minimize the first total deviation. Step 614L can determine the luminance channel forward shaping function based on the cumulative distribution function (CDF) matching between the histogram of the shaped SDR image luminance component 684L and the histogram of the input SDR image luminance component 580L.

[0081] Step 620L optimizes the backward shaping function of the luminance component and includes steps 622L and 624L. Step 622L processes individual pixels of the reconstructed HDR image luminance component 686L and the input HDR image luminance component 582L to determine a second total (across all pixels) deviation between the reconstructed HDR image luminance component 686L and the input HDR image luminance component 582L. Step 624L optimizes the pixel-based parameter 692L of the backward shaping function of the luminance channel to minimize the second total deviation. Step 624L can determine the luminance channel backward shaping function based on the CDF matching between the histogram of the reconstructed HDR image luminance component 686L and the histogram of the input HDR image luminance component 582L.

[0082] Once step 610L is completed, method 600 can proceed to step 550L, which outputs the luminance component 684L of the shaped SDR image. Once step 620L is completed, method 600 can proceed to step 552L, which outputs the pixel-based parameter 692L.

[0083] For the chroma channel, method 600 includes steps 610C, 620C, 630C, and 640C, which are embodiments of steps 510C, 520C, 530C, and 540C, respectively.

[0084] Step 610C optimizes the forward shaping function for the chroma components and includes steps 612C and 614C. For each chroma component, step 612C processes individual pixels of the shaped SDR image chroma component 684C0 / C1 and the input SDR image chroma component 580C0 / C1 to determine a first total (across all pixels) deviation between the shaped SDR image chroma component 684C0 / C1 and the input SDR image chroma component 580C0 / C1. Step 612C thus calculates the first total deviation for each chroma channel. Step 614C utilizes MMR to optimize (a) the pixel-based parameter 690C0 of the forward shaping function for the first chroma channel to minimize the first total deviation of the first chroma channel, and optimizes (b) the pixel-based parameter 690C1 of the forward shaping function for the second chroma channel to minimize the first total deviation of the second chroma channel.

[0085] Step 620C optimizes the backward shaping function of the chroma components and includes steps 622C and 624C. For each chroma component, step 622C processes individual pixels of the reconstructed HDR image chroma component 686C0 / C1 and the input HDR image chroma component 582C0 / C1 to determine a second total (across all pixels) deviation between the reconstructed HDR image chroma component 686C0 / C1 and the input HDR image chroma component 582C0 / C1. Step 622C thus calculates the second total deviation for each chroma channel. Step 624C utilizes MMR to optimize (a) the pixel-based parameter 692C0 of the backward shaping function for the first chroma channel to minimize the second total deviation of the first chroma channel, and optimizes (b) the pixel-based parameter 692C1 of the backward shaping function for the second chroma channel to minimize the second total deviation of the second chroma channel.

[0086] Step 630C is a decision step to evaluate whether the termination condition for the chroma components is met. If the termination condition is not met, method 600 proceeds to the next iteration of steps 640C and 610C, 620C, and 630C. Step 640C applies correction to one or both of the input SDR image chroma components 580C0 and 580C1. Steps 630C and 640C are pixel-based implementations of steps 530C and 540C, respectively. If step 630C finds that the termination condition is met, method 600 can proceed to steps 550C and 552C to output the shaped SDR image chroma components 684C0 and 684C1, and pixel-based parameters 692C0 and 692C1, respectively.

[0087] An example of method 600 is discussed in more detail below in “Example 2: Pixel-based MMR”.

[0088] Figures 7A to 7C A 3D bin-based method 700 is illustrated for generating forward and backward shaping functions for a color image using backpropagation only on the chroma channels, wherein each optimization iteration in the chroma channels is based on MMR. Method 700 is an embodiment of method 500. Method 700 can be executed by an iterative shaping function optimizer 114 (e.g., implemented in encoder 400). Figure 7A This is a flowchart used to create a 3D box. Figure 7B This is a flowchart of the forward and backward shaping functions used to generate the luminance components of the input SDR and HDR image pairs. Figure 7C This is a flowchart of the forward and backward shaping functions used to generate each chroma component of the input SDR and HDR image pairs.

[0089] Method 700 includes steps 702 and 704 (see...) Figure 7AStep 702 quantizes each of the luminance channel, the first chrominance channel, and the second chrominance channel. Step 704 bins the input HDR image 582 having a luminance component 582L, a first chrominance component 582C0, and a second chrominance component 582C1. More specifically, step 704 groups pixels in the input HDR image 582 that have the same quantization value in each of the luminance component 582L, the first chrominance component 582C0, and the second chrominance component 582C1 into 3D bins. For example, step 702 may quantize each color component channel in 100 pixel value bins numbered 0 to 99, and then step 704 may group all pixels of the input HDR image 582 into assigned 3D bins numbered 12, 46, and 33, with the pixel value of the luminance component 582L in bin 12 of the luminance channel, the pixel value of the first chroma component 582C0 in bin 46 of the first chroma channel, and the pixel value of the second chroma component 582C1 in bin 33 of the second chroma channel. Step 704 may also apply the 3D binning of the pixels of the input SDR image 580, thus defined by the input HDR image 582. In subsequent steps of method 700, the same 3D binning is used for the shaped SDR image generated by method 700.

[0090] In summary, step 704 generates image data 782 for the input HDR 3D bin and image data 780 for the input SDR 3D bin. The image data 782 for the input HDR 3D bin includes the luminance component 782L, the chrominance component 782C0, and the chrominance component 782C1 of the input HDR 3D bin. The term "chrominance component 782C" for the input HDR 3D bin unspecifically refers to one of the chrominance components 782C0 and 782C1 of the input HDR 3D bin. The image data 780 for the input SDR 3D bin includes the luminance component 782L, the chrominance component 780C0, and the chrominance component 780C1 of the input SDR 3D bin. The term "Input SDR 3D Box Chromaticity Component 780C" refers unspecifically to one of the input SDR 3D Box Chromaticity Components 780C0 and 780C1.

[0091] Compared to pixel-based method 600, 3D box-based method 700 saves memory space and provides faster processing and / or reduces the demand for computing resources because all seemingly similar pixels are grouped together. Steps 702 and 704 can be encoded in Figure 4 The machine-readable instructions 430 of the encoder 400.

[0092] After completing step 704, method 700 proceeds to the following... Figure 7BThe optimization of the brightness channel shown and as Figure 7C The optimization of the chroma channel is shown.

[0093] For the luminance component, method 700 includes steps 710L and 720L, which are embodiments of steps 510L and 520L, respectively.

[0094] Step 710L optimizes the forward shaping function for the luminance component and includes steps 712L and 714L. Steps 712L and 714L are similar to steps 612L and 614L, except that they operate on 3D bins instead of individual pixels. Therefore, step 710L generates 3D bin-shaped SDR luminance component data 784L to replace the shaped SDR image luminance component 684L, and optimizes the 3D bin-based parameters 790L of the forward shaping function for the luminance channel.

[0095] Step 720L optimizes the backward shaping function of the luminance component and includes steps 722L and 724L. Steps 722L and 724L are similar to steps 622L and 624L, except that they operate on 3D bins instead of individual pixels. Therefore, step 720L generates 3D bin-based parameters 792L for the backward shaping function of the luminance channel and operates using the reconstructed HDR luminance component data 786L from the 3D bins instead of the reconstructed HDR image luminance component 686L.

[0096] Once step 710L is completed, method 700 can proceed to step 550L, which outputs the shaped SDR chromaticity component data 784C of the 3D bins. Without departing from the scope of this disclosure, method 700 may include converting the shaped SDR luminance component data 784L of the 3D bins into a pixel-based representation of the shaped SDR luminance component, and outputting this pixel-based representation in step 550L. Once step 720L is completed, method 700 can proceed to step 552L, which outputs the 3D bin-based parameters 792L.

[0097] For the chroma channel, method 700 includes steps 710C, 720C, 730C, and 740C, which are embodiments of steps 510C, 520C, 530C, and 540C, respectively.

[0098] Step 710C optimizes the forward shaping function of the chroma components and includes steps 712C and 714C. Steps 712C and 714C are similar to steps 612C and 614C, except that they operate on 3D bins instead of individual pixels. Therefore, step 710C generates 3D bin-shaped SDR chroma component data 784C0 and 784C1 to replace the shaped SDR image chroma components 684C0 and 684C1, and optimizes the 3D bin-based parameters 790C0 and 790C1 of the forward shaping function of the first and second chroma channels, respectively.

[0099] Step 720C optimizes the backward shaping function of the chroma components and includes steps 722C and 724C. Steps 722C and 724C are similar to steps 622C and 624C, except that they operate on 3D bins instead of individual pixels. Therefore, step 720C generates 3D bin-based parameters 792C0 for the backward shaping function of the first chroma channel and 3D bin-based parameters 792C1 for the backward shaping function of the second chroma channel, and operates using the reconstructed HDR chroma component data 786C0 and 786C1 from the 3D bins instead of the reconstructed HDR image chroma components 686C0 and 686C1.

[0100] Step 730C is a decision step to evaluate whether the termination condition for the chroma components is met. If the termination condition is not met, method 700 proceeds to the next iteration of steps 740C, 710C, 720C, and 730C. Step 740C applies correction to one or both of the chroma components 780C0 and 780C1 of the input SDR 3D bin. Steps 730C and 740C are similar to steps 630C and 640C, respectively, except that they are based on 3D bin processing instead of pixel processing. If step 730C finds that the termination condition is met, method 700 can proceed to steps 550C and 552C to output the shaped SDR chroma component data 784C0 and 784C1 of the 3D bin and the 3D bin-based parameters 792C0 and 792C1, respectively. Without departing from the scope of this disclosure, method 700 may include converting the 3D binned shaped SDR chromaticity component data 784C0 and 784C1 into pixel-based representations of the shaped SDR chromaticity components, and outputting these pixel-based representations in step 550C.

[0101] An example of method 700 is discussed in more detail below in “Example 3: 3D box-based MMR”.

[0102] Without departing from the scope of this disclosure, the chroma channel can be implemented using a 3D box-based method 700 (according to...). Figure 7A and Figure 7BThe process can be handled using the flowchart outlined in the document, while the luminance channel can be processed using a pixel-based method 600 (according to...). Figure 6B The flowchart outlined in the document is used for processing.

[0103] Figure 8 A pixel-based method 800 for generating forward and backward shaping functions for a color image is illustrated, wherein optimization based on an iterative TPBS approximation and backward error propagation are employed in each color component channel. Method 800 is an embodiment of method 200 and method 300 can be implemented. Method 800 can be performed by an iterative shaping function optimizer 114 (e.g., implemented in encoder 400). Method 800 includes steps 810, 820, 830, and 840, which are embodiments of steps 210, 220, 230, and 240, respectively.

[0104] For each color component channel, method 800 takes the input SDR image 580 and the input HDR image 582 as inputs and performs one or more iterations of steps 810, 820, 830, and 840 to generate the forward and backward shaping functions for the considered color component channel, as well as the associated shaped SDR image data. If only one iteration is performed for a color component channel, step 840 is omitted from the processing of that color component channel.

[0105] Step 810 optimizes the forward shaping function for the considered color component channels and includes steps 812 and 814. Step 812 processes individual pixels of the associated color components (i.e., one of color components 580L, 580C0, and 580C1) of the shaped SDR image color component 784 and the input SDR image 580 to determine a first total (across all pixels) deviation between the shaped SDR image color component 784 and the color components of the input SDR image 580. Step 814 utilizes the TPBS approximation to optimize the pixel-based parameters 790 of the forward shaping function for the color component channels to minimize the first total deviation.

[0106] Step 820 optimizes the backward shaping function of the considered color component channels and includes steps 822 and 824. Step 822 processes individual pixels of the reconstructed HDR image color component 786 and the associated color components (i.e., one of color components 582L, 582C0, and 582C1) of the input HDR image 582 to determine a second total (across all pixels) bias between the reconstructed HDR image color component 786 and the color components of the input HDR image 582. Step 824 utilizes the TPBS approximation to optimize the pixel-based parameter 792 of the backward shaping function of the color component channels to minimize the second total bias.

[0107] Step 830 is a decision step to evaluate whether the termination condition for the color components is met. If the termination condition is not met, method 800 proceeds to the next iteration of steps 840 and 810, 820, and 830. Step 840 applies corrections to the color components of the considered input SDR image 580. Steps 830 and 840 are embodiments of steps 230 and 240, respectively. If step 830 finds that the termination condition is met, method 800 can proceed to steps 850 and 852. Step 850 outputs the shaped SDR image color components 784 of the considered color component channels optimized in the last iteration of step 810. Step 852 outputs the pixel-based parameters 792 of the considered color component channels optimized in the last iteration of step 820.

[0108] An example of method 800 is discussed in more detail below in “Example 4: Pixel-based TPBS approximation”.

[0109] In the same way that Method 700 is a pixel-based alternative to the 3D bin-based alternative of Method 600, Method 800 can be modified to process the image data in 3D bins. An example of this approach is discussed in more detail below in “Example 5: 3D Bin-Based TPBS Approximation”.

[0110] Example 1: Polynomial Integer Function

[0111] In this example, method 200 is a forward integer function. and backward shaping function This is performed for 4th-order polynomials, such that each of steps 210 and 220 optimizes five polynomial coefficients respectively. This optimization can be accomplished using least-squares minimization. While more advanced optimization algorithms are expected to provide better performance, this example is used here for illustration and can be implemented in scenarios where computational ease outweighs performance. The input data processed in this example of method 200 has only a single color component, equivalent to a luminance component, a single chrominance component, or grayscale image data.

[0112] The input HDR image 182 was generated using random pixel values. The input SDR image 180 was generated from the input HDR image 182 by cropping the dynamic range, introducing polynomial distortion, and adding noise. The resulting correspondence between HDR pixel values ​​and SDR pixel values ​​is plotted on [the graph]. Figure 9 Method 200 is applied to these examples, with input SDR image 180 and input HDR image 182, where 100 iterations are performed.

[0113] Figure 10For every 250th pixel of the input HDR image 182 and according to the number of iterations, the deviation between the input HDR image 182 and the reconstructed HDR image 186 generated in step 220 was plotted. Figure 10 Each curve in the graph corresponds to a different pixel. Figure 11 The pixel values ​​of every 250th pixel of the input SDR image 180 that was corrected in step 240 were plotted based on the number of iterations. Figure 12 For every 250th pixel of the input SDR image 180 and according to the number of iterations, the deviation between the input SDR image 180 and the shaped SDR image 184 corrected in step 240 is plotted. Figure 14 The total SDR distortion was plotted based on the number of iterations, and this total SDR distortion was calculated as follows: Figure 12 The sum of SDR deviations plotted in the image. Figure 13 The total HDR distortion was plotted based on the number of iterations and was calculated as follows: Figure 10 The sum of HDR biases plotted in the image. From Figure 10 and Figure 13 It is evident that the iterative optimization in method 200 converges and reduces the difference between the input HDR image 182 and the reconstructed HDR image 186. Meanwhile, from... Figure 11 , Figure 12 and Figure 14 It is evident that the improved consistency between the input HDR image 182 and the reconstructed HDR image 186 comes at the cost of introducing some distortion into the shaped SDR image 184. Depending on the application scenario and associated requirements, the combined weights w... v and w s It can be adjusted to define the desired trade-off between HDR distortion and SDR distortion.

[0114] Figure 15 The forward shaping function 1500 and correspondence 900 generated in the last iteration of step 210 are plotted. The forward shaping function 1500 solves the clipping problem in correspondence 900 while achieving reasonable consistency with the uncropped portion of correspondence 900. Since the forward shaping function avoids clipping, in scenarios where the display system is configured to display SDR images, the shaped SDR image 184 may be superior to the input SDR image 180. Figure 16The backward shaping function 1600 and correspondence 900 generated in the last iteration of step 220 are plotted. While the backward shaping function 1600 is indirectly based in part on the input SDR image 180, it does not exhibit the cropping problem found in correspondence 900 and further achieves reasonable consistency with the uncropped portion of correspondence 900. Both the forward and backward shaping functions show some residual distortion in the uncropped portion of correspondence 900. This distortion is likely at least partly a result of the relatively simple (4th-order polynomial) nature of the optimization applied in this example.

[0115] Example 2: Pixel-based MMR

[0116] Below, we outline an example of method 600.

[0117] Luminance channel:

[0118] In this example, the luminance channel (denoted as y) is a single-channel predictor. The forward shaping function in step 620L is a one-dimensional (1D) lookup table (LUT), and the backward shaping function in step 610L is an 8-term second-order polynomial predictor. The backward shaping function is also expressed as a 1D-LUT. Note that since the predictor is a 1D-LUT and can be easily obtained in a reversible manner, no iteration is involved in the luminance channel. We define the following notation:

[0119]

[0120]

[0121] and

[0122] Color channels:

[0123] The chroma channels (c0 and c1) are predicted using MMR. In the MMR forward shaping in step 610C, we employ all three color component channels from the HDR domain. As input to predict SDR chromaticity components and In the k-th iteration, the predicted (integer) value is We use these three SDR prediction values ​​as input to the backward shaping optimization in step 620C to predict HDR chroma values. The value predicted (reconstructed) in the k-th iteration is Note that some of the matrices used in each iteration do not change and can be computed in advance.

[0124] initialization:

[0125] In the first iteration (k=0), for all three channels (ch=y, c0, and c1), we set the corrected input SDR image as the original input SDR image, i.e. The MMR extension form of the i-th input HDR pixel is expressed as:

[0126]

[0127] Collect all P pixels to obtain the extended matrix:

[0128]

[0129] We can also define the following matrix, which will be used in the coefficient optimization process:

[0130]

[0131] We also prepared the input HDR chroma components in vector form:

[0132] and

[0133] Notice and They do not change during iteration because they represent the input HDR image. They can be pre-computed and stored in memory without needing to be recomputed.

[0134] Forward plastic surgery:

[0135] Forward MMR coefficients in channels c0 and c1 and The predicted SDR value in the k-th iteration can be obtained using the following formula.

[0136] as well as

[0137]

[0138] in,

[0139] and

[0140] In each iteration, we collect the corrected SDR values ​​from the previous iteration as a vector.

[0141] and These two vectors will be the target values ​​to be achieved in the k-th iteration of the forward prediction. Note that the brightness prediction value does not change in each iteration.

[0142] Define the following two vectors to facilitate discussion of coefficient optimization:

[0143] as well as

[0144]

[0145] The goal of coefficient optimization is to find the MMR coefficients that minimize the mean square error (MSE) between the corrected input SDR signal and the predicted (shaped) SDR signal in the k-th iteration.

[0146]

[0147]

[0148] The optimal MMR coefficients in forward shaping can be found as a least-squares solution:

[0149] as well as

[0150]

[0151] The optimal predicted SDR value in the k-th iteration can be expressed as:

[0152] as well as

[0153]

[0154] Retrograde plastic surgery:

[0155] In the k-th iteration, the MMR extension form of the predicted SDR pixel is expressed as:

[0156]

[0157] We can collect all P pixels together to form a matrix.

[0158]

[0159] Define the following matrix, which will be used in the coefficient optimization process:

[0160]

[0161] Backward MMR coefficients in channels c0 and c1 and The predicted HDR value in the k-th iteration can be obtained using the following formula.

[0162] as well as

[0163]

[0164] in,

[0165] and

[0166] We also define the following vectors to facilitate coefficient optimization:

[0167]

[0168]

[0169] The optimization objective is to find the MMR coefficients to minimize the MSE between the original input HDR signal and the predicted (reconstructed) HDR signal in the k-th iteration.

[0170]

[0171]

[0172] The optimal MMR coefficients in backward shaping can be found as a least-squares solution:

[0173] and

[0174]

[0175] The predicted HDR value can be expressed as

[0176] as well as

[0177]

[0178] Iterative control and calculation of the corrected input SDR signal in steps 630C and 640C:

[0179] At the end of each iteration, based on the reconstructed HDR signal and the input HDR signal, we determine the correction to the input SDR signal. and Quantity and These correction values ​​are and Functions:

[0180]

[0181] ε is the threshold used to determine whether to continue input SDR image correction in step 640C. α ch This is the convergence scaling factor, used as a trade-off between convergence speed and distortion. To avoid excessive changes in the input SDR signal, we can set the upper limit of the correction to...

[0182]

[0183] In step 640C, the corrected input SDR value is updated for the next iteration:

[0184]

[0185] Iteration termination:

[0186] As referenced above Figure 2 The iteration termination is performed as discussed in the section “Mathematical Description of an Embodiment of Method 200”.

[0187] Example 3: 3D Box-Based MMR

[0188] Below, we outline an example of method 700.

[0189] The basis of 3D binning:

[0190] make Let represent the three color values ​​of the i-th pixel from the input HDR signal in frame t. This represents the corresponding pixel in the input SDR image at pixel i in frame t. We use a fixed number of bins for each color component. To quantize an HDR image with three channel values ​​(Y, C0, and C1), we use minimum / maximum values ​​covering each dimension. Calculate the value using the unified partition boundary. 3D histogram. The quantization interval for each channel is...

[0191]

[0192] Represent the 3D histogram box as in, therefore, Total includes Each 3D box is indexed by a box index. The bin index is specified to represent the number of pixels with these 3-channel quantization values. To simplify the notation, we vectorize the 3D bin index {q} to the 1D index {q}.

[0193]

[0194] We also calculate the sum of each color component in the SDR for each 3D box. Let and These are the luminance and chrominance values ​​mapped in the SDR image domain, such that each of these bins contains the sum of all HDR luminance and chrominance (C0 and C1, respectively) pixel values, where the corresponding pixel value is located in that bin.

[0195] Assuming we have P pixels, these operations can be summarized in pseudocode as follows:

[0196]

[0197] The next step is to find the 3D histogram bins with a non-zero pixel count. In other words, we collect all non-zero entries to set... We calculate HDR and SDR The average value is summarized in the pseudocode as follows:

[0198]

[0199] Let P t express The number of elements in the middle.

[0200]

[0201] and

[0202]

[0203] Then, for For the elements in the array, we have mapping pairs and

[0204] optimization:

[0205] This example of method 700 then applies to 3D mapping pairs. and Application iterative MMR-based optimization, where, In forward plastic surgery, we will... As input for prediction chromaticity in (i.e.) and In the k-th iteration, the predicted (shaped SDR) value is: We use these three SDR predicted 3D values ​​as input to back-shaping to predict HDR chroma values. The value predicted in the k-th iteration is Note that some of the matrices used in each iteration do not change and can be computed in advance.

[0206] initialization:

[0207] In the first iteration (k=0), we set the SDR of the corrected input 3D bins to the original input SDR, that is,

[0208] For all 3 channels (ch = y, c0, and c1).

[0209] The extended form of the MMR of the i-th input HDR entry is expressed as:

[0210]

[0211] Collect all P t 10 entries to obtain the extended matrix:

[0212]

[0213] We also define the following matrix, which will be used in the coefficient optimization process:

[0214]

[0215] We have also prepared HDR chroma values ​​of the input 3D bins in vector form:

[0216] as well as

[0217] Notice, and They do not change during iteration because they represent HDR input values. They can be pre-computed and stored in memory without needing to be recomputed.

[0218] Forward plastic surgery:

[0219] Forward MMR coefficients in channels c0 and c1 and The predicted SDR value in the k-th iteration can be obtained using the following formula.

[0220]

[0221]

[0222] in,

[0223] and

[0224] In each iteration, we collect the corrected SDR values ​​from the previous iteration as a vector.

[0225] as well as

[0226] These two vectors will be the target values ​​to be achieved in the k-th iteration of the forward prediction process. Note that the brightness prediction value does not change in each iteration.

[0227] Define the following two vectors to facilitate discussion of coefficient optimization:

[0228] and

[0229]

[0230] The goal of coefficient optimization is to find MMR coefficients that minimize the MSE between the corrected input SDR signal and the predicted SDR signal in the k-th iteration.

[0231]

[0232]

[0233] The optimal MMR coefficients in forward shaping can be solved using the least squares solution:

[0234]

[0235]

[0236] The optimal predicted SDR value in the k-th iteration can be expressed as:

[0237] and

[0238]

[0239] Retrograde plastic surgery:

[0240] In the k-th iteration, the MMR extension form of the predicted SDR pixel is expressed as:

[0241]

[0242] We can collect all P pixels together to form a matrix.

[0243]

[0244] Define the following matrix, which will be used in the coefficient optimization process:

[0245]

[0246] Backward MMR coefficients in channels c0 and c1 and The predicted HDR value in the k-th iteration can be obtained using the following formula.

[0247] and

[0248]

[0249] in,

[0250] and

[0251] We also define the following vectors to facilitate coefficient optimization:

[0252]

[0253]

[0254] The optimization objective is to find MMR coefficients that minimize the MSE between the original input HDR entries and the predicted HDR entries in the k-th iteration.

[0255]

[0256]

[0257] The optimal MMR coefficients in backward shaping can be found as a least-squares solution:

[0258] and

[0259]

[0260] The predicted HDR value can be expressed as

[0261] and

[0262]

[0263] Iterative control and calculation of the corrected input SDR signal:

[0264] At the end of each iteration, based on the reconstructed HDR signal and the original input HDR signal, we determine the corrected input SDR signal. and Quantity and These correction values ​​are and Functions:

[0265]

[0266] The corrected input SDR value is updated for the next iteration:

[0267]

[0268] Iteration termination:

[0269] As referenced above Figure 2 The iteration termination is performed as discussed in the section “Mathematical Description of an Embodiment of Method 200”.

[0270] Example 4: Pixel-based TPBS approximation

[0271] Below, we outline an example of method 800.

[0272] initialization:

[0273] In the first iteration (k=0), we set the corrected input SDR as the original input SDR, that is,

[0274] For all 3 channels (ch = y, c0, and c1).

[0275] Recall that for the ch channel at the k-th iteration, TPBS is defined as

[0276]

[0277] in, It is a forward TPBS basis function, and These are the corresponding forward coefficients. Using all P pixels in a frame, we can construct the matrix form of the design matrix:

[0278]

[0279] We can also define the following matrix, which will be used in the coefficient optimization process:

[0280]

[0281] We have also prepared input HDR chroma values ​​in vector form:

[0282]

[0283] Notice, and They do not change during iteration because they represent HDR input values. They can be pre-computed and stored in memory without needing to be recomputed.

[0284] Forward plastic surgery:

[0285] Through the forward TPBS coefficient in the ch channel The predicted SDR value in the k-th iteration can be obtained using the following formula.

[0286]

[0287] in,

[0288]

[0289] In each iteration, we collect the corrected SDR values ​​from the previous iteration as a vector for each channel:

[0290]

[0291] These three vectors will represent the target values ​​to be achieved in the k-th iteration of the forward prediction. The following vectors for these three channels are defined to facilitate discussion of coefficient optimization:

[0292]

[0293] The goal of coefficient optimization is to find the TPBS coefficients that minimize the MSE between the corrected input SDR signal and the predicted SDR signal in the k-th iteration. For each channel, we solve the following optimization problem:

[0294]

[0295] The optimal TPBS coefficients in forward shaping can be found as a least-squares solution:

[0296]

[0297] The optimal predicted SDR value in the k-th iteration can be expressed as:

[0298]

[0299] Retrograde plastic surgery:

[0300] In the k-th iteration, the predicted TPBS value in HDR is represented as

[0301]

[0302] in, It is a backward TPBS basis function, and These are the corresponding backward coefficients. Using all P pixels in a frame, we can construct the matrix form of the design matrix:

[0303]

[0304] Define the following matrix, which will be used in the coefficient optimization process:

[0305]

[0306] Through the backward TPBS coefficient in the ch channel The predicted HDR value in the k-th iteration can be obtained using the following formula.

[0307]

[0308] in,

[0309]

[0310] We also define the following vectors to facilitate coefficient optimization:

[0311]

[0312] The optimization objective is to find the TPBS coefficients to minimize the MSE between the original input HDR signal and the predicted HDR signal in the k-th iteration.

[0313]

[0314] The optimal MMR coefficients in backward shaping can be found as a least-squares solution:

[0315]

[0316] The predicted HDR value can be expressed as

[0317]

[0318] Iterative control and calculation of the corrected input SDR signal:

[0319] At the end of each iteration, based on the reconstructed HDR signal and the original input HDR signal, we determine the corrected input SDR signal. Quantity The correction amount is and Functions:

[0320]

[0321] To avoid excessive changes in the input SDR signal, we can set the upper limit of the correction to...

[0322]

[0323] The corrected input SDR value is updated for the next iteration:

[0324]

[0325] Iteration termination:

[0326] As referenced above Figure 2 The iteration termination is performed as discussed in the section “Mathematical Description of an Embodiment of Method 200”.

[0327] Example 5: TPBS approximation based on 3D bins

[0328] Below, we outline an example of a modification of Method 800 to be based on 3D binning. The 3D binning is the same as in "Example 3: 3D Bin-Based MMR". Following this 3D binning process, we have mapping pairs and in In forward plastic surgery, we will... As input for prediction All channels (i.e. and In the k-th iteration, the predicted value is... We use the values ​​of these three SDR-predicted 3D bins as input to back-shaping to predict HDR values. The value predicted in the k-th iteration is Note that some of the matrices used in each iteration do not change and can be computed in advance.

[0329] initialization:

[0330] In the first iteration (k=0), we set the input SDR signal of the corrected 3D binning to the input SDR signal of the original 3D binning, that is,

[0331] For all 3 channels (ch = y, c0, and c1).

[0332] The predicted value is:

[0333]

[0334] Through all P in a frame t With these entries, we can construct the matrix form of the design matrix:

[0335]

[0336] We can also define the following matrix, which will be used in the coefficient optimization process:

[0337]

[0338] We have also prepared input HDR ch channels in vector form:

[0339]

[0340] Notice and They do not change during iteration because they represent input HDR values. They can be pre-calculated and stored in memory without needing to be recalculated.

[0341] Forward plastic surgery:

[0342] Through the forward TPBS coefficient in the ch channel The predicted SDR value in the k-th iteration can be obtained using the following formula.

[0343]

[0344] in,

[0345]

[0346] In each iteration, we collect the corrected SDR values ​​from the previous iteration as a vector.

[0347]

[0348] These three vectors will represent the target values ​​to be achieved in the k-th iteration of the forward prediction. The following vectors for these three channels are defined to facilitate discussion of coefficient optimization:

[0349]

[0350] The goal of coefficient optimization is to find the TPBS coefficients to minimize the MSE between the corrected input SDR signal and the predicted SDR signal in the k-th iteration.

[0351]

[0352] The optimal TPBS coefficients in forward shaping can be found as a least-squares solution:

[0353]

[0354] The optimal predicted SDR value in the k-th iteration can be expressed as:

[0355]

[0356] Retrograde plastic surgery:

[0357] In the k-th iteration, the predicted TPBS value in HDR is represented as

[0358]

[0359] in, It is a backward TPBS basis function, and These are the corresponding backward coefficients. Through all P values ​​in a frame... t With these entries, we can construct the matrix form of the design matrix:

[0360]

[0361] Define the following matrix, which will be used in the coefficient optimization process:

[0362]

[0363] Through the backward TPBS coefficient in the ch channel The predicted HDR value in the k-th iteration can be obtained using the following formula.

[0364]

[0365] in,

[0366]

[0367] We also define the following vectors to facilitate coefficient optimization:

[0368]

[0369] The optimization objective is to find the TPBS coefficients to minimize the MSE between the original input HDR signal and the predicted HDR signal in the k-th iteration.

[0370]

[0371] The optimal TPBS coefficients in backward shaping can be solved using the least squares solution:

[0372]

[0373] The predicted HDR value can be expressed as

[0374]

[0375] Iterative control and calculation of the corrected input SDR signal:

[0376] At the end of each iteration, based on the reconstructed HDR signal and the original input HDR signal, we determine the corrected input SDR signal. Quantity The correction amount is and Functions:

[0377]

[0378] The corrected input SDR value is updated for the next iteration:

[0379]

[0380] Iteration termination:

[0381] As referenced above Figure 2 The iteration termination is performed as discussed in the section “Mathematical Description of an Embodiment of Method 200”.

[0382] Feature combination

[0383] Without departing from its scope, the features described above and the features claimed below can be combined in various ways. For example, it will be understood that aspects of one image reshaping method and product described herein can be combined with or exchanged with features of another image reshaping method or product described herein. The following examples illustrate some possible non-limiting combinations of the embodiments described above. It should be clear that many other changes and modifications can be made to the methods, products, and systems described herein without departing from the spirit and scope of this disclosure:

[0384] (A1) A method for generating (a) a forward shaping function for compressing an input high dynamic range (HDR) image into a shaped standard dynamic range (SDR) image and (b) a backward shaping function for decompressing the shaped SDR image into a reconstructed HDR image, the method comprising: (i) optimizing the forward shaping function to minimize the deviation between the shaped SDR image and the input SDR image corresponding to the input HDR image; (ii) optimizing the backward shaping function to minimize the deviation between the reconstructed HDR image and the input HDR image; and (iii) applying correction to the input SDR image until a termination condition is met, and repeating the steps of optimizing the forward shaping function and the backward shaping function based on the corrected input SDR image.

[0385] (A2) In the method denoted as (A1), the step of applying the correction may include applying a bias-based correction value to each pixel of each color component channel in one or more color component channels of the input SDR image, wherein the correction value is proportional to the bias between the reconstructed HDR image and the input HDR image for the corresponding pixel and the corresponding color component channel, at least as long as the bias-based correction value is less than a predefined maximum correction value.

[0386] (A3) In the method denoted as (A2), for each pixel of each color component channel in one or more color component channels of the input SDR image, for the corresponding pixel and the corresponding color component channel, the correction value may be proportional to the deviation between the reconstructed HDR image and the input HDR image.

[0387] (A4) In either of the methods denoted as (A2) and (A3), at least until a predefined maximum number of iterations for optimizing the forward shaping function and the backward shaping function is reached, the termination condition may be that at least the deviation metric between the reconstructed HDR image and the input HDR image is not greater than a threshold amount.

[0388] (A5) In the method denoted as (A4), the deviation measure may be a weighted combination of (i) the deviation between the reconstructed HDR image and the input HDR image and (ii) the deviation between the shaped SDR image and the input SDR image.

[0389] (A6) In either of the methods denoted as (A4) and (A5), the deviation measure may be based solely on a subset of the color component channels that define each of the input SDR image and the input HDR image.

[0390] (A7) In the method denoted as (A6), the color component channel may include a luminance channel and two chrominance channels, and the subset may be the luminance channel.

[0391] (A8) In the method denoted as (A6), the color component channel may include a luminance channel and two chrominance channels, and the subset may be the two chrominance channels.

[0392] (A9) In any of the methods denoted as (A1) to (A8), the step of optimizing the backward shaping function may include applying the backward shaping function to the shaped SDR image generated by the optimized forward shaping function.

[0393] (A10) In any of the methods represented by (A1) to (A9), the step of optimizing the forward shaping function and applying correction to the input SDR image may utilize a local copy of the input SDR image, and the method may further include maintaining an uncorrected version of the input SDR image.

[0394] (A11) represents any of the methods from (A1) to (A10) that may include the steps of optimizing the forward shaping function and optimizing the backward shaping function for each of the multiple color component channels.

[0395] (A12) In the method denoted as (A11), the color component channels may include a luminance channel, a first chroma channel, and a second chroma channel, wherein (a) for the luminance channel, the method optimizes the forward and backward shaping functions only in a single iteration of the step of optimizing the forward and backward shaping functions, and (b) for each of the first and second chroma channels, the step of optimizing the forward and backward shaping functions utilizes multivariate multiple regression.

[0396] (A13) A method for encoding an HDR image may include, after receiving an input HDR image and an input SDR image, performing any of the methods represented by (A1) to (A12) to generate an iteratively optimized forward shaping function and an iteratively optimized backward shaping function. The method may further include outputting (i) the shaped SDR image generated in the last iteration of the step of optimizing the forward shaping function and (ii) parameters characterizing the backward shaping function optimized in the last iteration of the step of optimizing the backward shaping function.

[0397] (B1) A software product for generating (a) a forward shaping function for compressing an input high dynamic range (HDR) image into a shaped standard dynamic range (SDR) image and (b) a backward shaping function for decompressing the shaped SDR image into a reconstructed HDR image, the software product comprising a non-transitory computer-readable medium storing machine-readable instructions. These instructions include (i) a forward optimization instruction, which, when executed by the processor, causes the processor to optimize the forward shaping function to minimize the deviation between the shaped SDR image and the input SDR image corresponding to the input HDR image; (ii) a backward optimization instruction, which, when executed by the processor, causes the processor to optimize the backward shaping function to minimize the deviation between the reconstructed HDR image and the input HDR image; (iii) an SDR image correction instruction, which, when executed by the processor, causes the processor to apply correction to the input SDR image; and (iv) an iterative control instruction, which, when executed by the processor, causes the processor to evaluate whether a termination condition is met, and if the termination condition is not met, executes the SDR image correction instruction and re-executes the forward optimization instruction and the backward optimization instruction based on the corrected input SDR image.

[0398] (B2) In a software product denoted as (B1), an SDR image correction instruction can be configured to, when executed by a processor, cause the processor to apply a deviation-based correction value to each pixel of each color component channel in one or more color component channels of the input SDR image, wherein the correction value is proportional to the deviation between the reconstructed HDR image and the input HDR image for the corresponding pixel and the corresponding color component channel.

[0399] (B3) In either of the software products denoted as (B1) and (B2), the termination condition may include a threshold amount, and the iterative control instructions may be configured to, when executed by the processor, cause the processor to evaluate whether at least the deviation metric between the reconstructed HDR image and the input HDR image is less than the threshold amount.

[0400] (B4) In the software product denoted as (B3), the termination condition may further include a maximum number of iterations, and the iteration control instruction may be configured to, when executed by the processor, cause the processor to perform the following operations: if the maximum number of iterations has been reached, (a) stop re-executing the SDR image correction instruction, the forward optimization instruction, and the backward optimization instruction, and (b) output the parameters characterizing the backward shaping function and output the shaped SDR image.

[0401] (B5) In either of the software products represented as (B3) and (B4), the deviation measure may be a weighted combination of (i) the deviation between the reconstructed HDR image and the input HDR image and (ii) the deviation between the shaped SDR image and the input SDR image.

[0402] (B6) An encoder may include any of the software products represented as (B1) to (B5), a processor for executing machine-readable instructions of the software product, an input interface for receiving an input HDR image and an input SDR image, and an output interface for outputting (i) a shaped SDR image generated using an optimized forward shaping function and (ii) parameters characterizing the optimized backward shaping function.

[0403] Modifications may be made to the above systems, products, and methods without departing from their scope. Therefore, it should be noted that the contents contained in the above specification and shown in the accompanying drawings should be interpreted illustratively rather than restrictively. The following claims are intended to cover all statements made regarding the general and specific features described herein and the scope of the systems, products, and methods disclosed herein, and such statements may be said to fall within them.

Claims

1. A method for generating (a) a forward shaping function for compressing an input high dynamic range (HDR) image into a shaped standard dynamic range (SDR) image and (b) a backward shaping function for decompressing the shaped SDR image into a reconstructed HDR image, the method comprising: The forward shaping function is optimized to minimize the deviation between the shaped SDR image and the input SDR image corresponding to the input HDR image; The backward shaping function is optimized to minimize the deviation between the reconstructed HDR image and the input HDR image; as well as Until the termination condition is met: Apply correction to the input SDR image, and Based on the corrected input SDR image, the steps of optimizing the forward shaping function and the backward shaping function are repeated.

2. The method as described in claim 1, wherein, The steps for applying the correction include: A bias-based correction value is applied to each pixel of each color component channel in one or more color component channels of the input SDR image. For the corresponding pixel and the corresponding color component channel, the correction value is proportional to the bias between the reconstructed HDR image and the input HDR image.

3. The method of claim 2, wherein the deviation-based correction value is limited to less than a predefined maximum correction value.

4. The method of claim 2 or 3, wherein at least until a predefined maximum number of iterations for optimizing the forward shaping function and the backward shaping function is reached, the termination condition being that at least the deviation metric between the reconstructed HDR image and the input HDR image is not greater than a threshold amount.

5. The method of claim 4, wherein the deviation metric is a weighted combination of (i) the deviation between the reconstructed HDR image and the input HDR image and (ii) the deviation between the shaped SDR image and the input SDR image.

6. The method of claim 4, wherein the deviation metric is based only on a subset of the color component channels defining each of the input SDR image and the input HDR image.

7. The method of claim 6, wherein the color component channel comprises a luminance channel and two chroma channels, and the subset is the luminance channel.

8. The method of claim 6, wherein the color component channels include a luminance channel and two chroma channels, and the subset is the two chroma channels.

9. The method of claim 1 or 2, wherein the step of optimizing the backward shaping function comprises applying the backward shaping function to the shaped SDR image generated by the optimized forward shaping function.

10. The method of claim 1 or 2, wherein the steps of optimizing the forward shaping function and applying correction to the input SDR image utilize a local copy of the input SDR image, the method further comprising maintaining an uncorrected version of the input SDR image.

11. The method of claim 1 or 2, further comprising the steps of optimizing the forward shaping function and optimizing the backward shaping function for each of the plurality of color component channels.

12. The method of claim 11, wherein the color component channels include a luminance channel, a first chroma channel, and a second chroma channel, wherein: For the brightness channel, the method optimizes the forward shaping function and the backward shaping function only in a single iteration of the step of optimizing the forward shaping function and the backward shaping function; as well as For each of the first chroma channel and the second chroma channel, the step of optimizing the forward shaping function and the backward shaping function utilizes multivariate multiple regression.

13. A method for encoding HDR images, comprising: Perform the method as described in any one of claims 1 to 12 to generate an iteratively optimized forward integer function and an iteratively optimized backward integer function; Before performing the method as described in any one of claims 1 to 12, the input HDR image and the input SDR image are received; as well as After performing the method as described in any one of claims 1 to 12, the output is (i) the shaped SDR image generated in the last iteration of the step of optimizing the forward shaping function and (ii) parameters characterizing the backward shaping function optimized in the last iteration of the step of optimizing the backward shaping function.

14. A computer program having instructions that, when executed by a computing device or system, cause the computing device or system to perform the method as described in any one of claims 1 to 13.

15. An encoder, comprising: A non-transitory computer-readable medium storing the computer program as described in claim 14; A processor for executing the computer program; An input interface is provided for receiving the input HDR image and the input SDR image. as well as An output interface is provided for outputting (i) the shaped SDR image generated using the optimized forward shaping function, and (ii) parameters characterizing the optimized backward shaping function.

16. A non-transitory computer-readable storage medium having instructions that, when executed by a computing device or system, cause the computing device or system to perform the method as described in any one of claims 1 to 13.

Images