Neutral color protection for single-layer backward compatible codec

Abstract

A new method and a new system for processing a single-layer backward compatible codec with multi-channel multiple regression coefficients provided or directed in metadata, such that the coefficients have been biased to prevent shifting of neutral colors. A pseudo-neutral color block is used with a saturation weighting factor to bias the coefficient.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to European Patent Application 22 175 096.1 (case number: D22021EP) and U.S. Provisional Patent Application Serial No. 63 / 345,161 (case number: D22021USP1), both of which were filed on May 24, 2022, and are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to images and more particularly to preserving neutral colors in a single-layer backward compatible codec. Background Art

[0004] As used herein, the term "dynamic range" (DR) can relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance (luma)) in an image, for example, from dark gray (black) to the brightest white (highlight). In this sense, DR is related to "scene-referenced" intensity. DR can also relate to the ability of a display device to fully or approximately render an intensity range of a particular width. In this sense, DR is related to "display-referenced" intensity. Unless a particular sense is explicitly specified at any point in this specification to have a special meaning, it should be inferred that the term can be used in either sense, for example, interchangeably.

[0005] As used herein, the term "dynamic range" (DR) can relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance (luma)) in an image, for example, from dark gray (black) to the brightest white (highlight). In this sense, DR is related to "scene-referenced" intensity. DR can also relate to the ability of a display device to fully or approximately render an intensity range of a particular width. In this sense, DR is related to "display-referenced" intensity. Unless a particular sense is explicitly specified at any point in this specification to have a special meaning, it should be inferred that the term can be used in either sense, for example, interchangeably.

[0006] As used herein, the term high dynamic range (HDR) is associated with a DR width that spans 14-15 orders of magnitude across the human visual system (HVS). In practice, the DR that humans may perceive simultaneously across a wide range of intensities may be truncated to some extent compared to HDR. As used herein, the terms visual dynamic range (VDR) or enhanced dynamic range (EDR) may be associated, either individually or interchangeably, with the DR that the human visual system (HVS) including eye movements may perceive in a scene or image, allowing for some light adaptation changes across scenes or images. As used herein, VDR may be associated with a DR that spans 5 to 6 orders of magnitude. Therefore, while it may be narrower than a real scene with reference HDR, VDR or EDR still represents a wide DR width and may also be referred to as HDR.

[0007] In practice, an image includes one or more color components (e.g., luma Y and chroma Cb and Cr), where each color component is represented by n bits of precision per pixel (e.g., n=8). For example, using gamma luminance encoding, images with n≤8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images with n≥10 can be considered enhanced dynamic range images. HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating point formats, such as those developed by Industrial Light and Magic. TM ) TM File format.

[0008] Currently, most consumer desktop monitors support 200 to 300 cd / m 2 Or nits. Most consumer high-definition televisions (HDTVs) range from 300 to 500 nits, with newer models reaching 1,000 nits (cd / m 2 ). Therefore, this traditional display represents a lower dynamic range (LDR) relative to HDR, also known as standard dynamic range (SDR). Due to advanced capture devices (such as cameras) and HDR displays (for example, the PRM-4200 from Dolby Laboratories), TM With the advent of professional reference monitors), the availability of HDR content has grown, and HDR content may be color graded and displayed on monitors that support a higher dynamic range (e.g., from 1000 nits to 5000 nits or more).

[0009] The term "PQ" as used herein refers to perceived brightness amplitude quantization. The human visual system reacts to increased light levels in a very nonlinear manner. The ability of humans to see a stimulus is affected by the brightness of the stimulus, the size of the stimulus, the spatial frequencies that make up the stimulus, and the brightness level to which the eye adapts at a particular moment of viewing the stimulus. In some embodiments, the perceptual quantizer function maps a linear input grayscale level to an output grayscale level that better matches the contrast sensitivity threshold in the human visual system. An example PQ mapping function is described in SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays" (hereinafter referred to as "SMPTE"), which is incorporated herein by reference in its entirety, where a fixed stimulus size is given, for each brightness level (e.g., stimulus level, etc.), the minimum visible contrast step size at that brightness level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

[0010] As used herein, multi-channel multivariate regression (MMR) refers to a method that allows an encoder to approximate / predict a higher dynamic range image (e.g., HDR) given a lower dynamic range image (e.g., SDR) and an MMR model. An example is provided by U.S. Patent No. 8,811,490, “Multi-color channel multivariate regression predictor” by Guan-Ming Su et al., which is incorporated herein by reference in its entirety.

[0011] As used herein, single-layer backward compatible (SLBC) refers to a single-layer coding system that supports both higher dynamic range displays and lower dynamic range displays. This can be achieved by using MMR coefficients provided in metadata, either directly (e.g., coefficients as data) or indirectly (e.g., a pointer to one of multiple pre-generated MMR models). U.S. Patent No. 11,277,627 by Qing Song et al., "High-Fidelity Full Reference And High-Efficiency Reduced Reference Encoding In End-To-End Single-Layer Backward Compatible Encoding Pipeline," provides an example, which is incorporated herein by reference in its entirety.

[0012] As used herein, "d3DMT" refers to a dynamic 3D mapping table. The d3DMT is constructed from an HDR image and an SDR image and is used to forward shape the chroma codewords in the HDR image to shape the chroma codewords in the shaped SDR image to achieve relatively high (e.g., highest, etc.) perceived color fidelity. U.S. Patent No. 11,277,627 cited above gives an example.

[0013] WO 2021 / 076822 A1 discloses a method for encoding forward shaped image data in a video signal. A backward shaping mapping table is initially generated as an inverse of a forward shaping mapping table. The backward shaping mapping table is updated by applying a luminance forward mapping to the sampled luminance codewords, replacing the luminance codewords of the content mapping with the forward shaped luminance codewords. The luminance forward mapping is constructed by the forward shaping mapping table. The backward shaping mapping table and the luminance forward mapping are used to generate a backward shaping mapping for creating a reconstructed image from the forward shaped image. The forward shaped image and image metadata specifying the backward shaped mapping are encoded in the video signal. A receiving device of the video signal applies the backward shaping mapping to the forward shaped image to create a reconstructed image of a second dynamic range.

[0014] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section. Summary of the invention

[0015] The invention is defined by the independent claims. The dependent claims relate to optional features of certain embodiments. An embodiment for implementing neutral color protection includes a method for constructing a dynamic 3D mapping table (d3DMT) from a first reference image and a second reference image, the first reference image having a larger dynamic range than the second reference image; calculating a content color matrix based on the d3DMT; calculating content saturation from the d3DMT; calculating at least one weighting factor from the content saturation; constructing a neutral color set; calculating a neutral color matrix based on the neutral color set; combining the content color matrix, at least one weighting factor and the neutral color matrix to solve for multi-channel multiple regression (MMR) coefficients; and providing metadata containing data related to the MMR coefficients to a decoder to allow backward compatibility with a single layer bitstream.

[0016] This method can be programmed / built into an encoder as a codec.

[0017] The systems and methods are not limited to the embodiments described above, and further details and embodiments are provided in the description and drawings provided herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG. 1 shows an example of a prior art SLBC codec without neutral color protection.

[0019] Figure 2 An example of a SLBC codec with neutral color protection is shown.

[0020] Figure 3 Example graphs of color distortion (average) versus chroma saturation (d3DMT) for the Cb and Cr channels are shown.

[0021] Figure 4 Examples of saturation and minimum weighting factors for the Cb and Cr channels are shown.

[0022] Figure 5 Shows from Figure 4 Example of the resulting clipping function.

[0023] Figure 6 An example graph of neutral color region MAD with optimized weighting factors in Cb and Cr is shown.

[0024] Figure 7 An example graph of the entire image area MAD with optimized weighting factors in Cb and Cr is shown. DETAILED DESCRIPTION

[0025] As used herein, "neutral color" refers to the center (0.5 within the normalized range of 0 to 1) of all color axes (eg, Cb=0.5 and Cr=0.5 in the normalized YCbCr domain).

[0026] The SLBC algorithm consists of two paths, a forward path that maps an input high dynamic range image (e.g., HDR in this article) to a low dynamic range image (e.g., SDR in this article), and a backward path that maps the SDR back to HDR. Figure 1 shows an example SLBC based on using d3DMT to calculate the MMR coefficients for SLBC codec image prediction. A reference HDR image (110) and a corresponding reference SDR image (120) are used to construct a dynamic 3d mapping table (130). From this, a matrix (described as A in this article) and a vector (described as B in this article) (140) are calculated, which solves for the MMR coefficients (described as M in this article) (150).

[0027] Building d3DMT

[0028] The i-th pixel of the reference HDR image frame is represented as the three color channels (y, cb, cr) The corresponding reference SDR image is (The frame index t drops temporarily, but will be added back when solving for the MMR coefficients.) The pixel values ​​are normalized to [0 1], and there are P pixels in an image. The forward path (HDR to SDR) is used as an example here, but the backward path (SDR to HDR) can be derived in the same process.

[0029] For each component, indicate the number of vertical bars Q y , Q Cb , Q Cr Calculate (Q y ×Q Cb ×Q Cr )3D histogram. Let the 3D histogram bars in HDR be Ω Q，v Indicates that, where Q = [Q y , Q cb , Q Cr ].therefore, Contains the total (Q = Q y .Q Cb .Q Cr ) vertical bar such that the vertical bar index q = (q y ,q Cb ,q Cr ) indicates the number of pixels with these 3-channel quantized values.

[0030] For each channel, calculate the minimum value of the HDR image signal (L v，ch ) and maximum value (H v，ch )for:

[0031]

[0032]

[0033] Each channel is uniformly quantized to Q according to the minimum and maximum values ch Vertical bar. The range of vertical bar j is:

[0034] [L v，ch +j·g ch , L v，ch +(j+1)·g ch ), j∈{0, 1, …, Q ch -1} Formula (3)

[0035] in

[0036]

[0037] For a given input HDR value, the vertical bar index can be determined by the following formula:

[0038]

[0039] Calculate the HDR sum in each HDR 3D vertical bar as and Also calculate the SDR sum in each HDR 3D bar. Let and As the mapped SDR luminance and chrominance values, each vertical bar contains the sum of all SDR color channel pixel values, where the corresponding HDR pixel value is in that vertical bar.

[0040] Here's an example of this in action:

[0041] Step 1: Initialization of 3D SDR histogram and 3D mapped HDR chromaticity values:

[0042] Where q=(q y ,q Cb ,q Cr ) and q=0, ..., Q-1

[0043] Where q=(q y ,q Cb ,q Cr ) and q=0, ..., Q-1

[0044] Where q=(q y ,q Cb ,q Cr ) and q=0, ..., Q-1

[0045] Where q=(q y ,q Cb ,q Cr ) and q=0, ..., Q-1

[0046] Where q=(q y ,q Cb ,q Cr ) and q=0, ..., Q-1

[0047] Where q=(q y ,q Cb ,q Cr ) and q=0, ..., Q-1

[0048] Where q=(q y ,q Cb ,qCr ) and q=0, ..., Q-1

[0049] Step 2: Scan each pixel in HDR and SDR chrominance from each of the P color blocks (i):

[0050] / / HDR downsampled brightness quantization value

[0051] / / HDR chroma 0 quantization value

[0052] / / HDR chroma 1 quantization value

[0053] q=(q y ,q Cb ,q Cr )

[0054] / / 3D HDR Histogram

[0055] / / HDR Y value

[0056] / / HDR Cb value

[0057] / / HDR Cr value

[0058] / / SDR Y value

[0059] / / SDR Cb value

[0060] / / SDR Cr value

[0061] Now, find the 3D HDR histogram bins that have a non-zero number of pixels. In other words, discard all those bins that do not have any pixels. 0, q 1, …,q K-1 As K such vertical bars, calculate and The average value is as follows:

[0062] Set the SDR mapping value for each vertical bar (non-zero vertical bar index q i =(q y ,q Cb ,q Cr )), for each (i) of the K vertical bars:

[0063] / / Average HDR Y value

[0064] / / Average HDR Cb value

[0065] / / Average HDR Cr value

[0066] / / Average SDR Y value

[0067] / / Average SDR Cb value

[0068] / / Average SDR Cr value

[0069] All mapping pairs can be collected to build d3DMT.

[0070] Construction of MMR coefficient

[0071] The forward path is the conversion from HDR to SDR (higher dynamic range to lower dynamic range), and the backward path is from lower dynamic range to higher dynamic range. Different MMR coefficients can be used for the forward and backward paths, or the same MMR coefficient can be given to both paths based on which path is more important (for example, if HDR to SDR conversion is prioritized, only the forward path is calculated and then used for both paths).

[0072] Forward Path

[0073] The MMR extension for the i-th HDR entry in 3DMT is represented as

[0074]

[0075] All non-zero entries can be collected together as:

[0076]

[0077] The observed chrominance signal (for example, ch can be the Cb channel or the Cr channel) can be expressed as:

[0078]

[0079] The chth channel of the MMR coefficient is expressed as The predicted SDR signal is

[0080]

[0081] The optimized solution can be obtained through the least squares solution:

[0082]

[0083] It can be interpreted as:

[0084]

[0085] To simplify the discussion, the following notations are further used:

[0086]

[0087] The resulting forward coefficients are:

[0088]

[0089] Backward Path

[0090] The MMR coefficients of the backward path can be calculated by basically the same process as the forward path. Using the predicted SDR image, the predicted To the original HDR image Construct d3DMT. The MMR expansion form for the i-th entry can be expressed as:

[0091]

[0092] Collect all non-zero entries together as follows:

[0093]

[0094] The observed chrominance signal (eg, ch can be the Cb channel or the Cr channel) can be expressed as:

[0095]

[0096] The chth channel of the MMR coefficient is expressed as The predicted HDR signal is

[0097]

[0098] The optimized solution can be obtained by least squares solution

[0099]

[0100] This can be solved using a least squares solution algorithm.

[0101]

[0102] To simplify the discussion, the following notations are further used:

[0103]

[0104]

[0105] The MMR coefficients produced by the above process can be biased towards the majority of colors in the reference image, depending on the color distribution. This can result in tinting of neutral colors, which means that neutral colors are not accurately preserved. In general, the more saturated the image, the greater the chance of neutral color shift. This can be corrected by introducing a set of assumed neutral color patches during the MMR optimization process to provide an effective technical solution to mitigate the neutral color shift problem.

[0106] Neutral color protection

[0107] Figure 2 An embodiment of an improved SLBC algorithm that solves the problem of neutral color shift is shown. As in the previous process, the HDR image (210) and the SDR image (220) are used to create a d3DMT (230). However, two additional elements are added: 1) the saturation of the d3DMT is calculated (233), and a matrix of weighted values ​​w (235) is calculated from the saturation; and 2) a neutral color set is constructed from the neutral color blocks added to the image (243), and additional A matrices and b matrices are calculated from the blocks (245). The additional matrices (w, A NC ,b NC ) and the usual content color matrix (A CC and b CC )(250) are combined to solve for the MMR coefficients (260), which, as a result of the above, have been biased to protect neutral colors.

[0108] In other words, the left side of the flowchart (210, 220, 230, 233, 235, 240) shows the "natural image" path (for content color, or "CC"). Given an HDR-SDR natural image pair (210, 220), first a d3DMT is constructed (230). Based on the forward d3DMT, and (240). Similarly, d3DMT (230) is reused to calculate content saturation (233). The required weighting factors can be obtained by the function (235) Found. Based on the backward d3DMT, calculate and (240); and calculate content saturation (233).

[0109] The right side of the flow chart (243, 245) shows the (added) neutral color ("NC") pathway. (245). The backward path can be used with and and The same process is used to calculate .

[0110] In some embodiments, the system or method includes a neutral color path without a weighting factor. Figure 2 As shown, the system or method includes a natural image path and a neutral color path having weighting factors.

[0111] Then, the following formulas are used to combine (250) natural images and neutral color patches for the forward path and the backward path respectively:

[0112]

[0113]

[0114]

[0115]

[0116] The forward MMR coefficient and the backward MMR coefficient can be solved by the following formula:

[0117]

[0118]

[0119] In order to make the MMR coefficient optimization more inclined to neutral color preservation, modify the matrix and The modification (bias) starts with constructing N pseudo-neutral color patches that do not appear in the content (image).

[0120] Forward Path Bias

[0121] For the forward path, the i-th color block is assigned, i=0, ..., N-1, such as:

[0122]

[0123]

[0124]

[0125] and

[0126]

[0127]

[0128] The MMR expansion form for the i-th HDR neutral color block is

[0129]

[0130] All N color blocks can be collected together as:

[0131]

[0132] The observed chrominance signal (for example, ch can be the Cb channel or the Cr channel) can be expressed as:

[0133] Given the following matrix:

[0134]

[0135] The final A matrix and b vector will be weighted by the weighting factor w (F) Perform a linear combination.

[0136]

[0137]

[0138] The MMR coefficient of the forward path can be solved by the following formula:

[0139]

[0140] Backward Path Bias For the backward path, the i-th color block is assigned, i=0,...,N-1, such that

[0141]

[0142]

[0143]

[0144] and

[0145]

[0146]

[0147] The backward path is the same process. The corresponding reference SDR image is represented as

[0148]

[0149] All N color blocks can be collected together as:

[0150]

[0151] The observed chrominance signal (e.g., ch can be the Cb channel or the Cr channel) can be expressed as

[0152]

[0153] Given the following matrix:

[0154]

[0155] The final A matrix and b vector will be weighted by the weighting factor w (B) Perform a linear combination:

[0156]

[0157]

[0158] The MMR coefficient of the backward path can be solved as follows:

[0159]

[0160] Determine the weighting factors

[0161] As discussed in the previous sections, the forward path needs to be determined and / or backward path We can use empirical methods (experiments) to find the optimal solution, as shown in this paper.

[0162] Content saturation and neutral color deviation

[0163] Content saturation plays an important role in determining the weighting factors. Note that the neutral color (Cb,Cr) value in the YCbCr domain is (0.5,0.5). The distance between each color entry in to the nearest neutral color axis (0.5, 0.5) can be measured as

[0164]

[0165] This is the hue and saturation of an entry. For all valid non-zero For a cube, the average saturation can be calculated as:

[0166]

[0167] In a specific color space (such as R.709 / P3 / R.2020), the average saturation (β (F) ) can be determined within a container (e.g., R.2020). A wider gamut will have a higher average saturation.

[0168] To understand the effect of different values ​​of the weighting factor, we place N bThe grayscale bar of pixels is measured at each pixel. Since the position is known, the deviation of the grayscale bar from the neutral color point 0.5 between the final reconstructed HDR image and the neutral color is measured. The mean absolute difference (MAD) method is used for the chrominance channels from the neutral color block.

[0169] To simplify the design, set When the applied weighting factor is w t , the final reconstructed HDR pixel value is expressed as For pixels within the grayscale bar, MAD can be calculated as:

[0170]

[0171] Larger values ​​indicate a greater deviation from neutral colors and a greater likelihood of introducing non-neutral colors.

[0172] Computational Applications t (For example, ) and unquoted (e.g., ) between the reconstructed HDR and the source HDR for the entire chromaticity diagram. The difference between these two MADs is then calculated.

[0173]

[0174] A negative value means that the chroma accuracy is actually improved with less distortion. A positive value means that the chroma accuracy is reduced due to the introduction of the weighting factor.

[0175] Data Analysis - Empirical Methods for Determining Weighting Factors

[0176] First consider the case where there is no neutral color, that is, setting w t = 0. Figure 3 The content saturation of the two channels is shown in and distortion in grayscale bars Example correlation between . Content saturation has an effect on neutral color deviation. Content saturation The higher the value, the more chromaticity distortion The higher the value, the higher the chance of showing non-neutral color artifacts. Note that the images in this example are in 12-bit precision.

[0177] For each t test image, the minimum value of the weighting factor is searched so that the neutral color bias is less than the threshold δ:

[0178]

[0179] Figure 4 An example of the required (12-bit precision) and content saturation Note that the more saturated the content is, the higher the weighting factor will be needed to preserve neutral colors. Some situations do not require neutral color preservation (i.e. ),For example Figure 4 , where saturation is less than 0.1. Other cases may require different weighting values, such as in Figure 4 In the example above, saturation values ​​below 0.13 may use a weighting of 5, while higher saturation values ​​may require a weighting of 10 or 15.

[0180] Non-neutral color artifacts rarely show up in images from sets showing natural scenes, but it is more often found in very saturated synthetic datasets, especially in certain color spaces (e.g., R.2020).

[0181] The weighting factor selection method used

[0182] Saturation vs. minimum W plot (e.g., Figure 3 ) can be used to find a piecewise linear function that envelops / upper bounds the data points. The required weighting factor w can be determined t . Examples are listed below:

[0183]

[0184] Or it can be simply expressed as a shear function:

[0185]

[0186] Figure 5 The function (510) is shown as a straight line clipped below 0.05 and above 0.2. Different profiles may produce different curves / functions.

[0187] Figure 6 Shows the use of a piecewise function model for each frame An example of MAD is The MAD is within the tolerance range (δ=8) of 12-bit precision. Subjective testing revealed the invisibility of this small neutral color shift.

[0188] Figure 7 shows the use of optimized The entire image MAD For example, Figure 7 As shown, most frames show negative values, which means that using this neutral color protection method does improve the chromaticity. A few frames have values ​​greater than 0. But it is still less than (δ=8). The performance drops in MAD, but in an invisible range.

[0189] In some embodiments herein, a number of neutral color blocks (N) is multiplied by a weighting factor (W). In some embodiments, N is fixed and W can be adjusted to provide neutral color protection. In other embodiments, W can be fixed and the number of neutral color blocks can be increased (e.g., to W*N). Typically, fixing N is less computationally intensive. Fixing W bypasses the step of determining W and fixes W to 1, eliminating the need to multiply W by N in the calculation.

[0190] Hardware Implementation

[0191] An embodiment of the system is as part of a codec used in an encoder-decoder system. The MMR coefficients may be resolved at the decoder (a processor / machine configured to receive images - personal and / or video - and output a data stream with the encoded images and metadata) with metadata sent to the decoder, which includes the coefficient itself or includes a pointer to one of a set of predetermined coefficients stored in the decoder. For example, a decoder may include a library of MMR coefficient sets covering various expected bias levels, with the metadata informing the decoder which set to use for compatibility.

[0192] Therefore, as described herein, embodiments of the present invention may be directed to one or more of the exemplary embodiments listed below. Therefore, the present invention may be embodied in any form described herein, including but not limited to the following enumerated exemplary embodiments (EEE) that describe the structure, features, and functions of certain parts of the present invention:

[0193] EEE1. A method for encoding image data with neutral color protection, the method comprising: constructing a dynamic 3D mapping table (d3DMT) from a first reference image and a second reference image, the first reference image having a different dynamic range from the second reference image; calculating a content color matrix based on the d3DMT; calculating content saturation from the d3DMT; calculating at least one weighting factor from the content saturation; constructing a neutral color set; calculating a neutral color matrix based on the neutral color set; combining the content color matrix, the at least one weighting factor and the neutral color matrix to solve multi-channel multiple regression (MMR) coefficients; and providing metadata including data related to the MMR coefficients to a decoder to allow backward compatibility with a single-layer bitstream.

[0194] EEE2. The method according to EEE 1 is configured to provide an MMR coefficient, the MMR coefficient comprising data related to a forward path MMR coefficient and a backward path MMR coefficient, wherein the forward path MMR coefficient is determined independently of the backward path MMR coefficient.

[0195] EEE3. The method according to EEE 1, wherein: the method is configured to provide the MMR coefficient, and the MMR coefficient includes data related only to the backward path MMR coefficient.

[0196] EEE4. A method according to any one of EEE1 to 3, wherein the content color matrix is ​​in the form of A and b, wherein A is based on a vector of all non-zero vertical bars and b is based on a vector of all non-zero vertical bars and an observed chrominance signal.

[0197] EEE5. A method according to any one of EEE1 to 4, wherein the MMR coefficient is estimated using a least squares solution algorithm.

[0198] EEE6. The method according to EEE 5, wherein the MMR coefficient is A -1 The form of b.

[0199] EEE7. A method according to any one of EEE1 to 6, wherein the combination comprises multiplying a neutral color matrix among the neutral color matrices with one of the at least one weighting factors, and adding the product to a corresponding one of the content color matrices.

[0200] EEE8. A method according to any one of EEE1 to 7, wherein constructing the neutral color set includes constructing a plurality of pseudo-neutral color blocks that do not appear in the first reference image or the second reference image.

[0201] EEE9. The method according to any one of EEE1 to 8, wherein the at least one weighting factor comprises a forward path weighting factor and a backward path weighting factor.

[0202] EEE10. A method according to any one of EEE1 to 9, wherein the metadata comprises a pointer to a set of MMR coefficients stored in the decoder.

[0203] EEE11. A method according to any one of EEE1 to 10, wherein calculating at least one weighting factor includes comparing the content saturation with experimental data.

[0204] EEE12. The method according to EEE 11 further includes setting a linear function to envelop data points in the experimental data.

[0205] EEE13. An encoder configured to perform the method described in any one of EEE1 to 12, the encoder comprising: a processor; a signal input configured to receive the image; and a signal output configured to send the metadata.

[0206] EEE14. A method for encoding image data with neutral color protection, the method comprising: constructing a dynamic 3D mapping table (d3DMT) from a first reference image and a second reference image, the first reference image having a different dynamic range from the second reference image; calculating a content color matrix based on the d3DMT; constructing a neutral color set; calculating a neutral color matrix based on the neutral color set; combining the content color matrix and the neutral color matrix to solve multi-channel multiple regression (MMR) coefficients; and providing metadata including data related to the MMR coefficients to a decoder to allow backward compatibility with a single-layer bitstream.

[0207] Equivalents, extensions, alternatives and miscellaneous

[0208] Thus, example embodiments related to color conversion under coding efficiency constraints for HDR video encoding are described. In the foregoing description, embodiments of the present invention have been described with reference to many specific details, which may vary from implementation to implementation. Therefore, the only and exclusive indication of the present invention and the applicant's inventive intent is the set of claims issued in a specific form according to the present application, wherein such claim issuance includes any subsequent corrections. Any definitions expressly set forth herein for the terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes not expressly cited in the claims should not limit the scope of such claims in any way. Therefore, this specification and the drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A method for encoding image data with neutral color protection, the method comprising: constructing a dynamic 3D mapping table (d3DMT) from a first reference image and a second reference image, the first reference image having a different dynamic range than the second reference image; Calculate a content color matrix based on the d3DMT; calculating content saturation from the d3DMT; calculating at least one weighting factor from the content saturation; Build a neutral color set; Calculating a neutral color matrix based on the neutral color set; combining the content color matrix, the at least one weighting factor, and the neutral color matrix to solve for multi-channel multiple regression (MMR) coefficients; as well as Metadata containing data related to the MMR coefficients is provided to the decoder to allow backward compatibility for single layer bitstreams.

2. The method according to claim 1, wherein: The constructing of a dynamic 3D mapping table (d3DMT) from the first reference image and the second reference image includes determining a 3D histogram bin having a number of non-zero pixels.

3. The method according to claim 1 or 2, wherein: The content color matrix is ​​A CC and b CC of the form, where A CC Based on the vector of all non-zero vertical bars and b CC Based on the vector sum of all non-zero vertical bars and the observed chrominance signal.

4. The method according to any one of claims 1 to 3, wherein: The constructing the neutral color set includes generating a neutral color block vector, the neutral color block vector collecting a plurality of color blocks that do not appear in either the first reference image or the second reference image, and wherein the color blocks are configured to mitigate color shift.

5. The method according to any one of claims 1 to 4, wherein: The neutral color matrix is ​​A NC and b NC of the form, where A NC Based on the neutral color block vector and b NC An observed chromaticity signal represented by the center of all color axes based on the neutral color block vector and the neutral color.

6. The method according to any one of claims 1 to 5, wherein: The combining includes multiplying one of the neutral color matrices by one of the at least one weighting factor and adding the product to a corresponding one of the content color matrices.

7. The method according to any one of claims 1 to 6, wherein: The method is configured to provide MMR coefficients including data related to a forward path MMR coefficient and a backward path MMR coefficient, wherein the forward path MMR coefficient is determined independently of the backward path MMR coefficient.

8. The method according to any one of claims 1 to 7, wherein: The method is configured to provide the MMR coefficients, the MMR coefficients comprising data related only to backward path MMR coefficients.

9. The method according to any one of claims 1 to 8, wherein: The MMR coefficients were determined using a least squares solution algorithm.

10. The method according to claim 9, wherein: The MMR coefficient is A -1 The form of b.

11. The method according to any one of claims 1 to 10, wherein: The at least one weighting factor includes a forward path weighting factor and a backward path weighting factor.

12. The method according to any one of claims 1 to 11, wherein: The metadata includes a pointer to a set of MMR coefficients stored in the decoder.

13. The method according to any one of claims 1 to 12, wherein: Calculating at least one weighting factor includes comparing the content saturation to experimental data.

14. The method of claim 13, wherein calculating at least one weighting factor from the content saturation comprises finding a piecewise linear function to envelop data points in the experimental data, and wherein the piecewise linear function maps the measured content saturation to the at least one weighting factor.

15. An encoder, configured to perform the method according to any one of claims 1 to 14, the encoder comprising: processor; a signal input terminal configured to receive the image; as well as A signal output is configured to transmit the metadata.

Images