Determines the parameter set for the tone mapping curve

Abstract

The present invention generally relates to the field of video processing, and more specifically to high dynamic range (HDR) image processing. In particular, the present invention relates to a method for determining a parameter set for a tone mapping curve. The method comprises obtaining a plurality of parameter sets, each parameter set defining the tone mapping curve, each parameter set being derived from one of a plurality of HDR video frames. The method further comprises temporally filtering the plurality of parameter sets to obtain a temporally filtered parameter set.

Description

Technical Field

[0001] The present invention generally relates to the field of video processing, and more specifically to high dynamic range (HDR) video or image processing. In particular, the present invention relates to a method for determining a parameter set for a tone mapping curve for tone mapping an HDR video frame. Furthermore, the present invention relates to an encoder or decoder for encoding or decoding an HDR video frame, respectively. The encoder or decoder may be configured to perform the method. Background Art

[0002] In digital imaging, dynamic range can define the range of brightness of a captured scene, the limits of the brightness range that a particular digital camera or film can capture, or the brightness range supported by a display. In fact, the dynamic range of a typical real-world scene is usually around 10 -3 nits and 10 6 nits. In contrast, consumer displays typically have a much smaller dynamic range. If you want to display real-world scenes like this on such displays, you need to downscale the HDR to a lower dynamic range: a process called tone mapping. Tone mapping is typically non-linear.

[0003] In HDR image and video processing, the Perception Quantization (PQ) curve is often used to convert nits or cd / m 2 The optical signal is converted into an electrical signal between 0 and 1. The equation for a typical PQ curve is shown below:

[0004]

[0005] Wherein, L is the brightness value in the linear domain, ranging from 0 nits to 10000 nits, and L can be an R value, a G value, a B value, or a brightness component Y.

[0006] L' represents an electrical signal in the PQ domain and is included in a range [0, 1], which is generally referred to as a PQ value or a value in the PQ domain.

[0007]

[0008]

[0009]

[0010]

[0011]

[0012] The input of the PQ transfer function is an optical signal in the linear domain, and the output is an electrical signal in the PQ domain. Because there is a one-to-one mapping, if no quantization is applied, the input and output values ​​are effectively equivalent. They just reside in two different domains: the linear domain and the PQ domain.

[0013] The PQ Optical-Electro Transfer Function (PQ OETF) is often used for quantization. HDR images in the linear domain are first transferred to the PQ domain and then quantized to 10 or 12 bits. The PQ domain image is stored or compressed by the codec. Quantization in the PQ domain is more uniform to the human visual system, which is nonlinear. If quantization were performed in the linear domain, the perceived distortion would be much greater.

[0014] As mentioned above, the dynamic range and peak brightness of a display are typically smaller than those of a real-world scene or source HDR image or video. Therefore, tone mapping can be used to adapt, for example, an HDR video frame to the display. Non-linear curves can be used for tone mapping. If the tone mapping curve is the same for all HDR video frames in a video sequence, it is called static tone mapping. However, static tone mapping is not optimal because it does not adapt to the image statistics of each HDR video frame. In contrast, dynamic tone mapping adapts to the statistics of each HDR video frame, so the tone mapping curve varies from frame to frame. Dynamic tone mapping should fully exploit the dynamic range of the display, resulting in higher quality.

[0015] However, dynamic tone mapping is more difficult to control, and a common issue is flicker. Because image statistics vary from frame to frame, the tone mapping curve also varies from frame to frame. In specific applications, metadata contains key information or content characteristics to guide the tone mapping process, playing an important role as a "bridge" between the source (encoder) and the sink (decoder). However, in some cases, the content may vary significantly between consecutive frames, so the metadata and the corresponding tone mapping curve may also change. This can lead to display instability and a phenomenon known as "flicker."

[0016] Existing methods for addressing the aforementioned flicker problem are through standardized dynamic tone mapping schemes, such as SMPTE 2094-10, SMPTE 2094-20, SMPTE 2094-30, and SMPTE 2094-40, as described below. Dynamic tone mapping means that the tone mapping curve parameters are adaptively calculated, adapting to the changing image and therefore varying from frame to frame. Dynamic tone mapping more efficiently utilizes the dynamic range of the display. However, these existing methods do not satisfactorily address the flicker problem.

[0017] The conventional method 1400 proposed by Rafal Mantiuk et al. (“Display Adaptive Tone Mapping” in ACM Transactions on Graphics, Vol. 22, No. 3, August 2008) includes an anti-flicker filter for dynamic tone mapping and Figure 14 In method 1400, dynamic metadata is extracted 1401 from an HDR source (e.g., an HDR video frame), and a corresponding tone mapping curve is generated 1402 based on the metadata. Then, to combat flickering when the tone mapping curve changes rapidly, the tone mapping curve is temporally filtered 1403 before tone mapping 1404 is performed. Specifically, a windowed linear phase FIR digital filter is applied to the nodes of the tone mapping curve before obtaining the final tone mapping result.

[0018] However, the disadvantage of this approach is that the tone mapping curve of the previous frame needs to be stored in memory. In addition, temporal filtering of the complete tone mapping curve is a rather complex process. For example, if a 128-value look-up table (LUT) is used to represent the tone mapping curve and a filtering window size of 16 is used, 128 times 16 values ​​need to be stored in memory. Summary of the Invention

[0019] In view of the above problems and shortcomings, embodiments of the present invention aim to provide an efficient method for improving the quality of HDR videos or video frames, such as reducing flicker.

[0020] This object is achieved by the embodiments presented in the appended independent claims. Advantageous implementations of the embodiments are further defined in the dependent claims.

[0021] According to a first aspect, the present invention relates to a method for determining a parameter set for a tone mapping curve, wherein the method comprises: obtaining a plurality of parameter sets, wherein each parameter set defines a tone mapping curve, and each parameter set is derived based on an HDR video frame among a plurality of HDR video frames; performing time domain filtering on the plurality of parameter sets to obtain a parameter set after time domain filtering.

[0022] According to the method of the first aspect, instead of temporal filtering of the tone mapping curve itself, temporal filtering is performed on the parameter set that defines the tone mapping curve. Therefore, time and computing resources can be saved. In particular, the computational complexity and memory requirements are lower than when the entire chroma mapping curve is temporally filtered. In addition, a tone mapping curve can be obtained based on the parameter set of temporal filtering, by which the stability of the displayed content (in particular, HDR video) is improved, with reduced or even no flicker. In particular, flicker is reduced even when the scene changes rapidly between consecutive HDR video frames in the multiple HDR video frames.

[0023] The metadata may be or may include statistical data defining luminance characteristics of the HDR frame, e.g. derived or extracted from HDR frames and / or other HDR frames of the same scene.

[0024] The method may be performed (for example, in whole or in part) by an electronic device such as an encoder, a decoder, a system including an encoder and a decoder, an HDR system, an HDR television (TV), HDR color grading software, or an HDR video transcoder.

[0025] In an implementation of the first aspect, the performing time-domain filtering on the multiple parameter sets includes: calculating a weighted average or an average value of at least a portion of the parameters in the multiple parameter sets.

[0026] An efficient and effective method is provided for time-domain filtering of the parameter set.

[0027] In an implementation manner of the first aspect, the method further includes: generating the tone mapping curve according to the parameter set after the time domain filtering.

[0028] The tone mapping curve generated in this manner results in reduced flicker, particularly when the plurality of HDR video frames are displayed on a display having a lower dynamic range than the HDR video frames.

[0029] In an implementation manner of the first aspect, each parameter set directly or indirectly defines the tone mapping curve.

[0030] For example, the parameter set directly defining the tone mapping curve may include curve parameters. The parameter set indirectly defining the tone mapping curve may include metadata, and the curve parameters may be calculated based on the metadata.

[0031] In an implementation manner of the first aspect, each parameter set includes metadata of the HDR video frame or one or more curve parameters of the tone mapping curve.

[0032] In an implementation of the first aspect, each parameter set includes metadata extracted from the corresponding HDR video frame, and the temporally filtered parameter set includes the temporally filtered metadata.

[0033] In an implementation of the first aspect, the method further includes: calculating one or more curve parameters of the tone mapping curve according to the metadata after the time domain filtering.

[0034] In an implementation manner of the first aspect, the method further includes: generating the tone mapping curve according to the one or more curve parameters.

[0035] In this way, the flickering problem can be reduced based on the metadata after temporal filtering rather than the tone mapping curve itself.

[0036] In an implementation of the first aspect, each parameter set includes one or more curve parameters of the tone mapping curve calculated based on metadata extracted from the corresponding HDR video frame, and the temporally filtered parameter set includes one or more temporally filtered curve parameters.

[0037] In an implementation manner of the first aspect, the method further includes: generating the tone mapping curve according to the one or more time-domain filtered curve parameters.

[0038] In this way, the flicker problem can be reduced based on the curve parameters of the tone mapping curve after time domain filtering rather than the tone mapping curve itself.

[0039] In an implementation of the first aspect, the mapping curve is given by the following equation:

[0040]

[0041] Wherein, L is the brightness of the input pixel of the HDR video frame, m_n is a first value, in particular, m_n=1, m_m is a second value, in particular, m_m=2.4, m_b is a predetermined perceptual quantization (PQ) value, m_p is a brightness control factor, m_a is a scaling factor defining the maximum brightness of the output pixel, and the one or more curve parameters include m_p and m_a.

[0042] The tone mapping curve is also known as the Phoenix curve and results in particularly low flicker and stability when displaying HDR video frames on low dynamic range displays.

[0043] In an implementation of the first aspect, the method further includes transmitting or storing the temporally filtered parameter set as the multiple HDR video frames and supplementary / side information, for example, as basic metadata or artistic metadata.

[0044] In an implementation of the first aspect, the method further includes: obtaining a first parameter set for a first HDR video frame and pushing the first parameter set into a queue; obtaining a second parameter set for a second HDR video frame; detecting whether a scene change occurs between the first HDR video frame and the second HDR video frame, and if no scene change occurs, pushing the second parameter set into the queue; and calculating an average value of the parameter sets in the queue to obtain a parameter set after time domain filtering.

[0045] Provided is a simple and efficient method for performing time-domain filtering on the parameter set and obtaining the parameter set after time-domain filtering.

[0046] In an implementation of the first aspect, the method further includes clearing the queue if a scene change occurs.

[0047] For example, the method may detect scene changes based on one or more features (e.g., image statistics) of the HDR video frame, such as rapid / dramatic changes in the event of a scene change.

[0048] In an implementation manner of the first aspect, the method is performed by an encoder and / or a decoder.

[0049] According to a second aspect, the present invention relates to an encoder for encoding an HDR video frame, wherein the encoder is configured to perform the method according to the first aspect and / or any implementation of the first aspect.

[0050] According to a third aspect, the present invention relates to a decoder for decoding an HDR video frame, wherein the decoder is configured to perform the method according to the first aspect and / or any implementation of the first aspect.

[0051] According to a fourth aspect, the present invention relates to a system for generating a tone mapping curve, wherein the system comprises the encoder according to the second aspect and / or the decoder according to the third aspect.

[0052] The encoder, the decoder and the system achieve the same advantages and effects as described above for the first aspect and the method for implementing the same.

[0053] According to a fifth aspect, the present invention relates to a computer program, comprising program code for performing the method according to the first aspect and / or any of its implementations when executed by a processor, in particular when executed by a processor of an encoder according to the second aspect and / or by a processor of a decoder according to the third aspect.

[0054] According to a sixth aspect, the present invention provides a non-transitory storage medium storing executable program code, which, when executed by a processor, executes the method according to the first aspect and / or any implementation thereof.

[0055] It should be noted that all devices, elements, units and modules described in this application can be implemented in software or hardware elements or any combination thereof. The steps performed by the various entities described in this application and the functions to be performed by the various entities described are intended to refer to the various entities used to perform the various steps and functions. Even if in the description of the following specific embodiments, the specific functions or steps to be performed by the external entity are not reflected in the description of the specific detailed elements of the entity that performs the specific steps or functions, the technician should be aware that these methods and functions can be implemented in the corresponding software or hardware elements, or in any combination of such elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In conjunction with the accompanying drawings, the following description of specific embodiments will illustrate the above aspects and their implementation methods (various embodiments of the present invention), wherein:

[0057] Figure 1 A schematic diagram of a device (e.g., an encoder or decoder for encoding or decoding HDR video frames, respectively) provided by one embodiment is shown;

[0058] Figure 2 A schematic diagram illustrating a method for determining a parameter set of a tone mapping curve provided by an embodiment is shown;

[0059] Figure 3 Different exemplary tone mapping curves that can be generated by a decoder according to a determined parameter set are shown according to one embodiment;

[0060] Figure 4 A schematic diagram illustrating exemplary metadata for different modes of operation;

[0061] Figure 5 The performance comparison between two tone mapping curves provided by a conventional example and a tone mapping curve generated according to an embodiment is shown;

[0062] Figure 6 A schematic diagram of a process for performing time-domain filtering on multiple parameter sets provided by an embodiment is shown;

[0063] Figure 7 A schematic diagram of a method provided by an embodiment of the invention is shown, wherein metadata is time-domain filtered at the encoder side;

[0064] Figure 8 A schematic diagram of a method provided by an embodiment of the present invention is shown, wherein metadata is temporally filtered at the decoder side;

[0065] Figure 9 A schematic diagram of a method provided by an embodiment of the present invention is shown, wherein the curve parameters of the tone mapping curve are subjected to time domain filtering at the decoder side;

[0066] Figure 10 A schematic diagram of a method provided by an embodiment of the invention is shown, wherein metadata is time-domain filtered at the encoder side;

[0067] Figure 11 A schematic diagram of a method provided by an embodiment of the present invention is shown, wherein the curve parameters of the tone mapping curve are subjected to time domain filtering at the encoder side;

[0068] Figure 12 A schematic diagram of a method provided by an embodiment of the invention is shown, wherein metadata is time-domain filtered at the encoder side;

[0069] Figure 13 An example of an exemplary pipeline showing an HDR dynamic tone mapping process;

[0070] Figure 14 A schematic diagram shows a conventional method for solving the flicker problem by performing temporal filtering on a tone mapping curve. DETAILED DESCRIPTION

[0071] Figure 1 Schematic diagram of a device 100 provided in accordance with an embodiment of the present invention. The device 100 may be an encoder for encoding an HDR video frame 101. Alternatively, the device 100 may be a decoder for decoding an HDR video frame 101. The device 100 may be configured to perform a process for determining a tone mapping curve 300 (e.g., for an exemplary tone mapping curve 300, see Figure 3 ) parameter set method 200 (see also Figure 2 ). Specifically, the determined parameter set may also be used by a decoder to generate a tone mapping curve 300. The tone mapping curve 300 may then be used to tone map the HDR video frame 101.

[0072] Systems can also be formed (for example, see Figures 7 to 11), including at least one such encoder and / or one such decoder. In particular, the device 100 according to an embodiment of the present invention typically operates in such a system. In the system, an HDR video bitstream is typically sent from the encoder to the decoder. The bitstream may include HRD video frames 101 and various metadata.

[0073] It should be noted that the embodiments of the present invention can be Figure 13 The method 200 is implemented in blocks 1301 / 1302 or blocks 1303 / 1304 of the pipeline 1300 shown (particularly showing a system of an encoder and a decoder). In particular, the method 200 can be performed in these blocks, or performed on the decoder or encoder side.

[0074] The device 100 can be used to (refer to Figure 2 and Figure 3 ) obtains 201 a plurality of parameter sets 102, wherein each parameter set 102 defines a tone mapping curve 300, and each parameter set 102 is derived from one of the plurality of HDR video frames 101, for example, a previously processed HDR video frame 101 (encoder) or an encoded HDR video frame 101 (decoder). Furthermore, the apparatus 100 is configured to perform temporal filtering 202 on the plurality of parameter sets 102 to obtain temporally filtered parameter sets 103.

[0075] Thus, as will be explained in more detail later, each parameter set 102 may include metadata 402 for the corresponding HDR video frame 101 (e.g., see Figure 4 ,exist Figure 4 The metadata 402 may be extracted from the corresponding HDR video frame 101 (i.e., as the acquisition step 201). The metadata 402 may then be temporally filtered 202 to obtain temporally filtered metadata 402 (e.g., see Figure 7 ), and one or more curve parameters 502 of the tone mapping curve 300 can be calculated based on the temporally filtered metadata 403 (e.g., see Figure 4 , exemplary curve parameters 502 ). Furthermore, the tone mapping curve 300 may be generated based on one or more curve parameters 502 .

[0076] Alternatively, each parameter set 102 may include one or more curve parameters 502 of the tone mapping curve 300. The one or more curve parameters 502 of the tone mapping curve 300 may be calculated (i.e., as an acquisition step 201) based on the metadata 402 extracted from the corresponding HDR video frame 101. The curve parameters 502 may be temporally filtered 202 to obtain temporally filtered curve parameters 503 (e.g., see Figure 9), and then the tone mapping curve 300 can be generated according to the curve parameters 503 after time domain filtering.

[0077] Can be on the encoder or decoder side (thus acting as device 100, later on Figures 7 to 12 Temporal filtering 202 is performed on the parameter set 102 (i.e., metadata 402 or curve parameters 502) (as described in the various embodiments described above). Generating the curve parameters 502 from the temporally filtered metadata 402 can also be done at the encoder or decoder side. The tone mapping curve 300 is typically generated at the decoder side. Temporal filtering 202 of the parameter set 102, rather than the tone mapping curve 300 itself, yields various advantages, as previously described.

[0078] Device 100 (encoder or decoder) may include processing circuitry ( Figure 1 ), the processing circuit is used to perform, conduct or initiate various operations of the device 100 described herein. The processing circuit may include hardware and software. The hardware may include analog circuits or digital circuits, or both analog circuits and digital circuits. The digital circuit may include components such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP) or a multi-purpose processor. In one embodiment, the processing circuit includes one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code that, when executed by the one or more processors, causes the device 100 to perform, conduct or initiate the operations or methods described herein.

[0079] In particular, the device 100 may include a processor for executing a computer program, which includes a program code for executing the method 200, i.e., for controlling the device 100 to perform the above steps, i.e., obtaining 201 the parameter set 102 and performing time domain filtering 202 on the parameter set 102 to obtain the filtered parameter set 103.

[0080] According to an embodiment of the present invention, an exemplary tone mapping curve 300, also referred to herein as a “Phoenix curve,” may be given by the following equation:

[0081]

[0082] Wherein, L is the brightness of the input pixel of the HDR video frame 101, m_n is a first value, in particular, m_n=1, m_m is a second value, in particular, m_m=2.4, m_b is a predetermined perceptual quantization (PQ) value, m_p is a brightness control factor, and m_a is a scaling factor defining the maximum brightness of the output pixel.

[0083] Other embodiments using the "Phoenix Curve" may use other parameters, for example, m_m may be in the range of 1 to 5, and m_n may be in the range of 0.5 to 2.

[0084] Embodiments may use other non-linear tone mapping curves (besides the “Phoenix curve”), temporally filter the metadata used to define these other non-linear curves, or temporally filter the curve parameters directly for these other non-linear curves.

[0085] In particular, the above-mentioned curve parameters 502 may include parameters m_p and m_a.

[0086] The temporally filtered parameter set 103 can also be used by the decoder to generate the Phoenix tone mapping curve 300. Therefore, filtering the parameter set 102 can be performed by the encoder or the decoder. For example, the temporally filtered parameter set 103 can include the temporally filtered curve parameters 503, which include the parameters m_p and m_a.

[0087] Therefore, the parameter m_p has the physical meaning of a brightness control factor; the larger the m_p, the higher the brightness. Furthermore, m_a is a scaling factor that controls the maximum output brightness of the tone mapping performed using the tone mapping curve 300. The tone mapping curve 300 can be designed in the PQ domain. In other words, both the input L and the output of the tone mapping curve 300 can be referenced to the PQ value. The input L ranges from 0 to 1, where a PQ value of 0 is 0 nits in the linear domain and a PQ value of 1 is 10,000 nits in the linear domain. The output value ranges from 0 to a PQ value equal to or lower than the maximum display brightness of the PQ domain.

[0088] Figure 3Several examples 300A, 300B, and 300C of tone mapping curves 300 with different maximum input luminances and maximum display luminances are plotted in FIG. In particular, the exemplary tone mapping curves 300A, 300B, and 300C that can be generated by a decoder are generated based on different maximum input luminances (e.g., associated with the HDR video frame 101) and maximum display luminances (e.g., of the display to which the HDR video frame 101 will be tone mapped). Furthermore, for example, m_p=5.0. The tone mapping curve 300A can be generated by the decoder based on a maximum input luminance of 10,000 nits and a maximum display luminance of 500 nits. Furthermore, the tone mapping curve 300B can be generated by the decoder based on a maximum input luminance of 10,000 nits and a maximum display luminance of 1,000 nits. Furthermore, the tone mapping curve 300C can be generated by the decoder based on a maximum input luminance of 4,000 nits and a maximum display luminance of 1,000 nits.

[0089] There can be different modes of operation for encoder and decoder systems, and embodiments of the present invention can be applied in different ways. Depending on the mode of the system, the encoder and decoder can operate in different ways, and device 100 can be an encoder or a decoder. That is, method 200 according to an embodiment of the present invention can be performed in an encoder or a decoder, depending on the mode. Encoders and decoders generally have different roles in the mode.

[0090] For example, in conjunction with the China Ultra-HD Video Industrial Alliance (CUVA) HDR standard (CUVA HDR standard), the first mode may be referred to as "automatic mode" hereinafter, and the second mode may be referred to as "artistic mode" hereinafter. In both modes, the parameter set 102 defining the tone mapping curve 300 may be temporally filtered 202 as described above. The two modes of the CUVA HDR standard are briefly described as follows:

[0091] Automatic mode (mode flag tone_mapping_mode=0). In the first mode, the curve parameters 502 used to generate the tone mapping curve 300 are calculated based on the (base) metadata 402 at the decoder side. The metadata 402 (or the temporally filtered metadata 403 in some embodiments) is provided by the encoder to the decoder. The exemplary metadata 402 is Figure 4, and may include typical image statistics, for example, may include (with respect to one or more HDR video frames 101) a minimum luminance value, a maximum luminance value, an average luminance value, and / or a variance of luminance values. Metadata 402 (or in some embodiments, temporally filtered metadata 403) may include a minimum set of parameters sufficient to calculate curve parameters 502. For example, metadata 402 may include the following four parameters (referring to the CUVA HDR standard):

[0092] minimum_maxrgb_pq: The minimum of the maxrgb values ​​of all pixels in the frame. The value is in the PQ domain.

[0093] average_maxrgb_pq: The average of the maxrgb values ​​of all pixels in the frame.

[0094] variance_maxrgb_pq: The difference between the 90th percentile of the maxrgb values ​​of all pixels in the frame and the 10th percentile of the maxrgb values ​​of all pixels in the frame.

[0095] maximum_maxrgb_pq: The maximum value of the maxrgb values ​​of all pixels in the frame. The value is in the PQ domain.

[0096] The maxrgb value of a pixel is the maximum of the pixel's R, G, and B values. The value is in the PQ domain. All four parameters given above are values ​​in the PQ domain (hence each value name ends with _pq).

[0097] Artistic mode (mode flag tone_mapping_mode=1). In the second mode, one or more curve parameters 502 may be calculated using an algorithm, for example, at the content generation stage or at the transcoding stage. In the artistic mode, additional cubic spline parameters such as TH1, TH2, TH3, TH intensity and calculated "Phoenix" curve parameters 502 such as m_a, m_p, m_b may be used. Figure 4As exemplarily shown in FIG, calculated curve parameters 502 (or temporally filtered curve parameters 503 in some embodiments) can be added to metadata 402 (as further metadata 401, e.g., as "art mode metadata") to form extended metadata comprising metadata 402, further metadata 401, and optionally color metadata 404. The extended metadata can be sent to a decoder. The decoder can then be configured to directly perform tone mapping using the curve parameters 502 (or temporally filtered curve parameters 503 in some embodiments) (i.e., to generate a tone mapping curve 300 based on the curve parameters 502 or the temporally filtered curve parameters 503 and perform tone mapping using the generated tone mapping curve 300) if the curve parameters 502 or 503 are suitable for the display on which the HDR video frame 101 is to be displayed. The decoder can also be configured to discard these pre-calculated curve parameters 502 or 503 and fall back to automatic mode, i.e., the decoder can calculate new curve parameters 502 based on metadata 402. Any combination of these two cases can be employed. In particular, artistic mode means that the curve parameters 502 or 503 can be pre-calculated by the encoder and sent to the decoder as other metadata 401. Color-related metadata 404 can also be processed as described above, but they are optional.

[0098] It should be noted that "artistic mode" does not necessarily mean that an artist or colorist is involved. Artificial intelligence (AI) can replace artists or colorists in color grading and help determine curve parameters 502. Therefore, according to the above definition, the fundamental difference between automatic mode and artistic mode is that in automatic mode only (basic) metadata 402 is sent, while curve parameters 502 or 503 can be calculated on the encoder side and embedded in the extended metadata in artistic mode.

[0099] Below, refer to Figure 4 A method for calculating curve parameters 502, specifically parameters m_p and m_a, based on (base) metadata 402 is described. Besides the four parameters in metadata 402, all other variables may be intermediate variables or preset values. Thus, curve parameters 502, namely m_a and m_p, can be determined given the four parameters in exemplary base metadata 402. The following parameters refer to the CUVA HDR standard.

[0100] First, the median value of MAX1 and max_lum can be calculated. Here, two parameters of metadata 402, namely average_maxrgb and variance_maxrgb, can be used.

[0101]

[0102]

[0103] A and B are preset weighting factors, and MIN is the preset value for the lower threshold of max_lum. MaxRefDisplay is the peak luminance of the reference display. This is the display that colorists use to display HDR video during color grading. A typical reference display peak luminance might be 1000 nits or 4000 nits, though other values ​​are possible in practice.

[0104] Second, the parameter m_p can be calculated as follows:

[0105]

[0106] Where avgL is the average of the maxRGB values ​​of all pixels in the frame, TPL0 and TPH0 are the preset thresholds of avgL, PvalueH0 and PvalueH1 are the preset thresholds of m_p, and g0(w0) is the weight.

[0107] Third, the parameter m_p can be updated using max_lum:

[0108]

[0109] Among them, TPL1 and TPH1 are the preset thresholds of max_lum, PdeltaH1 and PdeltaL1 are the preset thresholds of m_p offset, and g1(w1) is the weight.

[0110] First, the parameter m_a can be calculated using m_p. In this step, another intermediate value H(L) is calculated.

[0111]

[0112] m_a=(MaxDISPLAY-MinDISPALY) / (H(MaxSource)-H(MinSource))

[0113] Among them, MaxDISPLAY and MinDISPLAY represent the maximum brightness and minimum brightness of the display, and MaxSource and MinSource represent the maximum brightness and minimum brightness (maxRGB value) of the source HDR video.

[0114] Advantageously, for example, the tone mapping curve 300 generated according to an embodiment of the present invention is more stable than a conventional tone mapping curve. Figure 5 As shown, C1 and C2 represent conventional tone mapping curves (graph C1 is the middle graph, C2 is the lower graph of the three graphs), and “Phoenix” represents a graph according to an embodiment of the present invention (e.g., Figure 3The tone mapping curve 300 (shown in the example of Figure 5 ). Because the "Phoenix" tone mapping curve 300 is inherently more stable than conventional tone mapping curves, temporal filtering of the entire tone mapping curve 300 need not be computationally intensive. Therefore, in accordance with embodiments of the present invention, temporal filtering 202 may be performed only on the parameter set 102 (either metadata 402 or curve parameters 502). Compared to temporally filtering the complete tone mapping curve 300, which would be stored in, for example, a 128-value look-up table (LUT), the metadata 402 may include only the four parameters described above. Thus, temporal filtering 202 on the metadata 402 may be, for example, 128 / 4 = 32 times more efficient than full filtering of the tone mapping curve 300.

[0115] Figure 6 A schematic diagram shows an exemplary process for obtaining a filtered parameter set 103, which can be performed by an encoder or decoder as a device 100 according to an embodiment of the present invention. As described above, the process works in the same manner as filtering metadata 402 and filtering curve parameters 502. The process includes the following steps:

[0116] First, at least a first parameter set 102 of a first HDR video frame 101 can be acquired 201, and the acquired first parameter set 102 can be pushed into a queue (not shown). Then, a second parameter set 102 of a second HDR video frame 101 can be acquired 201 (as shown). Then, if a scene change occurs between the first HDR video frame 101 and the second HDR video frame 101, the scene change can be detected (block 601). If no scene change occurs (no in block 601), the second parameter set 102 is pushed into the queue (block 602). If a scene change occurs (yes in block 601), the queue is cleared (block 603). In the first case (no in block 601), the average value of the parameter sets 102 in the queue can be calculated (block 604) as a temporal filtering step 202 to obtain the temporally filtered parameter set 103.

[0117] In one embodiment, time-domain filtering 202 of the plurality of parameter sets 102 includes computing a weighted average (eg, as performed in block 604 ), or an average of at least a portion of the parameters of the plurality of parameter sets 102 .

[0118] In particular, a weighted average of the queues may be calculated (block 604) to obtain the filtered parameter set 103 for the HDR video frame 101 in the time domain. The time domain filtering 202 may be performed according to the following equation:

[0119]

[0120] Where Q(k) is the kth value in the queue, w k is the weight. The sum of all weights is equal to 1. By default, all weights can be equal, that is, w = 1 / n. In one embodiment, a value closer to the current HDR video frame 101 ( Figure 6 ) assigns a greater weight to the parameter sets of HDR video frames 101 that are larger than the "second frame" in the queue. Tests have shown that these embodiments provide only a small quality gain compared to equal weights. Therefore, embodiments with equal gains provide an efficient implementation because they provide almost similar quality but with lower complexity. It should be noted that n may not always be the maximum length of the queue. At the beginning of the video or at an HDR video frame 101 where a scene cut (i.e., a scene change) occurs, n can be reset to 0 and then increased by 1 frame by frame until it reaches the maximum length of the queue. Once the queue is full, a "first in, first out" rule may be followed. In other words, the parameter set 102 for the oldest HDR video frame 101 in the queue can be popped out, and the parameter set 102 for the latest HDR video frame 101 can be pushed in.

[0121] Figure 6 The procedure has the following advantages: by temporally filtering 202 the parameter set 102, time and computing resources can be saved, and once the tone mapping curve 300 is calculated based on the temporally filtered parameter set 103, the stability of the displayed content can still be ensured without flickering. The queue length for temporal filtering 202 can depend on the frame rate and, by default, on the number of HDR video frames 101 in half a second to one second. For example, if the frame rate of the HDR video is 30 fps (frames per second), a reasonable queue length can be between 15 and 30. In one embodiment, a power of 2 can be used as the queue length, so in software, 32 can be used for 30 fps and 16 for 15 fps.

[0122] Figure 7 This procedure can be used to temporally filter 202 the (e.g., base) metadata 402, and temporally filter 202 the curve parameters 502 (e.g., parameters m_p and m_a). In the artistic mode, the curve parameters m_p and m_a can also be part of the enhanced metadata, and the enhanced metadata can also be temporally filtered.

[0123] While embodiments of the present invention include that temporal filtering 202 may be applied to both metadata 402 and curve parameters 502, filtering metadata 402 may be more beneficial for two reasons:

[0124] (1) Curve parameters 502, such as parameters m_a and m_p, may be nonlinear functions of metadata 402. Therefore, filtering these curve parameters 502 in the nonlinear domain may be more difficult to control. Metadata 402 may be more suitable for such linear filtering.

[0125] (2) In automatic mode, the filter curve parameters 502, such as the parameters m_a and m_p, are only available in the decoder. It may not be possible to filter the curve parameters 502 at the encoder side because they are not sent by the encoder to the decoder in automatic mode. However, the metadata 402 can be filtered at both the encoder and decoder sides in automatic mode.

[0126] The above embodiment provides the advantages of reduced computational complexity and storage requirements compared to temporal filtering of the entire tone mapping curve 300. In addition, potential flickering phenomena are avoided or reduced, and the stability of the displayed content can be guaranteed even when the scene changes rapidly between consecutive HDR video frames 101.

[0127] In the following, about Figures 7 to 9 Some specific embodiments of the first mode (eg, for the automatic mode) are described.

[0128] Figure 7 An embodiment is shown in which the apparatus 100 is an encoder in a coding system. In particular, metadata 402 (as parameter sets 102) is subjected to time domain filtering 202 at the encoder side.

[0129] Specifically, the encoder first extracts 701 (as an acquisition step 201) metadata 402 from a plurality of HDR video frames 101. The metadata 402 is then temporally filtered 202 at the encoder side. As for temporal filtering 202, the metadata 402 may be temporally filtered using the same Figure 6 After obtaining the filtered metadata 403 (as the filtered parameter set 103), the filtered metadata 403 is provided to the decoder. The decoder calculates 702 the curve parameters 502 based on the time-domain filtered metadata 403. In addition, the decoder generates 703 the tone mapping curve 300 based on the curve parameters 502.

[0130] Figure 8 An embodiment is shown where the apparatus 100 is a decoder in a codec system. In particular, metadata 402 (as parameter sets 102) is temporally filtered 202 at the decoder side.

[0131] Specifically, the encoder first extracts 701 metadata 402 from a plurality of HDR video frames 101 and then sends the metadata 402 to the decoder. The decoder receives (as an acquisition step 201) the metadata 402 and performs temporal filtering 202 on the metadata 402 to obtain temporally filtered metadata 403 (as a filtered parameter set 103). As for the temporal filtering 202 performed on the decoder side, the metadata 402 can be obtained using the same Figure 6 After obtaining the filtered metadata 403, the tone mapping curve parameters 502 are calculated 702 at the decoder side in order to generate 703 the tone mapping curve 300 at the decoder side.

[0132] Figure 9 An embodiment is shown where the apparatus 100 is a decoder in a codec system. In particular, the curve parameters 503 (as parameter set 102) are time-domain filtered 202 at the decoder side.

[0133] Specifically, the encoder first extracts 701 metadata 402 from a plurality of HDR video frames 101 and then sends it to the decoder. The decoder then calculates 702 (as the acquisition step 201) curve parameters 502 and then performs temporal filtering 202 on the curve parameters 502 to obtain temporally filtered curve parameters 503 (as the filtered parameter set 103). As for the temporal filtering 202, the curve parameters 502 can be obtained using the same Figure 6 After obtaining the filtered curve parameters 503 , the decoder generates 703 the tone mapping curve 300 according to the time-domain filtered curve parameters 503 .

[0134] In the following, about Figures 10 to 12 Some specific embodiments of the second mode (eg, art mode) are described.

[0135] Figure 10 An embodiment is shown in which the apparatus 100 is an encoder in a coding system. In particular, metadata 402 (as parameter sets) is temporally filtered 202 at the encoder side.

[0136] Specifically, the encoder first extracts 701 (as an acquisition step 201) metadata 402 from a plurality of HDR video frames 101. Then, temporal filtering 202 is performed on the encoder side to obtain filtered metadata 103 (as a parameter set 103 after temporal filtering). As for temporal filtering 202, the metadata 402 can be obtained using the same Figure 6The same processing procedure is shown in FIG. After obtaining the filtered metadata 403, the encoder calculates 702 the curve parameters 502 and can add 1000 the curve parameters to the metadata 402 as other metadata 401 to obtain extended metadata 1001. The extended metadata 1001 is sent to the decoder, which extracts the other metadata 401 therefrom, thereby extracting the curve parameters 502, and generates 703 the chroma mapping curve 300 based on these extracted curve parameters 502.

[0137] Figure 11 An embodiment is shown in which the apparatus 100 is an encoder in a coding system. In particular, the curve parameters 502 (as parameter set 102) are subjected to time domain filtering 202 at the encoder side.

[0138] Specifically, the encoder first extracts 701 metadata 402 from a plurality of HDR video frames 101. The encoder then calculates 702 (as the acquisition step 201) curve parameters 502. The encoder then performs temporal filtering 202 on the curve parameters 502 to obtain filtered curve parameters 503 (as the temporally filtered parameter set 103). As for the temporal filtering 202, the curve parameters 502 can be obtained using the same Figure 6 After obtaining the curve parameters 503 after temporal filtering, the encoder can add 1000 the curve parameters 503 after temporal filtering to the metadata 402 as other metadata 401 to obtain extended metadata 1001. The extended metadata 1001 is sent to the decoder, which extracts the other metadata 401 therefrom, thereby extracting the curve parameters 503 after temporal filtering, and generates 703 the chroma mapping curve 300 based on these curve parameters 503.

[0139] Figure 12 An embodiment is shown where the apparatus 100 is an encoder in a codec system. In particular, metadata 402 (as parameter sets 102) is temporally filtered 202 at the encoder side.

[0140] Figure 12Specifically, an example of temporally filtering metadata 402 for SMPTE 2094-10 is shown. In this embodiment, "fitting points" that are part of the metadata 402 (and therefore part of the filtered parameter set 103 after temporal filtering 403) can be used by the encoder to calculate 702 tone mapping curve parameters 502 c1, c2, and c3. Specifically, temporal filtering 202 can be applied to the fitting points, i.e., to obtain the temporally filtered fitting points. The temporally filtered metadata 402 is then sent to the decoder as additional metadata 401 within enhancement metadata 1001 (e.g., as embedded art mode metadata). Finally, the decoder generates 703 the tone mapping curve 300 based on the extracted additional metadata 401 and the resulting curve parameters 502.

[0141] Similarly, in SMPTE 2094-40, anchor points can be used to determine curve parameters 502, so in particular, temporal filtering 202 can also be applied to these anchor points. In SMPTE 2094-20 and 2094-30, three parameters 502 in the metadata 402 determine the tone mapping curve 300, including "ShadowGainControl", "MidtoneWidthAdjustmentFactor" and "HighlightGainControl", so temporal filtering 202 can be applied to these three parameters in particular.

[0142] In all the above embodiments, the metadata 402 may be dynamic metadata, i.e., the metadata 402 may change from frame 101 to frame 101. Furthermore, in all the above embodiments, the curve parameters 502 may be parameters m_a and m_p, which may be used to define the Phoenix tone mapping curve 300.

[0143] exist Figures 9 to 11 In the embodiment shown above, metadata 402 may be used to calculate curve parameters 502, for example before or after temporal filtering 202 of metadata 402. The cubic spline parameters and the calculated curve parameters 502 may then be combined as further metadata 401 embedded in enhanced metadata 1001.

[0144] Figure 13An example of a signal processing pipeline 1300 for implementing the HDR dynamic tone mapping process according to an embodiment of the present invention is shown. The system input is HDR video, for example, HDR video frames. Typically, this HDR video is output from a post-production stage, where a colorist uses a color grading system to edit the video for better quality or artistic purpose. HDR video has a high peak brightness, typically 1000 or 2000 nits, and in the near future, 4000 or 10,000 nits. Furthermore, the pixel values ​​of the video are in the PQ domain.

[0145] In the HDR pre-processing block 1301, the HDR video remains the same as the input. However, metadata is computable. Furthermore, in the HDR video encoding block 1302, the HDR video is compressed, for example, by a video codec, such as a video codec according to H.265 or any other video standard (national, international, or proprietary). In addition, the metadata is embedded in the header of the video stream that is sent from the encoder to the decoder (or stored on a storage medium for later retrieval by the decoder). In the HDR video decoding block 1303, the decoder receives the HDR video stream, decodes the compressed (or encoded) video, and extracts the metadata from the header.

[0146] Furthermore, in the HDR dynamic tone mapping block 1304, tone mapping is performed to adapt the HDR video to the display capacity.

[0147] For example, the HDR pre-processing block 1301 and / or the HDR dynamic tone mapping block 1304 may implement embodiments of the present invention.

[0148] The present invention has been described with reference to various exemplary embodiments and implementations. However, those skilled in the art will be able to understand and arrive at other variations by practicing the invention, studying the drawings, the present invention, and the appended claims. In the claims and the specification, the word "comprising" does not exclude other elements or steps, and "a" does not exclude a plurality of elements or steps. A single element or other unit may fulfil the functions of several entities or items described in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not mean that a combination of these measures cannot be effectively used.

Claims

1. A method for determining a parameter set of a tone mapping curve, characterized in that The method comprises: Obtaining a plurality of parameter sets, wherein each parameter set is used to define a tone mapping curve, and each parameter set is extracted from one of a plurality of high dynamic range (HDR) video frames; Performing time-domain filtering on the multiple parameter sets to obtain a time-domain filtered parameter set; The obtaining of multiple parameter sets includes: Obtain a first parameter set for a first HDR video frame, and push the first parameter set into a queue; Obtain a second parameter set for a second HDR video frame; detecting whether a scene change occurs between the first HDR video frame and the second HDR video frame, and if no scene change occurs, pushing the second parameter set into the queue; The performing time-domain filtering on the multiple parameter sets to obtain a parameter set after time-domain filtering includes: An average value of the parameter sets in the queue is calculated to obtain the parameter set after the time domain filtering.

2. The method according to claim 1, characterized in that The performing time-domain filtering on the multiple parameter sets includes calculating a weighted average or an average value of at least a portion of the parameters in the multiple parameter sets.

3. The method according to claim 1 or 2, characterized in that Also includes: The tone mapping curve is generated according to the parameter set after the time domain filtering.

4. The method according to any one of claims 1 to 3, characterized in that Each parameter set directly or indirectly defines the tone mapping curve.

5. The method according to any one of claims 1 to 4, characterized in that Each parameter set includes metadata of the HDR video frame or one or more curve parameters of the tone mapping curve.

6. The method according to any one of claims 1 to 5, characterized in that Each parameter set includes metadata extracted from the corresponding HDR video frame. The time-domain filtered parameter set includes time-domain filtered metadata.

7. The method according to claim 6, characterized in that Also includes: One or more curve parameters of the tone mapping curve are calculated according to the metadata after the temporal filtering.

8. The method according to claim 7, characterized in that Also includes: The tone mapping curve is generated according to the one or more curve parameters.

9. The method according to any one of claims 1 to 5, characterized in that Each parameter set comprises one or more curve parameters of the tone mapping curve, the one or more curve parameters being calculated based on metadata extracted from the corresponding HDR video frame, The time-domain filtered parameter set includes one or more time-domain filtered parameters.

10. The method according to claim 9, characterized in that Also includes: The tone mapping curve is generated according to the one or more time-domain filtered curve parameters.

11. The method according to any one of claims 1 to 10, characterized in that The tone mapping curve is defined by the following formula: ， Wherein, L is the brightness of the input pixel of the HDR video frame, m_n is the first value, m_m is the second value, m_b is the predetermined perceptual quantization PQ value, m_p is the brightness control factor, and m_a is the scaling factor defining the maximum brightness of the output pixel. The one or more curve parameters include m_p and m_a.

12. The method according to claim 11, characterized in that m_n=1, m_m=2.

4.

13. The method according to any one of claims 1 to 12, characterized in that Also includes: The parameter set after the temporal filtering and the multiple HDR video frames are transmitted or stored.

14. The method according to claim 1, wherein include: If a scene change occurs, the queue is cleared.

15. The method according to any one of claims 1 to 14, characterized in that The method is performed by an encoder and / or a decoder.

16. An encoder for encoding HDR video frames, characterized in that The encoder is used to: Perform the method according to any one of claims 1, 2, 4 to 7, 9 and 11 to 14.

17. A decoder for decoding HDR video frames, characterized in that The decoder is used to: Performing a method according to any one of claims 1 to 14.

18. A system for generating a tone mapping curve, characterized in that The system comprises: An encoder according to claim 16 and / or a decoder according to claim 17.

19. A non-volatile storage medium, characterized in that: The storage medium stores program instructions, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 14.

Images