Image reshaping using rate-distortion optimization in video coding

Abstract

The present disclosure relates to image reshaping with rate-distortion optimization in video coding. In view of a sequence of images in a first codeword representation, methods, processes and systems are presented for image reshaping with rate-distortion optimization, where the reshaping allows for encoding images in a second codeword representation that allows for more efficient compression than with the first codeword representation. Syntax methods for signaling reshaping parameters are also presented.

Description

[0001] Information related to divisional application

[0002] This application is a divisional application of Chinese invention patent application No. 201980013360.6, filed on February 13, 2019, entitled "Image Reshaping Using Rate Distortion Optimization in Video Coding".

[0003] Cross-reference of related applications

[0004] This application claims priority to U.S. Provisional Patent Application No. 62 / 792,122, filed January 14, 2019; U.S. Provisional Patent Application No. 62 / 782,659, filed December 20, 2018; U.S. Provisional Patent Application No. 62 / 772,228, filed November 28, 2018; U.S. Provisional Patent Application No. 62 / 739,402, filed October 1, 2018; U.S. Provisional Patent Application No. 62 / 726,608, filed September 4, 2018; U.S. Provisional Patent Application No. 62 / 691,366, filed June 28, 2018; and U.S. Provisional Patent Application No. 62 / 630,385, filed February 14, 2018, the entire contents of each of which are incorporated herein by reference. Technical Field

[0005] This invention generally relates to image and video coding. More specifically, embodiments of the invention relate to image reshaping using video coding. Background Technology

[0006] In 2013, the MPEG group within the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU) jointly released a draft of the HEVC (also known as H.265) video coding standard (reference [4]). Recently, the same group has gathered evidence to support the development of a next-generation coding standard that offers improved coding performance compared to existing video coding technologies.

[0007] As used in this article, the term 'bit depth' refers to the number of pixels used to represent one of the color components of an image. Traditionally, images are encoded with 8 bits per pixel per color component (e.g., 24 bits per pixel); however, modern architectures can now support higher bit depths, such as 10 bits, 12 bits, or more than 12 bits.

[0008] In conventional image pipelines, the captured image is quantized using a nonlinear photoelectric function (OETF) that converts linear scene light into a nonlinear video signal (e.g., gamma-coded RGB or YCbCr). Then, at the receiver, before being displayed on the monitor, the signal is processed by an electro-optical transfer function (EOTF) that converts the video signal values ​​into output screen color values. Such nonlinear functions include the conventional "gamma" curves documented in TU-R Rec.BT.709 and BT.2020, the "PQ (Perceptual Quantization)" curve described in SMPTE ST 2084, and the "HybridLog-Gamma" or "HLG" curve described in Rec.ITU-R BT.2100.

[0009] As used herein, the term "forward reshaping" refers to the process of mapping a digital image from its original bit depth and original codeword distribution or representation (e.g., gamma or PQ or HLG and the like) to an image with the same or different bit depth and different codeword distribution or representation. Reshaping allows for improved compressibility or improved image quality at a fixed bit rate. For example (but not limited to), reshaping can be applied to 10-bit or 12-bit PQ encoded HDR video to improve the coding efficiency of a 10-bit video coding architecture. In the receiver, after decompressing the reshaped signal, the receiver can apply an "inverse reshaping function" to restore the signal to its original codeword distribution. The inventors should understand that as next-generation video coding standards begin to develop, integrated image reshaping and improved coding techniques are expected. The method of the present invention can be applied to a variety of video content that includes (but is not limited to) content within the Standard Dynamic Range (SDR) and / or High Dynamic Range (HDR) range.

[0010] The methods described in this section are available methods, but not necessarily methods previously conceived or sought. Therefore, unless otherwise indicated, none of the methods described in this section should be considered prior art solely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the problems identified with respect to one or more methods should not be considered as having been identified in any prior art based on this section. Summary of the Invention

[0011] One aspect of this application relates to a method for reconstructing coded video data using one or more processors. The method includes: receiving a coded bitstream comprising one or more coded reshaped images represented as input codewords; receiving reshaping metadata for the one or more coded reshaped images in the coded bitstream, wherein the reshaping metadata includes parameters for generating a luminance-based chroma residual converter to calculate decoded chroma residuals and a forward reshaping function, wherein the forward reshaping function maps pixels of the image from a first codeword representation to the input codeword representation, wherein the reshaping metadata further includes: a first parameter indicating a minimum grouping index for reshaping; and a second parameter determining an active maximum grouping index for reshaping, wherein the active maximum grouping index is less than or equal to a predetermined maximum grouping index. The method comprises: a large grouping index, and determining the active maximum grouping index includes calculating the difference between the predetermined maximum grouping index and the second parameter; an absolute difference codeword value, which is used for each active group in the input codeword representation; and a positive or negative sign of the absolute difference codeword value, which is used for each active group in the input codeword representation; generating a forward reshaping function based on the reshaping metadata; generating an inverse reshaping function based on the reshaping metadata or the forward reshaping function, wherein the inverse reshaping function maps the pixels of the reshaping image from the input codeword representation to the first codeword representation; and decoding the encoded bitstream based on the forward reshaping function, the inverse reshaping function, and the chroma residual converter.

[0012] Another aspect of this application relates to a method for adaptively reshaping a video sequence using a processor. The method includes: using the processor to access an input image represented by a first codeword; generating reshaping parameters to represent a forward reshaping function that maps pixels of the input image to a second codeword representation; and generating an coded bitstream based on the input image and the forward reshaping function, wherein the reshaping parameters include: a first parameter determining an active maximum codeword block index for reshaping, wherein the active maximum codeword block index is less than or equal to a predetermined maximum codeword block index; a second parameter indicating a minimum codeword block index for reshaping; an absolute difference codeword value for each active codeword block in the second codeword representation; a sign of the absolute difference codeword value for each active block in the second codeword representation; and one or more luminance-based chrominance residual conversion parameters.

[0013] Another aspect of this application relates to a method for transmitting a bitstream generated by a video encoding device and used for reconstructing video. The method includes: transmitting the bitstream, wherein generating the bitstream includes: using a processor to access an input image represented as a first codeword; generating reshaping parameters to represent a forward reshaping function mapping pixels of the input image to a second codeword representation; and generating an coded bitstream based on the input image and the forward reshaping function, wherein the reshaping parameters include: a first parameter determining an active maximum codeword block index for reshaping, wherein the active maximum codeword block index is less than or equal to a predetermined maximum codeword block index; a second parameter indicating a minimum codeword block index for reshaping; an absolute difference codeword value for each active codeword block in the second codeword representation; a sign of the absolute difference codeword value for each active block in the second codeword representation; and one or more luminance-based chrominance residual conversion parameters. Attached Figure Description

[0014] The accompanying drawings illustrate embodiments of the invention by way of example and not limitation, and wherein like element symbols refer to similar elements, and wherein:

[0015] Figure 1A Describe an example process of a video delivery pipeline;

[0016] Figure 1B Describe an example process of compressing data using signal reshaping based on existing technologies;

[0017] Figure 2A An example architecture of an encoder utilizing in-hybrid loop reshaping according to an embodiment of the present invention is described;

[0018] Figure 2B An example architecture of a decoder utilizing in-hybrid loop reshaping according to an embodiment of the present invention is described;

[0019] Figure 2C An example architecture for utilizing reshaped intra-frame CU decoding according to an embodiment is described;

[0020] Figure 2D An example architecture for utilizing reshaped inter-frame CU decoding according to an embodiment is described;

[0021] Figure 2E An example architecture for intra-frame CU decoding within an inter-frame coded slice is described according to an embodiment for luma or chroma processing;

[0022] Figure 2F An example architecture for intra-frame CU decoding within an inter-frame coded slice is described according to an embodiment for chroma processing;

[0023] Figure 3AAn example process for encoding video using a reshaping architecture according to an embodiment of the present invention is described;

[0024] Figure 3B An example process for decoding video using a reshaping architecture according to an embodiment of the present invention is described;

[0025] Figure 4 An example process for reassigning codewords in a reshaping domain according to an embodiment of the present invention is described;

[0026] Figure 5 An example process for deriving a reshaping threshold according to an embodiment of the present invention is described;

[0027] Figure 6A , 6B 6C and 6D depiction based on Figure 5 The process described herein and the example data diagrams of embodiments of the present invention for deriving the reshaping threshold; and

[0028] Figure 6E An example of codeword allocation based on group variance according to an embodiment of the present invention is described. Detailed Implementation

[0029] This document describes signal reshaping and coding techniques for compressing images using rate-distortion optimization (RDO). In the following description, numerous specific details are set forth for illustrative purposes to provide a thorough understanding of the invention. However, it should be understood that the invention can be practiced without these specific details. In other instances, well-known structures and apparatuses have not been described exhaustively so as not to unnecessarily hinder, obscure, or confuse the invention.

[0030] Overview

[0031] The exemplary embodiments described herein relate to signal reshaping and video coding. In an encoder, a processor receives an input image with a first codeword representation to be reshaped into a second codeword representation (wherein the second codeword representation allows for more efficient compression than the first codeword representation), and generates a forward reshaping function that maps the pixels of the input image to the second codeword representation. To generate the forward reshaping function, the encoder: divides the input image into multiple pixel regions; assigns each of the pixel regions to one of multiple codeword groups based on a first luminance characteristic of each pixel region; calculates a grouping metric for each of the multiple codeword groups based on a second luminance characteristic of each of the pixel regions assigned to each codeword group; assigns a plurality of codewords in the second codeword representation to each codeword group based on the grouping metric and a rate-distortion optimization criterion; and generates the forward reshaping function in response to assigning codewords in the second codeword representation to each of the multiple codeword groups.

[0032] In another embodiment, in the decoder, the processor receives a coded bitstream syntax element characterized by a reshaping model, wherein the syntax element includes one or more of the following: a flag indicating the minimum codeword block index value for the reshaping construction process, a flag indicating the maximum codeword block index value for the reshaping construction process, a flag indicating the reshaping model distribution type (wherein the model distribution type is associated with a default block correlation parameter containing block importance values), or a flag indicating one or more differential block importance values ​​for adjusting the default block importance values ​​defined in the reshaping model distribution. The processor determines the default block importance value for each block and an allocation table of several default codewords assigned to each block based on the reshaping model distribution and the block importance values. Then, for each codeword block, the processor:

[0033] Its group importance value is determined by adding its default group importance value to its difference group importance value;

[0034] The number of codewords allocated to the codeword group is determined based on the group importance value and the allocation table; and

[0035] The positive reshaping function is generated based on the number of codewords assigned to each codeword group.

[0036] In another embodiment, in the decoder, a processor receives an encoded bitstream including one or more codeword-reshaped images represented by a first codeword and metadata associated with reshaping information of the encoded reshaped images. The processor generates an inverse reshaping function and a forward reshaping function based on the metadata associated with the reshaping information, wherein the inverse reshaping function maps pixels from the reshaped image represented by the first codeword to a second codeword representation, and the forward reshaping function maps pixels from the image represented by the second codeword to the first codeword representation. The processor extracts an encoded reshaping image comprising one or more coding units from the encoded bitstream, wherein for one or more coding units in the encoded reshaping image:

[0037] For the reshaping intra-frame coding unit (CU) in the coded reshaping image, the processor:

[0038] The first reshaping reconstruction sample of the CU is generated based on the reshaping residual in the CU and the first reshaping prediction sample.

[0039] The reshaping loop filter output is generated based on the first reshaping reconstruction sample and the loop filter parameters.

[0040] The inverse reshaping function is applied to the output of the reshaping loop filter to generate a decoded sample of the coding unit represented by the second codeword; and

[0041] The decoded sample of the encoding unit represented by the second codeword is stored in the reference buffer;

[0042] For the reshaping inter-frame coding unit in the reshaping image, the processor:

[0043] The positive reshaping function is applied to the prediction sample stored in the reference buffer in the second codeword representation to generate a second reshaping prediction sample;

[0044] The second reshaping reconstruction sample of the coding unit is generated based on the reshaping residual in the coding CU and the second reshaping prediction sample.

[0045] The reshaping loop filter output is generated based on the second reshaping reconstruction sample and the loop filter parameters.

[0046] The inverse reshaping function is applied to the output of the reshaping loop filter to generate samples of the coding unit represented by the second codeword; and

[0047] The sample of the encoding unit, represented by the second codeword, is stored in a reference buffer. Finally, the processor generates a decoded image based on the stored samples in the reference buffer.

[0048] In another embodiment, in the decoder, the processor receives an encoded bitstream including one or more encoded reshaped images represented as input codewords and reshaped metadata 207 of the one or more encoded reshaped images in the encoded bitstream. The processor generates a forward reshaped function 282 based on the reshaped metadata, wherein the forward reshaped function maps pixels from the image represented by the first codeword to the input codeword. The processor generates an inverse reshaped function 265-3 based on the reshaped metadata or the forward reshaped function, wherein the inverse reshaped function maps pixels from the reshaped image represented by the input codeword to the first codeword. The processor extracts an encoded reshaped image comprising one or more coding units from the encoded bitstream, wherein:

[0049] For the intra-frame coding units (intra-frame CUs) in the encoded reshaped image, the processor:

[0050] The reshaping reconstruction sample 285 of the intra-frame CU is generated based on the reshaping residual in the intra-frame CU and the intra-frame predicted reshaping prediction sample.

[0051] The inverse reshaping function 265-3 is applied to the reshaping reconstruction sample of the intra-frame CU to generate a decoded sample of the intra-frame CU represented by the first codeword;

[0052] The loop filter 270 is applied to the decoded sample of the intra-frame CU to generate the output sample of the intra-frame CU; and

[0053] The output sample of the intra-frame CU is stored in the reference buffer;

[0054] For the inter-frame coded CU (inter-frame CU) in the encoded reshaped image, the processor:

[0055] The positive reshaping function 282 is applied to the inter-frame prediction samples stored in the reference buffer in the first codeword representation to generate the reshaping prediction samples of the inter-frame CU represented by the input codeword;

[0056] The reshaping reconstruction sample of the inter-frame CU is generated based on the reshaping residual in the inter-frame CU and the reshaping prediction sample of the inter-frame CU.

[0057] The inverse reshaping function 265-3 is applied to the reshaping reconstruction sample of the inter-frame CU to generate a decoded sample of the inter-frame CU represented by the first codeword;

[0058] The loop filter 270 is applied to the decoded samples of the inter-frame CU to generate the output samples of the inter-frame CU; and

[0059] The output samples of the inter-frame CU are stored in the reference buffer; and

[0060] A decoded image represented by the first codeword is generated based on the output samples in the reference buffer.

[0061] Example video delivery processing pipeline

[0062] Figure 1A This describes an example process of a conventional video delivery pipeline 100, illustrating the various stages from video capture to video content display. A sequence of video frames 102 is captured or generated using image generation block 105. The video frames 102 may be digitally captured (e.g., via a digital camera) or generated by a computer (e.g., using computer animation) to provide video data 107. Alternatively, the video frames 102 may be captured onto film using a cinema camera. The film is converted to a digital format to provide video data 107. In production stage 110, the video data 107 is edited to provide a video production stream 112.

[0063] Next, the video data from video production stream 112 is provided to the processor in post-production editing block 115. Post-production editing block 115 may include adjusting or modifying the color or brightness of specific areas of the image to improve image quality or to achieve a specific look for the image according to the creative intent of the video creator. This is sometimes referred to as "color timing" or "color grading." Other edits (such as scene selection and sequencing, image cropping, adding computer-generated visual effects, etc.) can be performed in block 115 to produce a final version 117 of the production for release. During post-production editing 115, the video image is viewed on reference monitor 125.

[0064] After post-production 115, the video data from final production 117 can be transmitted to encoding block 120 for downstream transmission to decoding and playback devices (e.g., televisions, video boxes, cinemas, and the like). In some embodiments, encoding block 120 may include audio and video encoders (e.g., audio and video encoders defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats) to generate encoded bitstream 122. In a receiver, decoding unit 130 decodes encoded bitstream 122 to generate a decoded signal 132 representing the same or closely approximate signal 117. The receiver may be attached to a target display 140 that may have characteristics completely different from those of the reference display 125. In this case, display management signal 135 may be used to map the dynamic range of decoded signal 132 to the characteristics of the target display 140 by generating display mapping signal 137.

[0065] Signal reshaping

[0066] Figure 1B An example process of signal reshaping according to the prior art [2] is described. Given an input frame 117, a forward reshaping block 150 analyzes the input and encoding constraints and generates a codeword mapping function that maps the input frame 117 to a requantized output frame 152. For example, the input 117 may be encoded according to a specific electro-optic transfer function (EOTF) (e.g., gamma). In some embodiments, metadata may be used to transmit information about the reshaping process to downstream devices (e.g., decoders). As used herein, the term “metadata” refers to any auxiliary information transmitted as part of the encoded bitstream and that helps the decoder reproduce the decoded image. This metadata may include (but is not limited to) color space or gamut information, reference display parameters, and auxiliary signal parameters as described herein.

[0067] After encoding 120 and decoding 130, the decoded frame 132 can be processed by an inverse (or inverse) reshaping function 160, which converts the requantized frame 132 back to the original EOTF domain (e.g., gamma) for further downstream processing (e.g., the display management signal 135 discussed earlier). In some embodiments, the inverse reshaping function 160 can be integrated with the dequantizer in the decoder 130, for example, as part of the dequantizer in an AVC or HEVC video decoder.

[0068] As used herein, the term “reshaper” can refer to a forward or inverse reshaping function used when encoding and / or decoding digital images. Examples of reshaping functions are discussed in Reference [2]. In Reference [2], an in-loop block-based image reshaping method for high dynamic range video coding is proposed. The design allows block-based reshaping within the coding loop, but at the cost of increased complexity. Specifically, the design requires maintaining two sets of decoded image buffers: one set for inverse reshaping (or non-reshaping) decoded images, which can be used for both prediction without reshaping and output to the display; and another set for forward reshaping decoded images, which can only be used for prediction with reshaping. Although forward reshaping decoded images can be computed in real time, the complexity is very high, especially for inter-frame prediction (using motion compensation with subpixel interpolation). Generally speaking, display-image-buffer (DPB) management is complex and requires very careful attention, so inventors should understand that a simplified method for encoding video is desirable.

[0069] Reference [6] presents additional codec architectures based on reshaping, including an external out-of-loop reshaper, an architecture with an in-loop-only intra-frame reshaper, an architecture with an in-loop reshaper for predicting residuals, and a hybrid architecture combining both intra-frame in-loop reshaping and inter-frame residual reshaping. The primary goal of the proposed reshaping architectures is to improve subjective visual quality. Consequently, many of these methods will produce poor objective metrics, specifically the well-known peak signal-to-noise ratio (PSNR) metric.

[0070] In this invention, a novel reshaping mechanism is proposed based on Rate-Distortion Optimization (RDO). Specifically, when the target distortion metric is MSE (Mean Squared Error), the proposed reshaping mechanism improves both subjective visual quality and the utilized objective metric based on PSNR, Bjontegaard PSNR (BD-PSNR), or Bjontegaard Rate (BD-Rate). It should be noted that any of the proposed reshaping architectures can be applied, without loss of generality, to one or more of the luminance and chrominance components, or a combination of the luminance and chrominance components.

[0071] Reshaping based on rate distortion optimization

[0072] Considering a reshaped video signal represented by a bit depth of B bits in the color components (e.g., B = 10 for Y, Cb, and / or Cr), there exists a total of 2 B One usable codeword. Consider limiting the desired codeword range to [0, 2]. B Divide into N segments or groups, and make M k Let represent the number of codewords in the k-th segment or group after reshaping, such that, given the target bit rate R, the distortion D between the source image and the decoded or reconstructed image is minimized. Without loss of generality, D can be expressed as a measure of the sum of variances (SSE) between the corresponding pixel values ​​of the source input (Source(i,j)) and the corresponding pixel values ​​of the reconstructed image (Recon(i,j)):

[0073] D = SSE = ∑ i，j Diff(i,j) 2 (1)

[0074] in

[0075] Diff(i,j)=Source(i,j)-Recon(i,j).

[0076] The optimization reshaping problem can be rewritten as: find M k (k = 0, 1, ..., N-1) such that, given the bit rate R, D is minimized, where

[0077] Various optimization methods can be used to find solutions, but for real-time coding, the optimal solution can be very complex. This invention proposes a suboptimal but more practical analytical solution.

[0078] Without loss of generality, consider an input signal represented by a bit depth of B bits (e.g., B = 10), where the codeword is uniformly divided into N groups (e.g., N = 32). By default, each group is assigned to M. a =2 B / N codewords (for example, for N=32 and B=10, M a =32). Next, we will demonstrate a more efficient codeword allocation based on RDO through examples.

[0079] As used herein, the term "narrow range" [CW1, CW2] refers to the range of consecutive codewords between codewords CW1 and CW2, which is the full dynamic range [0, 2]. B A subset of [-1]. For example, in an embodiment, the narrow range may be defined as [16*2]. (B-8) 235*2 (B -8)(For example, for B = 10, the narrow range includes values ​​[64 940]). Assume the bit depth of the output signal is B. o If the dynamic range of the input signal is narrow, then in the content to be represented as "default" reshaping, we can extend the signal to the full range [0 2]. Bo -1]. Then, each group will have approximately M [units / groups]. f =CEIL((2 Bo / (CW2-CW1))*M a ) codewords, or for this example, if B o =B=10, then M f =CEIL((1024 / (940-64))*32) = 38 codewords, where CEIL(x) represents the top function, which maps x to the smallest integer greater than or equal to x. Without loss of generality, in the following examples, for simplicity, assume B... o =B.

[0080] For the same quantization parameter (QP), increasing the number of codewords in a block is equivalent to allocating more bits to encode the signal within the block, thus effectively reducing SSE or increasing PSNR. However, a uniform increase in codewords per block does not yield better results than no reshaping because the PSNR gain cannot outpace the increase in bit rate; that is, this is not a good trade-off for RDO. Ideally, we want to assign more codewords only to blocks that produce the optimal trade-off for RDO (i.e., a significant reduction in SSE (and an increase in PSNR) at the cost of a small increase in bit rate).

[0081] In this embodiment, RDO performance is improved through adaptive segmented reshaping mapping. The method can be applied to any type of signal, including both standard dynamic range (SDR) and high dynamic range (HDR) signals. Using the previous simple case as an example, the object of this invention is to improve M... a Or M f Assign it to each codeword segment or codeword group.

[0082] In the encoder, given the N codeword groups of the input signal, the average luminance variance of each group can be approximated as follows:

[0083] - The block variance (var) of each group bin (k)) and count value (c) bin The sum of (k) is initialized to zero, for example, for

[0084] For k = 0, 1, ..., N-1, var bin (k)=0 and c bin (k) = 0;

[0085] - Divide the image into L*L non-overlapping blocks (e.g., L=16);

[0086] - For each image block, calculate the average brightness of the block and the brightness variance of block i (e.g., Luma_mean(i) and Luma_var(i));

[0087] - Based on the average brightness of the block, the block is assigned to one of N groups. In an embodiment, if Luma_mean(i) is within the k-th segment of the input dynamic range, then the total group brightness variance of the k-th group is increased by the brightness variance of the newly assigned block, and the count value of the group is increased by one. That is, if the i-th pixel region belongs to the k-th group, then:

[0088] var bin (k)=var bin (k)+Luma_var(i); (2)

[0089] c bin (k)=c bin (k)+1.

[0090] - For each group, the average luminance variance of the group is calculated by dividing the sum of the block variances in the group by the count value, assuming the count value is not equal to 0; or if c bin If (k) is not 0, then:

[0091] var bin (k)=var bin (k) / c bin (k) (3)

[0092] Those skilled in the art will understand that we can use alternative metrics to characterize sub-blocks instead of luminance variance. For example, we can use the standard deviation of luminance values, weighted luminance variance, or luminance values, peak luminance, and the like.

[0093] In an embodiment, the following pseudocode describes an instance of how the encoder can use the computed metric of each group to adjust the group assignment.

[0094]

[0095] Among them TH U This indicates the predetermined upper threshold.

[0096] In another embodiment, the allocation can be performed as follows:

[0097]

[0098] TH0 and TH1 represent the predetermined lower threshold and upper threshold, respectively.

[0099] In another embodiment,

[0100]

[0101] Among them TH L This indicates the predetermined lower threshold.

[0102] The above example demonstrates how to select from two pre-selected numbers M f and M a Choose the number of codewords for each group. The threshold (e.g., TH) can be determined (for example, through exhaustive search or based on optimization rate distortion). U or TH L The threshold can also be adjusted based on the quantization parameter value (QP). In this example, for B=10, the threshold can be in the range of 1,000 to 10,000.

[0103] In this embodiment, to accelerate the processing, the following can be utilized: Lagrange The optimization method uses a set of fixed values ​​(i.e., {2,000, 3,000, 4,000, 5,000, 6,000, 7,000}) to determine the threshold. For example, for each TH(i) value in the set, using a predefined training cut, we can run a compression test with a fixed QP and compute the value of the objective function J defined as follows.

[0104] J(i)=D+λR。 (7)

[0105] Next, the optimal threshold can be defined as the TH(i) value in the group whose J(i) is the smallest.

[0106] In more general instances, we can predefine the lookup table (LUT). For example, in Table 1, the first row might delimit the grouping of measures (e.g., var). bin The LUT is a set of thresholds that divide the full range of (k) values ​​into several segments, and the second row defines the corresponding number of codewords (CWs) assigned in each segment. In the embodiment, one rule for constructing this LUT is: if the grouping variance is too large, then we need to use a large number of bits to reduce the SSE, therefore, we can assign less than M. a The codeword (CW) value. If the group variance is very small, then we can assign a value greater than M. a The CW value.

[0107] Table 1: Example LUTs for codeword allocation based on group variance threshold

[0108] <![CDATA[CW0]]> … <![CDATA[CW p-1 ]]> <![CDATA[CW p ]]> <![CDATA[CW p+1 ]]> … <![CDATA[CW q-1 ]]> <![CDATA[CW q ]]>

[0109] Using Table 1, the mapping from threshold to codeword can be obtained as follows:

[0110]

[0111] For example, given two thresholds and three codeword allocations, in the embodiment, for B=10, TH0=3,000, CW0=38, TH1=10,000, CW1=32 and CW2=28.

[0112] In another embodiment, the two thresholds TH0 and TH1 can be selected as follows: a) Considering TH1 to be a very large number (even infinite) and (for example) using RDO optimization in Equation (7) to select TH0 from a set of predetermined values. Given TH0, a second set of possible values ​​for TH1 is now defined (e.g., the set {10,000,15,000,20,000,25,000,30,000}), and Equation (7) is applied to identify the optimal value. The method can be repeatedly executed using a finite number of thresholds or until it converges.

[0113] We should note that after assigning codewords to blocks according to any of the previously defined schemes, M k The sum of the values ​​can exceed the maximum value of the available codewords (2). B Alternatively, there may be unused codewords. If unused codewords exist, we can simply decide to do nothing or assign them to a specific group. On the other hand, if the algorithm assigns more codewords than are available, we would like to (e.g.) readjust M by renormalizing the CW value. k Value. Alternatively, we can utilize existing M. k The value is used to generate the positive reshaping function, but then by using (∑ k M k ) / 2 B The output value of the reshaping function is then adjusted based on the conversion. Reference [7] also describes an example of codeword redistribution techniques.

[0114] Figure 4 This describes an instance procedure for allocating codewords to the reshaping domain according to the previously described RDO technique. In step 405, the desired reshaping dynamic range is divided into N groups. After dividing the input image into several non-overlapping blocks (step 410), for each block:

[0115] Step 415 calculates its luminance characteristics (e.g., mean and variance).

[0116] Step 420 assigns each image patch to one of the N groups.

[0117] Step 425: Calculate the average luminance variance for each group.

[0118] Given the values ​​calculated in step 425, in step 430, several codewords are assigned to each group according to one or more thresholds using any of the codeword allocation algorithms described in equations (4) to (8), for example. Finally, in step (435), the final codeword allocation can be used to generate a positive reshaping function and / or an inverse reshaping function.

[0119] In this embodiment, as an example (but not limited to), the following C code can be used to construct a forward LUT (FLUT):

[0120]

[0121] In this embodiment, the anti-LUT can be constructed as follows:

[0122]

[0123] According to the syntax, we can reuse the syntax proposed in previous applications, such as the piecewise polynomial pattern or parametric model in references [5] and [6]. Table 2 shows this example when N=32 for equation (4).

[0124] Table 2: Grammar for Reshaping Using the First Parameter Model

[0125]

[0126]

[0127] in:

[0128] `reshaper_model_profile_type` specifies the distribution type used in the reshaper construction process. A given distribution can provide information about the default values ​​used, such as the number of groups, the default group importance or priority, and the default codeword assignment (e.g., M). a and / or M f value).

[0129] `reshaper_model_scale_idx` specifies the exponent value of the scaling factor (denoted as ScaleFactor) used in the reshaper construction process. The value of ScaleFactor allows for improved control of the reshaping function to enhance overall coding efficiency.

[0130] `reshaper_model_min_bin_idx` specifies the minimum grouping index used in the reshaper construction process. The value of `reshaper_model_min_bin_idx` should be in the range of 0 to 31 (inclusive).

[0131] `reshaper_model_max_bin_idx` specifies the maximum grouping index used during the reshaper construction process. The value of `reshaper_model_max_bin_idx` should be in the range of 0 to 31 (inclusive).

[0132] `reshaper_model_bin_profile_delta[i]` specifies the difference value used to adjust the distribution of the i-th group during the reshaper construction process. The value of `reshaper_model_bin_profile_delta[i]` should be in the range of 0 to 1 (inclusive).

[0133] Table 3 depicts another embodiment that utilizes an alternative, more efficient syntax representation.

[0134] Table 3: Grammar Reshaping Using the Second-Parameter Model

[0135]

[0136]

[0137] in,

[0138] The reshaper_model_delta_max_bin_idx is set to equal the maximum allowed grouping exponent (e.g., 31) minus the maximum grouping exponent used in the reshaper construction process.

[0139] `reshaper_model_num_cw_minus1+1` specifies the number of codewords to be transmitted.

[0140] reshaper_model_delta_abs_CW[i] specifies the i-th absolute differential codeword value.

[0141] reshaper_model_delta_sign_CW[i] specifies the sign of the i-th differential codeword.

[0142] then:

[0143] reshaper_model_delta_CW[i]=(1-2*reshaper_model_delta_sign_CW[i])*reshaper_model_delta_abs_CW[i];

[0144] reshaper_model_CW[i]=32+reshaper_model_delta_CW[i].

[0145] `reshaper_model_bin_profile_delta[i]` specifies the difference value used to adjust the distribution of the i-th group during the reshaper construction process. When `reshaper_model_num_cw_minus1` equals 0, the value of `reshaper_model_bin_profile_delta[i]` should be in the range of 0 to 1. When `reshaper_model_num_cw_minus1` equals 1, the value of `reshaper_model_bin_profile_delta[i]` should be in the range of 0 to 2.

[0146] When reshaper_model_bin_profile_delta[i] is set to 0, CW = 32; when reshaper_model_bin_profile_delta[i] is set to 1, CW = reshaper_model_CW[0]; when reshaper_model_bin_profile_delta[i] is set to 2, CW = reshaper_model_CW[1]. In the embodiment, reshaper_model_num_cw_minus1 is allowed to be greater than 1 to allow ue(v) to be used to signal to reshaper_model_num_cw_minus1 and reshaper_model_bin_profile_delta[i] for more efficient encoding.

[0147] In another embodiment, as described in Table 4, the number of codewords in each block can be explicitly defined.

[0148] Table 4: Grammar Reshaping Using the Third Model

[0149]

[0150]

[0151] `reshaper_model_number_bins_minus1+1` specifies the number of groups used for the luminance component. In some embodiments, the number of groups being a power of 2 may be more efficient. The total number of groups can then be represented by its log2 representation, for example using an alternative parameter such as `log2_reshaper_model_number_bins_minus1`. For example, for 32 groups, `log2_reshaper_model_number_bins_minus1` = 4. `reshaper_model_bin_delta_abs_cw_prec_minus1+1` specifies the number of bits used for the representation of the syntax `reshaper_model_bin_delta_abs_CW[i]`.

[0152] reshaper_model_bin_delta_abs_CW[i] specifies the absolute differential codeword value for the i-th group.

[0153] reshaper_model_bin_delta_sign_CW_flag[i] specifies the sign of reshaper_model_bin_delta_abs_CW[i] as follows:

[0154] - If reshaper_model_bin_delta_sign_CW_flag[i] equals 0, then the corresponding variable RspDeltaCW[i] has a positive value.

[0155] Otherwise (reshaper_model_bin_delta_sign_CW_flag[i] is not equal to 0), the corresponding variable RspDeltaCW[i] has a negative value.

[0156] When reshaper_model_bin_delta_sign_CW_flag[i] does not exist, it is inferred to be equal to 0.

[0157] Variable RspDeltaCW[i]=(1-2*reshaper_model_bin_delta_sign_CW[i])*

[0158] reshaper_model_bin_delta_abs_CW[i];

[0159] The variable OrgCW is set to equal to (1 < <BitDepthY) / (reshaper_model_number_bins_minus1+1);

[0160] The variable RspCW[i] is derived as follows:

[0161] If reshaper_model_min_bin_idx<=i<=reshaper_model_max_bin_idx

[0162] Then RspCW[i] = OrgCW + RspDeltaCW[i].

[0163] Otherwise, RspCW[i] = 0.

[0164] In an embodiment, assuming that the codeword allocation is based on one of the earlier instances (e.g., equation (4)), instances of how to define the parameters in Table 2 include:

[0165] First, let's assume we assign "group importance" as follows:

[0166]

[0167] As used herein, the term "group importance" refers to a value assigned to each of N codeword groups to indicate the importance of all codewords in that group relative to the other groups during the reshaping process.

[0168] In this embodiment, we can set `default_bin_importance` from `reshaper_model_min_bin_idx` to `reshaper_model_max_bin_idx` to 1. We set the value of `reshaper_model_min_bin_idx` to the minimum grouping exponent, which makes M... k Not equal to 0. Set the value of reshaper_model_max_bin_idx to the maximum grouping exponent, which makes M k Not equal to 0. The reshaper_model_bin_profile_delta for each group within [reshaper_model_min_bin_idx reshaper_model_max_bin_idx] is the difference between bin_importance and default_bin_importance.

[0169] The following are examples of how to construct forward reshaping LUTs (FLUTs) and inverse reshaping LUTs (ILUTs) using the proposed parametric model:

[0170] 1) Divide the brightness range into N groups (e.g., N=32);

[0171] 2) Derive the group importance index for each group from the syntax. For example:

[0172]

[0173] 3) Automatically pre-assign codewords based on group importance:

[0174]

[0175] 4) Construct a forward reshaping LUT based on the codeword assignments for each group by accumulating the codewords assigned to each group. The sum should be less than or equal to the total codeword budget (e.g., 1024 for a 10-bit full range). (See the earliest C code for example).

[0176] 5) Construct the inverse reshaping LUT (see the earliest C code for example).

[0177] From a grammatical point of view, alternative methods can also be applied. The key is to explicitly or implicitly specify the number of codewords in each group (e.g., M). k Where k = 0, 1, 2, ..., N-1). In one embodiment, we can explicitly specify the number of codewords in each block. In another embodiment, we can specify codewords differentially. For example, the number of codewords in a block can be determined using the difference in the number of codewords between the current block and the previous block (e.g., M_Delta(k) = M(k) - M(k-1)). In another embodiment, we can specify the most frequently used number of codewords (i.e., M... M Furthermore, the number of codewords in each group is expressed as the difference between the number of codewords in each group and this number (e.g., M_Delta(k) = M(k) - M). M ).

[0178] In this embodiment, two reshaping methods are supported. One method is referred to as the "default reshaper," where M... f Assigned to all groups. The second method, referred to as “adaptive reshaper”, applies the adaptive reshaper described earlier. The two methods can be signaled to the decoder as described in reference [6] using special flags of the adaptive reshaper (e.g., sps_reshaper_adaptive_flag) (e.g., using sps_reshaper_adaptive_flag=0 and using sps_reshaper_adaptive_flag=1 with the default reshaper).

[0179] This invention can be applied to any reshaping architecture proposed in reference [6], such as an external reshaper, an in-loop-only in-frame reshaper, an in-loop residual reshaper, or an in-loop hybrid reshaper. As an example, Figure 2A and 2BAn example architecture for in-hybrid loop reshaping according to an embodiment of the present invention is depicted. Figure 2A In this architecture, elements are combined from both an intra-loop intra-frame reshaping architecture (top of the figure) and an intra-loop residual architecture (bottom of the figure). Under this architecture, reshaping is applied to image pixels for intra-frame slices and to prediction residuals for inter-frame slices. In encoder 200_E, two new blocks are added to the block-based conventional encoder (e.g., HEVC): one for estimating the forward reshaping function (e.g., based on...). Figure 4 The block 205, forward image reshaping block 210-1, and forward residual reshaping block 210-2 of the input video 117 apply forward reshaping to one or more of the color components or prediction residuals. In some embodiments, these two operations may be performed as part of a single image reshaping block. Parameter 207, which is associated with determining the inverse reshaping function in the decoder, may be passed to the lossless encoder block (e.g., CABAC 220) of the video encoder so that it can be embedded into the encoded bitstream 122. In intra-frame mode, intra-frame prediction 225-1, transform and quantization (T and Q), and inverse transform and inverse quantization (Q) are performed. -1 and T -1 All images are reshaped. In both modes, the stored image in the DPB 215 is always in the inverse reshaping mode, which requires an inverse image reshaping block (e.g., 265-1) or an inverse residual reshaping block (e.g., 265-2) before the loop filters 270-1 and 270-2. Figure 2A As depicted, the intra / inter-frame slice switching allows switching between two architectures depending on the type of slice to be encoded. In another embodiment, in-loop filtering for intra-frame slices can be performed prior to de-reshaping.

[0180] In decoder 200_D, the following new regular blocks are added to the block-based conventional decoder: block 250 (reshaper decoding) which reconstructs the forward and inverse reshape functions based on the encoded reshape function parameter 207; block 265-1 which applies the inverse reshape function to the decoded data; and block 265-2 which applies both the forward and inverse reshape functions to generate the decoded video signal 162. For example, in 265-2, the reconstructed value is given by Rec = ILUT(FLUT(Pred) + Res), where FLUT represents the forward reshape LUT and ILUT represents the inverse reshape LUT.

[0181] In some embodiments, the operations associated with blocks 250 and 265 may be combined into a single processing block. For example... Figure 2B As described, the intra / inter-frame slice switch allows switching between two modes depending on the slice type in the encoded video image.

[0182] Figure 3AAn example process 300_E is described in which video is encoded using a reshaping architecture (e.g., 200_E) according to an embodiment of the present invention. If reshaping is not enabled (path 305), then encoding 335 is performed as is known in prior art encoders (e.g., HEVC). If reshaping is enabled (path 310), then the encoder may have the option to apply a predetermined (default) reshaping function 315 or to adaptively determine a new reshaping function 325 based on image analysis 320 (e.g., ...). Figure 4 (As described in the document). After encoding image 330 using the reshaping architecture, the remainder of the encoding follows the same steps as the conventional encoding pipeline 335. If adaptive reshaping 312 is used, then metadata associated with the reshaping function is generated as part of the "encoded reshaper" step 327.

[0183] Figure 3B An example process 300_D of decoding video using a reshaping architecture (e.g., 200_D) according to an embodiment of the present invention is described. If reshaping is not enabled (path 340), then after decoding image 350, an output frame 390 is generated as in a conventional decoding pipeline. If reshaping is enabled (path 360), then the decoder determines to apply a predetermined (default) reshaping function 375 or adaptively determines a reshaping function 380 based on received parameters (e.g., 207). After decoding using the reshaping architecture 385, the remainder of the decoding follows a conventional decoding pipeline.

[0184] As described in reference [6] and earlier in this specification, the forward reshaping LUT FwdLUT can be constructed as a single unit, while the inverse reshaping LUT can be constructed using the forward reshaping LUT (FwdLUT) based on a reverse mapping. In an embodiment, the forward LUT can be constructed using piecewise linear interpolation. In the decoder, inverse reshaping can be accomplished either directly using the inverse LUT or also using linear interpolation. The piecewise linear LUT is constructed based on the input pivot point and the output pivot point.

[0185] Let (X1, Y1) and (X2, Y2) be the two input pivot points and their corresponding output values ​​for each group. Any input value X between X1 and X2 can be interpolated by the following equation:

[0186] Y=((Y2-Y1) / (X2-X1))*(X-X1)+Y1.

[0187] In the fixed-point implementation plan, the above equation can be rewritten as follows:

[0188] Y = ((m*X+2) FP_PREC-1 )>>FP_PREC)+c,

[0189] Where m and c represent the scalar and offset of linear interpolation, and FP_PREC is a constant related to fixed-point accuracy.

[0190] As an example, FwdLUT can be constructed as follows: making variables...

[0191] lutSize=(1<<BitDepthY).

[0192] Make variables

[0193] binNum=reshaper_model_number_bins_minus1+1,

[0194] and

[0195] binLen = lutSize / binNum.

[0196] For the i-th group, its two interval pivots (e.g., X1 and X2) can be derived as X1 = i * binLen and X2 = (i+1) * binLen. Next:

[0197]

[0198] FP_PREC delimits the fixed-point precision of the fractional part of the variable (e.g., FP_PREC = 14). In embodiments, binsLUT[] can be computed with a higher precision than FwdLUT. For example, binsLUT[] values ​​can be computed as 32-bit integers, while FwdLUT can be binsLUT values ​​truncated to 16 bits.

[0199] Adaptive threshold derivative

[0200] As previously described, during reshaping, one or more thresholds (e.g., TH, TH) can be utilized. U TH L (and similar methods) to adjust codeword allocation. In an embodiment, such thresholds may be adaptively generated based on content characteristics. Figure 5 An example process for deriving such a threshold according to an embodiment is described.

[0201] 1) In step 505, the brightness range of the input image is divided into N groups (e.g., N = 32). For example, N can also be represented as PIC_ANALYZE_CW_BINS.

[0202] 2) In step 510, we perform image analysis to calculate the brightness characteristics of each group. For example, we can calculate the percentage of pixels in each group (denoted as BinHist[b], b = 1, 2, ..., N), where

[0203] BinHist[b] = 100 * (total pixels in the group b) / (total pixels in the image), (10)

[0205] As previously discussed, another good measure of image characteristics is the mean variance (or standard deviation) of pixels in each group (denoted as BinVar[b]). In "block mode", BinVar[b] can be calculated as var in the steps described in the paragraphs leading to equations (2) and (3). bin (k). Alternatively, block-based computation can be modified using pixel-based computation. For example, it can be denoted as vf(i) (the variance associated with the group of pixels surrounding the i-th pixel in an m×m adjacent window (e.g., m=5), where the i-th pixel is located at the center of the window). For example, if

[0206]

[0207] W represents the area surrounding the i-th pixel with value x(i). N =The average value of the pixels in an m*m window (e.g., m=5), then

[0208]

[0209] Using a nonlinear mapping (e.g., vf(i) = log10(vf(i) + 1)) can suppress the dynamic range of the original variance values. Then, a variance factor can be used to calculate the average variance in each group.

[0210]

[0211] Where K b This indicates the number of pixels in group b.

[0212] 3) In step 515, the average group variance (and its corresponding exponent) (e.g., but not limited to) is sorted in descending order. For example, the sorted BinVar values ​​can be stored in BinVarSortDsd[b] and the sorted group exponents can be stored in BinIdxSortDsd[b]. As an example, using C code, the process can be described as follows:

[0213]

[0214]

[0215] Figure 6A The example plot depicts the sorted average grouped variance factor.

[0216] 4) In view of the grouped histogram values calculated in step 510, in step 520, we calculate and store a cumulative density function (CDF) according to the order of the sorted average group variances. For example, if the CDF is stored in the array BinVarSortDsdCDF[b], then in the embodiment:

[0217]

[0218] Figure 6B depicts an example of the calculated CDF based on Figure 6A data, as shown in Figure 605. The pair of CDF value and sorted average group variance ({x=BinVarSortDsd[b], y=BinVarSortDsdCDF[b]}) can be interpreted as: "there are y% of pixels in the image with variance greater than or equal to x" or "there are (100-y)% of pixels in the image with variance less than x".

[0219] 5) Finally, in step 525, in view of the CDF BinVarSortDsdCDF[BinVarSortDsd[b]] that varies with the sorted average group variance values, we can define a threshold based on the group variance and the cumulative percentage.

[0220] Figure 6C and 6D respectively show examples for determining a single threshold or two thresholds. When only one threshold (e.g., TH) is used, as an example, TH may be defined as "the average variance where k% of pixels have vf≥TH". Then, TH can be calculated by finding the intersection point of the CDF Figure 605 at k% (e.g., 610) (e.g., the BinVarSortDsd[b] value where BinVarSortDsdCDF=k%); for example, as shown in Figure 6C , for k=50, TH=2.5. Then, we can assign M f codewords to the groups with BinVar[b]<TH and assign M a codewords to the groups with BinVar[b]≥TH. As a rule of thumb, it is preferable to assign a larger number of codewords to groups with smaller variances (e.g., for a 10-bit video signal with 32 groups, M f >32>M a ).

[0221] When two thresholds are used, Figure 6D depicts an example of selecting TH L and TH U . For example, without loss of generality, TH Lmay be defined as the variance where 80% of the pixels have vf≥TH L (then, in this example, TH L =2.3), and TH U may be defined as the variance where 10% of all pixels have vf≥TH U (then, in this example, TH U =3.5). Given these thresholds, we may assign M f codewords to groups with BinVar[b]<TH L and assign M a codewords to groups with BinVar[b]≥TH U . For groups with BinVar between TH L and TH U , we may utilize the original number of codewords per group (e.g., 32 codewords when B=10).

[0222] The above techniques may be readily extended to cases with more than two thresholds. Relationships may also be used to adjust the number of codewords (M f , M a , etc.). As a rule of thumb, in low-variance groups, we should assign more codewords to improve PSNR (and reduce MSE); for high-variance groups, we should assign fewer codewords to save bits.

[0223] In an embodiment, if a parameter set (e.g., TH L , TH U , M a , M f , and the like) is manually obtained for specific content, for example, by exhaustive manual parameter tuning, then this automatic method may be applied to design a decision tree to classify each content to automatically set the optimal manual parameters. For example, content categories include movies, television, SDR, HDR, cartoons, nature, action, and the like.

[0224] To reduce complexity, various schemes may be utilized to constrain in-loop reshaping. If in-loop reshaping is adopted in a video coding standard, these constraints should be specified to ensure decoder simplification. For example, in an embodiment, luma reshaping may be deactivated for specific block coding sizes. For example, when nTbW*nTbH<TH, we may deactivate the intra and inter reshaper modes in inter slices, where the variable nTbW specifies the transform block width and the variable nTbH specifies the transform block height. For example, with TH=64, reshaping for both intra and inter modes of inter-coded slices (or tiles) is deactivated for blocks with sizes 4×4, 4×8, and 8×4.

[0225] Similarly, in another embodiment, we can disable the luma-based chroma residual conversion in the intra-frame mode in the inter-frame coded slice (or tile), or enable the segmentation tree when there are separate luma and chroma.

[0226] Interaction with other encoding tools

[0227] Loop Filtering

[0228] In reference [6], it is desired that the loop filter can operate in either the original pixel domain or the reshaped pixel domain. In one embodiment, it is suggested that loop filtering be performed in the original pixel domain (after image reshaping). For example, in hybrid in-loop reshaping architectures 200_E and 200_D, for intra-frame images, we would need to apply anti-reshaping 265-1 before the loop filter 270-1.

[0229] Figure 2C and Figure 2D Alternative decoder architectures 200B_D and 200C_D are described, in which inverse reshaping 265 is performed after loop filtering 270 and just before the decoded data is stored in the decoded image buffer (DPB) 260. In the proposed embodiment, the inverse residual reshaping formula for inter-frame slices is modified compared to the architecture in 200_D, and inverse reshaping is performed after loop filtering 270 (e.g., via the InvLUT() function or a lookup table). In this way, inverse reshaping is performed on both intra-frame and inter-frame slices after loop filtering, and the reconstructed pixels of both intra-frame coded CUs and inter-frame coded CUs before loop filtering are in the reshaping domain. After inverse reshaping 265, all output samples stored in the reference DPB are in the original domain. This architecture allows for both slice-based adaptation and CTU-based adaptation for in-loop reshaping.

[0230] like Figure 2C and Figure 2D As depicted, in the embodiment, loop filtering 270 is performed on both intra-frame coded CUs and inter-frame coded CUs in the reshaping domain, and inverse image reshaping 265 occurs only once to present a unified and simpler architecture for both intra-frame coded CUs and inter-frame coded CUs.

[0231] To decode the intra-coded CU 200B_D, intra-prediction 225 is performed on adjacent pixels of the reshaped image. Based on the residual Res and the predicted sample PredSample, the reconstructed sample 227 is derived as follows:

[0232] RecSample=Res+PredSample. (14)

[0233] Given the reconstructed sample 227, loop filtering 270 and inverse image reshaping 265 are applied to derive the RecSampleInDPB sample stored in DPB 260, where

[0234] RecSampleInDPB=InvLUT(LPF(RecSample)))=

[0235] =InvLUT(LPF(Res+PredSample))), (15)

[0236] InvLUT() represents the inverse reshaping function or inverse reshaping lookup table, and LPF() represents the loop filtering operation.

[0237] In traditional coding, inter-frame / intra-frame mode decision-making is based on calculating the distortion function (dfunc()) between the original and predicted samples. Instances of such functions include Sum of Variances (SSE), Sum of Absolute Differences (SAD), and others. When reshaping is used, CU prediction and mode decision-making are performed on the reshaping domain at the encoder side (not shown). That is, for mode decision-making,

[0238] Destraightening=dfunc(FwdLUT(SrcSample)-RecSample), (16)

[0239] Where FwdLUT() represents the forward reshaping function (or LUT) and SrcSample represents the original image sample.

[0240] For the inter-frame coding CU, at the decoder side (e.g., 200C_D), inter-frame prediction is performed using a reference image in the non-reshaping domain of the DPB. Then, in reconstruction block 275, reconstructed pixel 267 is derived as:

[0241] RecSample=(Res+FwdLUT(PredSmple)). (17)

[0242] Given the reconstructed sample 267, loop filtering 270 and inverse image reshaping 265 are applied to derive the RecSampleInDPB sample stored in the DPB, where

[0243] RecSampleInDPB=InvLUT(LPF(RecSample)))=InvLUT(LPF(Res+FwdLUT(PredSample)))). (18)

[0244] At the encoder side (not shown), assuming all neighboring samples (PredSample) used for prediction are already in the reshaping domain, intra-frame prediction is performed in the reshaping domain as follows:

[0245] Res=FwdLUT(SrcSample)-PredSample, (19a)

[0246] In the non-reshaping domain (i.e., directly using the reference image from the DPB), inter-frame prediction (e.g., using motion compensation) is performed, namely:

[0247] PredSample=MC(RecSampleinDPB), (19b)

[0248] Where MC() represents the motion compensation function. For motion estimation and fast mode decision-making that do not produce residuals, we can use the following equation to calculate the distortion:

[0249] Straightness=dfunc(SrcSample-PredSample).

[0250] However, for full-mode decisions that produce residuals, mode decisions are performed in the reshaping domain. That is, for full-mode decisions,

[0251] Distortion = dfunc(FwdLUT(SrcSample)-RecSample). (20)

[0252] Block-level adaptation

[0253] As previously explained, the proposed in-loop reshaper allows reshaping to be tuned at the CU level to, for example, set the variable CU_reshaper to on or off as needed. Under the same architecture, for inter-frame coded CUs, when CU_reshaper = off, reconstructed pixels need to be in the reshaping domain, even if the CU_reshaper flag is set to off for this inter-frame coded CU.

[0254] RecSample=FwdLUT(Res+PredSample), (21)

[0255] This ensures that intra-frame prediction always has neighboring pixels in the reshaping domain. The DPB pixel can be derived as:

[0256] RecSampleInDPB=InvLUT(LPF(RecSample)))=

[0257] =InvLUT(LPF(FwdLUT(Res+PredSample))). (twenty two)

[0258] For intra-frame coding CU, two alternative methods are proposed based on the coding process:

[0259] 1) Encode all intra-frame coded CUs using CU_reshaper= enabled. In this case, no additional processing is required because all pixels are already in the reshaping domain.

[0260] 2) Some intra-frame coded CUs can be encoded using CU_reshaper=off. In this case, with CU_reshaper=off, when applying intra-frame prediction, we need to apply inverse reshaping to adjacent pixels so that intra-frame prediction is performed in the original domain and the final reconstructed pixels are in the reshaping domain, i.e.:

[0261] RecSample=FwdLUT(Res+InvLUT(PredSample)), (23)

[0262] then

[0263] RecSampleInDPB=InvLUT(LPF(RecSample)))=

[0264] =InvLUT(LPF(FwdLUT(Res+InvLUT(PredSample))))). (twenty four)

[0265] Generally, the proposed architecture can be used in various combinations, such as intra-loop intra-frame reshaping only, intra-loop reshaping only for predicting residuals, or a hybrid architecture combining intra-loop intra-frame reshaping and inter-frame residual reshaping. For example, to reduce latency in the hardware decoding pipeline, for inter-frame slice decoding, we can perform intra-frame prediction (i.e., decode the intra-frame CU in the inter-frame slice) before inverse reshaping. Figure 2E The example architecture 200D_D of this embodiment is depicted. In the reconstruction module 285, for the inter-frame CU (e.g., Mux enabled output from 280 and 282), from equation (17):

[0266] RecSample=(Res+FwdLUT(PredSample)).

[0267] Where FwdLUT(PredSample) represents the output of the inter-frame predictor 280 followed by the forward reshaping 282. Otherwise, for the intra-frame CU (e.g., Mux enabled from the output of 284), the output of the reconstruction module 285 is:

[0268] RecSample=(Res+IPredSample),

[0269] IPredSample represents the output of intra-frame prediction block 284. Inverse reshaping block 265-3 is generated.

[0270] Y CU=InvLUT[RecSample].

[0271] Applying intra-frame prediction to inter-frame slices in the reshaping domain can also be applied to other embodiments, which include Figure 2C (where anti-reshaping is performed after loop filtering) and Figure 2D The embodiments described herein. In all such embodiments, particular attention needs to be paid to combining inter-frame / intra-frame prediction modes (i.e., when some samples are from inter-frame coded blocks and some samples are from intra-frame coded blocks during reconstruction), because inter-frame prediction is in the original domain, but intra-frame prediction is in the reshaping domain. When combining data from both inter-frame prediction coding units and intra-frame prediction coding units, prediction can be performed in either domain. For example, when combining inter-frame / intra-frame prediction modes is performed in the reshaping domain,

[0272] PredSampleCombined=PredSampeIntra+FwdLUT(PredSampleInter)

[0273] RecSample=Res+PredSampleCombined,

[0274] That is, the inter-frame coded samples in the original domain are reshaped before being added. Otherwise, when combining inter-frame / intra-frame prediction modes in the original domain, then:

[0275] PredSampleCombined=InvLUT(PredSampeIntra)+PredSampleInter

[0276] RecSample=Res+FwdLUT(PredSampleCombined),

[0277] That is, to reshape the intra-frame predicted samples back into the original domain.

[0278] Similar considerations can also be applied to the corresponding encoding embodiments, since the encoder (e.g., 200_E) contains a decoder loop that matches the corresponding decoder. As discussed earlier, equation (20) describes an embodiment in which mode decision-making is performed in the reshaping domain. In another embodiment, mode decision-making can be performed in the original domain, i.e.:

[0279] Distortion = dfunc(SrcSample-InvLUT(RecSample)).

[0280] For luminance-based chromaticity QP shift or chromaticity residual conversion, the average CU luminance value can always be calculated using the predicted value (rather than the reconstructed value) to minimize latency. Colorimetric QP deviation

[0281] As in reference [6], we can apply the same chromaDQP deviation process proposed to balance the luminance and chromaticity relationships caused by the reshaping curve. In the embodiment, we can derive segmented chromaDQP values ​​based on the codeword assignment of each group. For example:

[0282]

[0283] Encoder optimization

[0284] As described in reference [6], when lumaDQP is enabled, it is recommended to use pixel-based weighted distortion. When using reshaping, in the instance, the required weighting is adjusted based on the reshaping function (f(x)). For example:

[0285] W rsp =f′(x) 2 (26)

[0286] Where f′(x) represents the slope of the reshaping function f(x).

[0287] In another embodiment, we can derive the segment weighting directly based on the codeword assignment of each group. For example:

[0288] for the k-th bin,

[0289]

[0290] For the chroma components, the weighting can be set to 1 or a conversion factor sf. To reduce chroma distortion, sf can be set to a value greater than 1. To increase chroma distortion, sf can be set to a value greater than 1. In one embodiment, sf can be used to compensate for equation (25). Since chromaDQP can be set to only integers, we can use sf to adjust the fractional part of chromaDQP: therefore,

[0291] sf=2 ((chromaDQP-INT(chromaDQP)) / 3) .

[0292] In another embodiment, we can explicitly set the chromaQPOffset value or the image header in the image parameter set (PPS) to control chroma distortion.

[0293] The reshaper curve or mapping function does not need to be fixed for the entire video sequence. For example, it can be tuned based on the quantization parameter (QP) or the target bit rate. In one embodiment, we can utilize a more aggressive reshaper curve at lower bit rates and a less aggressive reshaping curve at relatively higher bit rates. For example, given 32 blocks in a 10-bit sequence, each block initially has 32 codewords. When the bit rate is relatively low, we can use codewords between

[2840] to select the codewords for each block. When the bit rate is high, we can select codewords between [31 33] for each block, or we can simply utilize the recognition reshaper curve.

[0294] Given slices (or tiles), slice (or tile) level reshaping can be performed in various ways that balance coding efficiency and complexity, including: 1) disabling reshaping only in intra-slices; 2) disabling reshaping in specific inter-slices (e.g., inter-slices at a specific time level, or inter-slices not used for reference images, or inter-slices considered unimportant reference images). This slice adaptation can also be QP / rate dependent, allowing different adaptation rules to be applied to different QPs or bit rates.

[0295] In the encoder, under the proposed algorithm, the variance of each block is calculated (e.g., BinVar(b) in equation (13)). Based on this information, codewords can be assigned based on the variance of each block. In one embodiment, BinVar(b) can be inversely linearly mapped to the number of codewords in each block b. In another embodiment, a nonlinear mapping (e.g., BinVar(b)) is used. 2 The sqrt(BinVar(b)) and similar methods can be used to inversely map the number of codewords in block b. Essentially, this method allows the encoder to apply arbitrary codewords to each block (beyond the simpler mappings previously used, where the encoder utilized two upper bounds M). f and M a (For example, see the following) Figure 6C (or three upper limits M) f 32 or M a (For example, see the following) Figure 6D (to allocate codewords in each group).

[0296] As an example Figure 6E Two codeword allocation schemes based on the BinVar(b) value are depicted. Graph 610 depicts codeword allocation using two thresholds, while graph 620 depicts codeword allocation using an inverse linear mapping, where the codeword allocation of a group is inversely proportional to its BinVar(b) value. For example, in an embodiment, the following code can be applied to derive the number of codewords (bin_cw) in a specific group:

[0297] alpha=(minCW-maxCW) / (maxVar-minVar);

[0298] beta=(maxCW*maxVar-minCW*minVar) / (maxVar-minVar);

[0299] bin_cw=round(alpha*bin_var+beta);,

[0300] Where minVar represents the minimum variance among all groups, maxVar represents the maximum variance among all groups, and minCW and maxCW represent the minimum and maximum number of codewords in each group, as determined by the reshaping model.

[0301] Luminance-based chromaticity QP offset correction

[0302] In reference [6], to compensate for the interaction between luminance and chrominance, an additional chrominance QP offset (denoted as chromaDQP or cQPO) and a luminance-based chrominance residual converter (cScale) are defined. For example:

[0303] chromaQP=QP_luma+chromaQPOffset+cQPO, (28)

[0304] Where chromaQPOffset represents the chroma QP offset, and QP_luma represents the luminance QP of the coding unit. As presented in reference [6], in the embodiment:

[0305]

[0306] Where FwdLUT′ represents the slope (first derivative) of FwdLUT(). For inter-frame slices, This represents the average predicted brightness value of the CU. For intra-frame slices, This represents the inverse reshaping value of the average predicted luma value of the CU. When dual-tree coding is used for the CU (i.e., the luma and chroma components have two separate coding trees and luma reconstruction is therefore available before chroma coding begins), the average reconstructed luma value of the CU can be used to derive the cQPO value. The cScale conversion factor is defined as:

[0307]

[0308] Where y = pow(2, x) means y = 2 x function.

[0309] Given that the luminance-derived QP value (denoted as qPi) and the final chrominance QP value (denoted as Qp) CThe nonlinear relationships between (e.g., see Tables 8 to 10, "Qp as a function of qPi for ChromaArrayType equal to 1" in reference [4]) C Specification of Qp C As a function of qPi for ChromaArrayType equalto 1), in the embodiment, cQPO and cScale can be further adjusted as follows.

[0310] The mapping between the adjusted luminance and chromaticity QP values ​​is represented as f_QPi2QPc() (e.g., in Tables 8 to 10 of reference [4]), then...

[0311] chromaQP_actual=f_QPi2QPc[chromaQP]=

[0312] =f_QPi2QPc[QP_luma+chromaQPOffset+cQPO]. (31)

[0313] To calculate the chroma residual, the ratio needs to be calculated based on the actual difference between the actual chroma codes (QPs) before and after applying cQPO:

[0314] QPcBase=f_QPi2QPc[QP_luma+chromaQPOffset];

[0315] QPcFinal=f_QPi2QPc[QP_luma+chromaQPOffset+cQPO]; (32)

[0316] cQPO_refine=QPcFinal-QpcBase;

[0317] cScale=pow(2,-cQPO_refine / 6).

[0318] In another embodiment, we can also absorb chromaQPOffset into cScale. For example:

[0319] QPcBase=f_QPi2QPc[QP_luma];

[0320] QPcFinal=f_QPi2QPc[QP_luma+chromaQPOffset+cQPO]; (33)

[0321] cTotalQPO_refine=QPeFinal-QpcBase;

[0322] cScale=pow(2,-cTotalQPO_refine / 6).

[0323] As an example, as described in reference [6], in the embodiments:

[0324] Setting CSCALE_FP_PREC = 16 indicates the precision parameter.

[0325] • Forward conversion: After the chromaticity residual is generated, but before transformation and quantization:

[0326] -C_Res=C_orig-C_pred

[0327] -C_Res_scaled=C_Res*cScale+(1<<(CSCALE_FP_PREC-1)))>>CSCALE_FP_PREC

[0328] • Inverse transformation: After inverse chroma quantization and inverse transformation, but before reconstruction:

[0329] -C_Res_inv=(C_Res_scaled<<CSCALE_FP_PREC) / cScale

[0330] -C_Reco = C_Pred + C_Res_inv;

[0331] In an alternative embodiment, the calculation for in-loop chromaticity reshaping can be expressed as follows. At the encoder side, the residual (CxRes = CxOrg - CxPred) for each chromaticity component Cx (e.g., Cb or Cr) of each CU or TU,

[0332]

[0333] Where CxResScaled is the converted Cb or Cr residual signal of the CU to be transformed and quantized. On the decoder side, CxResScaled is the converted chroma residual signal after inverse quantization and transformation, and...

[0334]

[0335]

[0336] The final reconstruction of the chromaticity components is

[0337] CxRec = CxPred + CxRes. (36)

[0338] This method allows the decoder to begin dequantization and transform operations for real-time chroma decoding after parsing. The cScale value used for the CU can be shared by the Cb and Cr components, and it can be derived from equations (29) and (30) as follows:

[0339]

[0340] in It is the average predicted luminance value of the current CU in the inter-frame slice (where dual-tree coding is not used and the reconstructed luminance is therefore unavailable), and It is the average reconstructed luminance value of the current CU in the intra-frame slice (where dual-tree coding is used). In the embodiment, the ratio is calculated and stored as a 16-bit fixed-point integer and the conversion operation at both the encoder and decoder sides is performed using fixed-point integer arithmetic. This represents the first derivative of the positive reshaping function. Assuming a piecewise linear representation of the curve, when Y belongs to the k-th group, FwdLUT(Y) = (CW[k] / 32). To reduce hardware latency, in another embodiment (see...) Figure 2E )middle, The average predicted luminance value of the current CU can be used from both intra-frame and inter-frame modes, regardless of the slice type or whether dual-tree is used. In another embodiment, the reconstructed CUs (e.g., the CUs in the top and / or left column of the current CU) from intra-frame and / or inter-frame modes can be used to derive the luminance. In another embodiment, higher-order syntax can be used to explicitly send region-based average, median and similar values, luminance values ​​or cScale values ​​in the bitstream.

[0341] The use of cScale is not limited to chromaticity residual conversion for in-loop reshaping. The same method can also be applied to out-of-loop reshaping. In out-of-loop reshaping, cScale can be used for chromaticity sample conversion. The operation is the same as the in-loop method.

[0342] On the encoder side, when calculating the chromaticity RDOQ, it is also necessary to calculate the λ correction value for chromaticity adjustment based on the correction offset (when using the QP offset or when using the chromaticity residual conversion):

[0343] Modifier=pow(2,-cQPO_refine / 3);

[0344] New_lambda=Old_lambda / Modifier. (38)

[0345] As mentioned in equation (35), utilizing cScale requires division in the decoder. To simplify the decoder implementation, we can decide to implement the same functionality using division in the encoder and apply simpler multiplication in the decoder. For example, making

[0346] cScaleInv = (1 / cScale)

[0347] Next, as an example, on the encoder

[0348] cResScale=CxRes*cScale=CxRes / (1 / cScale)=CxRes / cScaleInv,

[0349] And on the decoder

[0350] CxRes=cResScale / cScale=CxRes*(1 / cScale)=CxRes*cScaleInv.

[0351] In this embodiment, each luminance-dependent chromaticity conversion factor can be calculated for the corresponding luminance range in the piecewise linear (PWL) representation, rather than for each luminance codeword value. Therefore, the chromaticity conversion factor can be stored in a smaller LUT (e.g., with 16 or 32 items) (i.e., cScaleInv[binIdx]) instead of a 1024-item LUT (for 10-bit luminance codewords) (i.e., cScale[Y]). The conversion operations at both the encoder and decoder sides can be performed using fixed-point integer arithmetic as follows:

[0352] c'=sign(c)*((abs(c)*s+2 CSCALE_FP_PREC-1 )>>CSCALE_FP_PREC),

[0353] Where c is the chromaticity residual, s is the chromaticity residual conversion factor from cScaleInv[binIdx], binIdx is determined by the corresponding average luminance value, and CSCALE_FP_PREC is a constant value related to precision.

[0354] In embodiments, while the forward reshaping function can be represented using N equal segments (e.g., N = 8, 16, 32, and the like), the inverse representation will include nonlinear segments. From an implementation perspective, it would be desirable to also use equal segments to represent the inverse reshaping function; however, enforcing this representation can lead to a decrease in coding efficiency. As a compromise, in embodiments, we may be able to construct the inverse reshaping function using a “hybrid” PWL representation that combines equal and unequal segments. For example, when using 8 segments, we could first divide the entire range into two equal segments, and then subdivide each of these into 4 unequal segments. Alternatively, we could divide the entire range into 4 equal segments and then subdivide each of these into two unequal segments. Alternatively, we could first divide the entire range into several unequal segments, and then subdivide each unequal segment into several equal segments. Alternatively, we can first divide the entire range into two equal segments, and then subdivide each equal segment into equal subsegments, where the segment lengths in each group of subsegments are different.

[0355] For example, but not limited to, using 1,024 codewords, we can have: a) four segments each with 150 codewords and two segments each with 212 codewords, or b) eight segments each with 64 codewords and four segments each with 128 codewords. The general purpose of this segmentation combination is to reduce the number of comparisons required to identify the PWL segment index given a code value, thus simplifying the hardware and software implementation.

[0356] In the embodiments, for a more efficient implementation related to chromaticity residual conversion, the following changes may be enabled:

[0357] • When using a separate luminance / chrominance tree, disable chrominance residual conversion;

[0358] • Discontinue the chromaticity residual conversion for 2×2 chromaticity; and

[0359] • Use the predicted signal, rather than the reconstructed signal, for intra-frame and inter-frame coding units.

[0360] As an example, given Figure 2E The decoder 200D_D described in the document is used to process the luminance component. Figure 2F Describe the instance architecture 200D_DC used to process the corresponding chroma samples.

[0361] like Figure 2F As described in the text, when processing chroma, compared to Figure 2E The following changes will be made:

[0362] • Do not utilize forward and reverse reshaping blocks (282 and 265_3);

[0363] • A new chromaticity residual conversion block 288 exists to effectively replace the luminance inverse reshaping block 265_3; and

[0364] • Modify the reconstruction block 285-C to handle the color residuals in the original domain, as described in equation (36):

[0365] CxRec = CxPred + CxRes.

[0366] From equation (34), on the decoder side, let CxResScaled represent the extracted and transformed chroma residual signal after inverse quantization and transformation (before block 288), and let

[0367] CxRes = CxResScaled * C ScaleInv

[0368] This indicates the recalculated chromatic residual generated by the chromatic residual conversion block 288 used by reconstruction unit 285-C to calculate CxRec = CxPred + CxRes, where CxPred is generated by intra-frame prediction block 284 or inter-frame prediction block 280.

[0369] C for conversion unit (TU) ScaleInv The value can be shared by the Cb and Cr components and can be calculated as follows:

[0370] • If in intra-frame mode, then calculate the average of the intra-frame predicted luminance values;

[0371] • In inter-frame mode, the average of the predicted luminance values ​​for forward reshaping is calculated. That is, the average luminance value avg Y′ is calculated in the reshaping domain. TU ;and

[0372] • In the case of combination and intra-frame prediction, the average of the combined predicted brightness values ​​is calculated. For example, the combined predicted brightness values ​​can be calculated according to Appendix 2, section 8.4.6.6.

[0373] In this embodiment, we can apply a LUT based on avgY′ TU To calculate C ScaleInv Alternatively, given the piecewise linear (PWL) representation of the reshaping function, we can find the value belonging to the value avgY′ in the inverse PWL mapping. TU The index idx.

[0374] Next, C ScaleInv =cScaleInv[idx].

[0375] Currently, with the development of ITU and ISO, example implementations applicable to multi-functional video coding codecs (reference [8]) can be found in Appendix 2 (see, for example, section 8.5.5.1.2).

[0376] Disabling luminance-based chrominance residual conversion for intra-frame slices using dual-trees can lead to some loss of coding efficiency.

[0377] To improve the effect of color reshaping, the following methods can be used:

[0378] 1. The chromaticity conversion factor remains constant across the entire frame, depending on the average or median of the luminance sample values. This eliminates the TU-level dependency on luminance for chromaticity residual conversion.

[0379] 2. The chromaticity conversion factor can be derived using the reconstructed luminance values ​​from adjacent CTUs.

[0380] 3. The encoder can derive a chromaticity conversion factor based on the source luminance pixels and send it to the bitstream at the CU / CTU level (e.g., as an exponent representing a piecewise representation of the reshaping function). The decoder can then extract the chromaticity conversion factor from the reshaping function without depending on the luminance data.

[0381] The conversion factor for CTU can be derived and is only used for intra-frame transmission; however, it can also be used for inter-frame transmission. The additional transmission cost occurs only for intra-frame transmission and therefore has no impact on coding efficiency in random access.

[0382] 4. Chroma can be reshaped into luminance at the frame level, whereby a chroma reshaping curve is derived from a luminance reshaping curve based on correlation analysis between luminance and chroma. This completely eliminates chroma residual conversion.

[0383] delta_qp application

[0384] In AVC and HEVC, the parameter delta_qp allows modification of the QP value of a coded block. In this embodiment, we can derive the delta_qp value using the luma curve in the reshaper. We can derive the segmented lumaDQP value based on the codeword assignment for each group. For example:

[0385]

[0386] INT() can be CEIL(), ROUND(), or FLOOR(). The encoder can use a luminance function (e.g., average(luma), min(luma), max(luma), and similar) to find the luminance value of the block, and then use the corresponding lumaDQP value for the block. To obtain the rate-distortion benefit from equation (27), we can use weighted distortion and set it in the mode decision.

[0387]

[0388] Consideration of the number of remolding and grouping

[0389] In typical 10-bit video coding, at least 32 groups are preferably used for reshaping the codeword; however, to simplify the decoder implementation, in this embodiment, fewer groups can be used, i.e., 16 or even 8 groups. Given that the encoder can already use 32 groups to analyze the sequence and derive the distribution codewords, we can reuse the original 32-group codeword distribution and derive 16-group codewords by adding two corresponding 16-groups within each 32-group, i.e.:

[0390] for i = 0 to 15

[0391] CWIn16Bin[i]=CWIn32Bin[2i]+CWIn32Bin[2i+1].

[0392] For the chroma residual conversion factor, we can simply divide the codeword by 2 and point to a 32-block chromaScalingFactorLUT. For example, given...

[0393] CWIn32Bin

[32] ={0 0 33 38 38 38 38 38 38 38 38 38 38 38 38 38 38 33 3333 33 33 33 33 33 33 33 33 33 33 0 0},

[0394] The corresponding 16-group CW allocation is

[0395] CWIn16Bin

[16] ={0 71 76 76 76 76 76 76 71 66 66 66 66 66 66 0}.

[0396] This method can be extended to handle even fewer groups (i.e., 8), then

[0397] for i = 0 to 7

[0398] CWIn8Bin[i]=CWIn16Bin[2i]+CWIn16Bin[2i+1].

[0399] When using a narrow range of valid codewords (e.g., [64, 940] for a 10-bit signal and [64, 235] for an 8-bit signal), it should be noted that the first and last blocks are not considered for mapping to reserved codewords. For example, for a 10-bit signal with 8 blocks, each block would have 1024 / 8 = 128 codewords, and the first block would be [0, 127]; however, since the standard codeword range is [64, 940], the first block should only consider codewords [64, 127]. Special flags (e.g., video_full_range_flag = 0) can be used to inform the decoder that the range of the input video is greater than the full range [0, 235]. bitdepth -1] is narrower, and special care should be taken when processing the first and last groups to avoid generating illegal codewords. This applies to luminance and chrominance reshaping.

[0400] As examples (but not limited to), Appendix 2 provides example syntax structures and associated syntax elements for use with Figure 2C , Figure 2E and Figure 2F The implementation of the architecture described herein supports reshaping in the ISO / ITU Video Multifunction Codec (VVC) (Reference [8]), wherein the forward reshaping function comprises 16 segments.

[0401] References

[0402] The full text of each reference listed in this article is incorporated by way of citation.

[0403] [1] "Exploratory Test Model for HDRextension of HEVC", K. Minoo et al., MPEG Output File, JCTVC-W0092(m37732), 2016, San Diego, USA.

[0404] [2] PCT application PCT / US2016 / 025082, filed by GM. Su on March 30, 2016, entitled “In-Loop Block-Based Image Reshaping in High Dynamic Range Video Coding”, is also published as WO 2016 / 164235.

[0405] [3] U.S. Patent Application No. 15 / 410,563, filed January 19, 2017 by T. Lu et al., entitled “Content-Adaptive Reshaping for HighCodeword Representation Images”.

[0406] [4]ITU-T H.265, “High efficiency video coding”, ITU, December 2016.

[0407] [5] PCT application PCT / US2016 / 042229, filed by P. Yin et al. on July 14, 2016, entitled “Signal Reshaping and Coding for HDR and Wide Color Gamut Signals”, is also published as WO 2017 / 011636.

[0408] [6] PCT patent application PCT / US2018 / 040287 filed by T. Lu et al. on June 29, 2018, entitled “Integrated Image Reshaping and Video Coding”.

[0409] [7] J. Froehlich et al., “Content-Adaptive Perceptual Quantizer for High Dynamic Range Images”, U.S. Patent Application Publication No. 2018 / 0041759, February 8, 2018.

[0410] [8] "Versatile Video Coding (Draft 3)" uploaded by B. Bross, J. Chen and S. Liu on January 8, 2019, JVET output file, JVET-L1001, v9.

[0411] Example computer system implementation plan

[0412] Embodiments of the invention may be implemented using the following: computer systems, systems configured in electronic circuits and components, integrated circuit (IC) devices (e.g., microcontrollers), field-programmable gate arrays (FPGAs) or other configurable or programmable logic devices (PLDs), discrete-time or digital signal processors (DSPs), application-specific integrated circuits (ASICs), and / or devices comprising one or more of such systems, devices, or components. The computer and / or IC may perform, control, or execute instructions related to signal reshaping and encoding of images (e.g., the signal reshaping and encoding described herein). The computer and / or IC may calculate any of the various parameters or values ​​associated with the signal reshaping and encoding process described herein. Image and video embodiments may be implemented as hardware, software, firmware, and various combinations thereof.

[0413] Specific embodiments of the invention include a computer processor that executes software instructions that cause the processor to perform the methods of the invention. For example, one or more processors, such as a display, encoder, set-top box, transcoder, or the like, can implement the methods related to signal reshaping and encoding of images described above by executing software instructions in a processor-accessible program memory. The invention may also be provided in the form of a program product. The program product may include any non-transitory and tangible medium carrying a set of computer-readable signals including instructions that, when executed by a data processor, cause the data processor to perform the methods of the invention. Program products according to the invention may take any of a variety of non-transitory and tangible forms. Program products may include, for example, physical media, such as magnetic data storage media (including floppy disks, hard disk drives), optical data storage media (including CD-ROMs, DVDs), electronic data storage media (including ROMs, flash RAM), or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0414] When a component (e.g., software module, processor, assembly, device, circuit, etc.) is mentioned above, unless otherwise indicated, the reference to the component (including the reference to "component") shall be interpreted as including any component (e.g., functionally equivalent) that performs the function of the described component as an equivalent of the component, including components that are structurally not equivalent to the disclosed structure and perform the function of the illustrated exemplary embodiments of the invention.

[0415] Equivalence, extension, substitution and others

[0416] Therefore, exemplary embodiments related to high-efficiency signal reshaping and encoding of images are described. In the above description, embodiments of the invention have been described with reference to numerous specific details that may vary with implementation. Therefore, the unique indicators that the applicant wishes to consider as the invention for inventive purposes are the set of claims issued in a specific form by this application, such claims being issued in said specific form (which includes any subsequent corrections). Any definitions expressly set forth herein with respect to terms contained in such claims shall govern the meaning of such terms used in the claims. Therefore (but not limited to), elements, properties, features, advantages, or attributes not expressly listed in the claims shall in any way limit the scope of this claim. Therefore, this specification and drawings should be regarded as illustrative rather than limiting.

[0417] Enumerated Instance Examples

[0418] This invention may be embodied in any of the forms described herein, including (but not limited to) the following enumerated examples of embodiments (EEE) that describe the structure, features, and functions of some parts of the invention.

[0419] EEE 1. A method for adaptively reshaping a video sequence using a processor, the method comprising:

[0420] The processor is used to access the input image represented by the first codeword; and

[0421] Generate a forward reshaping function that maps pixels of the input image to a second codeword representation, wherein the second codeword representation allows for more efficient compression than the first codeword representation, wherein generating the forward reshaping function includes:

[0422] The input image is divided into multiple pixel regions;

[0423] Each pixel region is assigned to one of a plurality of codeword groups based on the first brightness characteristic of each pixel region.

[0424] The grouping metric of each of the plurality of codeword groups is calculated based on the second luminance characteristic of each of the pixel regions assigned to each codeword group;

[0425] The codewords represented by the second codeword are assigned to each codeword group according to the grouping metric and rate-distortion optimization criterion of each codeword group;

[0426] The positive reshaping function is generated in response to assigning a codeword represented by the second codeword to each of the plurality of codeword groups.

[0427] EEE 2. The method according to EEE1, wherein the first brightness characteristic of the pixel region includes the average brightness pixel value of the pixel region.

[0428] EEE 3. The method according to EEE1, wherein the second luminance characteristic of the pixel region includes the variance of the luminance pixel values ​​of the pixel region.

[0429] EEE 4. The method according to EEE3, wherein calculating the grouping metric of the codeword group includes: calculating the average of the variances of the luminance pixel values ​​of all pixel regions assigned to the codeword group.

[0430] EEE 5. The method according to EEE1, wherein allocating a plurality of codewords represented by the second codeword to the codeword group according to a grouping metric of the codeword group comprises:

[0431] If no pixel region is assigned to the codeword group, then no codeword is assigned to the codeword group.

[0432] If the grouping metric of the codeword group is lower than the upper threshold, then a first number of codewords are assigned; and

[0433] Otherwise, the second number of codewords are assigned to the codeword group.

[0434] EEE 6. According to the method described in EEE5, wherein for a first codeword representation having a depth of B bits and having B bits... o The second codeword representation with a depth of units and N codeword groups, wherein the first number of codewords includes M f =CEIL((2 Bo / (CW2-CW1))*M a And the second number of codewords includes M a =2 B / N, where CW1 < CW2 indicates [0 2 B The two codewords in [-1].

[0435] EEE 7. According to the method described in EEE6, where CW1 = 16 * 2 ( B-8) And CW2 = 235 * 2 (B-8) .

[0436] EEE 8. The method according to EEE 5, wherein determining the upper threshold includes:

[0437] Define a set of possible thresholds;

[0438] For each threshold in the group of thresholds:

[0439] A positive reshaping function is generated based on the threshold;

[0440] Encode and decode a set of input test frames according to the reshaping function and bit rate R to generate a set of output decoded test frames; and

[0441] Calculate the Total Rate Distortion Optimization (RDO) metric based on the input test frame and the decoded test frame; and

[0442] The upper threshold is selected from the possible thresholds of the group that minimizes the RDO metric.

[0443] EEE 9. The method according to EEE8, wherein calculating the RDO metric comprises: calculating J = D + λR, where D represents the distortion measurement between the pixel value of the input test frame and the corresponding pixel value in the decoded test frame, and λ represents the Lagrange multiplier.

[0444] EEE 10. The method according to EEE9, wherein D is a measurement of the sum of variances between the corresponding pixel values ​​of the input test frame and the decoded test frame.

[0445] EEE 11. The method according to EEE1, wherein assigning a plurality of codewords represented by the second codeword to the codeword group based on a grouping metric of the codeword group is based on a codeword allocation lookup table, wherein the codeword allocation lookup table defines two or more thresholds that divide the range of grouping metric values ​​into a plurality of segments and provides the grouping metric within each segment to the plurality of codewords assigned to the group.

[0446] EEE 12. The method according to EEE11, wherein, given the default codeword allocation for a group, fewer codewords than the default codeword allocation are assigned to groups with larger group sizes and more codewords than the default codeword allocation are assigned to groups with smaller group sizes.

[0447] EEE 13. According to the method of EEE12, wherein for a first codeword representation having B bits and N blocks, the default codeword allocation for each block is determined by M. a =2 B / N is given.

[0448] EEE 14. The method according to EEE1, further comprising generating reshaping information in response to the positive reshaping function, wherein the reshaping information includes one or more of the following:

[0449] A flag indicating the minimum codeword block index value used in the reshaping and reconstruction process.

[0450] A flag indicating the maximum codeword block index value used in the reshaping construction process.

[0451] A flag indicating the distribution type of the reshaping model, where each model distribution type is associated with a default grouping parameter, or

[0452] Used to adjust one or more difference values ​​of the default grouping related parameters.

[0453] EEE 15. The method according to EEE 5 further includes assigning a group importance value to each codeword group, wherein the group importance value is:

[0454] 0, if no codeword is assigned to the codeword group;

[0455] 2. If the first value of the codeword is assigned to the codeword group; and

[0456] Otherwise, it is 1.

[0457] EEE 16. The method according to EEE5, wherein determining the upper threshold comprises:

[0458] Divide the brightness range of the pixel values ​​in the input image into several groups;

[0459] For each group, a group histogram value and a mean group variance value are determined, wherein for each group, the group histogram value includes the number of pixels in the group out of the total number of pixels in the image, and the mean group variance value provides a measure of the mean pixel variance of the pixels in the group;

[0460] Sort the average group variance values ​​to generate a sorting table of average group variance values ​​and a sorting table of average group variance value indices.

[0461] The cumulative density function, varying according to the sorted average group variance, is calculated based on the sorted histogram values ​​and the index of the average group variance.

[0462] The upper threshold is determined based on a criterion satisfied by the value of the cumulative density function.

[0463] EEE 17. The method according to EEE 16, wherein calculating the cumulative density function includes calculating:

[0464]

[0465] Where b represents the number of groups, PIC_ANALYZE_CW_BINS represents the total number of groups, BinVarSortDsdCDF[b] represents the output of the CDF function for group b, BinHist[i] represents the group histogram value for group i, and BinIdxSortDsd[] represents the sorting table of the average group variance index.

[0466] EEE 18. The method according to EEE16, wherein the upper threshold is determined as the average group variance value that makes the CDF output k% of the pixels in the input image greater than or equal to the upper threshold.

[0467] EEE 19. The method described in EEE18, where k = 50.

[0468] EEE 20. In a decoder, a method for reconstructing a reshaping function, the method comprising:

[0469] Receive syntax elements characterized by a reshaping model in an encoded bitstream, wherein the syntax elements include one or more of the following

[0470] A flag indicating the minimum codeword block index value used in the reshaping construction process.

[0471] A flag indicating the maximum codeword block index value used in the remodeling construction process.

[0472] A flag indicating the distribution type of the reshaping model, wherein the model distribution type is associated with a default grouping correlation parameter that includes grouping importance values, or

[0473] A flag indicating one or more differential grouping importance values ​​used to adjust the default grouping importance values ​​defined in the reshaping model distribution;

[0474] Based on the reshaping model distribution, a default group importance value is determined for each group, and an allocation table is used to assign several default codewords to each group according to the importance value of the group.

[0475] Grouping for each codeword:

[0476] Its group importance value is determined by adding its default group importance value to its difference group importance value;

[0477] The number of codewords allocated to the codeword group is determined based on the group importance value and the allocation table; and

[0478] A positive reshaping function is generated based on the number of codewords allocated to each codeword group.

[0479] EEE 21. The method according to EEE 20, wherein M is determined using the allocation table. k (The number of codewords allocated to the k-th codeword block) further includes:

[0480]

[0481] Where Ma and M f It is an element of the allocation table, and bin_importance[k] represents the importance value of the k-th group.

[0482] EEE 22. In a decoder including one or more processors, a method for reconstructing encoded data, the method comprising:

[0483] Receive coded bit stream 122, the coded bit stream 122 including one or more coded reshaped images represented by a first codeword and metadata 207 related to the reshaping information of the coded reshaped images;

[0484] 250 anti-reshaping functions are generated based on the metadata associated with the reshaping information, wherein the anti-reshaping functions map pixels from the reshaping image represented by the first codeword to a second codeword representation;

[0485] A 250-fold forward reshaping function is generated based on the metadata associated with the reshaping information, wherein the forward reshaping function maps pixels from the image in the second codeword representation to the first codeword representation;

[0486] Extract a coded reshaped image comprising one or more coding units from the coded bitstream, wherein for one or more coding units in the coded reshaped image:

[0487] For the intra-frame coding unit (CU) in the coded reshaped image:

[0488] The first reshaping reconstruction sample 227 of the CU is generated based on the reshaping residual in the CU and the first reshaping prediction sample.

[0489] The 270 reshaping loop filter output is generated based on the first reshaping reconstruction sample and loop filter parameters.

[0490] The inverse reshaping function is applied at 265 to the output of the reshaping loop filter to generate a decoded sample of the encoded unit represented by the second codeword; and

[0491] The decoded sample of the encoding unit represented by the second codeword is stored in the reference buffer;

[0492] For the inter-frame coding units in the encoded reshaped image:

[0493] The positive reshaping function is applied to the prediction sample stored in the reference buffer in the second codeword representation to generate a second reshaping prediction sample;

[0494] The second reshaping reconstruction sample of the coding unit is generated based on the reshaping residual in the coding CU and the second reshaping prediction sample.

[0495] The reshaping loop filter output is generated based on the second reshaping reconstruction sample and the loop filter parameters.

[0496] The inverse reshaping function is applied to the output of the reshaping loop filter to generate samples of the coding unit represented by the second codeword; and

[0497] The sample of the encoded unit represented by the second codeword is stored in a reference buffer; and

[0498] The decoded image is generated based on the stored samples in the reference buffer.

[0499] EEE 23. An apparatus comprising a processor and configured to perform the method according to any one of EEE 1 to 22.

[0500] EEE 24. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing a method according to any one of EEE 1 to 22 using one or more processors.

[0501] Appendix 1

[0502] Example implementation of bubble sort

[0503]

[0504] Appendix 2

[0505] As an example, this appendix provides an example syntax structure and associated syntax elements according to the embodiments to support the reshaping of the Multifunctional Video Codec (VVC) (Reference [8]) under the current joint development by ISO and ITU. New syntax elements in existing draft versions are highlighted or explicitly mentioned. In the final description, equation numbers (such as (8-xxx)) represent placeholders that are updated as needed.

[0506] In the RBSP syntax of sequence parameter sets in 7.3.2.1

[0507]

[0508]

[0509]

[0510]

[0511] In 7.3.3.1 General Block Group Header Syntax

[0512]

[0513]

[0514]

[0515] Added a new syntax table block group reshaper model:

[0516]

[0517] The following semantics are added to the general sequence parameter set RBSP semantics:

[0518] A value of 1 for sps_reshaper_enabled_flag indicates that the reshaper is used in encoded video sequences (CVS). A value of 0 for sps_reshaper_enabled_flag indicates that the reshaper is not used in CVS.

[0519] The following semantics have been added to the tile group header syntax:

[0520] A `tile_group_reshaper_model_present_flag` value of 1 indicates that `tile_group_reshaper_model()` exists in the tile group header. A `tile_group_reshaper_model_present_flag` value of 0 indicates that `tile_group_reshaper_model()` does not exist in the tile group header. When `tile_group_reshaper_model_present_flag` does not exist, it is inferred that it is equal to 0.

[0521] A tile_group_reshaper_enabled_flag value of 1 indicates that the reshaper is enabled for the current tile group. A tile_group_reshaper_enabled_flag value of 0 indicates that the reshaper is not enabled for the current tile group. If tile_group_reshaper_enable_flag does not exist, it is inferred that it is equal to 0.

[0522] A value of 1 for `tile_group_reshaper_chroma_residual_scale_flag` indicates that chroma residual scaling is enabled for the current tile group. A value of 0 for `tile_group_reshaper_chroma_residual_scale_flag` indicates that chroma residual scaling is not enabled for the current tile group. If `tile_group_reshaper_chroma_residual_scale_flag` does not exist, it is assumed to be equal to 0.

[0523] Added the `tile_group_reshaper_model()` syntax.

[0524] `reshaper_model_min_bin_idx` specifies the minimum grouping (or segment) index used in the reshaper construction process. The value of `reshaper_model_min_bin_idx` should be in the range of 0 to `MaxBinIdx` (inclusive). The value of `MaxBinIdx` should be equal to 15.

[0525] `reshaper_model_delta_max_bin_idx` specifies the maximum allowed grouping (or segment) index `MaxBinIdx` minus the maximum grouping index used in the reshaper construction process. Set the value of `reshaper_model_max_bin_idx` to equal `MaxBinIdx - reshaper_model_delta_max_bin_idx`.

[0526] reshaper_model_bin_delta_abs_cw_prec_minus1+1 specifies the number of bits used in the representation of the syntax reshaper_model_bin_delta_abs_CW[i].

[0527] reshaper_model_bin_delta_abs_CW[i] specifies the absolute differential codeword value for the i-th group.

[0528] reshaper_model_bin_delta_sign_CW_flag[i] specifies the sign of reshaper_model_bin_delta_abs_CW[i] as follows:

[0529] – If reshaper_model_bin_delta_sign_CW_flag[i] equals 0, then the corresponding variable RspDeltaCW[i] is positive.

[0530] Otherwise (reshaper_model_bin_delta_sign_CW_flag[i] is not equal to 0), the corresponding variable RspDeltaCW[i] is negative.

[0531] If reshaper_model_bin_delta_sign_CW_flag[i] does not exist, it is inferred that it is equal to 0.

[0532] Variable RspDeltaCW[i]=(1-2*reshaper_model_bin_delta_sign_CW[i])*reshaper_model_bin_delta_abs_CW[i];

[0533] The variable RspCW[i] is exported using the following steps:

[0534] Set the OrgCW variable to equal (1 < <BitDepthY) / (MaxBinIdx+1)。

[0535] – If reshaper_model_min_bin_idx<=i<=reshaper_model_max_bin_idx,

[0536] Then RspCW[i] = OrgCW + RspDeltaCW[i].

[0537] Otherwise, RspCW[i] = 0.

[0538] If the value of BitDepthY is equal to 10, then the value of RspCW[i] should be in the range of 32 to 2*OrgCW-1.

[0539] The variable InputPivot[i] (where i is in the range from 0 to MaxBinIdx+1 (inclusive)) is derived as follows:

[0540] InputPivot[i] = i * COrgCW

[0541] The variables ReshapePivot[i] (where i is in the range from 0 to MaxBinIdx+1 (inclusive)), ScaleCoef[i], and InvScaleCoeff[i] (where i is in the range from 0 to MaxBinIdx (inclusive)) are derived as follows:

[0542]

[0543] The variable ChromaScaleCoef[i] (where i is in the range from 0 to MaxBinIdx (inclusive)) is derived as follows:

[0544]

[0545] Note: In an alternative implementation, we can combine the luminance and chrominance conversions, thus eliminating the need for ChromaResidualScaleLut[]. The chrominance conversion can then be implemented as follows:

[0546]

[0547] The following additions have been made to the weighted sample prediction process used for merging and intra-frame prediction. Highlighting the additions.

[0548] 8.4.6.6 Weighted Sample Prediction Process for Combining and Intra-Frame Prediction

[0549] The input for this process is:

[0550] – The width of the current encoded block, cbWidth

[0551] – The height of the current encoded block, cbHeight

[0552] – Two (cbWidth) × (cbHeight) arrays predSamplesInter and predSamplesIntra,

[0553] –Intra-frame prediction mode predModeIntra

[0554] – The variable cIdx specifies the color component index.

[0555] The output of this process is a (cbWidth)×(cbHeight) array of predicted sample values, predSamplesComb.

[0556] The variable bitDepth is exported as follows:

[0557] – If cIdx equals 0, then set bitDepth to equal BitDepthY.

[0558] Otherwise, set bitDepth to equal BitDepthC.

[0559] The predicted samples predSamplesComb[x][y] (where x = 0…cbWidth-1 and y = 0…cbHeight-1) are derived as follows:

[0560] – Weighted w is exported as follows:

[0561] – If predModeIntra is INTRA_ANGULAR50, then w is specified in Tables 8 to 10, where nPos equals y and nSize equals cbHeight.

[0562] Otherwise, if predModeIntra is INTRA_ANGULAR18, then w is specified in Tables 8 to 10, where nPos equals x and nSize equals cbWidth.

[0563] Otherwise, set w to equal 4.

[0564] – If cIdx equals 0, then predSamplesInter derives the following:

[0565] – If tile_group_reshaper_enabled_flag equals 1

[0566] Therefore, shiftY = 14

[0567] idxY=predSamplesInter[x][y]>>Log2(OrgCW)

[0568] predSamplesInter[x][y]=Clip1Y(ReshapePivot[idxY]

[0569] +(ScaleCoeff[idxY]*(predSamplesInter[x][y]-

[0570] InputPivot[idxY])+(1<<(shiftY–1)))>>shiftY)

[0571] (8-xxx)

[0572] Otherwise (tile_group_reshaper_enabled_flag equals 0)

[0573] predSamplesInter[x][y]=predSamplesInter[x][y]

[0574] – The predicted samples predSamplesComb[x][y] are derived as follows:

[0575]

[0576] Tables 8 to 10—Specifications of w varying according to position nP and size nS

[0577]

[0578] Add the following during image reconstruction:

[0579] 8.5.5 Image Reconstruction Process

[0580] The input for this process is:

[0581] – Specifies the position (xCurr, yCurr) of the top-left sample of the current block relative to the top-left sample of the current image component.

[0582] – Specifies the width and height of the current block using variables nCurrSw and nCurrSh, respectively.

[0583] – Specifies the color component cIdx of the current block.

[0584] – Specifies the (nCurrSw)×(nCurrSh) array predSamples for the predicted samples of the current block.

[0585] – Specifies the (nCurrSw)×(nCurrSh) array of residual samples for the current block, resSamples.

[0586] The following assignments are made based on the values ​​of the color component cIdx:

[0587] – If cIdx equals 0, then recSamples corresponds to the reconstructed image sample array SL and the function clipCidx1 corresponds to Clip1. Y .

[0588] Otherwise, if cIdx equals 1, then recSamples corresponds to the reconstructed chromaticity sample array S. Cb And the function clipCidx1 corresponds to Clip1 C .

[0589] Otherwise (cIdx equals 2), recSamples corresponds to the reconstructed chromaticity sample array S. Cr And the function clipCidx1 corresponds to Clip1 C .

[0590] When the value of `tile_group_reshaper_enabled_flag` is equal to 1, the (nCurrSw)×(nCurrSh) block of the reconstructed sample array `recSamples` at position (xCurr, yCurr) is exported as the mapping procedure specified in item 8.5.5.1. Otherwise, the (nCurrSw)×(nCurrSh) block of the reconstructed sample array `recSamples` at position (xCurr, yCurr) is exported as follows:

[0591]

[0592] Where i = 0…nCurrSw-1, j = 0…nCurrSh–1

[0593] 8.5.5.1 Image Reconstruction Using Mapping Process

[0594] This item specifies image reconstruction using a mapping process. 8.5.5.1.1 specifies image reconstruction using a mapping process for luminance sample values. 8.5.5.1.2 specifies image reconstruction using a mapping process for chrominance sample values.

[0595] 8.5.5.1.1 Image Reconstruction Using Mapping Process for Luminance Sample Values

[0596] The input for this process is:

[0597] - Specifies the (nCurrSw)×(nCurrSh) array predSamples for the brightness prediction samples of the current block.

[0598] - Specifies the (nCurrSw)×(nCurrSh) array of luminance residual samples resSamples for the current block.

[0599] The output of this process is:

[0600] -(nCurrSw)×(nCurrSh) Mapping brightness prediction sample array predMapSamples,

[0601] -(nCurrSw)×(nCurrSh) Reconstructs the luminance sample array recSamples.

[0602] The predMapSamples are exported as follows:

[0603] -If (CuPredMode[xCurr][yCurr]==MODE_INTRA)||(CuPredMode[xCurr][yCurr]==MODE_INTER && mh_intra_flag[xCurr][yCurr])

[0604] Then predMapSamples[xCurr+i][yCurr+j]=predSamples[i][j](8-xxx)

[0605] Where i=0...nCurrSw-1, i=0...nCurrSh-1

[0606] Otherwise, ((CuPredMode[xCurr][yCurr]==MODE_INTER && !mh_intra_flag[xCurr][yCurr])) applies as follows:

[0607] shiftY = 14

[0608] idxY=predSamples[i][j]>>Log2(OrgCW)

[0609] predMapSamples[xCurr+i][yCurr+j]=ReshapePivot[idxY]

[0610] +(ScaleCoeff[idxY]*(predSamples[i][j]-InputPivot[idxY])

[0611] +(1<<(shiftY-1)))>shiftY (8-xxx)

[0612] Where i = 0...nCurrSw-1, j = 0...nCurrSh-1

[0613] recSamples is exported as follows:

[0614] recSamples[xCurr+i][yCurr+j]=Clipl Y

[0615] (predMapSamples[xCurr+i][yCurr+j]+resSampoles[i][j]]) (8-xxx)

[0616] Where i = 0...nCurrSw-1, j = 0...nCurrSh-1

[0617] 8.5.5.1.2 Image Reconstruction Using Mapping Process for Chromaticity Sample Values

[0618] The input for this process is:

[0619] - Specifies the (nCurrSwx2)×(nCurrShx2) array mapping predMapSamples for the current block's mapped brightness prediction samples.

[0620] - Specifies the (nCurrSw)×(nCurrSh) array predSamples for the chromaticity prediction samples of the current block.

[0621] - Specifies the (nCurrSw)×(nCurrSh) array of chromaticity residual samples resSamples for the current block.

[0622] The output of this process is the reconstructed chroma sample array recSamples.

[0623] recSamples is exported as follows:

[0624] -if (!tile_group_reshaper_chroma_residual_scale_flag||((nCurrSw)x(nCurrSh)<=4))

[0625] Then recSamples[xCurr+i][yCurr+j]=Clip1C(predSamples[i][j]+resSamples[i][j]) (8-xxx)

[0626] Where i = 0...nCurrSw-1, j = 0...nCurrSh-1

[0627] - Otherwise (tile_group_reshaper_chroma_residual_scale_flag &&((nCurrSw)×(nCurrSh)>4)), the following applies:

[0628] The variable varScale is exported as follows:

[0629] 1.invAvgLuma=Clip1Y((∑i∑jpredMapSamples[(xCurr<<1)+i][(yCurr<<1)+j]+nCurrSw*nCurrSh*2) / (nCurrSw*nCurrSh*4))

[0630] 2. Derive the variable idxYInv by using the piecewise function exponent specified in item 8.5.6.2, which involves using the sample value invAvgLuma as input identification.

[0631] 3.varScale=ChromaScaleCoef[idxYInv]

[0632] recSamples is exported as follows:

[0633] - If tu_cbf_cIdx[xCurr][yCurr] equals 1, then the following applies:

[0634] shiftC = 11

[0635] recSamples[xCurr+i][yCurr+j]=ClipCidx1

[0636] (predSamples[i][j]+Sign(resSamples[i][j])

[0637] *((Abs(resSamples[i][j])*varScale+(1<<(shiftC-1)))>shiftC)) (8-xxx)

[0638] Where i = 0...nCurrSw-1, j = 0...nCurrSh-1

[0639] Otherwise (tu_cbf_cIdx[xCurr][yCurr] equals 0)

[0640] recSamples[xCurr+i][yCurr+j]=ClipCidx1(predSamples[i][j]) (8-xxx)

[0641] Where i = 0...nCurrSw-1, j = 0...nCurrSh-1

[0642] 8.5.6 Image Inverse Mapping Process

[0643] This item is invoked when the value of `tile_group_reshaper_enabled_flag` is equal to 1. The input is the reconstructed image brightness sample array S. L The output is the modified reconstructed image brightness sample array S' after the inverse mapping process. L .

[0644] The inverse mapping process for luminance sample values ​​is specified in 8.4.6.1.

[0645] 8.5.6.1 Image inverse mapping process of brightness sample values

[0646] The input to this process is the brightness position (xP, yP) that specifies the brightness sample position relative to the top-left brightness sample of the current image.

[0647] The output of this process is the inverse mapped luminance sample value invLumaSample.

[0648] The value of invLumaSample is derived by applying the following sequential steps:

[0649] 1. By calling the input S using the brightness sample value L [xP][yP]Identify the piecewise function exponent specified in item 8.5.6.2 to derive the variable idxYInv.

[0650] 2. The value of reshapeLumaSample is exported as follows:

[0651] shiftY = 14

[0652] invLumaSample=

[0653] InputPivot[idxYInv]+(InvScaleCoeff[idxYInv]*(S L [xP][yP]-ReshapePivot[idxYInv]) (8-xxx)

[0654] +(1<<(shiftY-1)))>>shiftY

[0655] 3.clipRange=((reshaper_model_min_bin_idx>0)&&(reshaper_model_max_bin_idx<MaxBinIdx));

[0656] When clipRange equals 1, the following applies:

[0657] minVal = 16 << (BitDepth) Y -8)

[0658] maxVal = 235 << (BitDepth) Y -8)

[0659] invLumaSample=Clip3(minVal, maxVal, invLumaSample)

[0660] Otherwise (clipRange equals 0), the following applies:

[0661] invLumaSample=ClipCidx1(invLumaSample)

[0662] 8.5.6.2 Identification of the piecewise function exponent for the luminance component

[0663] The input to this process is the brightness sample value S.

[0664] The output of this process is the exponent idxS that identifies the segment to which sample S belongs. The variable idxS is derived as follows:

[0665]

[0666] It should be noted that an alternative implementation for finding the identifier idxS is as follows:

[0667]

Claims

1. A method for reconstructing coded video data using one or more processors, the method comprising: Receive a coded bit stream comprising one or more coded reshaped images represented as input codewords; The reshaping metadata for the one or more coded reshaping images is received in the encoded bitstream, wherein the reshaping metadata includes parameters for generating a luminance-based chroma residual converter to calculate the decoded chroma residual and a forward reshaping function, wherein the forward reshaping function maps the pixels of the image from a first codeword representation to the input codeword representation, and wherein the reshaping metadata further includes: The first parameter indicates the minimum grouping index used in reshaping; The second parameter determines the active maximum grouping index used in the reshaping, wherein the active maximum grouping index is less than or equal to a predetermined maximum grouping index, and determining the active maximum grouping index includes calculating the difference between the predetermined maximum grouping index and the second parameter. Absolute differential codeword values, which are used for each active block in the input codeword representation; and The sign of the absolute difference codeword value is used for each active group in the input codeword representation; A positive reshaping function is generated based on the reshaping metadata; An inverse reshaping function is generated based on the reshaping metadata or the forward reshaping function, wherein the inverse reshaping function maps the pixels of the reshaping image from the input codeword representation to the first codeword representation; and The encoded bitstream is decoded based on the forward reshaping function, the inverse reshaping function, and the chroma residual converter.

2. The method of claim 1, wherein the forward reshaping function is reconstructed as a piecewise linear function having linear segments derived from the reshaping metadata.

3. A method for adaptively reshaping a video sequence using a processor, the method comprising: The processor is used to access the input image represented by the first codeword; Generate reshaping parameters to represent a forward reshaping function that maps the pixels of the input image to a second codeword representation; as well as An encoded bitstream is generated based on the input image and the forward reshaping function, wherein the reshaping parameters include: The first parameter determines the active maximum codeword block index for reshaping, wherein the active maximum codeword block index is less than or equal to a predetermined maximum codeword block index. The second parameter indicates the minimum codeword block index used in the reshaping process; Absolute differential codeword value, which is used for each active codeword block in the second codeword representation; The sign of the absolute difference codeword value is used for each active block in the second codeword representation; and One or more chromaticity residual conversion parameters based on luminance.

4. The method of claim 3, wherein the forward reshaping function comprises a piecewise linear function having linear segments derived from the reshaping parameters.

5. A method for transmitting a bit stream, the method comprising: Perform the method according to claim 3 to generate the bit stream; as well as Transmit the bit stream.

Images