Adaptive sample cutting

Abstract

A method of implementing sample clipping in a video coding system is provided. A video codec receives data to be encoded or decoded as a current block of a current image pixel of a video. A video codec signals or receives a first set of range definitions. A video codec encodes or decodes the current block by processing the received data in one or more codec stages, during which data samples generated by a first codec stage are limited by adaptive sample clipping within a first range of values defined by the first set of range definitions. The first set of range definitions may apply a shear function to data samples produced by the first codec stage to apply a maximum allowable value and a minimum allowable value to data samples produced by the first codec stage.

Description

[0001] Cross-referencing

[0002] This disclosure is part of a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 510,931, filed June 29, 2023. The contents of the above application are incorporated herein by reference. [Technical Field]

[0003] This disclosure generally relates to video encoding and decoding. In particular, this disclosure relates to a method for encoding pixel blocks through adaptive sample cropping. [Background Technology]

[0004] Unless otherwise stated herein, the methods described in this section are not prior art to the claims listed below, and are not admitted as prior art by virtue of their inclusion in this section.

[0005] High-Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a coding architecture similar to Discrete Cosine Transform (DCT) with hybrid block-based motion compensation. The basic unit used for compression is called a coding unit (CU), which is a 2Nx2N pixel block. Each CU can be recursively divided into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0006] Versatile Video Coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11. The input video signal is predicted from the reconstructed (REC) signal in the coded image region. The prediction residual signal is processed by block transform. The transform coefficients are quantized and entropy-coded in the bitstream along with other auxiliary information. The reconstructed signal is generated from the predicted signal and the reconstructed residual signal by inverse transforming the dequantized transform coefficients. The reconstructed signal is further processed by in-loop filtering to remove coding artifacts. The decoded image is stored in the frame buffer to predict future images in the input video signal.

[0007] In VVC, the coded image is divided into non-overlapping block regions represented by relevant coding tree units (CTUs). Leaf nodes of the coding tree correspond to coding units (CUs). The coded image can be represented by multiple slices, each containing an integer number of CTUs. The CTUs within a slice are processed in raster scan order. Double-prediction (B) slices can be decoded using either intra-frame or inter-frame prediction, using at most two motion vectors and a reference index to predict the sample value for each block. Predictive (P) slices are decoded using either intra-frame or inter-frame prediction, using at most one motion vector and a reference index to predict the sample value for each block. Intra-frame (I) slices are decoded using only intra-frame prediction.

[0008] CTUs can be partitioned into one or more non-overlapping coding units (CUs) using quadtree (QT) and nested multi-type-tree (MTT) structures to accommodate various local motion and texture features. CUs can be further partitioned into smaller CUs using one of five partitioning types: quadtree partitioning, vertical binary tree partitioning, horizontal binary tree partitioning, vertical center-side ternary tree partitioning, and horizontal center-side ternary tree partitioning.

[0009] Each CU contains one or more prediction units (PUs). The prediction unit, along with the associated CU syntax, serves as the basic unit of signaling predictor information. The specified prediction process is used to predict the values ​​of associated pixel samples within the PU. Each CU may contain one or more transform units (TUs) to represent prediction residual blocks. A transform unit (TU) consists of a transform block (TB) for the luminance sample and two corresponding chrominance sample transform blocks, each TB corresponding to a residual block sample from one color component. Integer transforms are applied to the transform blocks. The level values ​​of the quantization coefficients are entropy-encoded in the bitstream along with other auxiliary information. The terms coding tree block (CTB), coding block (CB), prediction block (PB), and transform block (TB) are used to specify two-dimensional sample arrays of monochromatic components associated with the CTU, CU, PU, ​​and TU, respectively. Therefore, a CTU consists of one luminance CTB, two chrominance CTBs, and associated syntax elements. A similar relationship exists for CUs, PUs, and TUs.

[0010] For each inter-frame predicted CU, motion parameters include motion vectors, reference image indices, and reference image list usage indices, as well as additional information for inter-frame predicted sample generation. Motion parameters can be passed explicitly or implicitly. When a CU is encoded in skip mode, the CU is associated with a PU and has no significant residual coefficients, no encoded motion vector increments, or reference image indices. A merging mode is specified where the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as additional plans introduced in the VVC. The merging mode can be applied to any inter-frame predicted CU. An alternative to the merging mode is explicit transmission of motion parameters, where the motion vectors, the corresponding reference image indices for each reference image list, the reference image list usage flags, and other necessary information are explicitly passed in each CU. [Summary of the Invention]

[0011] The following summary is for illustrative purposes only and is not intended to be limiting in any way. That is, the following summary aims to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious techniques described herein. Selected embodiments will be further described in detail below. Therefore, the following summary is not intended to identify the essential features of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter.

[0012] Some embodiments of this disclosure provide a method for implementing sample clipping in a video coding system. A video encoder receives data for a current block to be encoded or decoded into current image pixels of a video. The video encoder signals or receives a first set of range definitions. The video encoder encodes or decodes the current block by processing the received data in one or more coding stages, during which data samples generated in the first coding stage are constrained to a first numerical range defined by the first set of range definitions through adaptive sample clipping. The first set of range definitions can apply a clipping function to the data samples generated in the first coding stage to impose maximum and minimum permissible values ​​on the data samples generated in the first coding stage.

[0013] In some embodiments, the maximum and minimum allowed values ​​apply to both the luminance and chrominance components. In some embodiments, the maximum and minimum allowed values ​​apply only to samples of the luminance component and not to samples of the chrominance component. In some embodiments, the data samples generated by the second encoding stage are also limited to a first numerical range defined by the first set of ranges. In other words, the same set of maximum and minimum values ​​applies to multiple stages.

[0014] The video encoder may also signal a second set of range definitions so that the data samples generated by the second encoding stage are restricted to a second numerical range defined by the second set of range definitions. In other words, different sets of maximum and minimum values ​​can be applied to different stages. In some embodiments, the first set of range definitions is one of multiple sets of range definitions signaled by the encoder, and the encoder selects the first set of range definitions from the multiple sets of range definitions to apply to the data samples generated by the first encoding stage.

[0015] In some embodiments, a first set of range definitions is used for luminance mapping with chroma scaling (LMCS), that is, to define a first range that maps luminance values ​​from the original domain to the reshaped domain. The first set of range definitions may include a maximum and minimum value for defining the first range, as well as a number of non-zero codewords, and the number of non-zero codewords may not be a power of two. A second set of range definitions can be used to define a second range that maps luminance values ​​from the original domain to the reshaped domain.

[0016] In some embodiments, LMCS is disabled when adaptive sample clipping is applied, and adaptive sample clipping is disabled when LMCS is applied. In some embodiments, when LMCS is enabled, adaptive sample clipping is used in the in-loop filtering stage (e.g., deblocking filtering, adaptive loop filtering, sample adaptive shifting). In some embodiments, when LMCS is disabled, adaptive sample clipping is applied to the prediction or reconstruction stage of video coding. [Attached Image Description]

[0017] The accompanying drawings are included to provide a further understanding of the present disclosure and form part of it. These drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. It will be understood that the drawings are not necessarily to scale, as some components may be shown out of proportion to their actual dimensions in order to clearly illustrate the concepts of the present disclosure.

[0018] Figure 1 The luminance mapping architecture with chroma scaling (LMCS) in the video decoder is shown.

[0019] Figure 2 This concept explains how to define the reshaping domain of LMCS using maximum and minimum values.

[0020] Figure 3 An example of a video encoder that may implement adaptive sample cropping is shown.

[0021] Figure 4 This conceptually explains the part of a video encoder that implements controlled cropping.

[0022] Figure 5This conceptually explains the process of performing adaptive sample cropping during video encoding.

[0023] Figure 6 An example of a video decoder that may implement adaptive sample cropping is shown.

[0024] Figure 7 This conceptually explains the part of the video decoder that implements controlled cropping.

[0025] Figure 8 This conceptually explains the process of performing adaptive sample clipping during video decoding.

[0026] Figure 9 This describes, conceptually, an electronic system that implements certain embodiments of the present disclosure.

Detailed Implementation Methods

[0027] In the following detailed description, numerous specific details are set forth by way of example to provide a thorough understanding of the related techniques. Any variations, derivatives, and / or extensions of the techniques described herein are within the scope of this disclosure. In some cases, to avoid unnecessarily obscuring aspects of the techniques disclosed herein, well-known methods, procedures, components, and / or circuits related to one or more example implementations disclosed herein may be described at a relatively high level.

[0028] I. Luminance Mapping with Chroma Scaling (LMCS)

[0029] Luma mapping with chroma scaling (LMCS) is an encoding tool that runs as a processing block before loop filters (SAO, DBF, etc.) in a video coding loop. LMCS has two main component functions: 1) intra-loop mapping of the luma component based on an adaptive piecewise linear model, and 2) luma-dependent chroma residual scaling for the chroma component.

[0030] Figure 1This diagram illustrates the Luminosity Mapping with Chroma Scaling (LMCS) architecture in a video decoder. Some blocks correspond to processes applied in the mapping (or reshaping) domain; these include inverse quantization, inverse transform, intra-luminance prediction, and the addition of luminance prediction with the luminance residual. Other blocks in the diagram show where processes are applied in the original (i.e., unmapped) domain; these include loop filters such as deblocking filters, ALF and SAO, motion-compensated prediction, intra-luminance prediction, the addition of luminance prediction with the luminance residual, and storing the decoded image as a reference image. Some blocks in the diagram are LMCS function blocks, including forward and reverse mapping of the luminance signal and luminance-dependent chroma scaling processes. Like most other tools in VVC, LMCS can be enabled / disabled at the sequence level using the SPS flag.

[0031] Intra-loop mapping of the luminance component adjusts the dynamic range of the input signal by reallocating codewords within the dynamic range, thereby improving compression efficiency. Luminance mapping utilizes the forward mapping function FwdMap and the corresponding reverse mapping function InvMap. The FwdMap function uses a piecewise linear model with 16 equal parts for signaling. The InvMap function does not require signaling and is derived from the FwdMap function.

[0032] a. Brightness mapping using a piecewise linear model

[0033] The luminance mapping model can be signaled in the Adaptive Parameter Set (APS) syntax structure, with `aps_params_type` set to 1 (LMCS_APS). A maximum of four LMCS APSs can be used in an encoded video sequence. Only one LMCS APS can be used for a single image. The luminance mapping model uses a piecewise linear model for signaling. The piecewise linear model divides the dynamic range of the input signal into 16 equal parts, and for each part, its linear mapping parameters are represented by the number of codewords allocated to that part. Taking a 10-bit input as an example, by default, each of the 16 parts will be allocated 64 codewords. The signaled number of codewords is used to calculate the scaling factor and adjust the mapping function for that part accordingly. At the slice level, an LMCS enable flag is signaled to indicate... Figure 1 The LMCS procedure shown is applied to the current chip. If the current chip has LMCS enabled, an aps_id is signaled in the chip header to identify the APS carrying the luminance mapping parameters.

[0034] Each i-th part of the FwdMap piecewise linear model, i = 0…15, is defined by two input pivot points InputPivot[] and two output (mapping) pivot points MappedPivot[]. InputPivot[] and MappedPivot[] are calculated as follows (assuming 10-bit video):

[0035] (1) OrgCW = 64

[0036] (2) For i = 0:16, InputPivot[i] = i * OrgCW

[0037] (3) For i = 0:16, MappedPivot[i] is calculated as follows:

[0038] MappedPivot[0] = 0;

[0039] for(i=0;i<16;i++)

[0040] MappedPivot[i+1]=MappedPivot[i]+SignaledCW[i]

[0041] Where OrgCW is the number of raw codewords (the same for all i), and SignaledCW[i] is the number of signaling codewords in the i-th part.

[0042] like Figure 1 As shown, for an inter-frame coded block, motion compensation prediction is performed in the mapping domain. In other words, motion compensation prediction block Y is calculated based on the reference signal in the DPB. pred Then, the FwdMap function is applied to map the brightness prediction blocks in the original domain to the mapped domain, Y. ′ pred =FwdMap(Y pred For intra-coded blocks, the FwdMap function is not applied because intra-prediction is performed in the mapping domain. In calculating the reconstructed block Y... r Then, the InvMap function is applied to convert the reconstructed brightness values ​​in the mapped domain back to the reconstructed brightness values ​​in the original domain. The InvMap function is applied to luma blocks coded intra-frame and inter-frame.

[0043] The brightness mapping process (forward and / or reverse mapping) can be implemented using lookup tables (LUTs) or by on-the-fly computation. If using LUTs, FwdMapLUT and InvMapLUT can be pre-computed and pre-stored at the tile group level, and forward and reverse mapping can be simply implemented as FwdMap(Y pred =FwdMapLUT[Y pred ] and InvMap(Y r =InvMapLUT[Y rAlternatively, on-the-fly computation can be used. Take the forward mapping function FwdMap as an example. To determine which part a luminance sample belongs to, the sample value is right-shifted by 6 bits (corresponding to 16 equal parts). Then, the linear model parameters for that part are retrieved and applied on the fly to calculate the mapped luminance value. Let i be the part index, a1 and a2 be InputPivot[i] and InputPivot[i+1], and b1 and b2 be MappedPivot[i] and MappedPivot[i+1]. The FwdMap function is calculated as follows:

[0044] FwdMap(Y pred )=((b2–b1) / (a2–a1))*(Y pred –a1)+b1

[0045] The InvMap function can be computed on the fly in a similar manner. Typically, the parts in the mapping domain are not of equal size, so the most straightforward reverse mapping process requires comparison to determine which part the current sample value belongs to. This comparison increases the complexity of the decoder. For this reason, a bitstream constraint is imposed on the value of the output pivot point MappedPivot[i]:

[0046] Suppose the range of the mapping domain ([0, 1023] for a 10-bit video) is divided into 32 equal parts. If MappedPivot[i] is not a multiple of 32, then MappedPivot[i+1] and MappedPivot[i] cannot belong to the same part of the 32 equal-sized parts, i.e., MappedPivot[i+1] >> (BitDepth). Y -5) should not be equal to MappedPivot[i] >> (BitDepth) Y -5).

[0047] By using such bitstream constraints, the InvMap function can also be performed by simply right-shifting by 5 bits (corresponding to 32 equal-sized portions) to determine the portion to which a sample value belongs.

[0048] b. Luminance-dependent chromaticity residual scaling

[0049] Chroma residual scaling is used to compensate for the interaction between the luminance signal and its corresponding chrominance signal. Whether chroma residual scaling is enabled can be determined by chip-level signaling. If luminance mapping is enabled, an additional flag is signaled to indicate whether luminance-dependent chroma residual scaling is enabled. In some embodiments, luminance-dependent chroma residual scaling is disabled when luminance mapping is not used. Furthermore, luminance-dependent chroma residual scaling is always disabled for chrominance blocks with an area less than or equal to 4.

[0050] Chromaticity residual scaling depends on the average of the reconstructed adjacent luminance samples from the top and / or left of the current VPDU. If the current CU is inter 128x128, inter 128x64, or inter 64x128, the chromaticity residual scaling factor derived from the CU associated with the first VPDU is used for all chromaticity transform blocks within that CU. ScaleInv The value is calculated through the following steps: (avgYr represents the average value of the reconstructed adjacent brightness samples, such as...) Figure 1 As shown.

[0051] (1) Find the index Y of the piecewise linear model to which avgYr belongs using the InvMap function. idx .

[0052] (2)C ScaleInv =cScaleInv[Y Idx ], where cScaleInv[] is a 16-segment LUT pre-calculated based on the value of SignaledCW[i] and the offset value of the signaling for the chroma residual scaling process in APS.

[0053] Unlike brightness mapping performed on a sample basis, C ScaleInv It is a constant value for the entire chroma block. (Using C...) ScaleInv The application of chroma residual scaling is as follows:

[0054] Encoder end: C ResScale =C Res *C Scale =C Res / C ScaleInv

[0055] Decoder end: C Res =C ResScale / C Scale =C ResScale *C ScaleInv

[0056] c. LMCS encoding process

[0057] In some embodiments, LMCS data is signaled / encoded in the bitstream. The syntax elements of the encoded LMCS data include the minimum bin index, the delta maximum bin index, the codeword count variation within each interval (or range), and the chroma scaling correction offset. The LMCS data syntax table in APS is shown below:

[0058]

[0059] II. Controlled shearing / Adaptive sample shearing

[0060] Video systems can perform certain functions by adapting to the statistical characteristics of the input signal. This adaptability is evident in motion interpolation designs, which incorporate a set of filters to accommodate different signal characteristics, and in adaptive loop filter designs, which allow the encoder to design and transmit loop filters to the decoder. This additional adaptability provides higher coding efficiency for many sequences. Unfortunately, it also leads to an expansion of the dynamic range. For image sequences transmitted in the full range, such as [0, 255] in 8 bits, this dynamic range expansion is not a problem. Current decoding processes limit intermediate values ​​to the full range, so any dynamic range expansion is implicitly handled.

[0061] Problems arise when the input data does not use the full range. A very common scenario is that image data is stored with "broadcast-valid values". For example, the luminance signal might initially be in the range [16,235], while the chrominance signal might initially be in the range [16,240]. After being encoded by a video coding system, the reconstructed pixel values ​​are no longer within these ranges and may even exceed the range of the input values.

[0062] In some embodiments, the video coding system performs "controlled clipping" or "adaptive sample clipping." Specifically, the encoder transmits a known range of luma and chroma values ​​to the decoder, which uses these transmitted range values ​​as clipping points (e.g., as maximum and minimum values) to limit data samples to a range of certain defined allowable values. Clipping points can be operated at various stages of the encoding process, such as after prediction, after reconstruction, after deblocking, and after the adaptive loop filter process. These will be discussed below via reference. Figure 4 and Figure 7 Describe the video encoding stages that can be controlled by cutting.

[0063] In some embodiments, control clipping operates on the decoder based on the range (minimum and maximum values) of the original pixels in the current image. If a pixel value exceeds a specified range after reconstruction, the pixel value will be clipped to the minimum or maximum value. The minimum and maximum values ​​can be predicted before transmission.

[0064] In some embodiments, the video encoder uses a 4-stage control clipping mechanism, limiting the original value to the range [min_value, max_value] using the Clip3 function. In some embodiments, the Clip3 function can be defined according to the following equation:

[0065] Clipped_value=Clip3(min_value,max_value,orig_value)

[0066] Make Clipped_value =

[0067] if min_value ≤ orig_value ≤ max_value

[0068] If min_value is less than orig_value, then...

[0069] max_value if orig_value ≥ max_value

[0070] In some embodiments, the corresponding video encoder and decoder implement controlled clipping at various encoding and decoding stages. For example, in some embodiments, the video encoder and decoder implement controlled clipping after prediction, reconstruction, deblocking, and ALF. This will be explained below by reference. Figure 4 and Figure 7 Describes the controlled clipping implemented during the encoding phase of a video encoder.

[0071] In some embodiments, the minimum and maximum values ​​for controlled cropping can be defined at the Image Parameter Set (PPS) level or at the slice level. When using PPS-level adaptation, the minimum and maximum values ​​can be predefined either by signaling in the PPS or by setting the broadcast valid flag to 1. Slice-level adaptation can be enabled for both luma and chroma. When slice-level adaptation is enabled, the minimum and maximum values ​​passed in the PPS are used to predict those values ​​in the slice header. The syntax and semantics of this method are shown in the following syntax table:

[0072] Controlled cut syntax in PPS:

[0073]

[0074] Controlled cut syntax in the opening sequence:

[0075]

[0076] A controlled_clipping_flag value of 1 indicates that controlled clipping (CC) is enabled; a value of 0 indicates that controlled clipping is disabled.

[0077] A controlled_clipping_broadcast_legal_flag value of 1 indicates controlled clipping using predefined minimum and maximum pixel values; a value of 0 indicates controlled clipping using the transmitted minimum and maximum pixel values.

[0078] controlled_clipping_minY defines the minimum pixel value of the Y channel. When controlled_clipping_broadcast_legal_flag is equal to 1, controlled_clipping_minY shall be (16 << bit_depth_luma_minus8). When controlled_clipping_flag is equal to 0, controlled_clipping_minY shall be 0.

[0079] controlled_clipping_maxY defines the maximum pixel value of the Y channel. When controlled_clipping_broadcast_legal_flag is equal to 1, controlled_clipping_maxY shall be (235 << bit_depth_luma_minus8). When controlled_clipping_flag is equal to 0, controlled_clipping_maxY shall be (255 << bit_depth_luma_minus8).

[0080] controlled_clipping_minCr defines the minimum pixel value of the Cr channel. When controlled_clipping_broadcast_legal_flag is equal to 1, controlled_clipping_minCr shall be (16 << bit_depth_chroma_minus8). When controlled_clipping_flag is equal to 0, controlled_clipping_minCr shall be 0.

[0081] controlled_clipping_maxCr defines the maximum pixel value of the Cr channel. When controlled_clipping_broadcast_legal_flag is equal to 1, controlled_clipping_maxCr shall be (240 << bit_depth_chroma_minus8). When controlled_clipping_flag is equal to 0, controlled_clipping_maxCr shall be (255 << bit_depth_chroma_minus8).

[0082] The `controlled_clipping_sameC_data_flag` being equal to 1 indicates that the controlled clipping parameters in the PPS are the same for both chroma channels.

[0083] `controlled_clipping_minCb` defines the minimum pixel value for the Cb channel. When `controlled_clipping_broadcast_legal_flag` is equal to 1, `controlled_clipping_minCb` shall be (16 << bit_depth_chroma_minus8). When `controlled_clipping_sameC_data_flag` is equal to 1, `controlled_clipping_minCb` shall be `controlled_clipping_minCr`. When `controlled_clipping_flag` is equal to 0, `controlled_clipping_minCb` shall be 0.

[0084] `controlled_clipping_maxCb` defines the maximum pixel value for the Cb channel. When `controlled_clipping_broadcast_legal_flag` is equal to 1, `controlled_clipping_maxCb` shall be (240 << bit_depth_chroma_minus8). When `controlled_clipping_sameC_data_flag` is equal to 1, `controlled_clipping_maxCb` shall be `controlled_clipping_maxCr`. When `controlled_clipping_flag` is equal to 0, `controlled_clipping_maxCb` shall be (255 << bit_depth_chroma_minus8).

[0085] The `controlled_clipping_slice_controlY_flag` being equal to 1 indicates that slice-level luminance controlled clipping is enabled; being equal to 0 indicates that PPS-level luminance controlled clipping is enabled.

[0086] The `controlled_clipping_slice_controlC_flag` being equal to 1 indicates that slice-level chroma controlled clipping is enabled; being equal to 0 indicates that PPS-level chroma controlled clipping is enabled.

[0087] controlled_clipping_minY_slice_delta is used to derive controlled_clipping_minY_slice as follows.

[0088] controlled_clipping_minY_slice=controlled_clipping_minY+controlled_clipping_minY_slice_delta

[0089] When controlled_clipping_slice_controlY_flag equals 0 or controlled_clipping_flag equals 0, controlled_clipping_minY_slice should be controlled_clipping_minY. In the current slice, Y channel pixel values ​​smaller than controlled_clipping_minY_slice should be replaced with controlled_clipping_minY_slice.

[0090] The controlled_clipping_maxY_slice_delta is used to derive controlled_clipping_maxY_slice as follows.

[0091] controlled_clipping_maxY_slice=controlled_clipping_maxY+controlled_clipping_maxY_slice_delta

[0092] When controlled_clipping_slice_controlY_flag equals 0 or controlled_clipping_flag equals 0, controlled_clipping_maxY_slice should be controlled_clipping_maxY. In the current slice, channel Y pixel values ​​greater than controlled_clipping_maxY_slice should be replaced with controlled_clipping_maxY_slice.

[0093] controlled_clipping_minCr_slice_delta is used to derive controlled_clipping_minCr_slice as follows.

[0094] controlled_clipping_minCr_slice=controlled_clipping_minCr+controlled_clipping_minCr_slice_delta

[0095] When controlled_clipping_slice_controlC_flag equals 0 or controlled_clipping_flag equals 0, controlled_clipping_minCr_slice should be controlled_clipping_minCr. In the current slice, channel Cr pixel values ​​smaller than controlled_clipping_minCr_slice should be replaced with controlled_clipping_minCr_slice.

[0096] The controlled_clipping_maxCr_slice_delta is used to derive controlled_clipping_maxCr_slice as follows.

[0097] controlled_clipping_maxCr_slice=controlled_clipping_maxCr+controlled_clipping_maxCr_slice_delta

[0098] When controlled_clipping_slice_controlC_flag equals 0 or controlled_clipping_flag equals 0, controlled_clipping_maxCr_slice should be controlled_clipping_maxCr. In the current slice, channel Cr pixel values ​​greater than controlled_clipping_maxCr_slice should be replaced with controlled_clipping_maxCr_slice.

[0099] controlled_clipping_minCb_slice_delta is used to derive controlled_clipping_minCb_slice as follows.

[0100] controlled_clipping_minCb_slice=controlled_clipping_minCb+controlled_clipping_minCb_slice_delta

[0101] When controlled_clipping_slice_controlC_flag equals 0 or controlled_clipping_flag equals 0, controlled_clipping_minCb_slice should be controlled_clipping_minCb. In the current slice, channel Cb pixel values ​​smaller than controlled_clipping_minCb_slice should be replaced with controlled_clipping_minCb_slice.

[0102] The controlled_clipping_maxCb_slice_delta is used to derive controlled_clipping_maxCb_slice as follows.

[0103] controlled_clipping_maxCb_slice=controlled_clipping_maxCb+controlled_clipping_maxCb_slice_delta

[0104] When controlled_clipping_slice_controlC_flag equals 0 or controlled_clipping_flag equals 0, controlled_clipping_maxCb_slice should be controlled_clipping_maxCb. In the current slice, channel Cb pixel values ​​greater than controlled_clipping_maxCb_slice should be replaced with controlled_clipping_maxCb_slice.

[0105] Empirical observations suggest that this controlled shearing / adaptive sample shearing coding tool can lead to an expansion of the dynamic range of the reconstructed frame. This is problematic for sequences that do not utilize the entire input dynamic range, as the encoding process may produce pixel values ​​beyond the source dynamic range. To address this issue, in some embodiments, the shearing point in the TM is modified to account for the dynamic range of the input source (when controlled shearing is applied after prediction, reconstruction, deblocking, and adaptive loop filtering (ALF)). This provides higher coding efficiency for most sequences with only a small increase in complexity. (Empirically, the BD rate is reduced by 0.6% for the efficient random access configuration and by 0.4% for the efficient low-latency configuration.)

[0106] In some embodiments, minimum and maximum values ​​(for defining ranges) are defined only for specific color components, signaled by the encoder and decoded by the decoder. In one embodiment, minimum and maximum values ​​are defined only for the Y (luminance) range. In some embodiments, minimum and maximum values ​​are defined for Y (luminance) and flagged by the encoder signaling and decoded by the decoder (at the Adaptive Parameter Set (APS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Title Header (SH), or Picture Header (PH) level) to identify whether similar clipping is applied to the Cb / Cr (two chromaticity) components. In some embodiments, two components may share a range. In one embodiment, the encoder signales a separate flag, decoded by the decoder (at the APS, SPS, PPS, SH, or PH level), to indicate that all three components share the same range.

[0107] In some embodiments, adaptive sample clipping can be selectively applied to different stages of the encoding and decoding process. Specifically, the decoding process can apply adaptive sample clipping in the prediction stage (e.g., intra-frame prediction generation, inter-frame prediction generation), inverse quantization stage, inverse transform stage, in-loop filtering stage (e.g., deblocking, Sample Adaptive Offset (SAO), Adaptive Loop Filter (ALF)) and / or other decoding stages.

[0108] In some embodiments, multiple sets of minimum / maximum values ​​are defined and decoded by the encoder signaling and decoder, and each set of minimum / maximum clipping values ​​can be applied to one or more stages. For example, one set of minimum / maximum values ​​can be used for adaptive sample clipping in the ALF stage, while another set of minimum / maximum values ​​is used for adaptive sample clipping in the deblocking (DF) or SAO stage. In some embodiments, adaptive sample clipping can be applied to multiple stages with different minimum / maximum values, depending on the stage / tool ​​applied. In some embodiments, the minimum / maximum values ​​of adaptive sample clipping are used for one or more loop filtering stages (e.g., ALF, SAO, deblocking, etc.) with different minimum / maximum values, depending on the stage / tool.

[0109] In some embodiments, the in-loop filtering tool is modified to account for different minimum / maximum values ​​for adaptive sample clipping. In some embodiments, the encoder may signal additional flags, decoded by the decoder (at the APS, SPS, PPS, SH, or PH level), indicating whether one or more ranges are used. If a range is decided to be used (at the APS, SPS, PPS, SH, or PH level), any one of the available sets of minimum / maximum values ​​can be used. In some embodiments, the decision of whether to use one or another of the available sets of minimum / maximum values ​​at a certain stage depends on the tool. For example, in some embodiments, the ALF stage uses the minimum / maximum values ​​of set 1, while the DF or SAO stage uses the minimum / maximum values ​​of set 2.

[0110] In some embodiments, all tools use only one of multiple sets of minimum / maximum values ​​(called the global set) across all stages. This means the minimum / maximum range is defined in the encoder and then used globally throughout all encoding and decoding processes. In this case, the encoder signals the defined minimum / maximum value, which is then decoded by the decoder, and the decoded range is used throughout all decoding processes, whenever a clipping operation is needed to maintain the defined range of the decoded sample. In some embodiments, the encoder signals a separate flag, which is decoded by the decoder (at the APS, SPS, PPS, SH, or PH level), to instruct all tools to use one of multiple sets of minimum / maximum values ​​across all stages. In this case, a so-called "global" clipping range is used, e.g., for the entire sequence / image / slice, etc.

[0111] In some embodiments, adaptive sample clipping can be extended to the slice / tile / coding tree unit (CTU) row / coding unit (CU) / block level. In some embodiments, on / off control can be performed at any of the above levels. In some embodiments, the encoder can encode multiple sets of minimum / maximum values, and the decoder can decode / define and apply them at the slice / tile / CTU (row) / CU / block level. In this case, any method described in the present invention can be correspondingly extended.

[0112] In some methods, adaptive sample clipping is regarded as a mode of LMCS. When LMCS is enabled, an additional syntax element is signaled to indicate whether adaptive sample clipping is enabled. When adaptive sample clipping is enabled, all the above methods in the present invention can be used for sample clipping.

[0113] In some embodiments, the minimum / maximum values defined for adaptive sample clipping can be used in the LMCS stage. In some embodiments, the minimum and maximum values defined for adaptive sample clipping can replace the minimum and maximum values used in LMCS, so that when an operation that splits the luminance mapping into different ranges is applied, the range considered in LMCS is the range between the defined minimum and maximum values, rather than the range from 0 to ((1<<BitDepth)–1). In some embodiments, the number of ranges can be adjusted according to the minimum / maximum range. In some embodiments, the signaled minimum and maximum values replace lmcs_min_bin_idx and lmcs_delta_max_bin_idx, which are used to define the range and the number of non-zero codewords in the original and / or reshaped domain. (In other words, the signaled minimum and maximum values are used to calculate MappedPivot[i] and SignaledCW[i] for different parts / ranges of the reshaped domain, as described in Section I.a above.) In some embodiments, the minimum and maximum values are used to define lmcs_min_bin_idx and lmcs_delta_max_bin_idx (so no signaling is required for these syntax elements).

[0114] Figure 2This conceptually illustrates the use of maximum and minimum values ​​to define a reshaping domain for luma mapping with chroma scaling (LMCS). The figure shows the reshaping function that maps luma values ​​from the original domain to the reshaping domain. The reshaping function is defined by an input pivot point (InputPivot[i]) in the original domain and a mapping pivot point (MappedPivot[i]) in the reshaping domain. The input pivot is uniformly distributed in the original domain. The distribution of the mapping pivot adapts to different numbers of codewords within different ranges in the reshaping domain. In some embodiments, the distribution of the mapping pivot point may be defined by one or more sets of maximum and minimum values ​​described above for adaptive sample clipping. For example, in some embodiments, each set of maximum and minimum values ​​is used to define a range in the reshaping domain by, for example, specifying a mapping pivot point and the number of codewords within the range.

[0115] In some embodiments, the encoder may signal an additional delta syntax element and have it decoded by the decoder (e.g., at the APS, SPS, PPS, SH, PH, or APS level) to enable any minimum / maximum value for any number of codewords in the LMCS, not just multiples of 16. This would allow for a combination of (LMCS off + cut-on) and LMCS on scenarios. In some embodiments, the encoder may signal an additional flag and have it decoded by the decoder (at the APS, SPS, PPS, SH, PH, or APS level) to indicate whether one of multiple sets of minimum / maximum values ​​is used in the LMCS.

[0116] In some embodiments, LMCS and adaptive sample clipping are mutually exclusive. For example, in some embodiments, LMCS is disabled when adaptive sample clipping is applied. In some embodiments, adaptive sample clipping is disabled when LMCS is applied. In some embodiments, adaptive sample clipping is used in the in-loop filtering stage when LMCS is enabled. In one embodiment, when LMCS is disabled, adaptive sample clipping is further applied to the prediction and / or reconstruction stages.

[0117] In some embodiments, a separate syntax element is used to indicate whether LMCS or adaptive sample clipping is applied. In some embodiments, the encoder encodes at the encoder and the decoder decodes the separate syntax element, for example, at the APS, SPS, PPS, SH, PH, or APS level. In some embodiments, adaptive sample clipping is applied whenever LMCS is disabled / not applied, for example, at the APS, SPS, PPS, SH, or PH level.

[0118] In some embodiments, the range of the original pixels (minimum and maximum values) is signaled in a combination of SH, PH, APS, PPS, or LMCS syntax. In some embodiments, the encoder may signal the difference between the original minimum / maximum value and a predefined value, which is then decoded by the decoder. In this case, at the decoder end, to reconstruct the original values ​​of the minimum and maximum values, the predefined value needs to be known and added to the signaling value. In some embodiments, two separate predefined values ​​are used—one for the minimum value and another for the maximum value. In some embodiments, the encoder and decoder may use a set of predefined values, and the encoder may signal additional syntax elements, which are then decoded by the decoder, to identify which predefined value was used.

[0119] In some embodiments, predefined minimum and / or maximum values ​​are set for all color components. In some embodiments, predefined values ​​are set for the luminance component and the two chrominance components respectively. In some embodiments, separate minimum and maximum values ​​are set for each color component. In some embodiments, separate flags are signaled to indicate whether the minimum and / or maximum values ​​are shared by all color components (or the two chrominance components).

[0120] In some embodiments, predefined values ​​are used in SPS, PPS, SH, PH, or APS level signaling. In some embodiments, separate syntax elements (at SPS, PPS, SH, PH, or APS levels) are used to indicate whether separate signaling predefined values ​​are used or default values ​​are employed. In some embodiments, the encoder can enable / disable adaptive sample clipping at the slice (PH, image) level signaling / disable flags and decode them at the decoder. In some embodiments, the minimum and maximum values ​​of the current slice (image) are signaled only when the SH (PH, image) level flag is equal to 1 (not equal to 0).

[0121] In some embodiments, decisions can be made at the slice (or image head or image) level based on certain criteria, thus eliminating the need for signaling at the slice (or image head or image) level and allowing decisions to be made at the encoder and decoder without any additional signaling. In some embodiments, the encoder can enable / disable the SPS level flag for adaptive sample clipping via sequence-level signaling and decode at the decoder. In some embodiments, the slice (or image head or image) level flag is signaled only when the sequence-level flag is equal to 1 (not equal to 0).

[0122] In some embodiments, the minimum and maximum values ​​are defined using samples (i.e., the original unfiltered samples) only before Motion Compensated Temporal Filtering (MCTF). In other embodiments, samples after MCTF are used for adaptive cropping, and the minimum / maximum values ​​can be defined at the encoder and signaled by the encoder and decoded by the decoder. Other methods can also be used to define multiple sets of minimum / maximum ranges per frame.

[0123] III. Example Video Encoder

[0124] Figure 3 This example illustrates a video encoder 300 capable of adaptive sample cropping. As shown, the video encoder 300 receives an input video signal from a video source 305 and encodes it into a bitstream 395. The video encoder 300 has multiple components or modules for encoding the signal from the video source 305, including at least some modules selected from the following: a Transform (T) module 310, a Quantization (Q) module 311, an Inverse Quantization (IQ) module 314, an Inverse Transform (IT) module 315, an intra-frame estimation module 320, an intra-frame prediction module 325, a motion compensation module 330, a motion estimation module 335, an in-loop filter 345, a reconstructed image buffer 350, an MV buffer 365, an MV prediction module 375, and an entropy encoder 390. The motion compensation module 330 and the motion estimation module 335 are part of the inter-frame prediction module 340.

[0125] In some embodiments, modules 310 to 390 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 310 to 390 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 310 to 390 are illustrated as independent modules, some of these modules may be combined into a single module.

[0126] Video source 305 provides an uncompressed raw video signal, which represents the pixel data for each video frame. Subtractor 308 calculates the difference between the raw video pixel data from video source 305 and the predicted pixel data 313 from motion compensation module 330 or intra-frame prediction module 325, as the prediction residual 309. Transform module 310 converts the difference (or residual pixel data or residual signal 308) into transform coefficients (e.g., by performing a discrete cosine transform, or DCT for short). Quantization module 311 quantizes the transform coefficients into quantized data (or quantization coefficients) 312, which are then encoded into a bitstream 395 by entropy encoder 390.

[0127] The inverse quantization module 314 inverse-quantizes the quantized data (or quantization coefficients) 312 to obtain transform coefficients, and the inverse transform module 315 performs an inverse transform on the transform coefficients to generate a reconstruction residual 319. The reconstruction residual 319 is added to the predicted pixel data 313 to generate reconstructed pixel data 317. In some embodiments, the reconstructed pixel data 317 is temporarily stored in a line buffer (not shown) for intra-frame prediction and spatial MV prediction. The reconstructed pixels are filtered by an in-loop filter 345 and stored in a reconstructed image buffer 350. In some embodiments, the reconstructed image buffer 350 is external memory to the video encoder 300. In some embodiments, the reconstructed image buffer 350 is internal memory to the video encoder 300.

[0128] Intra-frame estimation module 320 performs intra-frame prediction based on reconstructed pixel data 317 to generate intra-frame prediction data. The intra-frame prediction data is provided to entropy encoder 390 for encoding into a bitstream 395. The intra-frame prediction data is also used by intra-frame prediction module 325 to generate predicted pixel data 313.

[0129] The motion estimation module 335 performs inter-frame prediction on reference pixel data of previously decoded frames stored in the reconstructed image buffer 350 by generating MVs. These MVs are provided to the motion compensation module 330 to generate predicted pixel data.

[0130] The video encoder 300 does not encode the complete actual MV in the bitstream. Instead, it uses MV prediction to generate a predicted MV and encodes the difference between the MV used for motion compensation and the predicted MV as residual motion data and stores it in the bitstream 395.

[0131] The MV prediction module 375 generates a predicted MV based on a reference MV generated for encoding the previous video frame (i.e., a motion-compensated MV used to perform motion compensation). The MV prediction module 375 retrieves the reference MV of the previous video frame from the MV buffer 365. The video encoder 300 stores the MV generated for the current video frame in the MV buffer 365 as a reference MV for generating the predicted MV.

[0132] The MV prediction module 375 creates a predicted MV using a reference MV. The predicted MV can be calculated through spatial MV prediction or temporal MV prediction. The difference (residual motion data) between the predicted MV and the motion-compensated MV (MC MV) of the current frame is encoded into a bitstream 395 by the entropy encoder 390.

[0133] The entropy encoder 390 encodes various parameters and data into a bitstream 395 using entropy coding techniques (such as context-adaptive binary arithmetic coding, CABAC, or Huffman coding). The entropy encoder 390 encodes various header elements, flags, quantization transform coefficients 312, and residual motion data as syntax elements into a bitstream 395. The bitstream 395 is then stored in a storage device or transmitted to the decoder via a communication medium such as a network.

[0134] The in-loop filter 345 performs filtering or smoothing operations on the reconstructed pixel data 317 to reduce coded artifacts, particularly at pixel block boundaries. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 345 include deblocking filtering (DBF), sample adaptive offset (SAO), and / or adaptive loop filtering (ALF). In some embodiments, luminance mapping with chroma scaling (LMCS) is performed before the loop filtering.

[0135] Figure 4 The controlled cropping portion of the video encoder 300 is conceptually illustrated. As shown in the figure, the encoder 300 implements controlled cropping in four stages: the post-prediction stage (stage 410, after the inter-frame prediction module 340 and the intra-frame prediction module 325), the post-reconstruction stage (stage 420, after the reconstructed pixel data 317), the post-deblocking filtering stage (stage 427, after the deblock (DB) filter 425), the post-SAO stage (stage 430, after the SAO 428), and the post-ALF stage (stage 440, after the ALF 435; ALF 435, SAO 428, and the deblocking filter 425 are part of the in-loop filter 345).

[0136] During prediction, the prediction reference buffers of the intra-frame prediction module 325 and the inter-frame prediction module 340 are used to store the predicted value 313 and generate a residual 309 with the original sequence. After prediction, controlled shearing can be applied to the generated prediction 313 in the post-prediction stage 410, which is used to generate the residual 309. The controlled shearing applied in the post-prediction stage 410 reduces the error level of the reconstructed residual 319.

[0137] Before reconstruction, the residual 309 is transformed and quantized into quantization coefficients 312 for transmission. The quantization coefficients 312 are then dequantized and inversely transformed into the reconstructed residual 319. This extends the dynamic range of the residual and also alters the dynamic range of the reconstructed pixel values. If the range is known in advance, the dynamic range of the reconstructed pixel values ​​can be limited by controlled clipping in the post-reconstruction stage 420. This reduces pixel errors when the reconstructed pixel values ​​exceed the limit.

[0138] Deblocking filtering and ALF alter the reconstructed pixel values ​​through filtering. The dynamic range of the filtered values ​​may change. Controlled cropping in post-deblocking filtering stage 430 and post-ALF stage 440 can also limit pixel values ​​to the correct range to minimize the expanded dynamic range.

[0139] In some embodiments, the maximum and minimum values ​​(or other types of range definitions) for constraining the sample values ​​at each controlled shearing phase 410-440 are provided to the entropy encoder 390 and signaled as syntax elements in the bitstream 395. In some embodiments, the maximum and minimum values ​​for LMCS operations are also provided to the entropy encoder 390 and signaled as syntax elements in the bitstream 395.

[0140] Figure 5 A process 500 for performing adaptive sample cropping during video encoding is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing encoder 300 execute process 500 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing encoder 300 executes process 500.

[0141] The encoder receives (at block 510) the data of the current block to be encoded into the current image pixels of the video.

[0142] Encoder signaling (at block 520) defines the first set of ranges. This first set of ranges can be enabled to apply to data samples of any other higher-level entity containing the current block within the same slice, the same brick, the same coding tree unit (CTU) row, the same coding unit, or the video coding hierarchy. In some embodiments, the first set of ranges (along with other flags related to adaptive sample clipping, as described in Section II above) can be signaled in the slice header, image header, adaptive parameter set (APS), and / or image parameter set (PPS).

[0143] The encoder encodes the current block (at block 530) by processing the received data in one or more encoding stages. In some embodiments, the encoding stages that perform adaptive sample clipping include post-prediction, post-reconstruction, post-deblocking artifacts, and post-ALF.

[0144] The encoder constrains (at block 540) the data samples generated by the first encoding stage to a first range defined by a first set of range definitions. The first set of range definitions can apply a shearing function to the data samples generated by the first encoding stage; specifically, it imposes maximum and minimum permissible values ​​on the data samples generated by the first encoding stage. In some embodiments, the maximum and minimum permissible values ​​apply to both the luma and chroma components. In some embodiments, the maximum and minimum permissible values ​​apply only to samples of the luma component and not to samples of the chroma component. In some embodiments, the data samples generated by the second encoding stage are also constrained to a first numerical range defined by the first set of range definitions. In other words, the same set of maximum and minimum values ​​is applied to multiple stages.

[0145] The video encoder can also signal a second set of range definitions, such that the data samples generated by the second encoding stage are constrained within a second numerical range defined by the second set of range definitions. In other words, different maximum and minimum values ​​can be applied to different stages. In some embodiments, the first set of range definitions is one of a set of multiple range definitions signaled by the encoder, and the encoder selects the first set of range definitions from the multiple sets of range definitions to apply to the data samples generated by the first encoding stage.

[0146] In some embodiments, a first set of range definitions is used for luminance mapping with chroma scaling (LMCS), that is, to define a first range that maps luminance values ​​from the original domain to the reshaped domain. The first set of range definitions may include a maximum and minimum value for defining the first range, as well as a number of non-zero codewords, and the number of non-zero codewords may not be a power of two. A second set of range definitions may be used to define a second range that maps luminance values ​​from the original domain to the reshaped domain.

[0147] In some embodiments, LMCS is disabled when adaptive sample cropping is applied; conversely, adaptive sample cropping is disabled when LMCS is applied. In some embodiments, when LMCS is enabled, adaptive sample cropping is used in the in-loop filtering stage (e.g., DBF, ALF, SAO). In some embodiments, when LMCS is disabled, adaptive sample cropping is applied to the prediction or reconstruction stage of video coding.

[0148] IV. Example Video Decoder

[0149] In some embodiments, the encoder may signal (or generate) one or more syntax elements in the bitstream, such that the decoder can parse the one or more syntax elements from the bitstream.

[0150] Figure 6An example video decoder 600 is illustrated, which can implement adaptive sample cropping. As shown, the video decoder 600 is an image or video decoding circuit that receives a bitstream 695 and decodes the contents of the bitstream into pixel data for video frames used for display. The video decoder 600 has several components or modules for decoding the bitstream 695, including some selected from inverse quantization module 611, inverse transform module 610, intra-frame prediction module 625, motion compensation module 630, in-loop filter 645, decoded image buffer 650, MV buffer 665, MV prediction module 675, and resolver 690. Motion compensation module 630 is part of inter-frame prediction module 640.

[0151] In some embodiments, modules 610-690 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 610-690 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 610-690 are shown as separate modules, some modules may be combined into a single module.

[0152] Parser 690 (or entropy decoder) receives bitstream 695 and performs preliminary parsing according to the syntax defined by the video coding or image coding standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 612. Parser 690 parses the various syntax elements using entropy coding techniques (e.g., context-adaptive binary arithmetic coding (CABAC) or Huffman coding).

[0153] The inverse quantization module 611 performs inverse quantization on the quantized data (or quantization coefficients) 612 to obtain transform coefficients, and the inverse transform module 610 performs inverse transform on the transform coefficients 616 to generate the reconstructed residual signal 619. The reconstructed residual signal 619 is added to the predicted pixel data 613 from the intra-frame prediction module 625 or the motion compensation module 630 to generate the decoded pixel data 617. The decoded pixel data is filtered by the in-loop filter 645 and stored in the decoded image buffer 650. In some embodiments, the decoded image buffer 650 is an external memory of the video decoder 600. In some embodiments, the decoded image buffer 650 is an internal memory of the video decoder 600.

[0154] Intra-frame prediction module 625 receives intra-frame prediction data from bitstream 695 and generates predicted pixel data 613 from decoded pixel data 617 stored in decoded image buffer 650 based on the data. In some embodiments, decoded pixel data 617 is also stored in a line buffer (not shown) for intra-frame image prediction and spatial MV prediction.

[0155] In some embodiments, the contents of the decoded image buffer 650 are used for display. The display device 605 may directly retrieve content from the decoded image buffer 650 for display, or retrieve the contents of the decoded image buffer into a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded image buffer 650 via pixel transfer.

[0156] The motion compensation module 630 generates predicted pixel data 613 from the decoded pixel data 617 stored in the decoded image buffer 650 based on the motion-compensated motion vectors (MC MVs). These motion-compensated motion vectors are decoded by adding the residual motion data received from the bitstream 695 to the predicted motion vectors received from the motion vector prediction module 675.

[0157] The motion vector prediction module 675 generates predicted motion vectors based on reference motion vectors generated for decoding previous video frames, such as motion-compensated motion vectors used for performing motion compensation. The motion vector prediction module 675 retrieves the reference motion vectors of the previous video frames from the motion vector buffer 665. The video decoder 600 stores the motion-compensated motion vectors generated for decoding the current video frame in the motion vector buffer 665 as reference motion vectors for generating the predicted motion vectors.

[0158] The in-loop filter 645 performs filtering or smoothing operations on the decoded pixel data 617 to reduce coding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 645 include a deblocking filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF). In some embodiments, a luminance mapping with chroma scaling (LMCS) is performed before the loop filter.

[0159] Figure 7A portion of a video decoder 600 implementing controlled cropping is conceptually illustrated. As shown, decoder 600 implements controlled cropping in four stages: post-prediction (stage 710, after inter-frame prediction module 640 and intra-frame prediction module 625), post-reconstruction (stage 720, after reconstructed pixel data 617), post-deblocking (stage 727, after deblocking filter 725), post-SAO (stage 730, after SAO 728), and post-ALF (stage 740, after ALF 735; ALF 735, SAO 728, and deblocking filter 725 are part of in-loop filter 645).

[0160] During prediction, the prediction reference buffers of the intra-frame prediction module 625 and the inter-frame prediction module 640 are used to store the predicted value 613. After prediction, control clipping can be applied to the generated prediction 613 in the post-prediction stage 710. The quantization coefficients 612 are inversely quantized and inversely transformed to the reconstruction residual 619. The prediction 613 and the reconstruction residual 619 are combined to generate the decoded pixel data 617. The control clipping applied in the post-prediction stage 710 reduces the error level of the decoded pixel data 617. This extends the dynamic range of the decoded pixel value 617. If the range is known in advance, the dynamic range of the decoded pixel value 617 can be limited by the control clipping in the post-reconstruction stage 720. This reduces pixel error when the decoded pixel value 617 exceeds the limit.

[0161] Deblocking and ALF alter the reconstructed pixel values ​​through filtering. The dynamic range of the filtered values ​​may change. Controlled clipping in post-deblocking stage 730 and post-ALF stage 740 can also limit pixel values ​​within the correct range to minimize the expanded dynamic range.

[0162] In some embodiments, the maximum and minimum values ​​(or other types of range definitions) for the constraint sample values ​​in each control shearing phase 710-740 are provided by the entropy decoder 690 and parsed as syntax elements from the bitstream 695. In some embodiments, the maximum and minimum values ​​for the LMCS operation are also provided by the entropy decoder 690 and parsed from the bitstream 695.

[0163] Figure 8 A process 800 for performing adaptive sample clipping during video decoding is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing decoder 600 execute process 800 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing decoder 600 executes process 800.

[0164] The decoder receives (in block 810) the data of the current block to be decoded into the current image pixels of the video.

[0165] The decoder receives (in block 820) the first set of range definitions. The first set of range definitions can be enabled to apply to data samples of any other high-level entity containing the current block within the same slice, the same brick, the same coding tree unit (CTU) row, the same coding unit, or the video coding hierarchy. In some embodiments, the first set of range definitions (along with other flags related to adaptive sample clipping described in Section II above) can be signaled in the slice header, image header, adaptive parameter set (APS), and / or image parameter set (PPS).

[0166] The decoder reconstructs (in block 830) the current block by processing the received data in one or more encoding / decoding stages. The decoder can then provide the reconstructed current block as part of the reconstructed current image for display. In some embodiments, the encoding / decoding stages that perform adaptive sample clipping include post-prediction, post-reconstruction, post-deblocking, and post-ALF.

[0167] The decoder constraint (in block 840) restricts the data samples generated by the first encoding / decoding stage to a first range defined by a first set of range definitions. The first set of range definitions can apply a clipping function to the data samples generated by the first encoding / decoding stage, specifically imposing maximum and minimum permissible values ​​on the data samples generated by the first encoding / decoding stage. In some embodiments, the maximum and minimum permissible values ​​apply to both the luma and chroma components. In some embodiments, the maximum and minimum permissible values ​​apply only to samples of the luma component and not to samples of the chroma component. In some embodiments, the data samples generated by the second encoding / decoding stage are also restricted to a first numerical range defined by the first set of range definitions. In other words, the same set of maximum and minimum values ​​applies to multiple stages.

[0168] The video decoder can also receive a second set of range definitions so that the data samples generated by the second encoding / decoding stage are restricted to a second numerical range defined by the second set of range definitions. In other words, different sets of maximum and minimum values ​​can be applied to different stages. In some embodiments, the first set of range definitions is one of multiple sets of range definitions in the decoder signaling, and the decoder selects the first set of range definitions from the multiple sets of range definitions to apply to the data samples generated by the first encoding / decoding stage.

[0169] In some embodiments, the first set of range definitions is used for LMCS, that is, to define a first range that maps luminance values ​​from the original domain to the reshaped domain. The first set of range definitions may include a maximum and a minimum value for defining the first range, as well as a number of non-zero codewords, and the number of non-zero codewords may not be a power of two. The second set of range definitions may be used to define a second range that maps luminance values ​​from the original domain to the reshaped domain.

[0170] In some embodiments, LMCS is disabled when adaptive sample clipping is applied, and adaptive sample clipping is disabled when LMCS is applied. In some embodiments, when LMCS is enabled, adaptive sample clipping is used in the in-loop filtering stage (e.g., DBF, ALF, SAO). In some embodiments, when LMCS is disabled, adaptive sample clipping is applied to the prediction or reconstruction stage of video decoding.

[0171] V. Example Electronic System

[0172] Many of the features and applications described above are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also known as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing unit to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. Computer-readable media do not include carrier waves and electronic signals transmitted via wireless or wired connections.

[0173] In this specification, the term "software" includes firmware residing in read-only memory or application programs stored in magnetic memory, which can be read into memory for processing by a processor. Furthermore, in some embodiments, multiple software inventions may be implemented as sub-parts of a larger program while remaining independent software inventions. In some embodiments, multiple software inventions may also be implemented as independent programs. Finally, any combination of independent programs that collectively implement the software inventions described herein is within the scope of this disclosure. In some embodiments, when a software program is installed on one or more electronic systems and runs, one or more specific machine implementations are defined that execute and perform the operations of the software program.

[0174] Figure 9 An electronic system 900 is conceptually illustrated, in which certain embodiments of the present disclosure are implemented. The electronic system 900 may be a computer (e.g., a desktop computer, personal computer, tablet computer, etc.), a telephone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 900 includes a bus 905, a processing unit 910, a graphics processing unit (GPU) 915, system memory 920, a network 925, a read-only memory 930, a permanent storage device 935, an input device 940, and an output device 945.

[0175] Bus 905 collectively represents all system, peripheral, and chipset buses that communicate with the numerous internal devices of electronic system 900. For example, bus 905 communicates with processing unit 910, GPU 915, read-only memory 930, system memory 920, and permanent storage device 935.

[0176] From these different memory units, processing unit 910 retrieves instructions for execution and processes data to perform the processes of this disclosure. The processing unit may be a single-processor or a multi-core processor, in different embodiments. Some instructions are passed to GPU 915 and executed by it. GPU 915 may offload various computational or supplementary image processing provided by processing unit 910.

[0177] Read-only memory (ROM) 930 stores static data and instructions used by processing unit 910 and other modules of the electronic system. On the other hand, permanent storage device 935 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 900 is powered off. Some embodiments of this disclosure use mass storage devices (e.g., disks or optical discs and their corresponding disk drives) as permanent storage device 935.

[0178] Other embodiments use removable storage devices (e.g., floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Similar to permanent storage device 935, system memory 920 is a read-write memory device. However, unlike storage device 935, system memory 920 is volatile read-write memory, such as random access memory. System memory 920 stores some instructions and data used by the processor during runtime. In some embodiments, processes according to this disclosure are stored in system memory 920, permanent storage device 935, and / or read-only memory 930. For example, various memory units include instructions for processing multimedia clips according to some embodiments. From these various memory units, processing unit 910 retrieves instructions to execute and processes data to perform the processes of some embodiments.

[0179] Bus 905 is also connected to input and output devices 940 and 945. Input device 940 enables a user to pass information and select commands to the electronic system. Input device 940 includes an alphanumeric keypad and pointing device (also known as a "cursor control device"), a camera (e.g., a webcam), a microphone, or similar devices for receiving voice commands. Output device 945 displays images generated by the electronic system or otherwise outputs data. Output device 945 includes printers and display devices, such as cathode ray tube (CRT) or liquid crystal display (LCD), and speakers or similar audio output devices. Some embodiments include devices such as touchscreens that function as both input and output devices.

[0180] Finally, as Figure 9 As shown, bus 905 also connects electronic system 900 to network 925 via a network adapter (not shown). In this way, the computer can become part of a computer network (e.g., a local area network (“LAN”), a wide area network (“WAN”), or an intranet, or a network of networks, such as the Internet). Any or all components of electronic system 900 can be used in conjunction with this disclosure.

[0181] Some embodiments include electronic components, such as microprocessors, storage, and memory, that store computer program instructions in a machine-readable or computer-readable medium (also referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Examples of such computer-readable media include RAM, ROM, read-only optical disc (CD-ROM), recordable optical disc (CD-R), rewritable optical disc (CD-RW), read-only digital versatile optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state drives, read-only and recordable... Optical discs, high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media can store a computer program that can be executed by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as code generated by a compiler, and files containing high-level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

[0182] While the foregoing discussion primarily concerns microprocessors or multi-core processors that execute software, many of the features and applications described above are implemented by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, these integrated circuits execute instructions stored on the circuit itself. Furthermore, some embodiments execute software stored in programmable logic devices (PLDs), read-only memory (ROM), or random access memory (RAM) devices.

[0183] The terms “computer,” “server,” “processor,” and “memory” as used in this specification and any claim of this application refer to electronic or other technical devices. These terms do not include individuals or groups. For the purposes of this specification, the term “display” or “show” means “displayed on an electronic device.” The terms “computer-readable medium,” “computer-readable media,” and “machine-readable medium” as used in this specification and any claim of this application are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, or any other transient signals.

[0184] While this disclosure has been described with reference to numerous specific details, those skilled in the art will recognize that this disclosure may be embodied in other specific forms without departing from its spirit. Furthermore, many figures (including...) Figure 5 and Figure 8 The process is conceptually described. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process may be implemented using several sub-processes or as part of a larger macro-process. Therefore, those skilled in the art will understand that this disclosure should not be limited to the foregoing illustrative details but should be defined by the appended claims.

[0185] Additional notes

[0186] The topics described herein sometimes illustrate different components contained within or connected to other components. It should be understood that the architectures shown are merely examples, and many other architectures can actually be implemented to achieve the same functionality. Conceptually, any arrangement of components to achieve the same functionality is effectively “associated” to achieve the desired function. Therefore, any two components combined in this document to achieve a particular function can be considered “associated” together to achieve the desired function, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be considered “operably connected” or “operably coupled” together to achieve the desired function, and any two components that can be so associated can also be considered “operably coupled” together to achieve the desired function. Specific examples of operational coupling include, but are not limited to, physically matable and / or physically interactive components and / or wirelessly interactive components and / or logically interactive and / or logically interactive components.

[0187] Furthermore, regarding the use of virtually any plural and / or singular terms in this document, those skilled in the art can appropriately translate them from plural to singular and / or from singular to plural depending on the context and / or application. For clarity, various singular / plural permutations may be explicitly listed herein.

[0188] Furthermore, those skilled in the art will understand that terms generally used herein, particularly in appended claims, such as the body of an appended claim, are generally considered "open" terms; for example, the term "comprising" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," and the term "including" should be interpreted as "including but not limited to," etc. Those skilled in the art will also understand that if a specific number of claim statements is intentional, that intention will be explicitly stated in the claims, and such intention does not exist without such statements. For example, to aid understanding, the following appended claims may contain the use of introductory phrases "at least one" and "one or more" to introduce claim statements. However, the use of these phrases should not be construed as implying that any particular claim introducing claim statements by the indefinite article "a" or "an" is limited to containing only one such statement, even if the same claim includes the introductory phrase "one or more" or "at least one" and indefinite articles such as "a" or "an," for example, "a" and / or "an" should be interpreted as "at least one" or "one or more"; the same applies to definite articles used to introduce claim statements. Furthermore, even if the specific number of claims is explicitly stated, those skilled in the art will recognize that such statements should be interpreted as at least the stated number. For example, the simple statement "two statements" without any other modifiers means at least two statements, or two or more statements. Additionally, in conventions similar to "at least one A, B, and C, etc.", this structure is generally understood by those skilled in the art to mean the convention. For example, "a system having at least one A, B, and C" will include, but is not limited to, a system having only A, a system having only B, a system having only C, a system with A and B together, a system with A and C together, a system with B and C together, and / or a system with A, B, and C together, etc. In conventions similar to "at least one A, B, or C, etc.", this structure is generally understood by those skilled in the art to mean the convention. For example, "a system having at least one A, B, or C" will include, but is not limited to, a system having only A, a system having only B, a system having only C, a system with A and B together, a system with A and C together, a system with B and C together, and / or a system with A, B, and C together, etc. Those skilled in the art will also understand that virtually any extractive word and / or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to include the possibility of one term, two terms, or two terms. For example, the phrase “A or B” will be understood to include the possibility of “A” or “B” or “A and B”.

[0189] As can be seen from the foregoing, the various implementations of this disclosure described herein are for illustrative purposes, and various modifications can be made without departing from the scope and spirit of this disclosure. Therefore, the various implementations disclosed herein are not intended to be limiting, and their true scope and spirit are indicated by the following claims.

Claims

1. A video coding method comprising: receiving data to be encoded or decoded as a current block of pixels of a current picture of a video; signaling or receiving a first set of range definitions; and encoding or decoding the current block by processing the received data in one or more coding stages, wherein data samples resulting from a first coding stage are limited within a first numerical range defined by the first set of range definitions.

2. The video coding method of claim 1, wherein the first set of range definitions applies a clipping function to the data samples resulting from the first coding stage.

3. The video coding method of claim 1, wherein the first set of range definitions imposes a maximum allowed value and a minimum allowed value on data samples resulting from the first coding stage.

4. The video coding method of claim 3, wherein the maximum and minimum allowed values apply to luma and chroma components.

5. The video coding method of claim 3, wherein the maximum and minimum allowed values apply only to samples of luma components and not to samples of chroma components.

6. The video coding method of claim 3, wherein a difference between the maximum and minimum allowed values and predefined maximum and minimum values is signaled by an encoder or decoded by a decoder.

7. The video coding method of claim 1, wherein data samples resulting from a second coding stage are limited within the first numerical range defined by the first set of range definitions.

8. The video coding method of claim 1, further comprising signaling or receiving a second set of range definitions, wherein data samples resulting from a second coding stage are limited within a second numerical range defined by the second set of range definitions.

9. The video coding method of claim 1, wherein the first set of range definitions is one of a plurality of sets of range definitions received or signaled, the method further comprising selecting the first set of range definitions from the plurality of sets of range definitions to apply to the data samples resulting from the first coding stage.

10. The video coding method of claim 1, wherein the first set of range definitions is enabled to apply to data samples of a same slice, a same tile, a same coding tree unit (CTU) row, or a same coding unit.

11. The video coding method of claim 1, wherein the first set of range definitions defines a first range mapping luma values from an original domain to a reshaped domain.

12. The video coding method of claim 10, wherein the first set of range definitions includes a maximum value and a minimum value for defining the first range and a number of non-zero codewords.

13. The video coding method of claim 11, wherein the number of non-zero codewords is not a power of two.

14. The video coding method of claim 10, wherein a second set of range definitions defines a second range mapping luma values from the original domain to the reshaped domain.

15. The video coding method of claim 1, wherein luma mapping with chroma scaling (LMCS) is disabled when adaptive sample clipping is applied, and adaptive sample clipping is disabled when LMCS is applied.

16. The video coding method of claim 1, wherein the adaptive sample clipping is used in an in-loop filtering stage when luma mapping with chroma scaling (LMCS) is enabled.

17. The video coding method of claim 1, wherein the adaptive sample clipping is applied in a prediction stage or a reconstruction stage when luma mapping with chroma scaling (LMCS) is disabled.

18. The video coding method of claim 1, wherein the first set of range definitions is signaled in at least one of a slice header, a picture header, an adaptation parameter set (APS), and a picture parameter set (PPS).

19. An electronic device, comprising: video coding circuitry configured to perform the following operations: receive data to be encoded or decoded as a current block of pixels of a current picture of a video; signal or receive a first set of range definitions; and encode or decode the current block by processing the received data in one or more coding stages, wherein data samples produced by a first coding stage are limited to a first range of values defined by the first set of range definitions.

20. A video decoding method, comprising: receiving data to be decoded as a current block of pixels of a current picture of a video; receiving a first set of range definitions; and reconstructing the current block by processing the received data in one or more coding stages, wherein data samples produced by a first coding stage are limited to a first range of values defined by the first set of range definitions.

21. A video encoding method, comprising: receiving data to be encoded as a current block of pixels of a current picture of a video; signaling a signal of a first set of range definitions; and encoding the current block by processing the received data in one or more coding stages, wherein data samples produced by a first coding stage are limited to a first range of values defined by the first set of range definitions.

Images