Multi-parameter adaptive loop filtering in video processing

Abstract

Apparatus, systems, and methods for video processing are described. In one exemplary aspect, a method for video processing includes: performing a conversion between a bitstream of video comprising one or more video regions and video, wherein a limiting parameter for filtering a reconstruction of a first video unit of the video region using a nonlinear adaptive loop filter is derived based on first side information included in the bitstream and an identifier of a first adaptive parameter set, and wherein filter coefficients for filtering the reconstruction of the first video unit using the nonlinear adaptive loop filter are derived based on second side information included in the bitstream and an identifier of the first adaptive parameter set.

Description

[0001] This patent application is a divisional application of the following invention patent application:

[0002] Application Number: 202080023970.7

[0003] Application date: March 24, 2020

[0004] Invention Title: Multi-parameter Adaptive Loop Filtering in Video Processing

[0005] Cross-references to related applications

[0006] In accordance with the patent law and / or rules applicable under the Paris Convention, this application aims to promptly claim priority and benefit from International Patent Application No. PCT / CN2019 / 079395, filed on March 24, 2019. For all purposes required by law, the entire disclosure of the aforementioned application is incorporated herein by reference as part of the disclosure of this application. Technical Field

[0007] This patent document relates to video processing technologies, equipment, and systems. Background Technology

[0008] Despite advancements in video compression, digital video still consumes the largest share of bandwidth on the internet and other digital communication networks. As the number of networked user devices capable of receiving and displaying video increases, the bandwidth demands for digital video are expected to continue to grow. Summary of the Invention

[0009] This document describes devices, systems, and methods related to digital video processing, such as devices, systems, and methods related to nonlinear adaptive loop filtering for video processing. The described methods can be applied to existing video codec standards (e.g., High Efficiency Video Coding (HEVC)) and future video codec standards (e.g., Versatile Video Coding (VVC)) or codecs.

[0010] In one representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: encoding video units of a video into coded video units; generating reconstructed samples from the coded video units; performing a clipping operation on the reconstructed samples, wherein the clipping parameter used in the clipping operation is a function of a clipping index and the bit depth of the reconstructed samples or the bit depth of the samples of the video unit; applying a nonlinear adaptive loop filter to the output of the clipping operation; and generating a codec representation of the video using the coded video units.

[0011] On the other hand, the disclosed technology can be used to provide a method for video processing. The method includes: parsing the codec representation of a video to obtain coded video units representing video units; generating reconstructed samples of the video units from the coded video units; performing a clipping operation on the reconstructed samples, wherein the clipping parameter used in the clipping operation is a function of a clipping index and the bit depth of the reconstructed samples or the bit depth of the video units; and applying a nonlinear adaptive loop filter to the output of the clipping operation to generate the final decoded video units.

[0012] On the other hand, the disclosed technology can be used to provide a method for video processing. The method includes: performing a codec representation of a video comprising one or more video regions and a conversion between the video and the video; and determining a limiting parameter for filtering the reconstruction of video units of the video regions using a nonlinear adaptive loop filter, wherein the determination is based on codec information of the video and / or the video regions and / or the video units.

[0013] On the other hand, the disclosed technology can be used to provide a method for video processing. The method includes: performing a codec representation of a video comprising one or more video regions and a conversion between the video and the representation; and determining a limiting parameter for filtering the reconstruction of video units of the video regions using a nonlinear adaptive loop filter, wherein the limiting parameter is a function of the color representation format.

[0014] On the other hand, the disclosed technology can be used to provide a method for video processing. The method includes: performing a codec representation of a video comprising one or more video regions and a conversion between the video and the video; and determining a limiting parameter for filtering the reconstruction of video units of the video regions using a nonlinear adaptive loop filter, wherein the limiting parameter depends on whether loop shaping (ILR) is applied to reconstruct the video units based on the representation of the video units in a first domain and a second domain and / or the scaled chroma residual of the chroma video units.

[0015] On the other hand, the disclosed technology can be used to provide a method for video processing. The method includes: performing a conversion between a codec representation of a video comprising one or more video regions and the video itself, wherein the codec representation includes first side information providing a limiting parameter for filtering the reconstruction of video units of the video regions using a nonlinear adaptive loop filter; wherein the first side information is signaled together with second side information indicating the filter coefficients used in the nonlinear adaptive loop filter.

[0016] On the other hand, the disclosed technology can be used to provide a method for video processing. This method includes: performing a codec representation of a video comprising one or more video regions and a conversion between the video and the codec representation, wherein the codec representation includes side information indicating multiple sets of clipping parameters used to filter the reconstruction of video units of the video regions using a nonlinear adaptive loop filter.

[0017] On the other hand, the disclosed technology can be used to provide a method for video processing. The method includes: performing a codec representation of a video comprising one or more video regions and a conversion between the video and the representation; wherein the codec representation includes side information providing one or more limiting parameters for filtering the reconstruction of chroma video units of the video regions using a nonlinear adaptive loop filter, wherein the one or more limiting parameters depend on the color format.

[0018] On the other hand, the disclosed technology can be used to provide a method for video processing. The method includes: performing a conversion between a codec representation of a video comprising one or more video regions and the video itself, wherein the codec representation includes side information providing a limiting parameter for filtering the reconstruction of video units of the video regions using an adaptive loop filter, wherein the execution includes generating filtered video units by applying a limiting operation to sample differences at the video region level.

[0019] In another representative aspect, the above method is implemented in the form of processor-executable code and stored in a computer-readable program medium.

[0020] In another representative aspect, an apparatus configured or operable to perform the above-described methods is disclosed. The apparatus may include a processor programmed to implement the methods.

[0021] In another representative aspect, one or more of the methods disclosed above may be implemented on the encoder side or the decoder side.

[0022] The above and other aspects and features of the disclosed technology are described in more detail in the accompanying drawings, description and claims. Attached Figure Description

[0023] Figure 1 An example of an encoder block diagram for video encoding and decoding is shown.

[0024] Figure 2A , Figure 2B and Figure 2C An example of the filter shape for a geometry transformation-based adaptive loop filter (GALF) is shown.

[0025] Figure 3 An example flowchart of the GALF encoder decision is shown.

[0026] Figures 4A to 4D An example subsampled Laplace calculation is shown for classification of adaptive loop filter (ALF).

[0027] Figure 5 An example of the shape of a brightness filter is shown.

[0028] Figure 6 An example of region partitioning for a Wide Video Graphic Array (WVGA) sequence is shown.

[0029] Figure 7 An exemplary flowchart of a decoding process with shaping is shown.

[0030] Figures 8A to 8C A flowchart illustrating an example method for video processing according to some embodiments of the disclosed technology is shown.

[0031] Figure 9 A flowchart illustrating an example method for video processing according to some embodiments of the disclosed technology is shown.

[0032] Figure 10A and Figure 10B This is a block diagram of an example hardware platform used to implement the video processing techniques described in this document. Detailed Implementation

[0033] Due to the increasing demand for higher resolution video, video encoding and decoding methods and technologies are ubiquitous in modern technology. Video codecs typically consist of electronic circuitry or software that compresses or decompresses digital video and are constantly being improved to provide higher encoding and decoding efficiency. Video codecs convert uncompressed video into a compressed format and vice versa. There are complex relationships between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end latency (delay). Compression formats typically conform to standard video compression specifications, such as the High Efficiency Video Codec (HEVC) standard (also known as H.265 or MPEG-H Part 2), the pending Universal Video Codec (VVC) standard, or other current and / or future video codec standards.

[0034] In some embodiments, reference software called the Joint Exploration Model (JEM) is used to explore future video coding and decoding technologies. In the JEM, sub-block-based prediction is employed in several coding and decoding tools, such as affine prediction, alternative temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), directional optical flow (BIO), frame-rate upconversion (FRUC), locally adaptive motion vector resolution (LAMVR), overlapped block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

[0035] Embodiments of the disclosed techniques can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve runtime performance. Section headings are used in this document to improve readability and are not in any way intended to limit the discussion or embodiments (and / or implementations) solely to the individual sections.

[0036] 1. Examples of color spaces and chromaticity subsampling

[0037] A color space, also known as a color model (or color system), is an abstract mathematical model that simply describes the range of colors as tuples of numbers, typically 3 or 4 values ​​or color components (e.g., RGB). Essentially, a color space is a refinement of a coordinate system and its subspaces.

[0038] For video compression, the most commonly used color spaces are YCbCr and RGB.

[0039] YCbCr, Y′CbCr, or Y Pb / Cb Pr / Cr, also known as YBCCR or Y'CBCR, is a family of color spaces used as part of the color imaging pipeline in video and digital photography systems. Y′ is the luminance component, while CB and CR are the blue and red chromaticity components. Y′ (with prime) differs from Y, where Y stands for luminance, meaning that the light intensity is non-linearly encoded based on gamma-corrected RGB primary colors.

[0040] Chromatic subsampling is a practice of encoding images by applying less resolution to chromaticity information than to luminance information. This takes advantage of the fact that the human visual system is less sensitive to chromatic aberration than to luminance.

[0041] 1.1 4:4:4 color format

[0042] Each of the three Y'CBcr components has the same sampling rate, so there is no chroma subsampling. This scheme is sometimes used in high-end film scanners and film post-production.

[0043] 1.2 4:2:2 color format

[0044] The two chroma components are sampled at half the luminance sampling rate; for example, the horizontal chroma resolution is halved. This reduces the bandwidth of the uncompressed video signal by one-third, resulting in almost no visual difference.

[0045] 1.3 4:2:0 color format

[0046] In the 4:2:0 mode, the horizontal sampling is doubled compared to 4:1:1, but in this scheme, the Cb and Cr channels are sampled only on each alternating line, thus halving the vertical resolution. Therefore, the data rate is the same. Cb and Cr are subsampled by a factor of 2 in both the horizontal and vertical directions. There are three variants of the 4:2:0 scheme with different horizontal and vertical addressing.

[0047] In MPEG-2, Cb and Cr are horizontally cosited. Cb and Cr are located between pixels in the vertical direction (in the gaps).

[0048] - In JPEG / JFIF, H.261 and MPEG-1, Cb and Cr are located in the gap, in the middle between alternating luminance samples.

[0049] In a 4:2:0DV configuration, Cb and Cr are co-located in the horizontal direction. In the vertical direction, they are co-located on alternating lines.

[0050] 2. Example of encoding / decoding process of typical video codec

[0051] Figure 1 An example of a VVC encoder block diagram is shown. (e.g.) Figure 1 As shown, the video coding process can begin with an input video that has undergone intra-frame prediction and / or motion estimation / motion compensation (ME / MC). These operations use feedback from reconstructed copies of previous codec portions of the video. The output of intra-frame prediction and / or ME / MC is differentially processed by a transform (T) operation, followed by a quantization (Q) operation, which is entropy-encoded into an output codec representation. In the feedback loop, the encoded representation (the output of the Q block) can undergo an inverse quantization (IQ) operation, followed by an inverse transform (IT) operation, to generate reconstructed samples of the encoded video block.

[0052] The reconstructed samples can be further processed through various "loop" filtering operations to generate reference samples, reference blocks, or reference images for further encoding. The loop filtering processing chain comprises three loop filtering blocks: a deblocking filter (DF), a sample adaptive offset (SAO), and an ALF. Unlike DF, which uses predefined filters, SAO and ALF utilize the original samples of the current image, reducing the mean square error between the original and reconstructed samples by adding an offset and by applying a finite impulse response (FIR) filter, respectively. The side information signaling for encoding and decoding informs the offset and filter coefficients. ALF is located at the final processing stage of each image and can be viewed as a tool to attempt to capture and repair artifacts generated in previous stages.

[0053] On the decoder side, several encoding operations are performed in reverse order to generate decoded and reconstructed video samples. For example, refer to... Figure 1 The decoder can parse the codec representation as the output of entropy encoding and decoding, and obtain the coded units or blocks of video. It then performs inverse quantization (IQ) and inverse transform (IT) operations on the coded units or blocks to generate reconstructed samples of the video units. The final decoded video units can be generated by applying the loop filtering operations described above regarding the feedback loop of the video encoder.

[0054] Example of an adaptive loop filter based on geometric transformation in 3JEM

[0055] In JEM, a geometric transformation-based adaptive loop filter (GALF) and a block-based filter are applied for adaptation. For the luminance component, one of 25 filters is selected for each 2×2 block based on the direction and activity of the local gradient.

[0056] 3.1 Example of filter shape

[0057] In JEM, up to three diamond filter shapes can be selected for the luminance component (e.g., Figure 2A , Figure 2B and Figure 2C As shown, these are 5×5, 7×7, and 9×9 rhombuses respectively. The image-level signaling notification index indicates the filter shape used for the luminance component. For the chrominance component in the image, a 5×5 rhombus is always used.

[0058] 3.1.1 Block Classification

[0059] Each 2×2 block is classified into one of 25 classes. The classification index C is based on a quantified value of its directionality D and activity. The derivation is as follows:

[0060]

[0061] To calculate D and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using the 1-D Laplacian operator:

[0062]

[0063] Indices i and j refer to the coordinates of the top-left sample point in the 2×2 block, and R(i,j) refers to the reconstructed sample point at coordinates (i,j).

[0064] Then the maximum and minimum values ​​of the horizontal and vertical gradients D are set as follows:

[0065]

[0066] Furthermore, the maximum and minimum values ​​of the gradients in the two diagonal directions are set as follows:

[0067]

[0068] To derive the value of the directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0069] Step 1. If and If both are true, then set D to 0.

[0070] Step 2. If If so, continue from step 3; otherwise, continue from step 4.

[0071] Step 3. If If so, set D to 2; otherwise, set D to 1.

[0072] Step 4. If If so, set D to 4; otherwise, set D to 3.

[0073] Activity value A is calculated as follows:

[0074]

[0075] A is further quantized to the range of 0 to 4 (inclusive), and the quantized value is represented as For the two chromaticity components in the image, no classification method is applied; that is, a single set of ALF coefficients is applied to each chromaticity component.

[0076] 3.1.2 Geometric Transformation of Filter Coefficients

[0077] Before filtering each 2×2 block, a geometric transformation, such as rotation or diagonal and vertical flip, is applied to the filter coefficients f(k,l), depending on the gradient value calculated for that block. This is equivalent to applying these transformations to samples within the filter's support region. The idea is to make the different blocks more similar by aligning their orientations when ALF is applied.

[0078] Three geometric transformations are introduced: diagonal flip, vertical flip, and rotation.

[0079] Diagonal flip: f D (k,l)=f(l,k),

[0080] Vertical flip: f V (k,l)=f(k,Kl-1), (9)

[0081] Rotation: f R (k,l)=f(Kl-1,k).

[0082] Here, K is the size of the filter, and 0 ≤ k, l ≤ K⁻¹ are the coefficient coordinates, such that position (0, 0) is at the top left corner and position (K⁻¹, K⁻¹) is at the bottom right corner. The transform is applied to the filter coefficients f(k, l) depending on the gradient values ​​calculated for this block. Table 1 summarizes the relationship between the transform and the four gradients in the four directions.

[0083] Table 1: Mapping of gradients and transformations calculated for a block

[0084] gradient value Transformation <![CDATA[g d2 <g d1 And g h <g v ]]> No transformation <![CDATA[g d2 <g d1 And g v <g h ]]> Flip diagonally <![CDATA[g d1 <g d2 And g h <g v ]]> Vertical flip <![CDATA[g d1 <g d2 And g v <g h ]]> Rotation

[0085] 3.1.3 Signaling of filter parameters

[0086] In JEM, the GALF filter parameters are signaled to the first CTU, i.e., after the stripe header and before the SAO parameters of the first CTU. Up to 25 sets of luminance filter coefficients can be signaled. To reduce bit overhead, filter coefficients from different categories can be merged. Furthermore, the GALF coefficients of the reference image are stored and can be reused as the GALF coefficients of the current image. The current image can optionally use the GALF coefficients stored for the reference image and bypass the GALF coefficient signaling. In this case, only the index of one of the reference images is signaled, and the stored GALF coefficients of the indicated reference image are inherited by the current image.

[0087] To support GALF temporal prediction, a candidate list of GALF filter banks is maintained. The candidate list is empty when decoding a new sequence begins. After decoding an image, a corresponding set of filters can be added to the candidate list. Once the size of the candidate list reaches the maximum allowed value (6 in the current JEM), a new set of filters overwrites the oldest set of filters in decoding order; that is, a first-in-first-out (FIFO) rule is applied to update the candidate list. To avoid duplication, a set of filters is only added to the list if the corresponding image does not use GALF temporal prediction. To support temporal scaling, there are multiple candidate lists of filter banks, and each candidate list is associated with a temporal layer. More specifically, each array assigned by a temporal layer index (TempIdx) can form a filter bank for previously decoded images whose TempIdx is equal to a lower TempIdx. For example, the k-th array is assigned to be associated with a TempIdx equal to k, and it contains only filter banks from images whose TempIdx is less than or equal to k. After a specific image is encoded or decoded, the filter bank associated with that image will be used to update those arrays associated with TempIdx equal to or higher.

[0088] Temporal prediction of GALF coefficients is used for inter-frame encoding / decoding to minimize signaling overhead. For intra-frame frames, temporal prediction is not available, and a set of 16 fixed filters is assigned to each class. To indicate the use of fixed filters, signaling informs each class of its flag, and, if necessary, also the index of the selected fixed filter. Even when a fixed filter is selected for a given class, the coefficients f(k,l) of an adaptive filter can still be sent for that class; in this case, the coefficients of the filter to be applied to the reconstructed image are the sum of the two sets of coefficients.

[0089] The filtering process for the luminance component can be controlled at the CU level. A signaling flag indicates whether GALF is applied to the luminance component of the CU. For the chrominance component, the application of GALF is indicated only at the image level.

[0090] 3.1.4 Filtering process

[0091] On the decoder side, when GALF is enabled for a block, each sample R(i,j) within the block is filtered to obtain the sample value R′(i,j), as shown below, where L represents the filter length, f m,n Let f(k,l) represent the filter coefficients, and let f(k,l) represent the decoding filter coefficients.

[0092]

[0093] 3.1.5 Determination process of encoder-side filter parameters

[0094] Figure 3 The entire encoder decision process for GALF is illustrated. For each luminance sample in a CU, the encoder decides whether to apply GALF, and the appropriate signaling flags are included in the strip header. For chroma samples, the decision to apply the filter is made at the picture level, not the CU level. Furthermore, chroma GALF for the picture is only checked if luminance GALF is enabled for the picture.

[0095] Example of a geometric transformation-based adaptive loop filter in 4VVC

[0096] Compared to JEM, the current design of GALF in VVC has the following main changes:

[0097] 1) Adaptive filter shapes have been removed. Only 7x7 filter shapes are allowed for the luminance component, while 5x5 filter shapes are allowed for the chrominance component.

[0098] 2) The time-domain predictions of the ALF parameters and the predictions from the fixed filter are both removed.

[0099] 3) For each CTU, the signaling notifies a bit flag to indicate whether ALF is enabled or disabled.

[0100] 4) Class index calculation is performed at a 4x4 level instead of a 2x2 level. Furthermore, as proposed in JVET-L0147, a subsampling Laplacian calculation method used for ALF classification is employed. More specifically, it is not necessary to calculate the horizontal / vertical / 45-degree diagonal / 135-degree gradient for each sample point within a block. Instead, 1:2 subsampling is utilized.

[0101] Example of a region-based adaptive loop filter in 5AVS2

[0102] ALF is the final stage of loop filtering. This process has two phases. The first phase is filter coefficient derivation. To train the filter coefficients, the encoder classifies the reconstructed pixels of the luma component into 16 regions and trains a set of filter coefficients for each category using the Wiener-Hopf equation to minimize the mean square error between the original and reconstructed frames. To reduce redundancy among these 16 sets of filter coefficients, the encoder adaptively merges them based on rate-distortion performance. At its maximum, 16 different filter banks can be assigned to the luma component, while only one filter bank is assigned to the chroma component. The second phase is filter decision, including frame-level and LCU-level decisions. First, the encoder decides whether to perform frame-level adaptive loop filtering. If frame-level ALF is enabled, the encoder further decides whether to perform LCU-level ALF.

[0103] 5.1 Filter Shape

[0104] The filter shape used in AVS-2 is a 7×7 cross shape superimposed with 3×3 squares, just as... Figure 5 The figures show the luminance and chromaticity components. Figure 5 Each square in the diagram corresponds to one sample point. Therefore, a total of 17 samples are used to derive the filtered value of the sample at position C8. Considering the overhead of the transmission coefficients, using a point-symmetric filter with only nine coefficients ({C0,C1,…,C8}) halves the number of filter coefficients and the number of multiplications in the filtering process. The point-symmetric filter can also reduce the computation for a single filtered sample by half; for example, there are only 9 multiplications and 14 additions for a single filtered sample.

[0105] 5.2 Region-based adaptive merging

[0106] To accommodate different encoding / decoding errors, AVS-2 employs a region-based multi-adaptive loop filter for the luma component. The luma component is divided into 16 roughly equal-sized basic regions, each aligned with the boundary of the largest coding unit (LCU), such as... Figure 6 As shown, a Wiener filter is derived for each region. The more filters used, the greater the distortion reduction; however, the number of bits used to encode these coefficients increases with the number of filters. To achieve optimal rate-distortion performance, these regions can be merged into smaller, larger regions that share the same filter coefficients. To simplify the merging process, each region is indexed according to a modified Hilbert order based on image prior correlation. Two regions with consecutive indices can be merged based on rate-distortion cost.

[0107] The decoder signaling should inform the mapping information between regions. In AVS-2, the number of basic regions is used to represent the merging result, and the filter coefficients are compressed sequentially according to their region order. For example, when {0,1}, {2,3,4}, {5,6,7,8,9} and the left basic region are merged into one region, only three integers are encoded and decoded to represent the merged graph, namely 2, 3, and 5.

[0108] 5.3 Signaling of Side Information

[0109] Several switch flags are also used. The sequence switch flag `adaptive_loop_filter_enable` controls whether the adaptive loop filter is applied to the entire sequence. The image switch flag `picture_alf_enble[i]` controls whether ALF is applied to the corresponding i-th image component. Only when `picture_alf_enble[i]` is enabled is the corresponding LCU level flag and filter coefficients for that color component transmitted. The LCU level flag `lcu_alf_enable[k]` controls whether ALF is enabled for the corresponding k-th LCU and is interleaved into the stripe data. The determination of the flags for different levels of adjustment is based on rate-distortion cost. This high flexibility further greatly improves the encoding and decoding efficiency of ALF.

[0110] In some embodiments, there may be up to 16 sets of filter coefficients for the luminance component.

[0111] In some embodiments, a set of filter coefficients may be transmitted for each chromaticity component (Cb and Cr).

[0112] GALF in 6VTM-4

[0113] In VTM4.0, the filtering process of the adaptive loop filter is performed as follows:

[0114] O(x, y) = ∑ (i，j) w(i,j).I(x+i,y+j) (11)

[0115] Here, sample I(x+i, y+j) are the input samples, O(x, y) are the filtered output samples (i.e., the filtering result), and w(i, j) represent the filter coefficients. In fact, VTM 4.0 uses integer algorithms to perform fixed-point precision calculations.

[0116]

[0117] Where L represents the filter length, and w(i,j) are the fixed-point filter coefficients.

[0118] 7. Nonlinear Adaptive Loop Filtering (ALF)

[0119] 7.1 Filtering and Reconstruction

[0120] Equation (11) can be revised without affecting encoding / decoding efficiency using the following expression:

[0121] O(x,y)=I(x,y)+∑ (i,j)≠(0，0) w(i,j).(I(x+i,y+j)-I(x,y)) (13)

[0122] Here, w(i,j) are the same filter coefficients as those in equation (11) (except that w(0,0) is equal to 1 in equation (13), while it is equal to 1-∑ in equation (11). (i,j)≠(0，0) w(i,j)].

[0123] 7.2 Modified Filter

[0124] Using the filter formula in (13) above, when the neighboring sample values ​​(I(x+i, y+j)) differ too much from the current sample value (I(x, y)) being filtered, we can reduce the influence of these neighboring sample values ​​by using a simple limiting function, thereby introducing a simple nonlinearity and making the AFL more efficient.

[0125] In this proposal, the ALF filter is modified as follows:

[0126] O′(x, y)=I(x, y)+∑ (i，j)≠(0，0) w(i,j).K(I(x+i,y+j)-I(x,y),k(i,j)) (14)

[0127] Here, K(d, b) = min(b, max(b, d)) is the limiting function, and k(i, j) is the limiting parameter, which depends on the (i, j) filter coefficients. The encoder performs optimization to find the optimal k(i, j).

[0128] In the JVET-N0242 implementation, a limiting parameter k(i, j) is specified for each ALF filter, and a limiting value is signaled for each filter coefficient. This means that up to 12 limiting values ​​can be signaled in the bitstream for each luminance filter, and up to 6 limiting values ​​can be signaled for each chrominance filter.

[0129] To limit signaling costs and encoder complexity, we restrict the evaluation of the clipping value to a small set of possible values. In this proposal, we use only four fixed values, which are the same for both the INTER and INTER chip groups.

[0130] Because the variance of local differences in luminance is typically higher than that in chrominance, we use two different sets of luminance and chrominance filters. We also include the maximum sample value in each set (here, 1024 for a bit depth of 10 bits), so clipping can be disabled if not necessary.

[0131] Table 2 provides a set of limiting values ​​used in the JVET-N0242 test. These four values ​​were selected by dividing the entire range of luminance sample values ​​(in 10-bit encoding / decoding) and the chrominance range from 4 to 1024 in the logarithmic domain into approximately equal parts.

[0132] More accurately, the brightness table for the limit value is obtained using the following formula:

[0133] Where M = 2 10 And N = 4.

[0134] Similarly, the colorimetric table for the limiting values ​​is obtained according to the following formula:

[0135] Where M = 2 10 N = 4 and A = 4.

[0136] Table 2: Authorized Limit Values

[0137]

[0138] The selected threshold value is encoded and decoded in the "alf_data" syntax element using the Golomb coding scheme corresponding to the index of the threshold value in Table 2 above. This coding scheme is the same as the coding scheme for the filter index.

[0139] Loop shaping (ILR) in 8JVET-M0427

[0140] In-loop reshaping (ILR) is also known as luminance mapping with chroma scaling (LMCS).

[0141] The basic idea of ​​loop shaping (ILR) is to transform the original (first domain) signal (predicted / reconstructed signal) to the second domain (shaping domain).

[0142] The loop luminance shaper is implemented as a pair of look-up tables (LUTs), but only one of the two LUTs needs to be signaled, as the other LUT can be calculated from the signaled LUT. Each LUT is a one-dimensional, 10-bit, 1024-entry mapping table (1D-LUT). One LUT is a forward LUT, FwdLUT, which takes the input luminance code value Y. i Mapped to the changed value Y r Y r =FwdLUT[Y i The other LUT is a reverse LUT, InvLUT, which will change the code value Y. r Mapped to ( Y represents i (The reconstructed value.)

[0143] 8.1 PWL Model

[0144] Conceptually, piecewise linear (PWL) is implemented as follows:

[0145] Let x1 and x2 be two input pivot points, and y1 and y2 be their corresponding output pivot points for a piece. The output value y for any input value x between x1 and x2 can be interpolated using the following equation:

[0146] y=((y2-y1) / (x2-x1))*(x-x1)+y1

[0147] In the fixed-point implementation, the equation can be rewritten as:

[0148] y=((m*x+2FP_PREC-1)>>FP_PREC)+c

[0149] Here, m is a scalar, c is the offset, and FP_PREC is a constant value that specifies the precision.

[0150] Note that in the CE-12 software, the PWL model is used to pre-compute the FwdLUT and InvLUT mapping tables with 1024 entries; however, the PWL model also allows for implementations that can compute the same mapping values ​​on the fly without pre-compiling the LUTs.

[0151] 8.2 Testing at the Fourth VVC Conference CE12-2

[0152] 8.2.1 Brightness Shaping

[0153] A test 2 for loop luminance shaping (i.e., CE12-2 in the proposal) provides a pipeline with lower complexity that also eliminates the decoding latency of block-by-block intra-prediction in inter-strip reconstruction. Intra-prediction is performed in the shaping domains of both inter-frame and intra-strip structures.

[0154] Regardless of the stripe type, intra-frame prediction is always performed in the integer domain. With this arrangement, intra-frame prediction can begin immediately after the previous TU reconstruction is complete. This arrangement also provides a unified process for intra-frame modes, rather than being stripe-dependent. Figure 7 A block diagram of the pattern-based CE12-2 decoding process is shown.

[0155] CE12-2 also tested a 16-segment piecewise linear (PWL) model for scaling of luminance and chrominance residuals, instead of the 32-segment PWL model of CE12-1.

[0156] In CE12-2, inter-frame strip reconstruction using a loop luminance shaper (the lighter shadow block indicates the signal in the shaping domain: luminance residual; intra-frame luminance prediction; and intra-frame luminance reconstruction)

[0157] 8.2.2 Luminance-dependent Chromaticity Residual Scaling

[0158] Luminance-dependent chroma residual scaling is a multiplication process implemented using fixed-point integer arithmetic. Chroma residual scaling compensates for the interaction between the luminance and chroma signals. Chroma residual scaling is applied at the TU level. More specifically, it applies to the following cases:

[0159] - For intra-frame calculations, average the reconstructed brightness.

[0160] - For inter-frame measurements, the predicted brightness is averaged.

[0161] The average is used to represent the index in the PWL model. The index represents the scaling factor cScaleInv. The chroma residual is multiplied by this number.

[0162] Note that the chroma scaling factor is calculated based on the predicted luminance value from the forward mapping, not the reconstructed luminance value.

[0163] 8.2.3 Signaling for ILR Side Information

[0164] Parameters (currently) are sent in the fragment header (similar to ALF). These reportedly require 40-100 bits. The following specification is based on JVET-L1001 version 9. Added syntax is highlighted in italics below.

[0165] The Sequence Parameter Set (RBSP) syntax in 7.3.2.1

[0166]

[0167] The general fragment header syntax in 7.3.3.1

[0168]

[0169]

[0170] Add a new syntax sheet group shaper model:

[0171]

[0172] Add the following semantics to the general sequence parameter set RBSP semantics:

[0173] A value of 1 for `sps_reshaper_enabled_flag` specifies that an integer shaper is used in a coded video sequence (CVS). A value of 0 for `sps_reshaper_enabled_flag` specifies that an integer shaper is not used in a CVS.

[0174] Add the following semantics to the fragment header syntax:

[0175] A `tile_group_reshaper_model_present_flag` value of 1 indicates that `tile_group_reshaper_model()` exists in the slice group header. A `tile_group_reshaper_model_present_flag` value of 0 indicates that `tile_group_reshaper_model()` does not exist in the slice group header. When `tile_group_reshaper_model_present_flag` does not exist, it is inferred to be equal to 0.

[0176] A tile_group_reshaper_enabled_flag value of 1 indicates that the shaper is enabled for the current tile group. A tile_group_reshaper_enabled_flag value of 0 indicates that the shaper is not enabled for the current tile group. When tile_group_reshaper_enable_flag does not exist, it is inferred to be equal to 0.

[0177] A `tile_group_reshaper_chroma_residual_scale_flag` value of 1 indicates that chroma residual scaling is enabled for the current tile group. A `tile_group_reshaper_chroma_residual_scale_flag` value of 0 indicates that chroma residual scaling is not enabled for the current tile group. When `tile_group_reshaper_chroma_residual_scale_flag` does not exist, it is inferred to be equal to 0.

[0178] Add the `tile_group_reshaper_model()` syntax.

[0179] `reshape_model_min_bin_idx` specifies the minimum binary bit (or segment) index to use during the shaper construction process. The value of `reshape_model_min_bin_idx` should be in the range of 0 to `MaxBinIdx` (inclusive). The value of `MaxBinIdx` should be equal to 15.

[0180] `reshape_model_delta_max_bin_idx` specifies the maximum allowed binary bit (or segment) index `MaxBinIdx` minus the maximum binary bit index to be used during the shaper construction process. The value of `reshape_model_max_bin_idx` is set to equal to `MaxBinIDx` – `reshape_model_delta_max_bin_idx`.

[0181] The increment of 1 in `reshaper_model_bin_delta_abs_cw_prec_minus1` specifies the number of bits used to represent the syntax `reshape_model_bin_delta_abs_CW[i]`.

[0182] reshape_model_bin_delta_abs_CW[i] specifies the absolute increment codeword value for the i-th binary bit.

[0183] `reshaper_model_bin_delta_sign_CW_flag[i]` specifies the sign of `reshape_model_bin_delta_abs_CW[i]`, as shown below:

[0184] – If reshape_model_bin_delta_sign_CW_flag[i] equals 0, then the corresponding variable RspDeltaCW[i] is positive.

[0185] Otherwise (reshape_model_bin_delta_sign_CW_flag[i] is not equal to 0), then the corresponding variable RspDeltaCW[i] is negative.

[0186] When reshape_model_bin_delta_sign_CW_flag[i] does not exist, it is inferred to be equal to 0.

[0187] Variable RspDeltaCW[i]=(1 2*reshape_model_bin_delta_sign_CW[i])*reshape_model_bin_delta_abs_CW[i];

[0188] The variable RspCW[i] is derived as follows:

[0189] The variable OrgCW is set to equal to (1 < <BitDepthY) / (MaxBinIdx+1)。

[0190] – If reshaper_model_min_bin_idx<=i<=reshaper_model_max_bin_idx,

[0191] Then RspCW[i]=OrgCW+RspDeltaCW[i].

[0192] Otherwise, RspCW[i] = 0.

[0193] If BitDepth Y If the value is 10, then the value of RspCW[i] should be in the range of 32 to 2*OrgCW-1.

[0194] The variable InputPivot[i] (where i is in the range of 0 to MaxBinIdx+1 (inclusive)) is derived as follows:

[0195] InputPivot[i] = i * OrgCW

[0196] The variables ReshapePivot[i] (i is in the range of 0 to MaxBinIdx+1 (inclusive)) and ScaleCoef[i] and InvScaleCoeff[i] (i is in the range of 0 to MaxBinIdx (inclusive)) are derived as follows:

[0197]

[0198] The variable ChromaScaleCoef[i] (where i is in the range of 0 to MaxBinIdx (inclusive)) is derived as follows:

[0199]

[0200] 8.2.4 Use of ILR

[0201] On the encoder side, each image (or group of images) is first converted to the integer domain. All encoding and decoding processes are performed in the integer domain. For intra-frame prediction, neighboring blocks are in the integer domain; for inter-frame prediction, reference blocks (generated from the original domain from the decoded image buffer) are first converted to the integer domain. The residuals are then generated and encoded / decoded into a bitstream.

[0202] After the entire image (or group of images) has been encoded / decoded, the samples in the integer domain are converted to the original domain, and then deblocking filters and other filters are applied.

[0203] Forward shaping of the predicted signal is disabled in the following cases:

[0204] - The current block is intra-frame encoded / decoded.

[0205] - The current block is encoded / decoded as CPR (Current Picture Reference, also known as Intra Block Copy (IBC)).

[0206] - The current block is encoded / decoded as a combined inter-intra mode (CIIP), and forward shaping is disabled for intra-predicted blocks.

[0207] 9. Disadvantages of existing implementation methods

[0208] The design of the non-linear ALF (NLALF) in JVET-N0242 has the following problems:

[0209] (1) It is designed for the 4:2:0 color format. For the 4:4:4 color format, the luminance and chrominance components may have similar importance. How to better apply NLALF is unknown.

[0210] (2) The limiting value is designed for the 10-bit case. How to define NLALF for other bit depths has not yet been studied.

[0211] (3) The interaction between loop shaping methods and NLALF has not been studied.

[0212] 10 Exemplary Methods for Improving Nonlinear Adaptive Loop Filtering

[0213] The embodiments of the currently disclosed technology overcome the shortcomings of existing implementations, thereby providing video codecs with higher encoding and decoding efficiency. Nonlinear adaptive loop filtering based on the disclosed technology can enhance existing and future video codec standards, as illustrated in the examples described below for various implementations. The examples of the disclosed technology provided below illustrate general concepts and are not intended to be limiting. In one example, the various features described in these examples can be combined unless explicitly stated otherwise.

[0214] 1. The parameters used in NLALF (e.g., the clipping parameters defined in Table 2) may depend on the encoding / decoding information.

[0215] a. Specify the parameters to be used in NLALF (e.g., the limiting parameters defined in Table 2).

[0216] This can depend on the temporal layer index / low-latency check flag / reference image.

[0217] 2. Multiple sets of NLALF parameters can be defined or signaled.

[0218] a. Alternatively, when multiple sets of NLALF parameters are signaled, they can be signaled in data units such as the Adaptation Parameter Set (APS) / slice header / video data unit.

[0219] b. In one example, signaling is used to notify NLALF parameters in a predictive manner.

[0220] i. For example, a set of NLALF parameters notified in a data unit (such as an APS, a slice, or a strip) is predicted by another set of NLALF parameters notified in the same data unit.

[0221] ii. For example, a set of NLALF parameters signaled in one data unit (such as an APS, a slice, or a stripe) is predicted by another set of NLALF parameters signaled in another data unit.

[0222] 3. The parameters used in NLALF (e.g., the clipping parameters defined in Table 2) may depend on the bit depth of the reconstructed samples before applying NLALF.

[0223] a. Alternatively, the parameters proposed for use in NLALF (e.g., the clipping parameters defined in Table 2) may depend on the input bit depth of the sample prior to encoding / decoding.

[0224] b. In one example, a parameter for a given bit depth can be derived from a parameter assigned to another bit depth.

[0225] i. In one example, a shift operation based on bit depth can be applied to derive a parameter for a given bit depth.

[0226] 4. The parameters used in NLALF (e.g., the limiting parameters defined in Table 2) may depend on the color representation format.

[0227] a. In one example, for the RGB case, the parameters have the same index for the G color component and for the B / R color component.

[0228] 5. The parameters used in NLALF (e.g., the limiting parameters defined in Table 2) may depend on whether the loop shaping (ILR) method is applied.

[0229] a. In one example, the parameters can differ when ILR is enabled or disabled.

[0230] 6. A proposal is made to store filter parameters (such as filter coefficients) and NLALF parameters (such as limiting parameters) together.

[0231] a. In one example, they can all be stored in APS.

[0232] b. In one example, when a video data unit (e.g., CTU / region / group) uses filter coefficients associated with an APS, the associated NLALF parameters can also be utilized.

[0233] c. Alternatively, for encoding / decoding a video data unit (e.g., CTU / region / group), when prediction based on filter coefficients associated with an APS is enabled, the associated NLALF parameters can also be used to predict the NLALF parameters of that video data unit based on the same APS.

[0234] 7. How NLALF processes chroma components can vary depending on the color format.

[0235] a. In one example, for a given color format (such as 4:4:4), the two chromaticity components can use different NLALF parameters.

[0236] 8. It proposes enabling or disabling ALF clipping at the sequence level, image level, strip level, group level, slice level, CTU level, CU level, or block level.

[0237] a. For example, the decoder can be signaled to whether clipping is enabled in the ALF, such as in the SPS, PPS, strip header, slice header, slice, CTU, CU or block signaling.

[0238] The above examples can be incorporated in the context of methods such as methods 810 to 840, which can be implemented at a video decoder or video encoder.

[0239] Figure 8A A flowchart of an exemplary method for video processing is shown. Method 810 includes, in step 812, encoding video units of a video into coded video units. Method 810 further includes, in step 813, generating reconstructed samples from the coded video units. Method 810 further includes, in step 814, performing a clipping operation on the reconstructed samples, wherein the clipping parameter used in the clipping operation is a function of the clipping index and the bit depth of the reconstructed samples or the bit depth of the samples of the video units. Method 810 further includes, in step 815, applying a nonlinear adaptive loop filter to the output of the clipping operation. Method 810 further includes, in step 816, generating a codec representation of the video using the coded video units.

[0240] Figure 8B A flowchart of an exemplary method for video processing is shown. Method 820 includes, in step 822, parsing the codec representation of the video to obtain coded video units representing video units of the video. Method 820 further includes, in step 823, generating reconstructed samples of the video units from the coded video units. Method 820 includes, in step 824, performing a clipping operation on the reconstructed samples, wherein the clipping parameter used in the clipping operation is a function of the clipping index and the bit depth of the reconstructed samples or the bit depth of the video units. Method 820 further includes, in step 825, applying a nonlinear adaptive loop filter to the output of the clipping operation to generate the final decoded video units.

[0241] Figure 8C A flowchart of an exemplary method for video processing is shown. Method 830 includes, in step 832, performing a conversion between a codec representation of a video comprising one or more video regions and the video itself. Method 830 further includes, in step 834, determining a limiting parameter for filtering the reconstruction of video units of the video regions using a nonlinear adaptive loop filter. In some embodiments, this determination is based on codec information of the video and / or video regions and / or video units. In some embodiments, the limiting parameter is a function of the color representation format. In some embodiments, the limiting parameter depends on whether loop shaping (ILR) is applied to reconstruct the video units based on the representation of the video units in a first domain and a second domain and / or the scaled chroma residual of the chroma video units.

[0242] Figure 9A flowchart of an exemplary method for video processing is shown. Method 840 includes performing a conversion between a codec representation of a video comprising one or more video regions and the video. In some embodiments, the codec representation includes providing first side information of clipping parameters for filtering the reconstruction of video cells of the video region using a nonlinear adaptive loop filter, and the first side information is signaled together with second side information indicating the filter coefficients used in the nonlinear adaptive loop filter. In some embodiments, the codec representation includes side information indicating multiple sets of clipping parameters for filtering the reconstruction of video cells of the video region using a nonlinear adaptive loop filter. In some embodiments, the codec representation includes providing side information of one or more clipping parameters for filtering the reconstruction of chroma video cells of the video region using a nonlinear adaptive loop filter, wherein the one or more clipping parameters depend on the color format. In some embodiments, the codec representation includes providing side information of clipping parameters for filtering the reconstruction of video cells of the video region using an adaptive loop filter, wherein the execution includes generating filtered video cells by applying a clipping operation to sample differences at the video region level.

[0243] Example implementations of 11 disclosed technologies

[0244] Figure 10A This is a block diagram of a video processing apparatus 900. Apparatus 900 can be used to implement one or more of the methods described herein. Apparatus 900 can be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. Apparatus 900 may include one or more processors 902, one or more memories 904, and video processing hardware 906. The processors 902(s) may be configured to implement one or more methods described in this document (including, but not limited to, method 800). The memories 904(s) may be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 906 may be used to implement some of the techniques described in this document in hardware circuitry.

[0245] Figure 10B This is another example of a block diagram of a video processing system in which the disclosed technology can be implemented. Figure 10BThis is a block diagram illustrating an example video processing system 4100 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 4100. System 4100 may include an input 4102 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10-bit multi-component pixel values, or in a compressed or encoded format. Input 4102 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, passive optical network (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0246] System 4100 may include codec component 4104, which may implement the various coding or encoding methods described in this document. Codec component 4104 may reduce the average bit rate of the video from input 4102 to the output of codec component 4104 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 4104 may be stored or transmitted via connected communication, as shown in component 4106. Component 4108 may use the bitstream (or codec) representation of the stored or transmitted video received at input 4102 to generate pixel values ​​or displayable video sent to display interface 4110. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that codec tools or operations are used at the codec and will be performed by the decoder as corresponding decoding tools or operations that are the reverse of the encoding result.

[0247] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The technologies described in this document can be found in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0248] Some embodiments of the disclosed technology involve making a decision or determination to enable a video processing tool or mode. In one example, when a video processing tool or mode is enabled, the encoder will use or implement that tool or mode in the processing of video blocks, but not necessarily modify the resulting bitstream based on the use of that tool or mode. That is, the conversion from video blocks to a video bitstream representation will be performed using that video processing tool or mode when it is enabled. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream knowing that it has been modified based on the video processing tool or mode. That is, the conversion from the video bitstream representation to video blocks will be performed using the video processing tool or mode enabled based on the decision or determination.

[0249] Some embodiments of the disclosed technology include making a decision or determination to disable a video processing tool or mode. In one example, when a video processing tool or mode is disabled, the encoder will not use the tool or mode in the conversion from video block to video bitstream representation. In another example, when a video processing tool or mode is disabled, the decoder will process the bitstream knowing that it has not been modified using a video processing tool or mode that is disabled based on the decision or determination.

[0250] In this document, the term "video processing" can refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm can be applied during the conversion from the pixel representation of the video to the corresponding bitstream representation, and vice versa. The bitstream representation of the current video block can, for example, correspond to bits co-located or distributed at different locations in the bitstream, as defined in the syntax. For example, macroblocks can be encoded based on the error residuals after transformation and encoding / decoding, and can also be encoded using bits from the header and other fields in the bitstream.

[0251] In some embodiments, the video encoding / decoding method may use, as referenced Figure 10A or Figure 10B The device described is implemented on a hardware platform.

[0252] Various technologies and embodiments can be described using the following terms-based format.

[0253] The first set of clauses describes certain characteristics and aspects of the disclosed technologies listed in the previous chapters.

[0254] 1. A method for video processing, comprising: determining a set of parameters for a current video block based on encoding and decoding information of the current video block; and reconstructing the current video block from a corresponding bitstream representation by performing a nonlinear filtering operation using the set of parameters.

[0255] 2. The method according to Clause 1, wherein the nonlinear filtering operation includes nonlinear adaptive loop filtering.

[0256] 3. The method according to Clause 1 or 2, wherein the set of parameters includes at least one limiting value for the luminance or chrominance component of the current video block.

[0257] 4. The method according to Clause 3, wherein the nonlinear filtering operation is based on a color format of chroma components.

[0258] 5. The method according to any one of Clauses 1 to 3, wherein the encoding / decoding information includes a temporal layer index, a low-latency check flag, or one or more reference pictures.

[0259] 6. The method according to any one of Clauses 1 to 3, wherein the encoding / decoding information includes the bit depth of the reconstructed sample prior to the nonlinear filtering operation.

[0260] 7. The method according to any one of Clauses 1 to 3, wherein the encoding / decoding information includes a color representation format.

[0261] 8. The method according to any one of Clauses 1 to 3, wherein the encoding / decoding information includes an instruction to apply a loop-shaping (ILR) method.

[0262] 9. The method according to any one of Clauses 1 to 3, wherein the corresponding bitstream representation includes multiple sets of parameters containing the set of parameters, and wherein the multiple sets of parameters are signaled in an Adaptive Parameter Set (APS), a slice header, or one or more video data units.

[0263] 10. The method according to any one of clauses 1 to 3, wherein the corresponding bitstream representation includes an adaptive parameter set (APS), the adaptive parameter set including the set of parameters and one or more filter coefficients associated with the nonlinear filtering operation.

[0264] 11. The method according to Clause 1 or 2, wherein the set of parameters includes one or more limiting values, and wherein the nonlinear filtering operation is performed at the sequence level, picture level, strip level, slice group level, slice level, codec tree unit (CTU) level, codec unit (CU) level, or block level.

[0265] 12. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 11.

[0266] 13. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method according to any one of clauses 1 to 11.

[0267] The second set of clauses describes certain features and aspects of the disclosed technologies listed in the preceding chapters, including, for example, Example Implementations 1 and 3-5.

[0268] 1. A video processing method, comprising: encoding video units of a video into coded video units; generating reconstructed samples from the coded video units; performing a clipping operation on the reconstructed samples, wherein the clipping parameter used in the clipping operation is a function of a clipping index and the bit depth of the reconstructed samples or the bit depth of the samples of the video units; applying a nonlinear adaptive loop filter to the output of the clipping operation; and generating a codec representation of the video using the coded video units.

[0269] 2. The method according to Clause 1, wherein the clipping index is signaled in the code-decode representation.

[0270] 3. A video processing method, comprising: parsing the encoded and decoded representation of a video to obtain an encoded video unit representing a video unit; generating reconstructed samples of the video unit from the encoded video unit; performing a clipping operation on the reconstructed samples, wherein the clipping parameter used in the clipping operation is a function of a clipping index and the bit depth of the reconstructed sample or the bit depth of the video unit; and applying a nonlinear adaptive loop filter to the output of the clipping operation to generate a final decoded video unit.

[0271] 4. The method described in Clause 3, wherein the clipping index is determined based at least on fields in the codec representation.

[0272] 5. The method described in Clause 3, wherein the limiting index is determined using predefined rules.

[0273] 6. The method according to Clause 1 or 3, wherein the limiting index and the bit depth of the reconstructed sample or the bit depth of the video unit are functions that return different values ​​for a given value of the limiting index based on the bit depth of the reconstructed sample or the bit depth of the video unit.

[0274] 7. The method according to Clause 1 or 3, wherein the mapping between the clipping index and the clipping parameter depends on the bit depth of the reconstructed sample or the bit depth of the video unit.

[0275] 8. The method according to Clause 1 or 3, wherein a first limiting value corresponding to a first limiting index at a given bit depth is derived based on a second limiting value corresponding to a second limiting index at another bit depth.

[0276] 9. The method according to Clause 8, wherein a shift operation of another bit depth is applied to derive a limiting parameter for a given bit depth.

[0277] 10. The method according to any one of clauses 1 to 9, wherein the encoding / decoding representation includes a limiting parameter that controls an upper or lower limit of the difference between two samples used in a nonlinear adaptive loop filter.

[0278] 11. A video processing method comprising: performing a codec representation of a video including one or more video regions and a conversion between the video and the video; and determining a limiting parameter for filtering the reconstruction of video units of the video regions using a nonlinear adaptive loop filter, wherein the determination is based on codec information of the video and / or the video regions and / or the video units.

[0279] 12. The method according to Clause 11, wherein the encoding / decoding information includes a time-domain layer index.

[0280] 13. The method according to Clause 11, wherein the encoding / decoding information includes a low-latency check flag.

[0281] 14. The method according to Clause 11, wherein the encoding / decoding information includes one or more reference images.

[0282] 15. The method according to any one of Clauses 11 to 14, wherein the video region includes video images.

[0283] 16. The method according to any one of Clauses 11 to 14, wherein the video unit includes an encoding / decoding unit.

[0284] 17. The method according to any one of clauses 11 to 16, wherein the limiting parameter controls an upper or lower limit of the difference between two samples used in the nonlinear adaptive loop filter.

[0285] 18. A video processing method comprising: performing a conversion between a video encoding / decoding representation and a video comprising one or more video regions; and determining a limiting parameter for filtering a reconstruction of video units of the video regions using a nonlinear adaptive loop filter, wherein the limiting parameter is a function of a color representation format.

[0286] 19. The method according to Clause 18, wherein, for RGB color format, the limiting parameter has the same index for the green component and for the blue or red component.

[0287] 20. A video processing method comprising: performing a conversion between a codec representation of a video including one or more video regions and between the video and the video; and determining a limiting parameter for filtering a reconstruction of video units of the video regions using a nonlinear adaptive loop filter, wherein the limiting parameter depends on whether loop shaping (ILR) is applied to reconstruct the video units based on the representation of the video units in a first domain and a second domain and / or the scaled chroma residual of the chroma video units.

[0288] 21. The method according to any one of Clauses 1 to 21, wherein the limiting parameter corresponds to the limiting value of the luminance component or chrominance component of the video unit.

[0289] 22. The method according to any one of clauses 1 to 21, wherein the method further comprises, during the conversion, generating filtered video cells by applying a nonlinear adaptive loop filter to the reconstruction of video cells, and using the filtered video cells to determine the prediction of another video cell of the video.

[0290] 23. The method according to any one of clauses 1 to 22, wherein performing the conversion includes generating a codec representation from the video.

[0291] 24. The method according to any one of clauses 1 to 22, wherein performing the conversion includes generating video from the codec representation.

[0292] 25. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform one or more of the methods described in claims 1 to 24.

[0293] 26. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing one or more of the methods described in accordance with clauses 1 to 24.

[0294] The third set of clauses describes certain features and aspects of the disclosed technologies listed in the preceding chapters, including, for example, Example Implementations 2 and 6-8.

[0295] 1. A video processing method comprising: performing a conversion between a codec representation of a video including one or more video regions and the video, wherein the codec representation includes first side information providing a limiting parameter for filtering the reconstruction of video cells of the video regions using a nonlinear adaptive loop filter; wherein the first side information is signaled together with second side information indicating the filter coefficients used in the nonlinear adaptive loop filter.

[0296] 2. The method according to Clause 1, wherein the first-side information and the second-side information are signaled in the same adaptive parameter set.

[0297] 3. The method according to Clause 1, wherein, where the video data unit uses at least some filter coefficients associated with the adaptive parameter set, the video data unit also uses a limiting parameter associated with the adaptive parameter set.

[0298] 4. The method according to Clause 1, wherein, when the conversion of a video data unit enables prediction based on at least some filter coefficients associated with an adaptive parameter set, the parameters associated with the adaptive parameter set are used to predict another parameter of another video data unit based on the adaptive parameter set.

[0299] 5. The method according to Clause 3 or 4, wherein the video data unit is a codec tree unit, a video region, or a group of segments.

[0300] 6. The method according to any one of Clauses 1 to 5, wherein the parameter corresponds to the limiting value of the luminance component or chrominance component of the video unit.

[0301] 7. A video processing method comprising: performing a conversion between a codec representation of a video including one or more video regions and the video, wherein the codec representation includes side information indicating multiple sets of limiting parameters for filtering the reconstruction of video units of the video regions using a nonlinear adaptive loop filter.

[0302] 8. The method according to Clause 7, wherein the edge information includes the multiple sets of limiting parameters.

[0303] 9. The method according to Clause 7, wherein the plurality of limiting parameters are known to the encoder and the decoder, and the side information includes an index of one or more limiting parameters in the plurality of limiting parameters.

[0304] 10. The method according to Clause 7, wherein the multiple sets of limiting parameters are included in the video data unit or the header of the video unit.

[0305] 11. The method according to Clause 10, wherein the video data unit comprises an adaptive parameter set, a slice group, or a strip.

[0306] 12. The method according to Clause 8, wherein one set of limiting parameters in a data unit signaling notification is predicted by another set of limiting parameters in the data unit signaling notification.

[0307] 13. The method according to Clause 7, wherein one set of limiting parameters in a set of signaling notifications in a data unit is predicted by another set of limiting parameters in a set of signaling notifications in another data unit.

[0308] 14. A video processing method comprising: performing a codec representation of a video including one or more video regions and a conversion between the video and the video; wherein the codec representation includes side information providing one or more limiting parameters for filtering the reconstruction of chroma video units of the video regions using a nonlinear adaptive loop filter, wherein the one or more limiting parameters depend on the color format.

[0309] 15. The method described in Clause 14, wherein, for a given color format, the two chromaticity components use different limiting parameters.

[0310] 16. The method described in Clause 15, wherein the specific color format is 4:4:4.

[0311] 17. The method according to any one of clauses 1 to 16, wherein the method further comprises, during the conversion, generating filtered video cells by applying a nonlinear adaptive loop filter to the reconstruction of video cells, and using the filtered video cells to determine the prediction of another video cell of the video.

[0312] 18. A video processing method comprising: performing a conversion between a codec representation of a video including one or more video regions and the video, wherein the codec representation includes side information providing a limiting parameter for filtering a reconstruction of video cells of the video regions using an adaptive loop filter, wherein the execution includes generating filtered video cells by applying a limiting operation to sample differences at the video region level.

[0313] 19. The method according to Clause 18, wherein the video region level is a sequence level, picture level, strip level, slice group level, slice level, codec tree unit level, codec unit level, or block level.

[0314] 20. The method according to Clause 18, wherein the instruction to enable clipping operation is signaled in the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), strip header, slice group header, slice, codec tree unit, codec unit, or block.

[0315] 21. The method according to any one of clauses 1 to 20, wherein the video region is a video image.

[0316] 22. The method according to any one of clauses 1 to 20, wherein the video unit is a codec unit or a transform unit or a strip or a codec tree or a codec tree row.

[0317] 23. The method according to any one of clauses 1 to 22, wherein performing the conversion includes generating a codec representation from the current block.

[0318] 24. The method according to any one of clauses 1 to 22, wherein performing the conversion includes generating the current block from the codec representation.

[0319] 25. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 24.

[0320] 26. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method according to any one of clauses 1 to 24.

[0321] Based on the foregoing, it should be understood that specific embodiments of the currently disclosed technology have been described for illustrative purposes, but various modifications can be made without departing from the scope of the invention. Therefore, the currently disclosed technology is not limited except for the appended claims.

[0322] The embodiments of the subject matter and functional operation described in this patent document can be implemented in various systems, digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium for execution by or control of the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a synthetic material that implements machine-readable propagation signals, or combinations thereof. The terms "data processing unit" or "data processing apparatus" encompass all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may include code that creates an operating environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof.

[0323] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to run on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a communication network.

[0324] The processes and logic described in this specification can be executed by one or more programmable processors running one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic can also be executed by special-purpose logic circuitry, and the device can be implemented as special-purpose logic circuitry, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0325] For example, processors suitable for running computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to one or more such mass storage devices to receive data from, transfer data to, or both. However, a computer does not require such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as electrical EPROMs, EEPROMs, and flash memory devices. The processor and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0326] This specification and accompanying drawings are to be considered exemplary only, where exemplary means example. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, the use of “or” is intended to include “and / or” unless the context clearly indicates otherwise.

[0327] Although this patent document contains numerous details, these details should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in this patent document within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, or even initially claimed in this way, in some cases one or more features from a claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof.

[0328] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or to perform all of the shown operations to obtain the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0329] Only a few implementations and examples have been described, and other implementations, enhancements and variations may be made based on what is described and shown in this patent document.

Claims

1. A method for processing video data, comprising: Perform a conversion between the bitstream of the video, which includes one or more video regions, and the video itself. The bitstream includes first-side information providing limiting parameters, which are used to filter the reconstruction of video units in a video region using a nonlinear adaptive loop filter; and The first side information and the second side information, which indicates the filter coefficients used in the nonlinear adaptive loop filter, are presented together in the same adaptive parameter set; and Wherein, when the video unit uses at least some of the filter coefficients associated with the adaptive parameter set, the video unit also uses a limiting parameter associated with the adaptive parameter set.

2. The method of claim 1, wherein, The video unit is a codec tree unit, a video region, or a group of segments.

3. The method according to claim 1, wherein, The limiting parameter corresponds to the limiting value of the luminance component or chrominance component of the video unit.

4. The method according to claim 1, wherein, The method further includes: During the conversion, a filtered video cell is generated by applying the nonlinear adaptive loop filter to the reconstruction of the video cell, and the filtered video cell is used to determine the prediction of another video cell of the video.

5. The method according to claim 1, wherein, The video area refers to video images.

6. The method according to claim 1, wherein, The video unit is an encoding / decoding unit, a transform unit, a strip, an encoding / decoding tree, or an encoding / decoding tree row.

7. The method according to claim 1, wherein, The limiting parameter is also derived based on the bit depth value of the video unit.

8. The method according to claim 1, wherein, The conversion includes encoding the video into the bitstream.

9. The method according to claim 1, wherein, The conversion includes decoding the video from the bitstream.

10. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: Perform a conversion between the bitstream of the video, which includes one or more video regions, and the video itself. in, The bitstream includes first-side information providing limiting parameters, which are used to filter the reconstruction of video units in a video region using a nonlinear adaptive loop filter; and The first side information and the second side information, which indicates the filter coefficients used in the nonlinear adaptive loop filter, are presented together in the same adaptive parameter set; and Wherein, when the video unit uses at least some of the filter coefficients associated with the adaptive parameter set, the video unit also uses a limiting parameter associated with the adaptive parameter set.

11. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: Perform a conversion between the bitstream of the video, which includes one or more video regions, and the video itself. in, The bitstream includes first-side information providing limiting parameters, which are used to filter the reconstruction of video units in a video region using a nonlinear adaptive loop filter; and The first side information and the second side information, which indicates the filter coefficients used in the nonlinear adaptive loop filter, are presented together in the same adaptive parameter set; and Wherein, when the video unit uses at least some of the filter coefficients associated with the adaptive parameter set, the video unit also uses a limiting parameter associated with the adaptive parameter set.

12. A non-transitory computer-readable recording medium storing a computer program / instructions and a bitstream thereon, wherein the computer program / instructions, when executed by a processor, implement a method for processing video data to generate the bitstream, the method comprising: Generate the bitstream of video comprising one or more video regions. The bitstream includes first-side information providing limiting parameters, which are used to filter the reconstruction of video units in a video region using a nonlinear adaptive loop filter; and The first side information and the second side information, which indicates the filter coefficients used in the nonlinear adaptive loop filter, are presented together in the same adaptive parameter set; and Wherein, when the video unit uses at least some of the filter coefficients associated with the adaptive parameter set, the video unit also uses a limiting parameter associated with the adaptive parameter set.

13. A method for storing a bitstream of video, comprising: Generate a bitstream of video that includes one or more video regions; as well as The bitstream is stored in a non-transitory computer-readable recording medium. The bitstream includes first-side information providing limiting parameters, which are used to filter the reconstruction of video units in a video region using a nonlinear adaptive loop filter; and The first side information and the second side information, which indicates the filter coefficients used in the nonlinear adaptive loop filter, are presented together in the same adaptive parameter set; and Wherein, when the video unit uses at least some of the filter coefficients associated with the adaptive parameter set, the video unit also uses a limiting parameter associated with the adaptive parameter set.

14. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instructions are executed by the processor, the processor causes the processor to perform the method according to any one of claims 2 to 9.

15. A non-transitory computer-readable medium storing instructions that, when executed, cause a processor to perform the method according to any one of claims 2 to 9.

Images