Clipping parameter derivation in adaptive loop filter

Abstract

This application relates to clipping parameter derivation in adaptive loop filters. Apparatuses, systems, and methods for time-domain prediction of parameters in non-linear adaptive loop filtering are described. In an example aspect, a method for visual media processing includes configuring one or more parameters of a clipping operation as part of a non-linear filtering operation for a current video block; and performing a conversion between the current video block and a bitstream representation of the current video block based on the one or more parameters, wherein the one or more parameters are coded according to a rule.

Description

[0001] This application is a divisional application of the invention patent application filed on April 15, 2020, with application number 202080028109.X and invention title "Derivation of Pruning Parameters in Adaptive Loop Filters".

[0002] Cross-reference to related applications

[0003] This application is made to claim priority and benefit in accordance with applicable patent law and / or the rules of the Paris Convention, International Patent Application No. PCT / CN2019 / 082626, filed on April 15, 2019. For all purposes under the law, the entire disclosure of the foregoing application is incorporated by reference as part of the disclosure of this application. Technical Field

[0004] This patent document relates to video encoding and decoding technologies, devices, and systems. Background Technology

[0005] Despite advancements in video compression, digital video accounts for the largest share of bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0006] This paper describes devices, systems, and methods related to digital video coding and decoding, and specifically, describes temporal prediction of parameters involved in nonlinear adaptive loop filtering. The described methods can be applied to existing video coding and decoding standards (e.g., High Efficiency Video Coding (HEVC)) and future video coding and decoding standards (e.g., Versatile Video Coding (VVC)) or codecs.

[0007] In one representative aspect, the disclosed technology can be used to provide a method for visual media processing. The method includes configuring one or more parameters for a clipping operation as part of a nonlinear filtering operation for a current video block; and performing a conversion between the current video block and its bitstream representation based on the one or more parameters, wherein the one or more parameters are encoded or decoded according to rules.

[0008] In another representative aspect, the disclosed technology can be used to provide a method for visual media processing. This method includes determining one or more parameters for a nonlinear filtering operation based on the characteristics of a current video block; and performing a conversion between the current video block and its bitstream representation based on the one or more parameters.

[0009] In another representative aspect, the disclosed technology can be used to provide a method for visual media processing. The method includes configuring one or more parameters for a clipping operation as part of a nonlinear filtering operation for a current video block; and performing a conversion between the current video block and a bitstream representation of the current video block based on the one or more parameters, wherein the one or more parameters are presented in the bitstream representation independently of the value of at least one filter coefficient associated with the nonlinear filtering operation.

[0010] In another representative aspect, the disclosed technology can be used to provide a method for visual media processing. The method includes configuring one or more parameters for a clipping operation as part of a nonlinear filtering operation for a current video block; and performing a conversion between the current video block and its bitstream representation based on the one or more parameters, wherein the current video block inherits filter coefficients from an i-th filter, and wherein a first rule associated with the inheritance of the one or more parameters of the clipping operation differs from a second rule associated with the inheritance of the filter coefficients.

[0011] In another representative aspect, the above methods are embodied in processor-executable code and stored in a computer-readable program medium.

[0012] In another representative aspect, video encoder devices can implement the methods described herein.

[0013] In another representative aspect, a video decoder device can implement the methods described herein.

[0014] The above and other aspects and features of the disclosed technology are described in more detail in the accompanying drawings, description and claims. Attached Figure Description

[0015] Figure 1 An example of an encoder block diagram for video encoding and decoding is shown.

[0016] Figure 2A , Figure 2B and Figure 2C An example of the filter shape for a geometry transformation-based adaptive loop filter (GALF) is shown.

[0017] Figure 3 An example flowchart of the GALF encoder decision is shown.

[0018] Figures 4A-4D An example subsampled Laplace calculation for adaptive loop filter (ALF) classification is shown.

[0019] Figure 5 An example of neighboring samples used in a bilateral filter is shown.

[0020] Figure 6 An example window showing two samples used in the coverage weight calculation is shown.

[0021] Figure 7 An example of a scan pattern is shown.

[0022] Figures 8A-8C A flowchart of an exemplary method for time-domain prediction of parameters in nonlinear adaptive loop filtering is shown.

[0023] Figure 9 This is a block diagram of an example hardware platform used to implement the video decoding or video encoding technologies described in this document.

[0024] Figure 10 This is a block diagram of an example video processing system in which the disclosed technology can be implemented.

[0025] Figure 11 A flowchart of an example method for visual media processing is shown.

[0026] Figure 12 A flowchart of an example method for visual media processing is shown.

[0027] Figure 13 A flowchart of an example method for visual media processing is shown.

[0028] Figure 14 A flowchart of an example method for visual media processing is shown. Detailed Implementation

[0029] Video encoding and decoding methods and technologies are ubiquitous in modern technology due to the increasing demand for higher resolution video. Video codecs typically consist of electronic circuitry or software that compresses or decompresses digital video and are constantly being improved to provide higher encoding and decoding efficiency. A video codec converts uncompressed video into a compressed format and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end latency (delay). Compression formats typically conform to standard video compression specifications, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), the pending Universal Video Coding (VVC) standard, or other current and / or future video codec standards.

[0030] Video codec standards have primarily evolved through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, while ISO / IEC developed MPEG-1 and MPEG-4. These two organizations jointly developed the H.262 / MPEG-2 video and H.264 / MPEG-4 Advanced Video Coding (AVC) standards, as well as the H.265 / HEVC standard. Since H.262, video codec standards have been based on a hybrid video codec architecture, utilizing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Group (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was established to work on the Universal Video Coding (VVC) standard, with the goal of reducing the bit rate by 50% compared to HEVC.

[0031] Implementations of the disclosed techniques can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve runtime performance. Section headings are used in this document to improve readability and are not in any way intended to limit the discussion or embodiments (and / or implementations) to the individual sections.

[0032] 1. Examples of color spaces and chromaticity subsampling

[0033] A color space, also known as a color model (or color system), is an abstract mathematical model that simply describes the range of colors as tuples of numbers, typically 3 or 4 values ​​or color components (e.g., RGB). Essentially, a color space is an elaboration of a coordinate system and its subspaces.

[0034] For video compression, the most commonly used color spaces are YCbCr and RGB.

[0035] YCbCr, Y'CbCr, or Y Pb / Cb Pr / Cr, also known as YBCCR or Y'CBCR, is a color space family used as part of the color image pipeline in video and digital photography systems. Y' is the luminance component, while CB and CR are the blue-difference and red-difference chromaticity components. Y' (which has a primary color) differs from Y, where Y is luminance, meaning the light intensity is non-linearly encoded based on gamma-corrected RGB primary colors.

[0036] Chromaticity subsampling is a practice that utilizes the fact that the human visual system is less sensitive to color differences than to brightness, and encodes images by applying a smaller resolution to chromaticity information than to brightness information.

[0037] 1.1 4:4:4 color format

[0038] Each of the three Y'CBcr components has the same sampling rate, so there is no chroma subsampling. This scheme is sometimes used in high-end cinema scanners and film post-production.

[0039] 1.2 4:2:2 color format

[0040] The two chroma components are sampled at half the luminance sampling rate; for example, the horizontal chroma resolution is halved. This reduces the bandwidth of the uncompressed video signal by one-third, with almost no parallax.

[0041] 1.3 4:2:0 color format

[0042] In 4:2:0, horizontal sampling is doubled compared to 4:1:1, but in this scheme, the Cb and Cr channels are sampled only on each alternating line, thus halving the vertical resolution. Therefore, the data rate is the same. Cb and Cr are subsampled by a factor of 2 in both the horizontal and vertical directions. There are three variations of the 4:2:0 scheme, each with different horizontal and vertical site selections.

[0043] In MPEG-2, Cb and Cr are horizontally cosited. Cb and Cr are located between pixels in the vertical direction (in the gaps).

[0044] In JPEG / JFIF, H.261, and MPEG-1, Cb and Cr are located in the gaps, in the middle between alternating luminance samples.

[0045] In a 4:2:0 DV configuration, Cb and Cr are co-located horizontally. Vertically, they are co-located alternately.

[0046] 2. Example of a typical video codec's encoding / decoding process

[0047] Figure 1 An example of a VVC encoder block diagram is shown, comprising three loop filtering blocks: a deblocking filter (DF), a sample adaptive offset (SAO), and an ALF. Unlike the DF, which uses predefined filters, the SAO and ALF utilize the raw samples of the current image, reducing the mean square error between the raw and reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively. The encoding / decoding side information signaling informs the offset and filter coefficients. The ALF is located at the final processing stage of each image and can be viewed as a tool attempting to capture and repair artifacts generated in previous stages.

[0048] 3 Examples of Geometric Transformation-Based Adaptive Loop Filters in JEM

[0049] In JEM, a geometric transformation-based adaptive loop filter (GALF) with block-based filter adaptation is applied. For the luminance component, one of 25 filters is selected for each 2×2 block based on the direction and activity of the local gradient.

[0050] 3.1 Examples of filter shapes

[0051] In JEM, up to three diamond filter shapes can be selected for the luminance component (e.g., Figure 2A , Figure 2B and Figure 2C As shown, these are 5×5, 7×7, and 9×9 rhombuses, respectively. The image-level signaling notification index indicates the filter shape used for the luminance component. For the chrominance component in an image, a 5×5 rhombus shape is always used.

[0052] 3.1.1 Block Classification

[0053] Each 2×2 block is classified into one of 25 classes. The classification index C is based on its directionality. and activities The quantization value is derived as follows:

[0054]

[0055] In order to calculate and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using the 1-D Laplacian operator:

[0056]

[0057] index and This refers to the coordinates of the top-left sample point in the 2×2 block, and Indicator coordinates Reconstructed sample points at the location.

[0058] Then, the gradients in the horizontal and vertical directions The maximum and minimum values ​​are set as follows:

[0059]

[0060] Furthermore, the maximum and minimum values ​​of the gradients in the two diagonal directions are set as follows:

[0061]

[0062] In order to derive directionality The values ​​are compared with each other and then compared with two thresholds. and Comparison:

[0063] Step 1. If and If both are true, then Set to 0.

[0064] Step 2. If If yes, continue from step 3; otherwise, continue from step 4.

[0065] Step 3. If ,but Set to 2; otherwise Set to 1.

[0066] Step 4. If ,but Set to 4; otherwise Set it to 3.

[0067] Activity Value Calculated as:

[0068]

[0069] It is further quantized to the range of 0 to 4 (inclusive), and the quantized value is represented as .

[0070] For the two chromaticity components in the image, no classification method is applied; that is, a single set of ALF coefficients is applied to each chromaticity component.

[0071] 3.1.2 Geometric Transformation of Filter Coefficients

[0072] Before filtering each 2×2 block, the filter coefficients are adjusted based on the gradient values ​​calculated for that block. Apply geometric transformations, such as rotation, diagonal flip, and vertical flip. This is equivalent to applying these transformations to samples within the filter's support region. The idea is to make these different blocks more similar by aligning their orientations.

[0073] Three geometric transformations are introduced: diagonal, vertical flip, and rotation.

[0074]

[0075] here, It is the size of the filter, and These are coefficient coordinates, which make the position... In the top left corner, and in position In the bottom right corner. Depending on the gradient values ​​calculated for that block, the transformation is applied to the filter coefficients. Table 1 summarizes the relationship between the transformation and the four gradients in the four directions.

[0076] Table 1: Mapping of gradients and transformations calculated for a block

[0077]

[0078] 3.1.3 Signaling of Filter Parameters

[0079] In JEM, the GALF filter parameters are signaled to the first CTU (coding tree block), i.e., after the stripe header and before the SAO parameters of the first CTU. Up to 25 sets of luminance filter coefficients can be signaled. To reduce bit overhead, filter coefficients from different categories can be merged. Furthermore, the GALF coefficients of the reference image are stored and can be reused as the GALF coefficients of the current image. The current image can optionally use the GALF coefficients stored for the reference image and bypass the GALF coefficient signaling. In this case, only the index of one of the reference images is signaled, and the stored GALF coefficients of the indicated reference image are inherited by the current image.

[0080] To support GALF temporal prediction, a candidate list of GALF filter sets is maintained. The candidate list is empty when decoding a new sequence begins. After an image is decoded, the corresponding filter set can be added to the candidate list. Once the size of the candidate list reaches the maximum allowed value (i.e., 6 in the current JEM), new filter sets overwrite the oldest filter sets in decoding order; that is, a first-in-first-out (FIFO) rule is applied to update the candidate list. To avoid duplication, a filter set is added to the list only if the corresponding image does not use GALF temporal prediction. To support temporal scaling, there are multiple candidate lists of filter sets, and each candidate list is associated with a temporal layer. More specifically, each array assigned by a temporal layer index (TempIdx) can form the filter set of previously decoded images whose TempIdx is equal to a lower TempIdx. For example, the k-th array is assigned to be associated with a TempIdx equal to k, and it contains only filter sets from images whose TempIdx is less than or equal to k. After a specific image is encoded or decoded, the filter set associated with that image will be used to update those arrays associated with TempIdx equal to or higher.

[0081] Temporal prediction of GALF coefficients is used for inter-frame encoding / decoding to minimize signaling overhead. For intra-frame frames, temporal prediction is not available, and a set of 16 fixed filters is assigned to each class. To indicate the use of fixed filters, signaling is used to notify each class of its flag, and, if necessary, the index of the selected fixed filter. Even when a fixed filter is selected for a given class, the coefficients f(k, l) of an adaptive filter can still be sent for that class; in this case, the coefficients of the filter to be applied to the reconstructed image are the sum of the two sets of coefficients.

[0082] The filtering process for the luminance component can be controlled at the CU (coding unit) level. Signaling flags indicate whether GALF is applied to the luminance component of the CU. For the chrominance component, the application of GALF is indicated only at the image level.

[0083] 3.1.4 Filtering Process

[0084] On the decoder side, when GALF is enabled for a block, for each sample within the block... Filtering is performed to obtain sample values. As shown below, where L represents the filter length. Represents the filter coefficients, and This represents the coefficients of the decoding filter.

[0085]

[0086] 3.1.5 Determination process of encoder-side filter parameters

[0087] Figure 3 The entire encoder decision process for GALF is illustrated. For each luminance sample in a CU, the encoder decides whether to apply GALF, and the appropriate signaling flags are included in the strip header. For chroma samples, the decision to apply the filter is made at the picture level, not the CU level. Furthermore, chroma GALF for the picture is only checked if luminance GALF is enabled for the picture.

[0088] 4. Examples of Geometric Transformation-Based Adaptive Loop Filters in VVC

[0089] Compared to JEM, the current design of GALF in VVC has the following main changes:

[0090] 1) Adaptive filter shapes have been removed. Only 7×7 filter shapes are allowed for the luminance component, while 5×5 filter shapes are allowed for the chrominance component.

[0091] 2) The time-domain predictions of the ALF parameters and the predictions from the fixed filter are both removed.

[0092] 3) For each CTU, the signaling notifies a bit of whether ALF is enabled or disabled.

[0093] 4) Class index calculation is performed at a 4×4 level instead of a 2×2 level. Furthermore, as proposed in JVET-L0147, a subsampling Laplacian calculation method used for ALF classification is employed. More specifically, it is not necessary to calculate the horizontal / vertical / 45-degree diagonal / 135-degree gradient for each sample point within a block. Instead, 1:2 subsampling is utilized.

[0094] In VTM4.0, the filtering process of the adaptive loop filter is performed as follows:

[0095]

[0096] Among them, sample points These are the input samples. These are the filtered output samples (i.e., the filtering result), and This represents the filter coefficients. In fact, VTM 4.0 uses integer algorithms to perform fixed-point precision calculations:

[0097]

[0098] Where L represents the filter length, and where, These are the filter coefficients with fixed-point precision.

[0099] 5. Nonlinear Adaptive Loop Filtering (ALF) in JVET-N0242

[0100] 5.1 Filter Reformation

[0101] Equation (11) can be revised with the following expression without affecting encoding and decoding efficiency:

[0102]

[0103] here, These are the same filter coefficients as those in equation (11) [except for] It equals 1 in equation (13), and it equals 1 in equation (11). ].

[0104] 5.2 Modified Filter

[0105] Using the filter formula in (13) above, when the neighboring sample values With the current sample value being filtered When the difference is too large, we can reduce the influence of these neighboring sample values ​​by using a simple clip function, thereby introducing a simple nonlinearity and making AFL more efficient.

[0106] In this proposal, the ALF filter is modified as follows:

[0107]

[0108] here, It is a limiting function, and It is the limiting parameter, which depends on Filter coefficients. The encoder performs optimization to find the optimal values. Note that the implementation of integer precision uses rounding. The shift.

[0109] In the JVET-N0242 implementation, a limiting parameter is specified for each ALF filter. For each filter coefficient, a limiting value is signaled. This means that up to 12 limiting values ​​can be signaled in the bitstream for each luminance filter, and up to 6 limiting values ​​can be signaled for each chrominance filter.

[0110] To limit signaling costs and encoder complexity, we restrict the evaluation of the clipping value to a small set of possible values. In this proposal, we use only four fixed values, which are the same for both INTER and INTRA tilegroups.

[0111] Because the variance of local differences in luminance is typically higher than that in chrominance, we use two different sets of filters for luminance and chrominance. We also include the maximum sample value in each set (here, 1024 for a bit depth of 10 bits), so clipping can be disabled if not necessary.

[0112] Table 2 provides a set of limiting values ​​used in the JVET-N0242 test. These four values ​​were selected by dividing the entire range of luminance sample values ​​(encoded in 10 bits) and the chrominance range from 4 to 1024 in the logarithmic domain into approximately equal parts.

[0113] More precisely, the brightness table for the limit value is obtained using the following formula:

[0114] AlfClip L Where M=2 10 N=4.

[0115] Similarly, the colorimetric table for the limiting values ​​is obtained according to the following formula:

[0116] AlfClip C Where M=2 10 N=4 and A=4.

[0117] Table 2: Authorized Limit Values

[0118]

[0119] The selected threshold value is encoded and decoded in the "alf_data" syntax element using the Golomb coding scheme corresponding to the index of the threshold value in Table 2 above. This coding scheme is the same as that used for the filter index.

[0120] 5.2.1 Syntax and Semantics

[0121] Due to NLALF, the newly introduced syntax changes (shown below in bold, italics, and underline) are as follows:

[0122] 7.3.4.3 Adaptive Loop Filter Data Syntax

[0123]

[0124]

[0125]

[0126] 6. CTU-based ALF in JVET-N0427

[0127] VTM4 employs an Adaptive Parameter Set (APS). Each APS contains a set of ALF filters for signaling notification, with a maximum of 32 APSs supported. This proposal tests strip-level time-domain filters. Slice groups can reuse ALF information from APSs to reduce overhead. APSs are updated as a first-in-first-out (FIFO) buffer.

[0128] For the luma component, when the ALF is applied to the luma CTB, it indicates a choice between 16 fixed filter sets, 5 time-domain filters, or 1 signaled filter set (signaled at the strip level). Only the filter set index is signaled. For a strip, only a new set of 25 filters can be signaled. If a new set is signaled for a strip, all luma CTBs in the same strip share that set. The fixed filter set can be used to predict new strip-level filter sets or as a candidate filter set for the luma CTB. The total number of filters is 64.

[0129] For chroma components, when the ALF is applied to the chroma CTB, if a new filter is notified by strip signaling, the CTB uses the new filter; otherwise, the nearest temporal chroma filter that satisfies the temporal scaling constraint is applied.

[0130] As a strip-level time-domain filter, APS is updated as a first-in-first-out (FIFO) buffer.

[0131] 7. Post-reconstruction filter

[0132] 7.1 Diffusion Filter (DF)

[0133] In JVET-L0157, a spread filter was proposed, in which the intra / inter-frame prediction signal of the CU can be further modified by the spread filter.

[0134] Uniform diffusion filter. The uniformly diffused filter is implemented by convolving the predicted signal with a fixed mask, which is given as... or , defined as follows.

[0135] In addition to the predicted signal itself, a row of reconstructed samples on the left and top of the block is used as input to the filtered signal, where the use of these reconstructed samples can be avoided in inter-frame blocks.

[0136] make It is the predicted signal over a given block obtained through intra-frame or motion-compensated prediction. To handle filter boundary points, the predicted signal needs to be extended to... This extended prediction can be formed in two ways:

[0137] Alternatively, as an intermediate step, a row of reconstructed samples from the left and top of the block is added to the prediction signal, and the resulting signal is then mirrored in all directions. Or, only the prediction signal itself is mirrored in all directions. The latter expansion is used for inter-frame blocks. In this case, only the prediction signal itself includes the expanded prediction signal. Input.

[0138] If you want to use a filter The following formula is proposed to replace the prediction signal.

[0139]

[0140] Using the aforementioned boundary extension. Here, the filter mask... Give as

[0141]

[0142] If you want to use a filter The following formula is proposed to replace the prediction signal.

[0143]

[0144] Here, the filter Give as

[0145]

[0146] Directional diffusion filter. Instead of using an adaptive signal diffusion filter, a directional filter and a horizontal filter with a fixed mask are used. and vertical filter More precisely, compared to the mask in the previous chapter. The corresponding uniform diffusion filter is simply restricted to being applied only in the vertical or horizontal direction. The vertical filter is implemented by applying a fixed filter mask to the predicted signal, and the horizontal filter is implemented by using a transposed mask. To achieve this, the fixed filter mask is as follows:

[0147]

[0148] 7.2 Bilateral filter (BF)

[0149] A bilateral filter is proposed in JVET-L0406, and it is always applied to luma blocks with non-zero transform coefficients and stripe quantization parameters greater than 17. Therefore, no signaling notification is required for the use of the bilateral filter. If a bilateral filter is applied, it is applied to the decoded samples immediately after the inverse transform. Furthermore, the filter parameters (i.e., weights) are explicitly derived from the encoding / decoding information.

[0150] The filtering process is defined as follows:

[0151]

[0152] here, It is the intensity of the current sample point, and It is the correction intensity of the current sample point. and These are the intensity and weighting parameters of the k-th neighboring sample point, respectively. Figure 5 The example depicts a current sample point and its four neighboring sample points (i.e., K=4).

[0153] More specifically, the weights associated with the k-th neighboring sample. The definition is as follows:

[0154]

[0155] here,

[0156] and .

[0157] here, This depends on the encoding / decoding mode and the codec block size. When the TU is further divided, the described filtering process is applied to intra-frame and inter-frame codec blocks to achieve parallel processing.

[0158] To better capture the statistical characteristics of the video signal and improve the performance of the filter, the weighting function derived from equation (2) is... The parameters are adjusted as shown in Table 4. It depends on the encoding / decoding mode and block segmentation parameters (minimum size).

[0159] Table 4: Different block sizes and encoding / decoding modes value

[0160]

[0161] To further improve encoding and decoding performance, when the TU is not partitioned, for an inter-frame codec block, the intensity difference between the current sample and one of its neighboring samples is replaced by the representative intensity difference between two windows covering the current sample and its neighboring samples. Therefore, the equation for the filtering process is modified as follows:

[0162]

[0163] here, and They represent respectively with and The value of the m-th sample point within the centered window. In this proposal, the window size is set to 3×3. Figure 6 The text describes the coverage. and An example of two windows.

[0164] 7.3 Hadamard transform domain filter (HF)

[0165] In JVET-K0068, a loop filter in the 1D Hadamard transform domain is applied at the CU level after reconstruction and has a multiplication-free implementation. The proposed filter is applied to all CU blocks that satisfy predefined conditions, and the filter parameters are derived from the encoding / decoding information.

[0166] Excluding 4×4 blocks, and if the stripe quantization parameter is greater than 17, the proposed filter is always applied to the lumen reconstruction block with non-zero transform coefficients. The filter parameters are explicitly derived from the encoding / decoding information. If the proposed filter is applied, it is executed on the decoded samples immediately after the inverse transform.

[0167] For each pixel from the reconstructed block, pixel processing includes the following steps:

[0168] ○ Scan the four neighboring pixels surrounding the current pixel according to the scanning mode.

[0169] ○ Read the 4-point Hadamard transform of the pixel

[0170] ○ Spectral filtering based on the following formula:

[0171]

[0172] Here, (i) is the index of the spectral component in the Hadamard spectrum, R(i) is the spectral component of the reconstructed pixel corresponding to the index, and σ is the filter parameter derived from the codec quantization parameter QP using the following equation:

[0173] .

[0174] Figure 7 The image shows an example of a scanning pattern, where A is the current pixel and {B, C, D} are the surrounding pixels.

[0175] For pixels located on the CU boundary, the scan mode is adjusted to ensure that all necessary pixels are within the current CU.

[0176] 8. Virtual Pipelining Data Unit (VPDU)

[0177] Virtual Pipeline Data Units (VPDUs) are defined as non-overlapping MxM-luminance (L) / NxN-chrominance (C) units in an image. In a hardware decoder, consecutive VPDUs are processed simultaneously by multiple pipeline stages; different stages process different VPDUs simultaneously. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so maintaining the VPDU size is important. In the HEVC hardware decoder, the VPDU size is set to the maximum transform block (TB) size. Increasing the maximum TB size from 32×32-L / 16×16-C (as in HEVC) to 64×64-L / 32×32-C (as in current VVC) can result in encoding / decoding gains, which is expected to produce a 4x increase in VPDU size (64×64-L / 32×32-C) compared to HEVC. However, in addition to quadtree (QT) codec unit (CU) partitioning, VVC also employs ternary tree (TT) and binary tree (BT) partitioning to achieve additional codec gain, and TT and BT partitioning can be recursively applied to 128×128-L / 64×64-C codec unit (CTU), which is said to result in a 16x VPDU size (128×128-L / 64×64-C) compared to HEVC.

[0178] In the current design of VVC, the VPDU size is defined as 64×64-L / 32×32-C.

[0179] 9. Disadvantages of existing implementation methods

[0180] The non-linear ALF (NLALF) design in JVET-N0242 has the following problems:

[0181] (1) The classification process of GALF relies on gradients and the Laplacian activity of reconstructed samples before applying ALF. However, the classification results may be inaccurate. For example, for one sample, the difference between it and its neighbors may be quite similar, while for another sample, the difference between it and a neighbor may be too different, and for all other samples, the difference may be too small. In both cases, they may be classified into a class index that may be unreasonable.

[0182] (2) The limiting parameter is associated with the filter coefficients. However, due to the filter merging process, a single filter can be used for multiple classes. Furthermore, for two blocks with the same class index (0…24 in the current GALF design), the filter coefficients and the limiting parameter are identical. However, these two blocks can have different characteristics; for example, the selected geometric transformations can be different. Using the same limiting parameter may be suboptimal.

[0183] (3) The index of the limiting parameter is signaled for each non-zero filter coefficient, which requires constructing the filter coefficients during the parse phase. This design is undesirable for hardware implementations.

[0184] (4) During temporal prediction, a block can inherit filter coefficients from previously encoded / decoded frames. How to handle the clipping parameter needs to be investigated.

[0185] (5) Limiting function Where b is the upper limit, -b is the lower limit, and d is the input. Setting the upper and lower limits to be equal might be suboptimal.

[0186] 10. An exemplary method for time-domain prediction of parameters in nonlinear ALF

[0187] The embodiments of the currently disclosed technology overcome the shortcomings of existing implementations, thereby providing video codecs with higher encoding and decoding efficiency. Based on the disclosed technology, temporal prediction of parameters in nonlinear adaptive loop filtering can enhance existing and future video codec standards, as illustrated in the examples described below for various implementations. The examples of the disclosed technology provided below illustrate general concepts and are not intended to be limiting. In one example, the various features described in these examples can be combined unless the contrary is explicitly stated.

[0188] In these examples, a filter can be associated with multiple filter coefficients. A filter set represents multiple filters. (By F) i Let F represent the i-th filter, and let F be the filter of i-th type. i k represents its associated filter coefficient, such as the variable k representing the relationship with F. i The associated k-th filter coefficient, for example, can correspond to Ck in Figure 2.

[0189] 1. Propose determining NLALF parameters (e.g., on / off control flags, limiting parameters) based on encoding and decoding information.

[0190] a. NLALF parameters (e.g., on / off control flags, limiting parameters) can depend on codec mode information.

[0191] i. In one example, which NLALF parameters to select can be determined by the encoding / decoding mode (such as intra-frame or non-intra-frame mode).

[0192] ii. In one example, which NLALF parameters to select can be determined by the encoding / decoding mode (such as intra-frame or inter-frame mode).

[0193] iii. In one example, which NLALF parameters to select can be determined by the codec mode (such as IBC or non-IBC mode).

[0194] b. NLALF parameters (e.g., on / off control flags, limiting parameters) can depend on the transformation information.

[0195] i. In one example, these parameters may depend on whether a transformation skip is applied.

[0196] c. NLALF parameters (e.g., on / off control flags, limiting parameters) can depend on residual information.

[0197] i. In one example, these parameters may depend on whether the block contains non-zero coefficients.

[0198] d. NLALF parameters (e.g., on / off control flags, limiting parameters) can depend on the group type / picture type.

[0199] e. NLALF parameters (e.g., on / off control flags, limiting parameters) can depend on temporal layer information / reference image information associated with a slice / group of slices / strip, etc.

[0200] i. In one example, these parameters may depend on whether all reference images are associated with a smaller POC value than the current image.

[0201] ii. In one example, these parameters may depend on whether all reference images are associated with a POC value that is smaller than or equal to the current image.

[0202] f. Propose determining NLALF parameters (e.g., on / off control flags, limiting parameters) based on reference images / motion information associated with a block.

[0203] 2. Propose determining NLALF parameters (e.g., on / off control flags, limiting parameters) based on geometric transformations.

[0204] a. In one example, for two MxN blocks, even if these parameters are associated with the same filter (e.g., due to the same class index), the associated NLALF parameters (e.g., the limiting parameters) can be different.

[0205] b. In one example, for a single filter coefficient, signaling can be used to indicate more than one limiting parameter.

[0206] i. In one example, the number of limiting parameters / or the index of the limiting parameters or other representations of the limiting parameters may depend on the number of allowed geometric transformations.

[0207] ii. In one example, predictive encoding / decoding can be applied to a limiting parameter / limiting parameter index associated with a filter parameter.

[0208] 1) In one example, the limiting parameter of a filter for a sample or block can be predicted by another limiting parameter of another filter for adjacent or non-adjacent samples or blocks in the spatial / temporal domain.

[0209] 3. It is proposed that the upper and lower limits of the amplitude limiting function can be unequal.

[0210] a. In one example, a signaling instruction can be used to notify an upper and lower limit of a limiting function.

[0211] i. Alternatively, predictive coding and decoding between the upper and lower bounds can also be applied.

[0212] 4. Propose direct encoding and decoding of fixed-length indications of limiting parameters (such as indexes).

[0213] a. In one example, each of these parameters can be encoded or decoded in N bits (e.g., N is set to 2).

[0214] i. In one example, N can be fixed;

[0215] ii. In one example, N can be notified by signaling;

[0216] iii. In one example, N can depend on encoding / decoding information such as QP, image size, etc.

[0217] b. Alternatively, these parameters can be encoded and decoded using a truncated unary method with the maximum value N.

[0218] i. In one example, N can be fixed;

[0219] ii. In one example, N can be notified by signaling;

[0220] iii. In one example, N can depend on encoding / decoding information such as QP, image size, etc.

[0221] c. Alternatively, for a filter / a set of filters, these parameters can be encoded and decoded in a fixed order using the Exponential-Golomb method.

[0222] d. Alternatively, these parameters can be encoded using run-length coding.

[0223] i. In one example, for each filter, the index of the limiting parameter can first be encoded as a “run”, and the number of consecutive identical limiting parameters is encoded as a “length”.

[0224] ii. In one example, for each k-th filter coefficient in all filters, with F i The index of the associated limiting parameter can first be encoded as a "run", and the number of the same limiting parameters in other filters is encoded as a "length".

[0225] e. In one example, predictive encoding and decoding can be applied with indications of limiting parameters (such as indexes).

[0226] i. In one example, predictive encoding / decoding can be applied to the limiting parameter within a filter.

[0227] ii. In one example, predictive encoding and decoding can be applied to the limiting parameters of different filters.

[0228] 1) In one example, predictive encoding and decoding can be applied to the limiting parameters of different filters for a color component.

[0229] 2) In one example, predictive encoding and decoding can be applied to the limiting parameters of different filters for multiple color components.

[0230] 3) In one example, predictive encoding and decoding can be applied to the limiting parameters of filters used for different points or blocks.

[0231] iii. In one example, predictive encoding and decoding can be applied to the clipping parameters of signaling notifications in different APSs.

[0232] 5. A method for decoupling the analysis of the limiting parameter and the construction of the filter coefficients is proposed.

[0233] a. In one example, the resolution of the limiting parameter (e.g., the index of the limiting parameter) is independent of the values ​​of the filter coefficients.

[0234] b. In one example, when the filter coefficients are equal to 0, the indication of the associated limiting parameters can still be signaled.

[0235] 6. When a block inherits filter coefficients from the i-th filter, the limiting parameters associated with the i-th filter may not be inherited.

[0236] a. In one example, when time-domain prediction is enabled for a block, signaling can be used to inform whether non-local ALF (applying clipping) is used, without directly inheriting the associated clipping parameters.

[0237] i. In one example, if it is determined that a limiting effect is to be applied, the associated limiting parameters can also be inherited.

[0238] b. In one example, assuming that filter coefficients are inherited / predicted from the i-th filter and clipping parameters are inherited / predicted from the j-th filter, then i may not be equal to j.

[0239] c. In one example, assuming filter coefficients are inherited / predicted from the i-th filter and clipping parameters are inherited / predicted from the j-th filter, then the i-th filter and the j-th filter can be associated with different filter sets.

[0240] i. In one example, the i-th filter can be associated with the first picture / slice group / slice / strip, and the j-th filter can be associated with the second picture / slice group / slice / strip.

[0241] ii. In one example, i is not equal to j. Alternatively, i is equal to j.

[0242] d. In one example, signaling can be used to indicate which filter the limiting parameter is associated with, such as the filter index.

[0243] e. In one example, signaling can be used to indicate which set of filters is associated with the limiting parameter, such as the APS index.

[0244] i. Alternatively, in addition, the filter index can be further signaled.

[0245] 7. During the classification process, the sample difference with a limited amplitude can be used instead of the sample difference directly.

[0246] a. In one example, during gradient calculation, either a limited sample difference or a limited gradient can be used.

[0247] b. In one example, during the active calculation process, the amplitude-limited sample difference or amplitude-limited gradient can be used.

[0248] c. In one example, the following equation can be used to calculate the vertical gradient:

[0249]

[0250] Clip1 and Clip2 are two clipping functions.

[0251] d. In one example, the following equation can be used to calculate the horizontal gradient:

[0252]

[0253] Clip1 and Clip2 are two clipping functions.

[0254] 8. Whether or not to apply amplitude limiting can depend on the location of the samples to be used in the filtering process (such as in section 5.2). ).

[0255] a. In one example, clipping can be disabled if the samples in the filter support are not located at the boundaries of CU / PU / TU / picture / slice / slice group.

[0256] b. In one example, if the sample points in the filter support are located at the boundaries of CU / PU / TU / picture / piece / piece group / CTU / virtual pipeline data unit (VPDU), then clipping can be applied.

[0257] c. Alternatively, whether to apply clipping can depend on the distance of the samples to be used during the filtering process from the boundaries of the CU / PU / TU / image / slice / slice group / CTU / VPDU (as described in Section 5.2). ).

[0258] i. In one example, the distance can be predefined (e.g., N pixels).

[0259] ii. In one example, distance can be notified by signaling.

[0260] 9. The filter shape (also known as filter support) used in the adaptive loop filter process can depend on the color representation.

[0261] a. In one example, when the color format is 4:4:4, the filter support for all components (such as Y, Cb, Cr) should be the same.

[0262] i. For example, the filter supports a 7×7 rhombus shape, such as Figure 2B As shown.

[0263] ii. For example, the filter supports a 5×5 rhombus shape, such as Figure 2A As shown.

[0264] b. In one example, the supported regions for all components when the color format is RGB.

[0265] i. For example, the filter supports a 7×7 rhombus shape, such as Figure 2BAs shown.

[0266] ii. For example, the filter supports a 5×5 rhombus shape, such as Figure 2A As shown.

[0267] The examples above can be incorporated into the context of the methods described below (e.g., methods 800, 810, and 820), which can be implemented at a video decoder or video encoder.

[0268] Figure 8A A flowchart of an exemplary method for video processing is shown. Method 800 includes, in step 802, determining one or more parameters for a nonlinear filtering operation based on the characteristics of the current video block.

[0269] Method 800 includes, in step 804, performing a conversion between the current video block and the bitstream representation of the current video block based on one or more parameters.

[0270] In some embodiments, the characteristic of the current video block is the encoding / decoding mode of the current video block. In the example, the encoding / decoding mode of the current video block is intra-frame mode, non-intra-frame mode, intra-block copy (IBC) mode, or non-IBC mode.

[0271] In some embodiments, the feature is transformation information. In an example, the transformation information includes an indication of transformation skipping applied to the current video block.

[0272] In some embodiments, the characteristic is residual information. In the example, the residual information includes zero-value coefficients in the current video block.

[0273] In some embodiments, the feature is a slice group type or an image type that includes a slice group or image of the current video block.

[0274] In some embodiments, the features are temporal layer information or reference information associated with a slice, group of slices, picture, or strip that includes the current video block.

[0275] In some embodiments, the feature is a reference image or motion information associated with the current video block.

[0276] In some embodiments, the property is a geometric transformation.

[0277] In some embodiments, one or more parameters include an on / off control flag or one or more parameters of a limiting function.

[0278] In some embodiments, the upper bound of the clipping function differs from the lower bound. In the example, predictive encoding / decoding is applied between the upper and lower bounds of the clipping function.

[0279] Figure 8BA flowchart of an exemplary method for video processing is shown. Method 810 includes, in step 812, configuring one or more parameters for a clipping operation as part of a nonlinear filtering operation for the current video block.

[0280] Method 810 includes, in step 814, performing a conversion between the current video block and its bitstream representation based on one or more parameters.

[0281] In some embodiments, one or more parameters are encoded and decoded with a fixed length of N bits. In other embodiments, one or more parameters are encoded and decoded with a maximum value of N using a truncated unary method. In the example, N is fixed. In another example, N is signaled. In yet another example, N is based on the encoding and decoding information of the current video block, wherein the encoding and decoding information includes quantization parameters or the size of the image including the current video block.

[0282] In some embodiments, for a filter or filter set, one or more parameters are encoded and decoded in a fixed order using the exponential-Columbus method.

[0283] In some embodiments, one or more parameters are encoded and decoded using run-length encoding and decoding.

[0284] In some embodiments, one or more parameters are signaled independently of the value of at least one filter coefficient.

[0285] In some embodiments, one or more parameters further include filter coefficients, wherein the current video block inherits filter coefficients from the i-th filter, and wherein one or more parameters of the clipping function are inherited from the j-th filter, which is different from the i-th filter.

[0286] Figure 8C A flowchart of an exemplary method for video processing is shown. Method 820 includes, in step 822, configuring a nonlinear filtering operation including a clipping operation.

[0287] Method 820 includes, in step 824, performing a conversion between the current video block and its bitstream representation based on a nonlinear filtering operation.

[0288] In some embodiments, method 820 further includes the step of performing a gradient calculation process, wherein the gradient calculation process uses a clipped sample difference or a clipped gradient generated using a clipping operation. In other embodiments, method 820 further includes the step of performing an active calculation process, wherein the active calculation process uses a clipped sample difference or a clipped gradient generated using a clipping operation. In the example, the clipped gradient includes a vertical gradient, wherein the vertical gradient is calculated as follows: In another example, the gradient for amplitude limiting includes a horizontal gradient, which is calculated as follows: , where clip1 and clip2 are the first and second clipping functions, respectively.

[0289] In some embodiments, performing the conversion includes filtering one or more samples of the current video block, and the operation of the clipping operation is configured based on the position of one or more samples.

[0290] In some embodiments, the location of one or more sample points is the boundary of a codec unit (CU), prediction unit (PU), transform unit (TU), picture, slice, slice group, codec tree unit (CTU), or virtual pipeline data unit (VPDU).

[0291] In some embodiments, within the context of methods 800, 810, and 820, the shape of the filter used in the nonlinear filtering operation is based on a color representation. In examples, the color representation includes a 4:4:4 color format or an RGB color format. In another example, the filter is a diamond-shaped filter.

[0292] In some embodiments, within the context of methods 800, 810, and 820, the nonlinear filtering operation is a nonlinear adaptive loop filtering operation.

[0293] 11. Example implementations of the disclosed technologies

[0294] Figure 9 This is a block diagram of a video processing apparatus 900. Apparatus 900 can be used to implement one or more of the methods described herein. Apparatus 900 can be embodied in a smartphone, tablet computer, computer, Internet of Things (IoT) receiver, etc. Apparatus 900 may include one or more processors 902, one or more memories 904, and video processing hardware 906. The processors (multiple) 902 can be configured to implement one or more methods described in this document (including, but not limited to, methods 800, 810, and 820). The memories (multiple memories) 904 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 906 can be used to implement some of the techniques described in this document in hardware circuitry.

[0295] In some embodiments, the video encoding / decoding method may use, as referenced Figure 9 The device described herein is implemented on a hardware platform.

[0296] Figure 10This is a block diagram illustrating an example video processing system 1000 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 1000. System 1000 may include an input 1002 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10-bit multi-component pixel values, or in a compressed or encoded format. Input 1002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, passive optical network (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0297] System 1000 may include codec component 1004, which may implement the various codec or encoding methods described in this document. Codec component 1004 may reduce the average bit rate of the video from input 1002 to output of codec component 1004 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 1004 may be stored or transmitted via connected communication, as shown in component 1006. Component 1008 may use the bitstream (or codec) representation of the stored or transmitted video received at input 1002 to generate pixel values ​​or displayable video sent to display interface 1010. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that codec tools or operations are used at the encoder and will be performed by the decoder as corresponding decoding tools or operations that are the reverse of the codec results.

[0298] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE, etc. The technologies described in this document can be found in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0299] Figure 11A flowchart of an example method for visual media processing is shown. The steps of this flowchart are discussed in conjunction with Example 4 in Section 10 of this document. In step 1102, the process configures one or more parameters for a clipping operation as part of a nonlinear filtering operation for the current video block. In step 1104, the process performs a conversion between the current video block and its bitstream representation based on one or more parameters, wherein the one or more parameters are encoded or decoded according to rules.

[0300] Figure 12 A flowchart of an example method for visual media processing is shown. The steps of this flowchart are discussed in conjunction with Example 1 in Section 10 of this document. In step 1202, the process determines one or more parameters for a nonlinear filtering operation based on the characteristics of the current video block. In step 1204, the process performs a conversion between the current video block and its bitstream representation based on one or more parameters.

[0301] Figure 13 A flowchart of an example method for visual media processing is shown. The steps of this flowchart are discussed in conjunction with Example 5 in Section 10 of this document. In step 1302, the process configures one or more parameters for a clipping operation as part of a nonlinear filtering operation for the current video block. In step 1304, the process performs a conversion between the current video block and its bitstream representation based on one or more parameters, wherein the one or more parameters are presented in the bitstream representation independently of the values ​​of at least one filter coefficient associated with the nonlinear filtering operation.

[0302] Figure 14 A flowchart of an example method for visual media processing is shown. The steps of this flowchart are discussed in conjunction with Example 6 in Section 10 of this document. In step 1402, the process configures one or more parameters for a clipping operation as part of a nonlinear filtering operation for the current video block. In step 1404, the process performs a conversion between the current video block and its bitstream representation based on one or more parameters, wherein the current video block inherits filter coefficients from the i-th filter, and wherein a first rule associated with the inheritance of one or more parameters of the clipping operation differs from a second rule associated with the inheritance of the filter coefficients.

[0303] The various embodiments discussed herein are now presented in a terms-based format.

[0304] A1. A method for visual media processing, comprising:

[0305] Configure one or more parameters for the clipping operation, which is part of the non-linear filtering operation, for the current video block; and

[0306] Based on one or more parameters, perform the conversion between the current video block and its bitstream representation, where one or more parameters are encoded or decoded according to rules.

[0307] A2. The method described in Clause A1, wherein the rule specifies encoding or decoding of one or more parameters with a fixed length of N bits.

[0308] A3. The method described in clause A1, wherein the rule specifies encoding or decoding of one or more parameters using a truncated unary method based on the maximum value N.

[0309] A4. The method described in any one or more of clauses A1-A3, wherein N is fixed.

[0310] A5. The method according to any one or more of clauses A1-A3, wherein N is signaled in the bitstream representation.

[0311] A6. The method according to any one or more of clauses A1-A3, wherein N is based on the encoding / decoding information of the current video block, wherein the encoding / decoding information includes quantization parameters or the size of an image including the current video block.

[0312] A7. The method described in Clause A1, wherein the nonlinear filtering operation is based on a filter, wherein a rule specifies that, for a filter or set of filters, one or more parameters are encoded or decoded in a fixed order using an exponential-Golomb method.

[0313] A8. The method described in Clause A1, wherein the rule specifies encoding or decoding of one or more parameters based on a run-length encoding / decoding method.

[0314] A9. The method described in Clause A5, wherein the rule further specifies that the run length of the run-length encoding / decoding method corresponds to the index of one or more parameters, and the length of the run-length encoding / decoding method corresponds to the number of identical consecutive parameters among one or more parameters.

[0315] A10. The method described in Clause A1, wherein the rule specifies encoding or decoding of one or more parameters based on predictive encoding or decoding.

[0316] A11. The method according to clause A10, wherein predictive encoding / decoding is applied to one or more parameters within a filter.

[0317] A12. The method according to clause A10, wherein predictive encoding / decoding is applied to one or more parameters in different filters.

[0318] A13. The method according to clause A12, wherein predictive encoding / decoding is applied to one or more parameters in different filters for a color component.

[0319] A14. The method according to clause A12, wherein predictive encoding / decoding is applied to one or more parameters in different filters for different color components.

[0320] A15. The method according to clause A12, wherein the predictive encoding / decoding is applied to one or more parameters in different filters for different samples of the current video block.

[0321] A16. The method according to clause A12, wherein predictive encoding / decoding is applied to one or more parameters in different filters for different video blocks.

[0322] A17. The method described in Clause A11, wherein one or more parameters are included as fields in different Adaptive Parameter Sets (APS).

[0323] A18. The method according to any one or more of clauses A1 to A17, wherein the nonlinear filtering operation is an adaptive loop filter (ALF) operation, wherein the ALF operation includes determining the filter index based on gradient calculations in different directions.

[0324] A19. The method described in any one or more of clauses A1 to A17, wherein one or more parameters include a limiting index.

[0325] B1. A method for visual media processing, comprising:

[0326] Based on the characteristics of the current video block, determine one or more parameters for the nonlinear filtering operation; and

[0327] Based on one or more parameters, perform the conversion between the current video block and its bitstream representation.

[0328] B2. The method described in Clause B1, wherein the characteristic of the current video block is the encoding / decoding mode of the current video block.

[0329] B3. The method described in Clause B2, wherein the encoding / decoding mode of the current video block is intra-frame mode, non-intra-frame mode, intra-block copy (IBC) mode, or non-IBC mode.

[0330] B4. The method described in clause B1, wherein the characteristic is transform information.

[0331] B5. The method according to clause B4, wherein the transformation information includes an indication of transformation skipping applied to the current video block.

[0332] B6. The method described in Clause B1, wherein the characteristic is residual information.

[0333] B7. The method described in Clause B6, wherein the residual information includes zero-value coefficients in the current video block.

[0334] B8. The method described in Clause B1, wherein the characteristic is a slice group type or a picture type that includes a slice group or picture of the current video block.

[0335] B9. The method described in Clause B1, wherein the feature is temporal layer information or reference information associated with a slice, group of slices, picture, or strip including the current video block.

[0336] B10. The method described in Clause B1, wherein the feature is a reference image or motion information associated with the current video block.

[0337] B11. The method described according to clause B1, wherein the property is a geometric transformation.

[0338] B12. A method according to any one or more of clauses B1 to B11, wherein one or more parameters include one or more parameters of the on / off control flag and / or the limiting function.

[0339] B13. The method described in Clause B12, wherein the upper limit of the limiting function is different from the lower limit of the limiting function.

[0340] B14. The method described in clause B13, wherein the predictive encoding / decoding is applied between the upper and lower limits of the clipping function.

[0341] B15. The method described in Clause B12, wherein the upper limit and lower limit of the clipping function are included as fields in the bitstream representation.

[0342] B16. The method according to clause B10, wherein the nonlinear filtering operation includes using a first filter and a second filter, and wherein one or more parameters of the second filter are predicted using one or more parameters of the first filter.

[0343] B17. The method according to clause B10, wherein the first filter and the second filter are applied to different sets of samples of the current video block.

[0344] B18. The method according to clause B10, wherein the first filter and the second filter are applied to samples associated with different video blocks.

[0345] B19. The method according to any one or more of clauses A1 to B18, wherein the shape of the filter used in the nonlinear filtering operation is based on the color representation of the sample points associated with the current video block.

[0346] B20. The method described in Clause B19, wherein the color representation includes a 4:4:4 color format or an RGB color format.

[0347] B21. The method described in clause B19, wherein the filter is a diamond filter.

[0348] B22. The method described in Clause B19, wherein the size of the diamond filter is 5x5 or 7x7.

[0349] B23. The method according to any one or more of clauses B1 to B22, wherein the nonlinear filtering operation is a nonlinear adaptive loop filtering operation.

[0350] B24. The method according to any one or more of clauses B1 to B23, wherein the nonlinear filtering operation is an adaptive loop filter (ALF) operation, wherein the ALF operation includes determining the filter index based on gradient calculations in different directions.

[0351] B25. The method described in any one or more of clauses B1 to B23, wherein one or more parameters include a limiting index.

[0352] B26. The method according to any one of clauses A1-B25, wherein the transformation includes generating a bitstream representation from the current video block.

[0353] B27. The method according to any one of clauses A1-B25, wherein the transformation includes generating pixel values ​​of the current video block from a bitstream representation.

[0354] B28. A video encoder apparatus including a processor configured to perform the methods described in any one or more of clauses A1-B25.

[0355] B29. A video decoder apparatus including a processor configured to implement the methods described in any one or more of clauses A1-B25.

[0356] B30. A computer-readable medium storing code embodying processor-executable instructions for implementing the methods described in any one or more of clauses A1-B25.

[0357] C1. A method for visual media processing, comprising:

[0358] Configure one or more parameters for the clipping operation, which is part of the non-linear filtering operation, for the current video block; and

[0359] Based on one or more parameters, a conversion between the current video block and its bitstream representation is performed, wherein the one or more parameters are presented in the bitstream representation independently of the value of at least one filter coefficient associated with the nonlinear filtering operation.

[0360] C2. The method according to clause C1, wherein one or more parameters are presented in the bitstream representation when the value of at least one filter coefficient is zero.

[0361] C3. The method according to any one or more of clauses C1-C2, wherein one or more parameters are presented in the bitstream representation, independent of the value of at least one filter coefficient.

[0362] C4. The method according to any one or more of clauses C1 to C3, wherein the nonlinear filtering operation is an adaptive loop filter (ALF) operation, wherein the ALF operation includes determining the filter index based on gradient calculations in different directions.

[0363] C5. The method described in any one or more of clauses C1 to C3, wherein one or more parameters include a limiting index.

[0364] D1. A method for visual media processing, comprising:

[0365] Configure one or more parameters for the clipping operation, which is part of the non-linear filtering operation, for the current video block; and

[0366] Based on one or more parameters, a conversion between the current video block and its bitstream representation is performed, wherein the current video block inherits filter coefficients from the i-th filter, and wherein a first rule associated with the inheritance of one or more parameters of the clipping operation differs from a second rule associated with the inheritance of the filter coefficients.

[0367] D2. The method described in accordance with clause D1, wherein the first rule specifies one or more parameters that exclude the clipping operation inherited from the i-th filter.

[0368] D3. The method described under Clause D1 further includes:

[0369] When temporal prediction is enabled for the current video block, determine whether to apply or disable the clipping operation.

[0370] D4. The method described under Clause D1 further includes:

[0371] When temporal prediction is enabled for the current video block, determine whether to apply or exclude the inheritance of one or more parameters for the clipping operation.

[0372] D5. The method described in Clause D1, wherein the first rule specifies one or more parameters of the limiting operation to be inherited from the j-th filter.

[0373] D6. The method according to clause D1, wherein the first rule specifies one or more parameters of the clipping operation inherited from the j-th filter, and wherein the j-th filter and the i-th filter are associated with different filter sets.

[0374] D7. The method described in Clause D6, wherein the j-th filter and the i-th filter are associated with different pictures and / or slice groups and / or slices and / or strips.

[0375] D8. The method described in clause D6, wherein the j-th filter is the same as the i-th filter.

[0376] D9. The method described in clause D5, wherein the j-th filter is different from the i-th filter.

[0377] D10. The method described in accordance with Clause D1, wherein the first rule specifies that one or more parameters of the clipping operation be included as fields in the bitstream representation.

[0378] D11. The method described in Clause D10, wherein the field includes an Adaptive Parameter Set (APS) index.

[0379] D12. The method according to any one or more of clauses D1-D11, wherein the limiting operation includes calculating the sampling difference of the limiting or the gradient of the limiting.

[0380] D13. The method according to clause D12, wherein the limiting gradient includes a vertical gradient, wherein the vertical gradient is calculated as...

[0381] V_(k,l)=|clip1(R(k,l)-R(k,l-1))+clip2(R(k,l)-R(k,l+1))|, where clip1 and clip2 are the first and second clipping functions, respectively, and R(i, j) represents the sample points of the current video block.

[0382] D14. The method according to clause D12, wherein the limiting gradient includes a horizontal gradient, wherein the horizontal gradient is calculated as...

[0383] H_(k,l)=|clip1(R(k,l)-R(k-1,l))+clip2(R(k,l)-R(k+1,l))|, where clip1 and clip2 are the first and second clipping functions, respectively, and R(i, j) represents the sample points of the current video block.

[0384] D15. The method described under any one or more of clauses D1 to D14 further includes:

[0385] Based on the location of the samples used in the filtering operation, determine whether to selectively enable or disable the clipping operation.

[0386] D16. The method according to clause D15, wherein the clipping operation is disabled if the sample point is not located at the boundary of one or more of the following: encoding / decoding unit, segmentation unit, transform unit, picture, slice, or slice group.

[0387] D17. The method according to clause D15, wherein clipping is enabled if the sample point is located at the boundary of one or more of the following: encoding / decoding unit, segmentation unit, transform unit, picture, slice, slice group, encoding / decoding tree unit, or virtual pipeline data unit.

[0388] D18. The method according to clause D15, wherein the location is the distance between the sample point and the boundary of one or more of the following: encoding / decoding unit, segmentation unit, transform unit, picture, slice, slice group, encoding / decoding tree unit, or virtual pipeline data unit.

[0389] D19. The method described in clause D18, wherein the distance is predefined.

[0390] D20. The method according to clause D18, wherein the distance is signaled in the bitstream representation.

[0391] D21. The method according to any one or more of clauses D1 to D20, wherein the shape of the filter used in the nonlinear filtering operation is based on the color representation of the sample points associated with the current video block.

[0392] D22. The method described in Clause D21, wherein the color representation includes a 4:4:4 color format or an RGB color format.

[0393] D23. The method described in clause D21, wherein the filter is a diamond filter.

[0394] D24. The method described in Clause D23, wherein the size of the diamond filter is 5x5 or 7x7.

[0395] D25. The method according to any one or more of clauses D1 to D24, wherein the nonlinear filtering operation is an adaptive loop filtering (ALF) operation, wherein the ALF operation includes determining the filter index based on gradient calculations in different directions.

[0396] D26. The method described in any one or more of clauses D1 to D24, wherein one or more parameters include a limiting index.

[0397] D27. The method according to any one of clauses C1-D26, wherein the transformation includes generating a bitstream representation from the current video block.

[0398] D28. The method according to any one of clauses C1-D26, wherein the transformation includes generating pixel values ​​of the current video block from the bitstream representation.

[0399] D29. A video encoder device including a processor configured to implement the method described in any one or more of clauses C1-D26.

[0400] D30. A video decoder apparatus including a processor configured to implement the methods described in any one or more of clauses C1-D26.

[0401] D31. A computer-readable medium storing code embodying processor-executable instructions for implementing the methods described in any one or more of clauses C1-D26.

[0402] In this document, the terms "video processing," "visual media processing," or "processing of visual media" can refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm can be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, and vice versa. For example, the bitstream representation of the current video block can correspond to co-occurring bits or bits distributed at different locations within the bitstream, as defined by the syntax. For example, a macroblock can be encoded based on the transformed and encoded / decoded error residuals, and also using bits in the header and other fields in the bitstream. Furthermore, during the conversion, the decoder can, based on this determination, parse the bitstream knowing that some fields may or may not be present, as described in the solutions above. Similarly, the encoder can determine whether certain syntax fields are included and generate the encoded / decoded representation accordingly by including or excluding these syntax fields from the encoded / decoded representation.

[0403] Based on the foregoing, it should be understood that specific embodiments of the currently disclosed technology have been described for illustrative purposes, but various modifications can be made without departing from the scope of the invention. Therefore, the currently disclosed technology is not limited except for the appended claims.

[0404] The implementation of the subject matter and functional operations described in this patent document can be implemented in various systems, digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium for execution by or control of the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a synthetic material that implements machine-readable propagation signals, or combinations thereof. The terms "data processing unit" or "data processing apparatus" encompass all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may include code that creates an operating environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof.

[0405] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to run on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a communication network.

[0406] The processes and logic flows described in this specification can be executed by one or more programmable processors running one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can also be implemented as dedicated logic circuitry, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0407] For example, processors suitable for running computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to one or more such mass storage devices to receive data from, transfer data to, or both. However, a computer does not require such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as electrical EPROMs, EEPROMs, and flash memory devices. The processor and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0408] This specification and accompanying drawings are to be considered exemplary only, where exemplary means example. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, the use of “or” is intended to include “and / or” unless the context clearly indicates otherwise.

[0409] Although this patent document contains numerous details, these details should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Some features described in this patent document within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, or even initially claimed in this way, in some cases one or more features from a claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof.

[0410] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or to perform all of the shown operations to obtain the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0411] Only a few implementations and examples have been described, and other implementations, enhancements and variations may be made based on what is described and shown in this patent document.

Claims

1. A method for processing video data, comprising: Determine to apply a nonlinear filtering operation to the current video region; Generate at least one first filter index for the current video region; The first filter coefficient set is derived based on at least one first filter index and at least one coefficient parameter syntax element; The first limiting parameter set is derived based on at least one first filter index and at least one filter limiting syntax element; The nonlinear filtering operation is performed based on the first set of filter coefficients and the first set of limiting parameters. as well as Perform the conversion between the current video region and the video bitstream; In this context, at least one filter clipping syntax element exists in the bitstream independently of the value of the first filter coefficient set. Wherein, the at least one filter limiting syntax element and the at least one coefficient parameter syntax element exist in the same adaptive parameter set, and The current video region includes codec tree blocks or stripes.

2. The method according to claim 1, wherein, The at least one filter clipping syntax element exists in the bitstream if at least one value of the first set of filter coefficients is zero.

3. The method according to claim 1, wherein, The at least one filter limiting syntax element includes a limiting index.

4. The method according to claim 1, wherein, The at least one coefficient parameter syntax element includes the absolute value of the coefficient.

5. The method according to claim 1, wherein, The at least one filter-limiting syntax element is encoded and decoded with a fixed length of 2 bits, and the at least one coefficient parameter syntax element is encoded and decoded in a fixed order using the exponential-Columbus model.

6. The method according to claim 1, wherein, The at least one first filter index is derived based on the difference between multiple sample points in different directions.

7. The method according to claim 6, wherein, The current video region is divided into multiple M... M video sub-regions, and for each M The difference between multiple samples in different directions is derived from the M video subregion, where M equals 2 or 4.

8. The method according to claim 7, wherein, The differences between the multiple sample points in different directions are derived based on a 1:N subsampling rate, where N is greater than 1.

9. The method according to claim 1, wherein, The conversion includes encoding the current video region into the bitstream.

10. The method according to claim 1, wherein, The conversion includes decoding the current video region from the bitstream.

11. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: Determine to apply a nonlinear filtering operation to the current video region; Generate at least one first filter index for the current video region; The first filter coefficient set is derived based on at least one first filter index and at least one coefficient parameter syntax element; The first limiting parameter set is derived based on at least one first filter index and at least one filter limiting syntax element; The nonlinear filtering operation is performed based on the first set of filter coefficients and the first set of limiting parameters. as well as Perform the conversion between the current video region and the video bitstream; In this context, at least one filter clipping syntax element exists in the bitstream independently of the value of the first filter coefficient set. Wherein, the at least one filter limiting syntax element and the at least one coefficient parameter syntax element exist in the same adaptive parameter set, and The current video region includes codec tree blocks or stripes.

12. The apparatus according to claim 11, wherein, The at least one filter clipping syntax element exists in the bitstream if at least one value of the first set of filter coefficients is zero.

13. The apparatus according to claim 11, wherein, The at least one filter limiting syntax element includes a limiting index.

14. The apparatus according to claim 11, wherein, The at least one first filter index is derived based on the difference between multiple sample points in different directions.

15. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: Determine to apply a nonlinear filtering operation to the current video region; Generate at least one first filter index for the current video region; The first filter coefficient set is derived based on at least one first filter index and at least one coefficient parameter syntax element; The first limiting parameter set is derived based on at least one first filter index and at least one filter limiting syntax element; The nonlinear filtering operation is performed based on the first set of filter coefficients and the first set of limiting parameters. as well as Perform the conversion between the current video region and the video bitstream; In this context, at least one filter clipping syntax element exists in the bitstream independently of the value of the first filter coefficient set. Wherein, the at least one filter limiting syntax element and the at least one coefficient parameter syntax element exist in the same adaptive parameter set, and The current video region includes codec tree blocks or stripes.

16. A non-transitory computer-readable recording medium having stored computer instructions and a bit stream thereon, wherein, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-8 to generate the bit stream.

17. A method for storing a video bitstream, comprising: The bit stream is generated by performing the method according to any one of claims 1-8; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images