Loop filter block flexible partitioning

Abstract

Embodiments of the present application disclose a method, apparatus and computer readable storage medium for performing in-loop filtering in a video coding process. The method comprises receiving image data, analyzing the image data, flexibly partitioning the image data into a plurality of in-loop filtering blocks (LFBs) to allow a size of an LFB in at least one of a first row and a first column in a same frame to be smaller than other LFBs in the same frame, and applying an in-loop filter to the plurality of LFBs.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 984,096, filed March 2, 2020, and U.S. Application No. 17 / 067,054, filed October 9, 2020, the disclosures of which are incorporated by reference in their entireties. TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to a set of advanced video coding techniques. More specifically, embodiments of the present disclosure can provide flexible picture partitioning for in-loop filter blocks. BACKGROUND

[0004] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. It was developed by the Alliance for Open Media (AOMedia) as a successor to VP9, an alliance of semiconductor companies, video on demand providers, video content publishers, software development companies, and web browser vendors founded in 2015. Many of the components of the AV1 project originated in previous research efforts by the alliance members. Individual contributors began experimenting with technology platforms several years ago: Xiph / Mozilla’s Daala announced code in 2010, Google’s experimental VP9 evolution project VP10 announced on September 12, 2014, and Cisco’s Thor announced on August 11, 2015. Building on the base code of VP9, AV1 incorporates additional technologies, several of which were developed in these experimental forms. The first version (version 0.1.0) of the AV1 reference codec was published on April 7, 2016. The Alliance announced the release of the AV1 bitstream specification, along with a reference software-based encoder and decoder, on March 28, 2018. On June 25, 2018, a confirmed version 1.0.0 of the specification was released. A confirmed version 1.0.0 of the specification was released on January 8, 2019, containing Errata 1. The AV1 bitstream specification includes a reference video codec.

[0005] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published version 1 of the H.265 / HEVC (High Efficiency Video Coding) standard in 2013, version 2 in 2014, version 3 in 2015, and version 4 in 2016. In 2015, the two standard organizations formed JVET (Joint Video Exploration Team) to explore the potential of developing the next video coding standard beyond HEVC. In October 2017, JVET released a Call for Proposals (CfP) on video compression with capabilities beyond HEVC. By February 15, 2018, a total of 22 CfP responses on standard dynamic range (SDR), 12 CfP responses on high dynamic range (HDR), and 12 CfP responses on 360 video categories were submitted, respectively. In April 2018, all received CfP responses were evaluated in the 122 MPEG / Tenth JVET Meeting. As a result of this meeting, JVET officially started the process of standardizing the next generation of video coding beyond HEVC. The new standard is named Versatile Video Coding (VVC), and JVET is renamed as Joint Video Expert Team. The current version of VTM (VVC Test Model) is VTM 7. SUMMARY

[0006] In the loop filtering process, the picture is partitioned into several equal-sized loop filtering blocks (except those in the last row / column), which can limit the loop filtering performance.

[0007] Embodiments of the present disclosure provide a solution to the above problem. For example, according to an embodiment, the loop filtering blocks can have different sizes, thereby improving the loop filtering performance.

[0008] According to an embodiment, a loop filtering method in a video coding process performed by at least one processor is provided, the method comprising: receiving image data; analyzing the image data; flexibly partitioning the image data into loop filtering blocks (LFBs); and applying a loop filter to the LFBs.

[0009] According to an embodiment, the flexible partitioning can be performed to allow a size of the LFBs in at least one of a first row and a first column in a same frame to be smaller than other LFBs within the same frame.

[0010] According to an embodiment, the flexible partitioning can be performed to allow a size of the LFBs in a first row to have a height smaller than LFBs disposed in a same frame, the LFBs disposed in the same frame not disposed in at least one of a last row and a last column in the same frame.

[0011] According to an embodiment, flexible partitioning can be performed to allow the size of LFBs in the first column to have a smaller width than LFBs disposed in the same frame that are not disposed in at least one of the last row and the last column in the same frame.

[0012] According to an embodiment, flexible partitioning can be performed to allow the size of LFBs in at least one of the first row and the first column to be smaller than other LFBs within the same frame and to not include the remaining portion of the partition in the last row and the last column within the same frame.

[0013] According to an embodiment, the height of LFBs in the first row and the width of LFBs in the first column can be hard coded.

[0014] According to an embodiment, the method can further include providing an indication of whether at least one of the LFBs in the first row and the first column has a smaller size than LFBs in other areas of the same frame.

[0015] According to an embodiment, the height of LFBs in the first row and the width of LFBs in the first column can be indicated using separate syntax.

[0016] According to an embodiment, the height of LFBs in the first row can be the same as the width of LFBs in the first column.

[0017] According to an embodiment, the remaining portion of the partition can not be included in the last row and the last column within the same frame. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a diagram of an adaptive loop filter (ALF) shape.

[0019] Figures 2A to 2D is a diagram of a downsampled Laplacian calculation.

[0020] Figure 3 is a diagram of modified block classification applied to luma components.

[0021] Figure 4 is a diagram of applying adaptive loop filtering (ALF) to one or more luma components at a virtual boundary.

[0022] Figure 5 is a diagram of LCU-aligned picture quadtree partitioning.

[0023] Figure 6 is a diagram of quadtree partitioning flags coded in z-order.

[0024] Figure 7A and Figure 7B is a diagram depicting placement of CC ALF.

[0025] Figure 8 is a schematic illustration of the position of a chroma sample relative to a luma sample.

[0026] Figure 9 is a schematic illustration of directional search for 8x8 blocks.

[0027] Figure 10 is a schematic illustration of subspace projection.

[0028] Figure 11 is a schematic illustration of loop filter block (LFB) partitioning.

[0029] Figure 12 is a schematic illustration of loop filter block (LFB) partitioning based on symmetric quadtree.

[0030] Figure 13 is a schematic illustration of loop filter block (LFB) partitioning based on asymmetric quadtree.

[0031] Figure 14 is a schematic illustration of loop filter block (LFB) partitioning.

[0032] Figure 15 is a schematic illustration of loop filter block (LFB) partitioning based on symmetric quadtree.

[0033] Figure 16 is a schematic illustration of loop filter block (LFB) partitioning based on asymmetric quadtree.

[0034] Figure 17 is a block diagram of a communication system according to an embodiment.

[0035] Figure 18 is a diagram of an arrangement of a G-PCC compressor and a G-PCC decompressor in an environment according to an embodiment.

[0036] Figure 19 is a diagram of a computer system suitable for implementing an embodiment. DETAILED DESCRIPTION

[0037] Embodiments described herein provide methods and apparatus for encoding and / or decoding image data.

[0038] [Adaptive loop filtering]

[0039] In VTM7, an adaptive loop filter (ALF) with block-based filter adaptation is applied. For the luma component, one out of 25 filters is selected for each 4x4 block based on the local gradient direction and activity.

[0040] [Filter shape]

[0041] In VTM7, a referenceFigure 1 Two diamond filters are used. A 7x7 diamond shape is applied to the luma component and a 5x5 diamond shape is applied to the chroma components.

[0042] [Block classification]

[0043] For the luma component, each 4x4 block is classified into one of 25 categories. The classification index C is derived based on its directionality D and quantization activity value . This is illustrated in Equation 1 (below).

[0044] Equation 1:

[0045]

[0046] To compute D and The gradients for horizontal, vertical and two diagonal directions are first computed using 1-D Laplacian equation as shown in Equations 2-5 (below).

[0047] Equation 2:

[0048]

[0049] Equation 3:

[0050]

[0051] Equation 4:

[0052]

[0053] Equation 5:

[0054]

[0055] Here, indices i and j refer to the coordinates of the top-left sample within the 4x4 block and R(i,j) indicates the reconstructed sample at coordinate (i,j).

[0056] To reduce the complexity of block classification, down-sampled 1-D Laplacian operator computation is applied. As shown in Figure 2, the same down-sampling positions are used for gradient computation of all directions. Here, Figure 2A Down-sampling positions for vertical gradient are depicted. Figure 2B Down-sampling positions for horizontal gradient are depicted. Figure 2C and Figure 2D Down-sampling positions for diagonal gradient are depicted.

[0057] The Dmax and Dmin values for the gradients of horizontal and vertical directions are then set according to Equation 6 (below):

[0058]

[0059] The maximum and minimum values of the two diagonal gradients are set according to equation 7 (below):

[0060]

[0061] To derive the value of the directionality D, these values are compared to each other and to two thresholds t1 and t2:

[0062] Step 1 : If and are true, then set D to 0.

[0063] Step 2: If continue with step 3; otherwise continue with step 4.

[0064] Step 3: If then set D to 2; otherwise set D to 1.

[0065] Step 4: If then set D to 4; otherwise set D to 3.

[0066] The activity value A is computed according to equation 8 (below):

[0067]

[0068] A is further quantized in the range from 0 to 4, inclusive, and the quantized value is denoted as

[0069] For chroma components in a picture, the classification method is not applied, i.e. a single set of ALF coefficients is applied to each chroma component.

[0070] [Geometric transformations of filter coefficients and clipping values]

[0071] Before filtering each 4x4 luma block, a geometric transformation such as a rotational flip, a diagonal flip and / or a vertical flip is applied to the filter coefficients f(k,l) and the corresponding filter clipping values c(k,l) depending on the gradient values computed for the block. This is equivalent to applying these transformations to the samples in the filter support region. The idea is to make different blocks to which ALF is applied more similar by aligning the directionality of the ALF.

[0072] Three geometric transformations are introduced, including diagonal flip, vertical flip and rotational flip: in equations 9-11 below.

[0073] Equation 9:

[0074] Diagonal: f D (k,l) = f(l,k), c D(k, l) = c(l, k),

[0075] Equation 10:

[0076] Vertical flip: f V (k, l) = f(k, K - l - 1), c V (k, l) = c(k, K - l - 1)

[0077] Equation 11:

[0078] Rotation: f R (k, l) = f(K - l - 1, k), c R (k, l) = c(K - l - 1, k)

[0079] Here, K is the filter size, and 0≤k, l≤K-1 are the coefficient coordinates, such that position (0,0) is at the top-left corner and position (K-1, K-1) is at the bottom-right corner. Depending on the gradient value computed for the block, a transform is applied to the filter coefficients f(k, l) and the clipping values c(k, l). The relationship between the four gradients and the four transforms is summarized in the table below.

[0080] Table 1: Mapping of gradients computed for a block and corresponding transforms.

[0081] Gradient value Transform g d2 g d1 and g h g v ]]> No transform g d2 g d1 and g v g h ]]> Diagonal g d1 g d2 and g h g v ]]> Vertical flip g d1 g d2 and g v g h ]]> Rotation

[0082] [Filter parameter signaling]

[0083] In VTM7, ALF filter parameters are signaled in an APS. Up to 25 sets of luma filter coefficients and clipping value indices, and up to 8 sets of chroma filter coefficients and clipping value indices can be signaled in one APS. To reduce the bit overhead, filter coefficients of different categories for the luma component can be merged. In the slice header, the index of the APS used for the current slice can be signaled. In VTM7, the signaling of ALF is CTU-based.

[0084] The clipping value indices decoded from the APS allow to determine the clipping values using the clipping value tables for the luma and chroma components. These clipping values depend on the internal bit depth. More precisely, the table of clipping values is obtained by:

[0085] Equation 12:

[0086] AlfClip = { round(2 B-α*n ) for n∈[0..N-1]}

[0087] Here, B is equal to the internal bit depth, a is a predefined constant value equal to 2.35, and N is equal to 4, which is the number of allowed clipping values in VVC. Table 2 shows the output of Equation 12:

[0088] Table 2: Specification of AlfClip depending on bitDepth and clipIdx

[0089]

[0090] In the slice header, up to 7 APS indices can be signaled to specify the set of luma filters used for the current slice. The filtering process can be further controlled at the CTB level. A flag is always signaled to indicate whether ALF is applied to the luma CTB or not. The luma CTB can select one filter from 16 fixed filter sets and the filter set in the APS. A filter set index is signaled for the luma CTB to indicate which filter set is applied. The 16 fixed filter sets are predefined and hard-coded in both the encoder and the decoder.

[0091] For chroma components, an APS index is signaled in the slice header to indicate that the chroma filter set is used for the current slice. At the CTB level, a filter index is signaled for each chroma CTB if there is more than one chroma filter set in the APS.

[0092] The filter coefficients are quantized with a norm equal to 128. To limit the multiplication complexity, bitstream conformance is applied such that the coefficient values at non-central positions are in the range of -27 to 27-1, inclusive. The central position coefficients are not signaled in the bitstream and are considered equal to 128.

[0093] In VVC Draft 7, the syntax and semantics of the clipping index and value are defined as follows:

[0094] alf_Luma_clip_idx[ sfldx ][ j ] specifies the clipping index of the clipping value to be used before multiplying the j-th coefficient of the signaled luma filter indicated by sfldx. The requirement of bitstream conformance is that the value of alf_Luma_clip_idx[ sfldx ][ j ] is in the range of 0 to 3, inclusive, where sfldx = 0.. alf_Luma_num_filters_signalled_minus1 and j = 0.. 11.

[0095] The luma filter clip value AlfClipL[ adaptation_parameter_set_id ] with the element AlfClipL[ adaptation_parameter_set_id ][ filtldx ][ j ] (where filtldx = 0.. NumAlfFilters - 1 and j = 0.. 11 ) is derived as specified in Table 2, depending on bitDepth set equal to BitDepthY and clipldx set equal to alf_Luma_clip_idx[ alf_Luma_coeff_delta_idx[ filtldx ][ j ] ][ j ].

[0096] The alf_Chroma_clip_idx[ altldx ][ j ] specifies the clip index of the clip value to be used before multiplying the j-th coefficient of the optional chroma filter by the index of altldx. The requirement for bitstream conformance is that the values of alf_Chroma_clip_idx[ altldx ][ j ] (where altldx = 0.. alf_Chroma_num_alt_filters_minus1 and j = 0.. 5 ) are in the range of 0 to 3, inclusive.

[0097] The chroma filter clip value AlfClipC[ adaptation_parameter_set_id ][ altldx ] with the element AlfClipC[ adaptation_parameter_set_id ][ altldx ][ j ] (where altldx = 0.. alf_Chroma_num_alt_filters_minus1 and j = 0.. 5 ) is derived as specified in Table, depending on bitDepth set equal to BitDepthC and clipldx set equal to alf_Chroma_clip_idx[ altldx ][ j ].

[0098] [Filtering process]

[0099] At the decoder side, when ALF is enabled for a CTB, each sample R(i,j) within the CU is filtered, resulting in a sample value R'(i,j) as shown in Equation 13 below:

[0100] R'(i,j) = R(i,j) + ((∑ k≠0 ∑ l≠0 f(k,l) x K(R(i+k,j+l)-R(i,j),c(k,l))+64) » 7)

[0101] Here, f(k, l) denotes the decoded filter coefficients, K(x, y) is a clipping function, and c(k, l) denotes the decoded clipping parameters. The variables k and l vary between and where L denotes the filter length. The clipping function K(x, y) = min(y, max(-y, x)) corresponds to the function Clip3(-y, y, x). By incorporating this clipping function, the loop filtering method becomes a non-linear process, referred to as non-linear ALF, as first proposed in JVET-N0242. The selected clipping values are encoded in the “alf_data” syntax element by using a Golomb encoding scheme corresponding to the indices of the clipping values in Table 2. This encoding scheme is the same as the one used for the filter indices.

[0102] [Virtual boundary filtering process for line buffer reproduction]

[0103] In VTM7, to reduce the line buffer requirement of ALF, a modified block classification and filtering is applied for samples near the horizontal CTU boundary. For this purpose, as shown in Figure 3 , a virtual boundary is defined as a line by shifting the horizontal CTU boundary with “N” samples. In Figure 3 , N is equal to 4 for the luma component and equal to 2 for the chroma components.

[0104] As illustrated in Figure 3 , the modified block classification is applied for the luma component. For the 1D Laplacian gradient calculation of 4x4 blocks, only samples above the virtual boundary are used in the virtual boundary. Similarly, for the 1D Laplacian gradient calculation of 4x4 blocks below the virtual boundary, only samples below the virtual boundary are used. Accordingly, the quantization of the activity value A is scaled by considering the reduced number of samples used in the 1D Laplacian gradient calculation.

[0105] For the filtering process, a symmetric padding operation at the virtual boundary is used for the luma and chroma components. As shown in Figure 4 , when the filtered sample is located below the virtual boundary, the neighboring sample located above the virtual boundary is padded. Meanwhile, the corresponding sample on the other side is also symmetrically padded.

[0106] [LCU-aligned picture quadtree partitioning]

[0107] To improve the coding efficiency, a coding unit synchronous picture quadtree based adaptive loop filter was proposed in JCTVC-C143. Here, the luma picture is partitioned into several multi-level quadtree partitions, and each partition boundary is aligned with the boundaries of the largest coding unit (LCU). Each partition has its own filtering process and can thus be referred to as a filtering unit (FU).

[0108] A 2-pass encoding process is described as follows. In the first pass, the quadtree partitioning mode and the best filter for each FU are determined. The filter distortion is estimated by the FFDE during the decision process. The reconstructed picture is filtered according to the determined quadtree partitioning mode of all FUs and the selected filters. In the second pass, the CU-level ALF on / off control is performed. According to the ALF on / off result, the first filtered picture is partially recovered by the reconstructed picture.

[0109] With a top-down partitioning strategy, the picture is partitioned into multiple levels of quadtree partitions using the rate-distortion criterion. Each partition is referred to as a filtering unit. The partitioning process aligns the quadtree partitions with the LCU boundaries. The encoding order of the FUs follows the z-scan order. For example, as shown in FIG. 1, a picture is partitioned into 10 FUs, and the encoding order is FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8, and FU9. Figure 5

[0110] To indicate the picture quadtree partitioning mode, a partitioning flag is encoded and sent in z-order. Figure 6 The quadtree partitioning mode corresponding to Figure 5 is shown.

[0111] The filter for each FU is selected from two filter sets based on the rate-distortion criterion. The first set has the 1 / 2 symmetric square and diamond filters newly derived for the current FU. The second set comes from a delay filter buffer; the delay filter buffer stores the filters previously derived for the FUs of the previous picture. The filter with the minimum rate-distortion cost from these two sets is selected for the current FU. Similarly, if the current FU is not a minimum FU and can be further partitioned into 4 sub-FUs, the rate-distortion costs of the 4 sub-FUs are calculated. By recursively comparing the rate-distortion costs of the partitioned and non-partitioned cases, the picture quadtree partitioning mode can be determined.

[0112] In JCTVC-C143, the maximum quadtree partitioning level is 2, which means the maximum number of FUs is 16. During the quadtree partitioning decision, the correlation values used to derive the Wiener coefficients of the 16 FUs (minimum FUs) at the bottom quadtree level can be reused. The remaining FUs can derive their Wiener filters from the correlation of the 16 FUs at the bottom quadtree level. Therefore, there is only one frame buffer access for deriving the filter coefficients of all FUs.

[0113] ​After determining the quad-tree partition mode, to further reduce the filtering distortion, CU-synchronous ALF on / off control is performed. By comparing the filtered distortion and the non-filtered distortion, a leaf CU can explicitly turn on / off ALF in its local region. Coding efficiency can be further improved by redesigning the filter coefficients according to the ALF on / off result. However, the redesign process requires additional frame buffer access. In the proposed CS-PQALF encoder design, there is no redesign process after the CU-synchronous ALF on / off decision to minimize the number of frame buffer accesses.

[0114] [Cross-Component Adaptive Loop Filter

[0115] In JVET-O0636, a cross-component adaptive loop filter (CC-ALF) is proposed. CC-ALF utilizes luma sample values to refine each chroma component. Figure 7A The arrangement of CC-ALF with respect to other loop filters is illustrated.

[0116] CC-ALF operates by applying a linear diamond filter ( Figure 7B ) to the luma channel of each chroma component. The filter coefficients are transmitted in APS, scaled by a factor of 2 10 and rounded for fixed-point representation. The application of the filter is controlled over a variable block size and is signaled by a context-coded flag received for each sample block. The block size as well as the CC-ALF enable flag are received at slice level for each chroma component. In contributions, the following block sizes (in chroma samples) are supported: 16x16, 32x32, 64x64.

[0117] The syntax changes for CC-ALF proposed in JVET-P1008 are described below.

[0118] Table 3: Syntax changes for CC-ALF proposed in JVET-P1008

[0119]

[0120] The semantic description of the CC-ALF related syntax is as follows:

[0121] When alf_ctb_cross_component_cb_idc[ xCtb » CtbLog2SizeY ][ yCtb » CtbLog2SizeY ] is equal to 0, this indicates that no cross-component Cb filter is applied to the block of Cb color component samples at luma location ( xCtb, yCtb ). When alf_cross_component_cb_idc[ xCtb » CtbLog2SizeY ][ yCtb » CtbLog2SizeY ] is not equal to 0, this indicates that the alf_cross_component_cb_idc[ xCtb » CtbLog2SizeY ][ yCtb » CtbLog2SizeY ]-th cross-component Cb filter is applied to the block of Cb color component samples at luma location ( xCtb, yCtb ).

[0122] When alf_ctb_cross_component_cr_idc[ xCtb » CtbLog2SizeY ][ yCtb » CtbLog2SizeY ] is equal to 0, this indicates that no cross-component Cr filter is applied to the block of Cr color component samples at luma location ( xCtb, yCtb ). When alf_cross_component_cr_idc[ xCtb » CtbLog2SizeY ][ yCtb » CtbLog2SizeY ] is not equal to 0, this indicates that the alf_cross_component_cr_idc[ xCtb » CtbLog2SizeY ][ yCtb » CtbLog2SizeY ]-th cross-component Cr filter is applied to the block of Cr color component samples at luma location ( xCtb, yCtb ).

[0123] [Position of chrominance samples]

[0124] Figure 8The relative positions of the indicated locations of the top-left chroma sample when Chroma format idc is equal to 1 (4:2:0 chroma format) and Chroma sample loc type top field or Chroma sample loc type bottom field is equal to the value of the variable ChromaLocType are illustrated. The area represented by the top-left 4:2:0 chroma sample (depicted as a large red square with a large red dot at its center) is shown relative to the area represented by the top-left luma sample (depicted as a small black square with a small black dot at its center). The areas represented by the neighboring luma samples are depicted as small gray squares with a small gray dot at their center. As an explanation of the notation, "X" denotes luma sample top; square denotes luma sample bottom. And the oval shape, triangle shape, and diamond shape denote chroma sample types.

[0125] [Constrained directional enhancement filter]

[0126] The primary goal of the in-loop constrained directional enhancement filter (CDEF) is to filter out coding artifacts while preserving the details of the image. In HEVC, the sample adaptive offset (SAO) algorithm achieves a similar goal by defining signal offsets for different classes of pixels. Unlike SAO, CDEF is a non-linear spatial filter. The design of the filter is constrained to be easily vectorizable (i.e., can be implemented with SIMD operations), which is not the case for other non-linear filters such as median filters and bilateral filters.

[0127] The CDEF design stems from the following observation. The amount of ringing in an encoded image tends to be roughly proportional to the quantization step size. The amount of detail is a property of the input image, but the minimum detail preserved in the quantized image also tends to be proportional to the quantization step size. For a given quantization step size, the amplitude of the ringing is typically smaller than the amplitude of the detail.

[0128] CDEF works by identifying the direction of each block, and then adaptively filtering along the identified direction, and to a lesser extent along directions that are rotated 45 degrees from the identified direction. The filter strength is explicitly signaled, which allows for high control over blurring. An efficient encoder search is designed for the filter strength. CDEF is based on two previously proposed in-loop filters, and the combined filter is used in the emerging AV1 codec.

[0129] [Direction search]

[0130] Directional search operates on the reconstructed pixels immediately after the deblocking filter. Since these pixels are available to the decoder, the direction does not need to be signaled. The search operates on 8x8 blocks, which are small enough to adequately handle non-linear edges while being large enough to reliably estimate the direction when applied to quantized images. Having a constant direction over an 8x8 region also makes vectorization of the filter easier. For each block, we determine the direction that best matches the pattern in the block by minimizing the sum of squared differences (SSD) between the quantized block and the closest perfectly directional block. A perfectly directional block is a block where all pixels along a line in one direction have the same value. Figure 9 An example of a directional search of an 8x8 block is shown.

[0131] [Non-linear low-pass directional filter]

[0132] The main reason to identify a direction is to align the filter taps along that direction to reduce ringing while preserving the directional edge or pattern. However, directional filtering alone sometimes does not adequately reduce ringing. It is also desirable to use filter taps on pixels that are not along the main direction. To reduce the risk of blurring, these extra taps are treated more conservatively. For this reason, CDEF defines primary and secondary taps. The full 2-D CDEF filter is represented in the following Equation 14:

[0133]

[0134] Here, D is a damping parameter, S (p) and S (s) are the strengths of the primary and secondary taps, respectively, and round(·) rounds away from zero, w k is the filter weight, and f(d, S, D) is a constraint function that operates on the difference between the filtered pixel and each neighboring pixel. For small difference values, f(d, S, D) = d, causing the filter to behave like a linear filter. When the difference value is large, f(d, S, D) = 0, which effectively ignores the filter tap.

[0135] [Loop restoration in AV1]

[0136] A set of in-loop restoration schemes are proposed for use in post-video coding deblocking to denoise and improve edge quality overall beyond traditional deblocking operations. These schemes are available on a per-intra frame basis for tiles of appropriate size. The particular schemes described are based on separable symmetric Wiener filters, a bi-directional self-guided filter with subspace projection, and a domain transform recursive filter. Because content statistics can vary significantly within a frame, these tools are integrated in a switchable intra-frame, where different tools can be triggered in different regions of the frame.

[0137] [Separable Symmetric Wiener Filter]

[0138] One promising restoration tool is the Wiener filter. Each pixel in a degraded frame can be reconstructed as a noncausal filtered version of the pixels within a w×w window, where w = 2r + 1 is odd for integers r. If the 2D filter taps are in column vector form by w... 2 The element vector F represents the filter parameters. Direct LMMSE optimization then results in the filter parameters being determined by F = H. -1 M is given, where H = E[XX] T ] is the autocovariance of x, w 2 The column vectorized version of the sample in a w×w window around the pixel, and M = E[YX T The target is to estimate the cross-correlation between x and the scalar source sample y. The encoder can estimate H and M based on the implementation in the deblocked frame and the source, and send the resulting filter F to the decoder. However, this not only affects the transmission of w... 2 Tapping incurs a considerable bit rate cost, and non-separable filtering makes decoding overly complex. Therefore, several additional constraints are imposed on the properties of F. First, F is constrained to be separable, allowing filtering to be implemented as separable horizontal and vertical w-tapped convolutions. Second, each horizontal and vertical filter is constrained to be symmetric. Third, it is assumed that the sum of the horizontal and vertical filter coefficients is 1.

[0139] [Dual self-guided filtering with subspace projection]

[0140] Guided filtering is one of the recent paradigms for image filtering that utilizes local linear models. An example is shown in Equation 15:

[0141] y = Fx + G

[0142] Here, a linear model is used to compute the filtered output y from the unfiltered sample x, where F and G are determined based on statistics of the degraded and guiding images near the filtered pixels. If the guiding image is identical to the degraded image, the resulting so-called self-guided filtering has an edge-preserving smoothing effect. The specific form of our proposed self-guided filtering depends on two parameters: the radius r and the noise parameter e, which are listed below:

[0143] Obtain the mean μ and variance σ of the pixels in the (2r+1)×(2r+1) windows surrounding each pixel. 2 This can be achieved efficiently using box filtering based on integral imaging.

[0144] Calculate for each pixel: f = σ 2 / (σ 2 +e); g=(1-f)μ

[0145] F and G for each pixel are computed as the average of the f and g values in a 3x3 window around the pixel for use.

[0146] The filtering is controlled by r and e, where higher r means higher spatial variance and higher e means higher range variance.

[0147] The principle of subspace projection is schematically illustrated in Figure 10 Even if none of the cheap recoveries X1, X2 is close to the source Y, a suitable multiplier {a, b} can make them closer to the source, provided they move slightly in the right direction.

[0148] [Domain Transform Recursive Filter]

[0149] Domain transform is a recent approach to edge-preserving image filtering that uses only 1-D operations, which can be much faster than other edge-aware processing methods. A recursive filtering entity is used, where the processing steps include horizontal left-to-right and right-to-left recursive 1st order filtering, followed by vertical top-to-bottom and bottom-to-top filtering, over a few (typically 3) iterations. Filter taps are obtained from the local horizontal and vertical gradients of the pixels and the iteration index.

[0150] [Joint Component Filtering (JCF)]

[0151] The Joint Component Filtering (JCF) method was proposed in previous IDF P20US056, which uses reconstructed samples from multiple color components as input and applies the output to multiple color components.

[0152] In-loop filters include, but are not limited to, ALF, SAO, deblocking, CDEF, CCALF, LR, and JCF. A loop filter block (LFB) refers to a unit on which an in-loop filtering process is applied. The LFB can be a block from the picture-level quadtree partitioning (e.g., as discussed above), or a CTU, or a smaller block such as 4x4 or 8x8.

[0153] The embodiments disclosed herein can be used individually or in any order in combination. Further, each embodiment (e.g., encoder and / or decoder) can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). For example, the one or more processors can execute a program stored in a non-volatile computer readable medium.

[0154] Certain embodiments will now be described. Starting from the top-left corner of a picture, a conventional loop filtering method partitions the picture into several equal-sized LFBs, except for the LFB in the last row / column, which is the remainder of the partitioning. That is, the LFB in the last row / column can have a different size than those in other areas of the picture.

[0155] As described above, an LFB is a loop filtering block, which can be different from a coding tree unit (CTU), which can also be referred to as a super block (SB). For example, an LFB can be used for processes associated with loop filtering, such as filtering and training filter coefficients. While a CTU / SB can be used as an LFB, generally an LFB cannot be used as a CTU / SB.

[0156] In some embodiments, the underlying partitioning of a CTU or SB can be further partitioned into smaller LFBs, which in turn can be affected by loop filtering. This can provide beneficial effects in improving the precision of loop filtering output.

[0157] Disclosed herein is a flexible partitioning of LFB (FPLFB), which allows the size of LFBs in the first row / column of a picture to be smaller than other LFBs.

[0158] More specifically, the height of LFBs in the first row and / or the width of LFBs in the first column of a picture can be smaller than the height and / or width of other LFBs (except for the LFB in the last row / column). The size of the LFB in the last row and / or column can also be smaller, i.e., the size of the remainder of the partitioning can be smaller.

[0159] In one embodiment, when LFBs have a fixed m*n size, the LFBs in the first row and column can have a k*l, k*n, or m*l size, where m and n are positive integers smaller than the picture width and height, and k and l are positive integers smaller than m and n. Figure 11 An example of LFB partitioning when the first row / column and the last row / column have smaller LFBs is depicted.

[0160] In one example, the proposed disclosure can be applied to the signaling and processing of CCALF, where the CTU is the basic unit for filtering, i.e., the CTU size is m*n.

[0161] In another example, the proposed disclosure can be applied to the ALF classification process with 4x4 luma LFBs as input.

[0162] In another example, the proposed disclosure can be applied to the CDEF edge direction derivation process with 8x8 luma LFBs as input.

[0163] In another example, the proposed disclosure can be applied to signaling and processing of JCF.

[0164] As briefly discussed above, Figure 11 An example of LFB partitioning is shown, where the first and last row / column have smaller LFBs (shown in gray) of m*n values, which can include 32x32, 64x64, 128x128.

[0165] In an embodiment, when LFBs are partitioned based on quad-tree, the LFBs in the first row and / or first column have smaller size than other LFBs in the same quad-tree partition level.

[0166] In an embodiment, the height of LFBs in the first row and the width of LFBs in the first column are determined first, and then the symmetric quad-tree partitioning is performed. The LFBs in the first row / column have smaller size than LFBs in the quad-tree region. Figure 12 An example of LFB partitioning based on symmetric quad-tree is depicted, when the first and last row / column have smaller LFBs. Here, the LFBs without padding indicate the partitioning based on symmetric quad-tree, and the LFBs padded with gray fill have smaller size than the quad-tree based LFBs.

[0167] In an embodiment, the asymmetric quad-tree partitioning is performed starting from the top-left corner of the frame. Figure 13 An example of asymmetric quad-tree partitioning is depicted. Here, the LFBs without padding indicate the partitioning based on asymmetric quad-tree, and the LFBs padded with gray in the last row / column are the remaining part of the picture-level quad-tree partitioning.

[0168] In an embodiment, the proposed method is applied to quad-tree based

[0169] According to an embodiment, only the first row and / or first column can have smaller LFBs, and the remaining part of the partitioning is not allowed in the last row / column. Figure 14 An example of LFB partitioning is depicted, when only the first row and first column have smaller LFBs. Figure 15 An example of LFB partitioning based on symmetric quad-tree is depicted, when only the first row / column have smaller LFBs. Figure 16 An example of asymmetric quad-tree partitioning is depicted.

[0170] According to an embodiment, different loop filters can use different FP LFB schemes.

[0171] In an embodiment, deblocking, CDEF, and LR can have different FP LFB schemes.

[0172] In an embodiment, deblocking, ALF, CCALF, and SAO can have different FP LFB schemes.

[0173] According to embodiments, different color components can use different FPLFB schemes.

[0174] In embodiments, Y, Cb and Cr can have different FPLFB schemes.

[0175] In embodiments, R, G and B can have different FPLFB schemes.

[0176] According to embodiments, the height of LFBs in the first row and the width of LFBs in the first column can be signaled in any parameter set (e.g., high-level syntax including DPS, VPS, SPS, PPS, APS, etc.), slice header, tile / tile group header, or SEI message.

[0177] According to embodiments, the size of LFBs in the first row and the first column can be hard-coded.

[0178] In one example, the height of LFBs in the first row can be 32, and the width of LFBs in the first column can be 32, and the rest of LFBs in the picture can be 128x128 (including LFBs in the last row and the last column).

[0179] According to embodiments, a binary flag can be signaled per sequence or per picture to indicate whether the first row LFBs have a smaller size; if yes, a separate syntax can be signaled to indicate the height of the first row. Another binary flag can be signaled per sequence or per picture to indicate whether the first column of LFBs have a smaller size. If yes, a separate syntax can be signaled to indicate the width of the first column.

[0180] In one example, two binary flags (lfb_first_row_smaller and lfb_first_column_smaller) are signaled in SPS to indicate whether the LFBs in the first row and the first column have a smaller size. If yes, lfb_first_row_height and lfb_first_column_width are signaled to indicate the height of the first row and the width of the first column.

[0181] According to embodiments, one separate syntax can be signaled to indicate the height of the first row, and another separate syntax can be signaled to indicate the width of the first column.

[0182] According to embodiments, one separate syntax is signaled to indicate the height of the first row and the width of the first column. That is, the height of the first row and the width of the first column can be the same.

[0183] According to embodiments, a separate syntax can be used to signal the specific in-loop filter partition, e.g. the height of the first and last row and / or the width of the first and last column, which is different from the syntax used to signal the partitioning of the coding tree unit (CTU), which can also be referred to as superblock (SB) (i.e. lower level partitioning). The in-loop filter partitioning can be the same or different from the lower level CTU and SB partitioning. In some embodiments, the in-loop filter partitioning can be employed using the lower level CTU and SB partitioning.

[0184] Also according to the present disclosure, the in-loop filter technique and in-loop filter partitioning can be applied to frames comprising CTBs / SBs partitioned according to a conventional partitioning scheme.

[0185] For example, the conventional partitioning scheme can be modified such that the first row and / or the first column have a different (e.g. smaller) size than the size assigned according to the conventional partitioning scheme. This will increase the overall precision of the applied in-loop filter. An environment in which embodiments can operate will now be described.

[0186] Figure 17 is a block diagram of a communication system 1700 according to an embodiment. The communication system 1700 can include at least two terminals 1710 and 1720 interconnected via a network 1750. For unidirectional transmission of data, a first terminal 1710 can code data at a local location for transmission to the second terminal 1720 via the network 1750. The second terminal 1720 can receive the coded data of the first terminal 1710 from the network 1750, decode the coded data and display the decoded data. Unidirectional data transmission can be common in media serving applications and the like.

[0187] Figure 17 A further pair of terminals 1730 and 1740 are further illustrated, which are provided to support bidirectional transmission of coded data that can occur, for example, during a video conference. For bidirectional transmission of data, each terminal 1730 or 1740 can code data captured at the local location for transmission to the other terminal via the network 1750. Each terminal 1730 or 1740 can also receive coded data transmitted by the other terminal, can decode the coded data, and can display the decoded data on a local display device.

[0188] In Figure 17In particular embodiments, terminals 1710-1740 can be illustrated as servers, personal computers, and smart phones, but the principles of the embodiments are not limited thereto. Embodiments apply to laptops, tablet computers, media players, and / or dedicated video conferencing equipment. Network 1750 represents any number of networks that convey coded data among terminals 1710-1740, including, for example, wireline and / or wireless communication networks. Communication networks 1750 can exchange data in circuit- switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of the present discussion, the architecture and topology of network 1750 can not be critical, unless otherwise explained in the following.

[0189] Figure 18 is an arrangement of G-PCC compressor 1803 and G-PCC decompressor 1810 in an environment according to an embodiment. The disclosed subject matter can be equally applied to other enabled applications, including, for example, video conferencing, digital TV, storing compressed data on digital media including CDs, DVDs, memory sticks, etc.

[0190] Streaming system 1800 can include a capture subsystem 1813, which can include a source 1801 such as a digital camera that creates, for example, uncompressed data 1802. Data 1802, which has a high data volume, can be processed by a G-PCC compressor 1803 coupled to source 1801. G-PCC compressor 1803 can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in greater detail below. Encoded data 1804, which has a lower data volume than data 1802, can be stored on a streaming server 1805 for future use. One or more streaming clients 1806 and 1808 can access streaming server 1805 to retrieve copies 1807 and 1809 of encoded data 1804. Client 1806 can include a G-PCC decompressor 1810 that decodes input copy 1807 of encoded data and creates output data 1811 that can be presented on display 1812 or other presentation device (not depicted). In some streaming systems, encoded data 1804, 1807, and 1809 can be encoded according to a video encoding / compression standard. Examples of such standards include those developed by MPEG for G-PCC.

[0191] The techniques described above can be implemented in a video encoder and / or decoder adapted for compression / decompression. The encoder and / or decoder can be implemented in hardware, software, or any combination thereof, and the software, if any, can be stored in one or more non-transitory computer-readable media. For example, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute programs stored in the non-transitory computer-readable media.

[0192] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 19 A computer system 900 suitable for implementing certain embodiments of the disclosure is shown.

[0193] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or like mechanisms to create code that can be executed by a computer CPU, GPU, or like mathematical and / or logical operation

[0194] These instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, internet of things devices, and the like.

[0195] Figure 19 The components of computer system 900 illustrated in FIG. 13 are examples and intended to be illustrative only. The computer software implementing embodiments of the disclosure is not limited to the examples of components shown in FIG. 13. The configuration of the components is also not intended to limit the scope of use or functionality of computer software implementing embodiments of the disclosure.

[0196] Computer system 900 can include certain human interface input devices. Such a human interface input device can be responsive to

[0197] The input human interface devices can include one or more of: a keyboard 901, a mouse 902, a trackpad 903, a touch screen 910, data g love, a joystick 905, a microphone 906, a scanner 907, a camera 908.

[0198] The computer system 900 can also include certain human interface output devices. Such human interface output devices can be stimulating one or more of the human senses of sight, touch, taste, smell, and hearing. For example, such devices can be visual output devices (e.g., display screens of any type); auditory output devices (e.g., speakers, headphones, and the like); tactile output devices (e.g., haptic feedback devices, vibration devices, and the like); taste and / or smell

[0199] The computer system 900 can also include human accessible storage devices and their associated media such as optical media (including CD / DVD ROM / RW 920 with CD / DVD 921), thumb-drive 922, removable hard drive or solid state drive 923, legacy magnetic

[0200] Those skilled in the art should also understand that, as used in connection with the presently disclosed subject matter, the term "computer-readable medium" does not encompass transmission media, carrier waves, or other transitory signals.

[0201] The computer system 900 can also include an interface to one or more communication networks. Networks can for example be wireless, wireline, optical. Networks can further be local, wide-area, metropolitan, vehicular and industrial, real-time, delay-tolerant, and so on. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LTE and the like, TV wireline or wireless wide area digital networks to include cable TV, satellite TV, and terrestrial TV, vehicular and industrial to include CANBus, and so forth. Certain networks commonly require external network interface adapters that attached to certain general purpose data ports or peripheral buses 949 (such as USB ports of the computer system 900); others are commonly integrated into the core of the computer system 900 by attachment to a system bus as described below (for example Ethernet interface into a PC computer system or cellular interface into a smartphone computer system). Using any of these networks, the computer system 900 can communicate with other entities. Such communication can be uni-directional, receive only (for example, broadcast TV), uni-directional send-only (for example, CANbus to certain CANbus devices), or bi-directional, for example to other computer systems using local or wide-area digital networks. Such communication can include communication to and from cloud computing environments 955. Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0202] The above human interface devices, human-accessible storage devices, and network interfaces 954 can be attached to the core 940 of the computer system 900.

[0203] The core 940 can include one or more Central Processing Units (CPU) 941, Graphics Processing Units (GPU) 942, specialized programmable processing units in the form of Field Programmable Gate Arrays (FPGA) 943, hardware accelerators for certain tasks 944, and so forth. These devices, along with Read-only memory (ROM) 945, Random-access memory (RAM) 946, internal mass storage such as internal non-user accessible hard drives, SSDs, and the like, can be connected through a system bus 948. In some computer systems, the system bus 948 can be implemented using one or more physical buses, such as physical address buses, parallel buses, external buses, across which different components of the computer system 900 can communicate. In some computer systems, the system bus 948 can be implemented partially or fully with one or more mat

[0204] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute certain instructions, and combinations of these instructions can constitute the aforementioned computer code. This computer code can be stored in ROM 945 or RAM 946. Transient data can also be stored in RAM 946, while permanent data can be stored, for example, in an internal mass storage device 947. Fast storage and retrieval of any memory device can be enabled by using a cache memory, which can be closely associated with one or more CPUs 941, GPUs 942, mass storage devices 947, ROM 945, RAM 946, etc.

[0205] Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be those specifically designed and constructed for the purposes of this disclosure, or they may be of types known and available to those skilled in the art of computer software.

[0206] By way of example and not limitation, a computer system having an architecture (900) and, in particular, a kernel (940) can provide functionality as a result of executing software contained in one or more tangible computer-readable media (including CPUs, GPUs, FPGAs, accelerators, etc.). Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of the kernel 940 having non-volatile properties (such as kernel internal mass storage 947 or ROM 945). Software implementing various embodiments of this disclosure can be stored in such devices and executed by the kernel 940. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can cause the kernel 940 and, in particular, the processors therein (including CPUs, GPUs, FPGAs, etc.) to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 946 and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide functionality as a result of hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 944), which may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. References to software may, where appropriate, include logic, and vice versa. References to computer-readable media may, where appropriate, include circuitry storing software for execution (such as integrated circuits (ICs)), circuitry containing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0207] While several non-limiting embodiments have been described, there are alterations, permutations, and various substitute equivalents, which fall within the scope of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods which, although not explicitly shown or described herein, embody the principles of the disclosure and, thus, are within the spirit and scope of the disclosure.

Claims

1. A method of performing in-loop filtering in a video coding process, the method comprising: The method comprises: receiving image data including a flag indicating whether there is a loop filter block (LFB) in a first portion of a frame that has a size smaller than other portions of the frame; in response to the flag indicating that there is an LFB in the first portion of the frame that has a size smaller than other portions of the frame, partitioning the image data into a plurality of LFBs such that at least a first row or a first column of the frame has an LFB that is smaller than other portions of the frame; and applying one or more loop filters to the plurality of LFBs.

2. The method of claim 1, wherein, The partitioning is performed to allow the size of the LFB in at least one of the first row and the first column within the same frame to be smaller than other LFBs within the same frame.

3. The method of claim 2, wherein, The partitioning is performed to allow the size of the LFB in the first row to have a height that is smaller than LFBs disposed in the same frame, the LFBs disposed in the same frame not being disposed in at least one of a last row and a last column in the same frame.

4. The method of claim 2, wherein, The partitioning is performed to allow the size of the LFB in the first column to have a width that is smaller than LFBs disposed in the same frame, the LFBs disposed in the same frame not being disposed in at least one of a last row and a last column in the same frame.

5. The method of claim 1, wherein, The partitioning is performed to allow the size of the LFB in at least one of the first row and the first column to be smaller than other LFBs within the same frame and not to include a remaining portion of the partitioning in a last row and a last column within the same frame.

6. The method of claim 2, wherein, The height of the LFB in the first row and the width of the LFB in the first column are hard-coded.

7. The method of claim 2, wherein, Further comprising providing an indication indicating whether at least one of the LFB in the first row and the first column has a size smaller than LFBs in other areas of the same frame.

8. The method of claim 2, wherein, Separate syntaxes are used to indicate the height of the LFB in the first row and the width of the LFB in the first column.

9. The method of claim 2, wherein, The height of the LFB in the first row is the same as the width of the LFB in the first column.

10. The method of claim 2, wherein, A remaining portion of the partitioning is not included in a last row and a last column within the same frame.

11. An apparatus for in-loop filtering in a video coding process, the apparatus comprising: The apparatus comprises: at least one memory configured to store program code; and at least one processor configured to access the at least one memory and operate according to the program code to implement the method of any of claims 1-10.

12. A non-transitory computer-readable storage medium storing instructions, the method comprising: The instructions cause the at least one processor to perform the method of any of claims 1-10.

13. A method of performing loop filtering in a video coding process, the method comprising: The method comprises: receiving a video bitstream including image data; partitioning the image data into a plurality of loop filter blocks (LFBs), wherein the plurality of loop filter blocks (LFBs) include: one or more loop filter blocks (LFBs) having a first size corresponding to a first row or a first column of a frame; and one or more loop filter blocks (LFBs) having a second size corresponding to other portions of the frame, the second size being greater than the first size; applying one or more loop filters to the plurality of loop filter blocks (LFBs).

Images