Padding method for in-loop filter based on neural network

Abstract

A padding method for a neural network based in-loop filter is provided. The method is implemented by a video coding device. The method includes determining padding dimensions for padding samples to be applied to a video unit of a video for in-loop filtering in real time, wherein d1, d2, d3 and d4 represent padding dimensions corresponding to top, bottom, left and right boundaries of the video unit, respectively; and performing conversion between the video unit and a bitstream of the video based on the determined padding dimensions. A corresponding video coding device and a non-transitory computer-readable recording medium are also disclosed.

Description

[0001] Cross-reference to related applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application 63 / 191,121, filed May 20, 2021, entitled “Method for Filling In-Loop Filters Based on Neural Networks,” which is incorporated herein by reference. Technical Field

[0003] This invention generally relates to video encoding and decoding, and more specifically, to in-loop filters in image / video encoding and decoding. Background Technology

[0004] Digital video consumes the largest share of bandwidth in the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] The disclosed aspects / embodiments provide a technique for real-time determination of the padding dimension of padding samples to be applied to video units for in-loop filtering. Therefore, the size of the padding region can be adjusted to accommodate different types of multi-layer bitstreams and / or different stripes of different temporal layers. Consequently, the video encoding / decoding process is improved compared to conventional video encoding / decoding techniques.

[0006] The first aspect relates to a method implemented by an encoding / decoding device. The method includes: determining in real-time the padding dimensions of padding samples for in-loop filtering of video to be applied to a video unit, wherein d1, d2, d3, and d4 represent the padding dimensions corresponding to the top, bottom, left, and right boundaries of the video unit, respectively; and performing a conversion between the video unit and the video bitstream based on the determined padding dimensions.

[0007] Optionally, in any of the foregoing aspects, another implementation of that aspect provides that d1, d2, d3, and d4 each have different values; d1, d2, d3, and d4 each have the same value; or d1 = d2 and d3 = d4.

[0008] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides an indication corresponding to the padding dimension being included in the bitstream.

[0009] Optionally, in any of the foregoing aspects, another implementation of the aspect provides at least one of the padding dimensions based on the position of the video unit relative to the parent video unit from which the video unit is segmented, and wherein the padding dimension corresponding to the outer boundary of the parent video unit is different from the padding dimension corresponding to the inner boundary of the parent video unit.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the at least one padding dimension is based on a size of the video unit, a color format, a channel type, a slice type, or a partition tree type.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the at least one padding dimension is based on which temporal layer the video unit belongs to or based on reference picture information.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the at least one padding dimension is based on a type of neural network (NN) in-loop filtering method applied to the video unit or whether the NN in-loop filtering method is applied to the video unit.

[0013] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the video unit has a video unit size that depends on at least one of a slice resolution and a quantization parameter (QP), and wherein the parent video unit is a slice.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the padding dimension defines a padding region around the video unit, and wherein the method further comprises padding the padding region with padding samples when at least one neighboring sample outside the video unit is unavailable.

[0015] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the padding dimension defines a padding region around the video unit, and wherein the method further comprises padding the padding region with padding samples when at least one neighboring sample outside the video unit is unavailable.

[0016] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the padding dimension defines a padding region around the video unit, and wherein the method further comprises padding the padding region with padding samples when at least one neighboring sample outside the video unit is unavailable.

[0017] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the padding method used to pad the padding region is based on a size of the video unit.

[0018] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the padding method used to generate the padding samples outside the video unit is based on a type of neural network (NN) filtering method applied to the video unit or whether the NN filtering method is applied to the video unit.

[0019] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the padding method used to generate the padding samples outside the video unit is based on a channel type, a slice type, or which temporal layer the video unit belongs to, and wherein a neural network (NN) filter is applied to the padding samples.

[0020] Optionally, in any of the preceding aspects, another implementation of this aspect provides that a granularity of a neural network (NN) filter model to be applied to the padded samples is predefined, contained in a bitstream, derived, or based on dimensions of the video unit.

[0021] Optionally, in any of the preceding aspects, another implementation of this aspect provides that a granularity of a neural network (NN) filter model to be applied to the padded samples is based on a quantization parameter (QP) and a resolution of the video unit.

[0022] Optionally, in any of the preceding aspects, another implementation of this aspect provides that binarization of a neural network (NN) filter model index corresponding to the NN filter model to be applied to the padded samples is based on a maximum number allowed for a higher level than the video unit, wherein the higher level is a slice, picture, or sequence level, and wherein the NN filter model index is binarized as a truncated unary code or a truncated binary code.

[0023] Optionally, in any of the preceding aspects, another implementation of this aspect provides that the converting includes encoding the video unit into a bitstream.

[0024] Optionally, in any of the preceding aspects, another implementation of this aspect provides that the converting includes decoding the video unit from a bitstream.

[0025] An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: determine, in real time, padding dimensions for padded samples to be applied to a video unit of a video for in-loop filtering, wherein d1, d2, d3, and d4 represent padding dimensions corresponding to top, bottom, left, and right boundaries of the video unit, respectively; and perform a conversion between the video unit and a video bitstream based on the determined padding dimensions.

[0026] A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing apparatus, wherein the method comprises: determining, in real time, padding dimensions for padded samples to be applied to a video unit of the video for in-loop filtering, wherein d1, d2, d3, and d4 represent padding dimensions corresponding to top, bottom, left, and right boundaries of the video unit, respectively; and generating the bitstream based on the determined padding dimensions.

[0027] For clarity, any of the foregoing examples can be combined with any one or more of the other foregoing examples to create new examples within the scope of the present disclosure.

[0028] These and other features will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, and the claims. Attached Figure Description

[0029] To gain a more complete understanding of this disclosure, reference is now made to the following brief description in conjunction with the accompanying drawings and specific embodiments, wherein like reference numerals denote like parts.

[0030] Figure 1 This is an example of raster scan strip segmentation of an image.

[0031] Figure 2 This is an example of rectangular strip segmentation in an image.

[0032] Figure 3 These are examples of images segmented into pieces, bricks, and rectangular strips.

[0033] Figure 4A This is an example of a codec tree block (CTB) that spans the bottom image boundary.

[0034] Figure 4B This is an example of a CTB that spans the right edge of the image.

[0035] Figure 4C This is an example of a CTB that spans the bottom right edge of the image.

[0036] Figure 5 This is an example of an encoder block diagram.

[0037] Figure 6 This is a diagram of the sample points within an 8×8 sample point block.

[0038] Figure 7 These are examples of pixels involved in filter on / off decisions and strong / weak filter selection.

[0039] Figure 8 Four one-dimensional (1-D) orientation patterns for edge offset (EO) sample point classification are shown.

[0040] Figure 9 An example of the filter shape for an adaptive loop filter (GALF) based on geometric transformation is shown.

[0041] Figure 10 An example of relative coordinates for 5×5 rhombus filter support is shown.

[0042] Figure 11 Another example of relative coordinates for 5×5 rhombus filter support is shown.

[0043] Figure 12A This is an example architecture of the proposed convolutional neural network (CNN) filter.

[0044] Figure 12B This is an example of the construction of a ResBlock.

[0045] Figure 13 is a diagram illustrating an example of uni-directional inter prediction.

[0046] Figure 14 is a diagram illustrating an example of bi-directional inter prediction.

[0047] Figure 15 is a diagram illustrating an example of layer-based prediction.

[0048] Figure 16 illustrates a padded video unit, where di, d2, d3, and d4 are padding dimensions for the top, bottom, left, and right boundaries, respectively.

[0049] Figure 17 illustrates mirror padding, where the gray blocks represent padding samples.

[0050] Figure 18 is a block diagram showing an example video processing system.

[0051] Figure 19 is a block diagram of a video processing device.

[0052] Figure 20 is a block diagram illustrating an example of a video coding system.

[0053] Figure 21 is a block diagram illustrating an example of a video encoder.

[0054] Figure 22 is a block diagram illustrating an example of a video decoder.

[0055] Figure 23 is a method for coding video data according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0056] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary design and implementations set forth herein, but can be modified in any manner within the scope of the appended claims and their equivalents.

[0057] The use of H.266 terminology in some descriptions is merely for ease of understanding and is not intended to limit the scope of the disclosed technology. Thus, the technology described herein is applicable to other video codec protocols and designs as well.

[0058] Video coding standards have evolved primarily through the development of the well-known International Telecommunication Union - Telecommunication (ITU-T) and International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) standards. The ITU-T produced H.261 and H.263, the ISO / IEC produced the Motion Picture Expert Group (MPEG)-l and MPEG-4 Visual, and the two organizations jointly produced the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / High Efficiency Video Coding (HEVC) standards.

[0059] Since H.262, video coding standards are based on a hybrid video coding structure where temporal prediction plus transform coding is utilized. To explore future video coding technologies beyond HEVC, the Video Coding Experts Group (VCEG) and MPEG jointly founded the Joint Video Exploration Team (JVET) in 2015. Since then, many new methods have been adopted by the JVET and put into the reference software named Joint Exploration Model (JEM).

[0060] In April 2018, the Joint Video Team (JVT) was formed by VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) to work on the Versatile Video Coding (VVC) standard with the goal of 50% bitrate reduction compared to HEVC. The VVC version 1 was finalized in July 2020.

[0061] Color spaces and chroma sub-sampling are discussed. A color space, also called a color model (or color system), is an abstract mathematical model that simply describes a range of colors as a tuple of numbers, typically 3 or 4 values or color components (e.g., red green blue (RGB)). Fundamentally, a color space is an exposition of a coordinate system and a subspace.

[0062] For video compression, the most commonly used color spaces are YCbCr and RGB. Y’CbCr or Y Pb / Cb Pr / Cr, also written as YC B C R or Y’C B C R , is a family of color spaces used as part of the color image pipeline in video and digital photography systems. Y’ is the luma component, C B and C R (also called Cb and Cr) are the blue-difference and red-difference chroma components. Y’ (with the prime) is different from Y, Y is the luminance, which means the light intensity is based on the gamma-corrected RGB primary non-linear encoding. Each color component (e.g., R, B, G, Y, etc.) can be referred to as a color channel or color channel type.

[0063] Chroma subsampling is a practice of encoding images by exploiting the fact that the human visual system is less sensitive to color differences than to luminance. The chroma information is encoded with lower resolution than the luminance.

[0064] Color formats such as 4:4:4, 4:2:2, and 4:2:0 are discussed.

[0065] For 4:4:4 chroma subsampling, the three Y’CbCr components have the same sampling rate, so there is no chroma subsampling. This scheme is sometimes used for high-end film scanners and film post-production.

[0066] For 4:2:2 chroma subsampling, two chroma components are sampled at half the luminance sampling rate: horizontal chroma resolution is halved. This reduces the bandwidth of the uncompressed video signal by one third, but there is little visual difference.

[0067] For 4:2:0 chroma subsampling, the horizontal sampling is doubled compared to 4:1:1, but in this scheme the vertical resolution is halved because the Cb and Cr channels are only sampled on every other line. Thus, the data rate is the same. Cb and Cr are subsampled by a factor of two in the horizontal and vertical directions, respectively. There are three variants of the 4:2:0 scheme with different horizontal and vertical addressing.

[0068] In MPEG-2, Cb and Cr are horizontally co-sited. Cb and Cr are located between pixels in the vertical direction (in the interstitial gaps). In Joint Photographic Experts Group (JPEG) / JPEG File Interchange Format (JFIF), H.261, and MPEG-1, Cb and Cr are located in the interstitial gaps between alternate luminance samples. In 4:2:0 DV, Cb and Cr are co-sited in the horizontal direction. In the vertical direction, they are co-sited on alternate lines.

[0069] Definitions of video units are provided. A picture is divided into one or more tile rows and one or more tile columns. A tile is a sequence of coding tree units (CTUs) that cover a rectangular area of the picture. A tile is divided into one or more bricks, each of which consists of multiple CTU rows within the tile. A tile that is not divided into multiple bricks is also referred to as a brick. However, a brick that is a true subset of a tile is not referred to as a tile. A slice either contains multiple tiles of a picture or contains multiple bricks of a tile.

[0070] Two slice modes are supported, namely, a raster-scan slice mode and a rectangular slice mode. In the raster-scan slice mode, a slice contains a sequence of tiles in a raster-scan of tiles of a picture. In the rectangular slice mode, a slice contains multiple bricks of a picture that collectively form a rectangular area of the picture. The bricks within a rectangular slice are arranged in the brick raster-scan order of the slice.

[0071] Figure 1 is an example of raster scan slice partitioning of picture 100, where the picture is partitioned into 12 tiles 102 and 3 raster scan slices 104. As shown, each tile 102 and raster scan slice 104 contains multiple CTUs 106.

[0072] Figure 2 is an example of rectangular slice partitioning of picture 200 according to the VVC specification, where the picture is partitioned into 24 tiles 202 (6 tile columns 203 and 4 tile rows 205) and 9 rectangular slices 204. As shown, each tile 202 and rectangular slice 204 contains multiple CTUs 206.

[0073] Figure 3 is an example of picture 300 partitioned into tiles, bricks, and rectangular slices according to the VVC specification, where picture 300 is partitioned into four tiles 302 (two tile columns 303 and two tile rows 305), eleven bricks 304 (the top-left tile contains one brick, the top-right tile contains five bricks, the bottom-left tile contains two bricks, and the bottom-right tile contains three bricks), and four rectangular slices 306.

[0074] CTU and coding tree block (CTB) sizes are discussed. In VVC, the coding tree unit (CTU) size, signaled in the sequence parameter set (SPS) by the syntax element log2_ctu_size_minus2, can be as small as 4x4. The sequence parameter set (SPS) syntax is as follows.

[0075]

[0076]

[0077]

[0078] log2_ctu_size_minus2 plus 2 specifies the luma coding tree block size of each ctu.

[0079] log2_min_luma_coding_block_size_minus2 plus 2 specifies the minimum luma coding block size.

[0080] The derivation of the variables CtbLog2SizeY, CtbSizeY, MinCbLog2SizeY, MinCbSizeY, MinTbLog2SizeY, MaxTbLog2SizeY, MinTbSizeY, MaxTbSizeY, PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC, and PicHeightInSamplesC is as follows.

[0081] CtbLog2SizeY = log2_ctu_size_minus2 + 2 (7-9)

[0082] CtbSizeY = 1 « CtbLog2SizeY (7-10)

[0083] MinCbLog2SizeY = log2_min_luma_coding_block_size_minus2 + 2 (7-11)

[0084] MinCbSizeY = 1 « MinCbLog2SizeY (7-12)

[0085] MinTbLog2SizeY = 2 (7-13)

[0086] MaxTbLog2SizeY = 6 (7-14)

[0087] MinTbSizeY = 1 « MinTbLog2SizeY (7-15)

[0088] MaxTbSizeY = 1 « MaxTbLog2SizeY (7-16)

[0089] PicWidthInCtbsY = Ceil(pic_width_in_luma_samples ÷ CtbSizeY) (7-17)

[0090] PicHeightInCtbsY = Ceil(pic_height_in_luma_samples ÷ CtbSizeY) (7-18)

[0091] PicSizeInCtbsY = PicWidthInCtbsY * PicHeightInCtbsY (7-19)

[0092] PicWidthInMinCbsY = pic_width_in_luma_samples / MinCbSizeY (7-20)

[0093] PicHeightInMinCbsY = pic_height_in_luma_samples / MinCbSizeY (7-21)

[0094] PicSizeInMinCbsY = PicWidthInMinCbsY * PicHeightInMinCbsY (7-22)

[0095] PicSizeInSamplesY = pic_width_in_luma_samples * pic_height_in_luma_samples (7-23)

[0096] PicWidthInSamplesC = pic_width_in_luma_samples / SubWidthC (7-24)

[0097] PicHeightInSamplesC = pic_height_in_luma_samples / SubHeightC (7-25)

[0098] Figure 4A is an example of a CTB that crosses the bottom picture boundary. Figure 4B is an example of a CTB that crosses the right picture boundary. Figure 4C is an example of a CTB that crosses the bottom-right picture boundary. In Figures 4A-4C K = M, L < N; K < M, L = N; K < M, L < N, respectively.

[0099] Reference is made to Figures 4A-4C CTUs in picture 400 are discussed. Assume that a CTB maximum coding unit (LCU) size, denoted by M x N (typically, M is equal to N, as defined in HEVC / VVC), and for CTBs located at picture (or slice or tile or other kind of type, taking picture boundaries as an example) boundaries, K x L samples are within the picture boundaries, where K < M or L < N. For Figures 4A-4C For those CTBs 402 depicted in, the CTB size is still equal to M x N, however, the bottom boundary / right side boundary of the CTB is outside picture 400.

[0100] The codec process of a typical video encoder / decoder (also called codec) is discussed. Figure 5 is an example of an encoder block diagram of VVC, which contains three in-loop filters: a Deblocking Filter (DF), a Sample Adaptive Offset (SAO) filter, and an Adaptive Loop Filter (ALF). Unlike DF, which uses a pre-defined filter, the SAO filter and the ALF utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a Finite Impulse Response (FIR) filter, respectively, with the offset and filter coefficients signaled by the codec using side information. The ALF is located at the last processing stage of each picture and can be considered as a tool that tries to capture and fix artifacts caused by the preceding stages.

[0101] Figure 5 is a schematic diagram of an encoder 500. The encoder 500 is suitable for implementing the VVC technique. The encoder 500 comprises three in-loop filters, namely a Deblocking Filter (DF) 502, a Sample Adaptive Offset (SAO) filter 504, and an ALF 506. Unlike the DF 502, which uses a pre-defined filter, the SAO filter 504 and the ALF 506 utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a FIR filter, respectively, with the offset and filter coefficients signaled by the codec using side information. The ALF 506 is located at the last processing stage of each picture and can be considered as a tool that tries to capture and fix artifacts caused by the preceding stages.

[0102] The encoder 500 further comprises an intra prediction component 508 and a motion estimation / compensation (ME / MC) component 510 configured to receive input video. The intra prediction component 508 is configured to perform intra prediction, while the ME / MC component 510 is configured to perform inter prediction using reference pictures obtained from a reference picture buffer 512. Residual blocks from either inter or intra prediction are fed to a transform component 514 and a quantization component 516 to generate quantized residual transform coefficients, which are fed to an entropy coding component 518. The entropy coding component 518 entropy encodes the prediction results and the quantized transform coefficients and sends them to a video decoder (not shown). The quantized components output from the quantization component 516 can be fed to an inverse quantization component 520, an inverse transform component 522, and a reconstruction (REC) component 524. The REC component 524 is capable of outputting pictures to the DF 502, the SAO 504, and the ALF 506 for filtering before the pictures are stored in the reference picture buffer 512.

[0103] The input to DF 502 is the reconstructed samples before the in-loop filter. First, the vertical edges in the picture are filtered. Then, the horizontal edges in the picture are filtered using the samples modified by the vertical edge filtering process as input. The vertical and horizontal edges in the CTBs of each CTU are processed separately on a coding unit basis. The vertical edges of the coding blocks in a coding unit are filtered starting from the left-hand side edges of the coding blocks, proceeding through the edges to the right-hand side of the coding blocks in their geometric order. The horizontal edges of the coding blocks in a coding unit are filtered starting from the top edges of the coding blocks, proceeding through the edges to the bottom of the coding blocks in their geometric order.

[0104] Figure 6 is an illustration 600 of samples 602 within an 8x8 block of samples 604. As shown, the illustration 600 includes horizontal and vertical block boundaries 606, 608 on an 8x8 grid, respectively. In addition, the illustration 600 depicts a non-overlapping block 610 of 8x8 samples, which can be deblocked in parallel.

[0105] The discussion of the boundary decision. The filter is applied to 8x8 block boundaries. In addition, it must be a transform block boundary or a coding sub-block boundary (e.g., due to the use of affine motion prediction, alternative temporal motion vector prediction (ATMVP)). For those that are not such boundaries, the filter is disabled.

[0106] The discussion of the computation of the boundary strength. For transform block boundaries / coding sub-block boundaries, if it is located in the 8x8 grid, the transform block boundary / coding sub-block boundary can be filtered, and the bS[xD i ][yD j ](where [xD i ][yD j ] denotes the coordinates) is defined in Table 1 and Table 2, respectively.

[0107] Table 1. Boundary strength (when SPS Intra Block Copy (IBC) is disabled)

[0108]

[0109] Table 2. Boundary strength (when SPS IBC is enabled)

[0110]

[0111] The discussion of the de-blocking decision for the luma component.

[0112] Figure 7is an example 700 of pixels involved in filter on / off decision and strong / weak filter selection. The wider and stronger luma filter is used only when condition 1, condition 2 and condition 3 are all TRUE. Condition 1 is the "large block condition". This condition checks whether the samples on the P side and the Q side belong to a large block, denoted by variables bSidePisLargeBlk and bSideQisLargeBlk, respectively. The definitions of bSidePisLargeBlk and bSideQisLargeBlk are as follows.

[0113] bSidePisLargeBlk = ((edge type is vertical and p0 belongs to a CU with width >= 32) || (edge type is horizontal and p0 belongs to a CU with height >= 32))? TRUE: FALSE

[0114] bSideQisLargeBlk = ((edge type is vertical and q0 belongs to a CU with width >= 32) || (edge type is horizontal and q0 belongs to a CU with height >= 32))? TRUE: FALSE

[0115] Based on bSidePisLargeBlk and bSideQisLargeBlk, condition 1 is defined as follows.

[0116] condition 1 = (bSidePisLargeBlk || bSideQisLargeBlk)? TRUE: FALSE

[0117] Next, if condition 1 is TRUE, condition 2 will be further checked. First, the following variables are derived.

[0118] dp0, dp3, dq0, dq3 are first derived as in HEVC

[0119] if (p side >= 32)

[0120] dp0 = (dp0 + Abs(p50 - 2*p40 + p30) + 1)>>1

[0121] dp3 = (dp3 + Abs(p53 - 2*p43 + p33) + 1)>>1

[0122] if (q side >= 32)

[0123] dq0 = (dq0 + Abs(q50 - 2*q40 + q30) + 1)>>1

[0124] dq3 = (dq3 + Abs(q53 - 2*q43 + q33) + 1)>>1

[0125] condition 2 = (d < β)? TRUE: FALSE

[0126] where d = dp0+ dq0+ dp3+ dq3.

[0127] If condition 1 and condition 2 are valid, then further check if any block uses sub-blocks.

[0128]

[0129] Finally, if both condition 1 and condition 2 are valid, the proposed deblocking method will check condition 3 (large block strong filter condition), which is defined as follows.

[0130] In condition 3 StrongFilterCondition, the following variables are derived.

[0131] dpq is derived as in HEVC.

[0132] sp3 = Abs(p3 - p0) is derived as in HEVC,

[0133]

[0134] sq3 = Abs(q0 - q3) is derived as in HEVC

[0135]

[0136] StrongFilterCondition = (dpq < (β » 2), sp3 + sq3 < (3 * β » 5), Abs(p0 - q0) < (5 * t C + 1) » 1)? TRUE : FALSE.

[0137] A stronger luma deblocking filter (designed for larger blocks) is discussed.

[0138] When the samples on either side of the boundary belong to a large block, a bilinear filter is used. A sample belonging to a large block is defined as when the width of the vertical edge >= 32 and the height of the horizontal edge >= 32.

[0139] The bilinear filter is shown as follows.

[0140] In the above HEVC deblocking, the block boundary samples pi for i = 0 to Sp - 1 and qj for j = 0 to Sq - 1 (pi and qi are the i-th sample in the row for filtering a vertical edge, or the i-th sample in the column for filtering a horizontal edge) are then replaced by linear interpolation as follows.

[0141] p i ′ = (f i * Middle s,t + (64 - f i ) * Ps + 32) » 6), clipped to p i ± tcPD i

[0142] q j ' = (g j * Middle s,t + (64 - g j ) * Q s + 32) » 6), clipped to q j ± tcPD j

[0143] where tcPD i and tcPD j The term is position dependent clipping (cliping), as described below; and g j , f i , Middle s,t , P s and Q s are given below.

[0144] The control of chroma deblocking is discussed.

[0145] A strong chroma filter is used on both sides of the block boundary. Here, the chroma filter is selected when both sides of the chroma edge are greater than or equal to 8 (chroma position) and the following three conditions are met. The first is the boundary strength and large block decision. The proposed filter can be applied when the block width or height in the chroma sample domain that is orthogonal to the block edge is equal to or greater than 8. The second and third decisions are basically the same as the HEVC luma deblocking decisions, which are the on / off decision and the strong filter decision, respectively.

[0146] In the first decision, the boundary strength (bS) is modified for chroma filtering and the conditions are checked sequentially. If the conditions are met, the remaining decisions with lower priority will be skipped.

[0147] Chroma deblocking is performed when bS is equal to 2 or when bS is equal to 1 when a large block boundary is detected.

[0148] The second and third decisions are basically the same as the HEVC luma strong filter decisions, as follows.

[0149] In the second decision: d is then derived as in HEVC luma deblocking. The second decision will be true when d is less than β.

[0150] In the third decision, StrongFilterCondition is derived as follows.

[0151] As derived in HEVC, sp3 = Abs(p3 - p0),

[0152] As derived in HEVC, sq3 = Abs(q0 - q3),

[0153] As in the HEVC design, StrongFilterCondition = (dpq < (β » 2), sp3 + sq3 < (β » 3), Abs(p0 - q0) < (5 * t C + 1) » 1).

[0154] Strong de-blocking filters for chroma are discussed. The following strong de-blocking filters for chroma are defined.

[0155] p2' = (3 * p3 + 2 * p2 + p1 + p0 + q0 + 4) » 3

[0156] p1' = (2 * p3 + p2 + 2 * p1 + p0 + q0 + q1 + 4) » 3

[0157] p0' = (p3 + p2 + p1 + 2 * p0 + q0 + q1 + q2 + 4) » 3

[0158] The proposed chroma filters perform de-blocking on a 4x4 grid of chroma samples.

[0159] Position dependent clipping (tcPD) is discussed. Position dependent clipping tcPD is applied to the output samples of the luma filtering process which involves the strong and long filters modifying 7, 5 and 3 samples at the boundaries. Assuming a quantization error distribution, it is suggested to increase the clipping values for samples which are expected to have higher quantization noise, thus expected to have a larger deviation of the reconstructed sample value from the true sample value.

[0160] For each P or Q boundary filtered with an asymmetric filter, a position dependent threshold table is selected from two tables provided as side information to the decoder, namely Tc7 and Tc3 in the following tables, depending on the result of the decision process in the boundary strength calculation.

[0161] Tc7 = {6, 5, 4, 3, 2, 1, 1}; Tc3 = {6, 4, 2};

[0162] tcPD = (Sp == 3)? tc3 : Tc7;

[0163] tcQD = (Sq == 3)? tc3 : Tc7;

[0164] For P or Q boundaries filtered with short symmetric filters, lower amplitude position dependent threshold values are applied.

[0165] Tc3 = {3, 2, 1};

[0166] After defining the threshold, the filtered p' is adjusted based on the tcP and tcQ clipping values. i and q' i The sample values ​​are clipped.

[0167] p” i =Clip3(p' i +tcP i ,p' i –tcP i ,p' i );

[0168] q” j =Clip3(q' j +tcQ j ,q' j –tcQ j ,q' j );

[0169] Where p' i and q' i These are the filtered sample values, p” i and q” j It is the output sample value after cropping, tcP i tcQ j The clipping threshold is derived from the VVC tc parameters, as well as tcPD and tcQD. The Clip3 function is the clipping function specified in VVC.

[0170] The adjustment of sub-blocks to remove blocks was discussed.

[0171] To achieve parallel-friendly deblocking using long filters and sub-block deblocking, the long filter is restricted to modifying a maximum of 5 samples on the side using sub-block deblocking (affine or ATMVP or decoder-side motion vector refinement (DMVR)), as shown in the long filter's brightness control. Furthermore, sub-block deblocking is adjusted such that sub-block boundaries on the 8×8 grid near the boundaries of the codec unit (CU) or implicit transform unit (TU) are restricted to modifying a maximum of two samples on each side.

[0172] The following applies to sub-block boundaries that are not aligned with the CU boundary.

[0173]

[0174]

[0175] Edges equal to 0 correspond to the CU boundary, edges equal to 2 or orthogonalLength-2 correspond to 8 samples from the sub-block boundary of the CU boundary, and so on. Implicit TU is true if implicit splitting of TU is used.

[0176] Sample adaptive offset is discussed. The input to SAO is the deblocked reconstructed samples (DB). The concept of SAO is to reduce the average sample distortion of a region by first classifying the region samples into multiple classes using a selected classifier, obtaining an offset for each class, and then adding the offset to each sample of the class, where the classifier index and the region's offset are coded in the bitstream. In HEVC and VVC, a region (the unit for SAO parameter signaling) is defined as a CTU.

[0177] HEVC employs two types of SAO that can meet the low complexity requirement. The two types are edge offset (EO) and band offset (BO), which are discussed in detail below. The index of the SAO type is coded (in the range of [0, 2]). For EO, the sample classification is based on the comparison between the current sample and the neighboring samples according to one-dimensional directional modes: horizontal, vertical, 135° diagonal, and 45° diagonal.

[0178] Figure 8 Four one-dimensional (1-D) directional modes 800 for EO sample classification are shown: horizontal (EO class = 0), vertical (EO class = 1), 135° diagonal (EO class = 2), and 45° diagonal (EO class = 3).

[0179] For a given EO class, each sample within a CTB is classified into one of five classes. The current sample value, labeled “c”, is compared to its two neighboring sample values along the selected 1-D mode. The classification rule for each sample is summarized in Table 3. Classes 1 and 4 are associated with local valleys and local peaks along the selected 1-D mode, respectively. Classes 2 and 3 are associated with concave and convex corners along the selected one-dimensional pattern, respectively. If the current sample does not belong to EO classes 1-4, it belongs to class 0 and no SAO is applied.

[0180] Table 3: Sample classification rule for edge offset

[0181]

[0182] Joint Exploration Model (JEM) based on geometric transform adaptive loop filter is discussed. The input to DB is the reconstructed samples after DB and SAO. The sample classification and filtering process is based on the reconstructed samples after DB and SAO.

[0183] In JEM, a geometric transform based adaptive loop filter (GALF) with block-based filter adaptation is applied. For the luma component, one of 25 filters is selected for each 2x2 block according to the direction and activity of local gradients.

[0184] Filter shapes are discussed. Figure 9 An example of GALF filter shapes 900 is shown, including a 5x5 diamond on the left, a 7x7 diamond on the right, and a 9x9 diamond in the middle. In JEM, up to three diamond filter shapes can be selected for the luma component (as shown in Figure 9 The index is signaled at picture level to indicate the filter shape used for the luma component. Each square represents a sample, and Ci (i is 0-6 (left), 0-12 (middle), 0-20 (right)) represents the coefficient applied to that sample. For the chroma components in a picture, a 5x5 diamond is always used.

[0185] Block classification is discussed. Each 2x2

[0186] block is classified into one of 25 classes. The classification index C is derived based on its directionality D and activity quantized values, as follows.

[0187]

[0188] To compute D and First, the gradients in horizontal, vertical and two diagonal directions are computed using a 1-D Laplacian operator.

[0189]

[0190]

[0191]

[0192]

[0193] The indices i and j refer to the coordinates of the top-left sample in the 2x2 block, and R(i,j) indicates the reconstructed sample at coordinate (i,j).

[0194] The maximum and minimum of the gradients in horizontal and vertical directions are then set as:

[0195]

[0196] And the maximum and minimum of the gradients in the two diagonal directions are set as:

[0197]

[0198] To derive the value of directionality D, these values are compared with each other and with two thresholds t1 and t2:

[0199] First step. If and are both true, then D is set to 0.

[0200] Second step. If Continue from step 3; otherwise continue from step 4.

[0201] Third step. If D is set to 2; otherwise D is set to 1.

[0202] Fourth step. If D is set to 4; otherwise D is set to 3.

[0203] The activity value A is computed as follows:

[0204]

[0205] A is further quantized to the range 0 to 4, inclusive, and the quantized value is denoted as

[0206] For both chroma components in a picture, the classification method is not applied, i.e. a single set of ALF coefficients is applied for each chroma component.

[0207] Geometrical transformations of filter coefficients are discussed.

[0208] Figure 10 An example of relative coordinates 1000 for a 5x5 diamond filter support is shown, respectively diagonal, vertical flip and rotation (from left to right).

[0209] Before filtering each 2x2 block, a geometrical transformation such as rotation or diagonal and vertical flip is applied to the filter coefficients f(k,l) associated with the coordinates (k,l) depending on the gradient value computed for this block. This is equivalent to applying these transformations to the samples in the filter support region. The idea is to make different blocks for which ALF is applied more similar by arranging the directionality of the different blocks.

[0210] Three geometrical transformations are introduced, including diagonal, vertical flip and rotation:

[0211] Diagonal: f D (k,l) = f(l,k),

[0212] Vertical flip: f V (k,l) = f(k,K-l-1) (9)

[0213] Rotation: f R (k,l) = f(K-l-1,k).

[0214] where K is the size of the filter, 0≤k,l≤K-1 are the coefficient coordinates, with position (0,0) at the top-left corner and (K-1,K-1) at the bottom-right corner. A transform is applied to the filter coefficients f(k,l) according to the gradient value computed for the block. Table 4 summarizes the mapping between the transform and the four gradients of the four directions.

[0215] Table 4: Mapping between the gradients computed for a block and the transform

[0216] Gradient value Transform g d2 g d1 g h g v ]]> No transform g d2 g d1 g v g h ]]> Diagonal g d1 g d2 g h g v ]]> Vertical flip g d1 g d2 and g v g h ]]> Rotation

[0217] Filter parameter signaling is discussed. In JEM, GALF filter parameters are signaled for the first CTU, i.e. after the slice header of the first CTU and before the SAO parameters. Up to 25 sets of luma filter coefficients can be signaled. To reduce the bit overhead, different categories of filter coefficients can be merged. Furthermore, the GALF coefficients of a reference picture are stored and allowed to be reused as GALF coefficients of the current picture. The current picture can choose to use the stored GALF coefficients of a reference picture and bypass the GALF coefficient signaling. In this case, only the index to one of the reference pictures is signaled and the stored GALF coefficients of the indicated reference picture are inherited by the current picture.

[0218] To support GALF temporal prediction, a candidate list of GALF filter sets is maintained. At the beginning of the decoding of a new sequence, the candidate list is empty. After the decoding of a picture, the corresponding filter set can be added to the candidate list. Once the size of the candidate list reaches the maximum allowed value (i.e. 6 in the current JEM), the new filter set overwrites the oldest set in the decoding order, i.e. a first-in-first-out (FIFO) rule is applied to update the candidate list. To avoid duplication, a set is only added to the list if the corresponding picture does not use GALF temporal prediction. To support temporal scalability, there are multiple candidate lists of filter sets and each candidate list is associated with a temporal layer. More specifically, each array assigned by a temporal layer index (TempIdx) can consist of filter sets of previously decoded pictures with lower TempIdx. For example, the k-th array is assigned to be associated with TempIdx equal to k and the k-th array only contains filter sets from pictures with TempIdx smaller than or equal to k. After a certain picture is coded, the filter set associated with the picture will be used to update those arrays associated with TempIdx equal to or higher.

[0219] Temporal prediction of the GALF coefficients is used for inter coded frames to minimize the signaling overhead. For intra frames, temporal prediction is not available and a set of 16 fixed filters is assigned to each class. To indicate the use of a fixed filter, a flag for each class is signaled and, if needed, the index of the selected fixed filter is also signaled. Even when a fixed filter is selected for a given class, the coefficients f(k,l) of an adaptive filter can still be transmitted for that class, in which case the coefficients of the filter that will be applied to the reconstructed picture are the sum of the two sets of coefficients.

[0220] The filtering process for the luma component can be controlled at the CU level. A flag is signaled to indicate whether GALF is applied to the luma component of a CU. For the chroma components, whether GALF is applied is only indicated at the picture level.

[0221] The filtering process is discussed. At the decoder side, when GALF is enabled for a block, each sample R(i,j) within the block is filtered, resulting in a sample value R'(i,j) as shown below, where L denotes the filter length, f m,n denotes the filter coefficients, and f(k,l) denotes the decoded filter coefficients.

[0222]

[0223] Figure 11 Another example of relative coordinates 1100 for a 5x5 diamond filter support is shown assuming the coordinates (i,j) of the current sample are (0,0). The samples in different coordinates that are filled with the same color are multiplied by the same filter coefficient.

[0224] The geometric transform based adaptive loop filter (GALF) in VVC is discussed. In VVC test model 4.0 (VTM4.0), the filtering process of the adaptive loop filter is performed as follows:

[0225] O(x,y) =∑ (i,j) w(i,j).I(x+i,y+j), (11)

[0226] where the sample I(x+i,y+j) is the input sample, O(x,y) is the filtered output sample (i.e., the filter result), and w(i,j) denotes the filter coefficient. In practice, in VTM4.0, it is implemented using integer operations to achieve fixed-point precision calculation

[0227]

[0228] where L denotes the filter length, and where w(i,j) is the filter coefficient in fixed-point precision.

[0229] The current design of GALF in VVC has the following major changes compared to JEM:

[0230] 1) The adaptive filter shape is removed. For luma component, only 7x7 filter shape is allowed, for chroma component, only 5x5 filter shape is allowed.

[0231] 2) The signaling of ALF parameters is removed from slice / picture level to CTU level.

[0232] 3) The calculation of class index is performed at 4x4 level instead of 2x2 level. In addition, the sub-sampling Laplacian calculation method for ALF classification is utilized as proposed in JVET-L0147. More specifically, there is no need to calculate horizontal / vertical / 45 diagonal / 135 degree gradient for each sample within a block. Instead, 1:2 sub-sampling is used.

[0233] The non-linear ALF in current VVC is discussed in relation to the filtering reconstruction.

[0234] Equation (11) can be re-expressed as the following expression without affecting the coding efficiency:

[0235] O(x, y) = I(x, y) +∑ (i,j)≠(0,0) w(i, j).(I(x+i, y+j)-I(x, y)), (13)

[0236] where w(i, j) is the filter coefficient same as in equation (11) [except that w(0,0) in equation (13) is equal to 1, while it is equal to 1-∑ (i,j)≠(0,0) w(i, j) in equation (11)].

[0237] Using the above filter formula of equation (13), VVC introduces non-linearity to make ALF more effective by using a simple clipping function to reduce the impact of neighboring sample values (I(x+i, y+j)) when they differ too much from the filtered current sample value (I(x, y)).

[0238] More specifically, the ALF filter is modified as follows:

[0239] O'(x, y) = I(x, y) +∑ (i,j)≠(0,0) w(i, j).K(I(x+i, y+j)-I(x, y), k(i, j)), (14)

[0240] where K(d, b) = min(b, max(-b, d)) is the clipping function, k(i, j) is the clipping parameter depending on the filter coefficient (i, j). The encoder performs optimization to find the best value k(i, j).

[0241] In the JVET-N0242 implementation, a clipping parameter k(i,j) is specified for each ALF filter, one clipping value is signaled per filter coefficient. This means that up to 12 clipping values can be signaled in the bitstream per luma filter, and up to 6 clipping values can be signaled per chroma filter.

[0242] To limit the signaling cost and encoder complexity, only 4 fixed values are used, which are the same for inter and intra.

[0243] Because the variance of local differences is typically higher for luma than for chroma, two different sets of clipping values are applied for luma and chroma filters. A maximum sample value in each set is also introduced (here 1024 for 10-bit bit-depth) so that clipping can be disabled when not necessary.

[0244] Table 5 provides the set of clipping values used in the JVET-N0242 tests. These 4 values are chosen by roughly equally dividing the full range of sample values for luma (coded in 10 bits) and the range from 4 to 1024 for chroma in the log domain.

[0245] More precisely, the luma table of clipping values is obtained by the following formula:

[0246] with M = 2 10并且 N = 4. (15)

[0247] Similarly, the chroma table of clipping values is obtained according to the following formula:

[0248] with M = 2 10 N = 4 and A = 4. (16)

[0249] Table 5: authorized clipping values

[0250]

[0251] The selected clipping values are coded in the “alf_data” syntax element by using a Golomb coding scheme corresponding to the index of the clipping values in the above table 5. The coding scheme is the same as for the coding of the filter indices.

[0252] Convolutional neural network based loop filters for video coding are discussed.

[0253] In deep learning, convolutional neural networks (CNNs, or ConvNets) are a class of deep neural networks most commonly used for analyzing visual images. They have very successful applications in image and video recognition / processing, recommender systems, image classification, medical image analysis, and natural language processing.

[0254] CNNs are regularized versions of multilayer perceptrons. Multilayer perceptrons generally mean fully connected networks, i.e., each neuron in one layer is connected to all neurons in the next layer. The "full connectivity" of these networks makes them prone to overfitting the data. Typical regularization methods include adding some form of weight magnitude measure to the loss function. CNNs take a different regularization approach: they exploit the hierarchical patterns in the data and assemble more complex patterns using smaller and simpler ones. Thus, CNNs are at the lower end of the spectrum in terms of connectivity and complexity.

[0255] CNNs use relatively little preprocessing compared to other image classification / processing algorithms. This means that the network learns the filters that are hand-designed in traditional algorithms. This independence from prior knowledge and human effort in feature design is a major advantage.

[0256] Deep learning based image / video compression generally has two meanings: purely neural network based end-to-end compression and traditional framework enhanced by neural network. The neural network based end-to-end compression is discussed in the following: Johannes Ballé, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” 2016 Picture Coding Symposium (PCS), pp. 1-5, IEEE, and Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszar, “Lossy image compression with compressive autoencoders,” arXiv preprint arXiv: 1703.00395 (2017).The traditional framework augmented by neural networks is discussed in Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao,“Fully Connected Network-Based Intra Prediction for Image Coding” IEEE Transactions on Image Processing 27, 7 (2018), 3236-3247; Yuanying Dai, Dong Liu, and Feng Wu,“A convolutional neural network approach for post-processing in HEVC intra coding” MMM. Springer, 28-39; Rui Song, Dong Liu, Houqiang Li, and Feng Wu,“Neural network-based arithmetic coding of intra prediction modes in HEVC” VCIP. IEEE, 1-4; and J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, W. Samek, H. Schwarz, D. Marpe, and T. Wiegand,“Neural network based intra prediction for video coding” Applications of Digital Image Processing XLI, Vol. 10752. International Society for Optics and Photonics, 1075213.

[0257] End-to-end compression usually adopts an autoencoder-like structure, implemented by either convolutional neural networks or recurrent neural networks. While relying purely on neural networks for image / video compression can avoid any manual optimization or hand-crafted design, the compression efficiency can not be satisfactory. Therefore, works in the second type of compression assist neural networks by replacing or enhancing certain modules to boost traditional compression frameworks. In this way, they can inherit the advantages of highly optimized traditional frameworks. For example, fully connected network-based intra prediction for image coding in HEVC as discussed in Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao, “Fully Connected Network-Based Intra Prediction for Image Coding,” IEEE Transactions on Image Processing 27, 7 (2018), p. 3236-3247.

[0258] Besides intra prediction, deep learning is also used to enhance other modules. For example, the in-loop filter of HEVC is replaced by a convolutional neural network in Yuanying Dai, Dong Liu, and Feng Wu, “A convolutional neural network approach for post-processing in HEVC intra coding,” MMM. Springer, 28-39. Rui Song, Dong Liu, Houqiang Li, and Feng Wu, “Neural network-based arithmetic coding of intra prediction modes in HEVC,” VCIP. IEEE, 1-4, which achieves satisfactory results. Neural networks are applied to improve the arithmetic coding engine.

[0259] In-loop filtering based on convolutional neural networks is discussed. In lossy image / video compression, the reconstructed frame is an approximation of the original frame, because the quantization process is irreversible, resulting in distortion of the reconstructed frame. To alleviate this distortion, a convolutional neural network can be trained to learn the mapping from the distorted frame to the original frame. In practice, training must be performed before deploying CNN-based in-loop filtering.

[0260] Training is discussed. The purpose of the training process is to find the best values of the parameters including weights and biases.

[0261] First, a codec (e.g., HM, JEM, VTM, etc.) is used to compress the training dataset to generate distorted reconstructed frames. Then, the reconstructed frames are fed into the CNN, and the cost is calculated using the output of the CNN and the ground truth frames (original frames). Commonly used cost functions include the sum of absolute differences (SAD) and the mean squared error (MSE). Next, the gradient of the cost with respect to each parameter is derived through a backpropagation algorithm. With the gradient, the parameter values can be updated. The above process is repeated until the convergence criterion is met. After the training is completed, the derived best parameters are saved for the inference phase.

[0262] The convolution process is discussed. In the convolution process, the filter moves over the picture from left to right and top to bottom, changing column by one pixel horizontally and row by one pixel vertically. The amount of movement between the application of the filter to the input image is called the stride, and it is almost always symmetric in the height and width dimensions. The default stride in two dimensions is (1, 1) for both height and width movement.

[0263] Figure 12A is an example architecture 1200 of the proposed CNN filter, Figure 12B is an example of the construction 1250 of a residual block (ResBlock). In most deep convolutional neural networks, the residual block is used as a basic module and is stacked several times to build the final network, where in one example, the residual block is obtained by combining a convolutional layer, a ReLU / PReLU activation function, and a convolutional layer, as shown in Figure 12B .

[0264] Inference is discussed. In the inference phase, the distorted reconstructed frames are fed into the CNN and processed by the CNN model whose parameters have been determined in the training phase. The input samples of the CNN can be the reconstructed samples before DB or after DB, or the reconstructed samples before SAO or after SAO, or the reconstructed samples before ALF or after ALF.

[0265] There are the following problems with the current CNN-based in-loop filter. First, the padding size is fixed for different types of different slices (e.g., intra mode slice (I slice), uni-prediction slice (P slice), and bi-prediction slice (B slice)). Second, the padding size is fixed for different temporal layers.

[0266] Techniques are disclosed herein that address one or more of the foregoing problems. For example, the invention provides techniques for determining, in real-time, padding dimensions for padding samples to be applied to a video unit for in-loop filtering. Thus, the size of the padding region can be adjusted to accommodate different types of multi-layer bitstreams and / or different slices of temporal layers. As a result, the video coding process is improved relative to conventional video coding techniques.

[0267] The following detailed examples should be considered illustrative of the general concepts explained herein. These examples are not to be construed as limiting. Moreover, these examples can be combined in any manner.

[0268] One or more neural network (NN) filter models are trained as part of in-loop filtering techniques or filtering techniques used in post-processing stages for reducing distortions caused during compression. Samples with different characteristics are processed by different neural network filter models. The invention sets forth how to decide on padding dimensions for different video units to achieve better performance and how to handle samples located at the boundaries of video units.

[0269] In this disclosure, a NN filter can be any kind of NN filter, e.g., a convolutional neural network (CNN) filter. In the following discussion, a NN filter can also be referred to as a non-CNN filter, e.g., a filter using a machine learning based solution.

[0270] In the following discussion, a video unit can be a picture sequence, a picture, a slice, a tile, a brick, a subpicture, a CTU / CTB, a CTU / CTB row, one or more CUs / coding blocks (CBs), one or more CTUs / CTBs, one or more virtual pipeline data units (VPDUs), a subregion within a picture / slice / tile / brick. A parent video unit (also referred to as a parent video unit) represents a unit larger than a video unit. Typically, a parent video unit will contain several video units, e.g., when a video unit is a CTU, a parent video unit can be a slice, a CTU row, multiple CTUs, etc. In some embodiments, a video unit can be a sample / pixel.

[0271] Figure 13 FIG. 13 is a schematic diagram showing an example of uni-directional inter prediction 1300. Uni-directional inter prediction 1300 can be used to determine motion vectors for encoded and / or decoded blocks created when partitioning a picture.

[0272] Unidirectional inter prediction 1300 employs a reference frame 1330 having a reference block 1331 to predict a current block 1311 in a current frame 1310. As shown, the reference frame 1330 can be temporally subsequent to the current frame 1310 (e.g., as a subsequent reference frame), but in some examples can also be temporally prior to the current frame 1310 (e.g., as an earlier reference frame). The current frame 1310 is an example frame / picture that is being encoded / decoded at a particular time. The current frame 1310 contains an object in the current block 1311 that matches an object in the reference block 1331 of the reference frame 1330. The reference frame 1330 is a frame used as a reference to encode the current frame 1310, and the reference block 1331 is a block in the reference frame 1330 that contains an object that is also contained in the current block 1311 of the current frame 1310.

[0273] The current block 1311 is any coding unit being encoded / decoded at a specified point in the coding process. The current block 1311 can be an entire partitioned block, or a sub-block when an affine inter prediction mode is employed. The current frame 1310 is separated from the reference frame 1330 by a temporal distance (TD) 1333. The TD 1333 indicates an amount of time between the current frame 1310 and the reference frame 1330 in a video sequence, and can be measured in units of frames. Prediction information for the current block 1311 can reference the reference frame 1330 and / or the reference block 1331 by a reference index that indicates a direction and a temporal distance between frames. Within the temporal segment represented by the TD 1333, an object in the current block 1311 moves from a location in the current frame 1310 to another location in the reference frame 1330 (e.g., a location of the reference block 1331). For example, the object can move along a motion trajectory 1313, which is a direction in which the object moves over time. A motion vector 1335 describes a direction and magnitude of the object’s motion along the motion trajectory 1313 over the TD 1333. Thus, the encoded motion vector 1335, the reference block 1331, and a residual that includes a difference between the current block 1311 and the reference block 1331 provide information sufficient to reconstruct the current block 1311 and position the current block 1311 in the current frame 1310.

[0274] Figure 14 is a diagram illustrating an example of bi-directional inter prediction 1400. Bi-directional inter prediction 1400 can be used to determine motion vectors for encoded and / or decoded blocks created when partitioning a picture.

[0275] Bidirectional inter prediction 1400 is similar to unidirectional inter prediction 1300, but employs a pair of reference frames to predict a current block 1411 in a current frame 1410. Thus, current frame 1410 and current block 1411 are substantially similar to current frame 1310 and current block 1311, respectively. Current frame 1410 is positioned in the time domain between a previous reference frame 1420 that precedes current frame 1410 in the video sequence and a subsequent reference frame 1430 that follows current frame 1410 in the video sequence. Previous reference frame 1420 and subsequent reference frame 1430 are substantially similar to reference frame 1330 in other respects.

[0276] Current block 1411 is matched to a previous reference block 1421 in previous reference frame 1420 and a subsequent reference block 1431 in subsequent reference frame 1430. This matching indicates that the object moves along motion trajectory 1413 and from a position at previous reference block 1421 to a position at subsequent reference block 1431 via current block 1411 over the course of the video sequence. Current frame 1410 is separated from previous reference frame 1420 by some previous temporal distance (TD0) 1423 and from subsequent reference frame 1430 by some subsequent temporal distance (TD1) 1433. TD0 1423 indicates, in units of frames, an amount of time between previous reference frame 1420 and current frame 1410 in the video sequence. TD1 1433 indicates, in units of frames, an amount of time between current frame 1410 and subsequent reference frame 1430 in the video sequence. Thus, the object moves along motion trajectory 1413 from previous reference block 1421 to current block 1411 over a period of time indicated by TD0 1423. The object also moves along motion trajectory 1413 from current block 1411 to subsequent reference block 1431 over a period of time indicated by TD1 1433. Prediction information for current block 1411 can reference previous reference frame 1420 and / or previous reference block 1421 and subsequent reference frame 1430 and / or subsequent reference block 1431 by indicating a pair of reference indices for direction and temporal distance between frames.

[0277] A previous motion vector (MV0) 1425 describes a direction and magnitude of motion of the object along motion trajectory 1413 over TD0 1423 (e.g., between previous reference frame 1420 and current frame 1410). A subsequent motion vector (MV1) 1435 describes a direction and magnitude of motion of the object along motion trajectory 1413 over TD1 1433 (e.g., between current frame 1410 and subsequent reference frame 1430). In this way, in bidirectional inter prediction 1400, current block 1411 can be coded and reconstructed by employing previous reference block 1421 and / or subsequent reference block 1431, MV0 1425, and MV1 1435.

[0278] In one embodiment, inter prediction and / or bi-directional inter prediction can be performed on a sample-by-sample (e.g., pixel-by-pixel) basis rather than on a block-by- block basis. That is, a motion vector can be determined for each sample in the current block 1411 that points to each sample in the previous reference block 1421 and / or the subsequent reference block 1431. In such an embodiment, Figure 14 The previous motion vector 1425 and the subsequent motion vector 1435 depicted in FIG. 15 represent multiple motion vectors corresponding to multiple samples in the current block 1411, the previous reference block 1421, and the subsequent reference block 1431.

[0279] In merge mode and advanced motion vector prediction (AMVP) mode, a candidate list is generated by adding candidate motion vectors to the candidate list in the order defined by the mode of the candidate list determination. Such candidate motion vectors can include motion vectors according to uni-directional inter prediction 1300, bi-directional inter prediction 1400, or a combination thereof. Specifically, when neighboring blocks are encoded, motion vectors are generated for these blocks. Such motion vectors are added to the candidate list for the current block, and a motion vector for the current block is selected from the candidate list. The motion vector can then be signaled as an index to the selected motion vector in the candidate list. A decoder can use the same process as the encoder to build the candidate list, and can determine the selected motion vector from the candidate list based on the signaled index. Thus, the candidate motion vectors include motion vectors generated according to uni-directional inter prediction 1300 and / or bi-directional inter prediction 1400, depending on which method was used when encoding such neighboring blocks.

[0280] A slice is an integer number of complete tiles or an integer number of consecutive complete coding tree units (CTU) rows within one tile of a picture that is exclusively contained in a single network abstraction layer (NAL) unit. When a slice contains one or more video units generated using intra prediction, the slice can be referred to as an I slice or I slice type. When a slice contains one or more video units generated using uni-directional inter prediction as shown in Figure 13 When a slice contains one or more video units generated using bi-directional inter prediction as shown in Figure 14 When a slice contains one or more video units generated using bi-directional inter prediction as shown in

[0281] Figure 15 is a diagram illustrating an example of layer-based prediction 1500. Layer-based prediction 1500 is compatible with uni-directional inter prediction and / or bi-directional inter prediction, but also performs between pictures in different layers.

[0282] Layer-based prediction 1500 is applied between pictures 1511, 1512, 1513, and 1514 in a different layer (also referred to as a temporal layer) and pictures 1515, 1516, 1517, and 1518. In the example shown, pictures 1511, 1512, 1513, and 1514 are part of layer N+1 1532, and pictures 1515, 1516, 1517, and 1518 are part of layer N 1531. A layer, such as layer N 1531 and / or layer N+1 1532, is a group of pictures that are all associated with similar values of characteristics, such as similar size, quality, resolution, signal-to-noise ratio, capacity, etc. In the example shown, layer N+1 1532 is associated with larger picture sizes than layer N 1531. Thus, in this example, pictures 1511, 1512, 1513, and 1514 in layer N+1 1532 have larger picture sizes (e.g., larger height and width, and thus more samples) than pictures 1515, 1516, 1517, and 1518 in layer N 1531. However, such pictures can be separated between layer N+1 1532 and layer N 1531 by other characteristics. While only two layers, layer N+1 1532 and layer N 1531, are shown, a group of pictures can be separated into any number of layers based on associated characteristics. Layer N+1 1532 and layer N 1531 can also be represented by a layer identifier (ID). A layer ID is a data item associated with a picture and indicates that the picture is part of the indicated layer. Thus, each picture 1511-1518 can be associated with a corresponding layer identifier (ID) to indicate which of layer N+1 1532 or layer N 1531 includes the corresponding picture.

[0283] Pictures 1511-1518 in different layers 1531-1532 are configured to be displayed alternately. As such, pictures 1511-1518 in different layers 1531-1532 can share the same temporal ID and can be included in the same access unit (AU) 1506. As used herein, an AU is a set of one or more coded pictures associated with the same display time domain for output from a decoded picture buffer (DPB). For example, a decoder can decode and display picture 1515 at the current display time if smaller pictures are desired, or a decoder can decode and display picture 1511 at the current display time if larger pictures are desired. As such, pictures 1511-1514 at higher layer N+1 1532 contain substantially the same picture data as corresponding pictures 1515-1518 at lower layer N 1531 (although the picture sizes are different). Specifically, picture 1511 contains substantially the same picture data as picture 1515, picture 1512 contains substantially the same picture data as picture 1516, and so on.

[0284] Pictures 1511-1518 can be coded by referring to other pictures 1511-1518 in the same layer N 1531 or layer N+1 1532. Coded pictures to another picture in the same layer results in inter-prediction 1523, which is compatible with uni-directional inter-prediction and / or bi-directional inter-prediction. Inter-prediction 1523 is depicted by solid arrows. For example, picture 1513 can be coded with inter-prediction 1523 by using one or both of pictures 1511, 1512, and / or 1514 in layer N+1 1532 as reference, where one picture is referenced for uni-directional inter-prediction and / or two pictures are referenced for bi-directional inter-prediction. Further, picture 1517 can be coded with inter-prediction 1523 by using one or both of pictures 1515, 1516, and / or 1518 in layer N 1531 as reference, where one picture is referenced for uni-directional inter-prediction and / or two pictures are referenced for bi-directional inter-prediction. When a picture is used as a reference picture for another picture in the same layer when performing inter-prediction 1523, the picture can be referred to as a reference picture. For example, picture 1512 can be a reference picture used to code picture 1513 according to inter-prediction 1523. In a multi-layer context, inter-prediction 1523 can also be referred to as intra-layer prediction. As such, inter-prediction 1523 is a mechanism of coding a sample of a current picture by referring to an indicated sample in a reference picture that is different from the current picture, where the reference picture and the current picture are in the same layer.

[0285] Pictures 1511-1518 can also be coded by referring to other pictures 1511-1518 in different layers. This process is referred to as inter-layer prediction 1521 and is represented by dashed arrows. Inter-layer prediction 1521 is a mechanism of coding a sample of a current picture by referring to an indicated sample in a reference picture, where the current picture and the reference picture are in different layers and thus have different layer IDs. For example, a picture in a lower layer N 1531 can be used as a reference picture to code a corresponding picture in a higher layer N+1 1532. As a specific example, picture 1511 can be coded according to inter-layer prediction 1521 with reference to picture 1515. In this case, picture 1515 is used as an inter-layer reference picture. An inter-layer reference picture is a reference picture used for inter-layer prediction 1521. In most cases, inter-layer prediction 1521 is constrained such that a current picture (e.g., picture 1511) can only use inter-layer reference pictures (e.g., picture 1515) that are included in the same AU 1506 and are in a lower layer. When multiple layers (e.g., more than two) are available, inter-layer prediction 1521 can code / decode a current picture based on multiple inter-layer reference pictures that are lower in level than the current picture.

[0286] The video encoder can encode the pictures 1511-1518 using layer-based prediction 1500, through many different combinations and / or permutations of inter-prediction 1523 and inter-layer prediction 1521. For example, picture 1515 can be coded according to intra-prediction. Pictures 1516-1518 can then be coded according to inter-prediction 1523 by using picture 1515 as a reference picture. In addition, picture 1511 can be coded according to inter-layer prediction 1521 by using picture 1515 as an inter-layer reference picture. Pictures 1512-1514 can then be coded according to inter-prediction 1523 by using picture 1511 as a reference picture. In this way, reference pictures can be used as single-layer reference pictures and inter-layer reference pictures for different coding mechanisms. By coding a higher layer N+1 1532 picture based on a lower layer N 1531 picture, the higher layer N+1 1532 can avoid employing intra-prediction, which has much lower coding efficiency than inter-prediction 1523 and inter-layer prediction 1521. In this way, the low coding efficiency of intra-prediction can be limited to the minimum / lowest quality pictures and, thus, to the least amount of video data coded. Pictures used as reference pictures and / or inter-layer reference pictures can be indicated in entries of a reference picture list(s) included in a reference picture list structure.

[0287] Figure 15 Each of the AUs 1506 can contain several pictures. For example, one AU 1506 can contain pictures 1511 and 1515. Another AU 1506 can contain pictures 1512 and 1516. In fact, each AU 1506 is a set of one or more coded pictures (e.g., for display to a user) associated with the same display time (e.g., same temporal ID) for output from a decoded picture buffer (DPB). Each access unit delimiter (AUD) 1508 is an indicator or data structure used to indicate the start of an AU (e.g., AU 1506) or a boundary between AUs.

[0288] The previous H.26x video coding family has provided support for scalability in profiles that are independent of single-layer coding. Scalable Video Coding (SVC) is a scalable extension of AVC / H.264 that supports spatial, temporal, and quality scalability. For SVC, a flag is signaled in each macroblock (MB) in an enhancement layer (EL) picture to indicate whether the EL MB is predicted using a colocated block from a lower layer. The prediction from the colocated block can include texture, motion vector, and / or coding mode. Implementations of SVC cannot directly reuse an unmodified H.264 / AVC implementation in their design. The SVC EL macroblock syntax and decoding process are different from the H.264 / AVC syntax and decoding process.

[0289] Scalable HEVC (SHVC) is an extension of the HEVC / H.265 standard that supports spatial and quality scalability, Multiview HEVC (MV-HEVC) is an extension of HEVC / H.265 that supports multiview scalability, 3D HEVC (3D-HEVC) is an extension of HEVC / H.264 that supports more advanced and efficient three-dimensional (3D) video coding than MV-HEVC. Note that temporal scalability is included as a component of the single-layer HEVC codec. The design of the multi-layer extension of HEVC adopts the idea that the decoded pictures used for inter-layer prediction come only from the same AU and are treated as long-term reference pictures (LTRPs) and are assigned reference indices in the reference picture lists together with other temporal reference pictures in the current layer. Inter-layer prediction (ILP) is implemented at the prediction unit (PU) level by setting the value of the reference index to refer to the inter-layer reference picture in the reference picture list.

[0290] It is worth noting that both the reference picture resampling and the spatial scalability features require resampling of a reference picture or a portion thereof. Reference picture resampling (RPR) can be implemented at the picture level or at the coding block level. However, when RPR is referred to as a coding feature, it is a feature for single-layer coding. Even so, from the codec design perspective, it is possible or even preferred to use the same resampling filter for the RPR feature for single-layer coding and the spatial scalability feature for multi-layer coding.

[0291] Copy or repeated padding can be used to extend a picture to a larger size. More specifically, a reference picture (e.g., the reference frame 1330 in FIG. 13A, or the previous reference frame 1420 or the subsequent reference frame 1430 in FIG. 13B) is extended to form a larger picture. For example, the boundary samples located at the left side boundary of the reference picture are copied to the left side of the reference picture, the boundary samples located at the right side boundary of the reference picture are copied to the right side of the reference picture, the boundary samples located at the top boundary of the reference picture are copied to the top of the reference picture, and the boundary samples located at the bottom boundary of the reference picture are copied to the bottom of the reference picture. These copied boundary samples located outside the reference picture are referred to as padded samples (also referred to as padding samples). Figure 13 Figure 14 For current picture coding, when the motion vector (e.g., the MV 1335) of a current block (e.g., the current block 1311) points to a reference block (e.g., the reference block 1331) that is (partially or completely) located outside the reference picture (e.g., the reference frame 1330), the prediction block of the current block is generated from the padded samples outside the reference picture boundary.

[0292] For current picture coding, when the motion vector (e.g., the MV 1335) of a current block (e.g., the current block 1311) points to a reference block (e.g., the reference block 1331) that is (partially or completely) located outside the reference picture (e.g., the reference frame 1330), the prediction block of the current block is generated from the padded samples outside the reference picture boundary.

[0293] ​Motion compensated padding is discussed in Y. W. Chen et al., “Description of SDR HDR and 360 video coding technology proposal by Qualcomm and Technicolor - low and high complexity versions,” JVET document JVET-J0021, 2018. When a decoder performs motion compensation, if a motion vector points to a block outside the reference frame boundary, a portion of the reference block is not available. To address this issue, padded samples can be used to extend or upscale the reference picture / frame. For each region of size 4xM or Mx4 along the boundary of the reference picture to be padded, M is the desired frame boundary extension, the motion vector is derived from the nearest 4x4 block intra. If the nearest 4x4 block is intra coded, a zero motion vector is used. If the nearest 4x4 block is coded with bi-directional inter prediction, only the motion vector pointing away from the frame boundary is used for the motion compensation of padding. After the motion vector derivation, the motion compensation is then performed to obtain the pixels in the padded region, while taking into account the average pixel value offset between the nearest 4x4 block and its corresponding block in the reference picture.

[0294] A discussion of padding dimensions is provided.

[0295] Example 1

[0296] 1. Figure 16 A padded video unit 1600 is shown, where d1, d2, d3, d4 are padding dimensions for the top, bottom, left, and right boundaries 1604 of the padded video unit 1600, respectively. The padding dimensions collectively form a padding region 1602 around the padded video unit 1600. As described above, the padding region 1602 includes padding samples used to predict samples within the padded video unit 1600. Samples within the padded video unit 1600 that abut one or more boundaries 1604 of the padded video unit 1600 can be referred to as boundary samples or neighboring samples.

[0297] In one embodiment, the padding size related parameters (e.g., dl, d2, d3, d4) are determined in real-time (also referred to as instantaneously). Real-time (or real-time) describes various operations in a computation or other process that guarantee a response time within a specified time (deadline), typically a relatively short time. A real-time process is typically a process that occurs within a time step of a defined maximum duration and is fast enough to affect the environment in which the process occurs, such as input to a computing system. In one embodiment, real-time means deciding and / or implementing dimensions in a predictive process.

[0298] a. In one example, dl, d2, d3, d4 are different.

[0299] b. In one example, dl, d2, d3, d4 are the same.

[0300] c. In one example, dl = d2, d3 = d4.

[0301] d. In one example, an indication of the padding size is signaled in the bitstream. In one embodiment, the indication is a hint, an initial parameter, or other value used to derive the padding size. The padding size refers to the size of the padding region 1602 and / or one or more padding dimensions.

[0302] e. In one example, at least one padding size related parameter is determined according to the position of the video unit relative to the parent video unit.

[0303] i. In one example, at least one of dl, d2, d3, d4 is different for video units located at the parent video unit boundary, while dl, d2, d3, d4 are the same for interior video units.

[0304] f. In one example, the padding size depends on the video unit size.

[0305] i. In one example, the padding size is the same for video units of different block sizes.

[0306] ii. In one example, a larger padding size is used for larger video units.

[0307] ii. In one example, the ratio of the padding size to the video unit size is a pre-defined constant.

[0308] g. In one example, the padding size depends on the color format (e.g., 4:4:4 or 4:2:0) and / or color channel type.

[0309] i. In one example, the padding size is the same for different channel types.

[0310] ii. In one example, the padding size is set larger for luma components and smaller for chroma components.

[0311] h. In one example, the padding size depends on the slice type and / or the split tree type (single tree or dual tree). A split tree is a structure that intuitively represents a division of a set into subsets such that each element in the set is in exactly one of the subsets. In a single tree, the split tree is shared by luma and chroma components. In a dual tree, there can be one split tree for luma components and another for chroma components.

[0312] i. In one example, the padding size is the same for different types of slices.

[0313] ii. In one example, the padding size is set larger for intra slices and smaller for inter slices.

[0314] i. In one example, the padding size depends on the temporal layer and / or the reference picture information.

[0315] i. In one example, the padding size is the same for different temporal layers.

[0316] ii. In one example, the padding size is set larger for low temporal layers (e.g., layers 1531 in FIG. 15) and smaller for high temporal layers (e.g., layers 1532 in FIG. 15). Figure 15 Figure 15

[0317] j. In one example, the padding size of a first video unit depends on the kind of NN filtering method applied to the first video unit.

[0318] k. In one example, the padding size depends on other decoding information (e.g., properties of a video unit, such as the percentage of intra coded samples).

[0319] l. In one example, the padding size depends on whether a NN filter is applied or not.

[0320] m. In one example, the padding dimensions d1, d2, d3, d4 are set to 8 for internal boundaries and to 0 for boundaries located at the boundary of the parent video unit, respectively. In one embodiment, the internal boundaries are the boundaries of the child video units within the parent video unit and are created by splitting the parent video unit.

[0321] ​​n.In one example, a video unit is set to a block (e.g., a coding block) whose size depends on the slice resolution and quantization parameter (QP). Resolution can refer to spatial resolution (e.g., size of display), temporal resolution (e.g., frame rate), or video quality (e.g., number of pixels displayed - 1920x1080, etc.).

[0322] o.In one example, a parent video unit (also referred to as a parent video unit) is set to a slice.

[0323] A discussion of padding methods is provided.

[0324] Example 2

[0325] 2.The padding method (e.g., how to generate sample values outside the video unit and / or whether to apply padding) can be decided in real-time (also referred to as on-the-fly).

[0326] a.In one example, whether to apply padding can depend on whether at least one or all of the samples outside the video unit are available.

[0327] i.In one example, when all samples in the padded region are available for all boundaries, e.g., top boundary, bottom boundary, left boundary, and right boundary, the samples are used directly without padding.

[0328] ii.In one example, when at least one sample in the padded region is not available for all boundaries, all neighboring samples are padded.

[0329] b.In one example, whether to apply padding can depend on whether at least one or all of the samples outside the video unit along a given direction (e.g., top / bottom / left / right) are available.

[0330] i.In one example, if all neighboring samples are available for a particular boundary (e.g., top boundary, bottom boundary, left boundary, or right boundary), the neighboring samples are used directly without padding.

[0331] ii.In one example, when at least one sample in the padded region is not available for a particular boundary, all neighboring samples for that boundary are padded.

[0332] c.In one example, available samples in the padded region are used directly without padding, and unavailable samples in the padded region can be padded.

[0333] d.In one example, the padding method can be one of zero padding, reflective padding, copy padding, constant padding, and mirror padding. In one embodiment, zero padding occurs when a border of all zero-valued samples (or pixels) is added around the edges of the input picture. This adds zero padding outside the picture, hence the name. In one embodiment, during reflective padding, values are filled with the "reflection" or "mirror" of the values directly opposite the edge of the "to be padded" shape (e.g., video unit). In one embodiment, during copy padding, when the selected reference block is outside or straddles the frame boundary of the reference frame, the frame boundary pixels are repeated. In one embodiment, the initial picture for reconstruction is padded with a uniform intensity border, which is referred to as constant padding. Figure 17 Mirror padding 1700 is shown, where the gray blocks represent padded samples. The numbers within each block are provided to illustrate the appropriate pattern used to implement mirror padding.

[0334] i.In one example, to apply mirror padding, the values outside the boundary are obtained by mirroring the video unit across its boundary, as shown. Figure 17

[0335] e.The padding method can depend on the video unit size.

[0336] f.In one example, an indication of the padding method is signaled in the bitstream.

[0337] g.In one example, at least one padding method dependent parameter (e.g., padding dimension) is determined according to the position of the video unit relative to the parent video unit.

[0338] h.In one example, the padding method of a first video unit depends on the kind of NN filtering method applied to the first video unit.

[0339] i.In one example, the padding method depends on other decoding information (e.g., characteristics of a video unit, such as percentage of intra coded samples).

[0340] j.In one example, the padding method depends on whether a NN filter is applied.

[0341] k.The padding method can depend on the channel type.

[0342] l.The padding method can depend on the slice type.

[0343] m.The padding method can depend on the temporal layer.

[0344] Example 3

[0345] ​3. In one example, the samples in the padded region are the unfiltered version, i.e., before applying the NN filter.

[0346] Example 4

[0347] 4. In one example, the samples in the padded region are the filtered version, i.e., after applying the NN filter.

[0348] The interference block size is discussed.

[0349] Example 5

[0350] 5. The granularity of the NN filter model selection can be different from the CTU size.

[0351] a. The granularity can be predefined, or the indication of the granularity can be signaled in the bitstream or derived on the fly.

[0352] b. In one example, the granularity depends on the QP and the resolution.

[0353] i. In one example, the granularity is coarser when the QP is larger or the resolution is higher.

[0354] ii. In one example, let q denote the sequence level QP and w denote the frame width. When q < 23, the granularity is 32x32. When 23≤q<29, w≤832, the granularity is 32x32. When 23≤q<29, w>832, the granularity is 64x64. When q≥29 and w≤832, the granularity is 128x128. When q≥29 and w>832, the granularity is 256x256.

[0355] The binarization of the neural network filter model index is discussed.

[0356] Example 6

[0357] The binarization of the NN filter model index of a first video unit, such as a coding block or region, can depend on the maximum number of models allowed at a level higher than the first video unit. Binarization is the process of dividing data into two groups and assigning one of two values to all members of the same group. This is typically achieved by defining a threshold and assigning the value 0 to all data points below the threshold and the value 1 to data points above the threshold.

[0358] a. In one example, the higher level is the slice / picture / sequence level.

[0359] b. In one example, the indication of the maximum number of models is signaled at the higher level, or is predefined or derived on the fly.

[0360] i.In one example, the indication of the maximum number of models is signaled in a picture header / slice header / picture parameter set (PPS) / sequence parameter set (SPS) / adaptation parameter set (APS).

[0361] c.In one example, the NN filter model index can be binarized into a truncated code, such as a truncated unary code or a truncated binary code. Unary coding is an entropy encoding that represents natural numbers n with n ones followed by a zero. Truncated unary is a variation of unary coding. Truncated binary or truncated binary coding is an entropy encoding that is typically used for uniform probability distributions with a finite alphabet.

[0362] Figure 18 is a block diagram illustrating an example video processing system 1800 in which various techniques disclosed herein can be implemented. Various implementations can include some or all of the components of the video processing system 1800. The video processing system 1800 can include an input 1802 for receiving video content. The video content can be received in a raw or uncompressed format, such as 8 or 10 bit multi-component pixel values, or can be in a compressed or encoded format. The input 1802 can represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical networks (PONs), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0363] The video processing system 1800 can include a codec component 1804, which can implement various coding or encoding methods described in this document. The codec component 1804 can reduce the average bitrate of video from the input 1802 to the output of the codec component 1804 to produce a coded representation of the video. Thus, the coding techniques are sometimes referred to as video compression or video transcoding techniques. The output of the codec component 1804 can be stored, or transmitted via a connected communication, as represented by component 1806. Component 1808 can use the stored or transmitted bitstream (or coded) representation of the video received at the input 1802 to generate pixel values or displayable video sent to a display interface 1810. The process of generating user-viewable video from a bitstream representation is sometimes referred to as video decompression. Furthermore, while certain video processing operations are referred to as “coding” operations or tools, it should be understood that the encoding tools or operations are used at an encoder and that corresponding decoding tools or operations, which reverse the results of the encoding, will be performed by a decoder.

[0364] Examples of a peripheral bus interface or a display interface can include a universal serial bus (USB) or a high-definition multimedia interface (HDMI) or a display port, etc. Examples of a storage interface can include a SATA (serial advanced technology attachment), a peripheral component interconnect (PCI), an integrated drive electronics (IDE) interface, etc. The techniques described in this document can be implemented in various electronic devices such as, for example, a mobile phone, a laptop, a smart phone, or another device capable of performing digital data processing and / or video display.

[0365] Figure 19 is a block diagram of a video processing apparatus 1900. The video processing apparatus 1900 can be used to implement one or more methods described herein. The video processing apparatus 1900 can be embodied in a smart phone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. The video processing apparatus 1900 can include one or more processors 1902, one or more memories 1904, and video processing hardware 1906 (also referred to as video processing circuitry). The processor(s) 1902 can be configured to implement one or more methods described in this document. The memory(ies) 1904 can be used for storing data and code used for implementing the methods and techniques described herein. The video processing hardware 1906 can be used to implement, in hardware circuitry, some of the techniques described in this document. In some embodiments, the video processing hardware 1906 can be partially or entirely within the processor 1902, such as a graphics processor.

[0366] Figure 20 is a block diagram illustrating an example of a video coding system 2000 that can utilize the techniques of this disclosure. As shown in Figure 20 The video coding system 2000 can include a source device 2010 and a destination device 2020. The source device 2010 generates encoded video data, which can be referred to as a video encoding device. The destination device 2020 can decode the encoded video data generated by the source device 2010, which can be referred to as a video decoding device.

[0367] The source device 2010 can include a video source 2012, a video encoder 2014, and an input / output (I / O) interface 2016.

[0368] Video source 2012 can include a source such as a video capture device, an interface to receive video data from a video content provider, and / or a computer graphics system for generating video data, or a combination thereof. Video data can comprise one or more pictures. Video encoder 2014 encodes video data from video source 2012 to generate a bitstream. The bitstream can include a sequence of bits that form a coded representation of the video data. The bitstream can include coded pictures and associated data. A coded picture is a coded representation of a picture. Associated data can include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 2016 can include a modulator / demodulator (modem) and / or a transmitter. Encoded video data can be transmitted directly to destination device 2020 via I / O interface 2016 by network 2030. Encoded video data can also be stored onto storage medium / server 2040 for access by destination device 2020.

[0369] Destination device 2020 can include I / O interface 2026, video decoder 2024, and display device 2022.

[0370] I / O interface 2026 can include a receiver and / or a modem. I / O interface 2026 can acquire encoded video data from source device 2010 or storage medium / server 2040. Video decoder 2024 can decode the encoded video data. Display device 2022 can display the decoded video data to a user. Display device 2022 can be integrated with destination device 2020 or can be external to destination device 2020. The destination device 2020 can be configured to interface with an external display device.

[0371] Video encoder 2014 and video decoder 2024 can operate according to a video compression standard, such as High Efficiency Video Coding (HEVC) standard, Versatile Video Coding (VVC) standard, and other current and / or further standards.

[0372] Figure 21 is a block diagram illustrating an example of a video encoder 2100, which can be Figure 20 video encoder 2014 in the video coding system 2000 described in FIG.

[0373] Video encoder 2100 can be configured to perform any or all of the techniques of this disclosure. In Figure 21 In examples, video encoder 2100 includes a number of functional components. The techniques described in this disclosure can be shared by the various components of video encoder 2100. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0374] The functional components of video encoder 2100 can include a partition unit 2101, a prediction unit 2102, a residual generation unit 2107, a transform unit 2108, a quantization unit 2109, an inverse quantization unit 2110, an inverse transform unit 2111, a reconstruction unit 2112, a buffer 2113, and an entropy encoding unit 2114. Prediction unit 2102 can include a mode selection unit 2103, a motion estimation unit 2104, a motion compensation unit 2105, and an intra-prediction unit 2106.

[0375] In other examples, video encoder 2100 can include more, less, or different functional components. In an example, prediction unit 2102 can include an intra-block copy (IBC) unit. The IBC unit can perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0376] Furthermore, some components, such as motion estimation unit 2104 and motion compensation unit 2105, can be highly integrated, but are represented separately for illustrative purposes. Figure 21 In an example, to perform inter-prediction, motion estimation unit 2104 can generate motion information for a current video block by comparing one or more reference frames from buffer 2113 to the current video block. Motion compensation unit 2105 can determine a predicted video block for the current video block based on the motion information and decoded samples from pictures other than the picture in which the current video block is associated, from buffer 2113.

[0377] Partition unit 2101 can partition a picture into one or more video blocks. Figure 20 Video encoder 2014 and video decoder 2024 can support various video block sizes.

[0378] Mode selection unit 2103 can select one of the coding modes (intra or inter), e.g., based on error results, and provide the resulting intra- or inter-coded block to residual generation unit 2107 for generation of residual block data, and to reconstruction unit 2112 for reconstruction of the encoded block for use as reference picture. In some examples, mode selection unit 2103 can select a combined intra- and inter-prediction (CIIP) mode in which the prediction is based on both an inter-prediction signal and an intra-prediction signal. In the case of inter-prediction, mode selection unit 2103 can also select a resolution for the motion vectors (e.g., sub-pixel or integer pixel precision) for the block.

[0379] To perform inter-prediction for a current video block, motion estimation unit 2104 can generate motion information for the current video block by comparing one or more reference frames from buffer 2113 to the current video block. Motion compensation unit 2105 can determine a predicted video block for the current video block based on the motion information and decoded samples from pictures other than the picture in which the current video block is associated, from buffer 2113.

[0380] Motion estimation unit 2104 and motion compensation unit 2105 can perform different operations on a current video block, e.g., depending on whether the current video block is an I slice, a P slice, or a B slice. I slices (or I frames) are the lowest in compression rate, but do not require other video frames for decoding. P slices (or P frames) can use data in previous frames for decompression, and are more easily compressed than I frames. B slices (or B frames) can use both previous and future frames as data references for the highest amount of data compression.

[0381] In some examples, motion estimation unit 2104 can perform uni-prediction on a current video block, and motion estimation unit 2104 can search for a reference video block for the current video block in reference pictures in list 0 or list 1. Motion estimation unit 2104 can then generate a reference index indicating a reference picture in list 0 or list 1 that contains the reference video block, and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 2104 can output the reference index, the prediction direction indicator, and the motion vector as motion information for the current video block. Motion compensation unit 2105 can generate a predicted video block for the current block based on the reference video block indicated by the motion information for the current video block.

[0382] In other examples, motion estimation unit 2104 can perform bi-prediction on a current video block, motion estimation unit 2104 can search for a reference video block for the current video block in reference pictures in list 0, and can also search for another reference video block for the current video block in reference pictures in list 1. Motion estimation unit 2104 can then generate a reference index indicating reference pictures in list 0 and list 1 that contain the reference video blocks, and a motion vector indicating a spatial displacement between the reference video blocks and the current video block. Motion estimation unit 2104 can output the reference index and the motion vector as motion information for the current video block. Motion compensation unit 2105 can generate a predicted video block for the current block based on the reference video blocks indicated by the motion information for the current video block.

[0383] In some examples, motion estimation unit 2104 can output a complete set of motion information for a decoder's decoding process.

[0384] In some examples, motion estimation unit 2104 can not output a complete set of motion information for a current video. Instead, motion estimation unit 2104 can signal motion information for a current video block with reference to motion information for another video block. For example, motion estimation unit 2104 can determine that the motion information for the current video block is sufficiently similar to the motion information for a neighboring video block.

[0385] In one example, the motion estimation unit 2104 can indicate, in a syntax structure associated with the current video block, a value that indicates to the video decoder 2024 that the current video block has the same motion information as another video block.

[0386] In another example, the motion estimation unit 2104 can identify, in a syntax structure associated with the current video block, another video block and a motion vector difference (MVD). The motion vector difference indicates a difference between a motion vector of the current video block and a motion vector of the indicated video block. The video decoder 2024 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0387] As described above, the video encoder 2014 can predictively signal motion vectors. Two examples of prediction signaling techniques that can be implemented by the video encoder 2014 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0388] The intra prediction unit 2106 can perform intra prediction on the current video block. When the intra prediction unit 2106 performs intra prediction on the current video block, the intra prediction unit 2106 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include a predicted video block and various syntax elements.

[0389] The residual generation unit 2107 can generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block for the current video block from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of samples in the current video block.

[0390] In other examples, there can be no residual data for the current video block, such as in skip mode, and the residual generation unit 2107 can not perform the subtraction operation.

[0391] The transform unit 2108 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0392] After the transform unit 2108 generates the transform coefficient video blocks associated with the current video block, the quantization unit 2109 can quantize the transform coefficient video blocks associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0393] Inverse quantization unit 2110 and inverse transform unit 2111 can apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 2112 can add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by prediction unit 2102 to produce a reconstructed video block associated with the current block for storage in buffer 2113.

[0394] After reconstruction unit 2112 reconstructs the video block, in-loop filtering operations can be performed to reduce video block artifacts in the video block.

[0395] Entropy encoding unit 2114 can receive data from other functional components of video encoder 2100. When entropy encoding unit 2114 receives data, entropy encoding unit 2114 can perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream that includes the entropy encoded data.

[0396] Figure 22 is a block diagram illustrating an example of a video decoder 2200 that can be Figure 20 the video coding system 2000 described in FIG. 1.

[0397] Video decoder 2200 can be configured to perform any or all of the techniques of this disclosure. In Figure 22 In the example of FIG. 1, video decoder 2200 includes a number of functional components. The techniques described in this disclosure can be shared among the various components of video decoder 2200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0398] In Figure 22 In the example of FIG. 1, video decoder 2200 includes an entropy decoding unit 2201, a motion compensation unit 2202, an intra prediction unit 2203, an inverse quantization unit 2204, an inverse transform unit 2205, a reconstruction unit 2206, and a buffer 2207. In some examples, video decoder 2200 can perform a decoding process generally reciprocal to the encoding process described with respect to video encoder 2014 Figure 20 ) described with respect to video encoder 2014.

[0399] Entropy decoding unit 2201 can retrieve an encoded bitstream. The encoded bitstream can include entropy encoded video data (e.g., encoded video data blocks). Entropy decoding unit 2201 can decode the entropy encoded video data, and, from the entropy decoded video data, motion compensation unit 2202 can determine motion information, including motion vectors, motion vector precision, reference picture list indices, and other motion information. Motion compensation unit 2202 can determine such information, for example, by performing AMVP and merge mode signaling.

[0400] Motion compensation unit 2202 can generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of the interpolation filter to be used with sub-pixel precision can be included in the syntax elements.

[0401] Motion compensation unit 2202 can use the interpolation filter as used by video encoder 2014 during encoding of the video block to calculate the interpolation of sub-integer pixels of the reference block. Motion compensation unit 2202 can determine the interpolation filter used by video encoder 2014 from the received syntax information and use the interpolation filter to generate the prediction block.

[0402] Motion compensation unit 2202 can use some of the syntax information to determine the size of the blocks used to encode the frame(s) and / or slice(s) of the coded video sequence, partitioning information describing how each macroblock of a picture of the coded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information to decode the coded video sequence.

[0403] Intra prediction unit 2203 can use, for example, intra prediction modes received in the bitstream to form the prediction block from spatially neighboring blocks. Inverse quantization unit 2204 inverse quantizes, i.e., de-quantizes, quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 2201. Inverse transform unit 2205 applies an inverse transform.

[0404] Reconstruction unit 2206 can add the residual block to the corresponding prediction block generated by motion compensation unit 2202 or intra prediction unit 2203 to form a decoded block. If desired, a deblocking filter can also be applied to the decoded block to filter out blockiness artifacts. The decoded video blocks are then stored in buffer 2207, which provides reference blocks for subsequent motion compensation / intra prediction, and also generates decoded video for presentation on a display device.

[0405] Figure 23 is a method 2300 for coding video data according to an embodiment of the present disclosure. The method 2300 can be performed by a coding device (e.g., an encoder) having a processor and a memory. The method 2300 can be implemented when determining padding dimensions to be used for padding samples to be applied to a video unit during in-loop filtering is needed.

[0406] In block 2302, the coding device determines, in real-time, padding dimensions to be used for padding samples applied to a video unit of a video for in-loop filtering. In one embodiment, dl, d2, d3, and d4 represent padding dimensions corresponding to the top, bottom, left, and right boundaries of the video unit, respectively.

[0407] In block 2304, the codec device performs a conversion between a video unit and a video bitstream based on the generated filtered samples. When implemented in an encoder, the conversion includes receiving a media file (e.g., a video unit) and encoding the media file into a bitstream. When implemented in a decoder, the conversion includes receiving a bitstream that includes a media file, and decoding the bitstream to obtain the media file.

[0408] In one embodiment, the method 2300 can utilize or incorporate one or more features or processes of other methods disclosed herein.

[0409] Next, a list of some embodiments preferred solutions is provided.

[0410] The following solution shows an example embodiment (e.g., Example 1) of the techniques discussed in this disclosure.

[0411] 1. A method of video processing, comprising: for a conversion between a video block of a video and a bitstream of the video, determining padding dimensions around the video block to apply in-loop filtering to the video block according to a rule; and performing the conversion based on the determining, wherein sizes defined by dl, d2, d3, and d4 are padding sizes for top, bottom, left, and right boundaries, respectively.

[0412] 2. The method of item 1, wherein the rule specifies that dl, d2, d3, d4 are different from each other.

[0413] 3. A method of video processing, comprising: for a conversion between a video block of a video and a bitstream of the video, determining whether or how to generate padding samples outside the video block to apply in-loop filtering to the video block according to a rule; and performing the conversion based on the determining.

[0414] 4. The method of item 3, wherein the rule specifies that whether to generate the padding samples depends on whether one or all of the samples outside the video unit are available.

[0415] 5. The method of item 3, wherein the rule specifies that whether to generate the padding samples depends on whether one or all of the samples outside the video unit along a given direction are available.

[0416] 6. The method of any of items 3-5, wherein the rule specifies that, before applying the in-loop filtering, available samples outside the video block are used without padding, and unavailable samples outside the video block are padded.

[0417] 7. The method of any of items 3-6, wherein the padding samples are samples before applying a neural network (NN) filter.

[0418] 8. The method of any of clauses 3-7, wherein the padded samples are samples after application of a neural network (NN) filter.

[0419] 9. A method of video processing, comprising: for a conversion between a video comprising video pictures and a bitstream representation of the pictures, determining a granularity of a neural network (NN) filter model used for in-loop filtering during the conversion based on a rule, and performing the conversion based on the determining, wherein the rule defines the granularity of the NN filter model independently of a size of a video block.

[0420] 10. The method of clause 9, wherein the rule defines the granularity is pre-defined.

[0421] 11. The method of any of clauses 9-10, wherein the rule defining the granularity depends on a quantization parameter (QP) or a frame resolution.

[0422] 12. A method of video processing, comprising: performing a conversion between a video comprising a video region containing video units and a bitstream of the video, wherein the conversion includes applying an in-loop neural network (NN) filter to a first video unit according to a rule, wherein the rule specifies that an indexing of a number of models of the NN filter according to a maximum number of models is allowed at a level of the video region.

[0423] 13. The method of clause 12, wherein the video region is a slice or a picture or a sequence.

[0424] 14. The method of any of clauses 12-13, wherein the maximum number of models allowed at the level of the video region is pre-determined or determined during the conversion.

[0425] 15. The method of any of clauses 12-13, wherein the indexing is binarized using a truncated code.

[0426] 16. The method of any of clauses 1-15, wherein the conversion comprises generating the bitstream from the video.

[0427] 17. The method of any of clauses 1-15, wherein the conversion comprises generating the video from the bitstream.

[0428] 18. A video decoding apparatus comprising a processor configured to implement a method recited in one or more of clauses 1 to 17.

[0429] 19. A video encoding apparatus comprising a processor configured to implement a method recited in one or more of clauses 1 to 17.

[0430] 20. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to implement the method of any one of items 1 to 17.

[0431] 21. A computer readable medium having a bitstream stored thereon, the bitstream being generated by the method of any one of items 1 to 17.

[0432] 22. A method of generating a bitstream, comprising: generating a bitstream using one or more of items 1 to 17, and writing the bitstream to a computer readable medium.

[0433] 23. The method, apparatus or system described in this document.

[0434] The following documents are incorporated by reference in their entirety:

[0435] [1] Johannes Ballé, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” in PCS IEEE (2016), 1-5.

[0436] [2] Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszar, “Lossy image compression with compressive autoencoders,” arXiv preprint arXiv:1703.00395 (2017).

[0437] [3] Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao, “Fully Connected Network-Based Intra Prediction for Image Coding,” IEEE Transactions on Image Processing 27, 7 (2018), 3236-3247.

[0438] [4] Yuanying Dai, Dong Liu, and Feng Wu, “A convolutional neural network approach for post-processing in HEVC intra coding”, MMM. Springer, 28-39.

[0439] [5] Rui Song, Dong Liu, Houqiang Li, and Feng Wu, “Neural network-based arithmetic coding of intra prediction modes in HEVC”, VCIP IEEE (2017), 1-4.

[0440] [6] J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, W. Samek, H. Schwarz, D. Marpe, and T. Wiegand, “Neural network based intra prediction for video coding”, Applications of Digital Image Processing XLI, Vol. 10752. International Society for Optics and Photonics, 1075213 (2018).

[0441] The disclosed and other solutions, examples, embodiments, modules and functional operations described herein can be realized in digital electronic circuitry, or in computer software, firmware, or hardware, including the structural equivalents of such software, firmware, or hardware, or in combinations of one or more of them. The disclosed and other embodiments can be realized as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.

[0442] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and

[0443] The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0444] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and compact disc read-only memory (CD-ROM) and digital versatile disc read-only memory (DVD-ROM) disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0445] Although this patent document contains many details, these should not be construed as limiting the scope of any subject matter or of any embodiment of the claimed subject matter but as merely describing features that are particular to certain embodiments of the particular technology. Certain features described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be deleted from the combination, and the claimed combination can also be directed to a sub-combination or a variation of a sub-combination.

[0446] Similarly, while operations are described in a particular, sequential order, this should not be understood as requiring that the operations be performed in that order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0447] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method of processing video data, comprising: determining in real time padding dimensions for padding samples to be applied to a video unit of a video for in-loop filtering, wherein 、 、 and respectively denote the padding dimensions corresponding to the top, bottom, left and right boundaries of the video unit; and performing a conversion between the video unit and a bitstream of the video based on the determined padding dimensions, wherein a padding method used to generate the padding samples outside the video unit is based on a channel type, a slice type, or which temporal layer the video unit belongs to, and wherein a neural network (NN) filter is applied to the padding samples.

2. The method of claim 1, wherein, , , and each have different values; or wherein , , and each have the same value; or wherein = = ​​ 3. The method of claim 1, wherein, One or more indications corresponding to the padding dimensions are included in the bitstream.

4. The method of claim 1, wherein, At least one of the padding dimensions is based on a position of the video unit relative to a parent video unit from which the video unit is partitioned, and wherein the padding dimension corresponding to an outer boundary of the parent video unit is different from the padding dimension corresponding to an inner boundary of the parent video unit.

5. The method of claim 1, wherein, At least one of the padding dimensions is based on a size, a color format, a channel type, a slice type, or a partition tree type of the video unit.

6. The method of claim 1, wherein, At least one of the padding dimensions is based on which temporal layer the video unit belongs to, or based on reference picture information.

7. The method of claim 1, wherein, At least one of the padding dimensions is based on a type of a neural network (NN) in-loop filtering method applied to the video unit or whether the NN in-loop filtering method is applied to the video unit.

8. The method of claim 1, wherein, The video unit has a video unit size that depends on at least one of a slice resolution and a quantization parameter (QP), and wherein the parent video unit is a slice.

9. The method of claim 1, wherein, The padding dimensions define a padding region around the video unit, and wherein the method further comprises padding the padding region with the padding samples when at least one neighboring sample outside the video unit is not available.

10. The method of claim 1, further comprising padding a boundary of the video unit with the padding samples when neighboring samples outside the video unit and along the boundary are not available.

11. The method of claim 1, wherein, The padding dimensions define a padding region around the video unit, and wherein the method further comprises padding the padding region using a padding method, the padding method comprising zero padding, reflection padding, copy padding, constant padding, or mirror padding.

12. The method of claim 11, wherein, The padding method used to pad the padding region is based on a size of the video unit.

13. The method of claim 1, wherein, The padding method used to generate the padding samples outside the video unit is based on a type of a neural network (NN) filtering method applied to the video unit or whether the NN filtering method is applied to the video unit.

14. The method of claim 1, wherein, A granularity of a neural network (NN) filter model to be applied to the padding samples is predefined, included in a bitstream, derived, or based on dimensions of the video unit.

15. The method of claim 1, wherein, A granularity of a neural network (NN) filter model to be applied to the padding samples is based on a quantization parameter (QP) and a resolution of the video unit.

16. The method of claim 1, wherein, A binarization of a neural network (NN) filter model index corresponding to a NN filter model to be applied to the padding samples is based on a maximum number allowed for a higher level than the video unit, wherein the higher level is a slice, picture or sequence level, and wherein the NN filter model index is binarized into a truncated unary code or a truncated binary code.

17. The method of claim 1, wherein, The conversion includes encoding the video unit into the bitstream.

18. The method of claim 1, wherein, The conversion includes decoding the video unit from the bitstream.

19. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: determining in real time padding dimensions for padding samples to be applied to a video unit of a video for in-loop filtering, wherein 、 、 and respectively denote padding dimensions corresponding to top, bottom, left and right boundaries of said video unit; and perform a conversion between the video unit and a bitstream of the video based on the determined padding dimension, wherein a padding method for generating the padding samples outside the video unit is based on a channel type, a slice type or which temporal layer the video unit belongs to, and wherein a neural network (NN) filter is applied to the padding samples.

20. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing apparatus, wherein the method comprises: determining in real time padding dimensions for padding samples to be applied to a video unit of the video for in-loop filtering, wherein 、 、 and respectively denote the padding dimensions corresponding to the top, bottom, left and right boundaries of the video unit; and generating the bitstream based on the determined padding dimension, wherein a padding method for generating the padding samples outside the video unit is based on a channel type, a slice type or which temporal layer the video unit belongs to, and wherein a neural network (NN) filter is applied to the padding samples.

Images