Configurable neural network model depth in neural network-based video coding

Abstract

A method implemented by a video coding device for configurable neural network model depth in neural network-based video coding. The method includes applying a neural network (NN) filter to unfiltered samples of a video unit to produce filtered samples, wherein the NN filter is based on a first NN filter model having a first depth or a second NN filter model having a second depth, wherein the depth comprises a number of residual blocks of the respective NN filter model, and wherein the second depth is different than the first depth. The method also includes performing a conversion between a video media file and a bitstream based on the filtered samples.

Description

[0001] Cross-references to related applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 219,328, filed July 7, 2021, entitled “External Attenuation in a Neural Network-Based Codec Tool for Video Codecs,” which is incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the encoding and decoding of images and videos. Background Technology

[0004] Digital video consumes the largest share of bandwidth on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] The disclosed aspects / embodiments provide one or more neural network (NN) filter models trained as encoding / decoding tools to improve the efficiency of video encoding / decoding. NN-based encoding / decoding tools can be used to replace or enhance one or more modules implemented by a video encoder / decoder (also called a codec). For example, NN models can be trained to provide additional intra-frame prediction modes, additional inter-frame prediction modes, transform kernels, and / or loop filters. Furthermore, NN models can be generated or designed by using external information such as prediction, partitioning, and quantization parameters (QP) as attention mechanisms(s).

[0006] The first aspect relates to a method implemented by an encoding / decoding device. The method includes: applying a neural network (NN) filter to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is based on a first NN filter model having a first depth or a second NN filter model having a second depth, wherein the depth includes the number of residual blocks of the respective NN filter model, and wherein the second depth is different from the first depth. The method further includes performing a conversion between a video media file and a bitstream based on the filtered samples.

[0007] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides that the video unit is a first video unit, wherein when the NN filter is applied to the first video unit, the NN filter uses the first NN filter model, and wherein when the NN filter is applied to a second video unit, the NN filter uses the second NN filter model.

[0008] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that determining whether to use the first NN filter model or the second NN filter model is based on a temporal layer of the video unit, a type of a slice or picture containing the video unit, a reference picture list matching the video unit, a color component of the video unit, a color format of the video unit, and a combination thereof.

[0009] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the NN filter uses the first NN filter model for a first video unit in a picture or slice, and wherein the NN filter uses the second NN filter model for a second video unit in the picture or slice.

[0010] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the NN filter uses the first NN filter model for a first video unit in a first picture or first slice, and wherein the NN filter uses the second NN filter model for a second video unit in a second picture or second slice.

[0011] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the NN filter uses the first NN filter model for a first video unit in a first temporal layer or a first type of slice, and wherein the NN filter uses the second NN filter model for a second video unit in a second temporal layer or a second type of slice.

[0012] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the NN filter uses the first NN filter model for a third video unit in a third temporal layer, wherein the first temporal layer and the third temporal layer are in a same subgroup of temporal layers, and wherein the second temporal layer is in a different subgroup of temporal layers.

[0013] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that determining whether to use the first NN filter model or the second NN filter model is based on whether the video unit is a coding tree unit (CTU), a coding tree block (CTB), a CTU row, a CTB row, a slice, a tile, a picture, a sequence, or a subpicture.

[0014] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the NN filter uses the first NN filter model for a first video unit in a first temporal layer, wherein the NN filter uses the second NN filter model for a second video unit in a second temporal layer, and wherein the first temporal layer is a higher temporal layer than the second temporal layer.

[0015] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the first depth is 8.

[0016] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the NN filter is one or more selected from the group consisting of an adaptive loop filter, a deblocking filter, and a sample adaptive offset filter.

[0017] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the first depth is 8, and wherein the second depth is 32.

[0018] Optionally, in any of the preceding aspects, another embodiment of the aspect provides signaling to a decoder of a depth of a NN filter model to be used.

[0019] Optionally, in any of the preceding aspects, another embodiment of the aspect further comprises signaling to a decoder of a depth of a NN filter model to be used using one selected from the group consisting of a supplemental enhancement information (SEI) message, a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a video parameter set (VPS), a picture header, and a slice header.

[0020] Optionally, in any of the preceding aspects, another embodiment of the aspect further comprises sending a depth of the first NN filter model to a decoder of a first video unit, and sending a depth of the second NN filter model to a decoder of a second video unit.

[0021] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the unfiltered samples include a luma component and a chroma component, and wherein the method further comprises signaling to a decoder different depths of the luma component and the chroma component.

[0022] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the NN filter is one or more selected from the group consisting of an adaptive loop filter, a deblocking filter, and a sample adaptive offset filter.

[0023] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the converting comprises generating the bitstream from the video media file.

[0024] Optionally, in any of the preceding aspects, another embodiment of the aspect provides that the converting comprises parsing the bitstream to obtain the video media file.

[0025] A second aspect relates to an apparatus for coding video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: apply a neural network, NN, filter to unfiltered samples of a video unit to produce filtered samples, wherein the NN filter is based on a first NN filter model having a first depth or a second NN filter model having a second depth, wherein the depth comprises a number of residual blocks of the respective NN filter model, and wherein the second depth is different from the first depth; and perform a conversion between a video media file and a bitstream based on the filtered samples.

[0026] A third aspect relates to a non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing apparatus, the method performed by the video processing apparatus comprising applying a neural network, NN, filter to unfiltered samples of a video unit to produce filtered samples, wherein the NN filter is based on a first NN filter model having a first depth or a second NN filter model having a second depth, wherein the depth comprises a number of residual blocks of the respective NN filter model, and wherein the second depth is different from the first depth; and generating the bitstream based on the filtered samples.

[0027] A fourth aspect relates to a method of storing a video bitstream performed by a video processing apparatus. The method performed by the video processing apparatus comprises applying a neural network, NN, filter to unfiltered samples of a video unit to produce filtered samples, wherein the NN filter is based on a first NN filter model having a first depth or a second NN filter model having a second depth, wherein the depth comprises a number of residual blocks of the respective NN filter model, and wherein the second depth is different from the first depth; and generating the bitstream based on the filtered samples.

[0028] For the sake of clarity, any one of the preceding embodiments can be combined with any one or more of the other preceding embodiments to form new embodiments within the scope of the present application.

[0029] These and other features will become more apparent from the following detailed description in which reference is made to the appended drawings and claims. BRIEF DESCRIPTION OF DRAWINGS

[0030] For a more complete understanding of the present disclosure, reference is now made to the following brief description of the drawings and detailed description in conjunction with the accompanying figures and claims.

[0031] Figure 1 is an example of a raster scan slice partition of a picture.

[0032] Figure 2 is an example of a rectangular slice partition of a picture.

[0033] Figure 3 is an example of picture partitioning into slices, tiles, and rectangular slices.

[0034] Figure 4A is an example of a coding tree block (CTB) that crosses a bottom picture boundary.

[0035] Figure 4B is an example of a CTB that crosses a right picture boundary.

[0036] Figure 4C is an example of a CTB that crosses a bottom-right picture boundary.

[0037] Figure 5 is an example of an encoder block diagram.

[0038] Figure 6 is an illustration of samples within an 8x8 sample block.

[0039] Figure 7 is an example of pixels involved in filter on / off decision and strong / weak filter selection.

[0040] Figure 8 shows four one-dimensional (1-D) directional schemes for edge offset (EO) sample classification.

[0041] Figure 9 shows an example of a geometric transform based adaptive loop filter (GALF) filter shape.

[0042] Figure 10 shows an example of relative coordinates for 5x5 diamond filter support.

[0043] Figure 11 shows another example of relative coordinates for 5x5 diamond filter support.

[0044] Figure 12A is an example architecture of the proposed CNN filter.

[0045] Figure 12B is an example of a residual block (ResBlock) construction.

[0046] Figure 13 is an example of a process of generating filtered samples based on a neural network filter model receiving coding parameters as input, according to various examples.

[0047] Figure 14 is an example of applying attention obtained using external information (e.g., coding parameters) to a feature map of a neural network filter model to provide a re-calibrated feature map, according to various examples.

[0048] Figure 15 is an example of a process of applying a neural network filter having a configurable depth to unfiltered samples of a video unit to generate filtered samples.

[0049] Figure 16A is a schematic block diagram of an architecture of a neural network filtering method according to various examples.

[0050] Figure 16B is a schematic block diagram of a neural network filter according to various examples. Figure 16A is a schematic block diagram of an attention residual block used in the architecture of

[0051] Figure 17 is a block diagram showing an example video processing system.

[0052] Figure 18 is a block diagram of a video processing device.

[0053] Figure 19 is a block diagram showing an example video coding system.

[0054] Figure 20 is a block diagram showing an example of a video encoder.

[0055] Figure 21 is a block diagram showing an example of a video decoder.

[0056] Figure 22 is a video data coding method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary design and implementations set forth herein, but can be modified in various ways within the scope of the appended claims and their equivalents.

[0058] The use of Versatile Video Coding (VVC), also known as H.266 terminology in certain descriptions is for ease of understanding only and is not intended to limit the scope of the disclosed technology. Accordingly, the technology described herein is applicable to other video codec protocols and designs as well.

[0059] This specification relates to video coding. In particular, this specification relates to loop filters in image / video coding. The disclosed examples can be applied individually or in various combinations for video bitstreams coded using existing video coding standards, such as the VVC standard, the High Efficiency Video Coding (HEVC) standard, or a yet-to-be-finalized standard (e.g., the third generation of Audio Video Coding Standard (AVS3)). The disclosed examples can also be applicable to future video coding standards, future video codecs, or as a post-processing method outside of the encoding / decoding process.

[0060] Video coding standards have evolved primarily through the development of the well-known International Telecommunication Union-Telecommunication (ITU-T) and International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) standards. The ITU-T developed H.261 and H.263, ISO / IEC developed Motion Pictures Expert Group (MPEG)-1 and MPEG-4 Visual, and the two organizations jointly developed the H.262 / MPEG-2 Video, H.264 / MPEG-4 Advanced Video Coding (AVC), and H.265 / High Efficiency Video Coding (HEVC) standards.

[0061] Since H.262, video coding standards are based on a hybrid video coding structure where temporal prediction plus transform coding is employed. To explore future video coding technologies beyond HEVC, the Video Coding Expert Group (VCEG) and MPEG jointly created the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to the reference software named Joint Exploration Model (JEM).

[0062] In April 2018, the Joint Video Expert Team (JVET) was created between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG), which is committed to researching the VVC standard, aiming for a 50% bit rate reduction compared to HEVC. VVC version 1 was completed in July 2020.

[0063] Color spaces and chroma subsampling are discussed. A color space, also called a color model (or color system), is an abstract mathematical model that simply describes a range of colors as a tuple of numerical values, usually 3 or 4 values or color components (e.g., red-green-blue (RGB)). At a fundamental level, a color space is an exposition of a coordinate system and subspace.

[0064] For video compression, the most commonly used color spaces are YCbCr and RGB. YCbCr, Y'CbCr, or Y Pb / Cb Pr / Cr, also written as YC B C R or Y' B C R is a series of color spaces used as part of the color image pipeline in video and digital photography systems. Y' is the luminance component, and CB and CR are the blue-difference and red-difference chrominance components. Y' (with the prime) is different from Y, which is luminance, meaning that the light intensity is based on gamma-corrected RGB primary non-linear encoding.

[0065] Chroma subsampling is a practice of encoding images with lower precision for chrominance information than for luminance information, exploiting the fact that the human visual system is less sensitive to color differences than to luminance differences.

[0066] For 4:4:4 chroma subsampling, each of the three Y'CbCr components has the same sampling rate, so there is no chroma subsampling. This scheme is sometimes used for high-end film scanners and film post-production.

[0067] For 4:2:2 chroma subsampling, the two chroma components are sampled at half the luminance sampling rate: horizontal chroma precision is halved. This reduces the bandwidth of the uncompressed video signal by one third, but there is little visual difference.

[0068] For 4:2:0 chroma subsampling, compared to 4:1:1, horizontal sampling is doubled, but vertical precision is halved because in this scheme the Cb and Cr channels are only sampled on every other line. Thus, the data rate is the same. Cb and Cr are subsampled by a factor of 2 in the horizontal and vertical directions, respectively. There are three variants of the 4:2:0 scheme with different horizontal and vertical addressing.

[0069] In MPEG-2, Cb and Cr are co-located horizontally. Cb and Cr are addressed between pixels in the vertical direction (gap addressing). In Joint Photographic Experts Group (JPEG) / JPEG File Interchange Format (JFIF), H.261, and MPEG-1, Cb and Cr are gap addressed, located in the middle of alternate luma samples. In 4:2:0 DV, Cb and Cr are co-located horizontally. In the vertical direction, they are co-located on alternate lines.

[0070] Definitions of video units are provided. A picture is divided into one or more tile rows and one or more tile columns. A tile is a sequence of coding tree units (CTUs) covering a rectangular region of a picture. A tile is divided into one or more bricks, each consisting of multiple CTU rows within the tile. A tile that is not partitioned into multiple bricks is also referred to as a brick. However, a brick that is a proper subset of a tile is not referred to as a tile. A slice includes multiple tiles of a picture or multiple bricks of a tile.

[0071] Two slice modes are supported, namely, raster-scan slice mode and rectangular slice mode. In the raster-scan slice mode, a slice includes a sequence of tiles in raster scan of a picture. In the rectangular slice mode, a slice includes multiple bricks of a picture that collectively form a rectangular region of the picture. The bricks within a rectangular slice are arranged in the order of brick raster scan of the slice.

[0072] Figure 1 An example of raster-scan slice partitioning for picture 100, where the picture is divided into twelve tiles 102 and three raster-scan slices 104. As shown, each tile 102 and slice 104 includes multiple CTUs 106.

[0073] Figure 2 An example of rectangular slice partitioning for picture 200 according to the VVC specification, where the picture is divided into twenty-four tiles 202 (six tile columns 203 and four tile rows 205) and nine rectangular slices 204. As shown, each tile 202 and slice 204 includes multiple CTUs 206.

[0074] Figure 3 An example of partitioning picture 300 into tiles, bricks, and rectangular slices according to the VVC specification, where the picture is divided into four tiles 302 (two tile columns 303 and two tile rows 305), eleven bricks 304 (one brick in the top-left tile, five bricks in the top-right tile, two bricks in the bottom-left tile, and three bricks in the bottom-right tile), and four rectangular slices 306.

[0075] CTU and coding tree block (CTB) sizes are discussed. In VVC, the coding tree unit (CTU) size, which is signaled in the sequence parameter set (SPS) by the syntax element log2_ctu_size_minus2, can be as small as 4x4. The sequence parameter set (SPS) raw byte sequence payload (RBSP) syntax is as follows.

[0076]

[0077]

[0078] log2_ctu_size_minus2 plus 2 specifies the luma coding tree block size of each CTU.

[0079] log2_min_luma_coding_block_size_minus2 plus 2 specifies the minimum luma coding block size.

[0080] The variables CtbLog2SizeY, CtbSizeY, MinCbLog2SizeY, MinCbSizeY, MinTbLog2SizeY, MaxTbLog2SizeY, MinTbSizeY, MaxTbSizeY, PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC and PicHeightInSamplesC are derived as follows:

[0081] CtbLog2SizeY = log2_ctu_size_minus2 + 2 (7-9)

[0082] CtbSizeY = 1 « CtbLog2SizeY (7-10)

[0083] MinCbLog2SizeY = log2_min_luma_coding_block_size_minus2 + 2 (7-11)

[0084] MinCbSizeY = 1 « MinCbLog2SizeY (7-12)

[0085] MinTbLog2SizeY = 2 (7-13)

[0086] MaxTbLog2SizeY = 6 (7-14)

[0087] MinTbSizeY = 1 « MinTbLog2SizeY (7-15)

[0088] MaxTbSizeY = 1 « MaxTbLog2SizeY (7-16)

[0089] PicWidthInCtbsY = Ceil( pic_width_in_luma_samples ÷ CtbSizeY ) (7-17)

[0090] PicHeightInCtbsY = Ceil( pic_height_in_luma_samples ÷ CtbSizeY ) (7-18)

[0091] PicSizeInCtbsY = PicWidthInCtbsY * PicHeightInCtbsY (7-19)

[0092] PicWidthInMinCbsY = pic_width_in_luma_samples ÷ MinCbSizeY (7-20)

[0093] PicHeightInMinCtbsY = pic_height_in_luma_samples ÷ MinCbSizeY (7-21)

[0094] PicSizeInMinCtbsY = PicWidthInMinCtbsY * PicHeightInMinCtbsY (7-22)

[0095] PicSizeInSamplesY = pic_width_in_luma_samples * pic_height_in_luma_samples (7-23)

[0096] PicWidthInSamplesC = pic_width_in_luma_samples ÷ SubWidthC (7-24)

[0097] PicHeightInSamplesC = pic_height_in_luma_samples ÷ SubHeightC (7-25)

[0098] Figure 4A is an example of a CTB that crosses the bottom picture boundary. Figure 4B is an example of a CTB that crosses the right picture boundary. Figure 4C is an example of a CTB that crosses the bottom-right picture boundary. In Figures 4A-4C , there are K = M, L < N; K < M, L = N; K < M, L < N, respectively.

[0099] Referring to Figures 4A-4CCTUs in picture 400 are discussed. Assume that M x N indicates the CTB / largest coding unit (LCU) size (typically M is equal to N as defined in HEVC / VVC), and for a CTB located at picture (or slice or tile or other type, taking picture boundary as an example) boundary, K x L samples are within the picture boundary, where K < M or L < N. For Figures 4A-4C For those CTBs 402 depicted in FIG. 4, the CTB size is still equal to M x N, however, the bottom boundary / right side boundary of the CTB is outside of picture 400.

[0100] The coding process of a typical video encoder / decoder (aka codec) is discussed. Figure 5 is an example of an encoder block diagram of VVC, which contains three in-loop filters: a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF). Unlike DF which uses a pre-defined filter, SAO and ALF exploit original samples of the current picture to reduce the mean square error between original and reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, with the help of coded side information signaling the offset and filter coefficients. ALF is located at the last processing stage of each picture and can be regarded as a tool trying to capture and fix artifacts established by previous stages.

[0101] Figure 5 is a schematic diagram of an encoder 500. Encoder 500 is suitable for implementing VVC techniques. Encoder 500 includes three in-loop filters, namely a deblocking filter (DF) 502, a sample adaptive offset (SAO) 504, and an ALF 506. Unlike DF 502 which uses a pre-defined filter, SAO 504 and ALF 506 exploit original samples of the current picture to reduce the mean square error between original and reconstructed samples by adding an offset and applying a FIR filter, respectively, with the help of coded side information signaling the offset and filter coefficients. ALF 506 is located at the last processing stage of each picture and can be regarded as a tool trying to capture and fix artifacts established by previous stages.

[0102] The encoder 500 also includes an intra prediction component 508 and a motion estimation / compensation (ME / MC) component 510 configured to receive input video. The intra prediction component 508 is configured to perform intra prediction while the ME / MC component 510 is configured to perform inter prediction with reference to reference pictures obtained from a reference picture buffer 512. Residual blocks from either inter or intra prediction are fed to a transform component 514 and a quantization component 516 to generate quantized residual transform coefficients that are fed to an entropy coding component 518. The entropy coding component 518 entropy codes the prediction results and quantized transform coefficients and sends them to a video decoder (not shown). Quantized components output from the quantization component 516 can be fed to a dequantization component 520, an inverse transform component 522, and a reconstruction (REC) component 524. The REC component 524 is capable of outputting pictures to the DF 502, the SAO 504, and the ALF 506 for filtering before the pictures are stored in the reference picture buffer 512.

[0103] The input to the DF 502 is the reconstructed samples before the in-loop filter. First, vertical edges in the picture are filtered. Then, horizontal edges in the picture are filtered with the samples modified by the vertical edge filtering process as input. The vertical and horizontal edges in the CTBs of each CTU are processed individually on a coding unit basis. The vertical edges of the coding blocks in a coding unit are filtered starting from the left edge of the coding blocks and proceeding through the edges to the right side of the coding blocks in their geometric order. The horizontal edges of the coding blocks in a coding unit are filtered starting from the top edge of the coding blocks and proceeding through the edges to the bottom side of the coding blocks in their geometric order.

[0104] Figure 6 is an illustration 600 of a of samples 602 within an 8x8 sample block 604. As shown, the illustration 600 includes horizontal block boundaries 606 and vertical block boundaries 608 on an 8x8 grid, respectively. In addition, the illustration 600 depicts an 8x8 sample non-overlapping block 610, which can be de-blocked in parallel.

[0105] A discussion of boundary decision is presented. The filter is applied to 8x8 block boundaries. In addition, it must be a transform block boundary or a coding sub-block boundary (e.g., due to the use of affine motion prediction, optional temporal motion vector prediction (ATMVP)). For those boundaries that are not such boundaries, the filter is disabled.

[0106] A discussion of boundary strength calculation is presented. For a transform block boundary / coding sub-block boundary, if it is located in an 8x8 grid, the transform block boundary / coding sub-block boundary can be filtered and bS[xD i ][yD jThe settings for [xDi][yDj] (where [xDi][yDj] represent coordinates) are defined in Table 1 and Table 2, respectively.

[0107] Table 1. Boundary Strength (when SPS IBC is disabled)

[0108]

[0109] Table 2. Boundary Strength (when SPS IBC is enabled)

[0110]

[0111]

[0112] The deblocking decision for the luminance component is discussed.

[0113] Figure 7 Example 700 shows the pixels involved in the filter on / off decision and strong / weak filter selection. A wider, stronger brightness filter is used only if conditions 1, 2, and 3 are all true. Condition 1 is the "bulk condition." This condition detects whether samples on the P-side and Q-side belong to a bulk, represented by the variables bSidePisLargeBlk and bSideQisLargeBlk, respectively. The definitions of bSidePisLargeBlk and bSideQisLargeBlk are as follows.

[0114] bSidePisLargeBlk = ((edge ​​type is vertical, and p0 belongs to CU with width >= 32) || (edge ​​type is horizontal, and p0 belongs to CU with height >= 32)) ? TRUE:FLASE

[0115] bSideQisLargeBlk = ((edge ​​type is vertical, and q0 belongs to CU with width >= 32) || (edge ​​type is horizontal, and q0 belongs to CU with height >= 32)) ? TRUE:FLASE

[0116] Based on bSidePisLargeBlk and bSideQisLargeBlk, condition 1 is defined as follows.

[0117] Condition 1=(bsidepislageblk||bsidepislageblk)? TRUE:FLASE

[0118] Next, if condition 1 is true, condition 2 will be further checked. First, the following variables are derived.

[0119] In HEVC, first export dp0, dp3, dq0, and dq3.

[0120] If (p-side is greater than or equal to 32)

[0121] dp0=(dp0+Abs(p50-2*p40+p30)+1)>>1

[0122] dp3=(dp3+Abs(p53-2*p43+p33)+1)>>1

[0123] If (q-side is greater than or equal to 32)

[0124] dq0=(dq0+Abs(q50-2*q40+q30)+1)>>1

[0125] dq3=(dq3+Abs(q53-2*q43+q33)+1)>>1

[0126] Condition 2 = (d < β)? TRUE: FALSE

[0127] Where d = dp0 + dq0 + dp3 + dq3.

[0128] If conditions 1 and 2 are valid, then further check whether any block uses sub-blocks.

[0129]

[0130] Finally, if both conditions 1 and 2 are valid, the proposed deblocking method will check condition 3 (large block strong filtering condition), which is defined as follows.

[0131] In condition 3 StrongFilterCondition, export the following variables.

[0132] dpq is derived in the same way as in HEVC.

[0133] sp3 = Abs(p3 - p0), as derived in HEVC.

[0134] If (p-side is greater than or equal to 32)

[0135] If (Sp == 5)

[0136] sp3=(sp3+Abs(p5-p3)+1)>>1

[0137] otherwise

[0138] sp3=(sp3+Abs(p7-p3)+1)>>1

[0139] sq3 = Abs(q0 - q3), as derived in HEVC if (q is greater than or equal to 32).

[0140] If (Sq == 5)

[0141] sq3=(sq3+Abs(q5-q3)+1)>>1

[0142] otherwise

[0143] sq3=(sq3+Abs(q7-q3)+1)>>1

[0144] In HEVC, StrongFilterCondition = (dpq < (β >> 2), sp3 + sq3 < (3 * β >> 5), and Abs(p0 - q0) < (5 * t). C +1)>>1)? TRUE:FALSE.

[0145] A stronger deblocking filter for brightness (designed for larger blocks) is discussed.

[0146] Bilinear filtering is used when samples on either side of the boundary belong to a large block. Samples belonging to a large block are defined when the width of the vertical edge is ≥32 and the height of the horizontal edge is ≥32.

[0147] Bilinear filters are listed below.

[0148] Then, in the above HEVC deblocking, for the block boundary sample points p from i = 0 to Sp-1... i And the block boundary sample points q from j=0 to Sq-1 i Replaced by linear interpolation, p i and q i This is to filter the i-th sample point in a row of vertical edges, or to filter the i-th sample point in a column of horizontal edges, as shown below.

[0149] p i ′=(f i *Middle s,t +(64-fi)*P s +32)>>6), clipped to p i ±tcPD i

[0150] q j ′=(g j *Middle s,t +(64-g j )*Q s +32)>>6), clipped to q j ±tcPD j

[0151] Among them, tcPD i and tcPDj The terminology is the position-related clipping described below, and g is given below. j f i Middle s,t P s and Q s .

[0152] The deblocking control of chroma was discussed.

[0153] A strong chroma filter is used on both sides of the block boundary. Here, the chroma filter is selected when both sides of the chroma edge are greater than or equal to 8 (chroma position), and the decision is based on the following three conditions: The first considers the boundary strength and the size of the block. The proposed filter can be applied when the width or height of the block orthogonal to the block edge in the chroma sampling domain is equal to or greater than 8. The second and third are essentially the same as the HEVC luma deblocking decision, namely the on / off decision and the strong filtering decision, respectively.

[0154] In the first decision, the boundary strength (bS) of the chroma filter is modified, and the conditions are checked sequentially. If a condition is met, the remaining lower-priority conditions are skipped.

[0155] Chromatic deblocking is performed when bS equals 2, or when a large block boundary is detected when bS equals 1.

[0156] The second and third conditions are essentially the same as the HEVC luminance strong filtering decision, as shown in the following example.

[0157] In the second condition: d is derived as in the HEVC luminance deblock. The second condition is TRUE when d is less than β.

[0158] In the third condition, StrongFilterCondition is deduced as follows.

[0159] For example, deriving dpq in HEVC.

[0160] sp3 = Abs(p3 - p0), as derived in HEVC.

[0161] sq3 = Abs(q0 - q3), as derived in HEVC.

[0162] In HEVC design, StrongFilterCondition = (dpq < (β >> 2), sp3 + sq3 < (β >> 3), Abs(p0 - q0) < (5 * t) C +1)>>1).

[0163] Strong deblocking filters for chroma are discussed. The following strong deblocking filters for chroma are defined.

[0164] p2′=(3*p3+2*p2+p1+p0+q0+4)>>3

[0165] p1′=(2*p3+p2+2*p1+p0+q0+q1+4)>>3

[0166] p0′=(p3+p2+p1+2*p0+q0+q1+q2+4)>>3

[0167] The proposed chromaticity filter performs deblocking on a 4×4 chromaticity sample grid.

[0168] Position-dependent limiting (tcPD) is discussed. tcPD is applied to the output samples of the brightness filtering process, which involves modifying the strong and long filters at the boundaries of 7, 5, and 3 samples. Assuming a quantization error distribution, it is proposed to increase the limiting value of samples expected to have higher quantization noise, thus resulting in a larger deviation between the expected reconstructed sample values ​​and the true sample values.

[0169] For each P or Q boundary filtered using an asymmetric filter, a position-related threshold table is selected from two tables (i.e., Tc7 and Tc3 in the table below) based on the results of the decision process in the boundary strength calculation, and it is provided to the decoder as edge information.

[0170] Tc7={6,5,4,3,2,1,1}; Tc3={6,4,2};

[0171] tcPD=(Sp==3)? Tc3:Tc7;

[0172] tcQD=(Sq==3)? Tc3:Tc7;

[0173] For P or Q boundaries filtered by a short symmetric filter, apply a lower amplitude position correlation threshold.

[0174] Tc3 = {3, 2, 1};

[0175] After defining the threshold, the filtered p' is adjusted based on the tcP and tcQ limiting values. i q' i The sample values ​​are limited.

[0176] p” i =Clip3(p' i +tcP i ,p' i –tcP i ,p' i );

[0177] q” j =Clip3(q' j +tcQj ,q' j –tcQ j ,q' j );

[0178] Where p' i and q' i It is the filtered sample value p” i and q" j It is the output sample value after clipping, tcP i tcP i It is the clipping threshold derived from the VVC tc parameters, as well as tcPD and tcQD. The function Clip3 is the clipping function defined in VVC.

[0179] Now we discuss sub-block deblocking adjustments. To achieve parallel-friendly deblocking using long filters and sub-block deblocking, the long filter is restricted to modifying a maximum of 5 samples on the side using sub-block deblocking (AFFINE or ATMVP or decoder-side motion vector refinement (DMVR)), as shown in the long filter's brightness control. Furthermore, sub-block deblocking is adjusted such that sub-block boundaries on the 8×8 grid near the codec unit (CU) or implicit TU boundary are restricted to modifying a maximum of two samples on each side.

[0180] The following applies to sub-block boundaries that are not aligned with the CU boundary.

[0181]

[0182] Where an edge equal to 0 corresponds to the CU boundary, an edge equal to 2 or orthogonalLength-2 corresponds to 8 sample points of the sub-block boundary from the CU boundary, and so on. If implicit partitioning of TU is used, then implicit TU is true.

[0183] Sample Adaptive Offset (SAO) is discussed. The input to SAO is the deblocked reconstructed samples (DB). The concept of SAO is to reduce the average sample distortion of a region by first classifying region samples into multiple categories using a selected classifier, obtaining an offset for each category, and then adding this offset to each sample of that category. The classifier index and the region offset are encoded and decoded in the bitstream. In HEVC and VVC, the region (the unit for SAO parameter signaling notification) is defined as a CTU.

[0184] HEVC employs two SAO types that meet low complexity requirements. These two types are Edge Offset (EO) and Band Offset (BO), which will be discussed in detail below. The index of the SAO type is encoded and decoded (within the range [0, 2]). For EO, ​​sample classification is based on a comparison between the current sample and its neighboring samples, according to a one-dimensional orientation scheme: horizontal, vertical, 135° diagonal, and 45° diagonal.

[0185] Figure 8 Four one-dimensional (1-D) orientation schemes 800 for EO sample point classification are shown: horizontal (EO classification = 0), vertical (EO classification = 1), 135° diagonal (EO classification = 2), and 45° diagonal (EO classification = 3).

[0186] For a given EO classification, each sample point within the CTB is classified into one of five categories. The current sample point value, labeled "c", is compared with its two neighboring sample point values ​​along the selected 1-D pattern. The classification rules for each sample point are summarized in Table 3. Categories 1 and 4 are associated with local valleys and local peaks along the selected 1-D pattern, respectively. Categories 2 and 3 are associated with concave and convex angles along the selected one-dimensional scheme, respectively. If the current sample point does not belong to EO categories 1-4, it belongs to category 0, and SAO is not applied.

[0187] Table 3: Sampling classification rules for edge offset

[0188]

[0189] An adaptive loop filter based on geometric transformation (SAO) is discussed in the Joint Exploration Model (JEM). The input to the DB is the reconstructed samples after DB and SAO. The sample classification and filtering processes are based on the reconstructed samples after DB and SAO.

[0190] In JEM, a geometric transformation-based adaptive loop filter (GALF) with block-based filter adaptation is applied. For the luminance component, one of 25 filters is selected for each 2×2 block based on the direction and effectiveness of the local gradient.

[0191] The shape of the filter was discussed. Figure 9 An example of a GALF filter shape 900 is shown, including a 5×5 rhombus on the left, a 7×7 rhombus in the middle, and a 9×9 rhombus on the right. In JEM, up to three rhombus filter shapes can be selected for the luminance component (e.g., ...). Figure 9 (As shown). The image-level signaling notification index indicates the filter shape used for the luminance component. Each square represents a sample point, and Ci (i is 0-6 (left), 0-12 (middle), 0-20 (right)) represents the coefficient applied to that sample point. For the chrominance component in the image, a 5×5 rhombus is always used.

[0192] Block classification was discussed. Each 2×2 block was divided into one of 25 classes. The classification index c was based on its directionality D and validity. The quantized values ​​are exported as shown below.

[0193]

[0194] To calculate D and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using the 1-D Laplacian operator.

[0195]

[0196]

[0197] H k,l =|2R(k,l)-R(k-1,l)-R(k+1,l)|,

[0198]

[0199]

[0200] Indices i and j refer to the coordinates of the top-left sample point in the 2×2 block, and R(i,j) represents the reconstructed sample point at coordinates (i,j).

[0201] The maximum and minimum values ​​of the gradients in the horizontal and vertical directions are then set as follows:

[0202]

[0203] The maximum and minimum values ​​of the gradients in the two diagonal directions are set as follows:

[0204]

[0205] To derive the values ​​of directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0206] Step 1. If and If true, then D is set to 0.

[0207] Step 2. If Continue from step 3; otherwise, continue from step 4.

[0208] Step 3. If D is set to 2; otherwise, D is set to 1.

[0209] Step 4. If D is set to 4; otherwise, D is set to 3.

[0210] The validity value A is calculated as follows:

[0211]

[0212] A is further quantized to the range of 0 to 4, and the quantized value is represented as

[0213] For the two chromaticity components in the image, no classification method is applied; that is, a single set of ALF coefficients is applied for each chromaticity component.

[0214] The geometric transformation of the filter coefficients is discussed.

[0215] Figure 10 The relative coordinates 1000 are shown for the 5×5 diamond filters that support (from left to right) diagonal, vertical flip, and rotation.

[0216] Before filtering each 2×2 block, geometric transformations such as rotation, diagonal flip, and vertical flip are applied to the filter coefficients f(k,l) associated with coordinates (k,l) based on the gradient values ​​calculated for that block. This is equivalent to applying these transformations to samples within the filter's support region. The idea is to make the different blocks applying ALF more similar by aligning their orientations.

[0217] Three geometric transformations are introduced: diagonal transformation, vertical flip, and rotation.

[0218]

[0219] Where K is the size of the filter, and 0 ≤ k, l ≤ K⁻¹ are the coefficient coordinates, such that position (0, 0) is located at the top left corner and position (K⁻¹, K⁻¹) is located at the bottom right corner. The transform is applied to the filter coefficients f(k, l) based on the gradient values ​​calculated for this block. Table 4 summarizes the relationship between the transform and the four gradients in the four directions.

[0220] Table 1: Mapping between gradients and transformations computed for a block

[0221]

[0222] Signaling notification of filter parameters is discussed. In JEM, GALF filter parameters are signaled for the first CTU, i.e., after the stripe header of the first CTU and before the SAO parameters. A maximum of 25 groups of luminance filter coefficients can be sent. To reduce bit overhead, filter coefficients from different categories can be merged. Furthermore, the GALF coefficients of a reference image are stored and can be reused as GALF coefficients for the current image. The current image can optionally use the GALF coefficients stored for the reference image and bypass GALF coefficient signaling. In this case, only the index of a reference image is signaled, and the stored GALF coefficients of the indicated reference image are inherited by the current image.

[0223] To support GALF temporal prediction, a candidate list of GALF filter sets is maintained. At the beginning of decoding a new sequence, the candidate list is empty. After decoding an image, the corresponding filter set can be added to the candidate list. Once the size of the candidate list reaches the maximum allowed value (i.e., 6 in the current JEM), new filter sets overwrite the oldest sets in decoding order; that is, a first-in-first-out (FIFO) rule is applied to update the candidate list. To avoid duplication, a set is added to the list only if the corresponding image does not use GALF temporal prediction. To support temporal scalability, multiple candidate lists of filter sets exist, and each candidate list is associated with a temporal layer. More specifically, each array assigned by the temporal layer index (TempIdx) can form the filter set of previously decoded images with a lower TempIdx. For example, the k-th array is assigned to be associated with a TempIdx equal to k, and the k-th array contains only the filter set from images with a TempIdx less than or equal to k. After an image is encoded or decoded, the set of filters associated with that image will be used to update the arrays associated with TempIdx that are equal to or higher than TempIdx.

[0224] Temporal prediction of GALF coefficients is used for inter-frame encoding / decoding to minimize signaling overhead. For intra-frame frames, temporal prediction is not available, and a set of 16 fixed filters is assigned to each class. To indicate the use of fixed filters, signaling informs the flag for each class and, if necessary, the index of the selected fixed filter. Even when a fixed filter is selected for a given class, the coefficients of the adaptive filter f(k,l) can still be sent for that class; in this case, the coefficients of the filter applied to the reconstructed image are the sum of the coefficients from both sets.

[0225] The filtering process for the luminance component can be controlled at the CU level. A signaling flag indicates whether GALF is applied to the luminance component of the CU. For the chrominance component, whether GALF is applied is only indicated at the image level.

[0226] The filtering process is discussed. On the decoder side, when GALF is enabled for a block, each sample R(i,j) within the block is filtered, producing sample values ​​R′(i,j) as shown below, where L represents the filter length, f m,n Let f(k,l) represent the filter coefficients, and let f(k,l) represent the decoding filter coefficients.

[0227]

[0228] Figure 11This example illustrates the relative coordinates supported by a 5×5 diamond filter, assuming the current sample point's coordinates (i, j) are (0, 0). Sample points at different coordinates, filled with the same color, are multiplied by the same filter coefficients.

[0229] The geometric transformation-based adaptive loop filter (GALF) for VVC is discussed. In VVC test model 4.0 (VTM4.0), the filtering process of the adaptive loop filter is performed as follows:

[0230] O(x,y)=∑ (i,j) w(i,j).I(x+i,y+j), (11)

[0231] Here, sample I(x+i,y+j) is the input sample, O(x,y) is the filtered output sample (i.e., the filtering result), and w(i,j) represents the filtering coefficients. In fact, VTM 4.0 uses integer arithmetic to achieve fixed-point precision calculations.

[0232]

[0233] Where L represents the filter length, and w(i,j) are the fixed-point filter coefficients.

[0234] Compared to JEM, the current design of GALF in VVC has the following main changes:

[0235] 1) Adaptive filter shapes have been removed. Only 7×7 filter shapes are allowed for the luminance component, and only 5×5 filter shapes are allowed for the chrominance component.

[0236] 2) Move the signaling notification of ALF parameters from the strip / picture level to the CTU level.

[0237] 3) The category index is calculated at a 4×4 level instead of a 2×2 level. Furthermore, as proposed in JVET-L0147, a subsampling Laplacian calculation method for ALF classification is utilized. More specifically, it is not necessary to calculate the horizontal / vertical / 45-degree diagonal / 135-degree gradient for each sample point within a block. Instead, 1:2 subsampling is used.

[0238] Regarding filtering and reconstruction, the nonlinear ALF in the current VVC is discussed.

[0239] Equation (11) can be reformulated without affecting encoding / decoding efficiency as follows:

[0240] O(x,y)=I(x,y)+∑ (i,j)≠(0,0) w(i,j).(I(x+i,y+j)-I(x,y)), (13)

[0241] Where w(i,j) are the same filter coefficients as in equation (11) [except that w(0,0) is equal to 1 in equation (13), while it is equal to 1-∑ in equation (11).] (i,j)≠(0,0) w(i,j)].

[0242] Using the above filter formula of equation (13), VVC introduces nonlinearity to reduce the impact when the neighboring sample values ​​(I(x+i,y+j)) differ too much from the current sample value (I(x,y)) being filtered by using a simple limiting function, thereby making ALF more effective.

[0243] More specifically, the ALF filter is modified as follows:

[0244] O′(x,y)=I(x,y)+∑ (i,j)≠(0,0) w(i,j).K(I(x+i,y+j)-I(x,y),k(i,j)), (14)

[0245] Where K(d,b)=min(b,max(-b,d)) is the limiting function, and k(i,j) is the limiting parameter that depends on the (i,j) filter coefficients. The encoder performs optimization to find the optimal k(i,j).

[0246] In the JVET-N0242 implementation, a limiting parameter k(i,j) is specified for each ALF filter, and a limiting value is signaled for each filter coefficient. This means that a maximum of 12 limiting values ​​can be signaled for each luminance filter in the bitstream, and a maximum of 6 limiting values ​​can be signaled for each chrominance filter.

[0247] To limit signaling notification costs and encoder complexity, only four fixed values ​​are used, which are the same for both INTER and INTRA stripes.

[0248] Because the local differences in luminance are typically greater than those in chrominance, two separate sets of luminance and chrominance filters are applied. The maximum sample value in each set (1024 for a 10-bit depth) is also introduced so that clipping can be disabled when not necessary.

[0249] Table 5 provides a set of limiting values ​​used in the JVET-N0242 test. These four values ​​were selected by roughly dividing the full range of luminance sample values ​​(encoded in 10 bits) and the chrominance range from 4 to 1024 in the logarithmic domain.

[0250] More precisely, the brightness table for the limit value is obtained using the following formula:

[0251]

[0252] Similarly, the colorimetric table for the limiting values ​​is obtained using the following formula:

[0253]

[0254] Table 5: Authorized Limit Values

[0255]

[0256] The selected threshold value is encoded and decoded in the "alf_data" syntax element using the Golomb coding scheme corresponding to the threshold value index in Table 5 above. This coding scheme is the same as that used for the filter index.

[0257] A convolutional neural network-based loop filter for video encoding and decoding is discussed.

[0258] In deep learning, convolutional neural networks (CNNs or ConvNets) are a type of deep neural network most commonly used for analyzing visual images. They have been very successful in image and video recognition / processing, recommender systems, image classification, medical image analysis, and natural language processing.

[0259] CNNs are a regularized version of multilayer perceptrons. Multilayer perceptrons typically mean fully connected networks, where each neuron in one layer is connected to all neurons in the next layer. This "full connectivity" makes them prone to overfitting data. Typical regularization methods involve adding some form of weight magnitude metric to the loss function. CNNs take a different approach to regularization. CNNs utilize hierarchical schemes in the data and assemble more complex methods using smaller and simpler schemes. Therefore, CNNs are at the lower extremes in terms of connectivity and complexity.

[0260] Compared to other image classification / processing algorithms, CNNs use relatively little preprocessing. This means the network learns filters that are hand-designed in traditional algorithms. This independence from existing knowledge and human effort in feature design is a major advantage.

[0261] Deep learning-based image / video compression generally has two meanings: end-to-end compression based entirely on neural networks, and traditional frameworks enhanced by neural networks. End-to-end compression based entirely on neural networks is discussed in Johannes Balle, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” 2016, Workshop on Picture Coding (PCS), pp. 1–5, Institute of Electrical and Electronics Engineers (IEEE), and in Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár, “Lossy image compression using compression autocoders,” arXiv prequel arXiv:2003.00395 (2017). The traditional framework enhanced by neural networks is discussed in the following articles: Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong and Wen Gao, “Intra-frame prediction for image coding and decoding based on fully connected networks”, IEEE Transactions on Image Processing 27, 7 (2018), 3236–3247; Yuanying Dai, Dong Liu and Feng Wu, “Post-processing convolutional neural network approach in HEVC intra-frame coding and decoding”, MMM.Springer, 28–39; Rui Song, Dong Liu, Houqiang Li and Feng Wu, “Neural network-based arithmetic coding and decoding of intra-frame prediction modes in HEVC”, VCIP.IEEE, 1–4; and J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, W. Samek, H. Schwarz, D. Marpe and T. Wiegand, “Neural network-based intra-frame prediction for video coding and decoding”, Digital Image Processing Applications XLI, Vol. 10752, International Society for Optics and Photonics, 1075213.

[0262] The first type typically employs an autoencoder-like structure, implemented through convolutional neural networks or recurrent neural networks. While relying solely on neural networks for image / video compression avoids any manual optimization or design, the compression efficiency may not be satisfactory. Therefore, research focused on the second type aims to enhance traditional compression frameworks by replacing or strengthening certain modules with neural networks. In this way, they can inherit the advantages of highly optimized traditional frameworks. For example, the fully connected network for intra-frame prediction proposed in HEVC is discussed in Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao, “Intra-frame Prediction Based on Fully Connected Networks for Image Encoding and Decoding,” IEEE Transactions on Image Processing 27, 7 (2018), pp. 3236-3247.

[0263] Besides intra-frame prediction, deep learning is also used to enhance other modules. For example, the loop filter in HEVC is replaced by a convolutional neural network, and satisfactory results were achieved in Yuanying Dai, Dong Liu, and Feng Wu, "A Convolutional Neural Network Approach for Post-Processing in HEVC Intra-Frame Encoding and Decoding," MMM. Springer, 28–39. Research in Rui Song, Dong Liu, Houqiang Li, and Feng Wu, "Neural Network-Based Arithmetic Encoding and Decoding for HEVC Intra-Frame Prediction Modes," VCIP. IEEE, 1–4, applies neural networks to improve the arithmetic encoding and decoding engine.

[0264] Loop filtering based on convolutional neural networks is discussed. In lossy image / video compression, the reconstructed frame is an approximation of the original frame, and because the quantization process is irreversible, this leads to distortion in the reconstructed frame. To mitigate this distortion, convolutional neural networks can be trained to learn the mapping from distorted frames to the original frames. In fact, training is necessary before deploying CNN-based loop filtering.

[0265] Training was discussed. The goal of the training process is to find the optimal values ​​for the parameters, including the weights and biases.

[0266] First, an encoder-decoder (e.g., HM, JEM, VTM, etc.) is used to compress the training dataset to generate distorted reconstructed frames. Then, the reconstructed frames are fed into a CNN, and the cost is calculated using the CNN's output and the ground truth frames (original frames). Common cost functions include Sum of Absolute Differences (SAD) and Mean Squared Error (MSE). Next, the gradient of the cost with respect to each parameter is derived using the backpropagation algorithm. The parameter values ​​are updated using the gradients. This process is repeated until the convergence criterion is met. After training is complete, the derived optimal parameters are saved for use in the inference phase.

[0267] The convolution process is discussed. During convolution, the filter moves across the image from left to right and from top to bottom, changing one pixel column for horizontal movement and one pixel row for vertical movement. The amount of movement by which the filter is applied across the input image is called the stride, and it is almost always symmetrical in the height and width dimensions. For height and width movements, the default (multiple) stride in two dimensions is (1, 1).

[0268] Figure 12A This is an example architecture 1200 of the proposed CNN filter, and Figure 12B This is an example of constructing a ResBlock (1250). In most deep convolutional neural networks, ResBlocks are used as basic modules and stacked several times to build the final network. In one example, ResBlocks are obtained by combining convolutional layers, ReLU / PReLU activation functions, and convolutional layers, such as... Figure 12B As shown.

[0269] Inference is discussed. During the inference phase, distorted reconstructed frames are fed into a CNN and processed by a CNN model whose parameters have been determined during training. The input samples to the CNN can be reconstructed samples before or after DB, or before or after SAO, or before or after ALF.

[0270] Current CNN-based loop filtering suffers from several issues. For example, the NN model does not use or otherwise leverages external information (e.g., information from the video codec, such as codec parameters and / or codec syntax, which are information beyond what the NN model itself generates) as attention or as an attention mechanism. Therefore, information including various codec parameters (e.g., predictions (e.g., prediction patterns, motion vectors, etc.), partitioning (e.g., segmentation information), etc.) may not be fully utilized, for example, to recalibrate feature maps. Furthermore, conventional NN models may have a fixed depth (e.g., the number of residual blocks in the NN model, as shown above). Figure 12A and Figure 12B (As shown in the examples), this can lead to overprocessing or underprocessing of video units, depending on various characteristics of the video unit. Overprocessing video units (e.g., by applying a neural network model with more residual blocks than needed) can be inefficient. Underprocessing video units (e.g., by applying a neural network model with too few residual blocks) can lead to degraded video encoding / decoding performance.

[0271] This document discloses techniques for addressing one or more of the aforementioned problems. For example, this disclosure provides one or more neural network (NN) filter models trained as encoding / decoding tools to improve the efficiency of video encoding / decoding. NN-based encoding / decoding tools can be used to replace or otherwise enhance modules involved in a video codec. For example, NN models can act as additional intra-frame prediction modes, inter-frame prediction modes, transform kernels, or loop filters. This disclosure also illustrates how to use multiple NN models, where at least some NN models have a different depth (e.g., the number of residual blocks) than the others. The selection of a particular NN model can be done in a predefined manner or derived on the fly, as will be further described below. It should be noted that NN models can be used as any encoding / decoding tool, such as NN-based intra / inter-frame prediction, NN-based super-precision, NN-based motion compensation, NN-based reference generation, NN-based fractional pixel interpolation, NN-based loop / post-filtering, etc.

[0272] The following list of embodiments should be considered as examples for explaining general concepts. These embodiments should not be interpreted narrowly. Furthermore, these embodiments can be combined in any way.

[0273] In this disclosure, the NN model can include any kind of NN architecture, such as a convolutional neural network (CNN) or a fully connected NN, or a combination of CNN and fully connected neural networks. In the discussion below, the NN model may also be referred to as a CNN model.

[0274] In the following discussion, a video unit can be a sequence, picture, strip, slice, tile, sub-picture, CTU / CTB, CTU / CTB line, one or more CU / codec block (CB), one or more CTU / CTB, one or more Virtual Pipeline Data Units (VPDUs), or a sub-region within a picture / strip / slice / tile. A parent video unit represents a unit larger than a video unit. Generally, a parent unit contains several video units; for example, when the video unit is a CTU, the parent unit can be a strip, a CTU line, multiple CTUs, etc. In some embodiments, a video unit can be a sample / pixel.

[0275] Figure 13This is an example of a process 1300 for generating filtered samples from an NN filter model based on received encoding / decoding parameters (e.g., extrinsic information of the NN filter model) as input. The NN filter model has an attention mechanism based on the encoding / decoding parameter input. The attention mechanism is configured to generate or acquire attention, which is useful for mimicking cognitive attention and effectively enhances certain parts of the data input to the NN filter while reducing others. For example, attention can be acquired by processing extrinsic information of the NN filter model, such as one or more encoding / decoding parameters. The attention is configured to be applied to one or more feature maps, for example, by weighting the feature maps according to the attention.

[0276] exist Figure 13 In the process 1300 shown, at least some unfiltered samples are provided as input to the NN filter. In one example, the unfiltered samples are samples (e.g., pixels) of video units that have not yet undergone any filtering or have not undergone a sufficient amount of filtering. Therefore, the output of the NN filter can be filtered samples. The output of the NN filter is also based on an NN filter model generated using codec parameter inputs. The codec parameter inputs may be related to the reconstruction of the video unit, the partitioning or segmentation scheme of the video unit, the prediction mode of the video unit, the quantization parameter (QP) associated with the video unit, and / or the boundary strength parameter of the boundary of the video unit.

[0277] For example, convolutional layers can be used to extract features from encoding / decoding parameters (e.g., extrinsic information), or from a combination of encoding / decoding parameters and internal information, such as features extracted within an NN filter model. At least one of the extracted features is used as attention obtained from the NN filter model.

[0278] Figure 14 This is an example of a process 1400 that applies attention, obtained using external information such as encoding / decoding parameters, to the feature maps of an NN filter model to provide recalibrated feature maps, based on several examples. The feature maps of the NN filter model are the result of applying filters to an input image (or feature maps provided as outputs from previous layers). For example, in each layer of the NN filter model, the feature map is the output of that layer. Therefore, feature maps produced by layers within the NN filter model can be referred to as intermediate feature maps, while feature maps produced by the final layer of the NN filter model can be referred to as final feature maps. For example, the final output of the NN filter model can be the final feature map.

[0279] exist Figure 14In the example, the operation performed by the convolutional layers(s) of the NN filter model is denoted as α, while external information (e.g., encoding / decoding parameters) is denoted as E, and internal information (e.g., feature maps extracted within the NN filter model) is denoted as I. The obtained attention is denoted as A. Procedure 1400 applies the attention A to the feature map G to generate a recalibrated feature map φ.

[0280] Therefore, operation α is applied to external information (e.g., encoding / decoding parameters), or a cascade of external and internal information, to extract or otherwise obtain attention A. Attention A is applied to feature map G to generate a recalibrated feature map φ. For example, applying attention A to feature map G effectively reweights feature map G (e.g., applying different weights to different elements of feature map G), and the recalibrated feature map φ is the reweighted feature map obtained by applying attention A to feature map G. In process 1400, N is the number of channels, W is the channel width, and H is the channel height.

[0281] exist Figure 14 In the NN filter model, the intermediate feature map is represented by G, where G∈R N×W×H The attention received is denoted as A, where A∈R. W×H This represents the attention received. In this paper, the symbol ∈ indicates that G is R. N×W×H Given an element from a set, and A is R W×H One element in the given set

[0282] For example, R is the domain to which a particular element in the feature map or attention belongs. R can be a domain that includes any real number, or a domain that includes any integer real number. In some examples, the feature map is a two-dimensional or three-dimensional array. Thus, the feature map G, such as the intermediate feature map G, where G∈R N×W×H The expression indicates that each element of the feature map G belongs to R, and that the feature map G has N×W×H elements. In this example, the values ​​of N, W, and H indicate that the feature map G is a three-dimensional array with dimensional values ​​of N, W, and H along each dimension, respectively. In another example, consider attention A, A∈R. W×H Each element of attention A belongs to R, and attention A has W×H elements. In this example, the values ​​of W and H indicate that attention A is a two-dimensional array with dimension values ​​of W and H along each dimension.

[0283] In one example, a recalibrated feature map is generated based on the following equation:

[0284] φ i,j,k =G i,j,k ×A j,k ,1≤i≤N,1≤j≤W,and1≤k≤H.

[0285] In another example, a recalibrated feature map is generated based on the following equation:

[0286] φ i,j,k =G i,j,k ×f(A j,k ), 1≤i≤N, 1≤j≤W, 1≤k≤H, and f represents the mapping function applied to each element of the attention.

[0287] In yet another example, a recalibrated feature map is generated based on the following equation:

[0288] φ i,j,k =G i,j,k ×f(A j,k )+G i,j,k , 1≤i≤N, 1≤j≤W, 1≤k≤H, and f represents the mapping function applied to each element of the attention.

[0289] As mentioned above, residual blocks are used as basic modules in neural network models, and multiple residual blocks can be stacked to build the final network. For example, residual blocks can be obtained by combining convolutional layers, ReLU / PreLU activation functions, and another convolutional layer, such as... Figure 12B As shown. Therefore, in at least some examples, attention is applied only to the last layer of the residual block (e.g., the feature map output of the last layer), and not to every layer of the residual block. Examples of applying attention in this way are also... Figure 16B As shown in the diagram, this will be discussed further below.

[0290] In another example, attention is applied only to a specified layer of the NN filter model (e.g., the feature map output by a specified layer), rather than to every layer of the NN filter model. The specified layer can be determined by the topology of the NN filter model.

[0291] Figure 15 This is an example of a process 1500 that applies a neural network filter with configurable depth to unfiltered samples of a video cell to generate filtered samples. Figure 15 In the process 1500 shown, at least some unfiltered samples are provided as input to the filtering system 1502. In one example, the unfiltered samples are samples (e.g., pixels) of video units that have not yet undergone any filtering or have not undergone a sufficient amount of filtering. Therefore, the output of the filtering system 1502 can be filtered samples.

[0292] The filtering system 1502 includes an NN filter 1504 based on an NN filter model with a first depth, and an NN filter 1506 based on an NN filter model with a second depth. Therefore, the depth of the filtering system 1502 or its NN filter model is configurable. NN filters 1504 and 1506 are similar to those described above, for example, regarding... Figure 12A and Figure 12B As described above, NN filters 1504 and 1506 include multiple residual blocks, and the number of residual blocks in each NN filter 1504 and 1506 is equal to the depth of that NN filter 1504 and 1506. Figure 15 The examples shown are illustrative, and in other examples, the filtering system 1502 may include more than two filters (e.g., the example of two filters shown is not limiting unless explicitly stated). Therefore, the applied NN filters can implement more than two depths.

[0293] As described above, in some examples, the NN filter model can have different depths, which can be selected in a predefined manner or derived on the fly. In some cases, the filtering system 1502 includes NN filters with configurable depths, represented by NN filter 1504 (based on a filter model with a first depth) and NN filter 1506 (based on a filter model with a second depth). In other cases, the filtering system 1502 includes multiple NN filters 1504, 1506, each based on an NN filter model with a different depth, and one of the NN filters 1504, 1506 is selected (e.g., pass filter selection logic 1508) as the NN filter based on various characteristics of the video unit or its unfiltered samples.

[0294] In various examples, an NN filter is applied to unfiltered samples to generate filtered samples. Figure 15 In this system, the depth of the NN filter model is selectable; therefore, the NN filter can be a first NN filter 1504 (using an NN filter model with a first depth) or a second NN filter 1506 (using an NN filter model with a second depth). The filtering system 1502 includes filter selection logic 1508 configured to determine the depth to be applied to unfiltered samples and select one of the NN filters 1504, 1506 (or the filter model used by them) based on the depth to be applied. For example, the filter selection logic 1508 can determine to apply the NN filter 1504 with the first depth to a first video unit and to apply the NN filter 1506 with the second depth to a second video unit.

[0295] Regardless of the depth of the NN filter model selected by filter selection logic 1508, an NN filter of a specific depth is applied to the unfiltered samples to generate filtered samples (e.g., the output of filtering system 1502). The conversion between video media files and bitstreams can then be performed based on the generated filtered samples, as described above.

[0296] In one example, filter selection logic 1508 is configured to determine depth based on one or more of the following: the temporal layer of the video unit, including the type of stripe or picture of the video unit (e.g., "i" for intra-frame, "e" for inter-frame, or "b" for bi-directional), a list of reference pictures that match the video unit, the color components of the video unit, and the color format of the video unit (e.g., luma-chroma or RGB).

[0297] In some examples, filter selection logic 1508 is configured to determine which NN filter models of different depths will be applied to different video units in an image or strip. For example, NN filter 1504 is selected for the first video unit in the image or strip, while NN filter 1506 is selected for the second video unit in the image or strip.

[0298] In other examples, filter selection logic 1508 is configured to determine which NN filter models with different depths will be applied to different video units in different pictures or stripes. For example, NN filter 1504 is selected for a first video unit in a first picture or first strip, while NN filter 1506 is selected for a second video unit in a second picture or second strip.

[0299] In other examples, filter selection logic 1508 is configured based on the temporal layer or stripe type of the video unit to determine which NN filter models of different depths will be applied to the video unit. In some cases, temporal layers are grouped into subgroups, and NN filter models of the same depth are applied to video units in temporal layers within a specific subgroup, while NN filter models of different depths are applied to video units in temporal layers within different subgroups.

[0300] In some examples, the filter selection logic 1508 is configured to determine the application of NN filter models with different depths based on whether the video unit is a codec tree unit (CTU), codec tree block (CTB), CTU line, CTB line, strip, slice, picture, sequence, or subpicture.

[0301] The depth of a specific video unit (or an NN filter model with that depth) can also be signaled, for example, sent by the encoder of the video unit to the decoder of the video unit. For example, supplemental enhancement information (SEI) messages, sequence parameter sets (SPS), picture parameter sets (PPS), adaptive parameter sets (APS), video parameter sets (VPS), picture headers, or stripe headers can be used to signal the depth to the decoder.

[0302] In some examples, signaling informs different depths (or NN filter models with that depth) for different video units. In other examples, such as when unfiltered samples contain luma and chroma components, signaling may be made individually for each of the luma and chroma components to inform a different depth.

[0303] A discussion of model selection is provided.

[0304] Example 1

[0305] 1. NN filter models can use external information such as reconstruction parameters, partitioning or segmentation parameters, prediction parameters, boundary strength parameters, QP, etc. (typically, codec parameters associated with the video unit being filtered) as input to obtain attention. For example, NN filter models have attention mechanisms based on codec parameter inputs.

[0306] a. In one example, the convolutional layers of the NN filter model are used to extract features from external information or a combination of external and internal information. The internal information refers to features extracted within the network, while the external information refers to other information that cannot be obtained from the features within the network, such as encoding / decoding parameters associated with the video unit being filtered. At least one of these extracted features will be used as attention. As described above, Figure 14 Illustrations of attention mechanisms based on some examples are provided, where the operation performed by (multiple) convolutional layers is denoted as α, external information is denoted as E, internal information is denoted as I, and the attention received is denoted as A.

[0307] i. In one example, E is one of the reconstruction parameters, partitioning or segmentation parameters, prediction parameters, boundary strength parameters, QP, etc.

[0308] ii. In one example, E can be any combination of reconstruction parameters, partitioning or segmentation parameters, prediction parameters, boundary strength parameters, QP, etc.

[0309] iii. In one example, I is the (multiple) intermediate feature maps of the NN model, which will be recalibrated by the acquired attention.

[0310] iv. In one example, A = α(E).

[0311] v. In one example, A = α(E, I), where E and I are first concatenated and then fed into the convolutional layer.

[0312] vi. In one example, A = α(E), where E is the cascade of the reconstructed image and the segmented image, α is a two-layer convolutional neural network, and A is a single-channel feature map with the same spatial precision as the feature map to which A will be applied.

[0313] b. In one example, the acquired attention is used to recalibrate the intermediate feature maps. The intermediate feature maps of the NN model are denoted as G, where G∈R. N×W×H Where N, W, and H are the number of channels, width, and height, respectively. The attention obtained is represented by A, where A∈R. W×H This represents the attention received. In this paper, the symbol ∈ indicates that G is R. N×W×H Given an element in a set, where A is R W×H An element in a given set.

[0314] i. In one example, the process of applying attention can be written as:

[0315] φ i,j,k =G i,j,k ×A j,k ,1≤i≤N,1≤j≤W,1≤k≤H, where φ is the recalibrated feature map.

[0316] ii. In one example, the process of applying attention can be written as:

[0317] φ i,j,k =G i,j,k ×f(A j,k Let φ be the recalibrated feature map, 1≤i≤N, 1≤j≤W, 1≤k≤H, and f represent the mapping function applied to each element of the attention. The mapping function can be a sigmoid function, a hyperbolic tangent (e.g., tanh) function, etc. For example, the sigmoid function is a function with a characteristic "S" shape or sigmoid curve. The hyperbolic tangent function is an example of the sigmoid function.

[0318] 1. In one example, different A and / or different f can be used for different channels of the feature map.

[0319] iii. In one example, the process of applying attention can be written as:

[0320] φ i,j,k =G i,j,k ×f(A j,k )+Gi,j,k , 1≤i≤N, 1≤j≤W, 1≤k≤H, where φ is the recalibrated feature map, and f represents the mapping function applied to each element of the attention. The mapping function can be a sigmoid function, a hyperbolic tangent (e.g., tanh) function, etc.

[0321] 1. In one example, different A and / or different f can be used for different channels of the feature map.

[0322] iv. In one example, attention operations can be applied to a specific layer within the network.

[0323] 1. In one example, when the network contains residual blocks, the attention operation is applied only to the feature map from the last layer of each residual block.

[0324] Example 2

[0325] 2. In the second embodiment, the external attention mechanism described herein can be applied in various ways. For example, the proposed external attention mechanism can be used in any neural network model for video encoding and decoding, such as neural network-based intra / inter-frame prediction, neural network-based super-precision, neural network-based motion compensation, neural network-based reference generation, neural network-based fractional pixel interpolation, neural network-based loop / post-filtering, etc.

[0326] a. In one example, an external attention mechanism is used for loop filtering based on a neural network.

[0327] Example 3

[0328] 3. In the third embodiment, the depth of the NN model can be specified in various ways. For example, the depth of the NN model (e.g., the number of residual blocks) can be predefined or derived on the fly.

[0329] a. In one example, the derivation may depend on encoding / decoding information, such as temporal layer, strip / image type, reference image list information, characteristics of the video unit to which the NN model is applied, color components, and color format.

[0330] b. In one example, the depth can adapt to different video units within a single image / strip.

[0331] c. In one example, the depth can adapt to two video units in two different images / strips.

[0332] d. In one example, the depth of the NN model can be the same or different for different temporal layers, different types of stripes, different stripes, different video units in different images, or any video unit.

[0333] e. In one example, the depth of the NN model can be the same for video units in different temporal layers or different types of stripes.

[0334] f. In one example, use a different network depth for each temporal layer or each type of stripe.

[0335] i. In one example, the NN model is shallower for video units in higher temporal layers.

[0336] 1. In one example, the NN model for higher time-domain layers includes several residual blocks, such as Figure 12A As shown. For higher time-domain layers (e.g., relative to lower time-domain layers), the number of residual blocks is 8.

[0337] ii. In one example, the NN model has a smaller depth for video units in inter-frame stripes.

[0338] 1. In one example, such as Figure 12A As shown, the NN model for inter-frame striping includes several residual blocks. The number of residual blocks is 8.

[0339] iii. In one example, the NN model is quite deep for video units in an intra-frame strip.

[0340] 1. In one example, the intra-striped NN model includes several residual blocks, such as Figure 12A As shown. The number of residual blocks is 32.

[0341] g. In one example, the temporal layer is grouped into multiple subgroups. For each subgroup, a network depth is used.

[0342] Example 4

[0343] 4. In the fourth embodiment, the depth indication of the NN model for the video unit depth can be signaled to the decoder.

[0344] a. In one example, the signaling notification for at least one depth of the NN model can be sent in SEI messages, SPS, PPS, APS, VPS, image headers, strip headers, CTUs, etc.

[0345] b. In one example, the decoder can be signaled with information about multiple depths of the NN model.

[0346] c. In one example, the depth of the NN model for luminance and the depth of the NN model for chrominance can be signaled to the decoder separately.

[0347] The first exemplary embodiment is discussed.

[0348] This first exemplary embodiment proposes a loop filtering method based on a convolutional neural network, incorporating adaptive model selection. The proposed deep loop filter with adaptive model selection (DAM) is an evolution of the previous contribution JVET-V0100, introducing a new network architecture into the VTM-11.0+NewMCTF codebase. Compared to VTM-11.0+NewMCTF, the proposed method demonstrates reduced BD rates for {Y, Cb, Cr} under AI, RA, and LDB configurations.

[0349] In a first exemplary embodiment, a convolutional neural network (CNN) is used to construct a loop filter to reduce distortion generated during compression. The network structure is a modification of the network structure discussed in Yue Li, Li Zhang, and Kai Zhang's "Deep Loop Filter with Adaptive Model Selection," JVET-V0100. Similar to JVET-V0100, residual blocks are used as base blocks and stacked several times to construct the final network. As a further development of JVET-V0100, an external attention mechanism is introduced into this contribution, thereby increasing expressive power at a similar model size. Furthermore, to handle different types of content, the individual networks are trained for different types of stripes and quality levels.

[0350] First exemplary embodiment and Figure 12A and 12B The illustrated embodiments are relevant. To improve the architecture, the following is now introduced. Figure 16A and Figure 16B This includes external attention mechanisms. Figure 16A This is a schematic block diagram of the architecture of NN filtering methods based on various examples. Figure 16B This is shown according to various examples in Figure 16A A schematic block diagram of the construction of the attention residual block 1650 used in the architecture 1600.

[0351] Except for attention residual block 1650, the rest of architecture 1600 is identical to that in JVET-V0100. The computation process in attention module 1650 can be written as follows:

[0352] F_out=F_in×f(Rec,Pred)+F_in

[0353] Where F_in and F_out represent the input and output of attention module 1650, respectively. Rec and Pred represent reconstruction and prediction, respectively. In this example, f consists of two convolutional layers, with an activation function applied after the first convolutional layer. The goal of f is to generate a spatial attention map from external information and then recalibrate the feature map F_in.

[0354] In the technique proposed in the first exemplary embodiment, each strip or block can determine whether to apply a CNN-based filter. When a CNN-based filter is determined to be applied to a strip / block, it can be further determined which model from a candidate model list comprising three models can be selected. For this purpose, each model is trained with a QP in the range {17, 22, 27, 32, 37, 42}. Given a test QP for a sequence, denoted as q, the candidate model list includes models trained with a QP equal to {q, q-5, q-10}. The selection process is based on the rate-distortion cost on the encoder side. If necessary, signaling instructions for on / off control and the model index are communicated in the bitstream. Further details regarding model selection are provided below.

[0355] The granularity of filter determination and model selection depends on accuracy and QP. In the proposed scheme, given higher accuracy and a larger QP, determination and selection can be performed over a larger region.

[0356] The candidate model list differs at different temporal levels. For lower temporal levels, if the ratio of intra-frame codec samples is not less than a threshold, the third candidate is replaced by an intra-slice NN filter model. For higher temporal levels, the third candidate, corresponding to the model at q-10, is replaced by the model at q+5.

[0357] For full-frame configurations, model selection is disabled, but on / off control is still retained.

[0358] Open-source machine learning frameworks, such as PyTorch, are useful for performing inference of the proposed CNN filters in VTM. Table 1-1 provides network information for the inference phase, as proposed by S. Liu, A. Segall, E. Alshina, R.-L. Liao, “General Test Conditions and Evaluation Procedures for JVET Based on Neural Network Video Encoding and Decoding Techniques”, JVET-V2016.

[0359] Table 1-1 Network information during the inference phase of the NN-based video encoding / decoding tool test

[0360]

[0361] PyTorch can also be used as a training platform. DIV2K (from https: / / data.vision.ee.ethz.ch / cvl / DIV2K / , R. Timofte, E. Agustsson, S. Gu, J. Wu, A. Ignatov, LVGool) and BVI-DVC (from Ma, Di, Fan Zhang, David R. Bull. "BVI-DVC: Training Database for Deep Video Compression," arXiv preprint arXiv:2003.13552 (2020)). These datasets were used to train I-strip and B-strip CNN filters, respectively. Different CNN models were trained to adapt to different QP points. Table 1-2 provides network information during the training phase, as proposed in S. Liu, A. Segall, E. Alshina, R.-L. Liao, "General Test Conditions and Evaluation Procedures for JVET Based on Neural Network Video Encoding and Decoding Techniques," JVET-V2016.

[0362] Table 1-2. Network Information During the Training Phase of the NN-Based Video Encoding / Decoding Tool Test

[0363]

[0364] Based on the general test conditions proposed in JVET-V2016 by S. Liu, A. Segall, E. Alshina, and R.-L. Liao, “General Test Conditions and Evaluation Procedures for JVET Based on Neural Network Video Coding and Decoding Technology,” the proposed CNN-based loop filtering method was tested on the new MCTF (from https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / - / tags / VTM-11.0) in VTM-11.0+. The new MCTF patch is available at https: / / vcgit.hhi.fraunhofer.de / jvet-ahg-nnvc / nnvc-ctc / - / tree / master / Software%20Patches.

[0365] When ALF (and CCALF) are placed after the proposed CNN-based filtering, deblocking filtering and SAO are disabled. Test results are shown in Tables 1-3 to 1-5. In the AI ​​configuration, the proposed method reduces the BD rate of Y, Cb, and Cr by an average of 9.12%, 22.39%, and 22.60%, respectively. In the RA configuration, the proposed method reduces the BD rate of Y, Cb, and Cr by an average of 12.32%, 27.48%, and 27.22%, respectively. In the LDB configuration, the proposed method results in an average BD rate reduction of %, %, and % for Y, Cb, and Cr, respectively.

[0366] Table 1-3. Performance of the proposed method on VTM11.0+ new MCTF(RA)

[0367]

[0368] Table 1-4. Performance of the proposed method on VTM11.0+ new MCTF(LDB)

[0369]

[0370] Table 1-5. Performance of the proposed method on VTM11.0+ new MCTF(AI)

[0371]

[0372]

[0373] A first exemplary embodiment proposes a CNN-based loop filtering method. The proposed CNN-based filtering method demonstrates useful encoding / decoding gain.

[0374] Figure 17 This is a block diagram illustrating an example of a video processing system 1700, in which various techniques disclosed herein can be implemented. Various implementations may include some or all of the components of the video processing system 1700. The video processing system 1700 may include an input 1702 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8-bit or 10-bit multi-component pixel values, or may be received in a compressed or encoded format. Input 1702 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

[0375] Video processing system 1700 may include codec component 1704, which implements the various codec or encoding methods described herein. Codec component 1704 can reduce the average bit rate of the video from input 1702 to the output of codec component 1704 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 1704 can be stored or transmitted via connected communication (as shown in component 1706). The stored or transmitted bitstream (or codec) representation of the video received at input 1702 can be used by component 1708 to generate pixel values ​​or to send displayable video to display interface 1710. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations for the reverse codec results will be performed by the decoder.

[0376] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), DisplayPort, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The technologies described herein can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0377] Figure 18 This is a block diagram of a video processing apparatus 1800. Apparatus 1800 can be used to implement one or more methods described herein. Apparatus 1800 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 1800 may include one or more processors 1802, one or more memories 1804, and video processing hardware 1806. The processors(multiple) 1802 can be configured to implement one or more methods described in this application. The memories(multiple) 1804 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 1806 can be used to implement some of the techniques described in this application using hardware circuitry. In some embodiments, hardware 1806 can be implemented wholly or partially within processor 1802, such as a graphics coprocessor.

[0378] Figure 19 This is a block diagram illustrating an example of a video codec system 1900 that can utilize the techniques disclosed herein. Figure 19As shown, the video encoding / decoding system 1900 may include a source device 1910 and a destination device 1920. The source device 1910 generates encoded video data and may be referred to as a video encoding device. The destination device 1920 can decode the encoded video data generated by the source device 1910 and may be referred to as a video decoding device.

[0379] The source device 1910 may include a video source 1912, a video encoder 1914, and an input / output (I / O) interface 1916.

[0380] Video source 1912 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems for generating video data, or combinations of these sources. Video data may include one or more images. Video encoder 1914 encodes the video data from video source 1912 to generate a bitstream. The bitstream may include bit sequences that form a codec representation of the video data. The bitstream may include codec images and associated data. A codec image is a codec representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax structures. I / O interface 1916 may include a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to destination device 1920 via network 1930 through I / O interface 1916. Encoded video data may also be stored on storage medium / server 1940 for access by destination device 1920.

[0381] The target device 1920 may include an I / O interface 1926, a video decoder 1924, and a display device 1922.

[0382] I / O interface 1926 may include a receiver and / or a modem.

[0383] I / O interface 1926 can acquire encoded video data from source device 1910 or storage medium / server 1940. Video decoder 1924 can decode the encoded video data. Display device 1922 can display the decoded video data to the user. Display device 1922 can be integrated with target device 1920, or it can be external to target device 1920, which is configured to interface with an external display device.

[0384] The video encoder 1914 and video decoder 1924 can operate according to video compression standards, such as the High Efficiency Video Coding (HEVC) standard, the Universal Video Coding (VVC) standard, and other current and / or further standards.

[0385] Figure 20 This is a block diagram illustrating an example of a video encoder 2000. The video encoder 2000 can be in... Figure 19The video encoder 1914 in the video codec system 1900 described herein.

[0386] The video encoder 2000 can be configured to perform any or all of the techniques disclosed herein. Figure 20 In the example, the video encoder 2000 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 2000. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0387] The functional components of the video encoder 2000 may include a segmentation unit 2001, a prediction unit 2002 (which may include a mode selection unit 2003, a motion estimation unit 2004, a motion compensation unit 2005, and an intra-frame prediction unit 2006), a residual generation unit 2007, a transform unit 2008, a quantization unit 2009, an inverse quantization unit 2010, an inverse transform unit 2011, a reconstruction unit 2012, a buffer 2013, and an entropy encoding / decoding unit 2014.

[0388] In other examples, the video encoder 2000 may include more, fewer, or different functional components. In one example, the prediction unit 2002 may include an intra-block copy (IBC) unit. The IBC unit can perform prediction in IBC mode, where at least one reference picture is the picture containing the current video block.

[0389] Furthermore, some components, such as the motion estimation unit 2004 and the motion compensation unit 2005, can be highly integrated, but for interpretable purposes... Figure 20 The example is shown separately.

[0390] The segmentation unit 2001 can segment an image into one or more video blocks. Figure 19 The video encoder 1914 and video decoder 1924 in the program can support various video block sizes.

[0391] The mode selection unit 2003 can select one of the encoding / decoding modes (intra-frame or inter-frame, e.g., based on error results) and provides the resulting intra-frame or inter-frame encoded / decoded block to the residual generation unit 2007 to generate residual block data, and provides it to the reconstruction unit 2012 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 2003 can select a combination of intra-frame and inter-frame prediction (CIIP) modes, where prediction is based on inter-frame prediction signals and intra-frame prediction signals. In the case of inter-frame prediction, the mode selection unit 2003 can also select the resolution of the motion vector (e.g., sub-pixel or integer pixel precision) for the coded block.

[0392] To perform inter-frame prediction on the current video block, motion estimation unit 2004 can generate motion information for the current video block by comparing one or more reference frames from buffer 2013 with the current video block. Motion compensation unit 2005 can determine the predicted video block for the current video block based on motion information and decoded samples from images other than those associated with the current video block from buffer 2013.

[0393] The motion estimation unit 2004 and the motion compensation unit 2005 can perform different operations on the current video block, for example, depending on whether the current video block is in an I-band, P-band, or B-band.

[0394] In some examples, motion estimation unit 2004 may perform unidirectional prediction for the current video block, and may search reference images in list 0 or list 1 to find a reference video block for the current video block. Motion estimation unit 2004 may then generate a reference index indicating that the reference image in list 0 or list 1 contains the reference video block, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 2004 may output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 2005 may generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0395] In other examples, motion estimation unit 2004 can perform bidirectional prediction for the current video block. Motion estimation unit 2004 can search for a reference video block for the current video block in the reference images in list 0, and can also search for another reference video block for the current video block in the reference images in list 1. Motion estimation unit 2004 can then generate reference indices indicating the reference images in lists 0 and 1, which contain reference video blocks and motion vectors indicating the spatial displacement between the reference video blocks and the current video block. Motion estimation unit 2004 can output the reference index and motion vector of the current video block as motion information for the current video block. Motion compensation unit 2005 can generate a predicted video block for the current video block based on the reference video blocks indicated by the motion information of the current video block.

[0396] In some examples, the motion estimation unit 2004 can output a complete set of motion information for the decoder to use in the decoding process.

[0397] In some examples, the motion estimation unit 2004 may not output the complete set of motion information for the current video. Instead, the motion estimation unit 2004 may signal the motion information of the current video block by referring to the motion information of another video block. For example, the motion estimation unit 2004 may determine that the motion information of the current video block is sufficiently similar to the motion information of neighboring video blocks.

[0398] In one example, the motion estimation unit 2004 may indicate a value in the syntactic structure associated with the current video block that indicates to the video decoder 1924 that the current video block has the same motion information as another video block.

[0399] In another example, motion estimation unit 2004 can identify another video block and motion vector difference (MVD) within the syntactic structure associated with the current video block. The motion vector difference represents the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 1924 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0400] As discussed above, the video encoder 1914 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 1914 include advanced motion vector prediction (AMVP) and merge pattern signaling notification.

[0401] Intra-prediction unit 2006 can perform intra-prediction on the current video block. When intra-prediction unit 2006 performs intra-prediction on the current video block, intra-prediction unit 1706 can generate prediction data for the current video block based on decoded samples from other video blocks in the same frame. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0402] The residual generation unit 2007 can generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.

[0403] In other examples, residual data for the current video block may not exist, such as in skip mode, and the residual generation unit 2007 may not perform subtraction operations.

[0404] The transform processing unit 2008 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0405] After the transform processing unit 2008 generates a transform coefficient video block associated with the current video block, the quantization unit 2009 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0406] The dequantization unit 2010 and the inverse transform unit 2011 can apply dequantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 2012 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 2002 to generate a reconstructed video block associated with the current block and store it in the buffer 2013.

[0407] After the video block is reconstructed by the reconstruction unit 2012, a loop filtering operation can be performed to reduce video block artifacts in the video block.

[0408] The entropy encoding unit 2014 can receive data from other functional components of the video encoder 2000. When the entropy encoding unit 2014 receives data, it can perform one or more entropy encoding / decoding operations to generate entropy-encoded data and output a bitstream including the entropy-encoded data.

[0409] Figure 21 This is a block diagram illustrating an example of a video decoder 2100. The video decoder 2100 can be... Figure 19 The video decoder 1924 in the video codec system 1900 shown in the figure.

[0410] The video decoder 2100 can be configured to perform any or all of the technologies disclosed herein. Figure 21 In the example, video decoder 2100 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of video decoder 2100. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0411] exist Figure 21 In the example, video decoder 2100 includes an entropy decoding unit 2101, a motion compensation unit 2102, an intra-frame prediction unit 2103, an inverse quantization unit 2104, an inverse transform unit 2105, a reconstruction unit 2106, and a buffer 2107. In some examples, video decoder 2100 may perform functions typically associated with video encoder 1914 (e.g., Figure 19 The encoding channel (pass) is the opposite of the decoding channel.

[0412] Entropy decoding unit 2101 can retrieve the encoded bitstream. The encoded bitstream may include entropy-coded video data (e.g., encoded video data blocks). Entropy decoding unit 2101 can decode the entropy-coded video data, and based on the entropy-coded video data, motion compensation unit 2102 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. For example, motion compensation unit 2102 can determine this information by executing AMVP and merge modes.

[0413] The motion compensation unit 2102 can generate motion compensation blocks, possibly performing interpolation based on an interpolation filter. The identifier of the interpolation filter used at sub-pixel precision can be included in the syntax element.

[0414] The motion compensation unit 2102 can use the interpolation filter used by the video encoder 1914 during the encoding of the video block to calculate the interpolated values ​​of the sub-integer pixels of the reference block. The motion compensation unit 2102 can determine the interpolation filter used by the video encoder 1914 based on the received syntax information and use the interpolation filter to generate the prediction block.

[0415] The motion compensation unit 2102 can use some syntax information to determine the size of the blocks used to encode the frames and / or stripes of the encoded video sequence, segmentation information describing how each macroblock of the picture of the encoded video sequence is segmented, a mode indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information of the decoded encoded video sequence.

[0416] Intra-prediction unit 2103 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 2104 performs inverse quantization, i.e., dequantization, on the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 2101. Inverse transform unit 2105 applies an inverse transform.

[0417] The reconstruction unit 2106 can add the residual block to the corresponding prediction block generated by the motion compensation unit 2102 or the intra-frame prediction unit 2103 to form a decoded block. If necessary, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 2107, which provides a reference block for subsequent motion compensation / intra-frame prediction and also generates decoded video for display device rendering.

[0418] Figure 22This is a video data encoding / decoding method 2200 according to an embodiment of the present invention. Method 2200 can be executed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 2200 can be implemented to provide an NN filter model with a configurable depth (e.g., the number of residual blocks of a filter model). For example, a first NN filter model has a first depth, and a second NN filter model has a second depth. Therefore, selecting either the first NN filter model or the second NN filter model makes the depth of the NN filter model configurable (e.g., a first depth or a second depth). Such an NN filter model allows different filter model depths to be applied to different video units, determined in a predefined or on-the-fly manner. As described above, in other examples, more than two filter models (e.g., more than two depths) can be implemented by the applied NN filter(s).

[0419] In block 2202, the encoding / decoding device applies a neural network (NN) filter to the unfiltered samples of the video unit to produce filtered samples. The NN filter is based on an NN filter model with a configurable depth, where the depth is the number of residual blocks in the NN filter model. In one embodiment, an unfiltered sample is a sample (or pixel) that has not undergone any filtering process or has not been sufficiently filtered. For example, an unfiltered sample has not been processed by any NN filter. As another example, an unfiltered sample has not been processed by an NN filter, an adaptive loop filter (ALF), a deblocking filter (DF), a sample adaptive offset (SAO) filter, or a combination thereof.

[0420] In block 2204, the codec unit performs conversion between the video media file and the bitstream based on the generated filtered samples.

[0421] When implemented in an encoder, the conversion involves receiving a media file (e.g., a video unit) and encoding the filtered samples into a bitstream. When implemented in a decoder, the conversion involves receiving a bitstream including the filtered samples and decoding the bitstream to obtain the filtered samples.

[0422] In one embodiment, method 2200 may utilize or combine one or more features or processes of other methods disclosed herein.

[0423] In one example, a non-transitory computer-readable medium stores a bitstream of video generated by a method (e.g., all or part of method 2200) performed by a video processing apparatus (e.g., video processing apparatus 1800 described above). For example, a bitstream can be generated by applying an NN filter to unfiltered samples of video cells to produce filtered samples. As described above, the NN filter is based on an NN filter model generated using a Quality Level Indicator (QI) input, and the bitstream is generated based on the filtered samples.

[0424] The following provides a list of preferred solutions for some embodiments.

[0425] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Example 1).

[0426] 1. A video processing method comprising: performing a rule-based conversion between a video comprising video blocks and a bitstream of the video, wherein the conversion comprises filtering at least some samples of the video blocks using a neural network (NN) filter, the neural network filter using one or more NN models, and wherein the rule specifies that the one or more NN models are equipped with an attention mechanism based on external information of the video.

[0427] 2. The method of claim 1, wherein the rule specifies that the convolutional layers of the NN are used to extract features from the external information, and the extracted features E are used as attention A in the one or more NN models.

[0428] 3. The method of claim 2, wherein the rule specifies the use of internal information I to determine A.

[0429] 4. The method of claim 3, wherein I comprises intermediate feature maps of one or more NN models.

[0430] 5. The method according to any one of claims 2-4, wherein the rule stipulates that A is obtained only from E, without using I.

[0431] 6. The method according to any one of claims 2-4, wherein the rule specifies that A is obtained by concatenating E and I.

[0432] 7. The method according to any one of claims 2-6, wherein the rule specifies the use of a two-layer convolutional neural network to obtain A.

[0433] 8. The method according to any one of claims 2-7, wherein the transformation comprises: recalibrating the intermediate feature map of the video using attention A.

[0434] 9. The method of claim 8, wherein the intermediate feature maps of the one or more NN models are represented as G∈R N×W×H Where N, W, and H are the number of channels, width, and height, respectively, and the attention A is represented as A∈R W×H .

[0435] 10. The method according to claim 9, wherein the process of applying attention is: φ i,j,k =G i,j,k ×Aj,k ,1≤i≤N,1≤j≤W,1≤k≤H, where φ is the recalibrated feature map.

[0436] 11. The method according to claim 10, wherein the process of applying attention is: φ i,j,k =G i,j,k ×f(A j,k ), 1≤i≤N, 1≤j≤W, 1≤k≤H, where φ is the recalibrated feature map and f represents the mapping function applied to each element of the attention.

[0437] 12. The method according to claim 11, wherein the function is a sigmoid function or a tanh function.

[0438] 13. The method according to claim 11 or 12, wherein different A and / or different f are used for different channels of the feature map.

[0439] 14. The method of claim 10, wherein the applied attention is: φ i,j,k =G i,j,k ×f(A j ,k )+G i,j,k ,1≤i≤N,1≤j≤W,1≤k≤H,where φ is the recalibrated feature map, and f represents the mapping function applied to each element of the attention.

[0440] 15. The method of claim 14, wherein different A and / or different f can be used for feature maps of different channels.

[0441] 16. The method according to claims 10-15, wherein the attention operation can be applied to a specified layer within the NN.

[0442] 17. The method according to any one of claims 10-16, wherein the rule specifies that when the NN contains residual blocks, the attention operation is applied only to the feature map from the last layer of each residual block.

[0443] 18. The method of claim 14, wherein the function is a sigmoid function or a tanh function.

[0444] 19. The method according to any one of claims 1-18, wherein the external information includes the video block partitioning scheme.

[0445] 20. The method according to claims 1-18, wherein the external information includes the prediction mode of the video block.

[0446] 21. The method according to any one of claims 1 to 18, wherein the external information includes quantization parameters associated with the video block.

[0447] 22. The method according to any one of claims 1-21, wherein the external information includes boundary strength parameters of the boundaries of the video block.

[0448] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Examples 3 and 4).

[0449] 23. The method according to any one of claims 1-21, wherein the depth of the one or more NN models is determined according to a depth rule.

[0450] 24. The method of claim 23, wherein the depth rule defines the depth as having a predetermined value.

[0451] 25. The method of claim 23, wherein the depth can be derived in an instantaneous manner.

[0452] 26. The method according to any one of claims 23-25, wherein the depth corresponds to the number of residual blocks.

[0453] 27. The method according to any one of claims 23-26, wherein the depth rule is based on the encoding and decoding information of the video block.

[0454] 28. The method according to any one of claims 23-27, wherein the depth rule is adaptive.

[0455] 29. The method according to any one of claims 23-28, wherein the depth rule causes the depth value to vary between different images.

[0456] 30. The method according to any one of claims 23-29, wherein one or more syntax elements indicating depth are included in the bitstream.

[0457] 31. The method according to any one of claims 23-30, wherein one or more syntax elements indicating depth are indicated to the decoder.

[0458] 32. The method according to any one of claims 30 to 31, wherein the one or more syntax elements comprise syntax elements indicating the depth of the luminance component and syntax elements indicating the depth of the chrominance component.

[0459] 33. The method according to any one of claims 1-32, wherein the conversion comprises generating a bitstream from the video.

[0460] 34. The method according to any one of claims 1-32, wherein the conversion comprises generating video from a bitstream.

[0461] 35. A video decoding apparatus, comprising a processor configured to implement the method of one or more of claims 1 to 34.

[0462] 36. A video encoding apparatus, comprising a processor configured to implement the method of one or more of claims 1 to 34.

[0463] 37. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 34.

[0464] 38. A computer-readable medium having a bitstream stored thereon, the bitstream being generated by the method of any one of claims 1 to 34.

[0465] 39. A method for generating a bitstream, comprising: generating a bitstream using one or more of claims 1-34, and writing the bitstream into a computer-readable medium.

[0466] 40. The methods, apparatus or systems described in this document.

[0467] The other solutions, examples, embodiments, modules, and functional operations disclosed in this document can be implemented in digital electronic circuits or computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more computer program instruction modules encoded on a computer-readable medium, which are executed or controlled by a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of substances affecting machine-readable propagation signals, or a combination thereof. The term "data processing apparatus" encompasses all means, devices, and machines that process data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof. The transmitted signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0468] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple harmonizing files (e.g., files storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected via a communication network.

[0469] The processes and logic described in this document can be executed by one or more programmable processors to execute one or more computer programs, thereby performing functions by manipulating input data and generating output. The processes and logic can also be executed by dedicated logic circuits, and can be implemented as dedicated logic circuits, such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits).

[0470] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into special-purpose logic circuitry.

[0471] Although this patent document contains numerous details, these details should not be construed as limiting any invention or the scope of the claims, but rather as a description of features that may be specific to particular embodiments of a particular invention. Certain features described in this patent document in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0472] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all shown operations to achieve the desired effect. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0473] Only some implementation methods and examples are described, and other implementation methods, enhancements and variations can be made based on the content described and shown in this patent document.

[0474] Although this patent document contains numerous details, these details should not be construed as limiting any invention or the scope of the claims, but rather as a description of features that may be specific to particular embodiments of a particular invention. Certain features described in this patent document in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0475] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all shown operations to achieve the desired effect. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0476] Only some implementation methods and examples are described, and other implementation methods, enhancements and variations can be made based on the content described and shown in this patent document.

Claims

1. A method implemented by a video encoding / decoding device, comprising: A neural network (NN) filter is applied to unfiltered samples of a video unit to generate filtered samples. The NN filter is based on a first NN filter model with a first depth or a second NN filter model with a second depth, wherein the depth includes the number of residual blocks of the corresponding NN filter model, and wherein the second depth differs from the first depth; wherein the depth signaling of the NN filter model to be used is communicated to the decoder; and The conversion between video media files and bitstreams is performed based on the filtered samples.

2. The method according to claim 1, wherein, The video unit is a first video unit, wherein when the NN filter is applied to the first video unit, the NN filter uses the first NN filter model, and wherein when the NN filter is applied to the second video unit, the NN filter uses the second NN filter model.

3. The method according to claim 1 further includes determining whether to use the first NN filter model or the second NN filter model based on the time layer of the video unit, the type of the strip or image containing the video unit, the list of reference images matching the video unit, the color components of the video unit, the color format of the video unit, and combinations thereof.

4. The method according to any one of claims 1-3, wherein, For a first video unit in an image or strip, the NN filter uses the first NN filter model, and for a second video unit in an image or strip, the NN filter uses the second NN filter model.

5. The method according to any one of claims 1-3, wherein, For a first video unit in a first image or a first strip, the NN filter uses the first NN filter model, and for a second video unit in a second image or a second strip, the NN filter uses the second NN filter model.

6. The method according to any one of claims 1-3, wherein, For a first video unit in a first temporal layer or a first type of stripe, the NN filter uses the first NN filter model, and for a second video unit in a second temporal layer or a second type of stripe, the NN filter uses the second NN filter model.

7. The method according to claim 6, wherein, For the third video unit in the third time-domain layer, the NN filter uses the first NN filter model, wherein the first time-domain layer and the third time-domain layer are in the same subgroup of the time-domain layer, and wherein the second time-domain layer is in a different subgroup of the time-domain layer.

8. The method according to any one of claims 1-3, wherein, The determination of whether to use the first NN filter model or the second NN filter model is based on whether the video unit is a codec tree unit (CTU), codec tree block (CTB), CTU row, CTB row, strip, slice, picture, sequence, or sub-picture.

9. The method according to any one of claims 1-3, wherein, For a first video unit in a first temporal layer, the NN filter uses the first NN filter model; for a second video unit in a second temporal layer, the NN filter uses the second NN filter model; and the first temporal layer is a higher temporal layer than the second temporal layer.

10. The method according to claim 9, wherein, The first depth is 8.

11. The method according to any one of claims 1-3, wherein, For a first video unit in an inter-frame stripe, the NN filter uses the first NN filter model, and for a second video unit in an intra-frame stripe, the NN filter uses the second NN filter model.

12. The method according to claim 11, wherein, The first depth is 8, and the second depth is 32.

13. The method according to any one of claims 1-3, further comprising: Use one of the following groups to signal the depth of the NN filter model to be used to the decoder: Supplemental Enhancement Information (SEI) message, Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Video Parameter Set (VPS), Picture header, and Strip header.

14. The method according to any one of claims 1-3, further comprising sending the depth of the first NN filter model to the decoder of the first video unit, and sending the depth of the second NN filter model to the decoder of the second video unit.

15. The method according to any one of claims 1-3, wherein, The unfiltered samples include a luminance component and a chrominance component, and the method further includes signaling to the decoder to inform the decoder of the different depths of the luminance component and the chrominance component.

16. The method according to any one of claims 1-3, wherein, The NN filter is selected from one or more of the group consisting of an adaptive loop filter, a deblocking filter, and a sample adaptive offset filter.

17. The method according to any one of claims 1-3, wherein, The conversion includes generating the bitstream from the video media file.

18. The method according to any one of claims 1-3, wherein, The conversion includes parsing the bitstream to obtain the video media file.

19. An apparatus for encoding and decoding video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: A neural network (NN) filter is applied to the unfiltered samples of a video unit to generate filtered samples. The NN filter is based on either a first NN filter model with a first depth or a second NN filter model with a second depth. The depth includes the number of residual blocks of the corresponding NN filter model, and wherein the second depth is different from the first depth; wherein the depth signaling of the NN filter model to be used is communicated to the decoder; and The conversion between video media files and bitstreams is performed based on the filtered samples.

20. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: A neural network (NN) filter is applied to the unfiltered samples of a video unit to generate filtered samples. The NN filter is based on either a first NN filter model with a first depth or a second NN filter model with a second depth. The depth includes the number of residual blocks of the corresponding NN filter model, and wherein the second depth is different from the first depth; wherein the depth signaling of the NN filter model to be used is communicated to the decoder; and The conversion between video media files and bitstreams is performed based on the filtered samples.

21. A method for storing a bitstream of video, comprising: A neural network (NN) filter is applied to the unfiltered samples of a video unit to generate filtered samples. The NN filter is based on either a first NN filter model with a first depth or a second NN filter model with a second depth. The depth includes the number of residual blocks of the corresponding NN filter model, and wherein the second depth is different from the first depth; wherein the depth signaling of the NN filter model to be used is communicated to the decoder; The bitstream is generated based on the filtered samples; and The bit stream is stored in a non-transitory computer-readable recording medium.

22. A video processing apparatus, comprising a processor, wherein, The processor is configured to perform the method as described in any one of claims 1-18.

23. A computer-readable medium having instructions stored thereon, wherein, When executed, the instructions cause the processor to perform the method according to any one of claims 1-18.

Images