Unified neural network loop filter signaling notification

Abstract

A unified neural network in-loop filter signaling and method implemented by a video encoding device includes applying a neural network (NN) filter to unfiltered samples of a video unit to generate filtered samples, where the NN filter is applied based on a syntax element of the video unit. The method also includes converting between a video media file and a bitstream based on the generated filtered samples.

Description

[0001] Cross-references to related applications

[0002] This patent application claims priority to International Application No. PCT / CN2021 / 087615 filed by Beijing ByteDance Network Technology Co., Ltd. on April 15, 2021; International Application No. PCT / CN2021 / 087915 filed by Beijing ByteDance Network Technology Co., Ltd. on April 16, 2021; U.S. Provisional Patent Application No. 63 / 176,871 filed by Lemon Inc. on April 19, 2021; and International Application No. PCT / CN2021 / 088480 filed by Beijing ByteDance Network Technology Co., Ltd. on April 20, 2021, all of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure generally relates to image and video encoding and decoding. Background Technology

[0004] Despite advancements in video compression, digital video still accounts for the largest share of bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required for digital video usage is expected to continue to grow. Summary of the Invention

[0005] The disclosed aspects / embodiments provide one or more neural network (NN) filter models trained as part of a loop filtering technique or a filtering technique used in a post-processing stage to reduce distortion caused during compression. Furthermore, samples with different characteristics are processed using different NN filter models. Additionally, the presence (e.g., application) of NN filter models can be controlled by different levels of syntax elements. For example, syntax elements(s) indicating whether an NN filter is applied may be at a first level (e.g., in the sequence header and / or sequence parameter set (SPS) of a video unit). Syntax elements(s) indicating whether an NN filter is applied may also be at a second level (e.g., in the picture header, picture parameter set (PPS), and / or strip header of a video unit). Further, syntax elements(s) indicating whether an NN filter is applied may be at a third level (e.g., syntax elements indicating the application of a patch, CTU, CTB, block, sub-picture, tile, or region of a video unit).

[0006] The first aspect relates to a method implemented by an encoding / decoding device. The method includes: applying a neural network (NN) filter to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is applied based on syntax elements of the video unit. The method further includes converting between a video media file and a bitstream based on the generated filtered samples.

[0007] Optionally, in any of the foregoing aspects, another embodiment of the aspect provides that the syntax element indicates at least one selected from the group consisting of: whether the NN filter is enabled, the number of NN filters to be applied, and the type of NN filter to be applied.

[0008] Optionally, in any of the foregoing aspects, another embodiment of this aspect provides a first level including a sequence level, and the syntax elements indicated in the first level are indicated in the sequence parameter set (SPS) and / or sequence header of the video unit; a second level including a picture level, and the syntax elements indicated in the second level are indicated in the picture header, picture parameter set (PPS) and / or stripe header of the video unit; and a third level including a subpicture level, and the syntax elements indicated in the third level are indications for a small block of the video unit, a codec tree unit (CTU) of the video unit, a codec tree block (CTB) of the video unit, a block of the video unit, a subpicture of the video unit, a slice of the video unit, a stripe of the video unit, or a region of the video unit.

[0009] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides that the syntax element is a first syntax element at the first level, the first syntax element indicating whether to adaptively select an NN filter at the second level to apply to the picture or stripe of the video unit.

[0010] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides that the syntax element is a first syntax element at the second level, the first syntax element being conditionally applied based on a second syntax element at the first level, wherein the NN filter is applied at the second level based on a flag that the second syntax element is true, and wherein the NN filter is not applied based on a flag that the second syntax element is false.

[0011] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides that the syntax element is a first syntax element at the second level, the first syntax element indicating whether to adaptively select an NN filter at the third level to apply to a sub-picture of the video unit, or the first syntax element indicating whether to control the use of the NN filter at the third level.

[0012] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that the syntax element is a first syntax element at the second level, and the first syntax element indicates whether the NN filter is adaptively selected at the second level, used at the second level, or applied at the second level; and wherein the first syntax element is signaled based on an indication of adaptively selecting the NN filter at the second level or an indication that the number of the NN filters is greater than 1.

[0013] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that the syntax element is a first syntax element at the third level, and the first syntax element is conditionally applied based on a second syntax element at the first level and / or a third syntax element at the second level, wherein the first syntax element is context coded, wherein based on a flag that one of the second syntax element and the third syntax element is true, the NN filter is applied at the third level based on the first syntax element, and wherein based on that one of the second syntax element and the third syntax element is false, the NN filter is not applied.

[0014] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that the syntax element is a first syntax element at the third level, and the first syntax element is signaled based on an indication of adaptively selecting the NN filter at the third level or an indication that the number of the NN filters is greater than 1.

[0015] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that the syntax element is signaled in response to enabling the NN filter for a picture or slice of the video unit, and wherein the NN filter is one of a plurality of (T) NN filters, and wherein the syntax element comprises an index (k).

[0016] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that the k-th NN filter is applied at the second level of the video unit based on the index k>=0 and k<T.

[0017] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that the NN filter is adaptively selected at the third level based on the index k>=T.

[0018] Optionally, in any of the preceding aspects, another embodiment of this aspect provides that the index k is limited to the range from 0 to (T–1).

[0019] Optionally, in any of the foregoing aspects, another embodiment of the aspect provides that the syntax element is encoded and decoded based on a context model selected based on the number of allowed NN filters, wherein the filter model index of the color component of the video unit is configured to specify one of K context models, and wherein one of the K context models is specified as the minimum of K-1 and binIdx, where binIdx is the index of the binary bit to be encoded and decoded.

[0020] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides encoding and decoding of the filter model indexes of the first and second color components of the video unit using the same set of context.

[0021] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides that the filter model index of the first color component of the video unit is encoded and decoded using a different set of contexts than the filter model index of the second color component of the video unit.

[0022] Optionally, in any of the foregoing aspects, another embodiment of that aspect provides signaling notification of the syntax element using context codec or bypass codec, or binarying the syntax element using fixed-length codec, unary codec, truncated unary codec, signaling notification unary codec, signed truncated unary codec, truncated binary codec, or exponential Golomb codec.

[0023] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides a conversion that includes generating the bitstream based on the video media file.

[0024] Alternatively, in any of the foregoing aspects, another embodiment of that aspect provides a conversion that includes parsing the bitstream to obtain the video media file.

[0025] The second aspect relates to an apparatus for encoding and decoding video data, including a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: apply a neural network (NN) filter to unfiltered samples of video units to generate filtered samples, wherein the NN filter is applied based on syntax elements of the video units; and convert between a video media file and a bitstream based on the generated filtered samples.

[0026] The third aspect relates to a non-transitory computer-readable medium storing a bitstream of video, the bitstream being generated by a method performed by a video processing apparatus. The method performed by the video processing apparatus includes: applying a neural network (NN) filter to unfiltered samples of video units to generate filtered samples, wherein the NN filter is applied based on syntax elements of the video units; and generating the bitstream based on the generated filtered samples.

[0027] For clarity, any of the foregoing embodiments may be combined with any one or more other foregoing embodiments to create new embodiments within the scope of this disclosure.

[0028] These and other features will become clearer from the following detailed description taken in conjunction with the accompanying drawings and claims. Attached Figure Description

[0029] For a more complete understanding of this disclosure, please refer to the following brief description in conjunction with the accompanying drawings and detailed description, wherein the same reference numerals represent the same elements.

[0030] Figure 1 This is an example of raster scan strip segmentation of an image.

[0031] Figure 2 This is an example of rectangular strip segmentation in an image.

[0032] Figure 3 These are examples of how an image is divided into slices, tiles, and rectangular strips.

[0033] Figure 4A This is an example of a codec tree block (CTB) that spans the bottom image boundary.

[0034] Figure 4B This is an example of a CTB that spans the right edge of the image.

[0035] Figure 4C This is an example of a CTB that spans the bottom right edge of the image.

[0036] Figure 5 This is an example of an encoder block diagram.

[0037] Figure 6 This is a diagram of the sample points within an 8×8 sample point block.

[0038] Figure 7 These are examples of pixels involved in filter on / off decisions and strong / weak filter selection.

[0039] Figure 8 Four one-dimensional (1-D) orientation schemes for EO sample point classification are shown.

[0040] Figure 9An example of the shape of an adaptive loop filter (GALF) based on geometric transformation is shown.

[0041] Figure 10 An example of relative coordinates for a 5×5 diamond filter support is shown.

[0042] Figure 11 Another example of relative coordinates for 5×5 rhombus filter support is shown.

[0043] Figure 12A This is an example architecture of the proposed CNN filter.

[0044] Figure 12B This is an example of constructing a ResBlock.

[0045] Figure 13 Examples of the process of generating filtered samples based on a neural network filter model with a received quality level indicator as input, based on various examples.

[0046] Figure 14 Examples of quality level indicators that are expanded (spanned) or tiled (tiles) into a two-dimensional array, based on various examples.

[0047] Figure 15 This is an example of feeding quality level indicators into multiple fully connected layers (e.g., fully connected layers of NN filters) to expand them into a one-dimensional vector, based on various examples.

[0048] Figure 16 Examples of video bitstreams based on various examples are shown.

[0049] Figure 17 These are example architectures of CNN filters based on various examples.

[0050] Figure 18 This is a block diagram illustrating an example video processing system.

[0051] Figure 19 This is a block diagram of a video processing device.

[0052] Figure 20 This is a block diagram illustrating an example video codec system.

[0053] Figure 21 This is a block diagram showing an example of a video encoder.

[0054] Figure 22 This is a block diagram illustrating an example of a video decoder.

[0055] Figure 23 This is a method for encoding and decoding video data according to embodiments of the present disclosure. Detailed Implementation

[0056] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the illustrative embodiments, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the full scope of the appended claims and their equivalents.

[0057] The Universal Video Codec (VVC), also known as H.266, is used in some descriptions for ease of understanding only and not to limit the scope of the disclosed technology. Therefore, the techniques described herein are also applicable to other video codec protocols and designs.

[0058] This specification relates to video encoding and decoding. Specifically, this description relates to loop filters in image / video encoding and decoding. The disclosed examples can be applied individually or in various combinations to video bitstreams encoded and decoded using existing video encoding and decoding standards (e.g., VVC, High Efficiency Video Codec (HEVC)) or pending standards (e.g., AVS3). The disclosed examples can also be applied to future video encoding and decoding standards, future video codecs, or as post-processing methods outside the encoding / decoding process.

[0059] Video codec standards have primarily evolved through the development of well-known standards from the International Telecommunication Union-Telecommunication (ITU-T) and the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). ITU-T developed H.261 and H.263, while ISO / IEC developed Moving Picture Experts Group (MPEG)-1 and MPEG-4 Vision. The two organizations jointly developed the H.262 / MPEG-2 video, H.264 / MPEG-4 Advanced Video Coding (AVC), and H.265 / High-Efficiency Video Coding (HEVC) standards.

[0060] Since H.262, video codec standards have been based on a hybrid video codec architecture, employing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Video Codec Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to reference software called the Joint Exploration Model (JEM).

[0061] In April 2018, a Joint Video Experts Team (JVET) was created between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) to research the VVC standard, aiming for a 50% bitrate reduction compared to HEVC. VVC version 1 was completed in July 2020.

[0062] Color spaces and chromaticity subsampling are discussed. A color space, also known as a color model (or color system), is an abstract mathematical model that simply describes a range of colors as tuples of numbers, typically 3 or 4 values ​​or color components (e.g., red, green, and blue (RGB)). Fundamentally, a color space is an explanation of a coordinate system and its subspaces.

[0063] For video compression, the most commonly used color spaces are YCbCr and RGB. YCbCr, Y'CbCr, or YPb / CbPr / Cr are also written as YC. B C R or Y'C B C R RGB is a color space family used as part of the color image pipeline in video and digital photography systems. Y' is the luminance component, while CB and CR are the blue and red difference chromaticity components. Y' (with an apostrophe) is different from Y, which is luminance, meaning that light intensity is based on gamma-corrected RGB primary color non-linear encoding.

[0064] Chromaticity subsampling is a practice that utilizes the fact that the human visual system is less sensitive to color differences than to brightness, and encodes images by applying lower precision to chromaticity information than to brightness information.

[0065] For 4:4:4 chroma subsampling, each of the three Y'CbCr components has the same sampling rate, therefore there is no chroma subsampling. This scheme is sometimes used in high-end film scanners and film post-production.

[0066] For 4:2:2 chroma subsampling, the two chroma components are sampled at half the luminance sampling rate: the horizontal chroma precision is halved. This reduces the bandwidth of the uncompressed video signal by one-third, but with almost no visual difference.

[0067] For 4:2:0 chroma subsampling, the horizontal sampling is doubled compared to 4:1:1, but the vertical precision is halved because the Cb and Cr channels are sampled only on each alternating line in this scheme. Therefore, the data rate is the same. Cb and Cr are subsampled by a factor of 2 in the horizontal and vertical directions, respectively. There are three variants of the 4:2:0 scheme with different horizontal and vertical addressing.

[0068] In MPEG-2, Cb and Cr are co-addressed horizontally. Cb and Cr are also addressed between pixels in the vertical direction (gap addressing). In Joint Picture Experts Group (JPEG) / JPEG File Exchange Format (JFIF), H.261, and MPEG-1, Cb and Cr are gap-addressed, located between alternating luma samples. In 4:2:0 DV, Cb and Cr are co-addressed horizontally. In the vertical direction, they are co-addressed on alternating lines.

[0069] The definition of a video unit is provided. An image is divided into one or more tile rows and one or more tile columns. A tile is a sequence of codec tree units (CTUs) that cover a rectangular area of ​​the image. A tile is divided into one or more bricks, each brick consisting of multiple CTU rows within the tile. A tile that is not divided into multiple bricks is also called a brick. However, a brick that is a proper subset of a tile is not called a tile. A stripe includes multiple tiles of an image or multiple bricks of a tile.

[0070] Two stripe modes are supported: raster scan stripe mode and rectangular stripe mode. In raster scan stripe mode, the stripe consists of a sequence of slices from a raster scan of the image. In rectangular stripe mode, the stripe consists of multiple tiles from the image, which together form a rectangular area of ​​the image. The tiles within the rectangular stripe are arranged according to the raster scan order of the stripe.

[0071] Figure 1 An example of raster scan strip segmentation for image 100, where the image is divided into twelve slices 102 and three raster scan strips 104. As shown, each slice 102 and strip 104 includes multiple CTUs 106.

[0072] Figure 2 This is an example of rectangular strip segmentation of image 200 according to the VVC specification, where the image is divided into twenty-four slices 202 (six slice columns 203 and four slice rows 205) and nine rectangular strips 204. As shown in the figure, each slice 202 and strip 204 includes multiple CTUs 206.

[0073] Figure 3 This is an example of dividing an image 300 into slices, tiles, and rectangular strips according to the VVC specification. The image is divided into four slices 302 (two slice columns 303 and two slice rows 305), eleven tiles 304 (the top left slice contains one tile, the top right slice contains five tiles, the bottom left slice contains two tiles, and the bottom right slice contains three tiles) and four rectangular strips 306.

[0074] The CTU and Codec Tree Block (CTB) sizes are discussed. In VVC, the CTU size, notified by the syntax element log2_ctu_size_minus2 in the Sequence Parameter Set (SPS) signaling, can be as small as 4×4. The Sequence Parameter Set Raw Byte Sequence Payload (RBSP) syntax is as follows.

[0075]

[0076]

[0077]

[0078] log2_ctu_size_minus2 plus 2 specifies the size of the luminance codec tree block for each CTU.

[0079] log2_min_luma_coding_block_size_minus2 plus 2 specifies the minimum luma encoding / decoding block size.

[0080] The variables CtbLog2SizeY, CtbSizeY, MinCbLog2SizeY, MinCbSizeY, MinTbLog2SizeY, MaxTbLog2SizeY, MinTbSizeY, MaxTbSizeY, PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC, and PicHeightInSamplesC are derived as follows:

[0081] CtbLog2SizeY=log2_ctu_size_minus2+2 (7-9)

[0082] CtbSizeY = 1 <CtbLog2SizeY (7-10)

[0083] MinCbLog2SizeY=log2_min_luma_coding_block_size_minus2+2 (7-11)

[0084] MinCbSizeY=1< <MinCbLog2SizeY (7-12)

[0085] MinTbLog2SizeY=2 (7-13)

[0086] MaxTbLog2SizeY=6 (7-14)

[0087] MinTbSizeY=1<<MinTbLog2SizeY (7-15)

[0088] MaxTbSizeY=1<<MaxTbLog2SizeY (7-16)

[0089] PicWidthInCtbsY=Ceil(pic_width_in_luma_samples÷CtbSizeY) (7-17)

[0090] PicHeightInCtbsY=Ceil(pic_height_in_luma_samples÷CtbSizeY) (7-18)

[0091] PicSizeInCtbsY=PicWidthInCtbsY*PicHeightInCtbsY (7-19)

[0092] PicWidthInMinCbsY=pic_width_in_luma_samples / MinCbSizeY (7-20)

[0093] PicHeightInMinCbsY=pic_height_in_luma_samples / MinCbSizeY (7-21)

[0094] PicSizeInMinCbsY=PicWidthInMinCbsY*PicHeightInMinCbsY (7-22)

[0095] PicSizeInSamplesY=pic_width_in_luma_samples*pic_height_in_luma_samples (7-23)

[0096] PicWidthInSamplesC=pic_width_in_luma_samples / SubWidthC (7-24)

[0097] PicHeightInSamplesC=pic_height_in_luma_samples / SubHeightC (7-25)

[0098] Figure 4A is an example of a CTB that crosses the bottom picture boundary. Figure 4B is an example of a CTB that crosses the right picture boundary. Figure 4C is an example of a CTB that crosses the bottom-right picture boundary. In Figures 4A-4C , there are respectively K=M, L<N; K<M, L=N; K<M, L<N.

[0099] reference Figures 4A-4C discusses the CTU in picture 400. Assume that the CTB / largest coding unit (LCU) size indicated by M×N (usually M is equal to N, as defined in HEVC / VVC), and for a CTB located at the boundary of a picture (or slice or tile or other types, taking the picture boundary as an example), K×L samples are within the picture boundary, where K<M or L<N. For Figures 4A-4C those CTBs 402 depicted in, the CTB size is still equal to M x N, however, the bottom boundary / right boundary of the CTB is outside the picture 400.

[0100] discusses the encoding and decoding process of a typical video encoder / decoder (also known as codec). Figure 5 is an example of an encoder block diagram of VVC, which includes three in-loop filtering blocks: a deblocking filter (DF), sample adaptive offset (SAO) and adaptive loop filter (ALF). Unlike DF which uses predefined filters, SAO and ALF utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding offsets and applying finite impulse response (FIR) filters respectively, with the help of encoding and decoding side information that signals the offsets and filter coefficients. ALF is located at the last processing stage of each picture, and can be regarded as a tool that attempts to capture and repair artifacts generated in previous stages.

[0101] Figure 5 is a schematic diagram of an encoder 500. The encoder 500 is suitable for implementing VVC technologies. The encoder 500 includes three in-loop filters, namely a deblocking filter (DF) 502, a sample adaptive offset (SAO) 504 and an ALF 506. Unlike the DF 502 which uses predefined filters, the SAO 504 and the ALF 506 utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding offsets and applying FIR filters respectively, with the help of encoding and decoding side information that signals the offsets and filter coefficients. The ALF 506 is located at the last processing stage of each picture, and can be regarded as a tool that attempts to capture and repair artifacts generated in previous stages.

[0102] The encoder 500 also includes an intra-frame prediction component 508 and a motion estimation / compensation (ME / MC) component 510, configured to receive input video. The intra-frame prediction component 508 is configured to perform intra-frame prediction, while the ME / MC component 510 is configured to perform inter-frame prediction using a reference image obtained from a reference image buffer 512. Residual blocks from inter-frame or intra-frame prediction are fed to a transform component 514 and a quantization component 516 to generate quantized residual transform coefficients, which are then fed to an entropy codec component 518. The entropy codec component 518 entropy codes and decodes the prediction results and the quantized transform coefficients and sends them to a video decoder (not shown). Quantized components output from the quantization component 516 can be fed to an inverse quantization component 520, an inverse transform component 522, and a reconstruction (REC) component 524. The REC component 524 is capable of outputting images to a DF 502, a SAO 504, and an ALF 506 for filtering before these images are stored in the reference image buffer 512.

[0103] The input to the DF 502 is the reconstructed samples prior to the loop filter. First, vertical edges in the image are filtered. Then, using the samples modified by the vertical edge filtering process as input, horizontal edges in the image are filtered. Vertical and horizontal edges in the CTB of each CTU are processed separately on a codec unit basis. Vertical edges of the codec block in the codec unit are filtered starting from the left edge of the codec block and proceeding geometrically towards the right edge of the codec block. Horizontal edges of the codec block in the codec unit are filtered starting from the top edge of the codec block and proceeding geometrically towards the bottom edge of the codec block.

[0104] Figure 6 This is a diagram 600 showing sample points 602 within an 8×8 sample block 604. As shown, diagram 600 includes horizontal block boundaries 606 and vertical block boundaries 608 on the 8×8 grid. Furthermore, diagram 600 depicts a non-overlapping 8×8 sample block 610, which can be deblocked in parallel.

[0105] Boundary decision-making is discussed. The filter is applied to 8×8 block boundaries. Furthermore, it must be a transform block boundary or an encoding / decoding sub-block boundary (e.g., due to the use of affine motion prediction, optional temporal motion vector prediction (ATMVP)). For those boundaries that are not such boundaries, the filter is disabled.

[0106] Boundary strength calculation was discussed. For transform block boundaries / encoder / decoder sub-block boundaries, if they are located in an 8×8 grid, the transform block boundaries / encoder / decoder sub-block boundaries can be filtered, and the bS[xD] of that edge... i ][yD jThe settings for [xDi][yDj] (where [xDi][yDj] represent coordinates) are defined in Table 1 and Table 2, respectively.

[0107] Table 1. Boundary Strength (when SPS IBC is disabled)

[0108]

[0109] Table 2. Boundary Strength (when SPS IBC is enabled)

[0110]

[0111]

[0112] The deblocking decision for the luminance component is discussed.

[0113] Figure 7 Example 700 shows the pixels involved in the filter on / off decision and strong / weak filter selection. A wider, stronger brightness filter is used only if conditions 1, 2, and 3 are all true. Condition 1 is the "bulk condition." This condition detects whether samples on the P-side and Q-side belong to a bulk, represented by the variables bSidePisLargeBlk and bSideQisLargeBlk, respectively. The definitions of bSidePisLargeBlk and bSideQisLargeBlk are as follows.

[0114] bSidePisLargeBlk = ((edge ​​type is vertical, and p0 belongs to CU with width >= 32) || (edge ​​type is horizontal, and p0 belongs to CU with height >= 32)) ? TRUE:FLASE

[0115] bSideQisLargeBlk = ((edge ​​type is vertical, and q0 belongs to CU with width >= 32) || (edge ​​type is horizontal, and q0 belongs to CU with height >= 32)) ? TRUE:FLASE

[0116] Based on bSidePisLargeBlk and bSideQisLargeBlk, condition 1 is defined as follows.

[0117] Condition 1=(bsidepislageblk||bsidepislageblk)? TRUE:FLASE

[0118] Next, if condition 1 is true, condition 2 will be further checked. First, the following variables are derived.

[0119] In HEVC, first export dp0, dp3, dq0, and dq3.

[0120] If (p-side is greater than or equal to 32)

[0121] dp0=(dp0+Abs(p50-2*p40+p30)+1)>>1

[0122] dp3=(dp3+Abs(p53-2*p43+p33)+1)>>1

[0123] If (q-side is greater than or equal to 32)

[0124] dq0=(dq0+Abs(q50-2*q40+q30)+1)>>1

[0125] dq3=(dq3+Abs(q53-2*q43+q33)+1)>>1

[0126] Condition 2 = (d < β)? TRUE: FALSE

[0127] Where d = dp0 + dq0 + dp3 + dq3.

[0128] If conditions 1 and 2 are valid, then further check whether any block uses sub-blocks.

[0129]

[0130] Finally, if both conditions 1 and 2 are valid, the proposed deblocking method will check condition 3 (large block strong filtering condition), which is defined as follows.

[0131] In condition 3 StrongFilterCondition, export the following variables.

[0132] dpq is derived in the same way as in HEVC.

[0133]

[0134]

[0135] In HEVC, StrongFilterCondition = (dpq < (β >> 2), sp3 + sq3 < (3 * β >> 5), and Abs(p0 - q0) < (5 * t). C +1)>>1)? TRUE:FALSE.

[0136] A stronger deblocking filter for brightness (designed for larger blocks) is discussed.

[0137] A bilinear filter is used when samples on either side of the boundary belong to a large block. Samples belonging to a large block are defined when the width of the vertical edge is greater than or equal to 32 and the height of the horizontal edge is greater than or equal to 32.

[0138] Bilinear filters are listed below.

[0139] Then, in the above HEVC deblocking, for the block boundary sample points p from i = 0 to Sp-1... i And the block boundary sample points q from j=0 to Sq-1 i Replaced by linear interpolation, p i and q i This is to filter the i-th sample point in a row of vertical edges, or to filter the i-th sample point in a column of horizontal edges, as shown below.

[0140] p i ′=(f i *Middle s,t +(64-f i )*P s +32)>>6), clipped to p i ±tcPD i

[0141] q j ′=(g j *Middle s,t +(64-g j )*Q s +32)>>6), clipped to q j ±tcPD j

[0142] Among them, tcPD i and tcPD j The terminology is the position-related clipping described below, and g is given below. j f i Middle s,t P s and Q s .

[0143] The deblocking control of chroma was discussed.

[0144] A strong chroma filter is used on both sides of the block boundary. Here, the chroma filter is selected when both sides of the chroma edge are greater than or equal to 8 (chroma position), and the decision is based on the following three conditions: The first condition is the decision regarding boundary strength and block size. The proposed filter can be applied when the width or height of the block orthogonal to the block edge in the chroma sampling domain is equal to or greater than 8. The second and third conditions are essentially the same as the HEVC luminance deblocking decision, namely the on / off decision and the strong filtering decision, respectively.

[0145] In the first decision, the boundary strength (bS) of the chroma filter is modified, and the conditions are checked sequentially. If a condition is met, the remaining lower-priority conditions are skipped.

[0146] Chromatic deblocking is performed when bS equals 2, or when a large block boundary is detected when bS equals 1.

[0147] The second and third conditions are essentially the same as the HEVC luminance strong filter decision, as shown below.

[0148] In the second condition: d is derived as in the HEVC luminance deblock. The second condition is TRUE when d is less than β.

[0149] In the third condition, StrongFilterCondition is deduced as follows.

[0150] For example, deriving dpq in HEVC

[0151] sp3 = Abs(p3 - p0), as derived in HEVC.

[0152] sq3 = Abs(q0 - q3), as derived in HEVC.

[0153] In HEVC design, StrongFilterCondition = (dpq < (β >> 2), sp3 + sq3 < (β >> 3), Abs(p0 - q0) < (5 * t) C +1)>>1).

[0154] Strong deblocking filters for chroma are discussed. The following strong deblocking filters for chroma are defined.

[0155] p2′=(3*p3+2*p2+p1+p0+q0+4)>>3

[0156] p1′=(2*p3+p2+2*p1+p0+q0+q1+4)>>3

[0157] p0′=(p3+p2+p1+2*p0+q0+q1+q2+4)>>3

[0158] The proposed chromaticity filter performs deblocking on a 4×4 chromaticity sample grid.

[0159] Position-dependent limiting (tcPD) is discussed. tcPD is applied to the output samples of the brightness filtering process, which involves modifying the strong and long filters at the boundaries of 7, 5, and 3 samples. Assuming a quantization error distribution, it is proposed to increase the limiting value of samples expected to have higher quantization noise, thus resulting in a larger deviation between the expected reconstructed sample values ​​and the true sample values.

[0160] For each P or Q boundary filtered using an asymmetric filter, a position-related threshold table is selected from two tables (i.e., Tc7 and Tc3 in the table below) that serve as edge information to the decoder, based on the result of the decision process in the boundary strength calculation.

[0161] Tc7={6,5,4,3,2,1,1}; Tc3={6,4,2};

[0162] tcPD=(Sp==3)? Tc3:Tc7;

[0163] tcQD=(Sq==3)? Tc3:Tc7;

[0164] For P or Q boundaries filtered by a short symmetric filter, apply a lower amplitude position correlation threshold.

[0165] Tc3 = {3, 2, 1};

[0166] After defining the threshold, the filtered p' is adjusted based on the tcP and tcQ limiting values. i q' i The sample values ​​are limited.

[0167] p” i =Clip3(p' i +tcP i ,p' i –tcP i ,p' i );

[0168] q” j =Clip3(q' j +tcQ j ,q' j –tcQ j ,q' j );

[0169] Among them, p' i and q' i It is the filtered sample value p” i and q" j It is the output sample value after amplitude limiting, tcP i The clipping threshold is derived from the VVCtc parameters, as well as tcPD and tcQD. The Clip3 function is the clipping function defined in VVC.

[0170] Now we discuss sub-block deblocking adjustments. To achieve parallel-friendly deblocking using long filters and sub-block deblocking, the long filter is restricted to modifying a maximum of 5 samples on the side using sub-block deblocking (AFFINE or ATMVP or decoder-side motion vector refinement (DMVR)), as shown in the long filter's brightness control. Furthermore, sub-block deblocking is adjusted such that sub-block boundaries on the 8×8 grid near the codec unit (CU) or implicit TU boundary are restricted to modifying a maximum of two samples on each side.

[0171] The following applies to sub-block boundaries that are not aligned with the CU boundary.

[0172]

[0173] An edge equal to 0 corresponds to the CU boundary, an edge equal to 2 or orthogonalLength-2 corresponds to 8 sample points of the sub-block boundary from the CU boundary, and so on. If implicit partitioning of TU is used, then implicit TU is true.

[0174] Sample Adaptive Offset (SAO) is discussed. The input to SAO is the deblocked reconstructed samples (DB). The concept of SAO is to reduce the average sample distortion of a region by first classifying region samples into multiple categories using a selected classifier, obtaining an offset for each category, and then adding this offset to each sample of that category. The classifier index and the region offset are encoded and decoded in the bitstream. In HEVC and VVC, the region (the unit for SAO parameter signaling notification) is defined as a CTU.

[0175] HEVC employs two SAO types that meet low complexity requirements. These two types are Edge Offset (EO) and Band Offset (BO), which will be discussed in detail below. The index of the SAO type is encoded and decoded (within the range [0, 2]). For EO, ​​sample classification is based on a comparison between the current sample and its neighboring samples, according to a one-dimensional orientation scheme: horizontal, vertical, 135° diagonal, and 45° diagonal.

[0176] Figure 8 Four one-dimensional (1-D) orientation schemes 800 for EO sample point classification are shown: horizontal (EO classification = 0), vertical (EO classification = 1), 135° diagonal (EO classification = 2), and 45° diagonal (EO classification = 3).

[0177] For a given EO classification, each sample point within the CTB is classified into one of five categories. The current sample point value, labeled "c", is compared with its two neighboring sample point values ​​along the selected 1-D pattern. The classification rules for each sample point are summarized in Table 3. Categories 1 and 4 are associated with local valleys and local peaks along the selected 1-D pattern, respectively. Categories 2 and 3 are associated with concave and convex angles along the selected one-dimensional scheme, respectively. If the current sample point does not belong to EO categories 1-4, it belongs to category 0, and SAO is not applied.

[0178] Table 3: Sampling classification rules for edge offset

[0179]

[0180] An adaptive loop filter based on geometric transformation (GEM) is discussed in the Joint Exploration Model (JEM). The input to the DB is the reconstructed samples after DB and SAO. The sample classification and filtering processes are based on the reconstructed samples after DB and SAO.

[0181] In JEM, a geometric transformation-based adaptive loop filter (GALF) with block-based filter adaptation is applied. For the luminance component, one of 25 filters is selected for each 2×2 block based on the direction and effectiveness of the local gradient.

[0182] The shape of the filter was discussed. Figure 9 An example of a GALF filter shape 900 is shown, including a 5×5 rhombus on the left, a 7×7 rhombus in the middle, and a 9×9 rhombus on the right. In JEM, up to three rhombus filter shapes can be selected for the luminance component (e.g., ...). Figure 9 (As shown). The image-level signaling notification index indicates the filter shape used for the luminance component. Each square represents a sample point, and Ci (i is 0–6 (left), 0–12 (middle), 0–20 (right)) represents the coefficient applied to that sample point. For the chrominance component in the image, a 5×5 rhombus is always used.

[0183] Block classification was discussed. Each 2×2 block was divided into one of 25 classes. The classification index c was based on its directionality D and validity. The quantized values ​​are exported as shown below.

[0184]

[0185] To calculate D and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using the 1-D Laplacian operator.

[0186]

[0187]

[0188]

[0189]

[0190] Indices i and j refer to the coordinates of the top-left sample point in the 2×2 block, and R(i,j) represents the reconstructed sample point at coordinates (i,j).

[0191] The maximum and minimum values ​​of the gradients in the horizontal and vertical directions are then set as follows:

[0192]

[0193] The maximum and minimum values ​​of the gradients in the two diagonal directions are set as follows:

[0194]

[0195] To derive the values ​​of directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0196] Step 1. If and If true, then D is set to 0.

[0197] Step 2. If Continue from step 3; otherwise, continue from step 4.

[0198] Step 3. If D is set to 2; otherwise, D is set to 1.

[0199] Step 4. If D is set to 4; otherwise, D is set to 3.

[0200] The validity value A is calculated as follows:

[0201]

[0202] A is further quantized to the range of 0 to 4, and the quantized value is represented as

[0203] For the two chromaticity components in the image, no classification method is applied; that is, a single set of ALF coefficients is applied for each chromaticity component.

[0204] The geometric transformation of the filter coefficients is discussed.

[0205] Figure 10 The relative coordinates 1000 are shown for the 5×5 diamond filters that support (from left to right) diagonal, vertical flip, and rotation.

[0206] Before filtering each 2×2 block, geometric transformations such as rotation, diagonal flip, and vertical flip are applied to the filter coefficients f(k,l) associated with coordinates (k,l) based on the gradient values ​​calculated for that block. This is equivalent to applying these transformations to samples within the filter's support region. The idea is to make the different blocks applying ALF more similar by aligning their orientations.

[0207] Three geometric transformations are introduced: diagonal transformation, vertical flip, and rotation.

[0208]

[0209] Where K is the size of the filter, and 0 ≤ k, l ≤ K⁻¹ are the coefficient coordinates, such that position (0, 0) is located at the top left corner and position (K⁻¹, K⁻¹) is located at the bottom right corner. The transform is applied to the filter coefficients f(k, l) based on the gradient values ​​calculated for this block. Table 4 summarizes the relationship between the transform and the four gradients in the four directions.

[0210] Table 4: Mapping between gradients and transformations computed for a block

[0211]

[0212] Signaling notification of filter parameters is discussed. In JEM, the GALF filter parameters are signaled for the first CTU, i.e., after the stripe header of the first CTU and before the SAO parameters. A maximum of 25 groups of luminance filter coefficients can be sent. To reduce bit overhead, filter coefficients from different categories can be merged. Furthermore, the GALF coefficients of a reference image are stored and can be reused as GALF coefficients for the current image. The current image can optionally use the GALF coefficients stored for the reference image and bypass the GALF coefficient signaling. In this case, only the index of a reference image is signaled, and the stored GALF coefficients of the indicated reference image are inherited by the current image.

[0213] To support GALF temporal prediction, a candidate list of GALF filter sets is maintained. At the beginning of decoding a new sequence, the candidate list is empty. After decoding an image, the corresponding filter set can be added to the candidate list. Once the size of the candidate list reaches the maximum allowed value (i.e., 6 in the current JEM), new filter sets overwrite the oldest sets in decoding order; that is, a first-in-first-out (FIFO) rule is applied to update the candidate list. To avoid duplication, a set is added to the list only if the corresponding image does not use GALF temporal prediction. To support temporal scalability, multiple candidate lists of filter sets exist, and each candidate list is associated with a temporal layer. More specifically, each array assigned by the temporal layer index (TempIdx) can form the filter set of previously decoded images with a lower TempIdx. For example, the k-th array is assigned to be associated with a TempIdx equal to k, and the k-th array contains only the filter set from images with a TempIdx less than or equal to k. After an image is encoded or decoded, the set of filters associated with that image will be used to update the arrays associated with TempIdx that are equal to or higher than TempIdx.

[0214] Temporal prediction of GALF coefficients is used for inter-frame encoding / decoding to minimize signaling overhead. For intra-frame frames, temporal prediction is not available, and a set of 16 fixed filters is assigned to each class. To indicate the use of fixed filters, signaling informs the flag for each class and, if necessary, the index of the selected fixed filter. Even when a fixed filter is selected for a given class, the coefficients of the adaptive filter f(k,l) can still be sent for that class; in this case, the coefficients of the filter applied to the reconstructed image are the sum of the coefficients from both sets.

[0215] The filtering process for the luminance component can be controlled at the CU level. A signaling flag indicates whether GALF is applied to the luminance component of the CU. For the chrominance component, whether GALF is applied is only indicated at the image level.

[0216] The filtering process is discussed. On the decoder side, when GALF is enabled for a block, each sample R(i,j) within the block is filtered, producing sample values ​​R′(i,j) as shown below, where L represents the filter length, f m,n Let f(k,l) represent the filter coefficients, and let f(k,l) represent the decoding filter coefficients.

[0217]

[0218] Figure 11This example illustrates the relative coordinates supported by a 5×5 diamond filter, assuming the current sample point's coordinates (i, j) are (0, 0). Sample points at different coordinates, filled with the same color, are multiplied by the same filter coefficients.

[0219] The geometric transformation-based adaptive loop filter (GALF) for VVC is discussed. In VVC test model 4.0 (VTM4.0), the filtering process of the adaptive loop filter is performed as follows:

[0220] O(x,y)=∑ (i,j) w(i,j).I(x+i,y+j), (11) where I(x+i,y+j) are the input samples, O(x,y) are the filtered output samples (i.e., the filtering result), and w(i,j) represent the filtering coefficients. In fact, in VTM4.0, it uses integer arithmetic to achieve fixed-point precision calculation.

[0221]

[0222] Where L represents the filter length, and w(i,j) are the fixed-point filter coefficients.

[0223] Compared to JEM, the current design of GALF in VVC has the following main changes:

[0224] 1) Adaptive filter shapes have been removed. Only 7×7 filter shapes are allowed for the luminance component, and only 5×5 filter shapes are allowed for the chrominance component.

[0225] 2) Move the signaling notification of ALF parameters from the strip / picture level to the CTU level.

[0226] 3) The category index is calculated at a 4×4 level instead of a 2×2 level. Furthermore, as proposed in JVET-L0147, a subsampling Laplacian calculation method for ALF classification is utilized. More specifically, it is not necessary to calculate the horizontal / vertical / 45-degree diagonal / 135-degree gradient for each sample point within a block. Instead, 1:2 subsampling is used.

[0227] Regarding filtering and reconstruction, the nonlinear ALF in the current VVC is discussed.

[0228] Equation (11) can be reformulated without affecting encoding / decoding efficiency as follows:

[0229] O(x,y)=I(x,y)+∑ (i,j)≠(0,0) w(i,j).(I(x+i,y+j)-I(x,y)), (13)

[0230] Where w(i,j) are the same filter coefficients as in equation (11) [except that w(0,0) is equal to 1 in equation (13), while it is equal to 1-∑ in equation (11). (i,j)≠(0,0) w(i,j)].

[0231] Using the above filter formula in (13), VVC introduces nonlinearity to reduce the impact of large differences between neighboring sample values ​​(I(x+i,y+j)) and the current sample value being filtered (I(x,y)) by using a simple limiting function, thereby making ALF more effective.

[0232] More specifically, the ALF filter is modified as follows:

[0233] O′(x,y)=I(x,y)+∑ (i,j)≠(0,0) w(i,j).K(I(x+i,y+j)-I(x,y),k(i,j)), (14)

[0234] Where K(d,b)=min(b,max(-b,d)) is the limiting function, and k(i,j) is the limiting parameter that depends on the (i,j) filter coefficients. The encoder performs optimization to find the optimal k(i,j).

[0235] In the JVET-N0242 implementation, a limiting parameter k(i,j) is specified for each ALF filter, and a limiting value is signaled for each filter coefficient. This means that a maximum of 12 limiting values ​​can be signaled for each luminance filter in the bitstream, and a maximum of 6 limiting values ​​can be signaled for each chrominance filter.

[0236] To limit signaling notification costs and encoder complexity, only four fixed values ​​are used, which are the same for both INTER and INTRA stripes.

[0237] Because the local differences in luminance are typically greater than those in chrominance, two separate sets of luminance and chrominance filters are applied. The maximum sample value in each set (1024 for a 10-bit depth) is also introduced so that clipping can be disabled when not necessary.

[0238] Table 5 provides a set of limiting values ​​used in the JVET-N0242 test. These four values ​​were selected by roughly dividing the full range of luminance sample values ​​(encoded in 10 bits) and the chrominance range from 4 to 1024 in the logarithmic domain.

[0239] More precisely, the brightness table for the limit value is obtained using the following formula:

[0240]

[0241] Similarly, the colorimetric table for the limiting values ​​is obtained using the following formula:

[0242]

[0243] Table 5: Authorized Limit Values

[0244]

[0245] The selected threshold value is encoded and decoded in the "alf_data" syntax element using the Golomb coding scheme corresponding to the threshold value index in Table 5 above. This coding scheme is the same as that used for the filter index.

[0246] A convolutional neural network-based loop filter for video encoding and decoding is discussed.

[0247] In deep learning, convolutional neural networks (CNNs or ConvNets) are a type of deep neural network most commonly used for analyzing visual images. They have been very successful in image and video recognition / processing, recommender systems, image classification, medical image analysis, and natural language processing.

[0248] CNNs are a regularized version of multilayer perceptrons. Multilayer perceptrons typically mean fully connected networks, where every neuron in one layer is connected to all neurons in the next layer. This "full connectivity" makes them prone to overfitting data. Typical regularization methods involve adding some form of weight magnitude metric to the loss function. CNNs take a different approach to regularization: they utilize hierarchical schemes in the data and assemble more complex methods using smaller and simpler schemes. Therefore, CNNs are at the lower extremes in terms of connectivity and complexity.

[0249] Compared to other image classification / processing algorithms, CNNs use relatively little preprocessing. This means the network learns filters that are hand-designed in traditional algorithms. This independence from existing knowledge and human effort in feature design is a major advantage.

[0250] Deep learning-based image / video compression generally has two meanings: end-to-end compression based entirely on neural networks, and traditional frameworks enhanced by neural networks. End-to-end compression based entirely on neural networks is discussed in Johannes Balle, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” 2016, Workshop on Picture Coding (PCS), pp. 1–5, Institute of Electrical and Electronics Engineers (IEEE), and in Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár, “Lossy image compression using compression autocoders,” arXiv prequel arXiv:1703.00395 (2017). The traditional framework enhanced by neural networks is discussed in the following articles: Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong and Wen Gao, “Intra-frame prediction for image coding and decoding based on fully connected networks”, IEEE Transactions on Image Processing 27, 7 (2018), 3236–3247; Yuanying Dai, Dong Liu and Feng Wu, “Post-processing convolutional neural network approach in HEVC intra-frame coding and decoding”, MMM. Springer, 28–39; Rui Song, Dong Liu, Houqiang Li and Feng Wu, “Neural network-based arithmetic coding and decoding of intra-frame prediction modes in HEVC”, VCIP. IEEE, 1–4; and J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, W. Samek, H. Schwarz, D. Marpe and T. Wiegand, “Neural network-based intra-frame prediction for video coding and decoding”, Digital Image Processing Applications XLI, Vol. 10752, International Society for Optics and Photonics, 1075213.

[0251] The first type typically employs an autoencoder-like structure, implemented through convolutional neural networks or recurrent neural networks. While relying solely on neural networks for image / video compression avoids any manual optimization or design, the compression efficiency may not be satisfactory. Therefore, research focused on the second type aims to enhance traditional compression frameworks by replacing or strengthening certain modules with neural networks. In this way, they can inherit the advantages of highly optimized traditional frameworks. For example, the fully connected network for intra-frame prediction proposed in HEVC is discussed in Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao, “Intra-frame Prediction Based on Fully Connected Networks for Image Encoding and Decoding,” IEEE Transactions on Image Processing 27, 7 (2018), pp. 3236-3247.

[0252] Besides intra-frame prediction, deep learning is also used to enhance other modules. For example, the loop filter in HEVC is replaced by a convolutional neural network, and satisfactory results were achieved in Yuanying Dai, Dong Liu, and Feng Wu, "A Convolutional Neural Network Approach for Post-Processing in HEVC Intra-Frame Encoding and Decoding," MMM. Springer, 28–39. Research in Rui Song, Dong Liu, Houqiang Li, and Feng Wu, "Neural Network-Based Arithmetic Encoding and Decoding for HEVC Intra-Frame Prediction Modes," VCIP. IEEE, 1–4, applies neural networks to improve the arithmetic encoding and decoding engine.

[0253] Loop filtering based on convolutional neural networks is discussed. In lossy image / video compression, the reconstructed frame is an approximation of the original frame, and because the quantization process is irreversible, this leads to distortion in the reconstructed frame. To mitigate this distortion, convolutional neural networks can be trained to learn the mapping from distorted frames to the original frames. In fact, training is necessary before deploying CNN-based loop filtering.

[0254] Training was discussed. The goal of the training process is to find the optimal values ​​for the parameters, including the weights and biases.

[0255] First, an encoder-decoder (e.g., HM, JEM, VTM, etc.) is used to compress the training dataset to generate distorted reconstructed frames. Then, the reconstructed frames are fed into a CNN, and the cost is calculated using the CNN's output and the ground truth frames (original frames). Common cost functions include Sum of Absolute Differences (SAD) and Mean Squared Error (MSE). Next, the gradient of the cost with respect to each parameter is derived using the backpropagation algorithm. The parameter values ​​are updated using the gradients. This process is repeated until the convergence criterion is met. After training is complete, the derived optimal parameters are saved for use in the inference phase.

[0256] The convolution process is discussed. During convolution, the filter moves across the image from left to right and from top to bottom, changing one pixel column for horizontal movement and one pixel row for vertical movement. The amount of movement by which the filter is applied across the input image is called the stride, and it is almost always symmetrical in the height and width dimensions. For height and width movements, the default (multiple) stride in two dimensions is (1, 1).

[0257] Figure 12A This is an example architecture 1200 of the proposed CNN filter, and Figure 12B This is an example of constructing a ResBlock (1250). In most deep convolutional neural networks, ResBlocks are used as basic modules and stacked several times to build the final network. In one example, ResBlocks are obtained by combining convolutional layers, ReLU / PReLU activation functions, and convolutional layers, such as... Figure 12B As shown.

[0258] Inference is discussed. During the inference phase, distorted reconstructed frames are fed into the CNN and processed by the CNN model, whose parameters have been determined during the training phase. The input samples to the CNN can be reconstructed samples before or after DB, or before or after SAO, or before or after ALF.

[0259] Current CNN-based loop filtering suffers from the following problems. First, a separate CNN model is trained for each quality level (e.g., quantization parameter (QP), constant ratio factor (CRF), or bit rate), resulting in a large number of CNN models. Second, when QPs are used as input to the neural network, they are first concatenated into a two-dimensional array of the same size as the block to be filtered, and then used as an additional input plane. Therefore, QP information may not be fully utilized. Third, the characteristics of an encoder-decoder block are not fully explored when constructing a candidate list containing multiple models for a given block.

[0260] This document discloses techniques for solving one or more of the aforementioned problems. For example, this disclosure provides one or more neural network (NN) filter models that are trained as part of a loop filtering technique or a filtering technique used in a post-processing stage to reduce distortion caused during compression. Furthermore, samples with different characteristics can be processed using different NN filter models.

[0261] This disclosure also describes how to design a unified NN filter model by feeding at least one indicator (which may be related to a quality level (e.g., QP, CRF, or bit rate, and is therefore a quality level indicator (QI)) as input to the NN filter), and how to construct a candidate list of multiple models containing coded blocks by considering the encoding and decoding statistics of the blocks. Other examples in this specification involve combining NN filter models with non-deep learning-based filtering (NDLF) methods.

[0262] Furthermore, the examples in this specification relate to controlling the presence (e.g., application) of NN filter models by using syntax elements at different levels. For example, syntax elements(s) indicating whether an NN filter(s) is applied may be at a first level (e.g., in the sequence header and / or sequence parameter set (SPS) of a video unit). Syntax elements(s) indicating whether an NN filter(s) is applied may also be at a second level (e.g., in the picture header, picture parameter set (PPS), and / or strip header of a video unit). Further, syntax elements(s) indicating whether an NN filter(s) is applied may be at a third level (e.g., indicative of a patch, CTU, CTB, block, subpicture, tile, or region of a video unit).

[0263] It should be noted that the concept of a unified NN filter model, which feeds Qi as input into the NN filter process, can also be extended to other NN-based encoding and decoding tools, such as NN-based super-resolution or motion compensation designs. In the example below, a neural network-based filtering technique is used as an example.

[0264] The following list of embodiments should be considered as examples to explain general concepts. These embodiments should not be interpreted narrowly. Furthermore, these embodiments can be combined in any way.

[0265] In this disclosure, the NN filter can be any type of NN filter, such as a convolutional neural network (CNN) filter. In the following discussion, the NN filter may also be referred to as a CNN filter.

[0266] In the following discussion, a video unit can be a sequence, picture, strip, slice, tile, sub-picture, CTU / CTB, CTU / CTB line, one or more CU / codec block (CB), one or more CTU / CTB, one or more Virtual Pipeline Data Units (VPDUs), or a sub-region within a picture / strip / slice / tile. A parent video unit represents a unit larger than a video unit. Generally, a parent unit will contain several video units; for example, when the video unit is a CTU, the parent unit can be a strip, a CTU line, multiple CTUs, etc. In some embodiments, a video unit can be a sample / pixel.

[0267] Figure 13 An example of the process for generating filtered samples for an NN filter model based on a received quality level indicator (QI) as input. Figure 13 In the process 1300 shown, at least some unfiltered samples are provided as input to the NN filter. In one example, the unfiltered samples are samples (e.g., pixels) of video units that have not yet undergone any filtering or have not undergone sufficient filtering. Therefore, the output of the NN filter can be filtered samples. The output of the NN filter is also based on the NN filter model generated using a Quality Level Indicator (QI) input. The QI input can be related to the quality level of the video unit, such as QP, CRF, or bitrate.

[0268] Figure 14 This is an example of process 1400 where quality level indicators are expanded or concatenated into a two-dimensional array, based on some examples. For instance, in process 1400, the two-dimensional array may have the same size as the video unit to be filtered (e.g., in each of the two dimensions) and serve as an additional input plane for the NN filter.

[0269] Figure 15 This is an example of process 1500 where a quality level indicator (QI) is fed into multiple fully connected layers (e.g., fully connected layers of an NN filter) to expand into a one-dimensional vector. In process 1500, the elements of the resulting one-dimensional vector can be used as channel scaling factors. In process 1500, channel scaling factors are useful for recalibrating the feature maps of the NN filter model (e.g., where N represents the number of channels, W represents the width, and H represents the height), and thus provide recalibrated feature maps in some examples.

[0270] Figure 16 An embodiment of video bitstream 1600 is shown. As used herein, video bitstream 1600 may also be referred to as a encoded / decoded video bitstream, bitstream, or a variant thereof. Figure 16As shown, bitstream 1600 includes one or more of the following: Decoding Capability Information (DCI) 1602, Video Parameter Set (VPS) 1604, Sequence Parameter Set (SPS) 1606, Picture Parameter Set (PPS) 1608, Picture Header (PH) 1610, and Picture 1614. Each of DCI 1602, VPS 1604, SPS 1606, and PPS 1608 can be collectively referred to as a parameter set. In this embodiment, Figure 16 Other parameter sets, not shown in the figure, may also be included in the bitstream 1600, such as, for example, the adaptive parameter set (APS), which is a syntax structure containing syntax elements applied to zero or more stripes determined by zero or more syntax elements found in the strip header.

[0271] As described above, embodiments of this specification control the presence (e.g., application) of NN filter models by using syntax elements at different levels. For example, syntax elements(s) indicating whether an NN filter is applied may be at a first level (e.g., in the sequence header and / or sequence parameter set (SPS) of a video unit). Syntax elements(s) indicating whether an NN filter is applied may also be at a second level (e.g., in the picture header, picture parameter set (PPS), and / or strip header of a video unit). Furthermore, syntax elements(s) indicating whether an NN filter is applied may be at a third level (e.g., syntax elements indicating a small block, CTU, CTB, block, sub-picture, tile, or region of a video unit).

[0272] DCI 1602, also known as a Decoder Parameter Set (DPS) or Decoder Parameter Set, is a syntactic structure containing syntax elements applied throughout the bitstream. DCI 1602 includes parameters that remain constant throughout the lifetime of the video bitstream (e.g., bitstream 1600), which can be translated to the lifetime of a session. DCI 1602 can include profile, level, and subprofile information to determine the maximum complexity interoperability point guaranteed never to be exceeded, even if video sequence splicing occurs within a session. It may also optionally include constraint flags indicating that the video bitstream will be constrained to the use of certain features indicated by the values ​​of those flags. This allows the bitstream to be marked as not using certain tools, which in particular allows for resource allocation in decoder implementations. Like all parameter sets, DCI 1602 exists at first reference and is referenced by the first picture in the video sequence, meaning it must be passed between the first Network Abstraction Layer (NAL) units in the bitstream. While multiple DCI 1602s may exist in bitstream 1600, the values ​​of the syntax elements within them must not be inconsistent at reference.

[0273] VPS 1604 includes decoding relevance or information for constructing the reference picture set used for the enhancement layer. VPS 1604 provides an overall view of the scalable sequence, including what type of operation points are provided, the tiers, levels, and grades of operation points, as well as some other high-level attributes of the bitstream that can be used as the basis for session negotiation and content selection.

[0274] SPS 1606 contains data common to all pictures in the picture sequence (SOP) (e.g., whether one or more NN filters are applied). SPS 1606 is a grammatical structure containing grammatical elements applied to zero or more complete codec layer video sequences (CLVS) as determined by the content of grammatical elements found in PPS 1608, referenced by grammatical elements found in each picture header 1612. Conversely, PPS 1608 contains data common to the entire picture 1614. PPS 1608 is a grammatical structure containing grammatical elements applied to zero or more complete codec pictures as determined by grammatical elements found in each picture header (e.g., PH 1612) (e.g., whether one or more NN filters are applied).

[0275] DCI 1602, VPS 1604, SPS 1606, and PPS 1608 are contained within different types of Network Abstraction Layer (NAL) units. An NAL unit is a syntax structure that contains an indication of the type of data to be followed (e.g., encoded / decoded video data). NAL units are categorized into Video Codec Layer (VCL) and non-VCL NAL units. VCL NAL units contain data representing sample values ​​in video frames, while non-VCL NAL units contain any associated additional information, such as parameter sets (important data that can be applied to multiple VCL NAL units) and supplementary enhancement information (timing information and other supplementary data that can enhance the usability of the decoded video signal but are not necessary for decoding sample values ​​in video frames).

[0276] PH 1612 is a syntactic structure that contains syntactic elements applied to all stripes (e.g., stripe 1618) of a picture (e.g., picture 1614).

[0277] Image 1614 is a monochrome luminance sample array, or a luminance sample array and two corresponding chrominance sample arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0278] Image 1614 can be a frame or a field. However, in a CVS 1616, all images 1614 are either frames or all images 1614 are fields. A CVS 1616 is a codec video sequence of each codec layer video sequence (CLVS) in the video bitstream 1600. It's important to note that CVS 1616 and CLVS are identical when the video bitstream 1600 comprises a single layer. CVS 1616 and CLVS differ only when the video bitstream 1600 comprises multiple layers.

[0279] Each image 1614 contains one or more stripes 1618. A stripe 1618 is an integer number of complete slices or an integer number of consecutive complete codec tree unit (CTU) rows within a slice of an image (e.g., image 1614). Each stripe 1618 is exclusively contained within a single NAL unit (e.g., a VCL NAL unit). A slice (not shown) is a rectangular region of CTUs within a specific slice column and a specific slice row in an image (e.g., image 1614). A CTU (not shown) is a codec tree block (CTB) of luminance samples, two corresponding CTBs of chrominance samples of an image with a three-sample array, or a CTB of samples of a monochrome image or an image encoded using three separate color planes and a syntax structure for encoding and decoding samples. A CTB (not shown) is an N×N sample block for a certain value N, such that dividing the components into CTBs is a partition. A block (not shown) is an M×N (M columns by N rows) sample (e.g., pixel) array, or an M×N transform coefficient array.

[0280] In this embodiment, each stripe 1618 includes a stripe header 1620. The stripe header 1620 is a portion of the encoded stripe 1618 that contains data elements belonging to all slices or CTU rows within the stripe represented in the stripe 1618. That is, the stripe header 1620 contains information about the stripe 1618, such as, for example, the stripe type, which reference image will be used, etc.

[0281] Image 1614 and its strip 1618 include data associated with the encoded or decoded image or video. Therefore, image 1614 and its strip 1618 can simply be referred to as the payload or data carried in bitstream 1600.

[0282] In one embodiment, (e.g., bitstream 1600) the first level is the sequence level, and the syntax elements indicated in the first level are indicated in the SPS 1606 and / or sequence header. In one embodiment, (e.g., bitstream 1600) the second level is the picture level, and the syntax elements indicated in the second level are indicated in the PPS 1608, PH 1612 and / or stripe header 1620. In one embodiment, (e.g., bitstream 1600) the third level is the subpicture level, and the syntax elements indicated in the third level are for video unit blocks, video unit CTUs, video unit CTBs, video unit blocks, video unit subpictures, video unit slices, video unit regions, etc.

[0283] In one embodiment, the syntax element is at a first level and indicates whether an NN filter can be adaptively selected at a second level to be applied to picture 1614 or strip 1618 of the video unit. In another embodiment, the syntax element is at a second level and indicates whether an NN filter can be adaptively selected at a third level to be applied to sub-pictures of the video unit. When the syntax element is at a second level, it can also indicate whether the use of the NN filter can be controlled at a third level (e.g., via a syntax element at a third level).

[0284] In one embodiment, syntax elements are encoded and decoded using context-based encoding and decoding (e.g., context-based adaptive binary arithmetic encoding and decoding (CABAC) or bypass encoding and decoding). In another embodiment, syntax elements are binary encoded and decoded using fixed-length encoding and decoding, unary encoding and decoding, truncated unary encoding and decoding, signaling-notification unary encoding and decoding, signed truncated unary encoding and decoding, truncated binary encoding and decoding, or exponential Golomb encoding and decoding.

[0285] Bitstream 1600 also includes one or more SEI messages containing supplemental enhancement information. SEI messages can contain various types of data, indicating the timing of video frames, or describing various properties of the encoded video, or how the encoded video is used or enhanced. SEI messages are also defined to contain arbitrary user-defined data. SEI messages do not affect the core decoding process but can indicate suggestions on how to post-process or display the video. Some other high-level properties of the video content are conveyed in Video Usability Information (VUI), such as an indication of the color space used to interpret the video content. As new color spaces have been developed, such as high dynamic range and wide color gamut videos, additional VUI identifiers have been added to indicate them.

[0286] Those skilled in the art will understand that, in practical applications, Bitstream 1600 may contain other parameters and information.

[0287] A discussion of model selection is provided.

[0288] Example 1

[0289] 1. The NN filter model can take at least one indicator that may be related to the quality level as input. This indicator is called the quality level indicator (QI).

[0290] a. In one example, the quality level indicator for a video cell is the quantization parameter (QP) associated with the video cell.

[0291] b. In one example, the quality level indicator for a video unit is the QP associated with the sequence to which the video unit belongs (e.g., the initial QP of signaling notification in SPS).

[0292] c. In one example, the quality level indicator for a video cell is the QP associated with the picture or strip to which the video cell belongs.

[0293] d. Alternatively, the QP can be replaced by a quantization step corresponding to the QP described in bullet points a, b, and c above. For example, the QI input can be the output of a quantization process applied to a video unit, where the quantization process is based on the QP of the video unit, the QP of a sequence containing video units, or the QP of a picture or strip containing video units.

[0294] e. In one example, the quality level indicators are first concatenated or expanded into a two-dimensional array with the same size as the video units to be filtered, and then treated as an additional input plane, for example... Figure 14 The splicing process is shown.

[0295] i. In one example, the video unit is CTU / CTB.

[0296] f. In one example, the quality level indicator is fed into several fully connected layers of the NN filter to be expanded into a one-dimensional vector, the elements of which will be used as channel scaling factors to recalibrate the feature maps of the NN filter model.

[0297] i. In one example, the feature map of the NN filter model is represented as f∈R N×W×H Let S∈R, where N, W, and H are the number of channels, width, and height, respectively. N Let F represent a one-dimensional vector obtained from the fully connected layer. Then, the above recalibration process can be written as: F i,j,k =f i,j,k ×S i ,1≤i≤N,1≤j≤W,1≤k≤H, where f is the recalibrated feature map, for example, Figure 15 As shown.

[0298] g. In one example, the quality level indicator is used as an additional input plane and a recalibration factor, i.e., aspects e and f can be combined.

[0299] h. In one example, the quality level indicator can be communicated from encoder signaling to the decoder, such as in the SPS, Picture Parameter Set (PPS), Picture header, Strip header, CTU, CU, or at any region level. In some examples, the PPS may include information about entropy encoding / decoding modes, strip groups, motion prediction, and deblocking filters.

[0300] i. In one example, f(QI) instead of QI can be fed into the NN, where f is any function.

[0301] j. In one example, the use of QI can depend on the color components.

[0302] i. For example, a first QI can be used for a first component (e.g., luminance), and a second QI can be used for a second component (e.g., chromaticity, such as Cb or Cr). The first QI can indicate the quality level of the first component, and the second QI can indicate the quality level of the second component.

[0303] ii. For example, the first QI can be used for a first component (e.g., luminance), and the first QI can also be used for a second component (e.g., chromaticity, such as Cb or Cr). The first QI can indicate the quality level of the first component.

[0304] iii. For example, QI can be used for NN filtering of the first component (e.g., luminance), but not for the second component (e.g., chrominance, such as Cb or Cr).

[0305] iv. For example, Qi can be used for NN filtering of all components.

[0306] k. In one example, the QI used to filter the first video unit can indicate the quality of the region or image containing the first video unit.

[0307] l. In one example, the QI used to filter the first video unit can indicate the quality of the region or image that does not contain the first video unit.

[0308] i. For example, QI can indicate the quality of the area or image referenced by the first video unit.

[0309] Example 2

[0310] 2. In the second embodiment, the construction of the filter candidate list may depend on the encoding and decoding statistics of the video unit (e.g., prediction mode, qp, stripe type, etc.).

[0311] a. In one example, the candidate list may depend on the ratio of the first set of samples in the video unit that are encoded and decoded by the codec tool (or codec mode) X.

[0312] i. In one example, X is a prediction mode, such as an intra-frame codec mode.

[0313] 1) Alternatively, X can be any other codec tool, such as any inter-frame prediction tool or transform / quantization codec tool.

[0314] ii. In one example, X is a variety of prediction modes, such as intra-frame / palette / IBC codec modes.

[0315] iii. In one example, if the ratio of intra-frame codec samples is not less than or greater than a threshold, then one of the candidates in the candidate list is the NN filter model corresponding to the candidate for the intra-frame codec.

[0316] iv. In one example, if the ratio of intra-frame codec samples is not less than or greater than a threshold, the first candidate is the NN filter model corresponding to the intra-frame stripe.

[0317] v. In the example above, the threshold can be predefined.

[0318] 1) In one example, the video unit is CTU / CTB.

[0319] 2) In one example, the threshold is set to 0.5.

[0320] 3) Alternatively, the threshold can be derived on the fly, for example, based on decoding information such as sequence or image or strip-level QP.

[0321] 4) Alternatively, the threshold can be exported instantly, for example, based on the color components.

[0322] b. In one example, the method described above in Example 2 may be applied conditionally.

[0323] i. In one example, the above methods(s) may be applied only to the luminance component.

[0324] ii. In one example, the above methods(s) can be applied to multiple color components.

[0325] 1) In one example, the encoding and decoding statistics of the luma block can be used to determine the candidate list of luma; and the encoding and decoding statistics of the chroma block can be used to determine the candidate list of chroma.

[0326] 2) In another example, the encoding and decoding statistics of the luma block can be used to determine a candidate list of luminance and chrominance.

[0327] iii. In one example, the above methods(s) can be applied to certain time-domain layers.

[0328] 1) In one example, they can be applied to time-domain layers with an ID greater than K (e.g., K=3).

[0329] 2) In one example, they can be applied to time-domain layers with ID less than K (e.g., K=4).

[0330] iv. In one example, the above methods(s) can be applied to certain image / strip types, such as P or B stripes.

[0331] c. The first set of samples can be located in the same picture containing the video unit to be filtered by the NN.

[0332] i. Alternatively, the first set of samples can be located in a different image than the video unit to be filtered by the NN. For example, the first set of samples can be located in the reference image of the video unit.

[0333] ii. Alternatively, the first set of samples may be in multiple images that are encoded / decoded prior to the image containing the video unit to be filtered.

[0334] Example 3

[0335] 3. In the third embodiment, one of the above-mentioned NN filters can be used exclusively with NDLF. For example, when an NN filter is applied, one or more NDLFs may no longer be applied, and vice versa.

[0336] a. In one example, NDLF may include one or more deblocking filters, SAO, cross-component (CC)-SAO, ALF, CC-ALF, luminance mapping with chroma scaling (LMCS), bilateral filters, transform domain filtering methods, etc.

[0337] b. In one example, the proposed neural network-based loop filter is used with various types of NDLFs, such as in combination with one or more NDLFs.

[0338] i. In one example, when the NN filter is applied, the original loop filters, such as Deblocking (DB), SAO, ALF, CC-SAO, and CC-ALF, are all turned off.

[0339] c. In one example, the proposed neural network-based filter is used exclusively with one or more types of NDLF.

[0340] i. In one example, an NN filter can be applied when ALF is disabled.

[0341] ii. In one example, when CC-ALF is disabled, the NN filter can be applied to the chroma component.

[0342] iii. In one example, when CC-SAO is disabled, the NN filter can be applied to the chroma component.

[0343] d. In one example, when neural network-based filtering is applied, the first type of NDLF is determined to be off. Therefore, when neural network-based filtering is enabled, information related to the first type of NDLF will not be signaled.

[0344] i. In one example, with neural network-based filtering enabled, if information related to the first type of NDLF is not signaled, it is inferred that the first type of NDLF is disabled.

[0345] ii. Alternatively, signaling notification may be provided with information related to the first type of NDLF, but such information is required to indicate that no signaling notification of the first type of NDLF is provided when NN-based filtering is enabled.

[0346] e. In one example, the above method can be applied to video units such as CTU, CTB, CTU / CTB lines, strips, slices, pictures, sequences, sub-pictures, etc.

[0347] Example 4

[0348] 4. In the fourth embodiment, the proposed NN filter can be used in conjunction with one or more NDLFs.

[0349] a. In one example, the proposed NN filter can be applied before or after a given NDLF (e.g., DB, SAO, CC-SAO, ALF, CC-ALF).

[0350] b. In one example, for a given video unit (e.g., sequence, picture, sub-picture, strip, slice, CTU, CTB, CU, prediction unit (PU), transform unit (TU), etc.), an NN filter can be applied when no NDLF is applied.

[0351] i. In one example, an NN filter can be applied to samples where a type of NDLF (e.g., ALF, CC-SAO, CC-ALF, etc.) is disabled.

[0352] ii. In one example, for samples where NN filtering is disabled, a type of NDLF (e.g., CC-SAO, ALF, CC-ALF, etc.) can be applied.

[0353] Example 5

[0354] 5. In the fifth embodiment, the interaction between the NN filter and (multiple) NDLFs may depend on the color format and / or color components.

[0355] a. For example, one of the NN filter and NDLF can be used exclusively for the first color format / component, but they can be used together for the second color format / component.

[0356] Example 6

[0357] 6. In the sixth embodiment, the interaction between the NN filter and (multiple) NDLFs may depend on the standard grade, hierarchy, level and / or constraint flags.

[0358] Example 7

[0359] 7. In the seventh embodiment, an NN-based filter, such as a bilateral filter, can be applied after a given loop filtering method.

[0360] Example 8

[0361] 8. In the eighth embodiment, an NN-based filter can be applied prior to a given loop filtering method, which may also include higher-level on / off control.

[0362] a. In one example, a higher level can refer to a strip level, a picture level, a sub-picture level, or a slice level.

[0363] b. In one example, the given loop filtering method could be SAO, CC-SAO, ALF, etc.

[0364] Example 9

[0365] 9. In the ninth embodiment, the application (e.g., using and / or enabling) of NN filter models, or the presence of (multiple) NN filter models, can be controlled by adding one or more syntax elements at a first level (e.g., sequence level, such as in SPS or sequence headers).

[0366] a. One or more syntax elements in the first level may include a first flag indicating whether the NN filter is enabled.

[0367] b. One or more syntax elements in the first level may include syntax elements indicating the number of NN filters that can be used.

[0368] i. The indication of the number of NN filters can only be signaled when the NN filters are enabled (e.g., when the first flag is true).

[0369] ii. The number of N filters can be encoded using the k-th exponent Golomb encoding / decoding.

[0370] 1. In one example, k equals 0 or 1.

[0371] 2. In one example, k can be a positive integer.

[0372] iii. The number of NN filters can be encoded and decoded using fixed-length codes.

[0373] The indication of the number of NN filters can be encoded and decoded using variable-length codes.

[0374] c. One or more syntax elements in the first level may include syntax elements that indicate the type or set of NN filters that can be used.

[0375] i. Syntax elements can be signaled only when the NN filter is enabled (e.g., when the first flag is true).

[0376] d. One or more syntax elements in the first level may include syntax elements that indicate the NN filters to be used in the first level.

[0377] e. One or more syntax elements in the first level may include syntax elements indicating whether NN filters can be adaptively selected in the second level.

[0378] f. Alternatively, one or more syntax elements may be at a second level (e.g., picture level, such as a picture header (PH) or picture parameter set (PPS), or stripe level, such as a stripe header (SH)) for further signaling notification. In the discussion below, PH can be replaced by SH.

[0379] i. Alternatively, further, signaling notification may be conditionally sent to syntax elements at the second level, for example, according to syntax elements for signaling notification at the first level.

[0380] ii. In one example, when the first flag is true, syntax elements related to the NN filter model (e.g., the use / enablement of the NN filter model and / or an indication of which NN filter model) will appear in the PPS or PH (e.g., at the second level).

[0381] 1. In one example, the indicator for the NN filter model of the video unit depends on the syntax element associated with the NN filter model in PPS or PH.

[0382] iii. In one example, when the first flag is false, there will be no syntax elements associated with the NN filter model in the second level, and the NN filter model will not be used for any video unit.

[0383] iv. One or more syntax elements in the second level may include syntax elements indicating the number of N filters that can be used.

[0384] 1. The number of NN filters can be encoded and decoded using the k-th exponent Golomb encoding / decoding.

[0385] a. In one example, k equals 0 or 1.

[0386] b. In one example, k can be a positive integer.

[0387] 2. The number of NN filters can be encoded and decoded using fixed-length codes.

[0388] 3. The number of NN filters can be encoded and decoded using variable-length code.

[0389] v. One or more syntax elements in the second level may include syntax elements that indicate the type or set of NN filters that can be used.

[0390] vi. One or more syntax elements in the second level may include syntax elements that indicate the NN filter to be used in the second level.

[0391] vii. One or more syntax elements in the second level may include syntax elements indicating whether NN filters can be adaptively selected in the third level.

[0392] 1. Alternatively, one or more syntax elements in the second level may include syntax elements indicating whether NN filters can be adaptively selected in the second level.

[0393] viii. One or more syntax elements in the second level may include syntax elements indicating whether NN filters can be applied to the second level.

[0394] ix. One or more syntax elements in the second level may include specific syntax elements that indicate whether the use of NN filters can be controlled in the third level (e.g., enabled or disabled).

[0395] 1. In one example, a specific syntax element is signaled only when it indicates that the NN filter can be applied to the second level.

[0396] x. One or more syntax elements in the second level may include syntax elements indicating whether NN filters are available for the second level.

[0397] 1. In one example, the signaling notifies the syntax element indicating that the NN filter will be used at the second level only when it indicates that the NN filter can be used at the second level.

[0398] 2. In one example, a syntax element indicating whether an NN filter can be adaptively selected at the second level is signaled only when it is indicated that the NN filter can be used at the second level.

[0399] xi. In one example, a syntax element indicating the NN filter to be used at the second level is signaled only when it is indicated that the NN filter can be adaptively selected at the second level.

[0400] xii. In one example, a syntax element indicating the NN filter to be used at the second level is signaled only when it is indicated that the number of NN filters is greater than 1.

[0401] xiii. If a CNN filter is enabled for the second level (e.g., a picture), a syntax element (e.g., picture_cnnlf_set_index[comIdx]) may be signaled. If it is equal to k, where k>=0 and k<T (e.g., T=3), the k-th CNN filter is used for the second level. If it is equal to T, the CNN filter is adaptively selected at the third level.

[0402] 1. Alternatively, or in addition, the syntax element may be restricted to a range, for example [0, number of NN filters – 1].

[0403] g. Alternatively, or in addition, one or more syntax elements at the third level (e.g., patch, CTU, CTB, block, sub-picture, slice, tile, and / or region containing multiple samples) may be further signaled.

[0404] i. Alternatively, or in addition, the syntax element at the third level may be conditionally signaled, for example according to the syntax element(s) signaled at the first level and / or the second level.

[0405] ii. Alternatively, a context may be used to encode and decode the syntax element at the third level.

[0406] iii. In one example, when the first flag is true, the syntax related to the NN filter model (e.g., the use / enable of the NN filter model and / or the indication of which NN filter model) will be present in the PH (picture header) or PPS.

[0407] 1. In one example, a second flag is signaled to indicate whether the NN filter is used at the third level.

[0408] a. In one example, the second flag is signaled only when it is indicated at the second level to control the use (e.g., enable or disable) of the NN filter at the third level.

[0409] b. In one example, the second flag is signaled only if the second level indicates that the NN filter can be adaptively selected at the third level.

[0410] c. In one example, the second flag is signaled only if a specific NN filter is selected at the second level.

[0411] d. In one example, the second flag is signaled only if the second level indicates that the NN filter can be applied to the second level.

[0412] e. In one example, the signaling informs the second flag, regardless of which NN filter is selected at the second level.

[0413] f. In one example, if the first flag is true, the second flag is always signaled.

[0414] g. In one example, if the second level indicates that the NN filter can be applied to the second level, then the signaling always informs the second flag.

[0415] h. Whether the signaling notification second flag depends on at least one of the following: strip or picture type; CTU mode; CTU size; and number of NN filters.

[0416] 2. In one example, the indicator for the NN filter model of the video unit depends on the syntax element associated with the NN filter model in PH or PPS.

[0417] 3. In one example, the indicator of the NN filter model is signaled only if the second flag is true.

[0418] 4. In one example, the indicator of the NN filter model is signaled only when the first or second level indicates that the filter should be adaptively selected at the third level.

[0419] iv. In one example, the signaling notifies the syntax element indicating the NN filter to be used in the third level only if it indicates that the NN filter can be adaptively selected in the third level.

[0420] 1. Alternatively, the signaling notifies the syntax element indicating the NN filter to be used in the third level only if the NN filter was not adaptively selected in the second level.

[0421] v. In one example, a signaling notification is sent only if the number of NN filters indicated is greater than 1, indicating the syntax element of the NN filters that will be used in the third level.

[0422] vi. In one example, when the first flag is false, there will be no syntax related to the NN filter model in the third level, and the NN filter model will not be used for any video unit.

[0423] vii. Alternatively, the choice of context model for encoding / decoding syntax elements may depend on the number of allowed filters.

[0424] 1. In one example, the filter model index for a given color component can use up to K contexts.

[0425] a. In one example, model selection can be defined as Min(binIdx, K-1), where binIdx is the index of the binary bits to be encoded or decoded.

[0426] 2. In one example, the filter model indices of two color components (e.g., U and V) can be encoded and decoded using the same set of context.

[0427] 3. In one example, the filter model indices for two color components (e.g., Y and U, or Y and V) can be encoded and decoded using different sets of context.

[0428] h. In one example, information that would be publicly disclosed in a second-level signaling notification may be provided in a third-level signaling notification.

[0429] i. In one example, the signaling notification in the first area / block / CTU that can be included in the second level is the public information to be signaled in the second level.

[0430] ii. In one example, the public information to be included in the signaling notification at the second level can be in the signaling notification of the first region / block / CTU with NN filtering enabled at the second level.

[0431] Example 10

[0432] 10. In the tenth embodiment, one or more syntax elements may exist for a video unit, and the values ​​of one or more syntax elements may indicate that an NN filter is adaptively applied to samples in the video unit, and / or the same model is applied to samples to which the NN filter is applied.

[0433] a. In one example, there may be only one syntax element, and the following applies:

[0434] i. For values ​​equal to K0 (e.g., K0 equals 0), disable the NN filter for all samples in the video unit.

[0435] ii. For values ​​equal to Ki (e.g., Ki equals M (M != K0)), enable NN filters for all samples in the video unit and apply the same filter model.

[0436] iii. For values ​​equal to Kj (e.g., Kj equals N (N!=K0 and N!=Ki)), adaptively enable the NN filter for all samples in the video unit and apply the same model.

[0437] 1. Alternatively, NN filters can be enabled or disabled for different regions within a video unit.

[0438] 2. Alternatively, the NN filter can be disabled for at least one region within a video unit.

[0439] 3. Alternatively, for regions within a video unit, further signaling can be used to indicate whether the NN filter is enabled or disabled.

[0440] 4. In one example, a video unit can be an image / strip / sub-image.

[0441] 5. In one example, the area within a video unit can be a CTU / CTB or another area containing more than one sample point / pixel.

[0442] b. In one example, multiple syntax elements may exist at the first level (e.g., image / strip), and the following applies:

[0443] i. In one example, the first syntax element (e.g., a 1-bit flag) can be used to indicate whether the NN filter is disabled for all samples in a video unit.

[0444] 1. Alternatively, a second syntax element may exist to indicate whether on / off control exists at the second level.

[0445] ii. Alternatively, the first syntax element (e.g., non-binary) may be used to indicate one of the following:

[0446] 1. Whether the NN filter is disabled for all samples in the video unit; whether all samples share the same model and model index; whether on / off control and / or model index can be further present in a second level (e.g., in CTU).

[0447] c. Alternatively, a second syntax element may exist to indicate whether on / off control exists at a second level (e.g., in CTU).

[0448] i. Alternatively, the second syntax element may be conditionally signaled based on the value of the first syntax element.

[0449] d. Alternatively, whether the signaling notification model index is in the second level (e.g., in CTU) may further depend on the value of one or more syntax elements.

[0450] i. In one example, if the value indicates that the same model is applied to those samples for which the NN filter is applied (whether adaptive or always enabled), then the indication of the model index in the second level is skipped.

[0451] Example 11

[0452] 11. In the eleventh embodiment, one or more syntax elements may exist at the region level to indicate whether all samples to which the NN filter has been applied share the same model, and the region size is no larger than the image / strip (e.g., CTU / CTB).

[0453] a. Alternatively, if one or more syntax elements indicate that the NN filter is applied to all regions, the model index can be further signaled.

[0454] i. In one example, the model index may be signaled before, together with, or with the first CTU that has applied the NN filter.

[0455] b. Alternatively, further signaling notifications may be provided indicating the on / off control of the NN filter for the region.

[0456] c. Alternatively, the region's filter model index may be conditionally signaled based on one or more syntax elements.

[0457] i. In one example, if one or more syntax elements indicate whether all samples to which the NN filter is applied share the same model, the indication of the filter model index for the region can be skipped.

[0458] Example 12

[0459] 12. In the twelfth embodiment, a first syntax element (e.g., alternative_tool_set) may be signaled at the first video unit (e.g., sequence header, SPS, picture header, PPS, strip header, etc.) to indicate the use of at least one optional codec tool.

[0460] a. In one example, a flag in alternative_tool_set indicates whether a particular optional codec tool is used (e.g., 0 indicates not used, 1 indicates used).

[0461] b. In one example, (alternative_tool_set&M) can be used to test whether a specific optional codec tool is used (e.g., (alternative_tool_set&M) equal to 0 indicates not using, (alternative_tool_set&M) equal to 1 indicates using). For example, M = 1, 2, 4, 8, 16, 32, 64, 128 or any 2 K .

[0462] i. In one example, (alternative_tool_set&1) can be used to test whether optional filtering is used (e.g., (alternative_tool_set&1) equal to 0 indicates that it is not used, and (alternative_tool_set&1) equal to 1 indicates that it is used).

[0463] c. In one example, some bits may be reserved.

[0464] i. For example, the highest (or lowest) N bits of alternative_tool_set can be retained, such as N=7.

[0465] d. For example, at least one optional encoding / decoding tool may be an NN-based encoding / decoding tool.

[0466] e. For example, optional filtering could be a neural network-based filtering method.

[0467] f. In one example, the optional encoding / decoding tool is an NN-based encoding / decoding method, such as NN-based superprecision, NN-based interpolation filter, NN-based intra-frame prediction, etc.

[0468] Example 13

[0469] 13. In the thirteenth embodiment, if there is no signaling notification, the above syntax elements can be set to default values ​​(e.g., 0).

[0470] Example 14

[0471] 14. In the fourteenth embodiment, different color components can be signaled separately to the above syntax elements.

[0472] a. For example, syntax elements can be indexed by color components.

[0473] Example 15

[0474] 15. In the fifteenth embodiment, the NN-based filtering or NN filtering method disclosed in this document may also be referred to as "optional filtering".

[0475] Example 16

[0476] 16. In the sixteenth embodiment, at least one syntax element described above can be signaled via context encoding / decoding.

[0477] a. Alternatively, at least one of the above syntax elements can be used for signaling notification via bypass encoding / decoding.

[0478] Example 17

[0479] 17. In the seventeenth embodiment, at least one of the above syntax elements may be binaryed using fixed-length encoding / decoding, unary encoding / decoding, truncated unary encoding / decoding, signed unary encoding / decoding, signed truncated unary encoding / decoding, truncated binary encoding / decoding, exponential Golomb encoding / decoding, or any other binary encoding / decoding method.

[0480] a. For example, syntax elements indicating the number of optional filters (or NN filters) (e.g., num_of_alternative_filter_minus1) can be binary-coded using unary encoding / decoding.

[0481] b. For example, unary encoding and decoding can be used to binary the syntax elements (e.g., picture_alternative_filter_set_index[comIdx]) that indicate optional filters (or NN filters) at the picture level.

[0482] c. For example, unary encoding and decoding can be used to binaryen syntax elements (e.g., picture_alternative_filter_set_index[comIdx]) that indicate optional filters (or NN filters) at the picture level.

[0483] d. For example, unary encoding / decoding can be used to binary the syntax elements (e.g., alternative_filter_lcu_set_index[compIdx][LcuIdx]) that indicate optional filters (or NN filters) in small blocks or CTUs or blocks.

[0484] Example 18

[0485] 18. In the eighteenth embodiment, at least one syntax element disclosed in the document can be used as a hook.

[0486] a. In one example, the decoder can parse syntax elements, but does not perform any further operations based on the syntax elements.

[0487] b. In one example, the decoder can parse the syntax elements, but ignores them.

[0488] c. In one example, the decoder can parse the syntax elements and record them.

[0489] d. In one example, the decoder can parse syntax elements and be used outside the encoding / decoding loop, such as in preprocessing or postprocessing.

[0490] Example 19

[0491] 19. In the nineteenth embodiment, the above method can be applied to any kind of NN-based encoding and decoding method.

[0492] a. In one example, they can be applied to a unified NN filtering method (loop or post-processing).

[0493] b. In one example, they can be applied to non-uniform NN filtering methods (loops or post-processing).

[0494] A first exemplary embodiment was discussed.

[0495] This first exemplary embodiment proposes a loop filtering method based on a convolutional neural network, in which adaptive model selection is introduced. The proposed deep loop filter with adaptive model selection (DAM) is an evolution of existing contributions proposed in JVET-U0068. Several variations and / or modifications are further introduced: porting the algorithm to VTM-11.0, improving the network structure, refining the candidate models, and adaptive inference size. Compared to VTM-11.0, the proposed method demonstrates a reduction in the BD rate of {Y, Cb, Cr} under AI, RA, and LDB configurations.

[0496] In this first exemplary embodiment, a convolutional neural network (CNN) is used to construct a loop filter to reduce distortion occurring during compression. The network structure is a modification of the network structure discussed in Yue Li, Li Zhang, and Kai Zhang, “AHG11: A Convolutional Neural Network-Based Loop Filter with Adaptive Model Selection,” JVET-U0068. Similar to JVET-U0068, residual blocks are used as basic modules and stacked several times to construct the final network. As a further development of JVET-U0068, the presented model is equipped with more residual blocks, but with fewer feature maps in each residual block, thereby increasing representational power at a similar model size. Furthermore, to handle different types of content, the individual networks are trained for different types of stripes and quality levels. In addition, some new features are introduced to further improve encoding and decoding performance.

[0497] First exemplary embodiment and Figure 12A and Figure 12BThe illustrated embodiment is relevant. To increase the receptive field and reduce complexity, the proposed method begins with a convolutional layer with a stride of 2. After this layer, the spatial precision of the feature map is reduced to half the input size in both the horizontal and vertical directions. The feature map output from the first convolutional layer then passes through several sequentially stacked residual blocks. In this first exemplary embodiment, there may be 32 residual blocks instead of... Figure 12A The 16 shown are used as inputs to the last convolutional layer, which takes the feature map from the last residual block as input and produces four N×N feature maps. Finally, a fusion layer is used to generate the filtered image with the same spatial precision as the input to the CNN (e.g., 2N×2N). Other details related to the network architecture are shown below:

[0498] For all convolutional layers, a 3×3 kernel size is used. For the inner convolutional layers, the number of feature maps is M (M=96). For the activation function, PReLU is used.

[0499] Different groups of models were trained using I-band and B-band respectively.

[0500] When training CNN filters for intra-strip, prediction and partitioning information are additionally fed into the network.

[0501] When training CNN filters for inter-frame stripes, prediction information is additionally fed into the network.

[0502] In the technique proposed in the first exemplary embodiment, each strip or block can determine whether to apply a CNN-based filter. When a CNN-based filter is determined to be applied to a strip / block, it can be further determined which model from a candidate model list including three models. For this purpose, a single model is trained with a QP in {17, 22, 27, 32, 37, 42}. Given a test QP, denoted as q, for the encoder-decoder sequence, the candidate model list includes models trained with a QP equal to {q, q-5, q-10}. The selection process is based on the rate-distortion cost at the encoder end. If necessary, signaling instructions for on / off control and model indexes are communicated in the bitstream.

[0503] To achieve higher representational power while maintaining a similar model size, the number of residual blocks was increased from 16 to 32, while the number of feature maps in each residual block was reduced from 128 to 96.

[0504] The granularity of filter determination and model selection depends on accuracy and QP. In the proposed scheme, given higher accuracy and a larger QP, determination and selection can be performed over a larger region.

[0505] The candidate model list differs across different temporal layers. For lower temporal layers, if the ratio of intra-frame codec samples is not less than a threshold, the third candidate is replaced by an intra-frame striped NN filter model. For higher temporal layers, the third candidate, corresponding to the model at q-10, is replaced by the model at q+5.

[0506] For full-frame configurations, model selection is disabled, but on / off control is still retained.

[0507] Open-source machine learning frameworks, such as PyTorch, can be used to perform inference of the CNN filters proposed by VTM. As suggested in S. Liu, A. Segall, E. Alshina, R.-L. Liao, “General Test Conditions and Evaluation Procedures for JVET Based on Neural Networks for Video Coding and Decoding”, JVET-U 2016, Table 1-1 provides network information for the inference phase.

[0508] Table 1-1. Network information tested by NN-based video encoding / decoding tools during the inference phase.

[0509]

[0510] PyTorch can also be used as a training platform. The DIV2K dataset (from https: / / data.vision.ee.ethz.ch / cvl / DIV2K / , R. Timofte, E. Agustsson, S. Gu, J. Wu, A. Ignatov, LVGool) and the BVI-DVC dataset (from Ma, Di, Fan Zhang, and David R. Bull. "BVI-DVC: Training Database for Deep Video Compression," arXiv preprint arXiv:2203.13552(2020)) were used to train I-strip and B-strip CNN filters, respectively. Different CNN models were trained to adapt to different QP points. As suggested in S. Liu, A. Segall, E. Alshina, R.-L. Liao, "General Test Conditions and Evaluation Procedures for Neural Network-Based Video Encoding and Decoding Techniques in JVET," JVET-U 2016, Table 1-2 provides network information for the training phase.

[0511] Table 1-2. Network information from the NN-based video encoding / decoding tool test during the training phase.

[0512]

[0513] Based on the general test conditions recommended in JVET, “General Test Conditions and Evaluation Procedures for Video Coding Techniques Based on Neural Networks”, JVET-U2016, the proposed CNN-based loop filtering method was tested on VTM-11.0 (from https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / - / tags / VTM-11.0). Deblocking filtering and SAO were disabled when ALF (and CCALF) were placed after the proposed CNN-based filtering. Test results are shown in Tables 1-3 to 1-5. Under the AI ​​configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr. Under the RA configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr. Under the LDB configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr.

[0514] Table 1-3. Performance (RA) of the proposed method on VTM11.0

[0515]

[0516] Table 1-4. Performance (LDB) of the proposed method on VTM11.0

[0517]

[0518] Table 1-5. Performance (AI) of the proposed method on VTM11.0

[0519]

[0520] This first exemplary embodiment proposes a CNN-based loop filtering method. The proposed CNN-based method demonstrates useful encoding / decoding gain.

[0521] A second exemplary embodiment is discussed, which provides a conditional loop filtering solution with parameter selection.

[0522] Specifically, this second exemplary method proposes a conditional loop filtering approach. It is developed from existing contributions presented in JVET-U0068. The proposed CNN architecture, conditional on QP-based auxiliary parameters, generates a unified model for processing different quality levels. To better capture local image characteristics, the proposed technique adaptively selects auxiliary parameters at both the block and strip levels. Compared to VTM-11.0, the proposed method demonstrates a reduction in the BD rate of {Y, Cb, Cr} under AI, RA, and LDB configurations.

[0523] In the second exemplary embodiment, a convolutional neural network (CNN) is used to construct a loop filter to reduce distortion generated during compression. The network structure is a modification of the network structure discussed in Yue Li, Li Zhang, and Kai Zhang, “AHG11: A Loop Filter Based on a Convolutional Neural Network with Adaptive Model Selection,” JVET-U0068. Similar to JVET-U0068, residual blocks are used as basic modules and stacked several times to construct the final network. As a further development (or modification) of JVET-U0068, the second exemplary embodiment introduces QP-based auxiliary parameters as input to the network (e.g., to an NN filter), resulting in a unified filter model capable of handling various quality levels. Furthermore, several new features are introduced to further improve encoding and decoding performance.

[0524] The second exemplary embodiment uses the proposed CNN filtering method, such as Figure 17 As shown. Figure 17 This is example architecture 1700 of the proposed CNN filter. The convolutional part is similar to that proposed in JVET-U0068. The process of feeding QPs into the network is twofold. First, the QPs are concatenated into a two-dimensional array with the same size as the block to be filtered. Then, the two-dimensional array is treated as an additional input plane. Second, the QPs are fed into fully connected layers to expand into a one-dimensional vector, whose elements are used as channel scaling factors to recalibrate the feature maps. In example architecture 1700, M represents the number of feature maps (e.g., M = 96 in some examples), and N represents the number of samples in one dimension.

[0525] In a second exemplary embodiment, each strip or block may determine whether to apply a CNN-based filter. When a CNN-based filter is determined to be applied to a strip / block, it can be further determined which auxiliary parameter from a candidate set with three possible auxiliary parameters. Given a test QP for the encoder-decoder sequence, denoted as q, the candidate set includes three auxiliary parameters. For lower and higher time-domain layers, the candidate sets may be predefined as {q, q-5, q-10} and {q, q-5, q+5}, respectively. The selection process is based on the rate-distortion cost at the encoder end. If necessary, signaling instructions for on / off control and the auxiliary parameter index are communicated in the bitstream. Further details regarding parameter selection are listed below.

[0526] 1. The granularity of filter determination and auxiliary parameter selection depends on image precision and QP. In the proposed scheme, given higher precision and a larger QP, determination and selection will be performed over a larger area.

[0527] 2. Intra-strip NN filters can be used for inter-strip. For lower temporal layers, if the ratio of intra-encoding / decoding samples is not less than a threshold, the inter-strip NN filter input with a third auxiliary parameter candidate is replaced by the intra-strip NN filter model.

[0528] 3. For full-frame configuration, parameter selection is disabled, but on / off control is still retained.

[0529] Open-source machine learning frameworks, such as PyTorch, are useful for performing inference on CNN filters proposed by VTM. As suggested in S. Liu, A. Segall, E. Alshina, R.-L. Liao, “General Test Conditions and Evaluation Procedures for JVET Based on Neural Networks for Video Coding and Decoding”, JVET-U 2016, Table 2-1 provides network information for the inference phase.

[0530] Table 2-1. Network information tested by NN-based video encoding / decoding tools during the inference phase.

[0531]

[0532] PyTorch can also be used as a training platform. The DIV2K dataset (from https: / / data.vision.ee.ethz.ch / cvl / DIV2K / , R. Timofte, E. Agustsson, S. Gu, J. Wu, A. Ignatov, LVGool) and the BVI-DVC dataset (from Ma, Di, Fan Zhang, and David R. Bull. "BVI-DVC: Training Database for Deep Video Compression," arXiv preprint arXiv:2203.13552 (2020)) were used to train I-strip and B-strip CNN filters, respectively. A unified CNN model was trained to handle different QP points. As suggested in S. Liu, A. Segall, E. Alshina, R.-L. Liao, "General Test Conditions and Evaluation Procedures for Neural Network-Based Video Encoding and Decoding Techniques in JVET," JVET-U 2016, Table 2-2 provides network information for the training phase.

[0533] Table 2-2. Network information from the NN-based video encoding / decoding tool test during the training phase.

[0534]

[0535] Based on the general test conditions defined above, the proposed CNN-based loop filtering method was tested on VTM-11.0 as described above. Deblocking filtering and SAO were disabled when ALF (and CCALF) were placed after the proposed CNN-based filtering. Test results are shown in Tables 2-3 to 2-5. Under the AI ​​configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr. Under the RA configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr. Under the LDB configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr.

[0536] Table 2-3. Performance (RA) of the proposed method on VTM11.0

[0537]

[0538] Table 2-4. Performance (LDB) of the proposed method on VTM11.0

[0539]

[0540] Table 2-5. Performance (AI) of the proposed method on VTM11.0

[0541]

[0542] This second exemplary embodiment proposes a CNN-based loop filtering method. The proposed CNN-based loop filtering method demonstrates useful encoding / decoding gain.

[0543] A third exemplary embodiment was discussed.

[0544] The third exemplary embodiment proposes a loop filtering method based on a convolutional neural network, in which a uniform model is trained to handle various quality levels. It is an extension and / or modification of the existing contribution JVET-U0068. The proposed CNN architecture uses QP as input. To better capture the local features of the image, the proposed technique enables adaptive model selection at both the block and strip levels. Compared to VTM-11.0, the proposed method demonstrates a reduction in BD-rate for {Y, Cb, Cr} under AI, RA, and LDB configurations.

[0545] In this third exemplary embodiment, a convolutional neural network (CNN) is used to construct a loop filter to reduce distortion occurring during compression. The network structure is a modification of the network structure discussed in Yue Li, Li Zhang, and Kai Zhang, “AHG11: A Convolutional Neural Network-Based Loop Filter with Adaptive Model Selection,” JVET-U0068. Similar to JVET-U0068, residual blocks are used as basic modules and stacked several times to construct the final network. As a further development (or modification) of JVET-U0068, the proposed model takes QP as input, resulting in a unified model capable of handling various quality levels. Furthermore, several new features are introduced to further improve encoding and decoding performance.

[0546] The third exemplary embodiment uses, as Figure 17 The proposed CNN filtering method is shown and described above. To increase the receptive field and reduce complexity, the proposed method begins with a convolutional layer with a stride of 2. After this layer, the spatial precision of the feature map is reduced to half the input size in both the horizontal and vertical directions. The feature map output from the first convolutional layer is then passed through several sequentially stacked residual blocks. The last convolutional layer takes the feature map from the last residual block as input and produces 4 N×N feature maps. Finally, a mixing layer is used to produce the filtered image with the same spatial precision as the CNN input (e.g., 2N×2N). Other details related to the network architecture are shown below:

[0547] 1. Use a 3×3 kernel size for all convolutional layers. For inner convolutional layers, the number of feature maps is 96. Use PReLU as the activation function.

[0548] 2. Train different groups of models for I-band and B-band respectively.

[0549] 3. When training CNN filters for intra-frame stripes, prediction and partitioning information are additionally fed into the network.

[0550] 4. When training CNN filters for inter-frame stripes, prediction information is additionally fed into the network.

[0551] In a third exemplary embodiment, each strip or block can determine whether to apply a CNN-based filter. When a CNN-based filter is determined to be applied to a strip / block, it can be further determined which model from a candidate model list comprising three models can be selected. Thus, given a test QP, denoted as q, for the encoder-decoder sequence, the candidate model list includes three models corresponding to {q, q-5, q-10}. The selection process is based on the rate-distortion cost on the encoder side, and, if necessary, an indication of on / off control and a model index are signaled in the bitstream.

[0552] like Figure 17 As shown, QP is first concatenated into a two-dimensional array with the same size as the block to be filtered, and then treated as an additional input plane. Furthermore, QP is fed into a fully connected layer to expand into a one-dimensional vector, whose elements are used as channel scaling factors to recalibrate the feature maps.

[0553] The choice of granularity for the model depends on accuracy and QP. To achieve higher accuracy and a larger QP, a larger block size needs to be considered.

[0554] The candidate model list differs across different temporal layers. For lower temporal layers, if the ratio of intra-frame codec samples is not less than a threshold, the third candidate is replaced by an intra-frame striped NN filter model. For higher temporal layers, the third candidate, i.e., the q-10 model, is replaced by the q+5 model.

[0555] For full-frame configurations, model selection is disabled, but on / off control is still retained.

[0556] Open-source machine learning frameworks, such as PyTorch, can be used to perform inference of the CNN filters proposed by VTM. As suggested in S. Liu, A. Segall, E. Alshina, R.-L. Liao, “General Test Conditions and Evaluation Procedures for JVET Based on Neural Networks for Video Coding and Decoding”, JVET-U 2016, Table 3-1 provides network information for the inference phase.

[0557] Table 3-1. Network information tested by NN-based video encoding / decoding tools during the inference phase.

[0558]

[0559] PyTorch can also be used as a training platform. The DIV2K dataset (from https: / / data.vision.ee.ethz.ch / cvl / DIV2K / , R. Timofte, E. Agustsson, S. Gu, J. Wu, A. Ignatov, LVGool) and the BVI-DVC dataset (from Ma, Di, Fan Zhang, and David R. Bull. "BVI-DVC: Training Database for Deep Video Compression," arXiv preprint arXiv:2203.13552 (2020)) were used to train I-strip and B-strip CNN filters, respectively. A unified CNN model was trained to handle different QP points. As suggested in S. Liu, A. Segall, E. Alshina, R.-L. Liao, "General Test Conditions and Evaluation Procedures for Neural Network-Based Video Coding and Decoding Techniques in JVET," JVET-U 2016, Table 3-2 provides network information for the training phase.

[0560] Table 3-2. Network information from the NN-based video encoding / decoding tool test during the training phase.

[0561]

[0562] Based on the general test conditions defined above, the proposed CNN-based loop filtering method was tested on VTM-11.0 as described above. Deblocking filtering and SAO were disabled when ALF (and CCALF) were placed after the proposed CNN-based filtering. Test results are shown in Tables 3-3 to 3-5. Under the AI ​​configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr. Under the RA configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr. Under the LDB configuration, the proposed method demonstrates reduced BD rates for Y, Cb, and Cr.

[0563] Table 3-3. Performance (RA) of the proposed method on VTM11.0

[0564]

[0565] Table 3-4. Performance (LDB) of the proposed method on VTM11.0

[0566]

[0567] Table 3-5. Performance (AI) of the proposed method on VTM11.0

[0568]

[0569] A fourth exemplary embodiment is discussed. In this fourth exemplary embodiment, changes, modifications, and / or improvements to the current AVS specification are described. In the tables below, relative to the AVS specification, these changes, modifications, and / or improvements are indicated by italics for deleted text and by underlines for added text.

[0570] Table 4-1. Sequence Headers

[0571]

[0572]

[0573]

[0574]

[0575]

[0576] Table 4-2. Intra-frame image headers

[0577]

[0578]

[0579]

[0580]

[0581] Table 4-3. Inter-frame image headers

[0582]

[0583]

[0584]

[0585]

[0586]

[0587] Table 4-4. Small Pieces

[0588]

[0589]

[0590]

[0591] A fifth exemplary embodiment is discussed. In the fifth exemplary embodiment, changes, modifications, and / or improvements to the current AVS specification are described. In the tables below, relative to the AVS specification, these changes, modifications, and / or improvements are indicated by italics for deleted text and by underlines for added text.

[0592] Table 5-1. Sequence Headers

[0593]

[0594]

[0595]

[0596]

[0597]

[0598] Table 5-2. Intra-frame image headers

[0599]

[0600]

[0601]

[0602]

[0603]

[0604] Table 5-3. Inter-frame image headers

[0605]

[0606]

[0607]

[0608]

[0609]

[0610] Table 5-4. Small blocks or strips

[0611]

[0612]

[0613]

[0614] A sixth exemplary embodiment is discussed. In the sixth exemplary embodiment, changes, modifications, and / or improvements to the current AVS specification are described. In the tables below, relative to the AVS specification, these changes, modifications, and / or improvements are indicated by italics for deleted text and by underlines for added text.

[0615] Table 6-1. Sequence Headers

[0616]

[0617]

[0618]

[0619]

[0620]

[0621] Table 6-2. Intra-frame Prediction Image Headers

[0622]

[0623]

[0624]

[0625]

[0626]

[0627] Table 6-3. Inter-frame Prediction Image Headers

[0628]

[0629]

[0630]

[0631]

[0632]

[0633] Table 6-4. Small blocks or strips

[0634]

[0635]

[0636]

[0637]

[0638] The sequence header is discussed, where the enhanced temporal motion vector prediction and motion vector angle prediction enable flags, `etmvp_mvap_enable_flag`, can be binary variables. A value of "1" indicates that enhanced temporal motion vector prediction and motion vector angle prediction can be used; a value of "0" indicates that enhanced temporal motion vector prediction and motion vector angle prediction should not be used. The value of `etmvpMvapEnableFlag` is equal to the value of `etmvp_mvap_enable_flag`. If `etmvp_mvap_enable_flag` is not present in the bitstream, then the value of `etmvpMvapEnableFlag` is 0.

[0639] In the sixth exemplary embodiment, the optional toolset is represented as `alternative_tool_set`, which can be an 8-bit unsigned integer. This indicates whether the optional toolset is used. If `alternative_tool_set` does not exist in the bitstream, its value should be '00000000'. A value of "1" in the last bit of `alternative_tool_set` indicates that selective filtering can be used; a value of "0" in the last bit indicates that selective filtering should not be used. The value of `AlternativeFilterEnableFlag` is equal to the value of `alternative_tool_set` & 0x01. If `alternative_tool_set` does not exist in the bitstream, its value is equal to 0. The other bits are reserved.

[0640] The number of selective filter models is discussed, which can be represented as num_of_alternative_filter_minus1, indicating the number of filter models available for selective filtering. The resolution process will be further described below. The value of NumOfAlternativeFilter is equal to the value of num_of_alternative_filter_minus1 plus 1. If num_of_alternative_filter_minus1 does not exist in the bitstream, the value of NumOfAlternativeFilter is equal to 0.

[0641] An enhanced adaptive correction filter is discussed, which can be represented as an enable flag `ealf_enable_flag`, which can be a binary variable. A value of "1" indicates that the enhanced adaptive correction filter should be used; a value of "0" indicates that the enhanced adaptive correction filter should not be used. The value of `EalfEnableFlag` is equal to the value of `ealf_enable_flag`. If `ealf_enable_flag` is not present in the bitstream, the value of `EalfEnableFlag` is 0.

[0642] Intra-frame picture headers / inter-frame picture headers are discussed. Picture-level adaptive correction filtering enable flags (e.g., picture_alf_enable_flag[compIdx]) can be binary variables. A value of "1" indicates that adaptive correction filtering can be used for the compIdx-th component of the current image; a value of "0" indicates that adaptive correction filtering should not be used for the compIdx-th component of the current image. The value of PictureAlfEnableFlag[compIdx] is equal to the value of picture_alf_enable_flag[compIdx].

[0643] The image-level selective filtering enable flag (e.g., `picture_alternative_filter_enable_flag[compIdx]`) can be a binary variable. A value of "1" indicates that the `compIdx`-th component of the current image can be filtered by the neural network; a value of "0" means that the `compIdx`-th component of the current image should not be filtered by the neural network. The value of `PictureAlternativeFilterEnableFlag[compIdx]` is equal to the value of `picture_alternative_filter_enable_flag[compIdx]`. If `picture_alternative_filter_enable_flag[compIdx]` is not present in the bitstream, then the value of `PictureAlternativeFilterEnableFlag[compIdx]` is 0.

[0644] In this sixth exemplary embodiment, the picture-level selective filter adaptation flag (e.g., picture_alternative_filter_adaptive_flag[compIdx]) can be a binary variable. A value of "1" indicates that adaptive selective filtering can be used for the compIdx-th component of the current image; a value of "0" indicates that adaptive selective filtering should not be used for the compIdx-th component of the current image. The value of PictureAlternativeFilterAdaptiveFlag[compIdx] is equal to the value of picture_alternative_filter_adaptive_flag[compIdx]. If picture_alternative_filter_adaptive_flag[compIdx] does not exist in the bitstream, then the value of PictureAlternativeFilterAdaptiveFlag[compIdx] is 0.

[0645] The image-level selective filter model index (e.g., `picture_alternative_filter_set_index[compIdx]`) indicates the model index of the `compIdx`-th component of the current image using selective filtering. The value of `PictureAlternativeFilterSetIndex[compIdx]` is equal to the value of `picture_alternative_filter_set_index[compIdx]`. The parsing process will be further described below. If `picture_alternative_filter_set_index[compIdx]` does not exist in the bitstream, the value of `PictureAlternativeFilterSetIndex[compIdx]` is -1.

[0646] The image-level intra-block copy prediction enable flag (e.g., `picture_ibc_enable_flag`) can be a binary variable. A value of '1' indicates that the current image can use intra-block copy prediction; a value of '0' indicates that the current image should not use intra-block copy prediction. The value of `PictureIbcEnableFlag` is equal to the value of `picture_ibc_enable_flag`. If `picture_ibc_enable_flag` is not present in the bitstream, the value of `PictureIbcEnableFlag` is 0.

[0647] The description further details slices and / or stripes. The maximum codec unit adaptive correction filter enable flag (e.g., alf_lcu_enable_flag[compIdx][LcuIdx]) can be a binary variable. A value of "1" indicates that the sample of the LcuIdx-th LCU compIdx component should use adaptive correction filtering; a value of "0" indicates that the sample of the LcuIdx-th LCU compIdx component should not use adaptive correction filtering. The value of AlfLCUEnableFlag[compIdx][LcuIdx] is equal to the value of alf_lcu_enable_flag[compIdx][LcuIdx].

[0648] The maximum codec unit selective filtering enable flag (e.g., alternative_filter_lcu_enable_flag[compIdx][LcuIdx]) can be a binary variable. A value of "1" indicates that the sample of the LcuIdx-th maximum codec unit compIdx component should use selective filtering; a value of "0" indicates that the sample of the LcuIdx-th maximum codec unit compIdx component should not use neural network filtering. The resolution process will be further described below. The value of AlternativeFilterLcuEnableFlag[compIdx][LcuIdx] is equal to the value of alternative_filter_lcu_enable_flag[compIdx][LcuIdx]. If alternative_filter_lcu_enable_flag[compIdx][LcuIdx] does not exist in the bitstream, then the value of AlternativeFilterLcuEnableFlag[compIdx][LcuIdx] is 0.

[0649] In this sixth exemplary embodiment, the maximum codec unit neural network filter model index (e.g., alternative_filter_lcu_set_index[compIdx][LcuIdx]) indicates the model index of the selective filter used for the samples of the maximum codec unit component LcuIdx (the resolution process is further described below), and its value ranges from 0 to NumOfAlternativeFilter-1. The value of AlternativeFilterLCUSetIndex[compIdx][LcuIdx] is equal to the value of alternative_filter_lcu_set_index[compIdx][LcuIdx]. If alternative_filter_lcu_set_index[compIdx][LcuIdx] does not exist in the bitstream, then the value of AlternativeFilterLCUSetIndex[compIdx][LcuIdx] is -1.

[0650] The padding bits (e.g., aec_lcu_stuffing_bit) of the maximum codec unit in the high-entropy codec include one or more padding bits. The value of aec_lcu_stuffing_bit of the last maximum codec unit of the stripe should be '1', and the resolution process will be described further below.

[0651] The various parsing processes are now described. When parsing a k-order exponential Golomb code, the first non-zero bit is found from the current position in the bitstream, and the number of zero bits found is recorded as leadingZeroBits. Then, CodeNum is calculated based on leadingZeroBits. This can be described in pseudocode as follows:

[0652]

[0653] Table 6-5 shows the structures of exponential Golomb codes of orders 0, 1, 2, and 3. The bit binary string of an exponential Golomb code consists of a "prefix" and a "suffix". The prefix is ​​composed of consecutive "0"s and "1"s (leadingZeroBits). The suffix is ​​composed of leadingZeroBits + k bits, which is the binary string xi in the table, where xi has a value of '0' or '1'.

[0654] Table 6-5. k-order exponential Golomb codes

[0655]

[0656] When parsing the grammatical elements described by ue(v) and se(v), zero-order exponent Golomb codes are used, and the parsing process is as follows:

[0657] ue(v): The value of the syntax element is equal to CodeNum

[0658] se(v): Calculate the value of the syntax element according to the mapping relationship of signed exponent Golomb codes given in Table 6-6.

[0659] Table 6-6. Mapping relationship between se(v) and CodeNum

[0660] 0 0 1 1 2 –1 3 2 4 –2 5 3 6 –3 k <![CDATA[(–1) k+1 ×Ceil(k÷2)]]>

[0661] When parsing the syntax elements described by ae(v), all binary symbol models and entropy codecs are first initialized before parsing the stripes, as described below. Then, the binary symbol strings are parsed sequentially from the bitstream, as described below. Finally, the values ​​of the syntax elements described by ae(v) are obtained from the binary symbol strings, as described below.

[0662] Initialize the entropy codec if any of the following conditions exist.

[0663] Conditions: The current codec unit contains only the luma component, and IntraLumaPredMode is equal to "Intra_Luma_PCM", or the current codec unit contains both luma and chroma components, and IntraLumaPredMode is equal to "Intra_Luma_PCM", and IntraChromaPredMode is not equal to "Intra_Chroma_PCM". After parsing pcm_coeff of the luma coded block, initialize the entropy codec.

[0664] Conditions: The current codec unit contains only chroma components, and IntraChromaPredMode is equal to "Intra_Chroma_PCM", or the current codec unit contains both luma and chroma components, and IntraLumaPredMode is equal to "Intra_Luma_PCM" and IntraChromaPredMode is equal to "Intra_Chroma_PCM". After parsing pcm_coeff of the chroma codec block, initialize the entropy codec.

[0665] Table 8.3.3.2.1 - Deriving the Binary Symbolic Model (Referencing each section in the AVS specification)

[0666]

[0667]

[0668]

[0669]

[0670] Table 8.3.3.3.1 - Parsing Process (Refer to the AVS specification for references to each chapter)

[0671]

[0672]

[0673]

[0674]

[0675]

[0676]

[0677] The levels defined in this chapter are shown in Table B.2.

[0678] 0x00 prohibit 0x20 Baseline 8-bit level (primary 8-bit level) 0x22 Baseline 10-bit level (Main 10-bit level) 0x30 Enhanced 8-bit level (higher 8-bit tier) 0x32 Enhanced by 10 bits (10-bit higher tier) other reserve

[0679] For a given grade, different levels support the same subset of syntax.

[0680] A base 8-bit bitstream should meet the following conditions:

[0681] The value of profile_id should be 0x20.

[0682] The value of progressive_sequence is "1".

[0683] The value of chroma_format should be "01".

[0684] The value of sample_precision should be "001".

[0685] Between the start code of a video sequence and the end code of the first video sequence, or between the start code of a video sequence and the first video editing code, the PictureStructure value of all encoded and decoded pictures should be the same.

[0686] The progressive_frame value for all encoded and decoded images should be the same between the start code of a video sequence and the end code of the first subsequent video sequence, or between the start code of a video sequence and the first subsequent video edit code.

[0687] The value of MiniSize should be 8.

[0688] The value of log2_lcu_size_minus2 should be in the range of 3 to 5.

[0689] The value of log2_min_cu_size_minus2 should be 0.

[0690] The value of patch_width_minus1 should be PictureWidthInLCU-1.

[0691] The value of stable_patch_flag should be "1".

[0692] The value of ref_colocated_patch_flag should be "0".

[0693] The value of uniform_patch_flag should be "1".

[0694] Small patches belonging to the same image should appear in ascending order based on the patch_index value.

[0695] If any side of the codec block is longer than 64, you can only choose either skip mode or inter-frame prediction mode for the codec block. If inter-frame prediction mode is selected, the value of ctp_zero_flag should be '1'.

[0696] If the number of luminance samples in the current codec unit is less than 64, the current codec unit should not use skip mode or direct mode, and the prediction reference mode of the prediction unit of the current codec unit should not be "PRED_List01".

[0697] The value of MaxSplitTimes should be 6.

[0698] The value of MaxPartRatio should be 8.

[0699] The value of log2_max_bt_size_minus2 should be 4 or 5.

[0700] If the width and height of the LCU of the image are both equal to 128, then the LCU should be partitioned using a quadtree and the value of qt_split_flag should be "1".

[0701] If the width of a codec tree node is equal to 128 and the height is equal to 64, and the codec tree node is split, then the codec tree node should be split using a vertical binary tree, and the value of bet_split_dir_flag should be "1".

[0702] If the width of a codec tree node is 64 and the height is 128, and the codec tree node is split, then the codec tree node should be split using a horizontal binary tree, and the value of bet_split_dir_flag should be "0".

[0703] If the library_stream_flag of the current sequence is "1", then all images in the sequence should be I-images.

[0704] If the syntax element duplicate_sequence_header_flag exists in the bitstream, then the value of this syntax element should be "1".

[0705] The decoded image buffer can contain at most one knowledge image.

[0706] The value of NumOfUpdatedLibrary should be 1.

[0707] The value of MinLibraryUpdatePeriod should be 1.

[0708] The value of MAX_TEMPORAL_ID should be 7.

[0709] The classification restrictions specified in B.3.2.

[0710] Supported versions include: 2.0.15, 2.0.30, 2.0.60, 4.0.30, 4.0.60, 6.0.30, 6.2.30, 6.0.60, 6.2.60, 6.0.120, 6.2.120, 8.0.30, 8.2.30, 8.0.60, 8.2.60, 8.0.120, 8.2.120, 10.0.30, 10.2.30, 10.0.60, 10.2.60, 10.0.120, and 10.2.120.

[0711] A base 10-bit bitstream should meet the following conditions:

[0712] The value of profile_id should be 0x22.

[0713] The value of progressive_sequence should be "1".

[0714] The value of chroma_format should be "01".

[0715] The value of sample_precision should be "001" or "010".

[0716] The value of encoding_precision should be "001" or "010".

[0717] Between the start code of a video sequence and the end code of the first video sequence, or between the start code of a video sequence and the first video editing code, the PictureStructure value of all encoded and decoded pictures should be the same.

[0718] The progressive_frame value for all encoded and decoded images should be the same between the start code of a video sequence and the end code of the first subsequent video sequence, or between the start code of a video sequence and the first subsequent video edit code.

[0719] The value of MiniSize should be 8.

[0720] The value of log2_lcu_size_minus2 should be in the range of 3 to 5.

[0721] The value of log2_min_cu_size_minus2 should be 0.

[0722] The value of patch_width_minus1 should be PictureWidthInLCU-1.

[0723] The value of stable_patch_flag should be "1".

[0724] The value of ref_colocated_patch_flag should be "0".

[0725] The value of uniform_patch_flag should be "1".

[0726] Small patches belonging to the same image should appear in ascending order based on the patch_index value.

[0727] If any side of the codec block is longer than 64, you can only choose either skip mode or inter-frame prediction mode for the codec block. If inter-frame prediction mode is selected, the value of ctp_zero_flag should be '1'.

[0728] If the number of luminance samples in the current codec unit is less than 64, the current codec unit should not use skip mode or direct mode, and the prediction reference mode of the prediction unit of the current codec unit should not be "PRED_List01".

[0729] The value of MaxSplitTimes should be 6.

[0730] The value of MaxPartRatio should be 8.

[0731] The value of log2_max_bt_size_minus2 should be 4 or 5.

[0732] If the width and height of the LCU of the image are both equal to 128, then the LCU should be partitioned using a quadtree and the value of qt_split_flag should be "1".

[0733] If the width of a codec tree node is equal to 128 and the height is equal to 64, and the codec tree node is split, then the codec tree node should be split using a vertical binary tree, and the value of bet_split_dir_flag should be "1".

[0734] If the width of a codec tree node is 64 and the height is 128, and the codec tree node is split, then the codec tree node should be split using a horizontal binary tree, and the value of bet_split_dir_flag should be "0".

[0735] If the library_stream_flag of the current sequence is "1", then all images in the sequence should be I-images.

[0736] If the syntax element duplicate_sequence_header_flag exists in the bitstream, then the value of this syntax element should be "1".

[0737] The decoded image buffer can contain at most one knowledge image.

[0738] The value of NumOfUpdatedLibrary should be 1.

[0739] The value of MinLibraryUpdatePeriod should be 1.

[0740] The value of MAX_TEMPORAL_ID should be 7.

[0741] Classification restrictions specified in B.3.2

[0742] Supported versions include: 2.0.15, 2.0.30, 2.0.60, 4.0.30, 4.0.60, 6.0.30, 6.2.30, 6.0.60, 6.2.60, 6.0.120, 6.2.120, 8.0.30, 8.2.30, 8.0.60, 8.2.60, 8.0.120, 8.2.120, 10.0.30, 10.2.30, 10.0.60, 10.2.60, 10.0.120, and 10.2.120.

[0743] To enhance an 8-bit bitstream, the following conditions must be met:

[0744] The value of profile_id should be 0x30.

[0745] The value of progressive_sequence should be "1".

[0746] The value of chroma_format should be "01".

[0747] The value of sample_precision should be "001".

[0748] Between the start code of a video sequence and the end code of the first video sequence, or between the start code of a video sequence and the first video editing code, the PictureStructure value of all encoded and decoded pictures should be the same.

[0749] The progressive_frame value for all encoded and decoded images should be the same between the start code of a video sequence and the end code of the first subsequent video sequence, or between the start code of a video sequence and the first subsequent video edit code.

[0750] The value of MiniSize should be 8.

[0751] The value of log2_lcu_size_minus2 should be in the range of 3 to 5.

[0752] The value of log2_min_cu_size_minus2 should be 0.

[0753] The value of stable_patch_flag should be "1".

[0754] The value of ref_colocated_patch_flag should be "0".

[0755] The value of uniform_patch_flag should be "1".

[0756] Small patches belonging to the same image should appear in ascending order based on the patch_index value.

[0757] If any side of the codec block is longer than 64, you can only choose either skip mode or inter-frame prediction mode for the codec block. If inter-frame prediction mode is selected, the value of ctp_zero_flag should be '1'.

[0758] If the number of luminance samples in the current codec unit is less than 64, the current codec unit should not use skip mode or direct mode, and the prediction reference mode of the prediction unit of the current codec unit should not be "PRED_List01".

[0759] The value of MaxSplitTimes should be 6.

[0760] The value of MaxPartRatio should be 8.

[0761] The value of log2_max_bt_size_minus2 should be 4 or 5.

[0762] If the width and height of the LCU of the image are both equal to 128, then the LCU should be partitioned using a quadtree and the value of qt_split_flag should be "1".

[0763] If the width of a codec tree node is equal to 128 and the height is equal to 64, and the codec tree node is split, then the codec tree node should be split using a vertical binary tree, and the value of bet_split_dir_flag should be "1".

[0764] If the width of a codec tree node is 64 and the height is 128, and the codec tree node is split, then the codec tree node should be split using a horizontal binary tree, and the value of bet_split_dir_flag should be "0".

[0765] If the library_stream_flag of the current sequence is "1", then all images in the sequence should be I-images.

[0766] If the syntax element duplicate_sequence_header_flag exists in the bitstream, then the value of this syntax element should be "1".

[0767] The decoded image buffer can contain at most one knowledge image.

[0768] The value of NumOfUpdatedLibrary should be 1.

[0769] The value of MinLibraryUpdatePeriod should be 1.

[0770] The value of MAX_TEMPORAL_ID should be 7.

[0771] Classification restrictions specified in B.3.2

[0772] Supported versions include: 2.0.15, 2.0.30, 2.0.60, 4.0.30, 4.0.60, 6.0.30, 6.2.30, 6.4.30, 6.6.30, 6.0.60, 6.2.60, 6.4.60, 6.6.60, 6.0.120, 6.2.120, 6.4.120, 6.6.120, 8.0.30, 8.2.30, 8.4.30, 8.6.30, 8 0.60,8.2.60,8.4.60,8.6.60,8.0.120,8.2.120,8.4.120,8.6.120,10.0.30,10.2.30,10.4.30,10.6.30,10.0.60,10.2.60,10.4.60,10.6.60,10.0.120,10.2.120,10.4.120,and10.6.120.

[0773] To enhance a 10-bit bitstream, the following conditions must be met:

[0774] The value of profile_id should be 0x32.

[0775] The value of progressive_sequence should be "1".

[0776] The value of chroma_format should be "01".

[0777] The value of sample_precision should be "001" or "010".

[0778] The value of encoding_precision should be "001" or "010".

[0779] Between the start code of a video sequence and the end code of the first video sequence, or between the start code of a video sequence and the first video editing code, the PictureStructure value of all encoded and decoded pictures should be the same.

[0780] The progressive_frame value for all encoded and decoded images should be the same between the start code of a video sequence and the end code of the first subsequent video sequence, or between the start code of a video sequence and the first subsequent video edit code.

[0781] The value of MiniSize should be 8.

[0782] The value of log2_lcu_size_minus2 should be in the range of 3 to 5.

[0783] The value of log2_min_cu_size_minus2 should be 0.

[0784] The value of stable_patch_flag should be "1".

[0785] The value of ref_colocated_patch_flag should be "0".

[0786] The value of uniform_patch_flag should be "1".

[0787] Small patches belonging to the same image should appear in ascending order based on the patch_index value.

[0788] If any side of the codec block is longer than 64, you can only choose either skip mode or inter-frame prediction mode for the codec block. If inter-frame prediction mode is selected, the value of ctp_zero_flag should be '1'.

[0789] If the number of luminance samples in the current codec unit is less than 64, the current codec unit should not use skip mode or direct mode, and the prediction reference mode of the prediction unit of the current codec unit should not be "PRED_List01".

[0790] The value of MaxSplitTimes should be 6.

[0791] The value of MaxPartRatio should be 8.

[0792] The value of log2_max_bt_size_minus2 should be 4 or 5.

[0793] If the width and height of the LCU of the image are both equal to 128, then the LCU should be partitioned using a quadtree and the value of qt_split_flag should be "1".

[0794] If the width of a codec tree node is equal to 128 and the height is equal to 64, and the codec tree node is split, then the codec tree node should be split using a vertical binary tree, and the value of bet_split_dir_flag should be "1".

[0795] If the width of a codec tree node is 64 and the height is 128, and the codec tree node is split, then the codec tree node should be split using a horizontal binary tree, and the value of bet_split_dir_flag should be "0".

[0796] If the library_stream_flag of the current sequence is "1", then all images in the sequence should be I-images.

[0797] If the syntax element duplicate_sequence_header_flag exists in the bitstream, then the value of this syntax element should be "1".

[0798] The decoded image buffer can contain at most one knowledge image.

[0799] The value of NumOfUpdatedLibrary should be 1.

[0800] The value of MinLibraryUpdatePeriod should be 1.

[0801] The value of num_of_alternative_filter_minus1 should be in the range of 0 to 3.

[0802] The value of MAX_TEMPORAL_ID should be 7.

[0803] Classification restrictions specified in B.3.2

[0804] Supported versions include: 2.0.15, 2.0.30, 2.0.60, 4.0.30, 4.0.60, 6.0.30, 6.2.30, 6.4.30, 6.6.30, 6.0.60, 6.2.60, 6.4.60, 6.6.60, 6.0.120, 6.2.120, 6.4.120, 6.6.120, 8.0.30, 8.2.30, 8.4.30, 8.6.30, 8 0.60, 8.2.60, 8.4.60, 8.6.60, 8.0.120, 8.2.120, 8.4.120, 8.6.120, 10.0.30, 10.2.30, 10.4.30, 10.6.30, 10.0.60, 10.2.60, 10.4.60, 10.6.60, 10.0.120, 10.2.120, 10.4.120, and 10.6.120.

[0805] A seventh exemplary embodiment is discussed. In the seventh exemplary embodiment, changes, modifications, and / or improvements to the current AVS specification are described. In the tables below, relative to the AVS specification, these changes, modifications, and / or improvements are indicated by italics for deleted text and by underlines for added text.

[0806] Table 7-1. Sequence Headers

[0807]

[0808]

[0809]

[0810]

[0811] Table 7-2. Intra-frame image headers

[0812]

[0813]

[0814]

[0815]

[0816] Table 7-3. Inter-frame image headers

[0817]

[0818]

[0819]

[0820]

[0821]

[0822] Table 7-4. Small blocks or strips

[0823]

[0824]

[0825]

[0826] The sequence header is discussed further. The enhanced sample offset compensation enable flag (e.g., esao_enable_flag) can be a binary variable. A value of "1" indicates that enhanced sample offset compensation can be used; a value of "0" indicates that enhanced sample offset compensation should not be used. The value of EsaoEnableFlag is equal to the value of esao_enable_flag. If esao_enable_flag is not present in the bitstream, the value of EsaoEnableFlag is 0. If the value of EsaoEnableFlag is 1, then the value of SaoEnableFlag should be 0.

[0827] The enhanced temporal motion vector prediction and motion vector angle prediction enable flags (e.g., etmvp_mvap_enable_flag) can be binary variables. A value of "1" indicates that enhanced temporal motion vector prediction and motion vector angle prediction can be used; a value of "0" indicates that enhanced temporal motion vector prediction and motion vector angle prediction should not be used. The value of EtmvpMvapEnableFlag is equal to the value of etmvp_mvap_enable_flag. If etmvp_mvap_enable_flag is not present in the bitstream, the value of EtmvpMvapEnableFlag is 0.

[0828] In the seventh exemplary embodiment, the neural network toolset (e.g., nn_tools_set_hook) may be an 8-bit unsigned integer. This identifies whether a neural network is being used. If nn_tools_set_hook is not present in the bitstream, its value should be '00000000'. These bits should be ignored during the decoding process.

[0829] The last bit of `nn_tools_set_hook` is '1', indicating that neural network filtering can be used; the last bit is '0', indicating that neural network filtering should not be used. The value of `NnFilterEnableFlag` is equal to the value of `nn_tools_set_hook` & 0x01. If `nn_tools_set_hook` is not present in the bitstream, the value of `NnFilterEnableFlag` is 0. Other bits are reserved.

[0830] The number of neural network filter models, `num_of_nn_filter_minus1`, indicates the number of filter models available for neural network filtering. An example of the parsing process is described further below. The value of `NumOfNnFilter` is equal to the value of `num_of_nn_filter_minus1` plus 1. If `num_of_nn_filter_minus1` is not present in the bitstream, the value of `NumOfNnFilter` is equal to 0. These bits should be ignored during the decoding process.

[0831] The enhanced adaptive correction filter enable flag (e.g., ealf_enable_flag) can be a binary variable. A value of "1" indicates that the enhanced adaptive correction filter should be used; a value of "0" indicates that the enhanced adaptive correction filter should not be used. The value of EalfEnableFlag is equal to the value of ealf_enable_flag. If ealf_enable_flag is not present in the bitstream, the value of EalfEnableFlag is 0.

[0832] The image reordering delay (e.g., `output_reorder_delay`) can be a 5-bit unsigned integer. It represents the reordering delay caused by inconsistencies between the image encoding / decoding order and the display order, measured in units of decoded images. Because the display time of decoded images is related to the values ​​of syntax elements such as `progressive_sequence`, `progressive_frame`, `repeat_first_field`, and `picture_structure`, the absolute length of this time is not fixed, but the number of decoded images displayed within this time is constant. When `low_delay` is '0', the value of `OutputReorderDelay` is equal to the value of `output_reorder_delay`; when `low_delay` is '1', the value of `OutputReorderDelay` is 0.

[0833] Now let's discuss intra-frame image headers / inter-frame image headers. The weighted quantization parameter index (e.g., weight_quant_param_index) can be a 2-bit unsigned integer. The weighted quantization parameter index for the current image. Reserved value "11".

[0834] The weighted quantization matrix model (e.g., weight_quant_model) can be a 2-bit unsigned integer specifying the distribution model of the weighted quantization parameters. The value "11" is a reserved value. If weight_quant_model exists in the bitstream of the current image header, the value of WeightQuantModel is equal to the value of weight_quant_model.

[0835] The weighted quantization parameter delta 1 is weight_quant_param_delta1[i]. The weighted quantization parameter delta 2 is weight_quant_param_delta2[i].

[0836] The increment of the weighted quantization parameter for the current image, ranging from -128 to 127. See below for the parsing process. For the decoding process, see section 9.2.6 of the AVS specification. If weight_quant_param_delta1[i] or weight_quant_param_delta2[i] does not exist in the bitstream of the current image header, then the value of weight_quant_param_delta1[i] or weight_quant_param_delta2[i] is 0.

[0837] The image-level adaptive correction filtering enable flag (e.g., picture_alf_enable_flag[compIdx]) can be a binary variable. A value of "1" indicates that adaptive correction filtering can be used for the compIdx-th component of the current image; a value of "0" indicates that adaptive correction filtering should not be used for the compIdx-th component of the current image. The value of PictureAlfEnableFlag[compIdx] is equal to the value of picture_alf_enable_flag[compIdx].

[0838] In the seventh exemplary embodiment, the picture-level neural network filtering enable flag (e.g., picture_nn_filter_enable_flag[compIdx]) can be a binary variable. A value of "1" indicates that the compIdx-th component of the current image can be filtered by the neural network; a value of "0" means that the compIdx-th component of the current image should not be filtered by the neural network. The value of PictureNnFilterEnableFlag[compIdx] is equal to the value of picture_nn_filter_enable_flag[compIdx]. If picture_nn_filter_enable_flag[compIdx] does not exist in the bitstream, then the value of PictureNnFilterEnableFlag[compIdx] is 0. These bits should be ignored during the decoding process.

[0839] The image-level selective filtering adaptive flag (e.g., `picture_nn_filter_adaptive_flag[compIdx]`) can be a binary variable. A value of "1" indicates that the `compIdx`-th component of the current image can use adaptive neural network filtering; a value of "0" indicates that the `compIdx`-th component of the current image should not use adaptive neural network filtering. The value of `PictureNnFilterAdaptiveFlag[compIdx]` is equal to the value of `picture_nn_filter_adaptive_flag[compIdx]`. If `picture_nn_filter_adaptive_flag[compIdx]` is not present in the bitstream, then the value of `PictureNnFilterAdaptiveFlag[compIdx]` is 0. These bits should be ignored during the decoding process.

[0840] The image-level god-level network filter model index (e.g., picture_nn_filter_set_index[compIdx]) represents the model index of the compIdx component of the current image filtered using a neural network. See the parsing procedure below; values ​​range from 0 to NumOfNnFilter-1. The value of PictureNnFilterSetIndex[compIdx] is equal to the value of picture_nn_filter_set_index[compIdx]. If picture_nn_filter_set_index[compIdx] does not exist in the bitstream, the value of PictureNnFilterSetIndex[compIdx] is -1. These bits should be ignored during decoding.

[0841] The image-level intra-block copy prediction enable flag (e.g., `picture_ibc_enable_flag`) can be a binary variable. A value of '1' indicates that the current image can use intra-block copy prediction; a value of '0' indicates that the current image should not use intra-block copy prediction. The value of `PictureIbcEnableFlag` is equal to the value of `picture_ibc_enable_flag`. If `picture_ibc_enable_flag` is not present in the bitstream, the value of `PictureIbcEnableFlag` is 0.

[0842] Now let's discuss pieces. The LCU cross-component sample offset compensation set index (e.g., ccsao_lcu_set_index[compIdx][LcuIdx]) indicates the cross-component sample offset compensation group that should be used for the samples of the LcuIdx maximum codec unit compIdx component, as explained below. The value of CcsaoLcuSetIndex[compIdx][LcuIdx] is equal to the value of ccaso_lcu_set_idx[compIdx][LcuIdx]. The value of ccaso_lcu_set_idx[compIdx][LcuIdx] ranges from 0 to 3. If ccaso_lcu_set_idx[compIdx][LcuIdx] does not exist in the bitstream, then the value of CcsaoLcuSetIndex[compIdx][LcuIdx] is 0.

[0843] The maximum codec unit adaptive correction filter enable flag (e.g., alf_lcu_enable_flag[compIdx][LcuIdx]) can be a binary variable. A value of "1" indicates that the sample of the LcuIdx-th LCU compIdx component should use adaptive correction filtering; a value of "0" indicates that the sample of the LcuIdx-th LCU compIdx component should not use adaptive correction filtering. The value of AlfLCUEnableFlag[compIdx][LcuIdx] is equal to the value of alf_lcu_enable_flag[compIdx][LcuIdx]

[0844] In this seventh exemplary embodiment, the maximum codec unit neural network filter enable flag (e.g., nn_filter_lcu_enable_flag[compIdx][LcuIdx]) can be a binary variable. A value of "1" indicates that a neural network should be used to filter the samples of the LcuIdx-th maximum codec unit compIdx component; a value of "0" indicates that a neural network should not be used to filter the samples of the LcuIdx-th maximum codec unit compIdx component. The parsing process will be further described below. The value of NnFilterLcuEnableFlag[compIdx][LcuIdx] is equal to the value of nn_filter_lcu_enable_flag[compIdx][LcuIdx]. If nn_filter_lcu_enable_flag[compIdx][LcuIdx] does not exist in the bitstream, then the value of NnFilterLcuEnableFlag[compIdx][LcuIdx] is 0. These bits should be ignored during the decoding process.

[0845] The model index of the neural network filter for the largest encoder-decoder unit (e.g., nn_filter_lcu_set_index[compIdx][LcuIdx]) indicates the model index of the neural network filter used for the sample of the LcuIdx-th largest encoder-decoder unit (compIdx component). The resolution process is described in 8.3, and the value ranges from 0 to NumOfNnFilter-1. The value of NnFilterLCUSetIndex[compIdx][LcuIdx] is equal to the value of nn_filter_lcu_set_index[compIdx][LcuIdx]. If nn_filter_lcu_set_index[compIdx][LcuIdx] does not exist in the bitstream, then the value of NnFilterLCUSetIndex[compIdx][LcuIdx] is -1. These bits should be ignored during the decoding process.

[0846] The padding bit (e.g., aec_lcu_stuffing_bit) of the maximum codec unit in the high-entropy codec can include padding bits. The value of aec_lcu_stuffing_bit of the last maximum codec unit of the stripe should be '1', and the parsing process will be described further below.

[0847] A small block end code (such as patch_end_code) may include the bit string "0x0000018F" and mark or indicate the end of the strip.

[0848] The various parsing processes are now described. When parsing a k-order exponential Golomb code, the first non-zero bit is found from the current position in the bitstream, and the number of zero bits found is recorded as leadingZeroBits. Then, CodeNum is calculated based on leadingZeroBits. This is described in pseudocode as follows:

[0849]

[0850] Table 7-5 shows the structures of exponential Golomb codes of order 0, 1, 2, and 3. The bit string of an exponential Golomb code consists of a prefix and a suffix. The prefix is ​​composed of consecutive leadingZeroBits of 0s and 1s. The suffix is ​​composed of leadingZeroBits + k bits, which is the binary string xi in the table, where xi has a value of '0' or '1'.

[0851] Table 7-5. k-order exponential Golomb codes

[0852]

[0853]

[0854] When parsing the grammatical elements described by ue(v) and se(v), zero-order exponent Golomb codes are used, and the parsing process is as follows:

[0855] ue(v): The value of the syntax element is equal to CodeNum;

[0856] se(v): Calculate the value of the syntax element according to the mapping relationship of signed exponent Golomb codes given in Table 7-6.

[0857] Table 7-6. Mapping relationship between se(v) and CodeNum

[0858] 0 0 1 1 2 –1 3 2 4 –2 5 3 6 –3 k <![CDATA[(–1) k+1 ×Ceil(k÷2)]]>

[0859] When parsing the syntax elements described by ae(v), all binary symbol models and entropy codecs are first initialized before parsing the stripes, as described below. Then, the binary symbol strings are parsed sequentially from the bitstream, as described below. Finally, the values ​​of the syntax elements described by ae(v) are obtained from the binary symbol strings, as described below.

[0860] Initialize the entropy codec if any of the following conditions exist (see 8.3.2.2).

[0861] If the current codec unit contains only the luma component and IntraLumaPredMode is equal to "Intra_Luma_PCM", or if the current codec unit contains both luma and chroma components and IntraLumaPredMode is equal to "Intra_Luma_PCM" and IntraChromaPredMode is not equal to "Intra_Chroma_PCM", then initialize the entropy codec after parsing pcm_coeff of the luma coded block.

[0862] If the current codec unit contains only chroma components and IntraChromaPredMode is equal to "Intra_Chroma_PCM", or if the current codec unit contains both luma and chroma components and IntraLumaPredMode is equal to "Intra_Luma_PCM" and IntraChromaPredMode is equal to "Intra_Chroma_PCM", initialize the entropy codec after parsing pcm_coeff of the luma coded block.

[0863] Table 8.3.3.2.1 - Deriving the Binary Symbolic Model (Referencing each section in the AVS specification)

[0864]

[0865]

[0866]

[0867] Table 8.3.3.3.1 - Analysis Process

[0868]

[0869]

[0870]

[0871]

[0872]

[0873]

[0874]

[0875] To enhance an 8-bit bitstream, the following conditions must be met:

[0876] The value of profile_id should be 0x30.

[0877] The value of progressive_sequence should be "1".

[0878] The value of chroma_format should be "01".

[0879] The value of sample_precision should be "001".

[0880] Between the start code of a video sequence and the end code of the first video sequence, or between the start code of a video sequence and the first video editing code, the PictureStructure value of all encoded and decoded pictures should be the same.

[0881] The progressive_frame value for all encoded and decoded images should be the same between the start code of a video sequence and the end code of the first subsequent video sequence, or between the start code of a video sequence and the first subsequent video edit code.

[0882] The value of MiniSize should be 8.

[0883] The value of log2_lcu_size_minus2 should be in the range of 3 to 5.

[0884] The value of log2_min_cu_size_minus2 should be 0.

[0885] The value of stable_patch_flag should be "1".

[0886] The value of ref_colocated_patch_flag should be "0".

[0887] The value of uniform_patch_flag should be "1".

[0888] Small patches belonging to the same image should appear in ascending order based on the patch_index value.

[0889] If any side of the codec block is longer than 64, you can only choose either skip mode or inter-frame prediction mode for the codec block. If inter-frame prediction mode is selected, the value of ctp_zero_flag should be '1'.

[0890] If the number of luminance samples in the current codec unit is less than 64, the current codec unit should not use skip mode or direct mode, and the prediction reference mode of the prediction unit of the current codec unit should not be "PRED_List01".

[0891] The value of MaxSplitTimes should be 6.

[0892] The value of MaxPartRatio should be 8.

[0893] The value of log2_max_bt_size_minus2 should be 4 or 5.

[0894] If the width and height of the LCU of the image are both equal to 128, then the LCU should be partitioned using a quadtree and the value of qt_split_flag should be "1".

[0895] If the width of a codec tree node is equal to 128 and the height is equal to 64, and the codec tree node is split, then the codec tree node should be split using a vertical binary tree, and the value of bet_split_dir_flag should be "1".

[0896] If the width of a codec tree node is 64 and the height is 128, and the codec tree node is split, then the codec tree node should be split using a horizontal binary tree, and the value of bet_split_dir_flag should be "0".

[0897] If the library_stream_flag of the current sequence is "1", then all images in the sequence should be I-images.

[0898] If the syntax element duplicate_sequence_header_flag exists in the bitstream, then the value of this syntax element should be "1".

[0899] The decoded image buffer can contain at most one knowledge image.

[0900] The value of NumOfUpdatedLibrary should be 1.

[0901] The value of MinLibraryUpdatePeriod should be 1.

[0902] The value of MAX_TEMPORAL_ID should be 7.

[0903] In the seventh exemplary embodiment, the value range of num_of_nn_filter_minus1 should be 0 to 3.

[0904] In the seventh exemplary embodiment, the value of MAX_TEMPORAL_ID should be 7.

[0905] In the seventh exemplary embodiment, the classification restrictions specified in B.3.2 apply.

[0906] Supported versions include: 2.0.15, 2.0.30, 2.0.60, 4.0.30, 4.0.60, 6.0.30, 6.2.30, 6.4.30, 6.6.30, 6.0.60, 6.2.60, 6.4.60, 6.6.60, 6.0.120, 6.2.120, 6.4.120, 6.6.120, 8.0.30, 8.2.30, 8.4.30, 8.6.30, 8 0.60,8.2.60,8.4.60,8.6.60,8.0.120,8.2.120,8.4.120,8.6.120,10.0.30,10.2.30,10.4.30,10.6.30,10.0.60,10.2.60,10.4.60,10.6.60,10.0.120,10.2.120,10.4.120,and10.6.120.

[0907] To enhance a 10-bit bitstream, the following conditions must be met:

[0908] The value of profile_id should be 0x32.

[0909] The value of progressive_sequence should be "1".

[0910] The value of chroma_format should be "01".

[0911] The value of sample_precision should be "001" or "010".

[0912] The value of encoding_precision should be "001" or "010".

[0913] Between the start code of a video sequence and the end code of the first video sequence, or between the start code of a video sequence and the first video editing code, the PictureStructure value of all encoded and decoded pictures should be the same.

[0914] The progressive_frame value for all encoded and decoded images should be the same between the start code of a video sequence and the end code of the first subsequent video sequence, or between the start code of a video sequence and the first subsequent video edit code.

[0915] The value of MiniSize should be 8.

[0916] The value of log2_lcu_size_minus2 should be in the range of 3 to 5.

[0917] The value of log2_min_cu_size_minus2 should be 0.

[0918] The value of stable_patch_flag should be "1".

[0919] The value of ref_colocated_patch_flag should be "0".

[0920] The value of uniform_patch_flag should be "1".

[0921] Small patches belonging to the same image should appear in ascending order based on the patch_index value.

[0922] If any side of the codec block is longer than 64, you can only choose either skip mode or inter-frame prediction mode for the codec block. If inter-frame prediction mode is selected, the value of ctp_zero_flag should be '1'.

[0923] If the number of luminance samples in the current codec unit is less than 64, the current codec unit should not use skip mode or direct mode, and the prediction reference mode of the prediction unit of the current codec unit should not be "PRED_List01".

[0924] The value of MaxSplitTimes should be 6.

[0925] The value of MaxPartRatio should be 8.

[0926] The value of log2_max_bt_size_minus2 should be 4 or 5.

[0927] If the width and height of the LCU of the image are both equal to 128, then the LCU should be partitioned using a quadtree and the value of qt_split_flag should be "1".

[0928] If the width of a codec tree node is equal to 128 and the height is equal to 64, and the codec tree node is split, then the codec tree node should be split using a vertical binary tree, and the value of bet_split_dir_flag should be "1".

[0929] If the width of a codec tree node is 64 and the height is 128, and the codec tree node is split, then the codec tree node should be split using a horizontal binary tree, and the value of bet_split_dir_flag should be "0".

[0930] If the library_stream_flag of the current sequence is "1", then all images in the sequence should be I-images.

[0931] If the syntax element duplicate_sequence_header_flag exists in the bitstream, then the value of this syntax element should be "1".

[0932] The decoded image buffer can contain at most one knowledge image.

[0933] The value of NumOfUpdatedLibrary should be 1.

[0934] The value of MinLibraryUpdatePeriod should be 1.

[0935] The value of MAX_TEMPORAL_ID should be 7.

[0936] In the seventh exemplary embodiment, the value range of num_of_nn_filter_minus1 should be 0 to 3.

[0937] In the seventh exemplary embodiment, the classification restrictions specified in B.3.2 apply.

[0938] In the seventh exemplary embodiment, the supported levels include: 2.0.15, 2.0.30, 2.0.60, 4.0.30, 4.0.60, 6.0.30, 6.2.30, 6.4.30, 6.6.30, 6.0.60, 6.2.60, 6.4.60, 6.6.60, 6.0.120, 6.2.120, 6.4.120, 6.6.120, 8.0.30, 8.2.30, and 8.4.3. 0,8.6.30,8.0.60,8.2.60,8.4.60,8.6.60,8.0.120,8.2.120,8.4.120,8.6.120,10.0.30,10.2 .30,10.4.30,10.6.30,10.0.60,10.2.60,10.4.60,10.6.60,10.0.120,10.2.120,10.4.120and 10.6.120.

[0939] Back Figure 18 This diagram illustrates a block diagram of an example video processing system 1800, in which various techniques disclosed herein can be implemented. Various implementations may include some or all of the components of the video processing system 1800. The video processing system 1800 may include an input 1802 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8-bit or 10-bit multi-component pixel values, or may be received in a compressed or encoded format. The input 1802 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

[0940] Video processing system 1800 may include codec component 1804, which implements the various codec or encoding methods described herein. Codec component 1804 can reduce the average bit rate of the video from input 1802 to the output of codec component 1804 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 1804 can be stored or transmitted via connected communication (as shown in component 1806). The stored or transmitted bitstream (or codec) representation of the video received at input 1802 can be used by component 1808 to generate pixel values ​​or to send displayable video to display interface 1810. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations for the reverse codec results will be performed by the decoder.

[0941] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The techniques described herein can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0942] Figure 19 This is a block diagram of a video processing apparatus 1900. Apparatus 1900 can be used to implement one or more methods described herein. Apparatus 1900 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 1900 may include one or more processors 1902, one or more memories 1904, and video processing hardware 1906. The processors(multiple) 1902 can be configured to implement one or more methods described in this application. The memories(multiple) 1904 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 1906 can be used to implement some of the techniques described in this application using hardware circuitry. In some embodiments, hardware 1906 may be implemented wholly or partially within processor 1902, such as a graphics coprocessor.

[0943] Figure 20 This is a block diagram illustrating an example video codec system 2000 that can utilize the techniques disclosed herein. Figure 20 As shown, the video encoding / decoding system 2000 may include a source device 2010 and a destination device 2020. The source device 2010 generates encoded video data, which can be referred to as a video encoding device. The destination device 2020 can decode the encoded video data generated by the source device 2010 and can be referred to as a video decoding device.

[0944] The source device 2010 may include a video source 2012, a video encoder 2014, and an input / output (I / O) interface 2016.

[0945] Video source 2012 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems for generating video data, or combinations of these sources. Video data may include one or more images. Video encoder 2014 encodes the video data from video source 2012 to generate a bitstream. The bitstream may include bit sequences that form a codec representation of the video data. The bitstream may include codec images and associated data. A codec image is a codec representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax structures. I / O interface 2016 may include a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to destination device 2020 via network 2030 through I / O interface 2016. Encoded video data may also be stored on storage medium / server 2040 for access by destination device 2020.

[0946] The target device 2020 may include an I / O interface 2026, a video decoder 2024, and a display device 2022.

[0947] The I / O interface 2026 may include a receiver and / or a modem.

[0948] I / O interface 2026 can acquire encoded video data from source device 2010 or storage medium / server 2040. Video decoder 2024 can decode the encoded video data. Display device 2022 can display the decoded video data to the user. Display device 2022 can be integrated with target device 2020, or it can be external to target device 2020, which is configured to interface with an external display device.

[0949] The video encoder 2014 and video decoder 2024 can operate according to video compression standards such as the High Efficiency Video Coding (HEVC) standard, the Universal Video Coding (VVC) standard, and other current and / or further standards.

[0950] Figure 21 This is a block diagram illustrating an example of a video encoder 2100, which can be in... Figure 20 The video encoder 2014 in the video codec system 2000 described in the document.

[0951] The video encoder 2100 can be configured to perform any or all of the techniques disclosed herein. Figure 21 In the example, the video encoder 2100 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 2100. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0952] The functional components of the video encoder 2100 may include a segmentation unit 2101, a prediction unit 2102 (which may include a mode selection unit 2103, a motion estimation unit 2104, a motion compensation unit 2105, and an intra-frame prediction unit 2106), a residual generation unit 2107, a transform unit 2108, a quantization unit 2109, an inverse quantization unit 2110, an inverse transform unit 2111, a reconstruction unit 2112, a buffer 2113, and an entropy encoding / decoding unit 2114.

[0953] In other examples, the video encoder 2100 may include more, fewer, or different functional components. In one example, the prediction unit 2102 may include an intra-block copy (IBC) unit. The IBC unit can perform prediction in IBC mode, where at least one reference picture is the picture containing the current video block.

[0954] Furthermore, some components, such as the motion estimation unit 2104 and the motion compensation unit 2105, can be highly integrated, but for interpretable purposes... Figure 21 The example is shown separately.

[0955] The segmentation unit 2101 can segment an image into one or more video blocks. Figure 20 The video encoder 2014 and video decoder 2024 in the software can support various video block sizes.

[0956] The mode selection unit 2103 can select one of the encoding / decoding modes (intra-frame or inter-frame, e.g., based on error results) and provide the resulting intra-frame or inter-frame encoded / decoded block to the residual generation unit 2107 to generate residual block data, and to the reconstruction unit 2112 to reconstruct the encoded block for use as a reference picture. In some examples, the mode selection unit 2103 can select a combination of intra-frame and inter-frame prediction (CIIP) modes, where prediction is based on inter-frame prediction signals and intra-frame prediction signals. In the case of inter-frame prediction, the mode selection unit 2103 can also select the resolution of the motion vector (e.g., sub-pixel or integer pixel precision) for the block.

[0957] To perform inter-frame prediction on the current video block, motion estimation unit 2104 can generate motion information for the current video block by comparing one or more reference frames from buffer 2113 with the current video block. Motion compensation unit 2105 can determine the predicted video block for the current video block based on motion information and decoded samples from images other than those associated with the current video block from buffer 2113.

[0958] The motion estimation unit 2104 and the motion compensation unit 2105 can perform different operations on the current video block, for example, depending on whether the current video block is in an I-band, P-band, or B-band.

[0959] In some examples, motion estimation unit 2104 may perform unidirectional prediction for the current video block, and may search reference images in list 0 or list 1 to find a reference video block for the current video block. Motion estimation unit 2104 may then generate a reference index indicating that the reference image in list 0 or list 1 contains the reference video block, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 2104 may output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 2105 may generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0960] In other examples, motion estimation unit 2104 can perform bidirectional prediction for the current video block. Motion estimation unit 2104 can search for a reference video block for the current video block in the reference images in list 0, and can also search for another reference video block for the current video block in the reference images in list 1. Motion estimation unit 2104 can then generate reference indices indicating the reference images in lists 0 and 1, which contain reference video blocks and motion vectors indicating spatial displacement between the reference video blocks and the current video block. Motion estimation unit 2104 can output the reference index and motion vector of the current video block as motion information for the current video block. Motion compensation unit 2105 can generate a predicted video block for the current video block based on the reference video blocks indicated by the motion information of the current video block.

[0961] In some examples, the motion estimation unit 2104 can output a complete set of motion information for the decoder to use in the decoding process.

[0962] In some examples, motion estimation unit 2104 may not output the complete set of motion information for the current video. Instead, motion estimation unit 2104 may signal the motion information of the current video block to another video block. For example, motion estimation unit 2104 may determine that the motion information of the current video block is sufficiently similar to the motion information of neighboring video blocks.

[0963] In one example, the motion estimation unit 2104 may indicate a value in the syntactic structure associated with the current video block that indicates to the video decoder 2024 that the current video block has the same motion information as another video block.

[0964] In another example, motion estimation unit 2104 can identify another video block and motion vector difference (MVD) within the syntactic structure associated with the current video block. The motion vector difference represents the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 2124 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0965] As discussed above, Video Encoder 2014 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by Video Encoder 2014 include Advanced Motion Vector Prediction (AMVP) and Merge Pattern Signaling Notification.

[0966] Intra-prediction unit 2106 can perform intra-prediction on the current video block. When intra-prediction unit 1906 performs intra-prediction on the current video block, intra-prediction unit 2106 can generate prediction data for the current video block based on decoded samples of other video blocks in the same frame. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0967] The residual generation unit 2107 can generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.

[0968] In other examples, residual data for the current video block may not exist, for example in skip mode, and residual generation unit 2107 may not perform subtraction operations.

[0969] The transform processing unit 2108 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0970] After the transform processing unit 2108 generates a transform coefficient video block associated with the current video block, the quantization unit 2109 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0971] The dequantization unit 2110 and the inverse transform unit 2111 can apply dequantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 2112 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 2102 to generate a reconstructed video block associated with the current block for storage in the buffer 2113.

[0972] After the video block is reconstructed by the reconstruction unit 2112, a loop filtering operation can be performed to reduce video block artifacts in the video block.

[0973] The entropy encoding / decoding unit 2114 can receive data from other functional components of the video encoder 2100. When the entropy encoding / decoding unit 2114 receives data, it can perform one or more entropy encoding / decoding operations to generate entropy-encoded data and output a bitstream including the entropy-encoded data.

[0974] Figure 22 This is an example block diagram illustrating the video decoder 2200. The video decoder 2200 can be... Figure 20 The video decoder 2024 in the video codec system 2000 shown in the figure.

[0975] The video decoder 2200 can be configured to perform any or all of the technologies disclosed herein. Figure 22 In the example, the video decoder 2200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 2200. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0976] exist Figure 22 In the example, video decoder 2200 includes an entropy decoding unit 2201, a motion compensation unit 2202, an intra-frame prediction unit 2203, an inverse quantization unit 2204, an inverse transform unit 2205, a reconstruction unit 2206, and a buffer 2207. In some examples, video decoder 2000 may perform functions typically associated with video encoder 1814 (e.g., Figure 18 The encoding channel (pass) is the opposite of the decoding channel.

[0977] Entropy decoding unit 2201 can retrieve the encoded bitstream. The encoded bitstream may include entropy-coded video data (e.g., encoded video data blocks). Entropy decoding unit 2201 can decode the entropy-coded video data, and based on the entropy-coded video data, motion compensation unit 2202 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. For example, motion compensation unit 2202 can determine this information by executing AMVP and merge modes.

[0978] The motion compensation unit 2202 can generate motion compensation blocks, possibly performing interpolation based on an interpolation filter. The identifier of the interpolation filter used at sub-pixel precision can be included in the syntax element.

[0979] The motion compensation unit 2202 can use the interpolation filter used by the video encoder 2014 during the encoding of the video block to calculate the interpolated values ​​of the sub-integer pixels of the reference block. The motion compensation unit 2202 can determine the interpolation filter used by the video encoder 2014 based on the received syntax information and use the interpolation filter to generate the prediction block.

[0980] The motion compensation unit 2202 can use some syntax information to determine the size of the blocks used to encode the frames and / or stripes of the encoded video sequence, segmentation information describing how each macroblock of the picture of the encoded video sequence is segmented, a mode indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information of the decoded encoded video sequence.

[0981] Intra-prediction unit 2203 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Dequantization unit 2204 dequantizes, i.e., inverse quantization, the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 2201. Inverse transform unit 2205 applies an inverse transform.

[0982] The reconstruction unit 2206 can add the residual block to the corresponding prediction block generated by the motion compensation unit 2202 or the intra-frame prediction unit 2203 to form a decoded block. If necessary, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 2207, which provides a reference block for subsequent motion compensation / intra-frame prediction and also generates decoded video for display device rendering.

[0983] Figure 23This is a method 2300 for encoding and decoding video data according to an embodiment of the present invention. Method 2300 can be executed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 2300 can be implemented to control the presence (e.g., application) of an NN filter model through various levels of syntax elements. For example, syntax elements(s) indicating whether an NN filter(s) is applied may be at a first level (e.g., in the sequence header and / or sequence parameter set (SPS) of a video unit). Syntax elements(s) indicating whether an NN filter(s) is applied may also be at a second level (e.g., in the picture header, picture parameter set (PPS), and / or stripe header of a video unit). Furthermore, syntax elements(s) indicating whether an NN filter(s) is applied may be at a third level (e.g., syntax elements indicating a small block, CTU, CTB, block, sub-picture, tile, or region of a video unit).

[0984] In block 2302, the encoding / decoding device applies a neural network (NN) filter to the unfiltered samples of the video unit to generate filtered samples. The NN filter is applied based on the syntax elements of the video unit. In one embodiment, unfiltered samples are samples (or pixels) that have not undergone any filtering process or have not been sufficiently filtered. For example, unfiltered samples have not been processed by any NN filter. As another example, unfiltered samples have not been processed by an NN filter, an adaptive loop filter (ALF), a deblocking filter (DF), a sample adaptive offset (SAO) filter, or a combination thereof.

[0985] In block 2304, the codec device converts between the video media file and the bitstream based on the generated filtered samples.

[0986] When implemented in an encoder, the conversion involves receiving a media file (e.g., a video unit) and encoding the filtered samples into a bitstream. When implemented in a decoder, the conversion involves receiving a bitstream comprising the filtered samples and decoding the bitstream to obtain the filtered samples.

[0987] In one embodiment, method 2300 may utilize or combine one or more features or processes of other methods disclosed herein.

[0988] In one example, a non-transitory computer-readable medium stores a bitstream of video generated by a method (e.g., all or part of method 2300) performed by a video processing apparatus (e.g., video processing apparatus 1900 described above). For example, a bitstream can be generated by applying an NN filter to unfiltered samples of video units to generate filtered samples. As described above, the NN filter is applied based on the syntax elements of the video units, and a bitstream is generated based on the filtered samples.

[0989] The following provides a list of preferred solutions for some embodiments.

[0990] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Example 1).

[0991] 1. A method implemented by a video codec apparatus, comprising: for a conversion between video units and a bitstream of video, applying parameters of a loop filter to the video units according to rules; and performing the conversion based on a determination, wherein the determination is based on a quality level indicator (QI) input.

[0992] 2. The method according to claim 1, wherein the Qi of the video unit is the quantization parameter of the video unit.

[0993] 3. The method according to claim 1, wherein the Qi of the video unit is the quantization parameter value of the video.

[0994] 4. The method according to claim 1, wherein the QI of the video unit is a quantization step size applied to the video unit.

[0995] 5. The method of claim 1, wherein the conversion is performed by concatenating the Qi indicator inputs into a two-dimensional array of the same size as the video unit and using it as the determined input plane.

[0996] 6. The method according to any one of claims 1-5, wherein the QI indicator input is indicated in the bit stream.

[0997] 7. The method of claim 1, wherein the function for determining the QI indicator is used.

[0998] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Example 2).

[0999] 8. A video processing method comprising: for conversion between video units and a bitstream of video, constructing a filter candidate list applicable to loop filtering of the video units according to rules; and performing the conversion based on the determination; wherein the rules specify that at least one entry in the list is based on a computational neural network (CNN) model.

[1000] 9. The method of claim 8, wherein the list is constructed in response to encoding / decoding statistics of the video units.

[1001] 10. The method of claim 9, wherein the encoding / decoding statistics include the ratio of a first set of video samples associated with the video unit to the samples of the video unit, the first set of video samples being encoded / decoded using a predetermined video encoding / decoding tool (referred to as tool X).

[1002] 11. The method of claim 10, wherein tool X comprises intra-frame prediction, palette prediction, or intra-frame block copy prediction mode.

[1003] 12. The method according to any one of claims 8-11, wherein the rule further responds to the encoding / decoding conditions of the video unit.

[1004] 13. The method of claim 12, wherein the encoding / decoding condition is that the video unit belongs to the luminance component of the video.

[1005] 14. The method according to claim 12, wherein the encoding / decoding condition is that the video unit belongs to a certain temporal layer of the video.

[1006] 15. The method of claim 10, wherein the first set of video samples is in an image including the video unit.

[1007] 16. The method of claim 10, wherein the first set of video samples are in one or more images different from the image including the video unit.

[1008] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Example 3).

[1009] 17. A video processing method comprising: for a conversion between video units of a video and a bitstream of a video, determining, according to rules, whether or how to enable a neural network (NN)-based filter for the video unit; and performing the conversion based on the determination; wherein the rules depend on whether a non-deep learning-based filter (NDLF) is enabled for the video unit.

[1010] 18. The method of claim 17, wherein the NDLF filtering comprises deblocking filtering or sample adaptive offset filtering or cross-component sample adaptive offset or adaptive loop filtering or cross-component adaptive loop filtering or luminance mapping with chroma scaling or bilateral filtering or transform domain filtering.

[1011] 19. The method according to any one of claims 17-18, wherein the rule specifies one or more NDLF filter types that are mutually exclusive with the NN filter.

[1012] 20. The method of claim 17, wherein the rule specifies one or more NDLF filter types that are allowed to be enabled together with the NN filter.

[1013] 21. The method according to any one of claims 17-20, wherein the rule depends on the color format of the video or the quality, level, or grade of the video.

[1014] 22. The method according to any one of claims 1-21, wherein the video unit comprises an encoding / decoding unit, a transformation unit, a prediction unit, and stripes or sub-pictures.

[1015] 23. The method according to any one of claims 1-21, wherein the video unit is a codec block, a video strip, a video picture, a video strip, or a video subpicture.

[1016] 24. The method according to any one of claims 1-23, wherein the conversion comprises generating the video from the bitstream or generating the bitstream from the video.

[1017] 25. A video decoding apparatus comprising a processor configured to implement the method of one or more of claims 1 to 24.

[1018] 26. A video encoding apparatus comprising a processor configured to implement the method of one or more of claims 1 to 24.

[1019] 27. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 24.

[1020] 28. A method, apparatus or bit stream generated according to the disclosed method or system described in this document.

[1021] The other solutions, examples, embodiments, modules, and functional operations disclosed in this document can be implemented in digital electronic circuits or computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more computer program instruction modules encoded on a computer-readable medium, which are executed or controlled by a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of substances affecting machine-readable propagation signals, or a combination thereof. The term "data processing apparatus" encompasses all means, devices, and machines that process data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof. The transmitted signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[1022] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple harmonizing files (e.g., files storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected via a communication network.

[1023] The processes and logic described in this document can be executed by one or more programmable processors to execute one or more computer programs, thereby performing functions by manipulating input data and generating output. The processes and logic can also be executed by dedicated logic circuits, and can be implemented as dedicated logic circuits, such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits).

[1024] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into special-purpose logic circuitry.

[1025] Although this patent document contains numerous details, these details should not be construed as limiting any invention or the scope of the claims, but rather as a description of features that may be specific to particular embodiments of a particular invention. Certain features described in this patent document in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[1026] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all shown operations to achieve the desired effect. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[1027] Only some implementation methods and examples are described, and other implementation methods, enhancements and variations can be made based on the content described and shown in this patent document.

[1028] Although this patent document contains numerous details, these details should not be construed as limiting any invention or the scope of the claims, but rather as a description of features that may be specific to particular embodiments of a particular invention. Certain features described in this patent document in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[1029] Similarly, although the operations are depicted in a specific order in the figures, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[1030] Only a few implementations and examples have been described, and other implementations, improvements and modifications may be made based on the descriptions and instructions in this patent document.

Claims

1. A method implemented by a video encoding / decoding device, comprising: A neural network (NN) filter is applied to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is applied based on the syntax elements of the video unit. as well as Based on the generated filtered samples, conversion is performed between the video media file and the bitstream; When the NN filter is enabled for the video unit, the syntax element includes an index k, where the NN filter is one of T NN filters; The syntax elements are encoded and decoded based on a context model, which is selected based on the number of allowed NN filters. The filter model index of the color component of the video unit is configured to specify one of K context models, and one of the K context models is specified as the minimum value of K-1 and binIdx, where binIdx is the index of the binary bit to be encoded and decoded.

2. The method according to claim 1, wherein, The syntax element indicates selection from at least one of the following: whether the NN filter is enabled, the number of NN filters to be applied, and the type of the NN filter to be applied.

3. The method according to claim 1, wherein, The first level includes the sequence parameter set (SPS) and / or the sequence header. The second level includes image headers, image parameter sets (PPS), and / or strip headers. The third level includes the video unit's patch, the video unit's codec tree unit (CTU), the video unit's codec tree block (CTB), the video unit's block, the video unit's sub-picture, the video unit's slice, the video unit's strip, or the video unit's region.

4. The method according to claim 3, wherein, The syntax element is a first syntax element at the first level, which indicates whether the NN filter is adaptively selected at the second level to be applied to the picture or strip of the video unit.

5. The method according to claim 3, wherein, The syntax element is a first syntax element at the second level, which is conditionally applied based on a second syntax element at the first level, wherein the NN filter is applied at the second level based on a flag indicating that the second syntax element is true, and wherein the NN filter is not applied based on a flag indicating that the second syntax element is false.

6. The method according to claim 3, wherein, The syntax element is a first syntax element at the second level, which indicates whether the NN filter is adaptively selected at the third level to be applied to the sub-pictures of the video unit, or whether the use of the NN filter is controlled at the third level.

7. The method according to claim 3, wherein, The syntax element is a first syntax element at the second level, which indicates whether the NN filter is adaptively selected, used, or applied at the second level; and wherein the first syntax element is transmitted via signaling based on an indication of adaptively selecting the NN filter at the second level or an indication that the number of NN filters is greater than 1.

8. The method according to claim 3, wherein, The syntax element is a first syntax element at the third level, which is conditionally applied based on a second syntax element at the first level and / or a third syntax element at the second level, wherein the first syntax element is encoded using context.

9. The method according to claim 3, wherein, The syntax element is the first syntax element at the third level, which is transmitted via signal based on an indication of adaptively selecting the NN filter at the third level or an indication that the number of the NN filters is greater than 1.

10. The method of claim 3, further comprising: Based on the index k >= 0 and k < T, the k-th NN filter is applied in the second level.

11. The method according to claim 3, further comprising: Based on the index k >= T, the NN filter is adaptively selected at the third level.

12. The method according to claim 1, wherein, The index k is restricted to the range of 0 to (T–1).

13. The method according to claim 1, wherein, The filter model indexes of the first and second color components of the video unit are encoded and decoded using the same set of context.

14. The method according to claim 1, wherein, The filter model index of the first color component of the video unit is encoded and decoded using a different set of contexts than the filter model index of the second color component of the video unit.

15. The method according to claim 1, wherein, The syntax element can be transmitted via signaling using contextual encoding or bypass encoding, or binaryized using fixed-length encoding, unary encoding, rounding unary encoding, signed unary encoding, signed rounding unary encoding, rounding binary encoding, or exponential Golomb encoding.

16. The method according to any one of claims 1-15, wherein, The conversion includes generating the bitstream based on the video media file.

17. The method according to any one of claims 1-15, wherein, The conversion includes parsing the bitstream to obtain the video media file.

18. An apparatus for encoding and decoding video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, it causes the processor to: A neural network (NN) filter is applied to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is applied based on the syntax elements of the video unit; and Based on the generated filtered samples, the conversion is performed between the video media file and the bitstream; When the NN filter is enabled for the video unit, the syntax element includes an index k, where the NN filter is one of T NN filters; The syntax elements are encoded and decoded based on a context model, which is selected based on the number of allowed NN filters. The filter model index of the color component of the video unit is configured to specify one of K context models, and one of the K context models is specified as the minimum value of K-1 and binIdx, where binIdx is the index of the binary bit to be encoded and decoded.

19. A method for generating a video bitstream, comprising: The bitstream is generated by performing the following video processing method; as well as The bitstream is stored in a non-transitory computer-readable recording medium. The video processing method includes: A neural network (NN) filter is applied to unfiltered samples of a video unit to generate filtered samples, wherein the NN filter is applied based on the syntax elements of the video unit; and The bit stream is generated based on the generated filtered samples; When the NN filter is enabled for the video unit, the syntax element includes an index k, where the NN filter is one of T NN filters; The syntax elements are encoded and decoded based on a context model, which is selected based on the number of allowed NN filters. The filter model index of the color component of the video unit is configured to specify one of K context models, and one of the K context models is specified as the minimum value of K-1 and binIdx, where binIdx is the index of the binary bit to be encoded and decoded.

Images