Unified neural network for loop filtering for video coding and decoding

Abstract

A mechanism for processing video data is disclosed. The mechanism includes determining to apply a neural network (NN) model to the visual media data. The NN model is configured to receive as input a slice type indicator (STI) corresponding to a slice type. A conversion between the visual media data and the bitstream is performed based on the NN model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 478,309, filed on January 3, 2023, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the generation, storage, and use of digital audio-visual media information in file format. Background Art

[0004] Digital video accounts for the largest share of bandwidth used on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the demand for bandwidth used by digital video is likely to continue to grow. Summary of the Invention

[0005] A first aspect relates to a method for processing video data, comprising: determining to apply a neural network (NN) model to visual media data, wherein the NN model is configured to receive a slice type indicator (STI) corresponding to a slice type as input; and performing conversion between the visual media data and a bitstream based on the NN model.

[0006] Optionally, in any one of the above aspects, another embodiment of the aspect provides deriving the STI from the stripe type.

[0007] Optionally, in any one of the above aspects, another embodiment of the aspect provides that when the slice type is an intra-frame prediction (I) slice, the STI is set to a, when the slice type is a bidirectional inter-frame prediction (B) slice, the STI is set to b, and when the slice type is a unidirectional inter-frame prediction (P) slice, the STI is set to c, where a, b and c are all constants.

[0008] Optionally, in any of the above aspects, another embodiment of the aspect provides that b is equal to c.

[0009] Optionally, in any of the above aspects, another embodiment of the aspect provides that b is equal to -a, and c is equal to -a.

[0010] Optionally, in any of the above aspects, another embodiment of the aspect provides that a is equal to 1, b is equal to -1, and c is equal to -1.

[0011] Optionally, in any of the above aspects, another embodiment of the aspect provides that a is equal to 1, b is equal to 0.5, and c is equal to 0.5.

[0012] Optionally, in any of the above aspects, another embodiment of the aspect provides tiling or expanding the STI into a two-dimensional (2D) array having the same size as the video unit of the visual media data, and after the tiling or the expanding, treating the STI as an additional input.

[0013] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the video unit includes a codec tree unit (CTU) or a codec tree block (CTB).

[0014] Optionally, in any of the above aspects, another embodiment of the aspect provides that the STI is included in a sequence parameter set (SPS), picture parameter set (PPS), picture header, slice header, CTU, codec unit (CU) or region level of a bitstream generated by an encoder or received by a decoder.

[0015] Optionally, in any of the above aspects, another embodiment of the aspect provides that the STI includes f(STI), where f is an arbitrary function.

[0016] Optionally, in any of the above aspects, another embodiment of the aspect provides that the STI is used for one or more color components.

[0017] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the STI includes a first STI and a second STI, wherein the first STI is used for a first color component and the second STI is used for a second color component.

[0018] Optionally, in any of the above aspects, another embodiment of the aspect provides that the first color component includes a luminance color component, and wherein the second color component includes a chrominance color component, such as a blue difference color component (Cb) or a red difference color component (Cr).

[0019] Optionally, in any one of the above aspects, another implementation of the aspect provides that the first STI indicates a quality level of the first color component.

[0020] Optionally, in any one of the above aspects, another implementation of the aspect provides that the second STI indicates a quality level of the second color component.

[0021] Optionally, in any of the above aspects, another embodiment of the aspect provides using the STI for NN filtering of a first color component but not for NN filtering of a second color component, wherein the first color component includes a luminance color component and the second color component includes a chrominance color component, such as a blue difference color component (Cb) or a red difference color component (Cr).

[0022] Optionally, in any of the above aspects, another embodiment of the aspect provides using the STI for NN filtering of all color components.

[0023] Optionally, in any one of the above aspects, another embodiment of the aspect provides performing a specific operation when a picture is divided into multiple slices, wherein the specific operation includes the NN model using the slice type of a single slice of the multiple slices for the entire picture.

[0024] Optionally, in any of the above aspects, another embodiment of the aspect provides that the single slice is the first slice of the picture.

[0025] Optionally, in any one of the above aspects, another embodiment of the aspect provides performing a specific operation when the picture is divided into multiple slices, wherein the specific operation includes the NN model using a slice type of a slice among the multiple slices for a sample point in the slice.

[0026] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the NN model is configured to receive a codec mode indicator (CMI) corresponding to the codec mode as input.

[0027] Optionally, in any one of the above aspects, another implementation of the aspect provides deriving the CMI from the coding mode.

[0028] Optionally, in any one of the above aspects, another implementation of the aspect provides that the CMI indicates an intra-frame prediction mode or an inter-frame prediction mode.

[0029] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the CMI indicates an intra block copy mode or a palette mode.

[0030] Optionally, in any of the above aspects, another embodiment of the aspect provides that the CMI includes a two-dimensional (2D) array having the same resolution as the video unit of the visual media data to be filtered, and wherein each sample in the 2D array is represented by a value reflecting a codec mode associated with the codec unit (CU) to which the sample belongs.

[0031] Optionally, in any one of the above aspects, another embodiment of the aspect provides that i and j are vertical indices and horizontal indices inside the 2D array, wherein intra_pred_mode is the intra-frame prediction mode associated with the coding unit (CU) to which the sample point (i, j) belongs, inter_pred_mode is the inter-frame prediction mode associated with the CU to which the sample point (i, j) belongs, and the CMI (i, j) is derived as follows: when the sample point (i, j) belongs to the intra-frame coded CU, the CMI (i, j) is set to the intra_pred_mode; and when the sample point (i, j) belongs to the inter-frame coded CU, the CMI (i, j) is set to the inter_pred_mode.

[0032] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the intra-frame prediction mode includes skip mode, merge mode or advanced motion vector prediction (AMVP) mode.

[0033] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the CMI is included in a sequence parameter set (SPS), picture parameter set (PPS), picture header, slice header, CTU, codec unit (CU) or region level of a bitstream generated by an encoder or received by a decoder.

[0034] Optionally, in any of the above aspects, another embodiment of the aspect provides that the CMI includes f(CMI), wherein f is an arbitrary function.

[0035] Optionally, in any of the above aspects, another embodiment of the aspect provides that the CMI is used for one or more color components.

[0036] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the CMI includes a first CMI and a second CMI, wherein the first CMI is used for a first color component, and wherein the second CMI is used for a second color component.

[0037] Optionally, in any of the above aspects, another embodiment of the aspect provides that the first color component includes a luminance color component, and wherein the second color component includes a chrominance color component, such as a blue difference color component (Cb) or a red difference color component (Cr).

[0038] Optionally, in any one of the above aspects, another implementation manner of the aspect provides that the first CMI indicates a quality level of the first color component.

[0039] Optionally, in any one of the above aspects, another implementation manner of the aspect provides that the second CMI indicates a quality level of the second color component.

[0040] Optionally, in any of the above aspects, another embodiment of the aspect provides using the CMI for NN filtering of a first color component but not for NN filtering of a second color component, wherein the first color component includes a luminance color component and wherein the second color component includes a chrominance color component, such as a blue difference color component (Cb) or a red difference color component (Cr).

[0041] Optionally, in any of the above aspects, another embodiment of the aspect provides using the CMI for NN filtering of all color components.

[0042] Optionally, in any of the above aspects, another embodiment of the aspect provides that the NN model is configured to receive as input encoded information, reconstructed information, information derived from the encoded information, or some combination thereof.

[0043] Optionally, in any of the above aspects, another embodiment of the aspect provides that the encoded information includes a quantization parameter (QP), or the encoded information is derived based on the QP.

[0044] Optionally, in any one of the above aspects, another implementation of the aspect provides that the encoded information includes a prediction direction, or the encoded information is derived based on the prediction direction.

[0045] Optionally, in any one of the above aspects, another implementation of the aspect provides that the encoded information includes a reference picture index, or the encoded information is derived based on the reference picture index.

[0046] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the encoded information includes a picture order count (POC), or the encoded information is derived based on the POC.

[0047] Optionally, in any one of the above aspects, another implementation of the aspect provides that the encoded information includes the number of reference pictures, or the encoded information is derived based on the number of reference pictures.

[0048] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the encoded information includes a time domain layer identifier (ID), or the encoded information is derived based on the time domain layer ID.

[0049] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the encoded information includes a motion vector, or the encoded information is derived based on the motion vector.

[0050] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the encoded information includes motion vector differences, or the encoded information is derived based on the motion vector differences.

[0051] Optionally, in any one of the above aspects, another implementation of the aspect provides that the encoded information includes a transform type, or the encoded information is derived based on the transform type.

[0052] Optionally, in any one of the above aspects, another implementation of the aspect provides that the encoded information includes residual information, or the encoded information is derived based on the residual information.

[0053] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the coded information includes a coded block flag (CBF), or the coded information is derived based on the CBF.

[0054] Optionally, in any of the above aspects, another embodiment of the aspect provides that the encoded information is derived based on whether other loop filters have been enabled, and wherein the other loop filters include a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a cross-component ALF or a bilateral filter.

[0055] Optionally, in any of the above aspects, another embodiment of the aspect provides that the encoded information is derived based on whether the sample point is modified by other loop filters, and wherein the other loop filters include a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a cross-component ALF or a bilateral filter.

[0056] Optionally, in any of the above aspects, another embodiment of the aspect provides that two or more types of the encoded and decoded information are input into the NN model together.

[0057] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the STI, the CMI and the encoded and decoded information are input into the NN model together.

[0058] Optionally, in any of the above aspects, another embodiment of the aspect provides that only one of the STI, the CMI and the encoded and decoded information is used by the NN model.

[0059] Optionally, in any of the above aspects, another embodiment of the aspect provides that a syntax element is included in the bitstream to indicate whether the STI, the CMI, the encoded information or some combination thereof is used by the NN model.

[0060] Optionally, in any of the above aspects, another embodiment of the aspect provides that a syntax element is included in the bitstream to indicate how the STI, the CMI, the encoded information or some combination thereof is used by the NN model.

[0061] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the syntax element is binarized into a flag, a fixed-length code, an exponential-Golomb code, a unary code or a truncated unary code, and wherein the syntax element is signed or unsigned.

[0062] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the syntax element is encoded or decoded using at least one context model or is bypassed.

[0063] Optionally, in any one of the above aspects, another implementation of the aspect provides that the syntax element is encoded and decoded in a conditional manner.

[0064] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the syntax element is included in the bitstream only when the STI, the CMI, the encoded and decoded information corresponding to the syntax element is applicable or enabled.

[0065] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the syntax element is included at a block level, a sequence level, a picture group level, a picture level, a slice level or a slice group level of a bitstream, or is included in a codec tree unit (CTU), a codec unit (CU), a transform unit (TU), a prediction unit (PU), a codec tree block (CTB), a codec block (CB), a transform block (TB) or a prediction block (PB) of a bitstream.

[0066] Optionally, in any one of the above aspects, another embodiment of the aspect provides that the syntax element is included in a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header or a slice group header of a bitstream.

[0067] Optionally, in any one of the above aspects, another embodiment of the aspect provides indicating whether and / or how to apply the method at the block level, sequence level, picture group level, picture level, slice level or slice group level of the bitstream, or indicating whether and / or how to apply the method in the codec tree unit (CTU), codec unit (CU), transform unit (TU), prediction unit (PU), codec tree block (CTB), codec block (CB), transform block (TB) or prediction block (PB) of the bitstream.

[0068] Optionally, in any one of the above aspects, another embodiment of the aspect provides an indication of whether and / or how to apply the method in a sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependency parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptation parameter set (APS), slice header or slice group header of a bitstream.

[0069] Optionally, in any of the above aspects, another embodiment of the aspect provides whether and / or how to apply the method depends on the block size, color format, single tree partitioning, double tree partitioning, color component, the slice type, picture type or other encoded and decoded information.

[0070] Optionally, in any of the above aspects, another embodiment of the aspect provides that one or more of the methods are used in another encoding and decoding tool that performs chroma fusion.

[0071] A second aspect relates to a method for processing video data, comprising: determining to apply a neural network (NN) model to visual media data, wherein the NN model is configured to receive a type indicator corresponding to a type as input, wherein the type includes a picture type, a frame type, or a block type; and performing conversion between the visual media data and a bitstream based on the NN model.

[0072] Optionally, in any of the above aspects, another embodiment of the aspect provides that the converting includes encoding the media data into a bitstream.

[0073] Optionally, in any of the above aspects, another embodiment of the aspect provides that the converting includes decoding the media data from a bitstream.

[0074] A third aspect relates to an apparatus for processing media data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any disclosed embodiment.

[0075] A fourth aspect relates to a non-transitory computer-readable medium, comprising a computer program product for use with a video codec device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium, such that when the instructions are executed by a processor, the video codec device performs the method of any disclosed embodiment.

[0076] A fifth aspect relates to a non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing apparatus, wherein the method includes any disclosed method.

[0077] A sixth aspect relates to a method for storing a bitstream of a video, including any method of any disclosed method.

[0078] The seventh aspect relates to the method, device or system described in the present disclosure.

[0079] For purposes of clarity, any of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0080] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] For a more complete understanding of this disclosure, reference is now made to the following brief description, which is taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0082] Figure 1 An example picture divided into raster scan strips is shown.

[0083] Figure 2 An example picture segmented into rectangular scan strips is shown.

[0084] Figure 3 An example picture segmented into bricks is shown.

[0085] FIG4 shows an example of a CTB that crosses a picture boundary.

[0086] Figure 5 An example of an encoder block diagram is shown.

[0087] Figure 6 Examples of block boundaries in a picture are shown.

[0088] Figure 7 An example of pixels involved in filter usage is shown.

[0089] Figure 8 Directional patterns for edge offset (EO) sample classification are shown.

[0090] Figure 9 Example geometric transformation based adaptive loop filter (GALF) filter shapes are shown.

[0091] Figure 10 An example of relative coordinates for a 5x5 diamond filter support is shown.

[0092] Figure 11 An example of relative coordinates for a 5x5 diamond filter support is shown.

[0093] Figure 12A An example convolutional neural network (CNN) is shown.

[0094] Figure 12B The rectified linear unit (ReLU) / parameterized rectified linear unit (PReLU) activation function and convolutional layer are shown.

[0095] Figure 13 An example of tiling the slice type indicators (STI) into a two-dimensional array is shown.

[0096] Figure 14 is a block diagram illustrating an example video processing system.

[0097] Figure 15is a block diagram of an example video processing device.

[0098] Figure 16 is a flow chart of an example method of video processing.

[0099] Figure 17 is a block diagram illustrating an example video encoding and decoding system.

[0100] Figure 18 is a block diagram illustrating an example encoder.

[0101] Figure 19 is a block diagram illustrating an example decoder.

[0102] Figure 20 is a schematic diagram of an example encoder. DETAILED DESCRIPTION

[0103] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or yet to be developed. The present disclosure should in no way be limited to the illustrative implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims, and their full scope of equivalents.

[0104] The section headings used in this disclosure are for ease of understanding and are not intended to limit the applicability of the techniques and embodiments disclosed in each section to that section. In addition, the techniques described herein are applicable to other video codec protocols and designs.

[0105] 1. Preliminary Discussion

[0106] The present disclosure relates to video coding and decoding technology. Specifically, it relates to loop filters in image / video coding and decoding. It can be applied to video coding and decoding standards such as High Efficiency Video Codec (HEVC), Versatile Video Codec (VVC) or the third generation Audio Video Standard (AVS3). It can also be applied to other video coding and decoding technologies, video codecs and / or used as a post-processing method outside the encoding and decoding process.

[0107] 2. Video codec standards

[0108] Video codec standards have evolved primarily through the development of standards from the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the International Organization for Standardization and the International Electrotechnical Commission (ISO / IEC). ITU-T produced the H.261 and H.263 standards, ISO / IEC produced the Moving Picture Experts Group (MPEG)-1 and MPEG-4 Vision, and the two organizations jointly produced the H.262 / MPEG-2 Video standard, the H.264 / MPEG-4 Advanced Video Coding (AVC) standard, and the H.265 / HEVC[1] standard. Starting with H.262, video codec standards are based on a hybrid video codec structure that utilizes temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Video Coding Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET). JVET adopted many methods and incorporated them into a reference software called the Joint Exploration Model (JEM)[2]. The JVET between the Video Coding Experts Group (VCEG) (Q6 / 16) and the ISO / IEC Joint Technical Committee (JTC1) SC29 / WG11 (eg, MPEG) was created to work on the VVC standard with the goal of reducing bitrate by 50% compared to HEVC.

[0109] A sample version of the VVC draft, Versatile Video Codec (Draft 10), can be found at: http: / / phenix.it-sudparis.eu / jvet / doc_end_user / current_document.php?id=10399. A sample version of the VVC reference software, named VTM, can be found at: https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / - / tags / VTM-10.0.

[0110] 2.1 Color Space and Chroma Downsampling

[0111] A color space, also known as a color model (or color system), is a mathematical model that describes the range of colors as a tuple of numbers, such as 3 or 4 values ​​or color components (e.g., RGB). Generally speaking, a color space is a refinement of a coordinate system and subspaces. For video compression, the most commonly used color spaces are luminance, blue-difference chroma, and red-difference chroma (YCbCr) and red, green, blue (RGB).

[0112] YCbCr, Y'CbCr, or Y Pb / Cb Pr / Cr, also written as YCBCR or Y'CBCR, is a family of color spaces used as part of the color image pipeline in video and digital photography systems. Y' is the luma component, and CB and CR are the blue-difference and red-difference chroma components. Y' (with a prime) is distinguished from Y, which is luma, meaning that light intensity is encoded nonlinearly based on the gamma-corrected RGB primaries.

[0113] Chroma downsampling is the practice of encoding an image at a lower resolution for chroma information than for luminance information, taking advantage of the fact that the human visual system is less sensitive to differences in color than to luminance. 2.1.1 4:4:4

[0115] In 4:4:4, each of the three Y'CbCr components has the same sampling rate. Therefore, there is no chroma downsampling. This scheme is sometimes used in high-end film scanners and film post-production. 2.1.2 4:2:2

[0117] In 4:2:2, the two chroma components are sampled at half the sample rate of luma. The horizontal chroma resolution is halved. This reduces the bandwidth of the uncompressed video signal by one-third with little visual difference. 2.1.3 4:2:0

[0119] In 4:2:0, horizontal sampling is doubled compared to 4:1:1, but vertical resolution is halved because the Cb and Cr channels are sampled only on alternate lines. Therefore, the data rate remains the same. Cb and Cr are downsampled by a factor of 2 both horizontally and vertically. There are three variants of the 4:2:0 scheme, with different horizontal and vertical positions.

[0120] In MPEG-2, Cb and Cr are co-located horizontally. Cb and Cr are located between pixels vertically (in interstitial positions). In Joint Photographic Experts Group (JPEG) / JPEG File Interchange Format (JFIF), H.261, and MPEG-1, Cb and Cr are located in interstitial positions, in the middle of alternate luma samples. In 4:2:0 DV, Cb and Cr are co-located horizontally. Vertically, they are co-located on alternate lines.

[0121] 2.2 Definition of Video Unit

[0122] A picture is divided into one or more slice rows and one or more slice columns. A slice is a sequence of codec tree units (CTUs) that covers a rectangular area of ​​a picture. A slice can be divided into one or more bricks, each brick consisting of a certain number of CTU rows within the slice. Slices that are not split into multiple bricks may also be called bricks. However, bricks that are proper subsets of a slice may not be called slices. A slice contains multiple slices of a picture or multiple bricks of a slice.

[0123] Two striping modes are supported: raster scan striping and rectangular striping. In raster scan striping, a strip contains a sequence of slices from a raster scan of the image. In rectangular striping, a strip contains multiple tiles of the image, which together form a rectangular region of the image. The tiles within a rectangular strip are in the order in which the tiles in the strip were raster scanned. Figure 1 An example of raster scan stripe partitioning of a picture is shown, where the picture is divided into 12 slices and 3 raster scan stripes.

[0124] Figure 2 An example of rectangular strip partitioning of a picture is shown, where the picture is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0125] Figure 3 An example of a picture partitioned into slices, bricks, and rectangular strips is shown, where the picture is divided into 4 slices (2 slice columns and 2 slice rows), 11 bricks (the upper left slice contains 1 brick, the upper right slice contains 5 bricks, the lower left slice contains 2 bricks, and the lower right slice contains 3 bricks), and 4 rectangular strips.

[0126] 2.2.1 CTU / CTB Size

[0127] In VVC, the CTU size signaled in the sequence parameter set (SPS) by the syntax element log2_ctu_size_minus2 can be as small as 4x4.

[0128] 7.3.2.3 Sequence Parameter Set RBSP Syntax

[0129]

[0130]

[0131]

[0132] log2_ctu_size_minus2 plus 2 specifies the luma coding tree block size for each CTU. log2_min_luma_coding_block_size_minus2 plus 2 specifies the minimum luma coding block size. The variables CtbLog2SizeY, CtbSizeY, MinCbLog2SizeY, MinCbSizeY, MinTbLog2SizeY, MaxTbLog2SizeY, MinTbSizeY, MaxTbSizeY, PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC, and PicHeightInSamplesC are derived as follows:

[0133] CtbLog2SizeY = log2_ctu_size_minus2 + 2 (7-9)

[0134] CtbSizeY = 1 << CtbLog2SizeY (7-10)

[0135] MinCbLog2SizeY = log2_min_luma_coding_block_size_minus2 + 2 (7-11)

[0136] MinCbSizeY = 1 << MinCbLog2SizeY (7-12)

[0137] MinTbLog2SizeY = 2 (7-13)

[0138] MaxTbLog2SizeY = 6 (7-14)

[0139] MinTbSizeY = 1 << MinTbLog2SizeY (7-15)

[0140] MaxTbSizeY = 1 << MaxTbLog2SizeY (7-16)

[0141] PicWidthInCtbsY = Ceil( pic_width_in_luma_samples÷CtbSizeY) (7-17)

[0142] PicHeightInCtbsY = Ceil( pic_height_in_luma_samples ÷ CtbSizeY ) (7-18)

[0143] PicSizeInCtbsY = PicWidthInCtbsY * PicHeightInCtbsY (7-19)

[0144] PicWidthInMinCbsY = pic_width_in_luma_samples / MinCbSizeY (7-20)

[0145] PicHeightInMinCbsY = pic_height_in_luma_samples / MinCbSizeY (7-21)

[0146] PicSizeInMinCbsY = PicWidthInMinCbsY * PicHeightInMinCbsY (7-22)

[0147] PicSizeInSamplesY = pic_width_in_luma_samples * pic_height_in_luma_samples (7-23)

[0148] PicWidthInSamplesC = pic_width_in_luma_samples / SubWidthC (7-24)

[0149] PicHeightInSamplesC = pic_height_in_luma_samples / SubHeightC (7-25)

[0150] 2.2.2 CTUs in the Picture

[0151] Assume that the CTB / largest coding unit (LCU) size is indicated by M×N (usually M equals N), and for the coding tree blocks (CTBs) located at the picture boundary (or slice or strip or other type of boundary, with the picture boundary as an example), K×L samples are within the picture boundary, where K < M or L < N. For those CTBs as shown in Figure 4, the CTB size is still equal to M×N. However, the bottom boundary / right boundary of the CTB is outside the picture.

[0152] 2.3 Coding and Decoding Processes of an Example Video Codec

[0153] Figure 5 An example of a VVC encoder block diagram is shown, which contains three loop filtering blocks: deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF). Unlike DF, which uses a predefined filter, SAO and ALF use the original samples of the current picture to reduce the mean square error between the original and reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, and using the encoded side information to signal the offset and filter coefficients. ALF is located in the last processing stage of each picture and can be seen as a tool that attempts to capture and repair artifacts caused by previous stages.

[0154] 2.4 Deblocking Filter (DB)

[0155] Figure 6 An example of a block boundary in a picture is shown. The input to DB is the reconstructed samples before the loop filter.

[0156] Vertical edges in the image are filtered first. Horizontal edges in the image are then filtered using the samples modified by the vertical edge filtering process as input. Vertical and horizontal edges in the CTBs of each CTU are processed separately on a codec unit basis. Vertical edges of a codec block within a codec unit are filtered starting with the edge on the left-hand side of the codec block and proceeding in geometric order through the edges toward the right-hand side of the codec block. Horizontal edges of a codec block within a codec unit are filtered starting with the edge at the top of the codec block and proceeding in geometric order through the edges toward the bottom of the codec block.

[0157] 2.4.1 Boundary Decision

[0158] Filtering is applied to 8x8 block boundaries. In addition, such boundaries must be transform block boundaries or codec subblock boundaries, such as those caused by the use of affine motion prediction (ATMVP). For other boundaries, deblocking filtering is disabled.

[0159] 2.4.2 Boundary strength calculation

[0160] For transform block boundaries / codec sub-block boundaries, if the boundary lies in an 8x8 grid, the boundary can be filtered and the settings of bS[xDi][yDj] (where [xDi][yDj] represents the coordinates) of the edge are defined as shown in Table 1 and Table 2 respectively.

[0161]

[0162] Table 1 Boundary Strength (When Sequence Parameter Set (SPS) Intra Block Copy (IBC) is Disabled)

[0163]

[0164] Table 2 Boundary Strength (When SPS IBC is Enabled)

[0165] 2.4.3 Deblocking Decision for Luminance Component

[0166] The deblocking decision process is described in this subsection. Figure 7 An example of pixels involved in filter usage is shown.

[0167] The wider, stronger luma filter is used only when conditions 1, 2, and 3 are all true. Condition 1 is the "large block condition." This condition checks whether the samples on the P and Q sides belong to large blocks, represented by the variables bSidePisLargeBlk and bSideQisLargeBlk, respectively. bSidePisLargeBlk and bSideQisLargeBlk are defined as follows.

[0168] bSidePisLargeBlk = ((edge ​​type is vertical and p0 belongs to a CU with width >= 32) || (edge ​​type is horizontal and p0 belongs to a CU with height >= 32))? True: False

[0169] bSideQisLargeBlk = ((edge ​​type is vertical and q0 belongs to a CU with width >= 32) || (edge ​​type is horizontal and q0 belongs to a CU with height >= 32))? True: False

[0170] Based on bSidePisLargeBlk and bSideQisLargeBlk, condition 1 is defined as follows:

[0171] Condition 1 = (bSidePisLargeBlk || bSidePisLargeBlk)? True: False

[0172] Next, if condition 1 is true, condition 2 will be further checked. First, the following variables are derived:

[0173] First, derive dp0, dp3, dq0, dq3 according to the HEVC method

[0174] if (p side is greater than or equal to 32)

[0175] dp0=(dp0+Abs(p50-2*p40+p30)+1)>>1

[0176] dp3=(dp3+Abs(p53-2*p43+p33)+1)>>1

[0177] if (q side is greater than or equal to 32)

[0178] dq0=(dq0+Abs(q50-2*q40+q30)+1)>>1

[0179] dq3=(dq3+Abs(q53-2*q43+q33)+1)>>1

[0180] Condition 2 = (d < β)? True: False

[0181] Where d = dp0 + dq0 + dp3 + dq3.

[0182] If conditions 1 and 2 are valid, we further check whether any of the blocks use sub-blocks:

[0183]

[0184]

[0185] Finally, if both conditions 1 and 2 are valid, the deblocking method will check condition 3 (bulk strong filter condition), which is defined as follows. In condition 3 StrongFilterCondition, the following variables are derived:

[0186] Derived dpq in the same way as HEVC.

[0187] Derived according to HEVC method sp3 = Abs (p3-p0)

[0188]

[0189] According to the HEVC method, sq3 = Abs(q0-q3)

[0190]

[0191] According to HEVC, StrongFilterCondition = (dpq is less than (β>>2), sp3+sq3 is less than (3*β>>5), and Abs(p0-q0) is less than (5*tC+1)>>1)? True: False.

[0192] 2.4.4 Stronger Deblocking Filter for Luma (Designed for Larger Blocks)

[0193] When the samples on either side of the boundary belong to a large block, a bilinear filter is used. When the width of the vertical edge is >= 32, and when the height of the horizontal edge is >= 32, the sample is defined as belonging to a large block. The bilinear filter is listed as follows. Then, the block boundary samples pi (i=0 to Sp-1) and qi (j=0 to Sq-1) in the HEVC deblocking described above (pi and qi are the i-th sample in the row for filtering the vertical edge, or the i-th sample in the column for filtering the horizontal edge) are replaced by the following linear interpolation:

[0194] p i ′=(f i *Middle s,t +(64-fi i )*P s +32)>>6), limit to p i ±tcPD i

[0195] q j ′=(g j *Middle s,t +(64-g j )*Q s +32)>>6), limit to q j ±tcPD j

[0196] tcPD i and tcPD j The term is position-dependent clipping as described above, and g j 、f i 、Middle s,t 、P s and Q s Given below.

[0197] 2.4.5 Chroma Deblocking Control

[0198] A chroma strong filter is applied on both sides of the block boundary. Here, the chroma filter is selected when the distance between the two sides of the chroma edge is greater than or equal to 8 (chroma position), and the following decision with three conditions is met: The first decision is for the boundary strength and large block decision. The filter can be applied when the block width or height orthogonally crossing the block edge is equal to or greater than 8 in the chroma sample domain. The second and third decisions are essentially the same as the HEVC luma deblocking decisions, which are the on / off decision and the strong filter decision, respectively.

[0199] In the first decision, the boundary strength (bS) is modified for chroma filtering and the conditions are checked sequentially. If a condition is met, the remaining conditions with lower priority are skipped. When bS is equal to 2, or when bS is equal to 1 in the case of a large block boundary detected, chroma deblocking is performed. The second and third conditions are essentially the same as the HEVC luma strong filter decision below.

[0200] In the second condition, d is derived in the same way as HEVC luma deblocking. When d is less than β, the second condition will be true. In the third condition, StrongFilterCondition is derived as follows:

[0201] Derived dpq in the same way as HEVC.

[0202] Derived according to HEVC method sp3 = Abs (p3-p0)

[0203] According to the HEVC method, sq3 = Abs(q0-q3)

[0204] According to HEVC design, StrongFilterCondition = (dpq less than (β>>2), sp3+sq3 less than (β>>3), and Abs(p0-q0) less than (5*tC+1)>>1)

[0205] 2.4.6 Strong Deblocking Filter for Chroma

[0206] The strong deblocking filter for chroma is defined as follows:

[0207] p2′=(3*p3+2*p2+p1+p0+q0+4)>>3

[0208] p1′=(2*p3+p2+2*p1+p0+q0+q1+4)>>3

[0209] p0′=(p3+p2+p1+2*p0+q0+q1+q2+4)>>3

[0210] The example chroma filter performs deblocking on a 4x4 chroma sample grid.

[0211] 2.4.7 Position-dependent limiting

[0212] Position-dependent clipping (tcPD) is applied to the output samples of the luma filtering process, which involves modifying strong and long filters of 7, 5, and 3 samples at the boundaries. Assuming the quantization error distribution, the clipping value can be increased for samples expected to have higher quantization noise, so the reconstructed sample values ​​are expected to deviate more from the true sample values.

[0213] For each P or Q boundary filtered with the asymmetric filter, a position-dependent threshold table is selected from two tables (e.g., Tc7 and Tc3 in the following table) provided to the decoder as side information, according to the result of the decision-making process:

[0214] Tc7={6,5,4,3,2,1,1}; Tc3={6,4,2};

[0215] tcPD=(Sp==3)? Tc3:Tc7;

[0216] tcQD=(Sq==3)? Tc3:Tc7;

[0217] For P or Q boundaries filtered with a short symmetric filter, a lower magnitude position-dependent threshold is applied:

[0218] Tc3={3,2,1};

[0219] After defining the thresholds, the filtered p'i and q'i sample values ​​are clipped according to the tcP and tcQ clipping values:

[0220] p”i=Clip3(p’i+tcPi,p’i–tcPi,p’i);

[0221] q”j=Clip3(q’j+tcQj,q’j–tcQj,q’j);

[0222] where p'i and q'i are the filtered sample values, p"i and q"j are the clipped output sample values, and tcPitcPi is the clipping threshold derived from the VVC tc parameters and tcPD and tcQD. Function Clip3 is the clipping function as specified in VVC.

[0223] 2.4.8 Sub-block Deblocking Adjustment

[0224] To achieve parallel-friendly deblocking using both long filters and sub-block deblocking, the long filter is restricted to modifying at most 5 samples on one side when using sub-block deblocking (AFFINE or Advanced Temporal Motion Vector Prediction (ATMVP) or Decoder-Side Motion Vector Refinement (DMVR)), as shown in the luma control of the long filter. In addition, sub-block deblocking is adjusted so that sub-block boundaries on the 8x8 grid close to the codec unit (CU) or implicit transform unit (TU) boundary are restricted to modifying at most two samples on each side.

[0225] The following applies to sub-block boundaries that are not aligned with CU boundaries.

[0226] If (block Q's mode == SUBBLOCKMODE && edge != 0) {

[0227] if(!(implicitTU&&(edge==(64 / 4))))

[0228] if(edge==2||edge==(orthogonalLength-2)||edge==(56 / 4)||edge==(72 / 4))

[0229] Sp = Sq = 2;

[0230] else

[0231] Sp = Sq = 3;

[0232] else

[0233] Sp=Sq=bSideQisLargeBlk? 5:3

[0234] }

[0235] Where edge equals 0 corresponds to a CU boundary, edge equals 2 or equal to orthogonalLength-2 corresponds to a sub-block boundary 8 samples away from the CU boundary, etc. If implicit partitioning of TUs is used, implicitTU is true.

[0236] 2.5SAO

[0237] The input of SAO is the reconstructed samples after DB. The concept of SAO is to reduce the average sample distortion of the region by first classifying the region samples into multiple categories using a selected classifier, obtaining an offset for each category, and then adding the offset to each sample of the category, where the classifier index and the offset of the region are encoded and decoded in the bitstream. In HEVC and VVC, the region (the unit used for SAO parameter signaling) is defined as a CTU.

[0238] HEVC uses two SAO types that can meet low complexity requirements. These two types are edge-offset (EO) and band-offset (BO), which are discussed in further detail below. The index of the SAO type is encoded and decoded (it is in the range of [0,2]). For EO, ​​sample classification is based on the comparison between the current sample and the neighboring samples according to a one-dimensional (1-D) directional mode (horizontal, vertical, 135° diagonal, and 45° diagonal).

[0239] Figure 8 Directional patterns for EO sample classification are shown.

[0240] For a given EO class, each sample point within the CTB is classified into one of five categories. The current sample point value (labeled "c") is compared with its two nearest neighbors along the selected 1-D pattern. The classification rules for each sample point are summarized in Table 1. Categories 1 and 4 are associated with local valleys and local peaks, respectively, along the selected 1-D pattern. Categories 2 and 3 are associated with concave corners and convex corners, respectively, along the selected 1-D pattern. If the current sample point does not belong to EO categories 1-4, it is class 0 and SAO is not applied.

[0241] Table 3: Sample point classification rules for edge offset

[0242] category condition 1 c < a and c < b 2 (c<a&&c==b)||(c==a&&c<b) 3 (c>a&&c==b)||(c==a&&c>b) 4 c>a&&c>b 5 None of the above

[0243] 2.6 Geometric Transformation-Based Adaptive Loop Filter in JEM

[0244] The input of DB is the reconstructed samples after DB and SAO. The sample classification and filtering process is based on the reconstructed samples after DB and SAO.

[0245] In JEM, a geometric transform-based adaptive loop filter (GALF) with block-based filter adaptation is applied [3]. For the luminance component, one of 25 filters is selected for each 2x2 block based on the direction and activity of the local gradient.

[0246] 2.6.1 Filter Shape

[0247] Figure 9 GALF filter shapes are shown. In JEM, up to three diamond filter shapes can be selected for the luminance component (e.g. Figure 9 (as shown). An index is signaled at the picture level to indicate the filter shape for the luma component. Each square represents a sample, and Ci (i is 0 to 6 (left), 0 to 12 (center), 0 to 20 (right)) represents the coefficient to be applied to the sample. For the chroma components in a picture, a 5×5 diamond shape is always used.

[0248] 2.6.1.1 Block Classification

[0249] Each 2×2 block is classified into one of 25 categories. The classification index C is based on its directionality D and activity The quantized value of is derived as follows:

[0250]

[0251] To calculate D and First, use the 1-D Laplacian to calculate the gradient in the horizontal, vertical, and two diagonal directions:

[0252]

[0253] The indices i and j refer to the coordinates of the top left sample in the 2×2 block, and R(i,j) indicates the reconstructed sample at coordinate (i,j). The maximum and minimum values ​​of the gradient D in the horizontal and vertical directions are set to:

[0254]

[0255] And the maximum and minimum values ​​of the gradients in the two diagonal directions are set to:

[0256]

[0257] To derive the value of the directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0258] Step 1. If and are both true, then D is set to 0.

[0259] Step 2. If Then continue with step 3; otherwise, continue with step 4.

[0260] Step 3. If Then D is set to 2; otherwise D is set to 1.

[0261] Step 4. If Then D is set to 4; otherwise D is set to 3.

[0262] The activity value A is calculated as:

[0263]

[0264] A is further quantized to the range of 0 to 4 (inclusive), and the quantized value is expressed as For the two chroma components in a picture, no classification method is applied, ie, a single set of ALF coefficients is applied for each chroma component.

[0265] 2.6.1.2 Geometric Transformation of Filter Coefficients

[0266] Figure 10 An example of relative coordinates for a 5×5 diamond filter support is shown. Before filtering each 2×2 block, geometric transformations (such as rotations or diagonal and vertical flips) are applied to the filter coefficients f(k, l) associated with the coordinates (k, l) according to the gradient values ​​calculated for that block. This is equivalent to applying these transformations to the samples in the filter support area. The idea is to make different blocks with the ALF applied more similar by aligning their directionality.

[0267] Three geometric transformations are introduced, including diagonal, vertical flip and rotation:

[0268] Diagonal: f D (k,l)=f(l,k),(9)Vertical flip:f V (k,l)=f(k,Kl-1),

[0269] Rotation: f R (k,l)=f(Kl-1,k).

[0270] Where K is the size of the filter, and 0≤k,l≤K-1 are the coefficient coordinates, such that position (0,0) is in the upper left corner and position (K-1,K-1) is in the lower right corner. The transform is applied to the filter coefficients f(k,l) according to the gradient value calculated for that block. Table 4 summarizes the relationship between the transform and the four gradients in the four directions. Figure 10 The transformed coefficients for each position based on a 5x5 diamond are shown.

[0271] Gradient value Transform <![CDATA[g d2 <g d1 And g h <g v ]]> No transformation <![CDATA[g d2 <g d1 And g v <g h ]]> diagonal <![CDATA[g d1 <g d2 And g h <g v ]]> Flip vertically <![CDATA[g d1 <g d2 And g v <g h ]]> Rotation

[0272] Table 4 shows the mapping between the gradient and transformation calculated by a block.

[0273] 2.6.1.3 Filter Parameter Signaling

[0274] In JEM, GALF filter parameters are signaled for the first CTU, i.e., after the slice header and before the SAO parameters for the first CTU. Up to 25 sets of luma filter coefficients can be signaled. To reduce bit overhead, filter coefficients from different classifications can be merged. In addition, the GALF coefficients of the reference picture are stored and can be reused as the GALF coefficients for the current picture. The current picture can choose to use the GALF coefficients stored for the reference picture and bypass GALF coefficient signaling. In this case, only the index indicating one of the reference pictures is signaled, and the stored GALF coefficients of the indicated reference picture are inherited for the current picture.

[0275] To support GALF temporal prediction, a candidate list of GALF filter banks is maintained. When decoding a new sequence begins, the candidate list is empty. After decoding a picture, the corresponding filter bank can be added to the candidate list. Once the candidate list size reaches the maximum allowed value (e.g., 6 in JEM), the new filter bank overwrites the oldest bank in decoding order, i.e., the candidate list is updated using a first-in-first-out (FIFO) rule. To avoid duplication, a bank can be added to the list only if the corresponding picture does not use GALF temporal prediction. To support temporal scalability, multiple candidate lists of filter banks exist, and each candidate list is associated with a temporal layer. More specifically, each array assigned by a temporal layer index (TempIdx) can constitute a filter bank from a previously decoded picture with a TempIdx equal to or lower than k. For example, the kth array is assigned to be associated with a TempIdx equal to k, and the kth array only contains filter banks from pictures with a TempIdx less than or equal to k. After encoding and decoding a particular picture, the filter bank associated with that picture will be used to update those arrays associated with TempIdx equal to or higher than TempIdx.

[0276] Temporal prediction of the GALF coefficients is used for inter-frame codec frames to minimize signaling overhead. For intra frames, temporal prediction is not available, and a set of 16 fixed filters is assigned to each class. To indicate the use of fixed filters, a flag for each class is signaled, and if necessary, the index of the selected fixed filter is also signaled. Even when a fixed filter is selected for a given class, the coefficients of the adaptive filter f(k,l) for that class can still be sent, in which case the coefficients of the filter to be applied to the reconstructed image are the sum of the two sets of coefficients.

[0277] The filtering process of the luma component can be controlled at the CU level. A signal is transmitted to indicate whether GALF is applied to the luma component of the CU. For chroma components, whether GALF is applied is only indicated at the picture level.

[0278] 2.6.1.4 Filtering process

[0279] At the decoder side, when GALF is enabled for a block, each sample R(i,j) in the block is filtered to obtain the sample value R′(i,j) as shown below, where L represents the filter length and f m,n represents the filter coefficient, and f(k, l) represents the decoded filter coefficient.

[0280]

[0281] Figure 11The example of relative coordinates used for a 5x5 diamond filter support is shown, assuming that the coordinates (i, j) of the current sample point are (0, 0). Samples in different coordinates filled with the same color are multiplied by the same filter coefficient.

[0282] 2.7 Geometric Transformation-Based Adaptive Loop Filter (GALF) in VVC

[0283] 2.7.1 GALF in VTM-4

[0284] In VTM4.0, the filtering process of the adaptive loop filter is performed as follows:

[0285] O(x,y)=∑ (i,j) w(i,j).I(x+i,y+j), (11)

[0286] Wherein, the sample point I(x+i,y+j) is the input sample point, O(x,y) is the output sample point after filtering (i.e., the filtering result), and w(i,j) represents the filter coefficient. In practice, in VTM4.0, it uses integer arithmetic to implement fixed-point precision calculations:

[0287]

[0288] where L represents the filter length and w(i,j) are the filter coefficients in fixed-point precision.

[0289] Compared with GALF in JEM, the example design of GALF in VVC has the following major changes:

[0290] 1) Remove adaptive filter shape. For luma components, only 7x7 filter shape is allowed, while for chroma components, 5x5 filter shape is allowed.

[0291] 2) Remove the signaling of ALF parameters from the slice / picture level to the CTU level.

[0292] 3) Class index calculation is performed at the 4x4 level instead of the 2x2 level. Additionally, as proposed in JVET-L0147, a downsampled Laplacian calculation method is utilized for ALF classification. More specifically, the horizontal / vertical / 45° diagonal / 135° gradients for each sample point within a block do not need to be calculated. Instead, 1:2 downsampling is utilized.

[0293] 2.8 Nonlinear ALF in VVC

[0294] 2.8.1 Nonlinear Filtering Reconstruction

[0295] Equation (11) can be reconstructed into the following expression without affecting the coding efficiency:

[0296] O(x,y)=I(x,y)+∑ (i,j)≠(0,0) w(i,j).(I(x+i,y+j)-I(x,y)) (13)

[0297] where w(i,j) are the same filter coefficients as in equation (11) [except that w(0,0) is equal to 1 in equation (13) and 1-∑ (i,j)≠(0,0) w(i,j)].

[0298] Using the above filter formula (13), VVC introduces nonlinearity by using a simple clipping function to reduce the influence of neighboring sample values ​​when the neighboring sample values ​​(I(x+i,y+j))) differ too much from the current sample value being filtered (I(x,y)), thereby making the ALF more efficient. More specifically, the ALF filter is modified as follows:

[0299] O′(x,y)=I(x,y)+∑ (i,j)≠(0,0) w(i,j).K(I(x+i,y+j)-I(x,y),k(i,j)) (14)

[0300] where K(d,b)=min(b,max(-b,d)) is the clipping function and k(i,j) is the clipping parameter, which depends on the (i,j) filter coefficients. The encoder performs an optimization to find the best k(i,j).

[0301] In the JVET-N0242 implementation, a clipping parameter k(i,j) is specified for each ALF filter, with one clipping value signaled for each filter coefficient. This means that up to 12 clipping values ​​can be signaled in the bitstream for each luma filter, and up to 6 clipping values ​​can be signaled in the bitstream for each chroma filter. To limit signaling cost and encoder complexity, only 4 fixed values ​​are used that are the same for both inter and intra slices.

[0302] Because the variance of local differences for luma is typically higher than for chroma, two different groups are applied to luma and chroma filters. A maximum sample value in each group (here 1024 for a 10-bit bit depth) is also introduced so that clipping can be disabled if not necessary.

[0303] The set of clipping values ​​used in the JVET-N0242 test is provided in Table 5. The four values ​​are selected by dividing the entire range of sample values ​​for luma (encoded on 10 bits) and the range from 4 to 1024 for chroma roughly equally in the logarithmic domain. More precisely, the luma table of clipping values ​​has been obtained by the following formula:

[0304]

[0305] Similarly, the chromaticity table of the clipped value is obtained according to the following formula:

[0306]

[0307] Table 5: Authorized limit values

[0308]

[0309] The selected clipping value is encoded in the "alf_data" syntax element by using the Golomb coding scheme corresponding to the index of the clipping value in the above Table 5. This coding scheme is the same as the coding scheme used for the filter index.

[0310] 2.9 Convolutional Neural Network-Based Loop Filters for Video Codecs

[0311] 2.9.1 Convolutional Neural Networks

[0312] In deep learning, convolutional neural networks (CNNs or ConvNets) are a type of deep neural network that are most commonly used to analyze visual images. They have been very successful in image and video recognition / processing, recommendation systems, image classification, medical image analysis, and natural language processing.

[0313] CNNs are regularized versions of multilayer perceptrons. Multilayer perceptrons are often referred to as fully connected networks, meaning that every neuron in one layer is connected to every neuron in the next layer. The "fully connected" nature of these networks makes them prone to overfitting the data. Typical regularization methods involve adding some form of magnitude measure of the weights to the loss function. CNNs take a different approach to regularization: they exploit hierarchical patterns in the data and use smaller, simpler patterns to assemble more complex patterns. Therefore, on a scale of connectivity and complexity, CNNs are at the lower end of the spectrum.

[0314] Compared to other image classification / processing algorithms, CNNs use relatively little preprocessing. This means the network learns filters that are manually designed in traditional algorithms. This lack of reliance on prior knowledge and manual feature design is a major advantage.

[0315] 2.9.2 Deep Learning for Image / Video Encoding and Decoding

[0316] Image / video compression based on deep learning generally has two meanings: pure neural network-based end-to-end compression [1,2] and frameworks enhanced by neural networks [3,4,5,6]. The first category usually adopts a structure similar to an autoencoder, implemented by a convolutional neural network or a recurrent neural network. Although relying purely on neural networks for image / video compression can avoid any manual optimization or artificial design, the compression efficiency may not be satisfactory. Therefore, the work distributed in the second category uses neural networks as an auxiliary and enhances the traditional compression framework by replacing or enhancing some modules. In this way, they can inherit the advantages of highly optimized frameworks. For example, J.Li et al. proposed a fully connected network for intra-frame prediction in HEVC [3]. In addition to intra-frame prediction, deep learning is also used to enhance other modules. For example, Y.Dai et al. replaced the loop filter of HEVC with a convolutional neural network and achieved promising results [4]. The work in [5] applied neural networks to improve the arithmetic codec engine.

[0317] 2.9.3 Loop Filtering Based on Convolutional Neural Networks

[0318] In lossy image / video compression, the reconstructed frame is an approximation of the original frame because the quantization process is irreversible, resulting in distortion in the reconstructed frame. To mitigate this distortion, a convolutional neural network can be trained to learn the mapping from the distorted frame to the original frame. In practice, training must be performed before deploying the CNN-based loop filter.

[0319] 2.9.3.1 Training

[0320] The goal of the training process is to find optimal values ​​for the parameters including weights and biases.

[0321] First, the training dataset is compressed using a codec (e.g., HM, JEM, VTM, etc.) to generate distorted reconstructed frames.

[0322] The reconstructed frame is then fed into the CNN, and the cost is calculated using the CNN output and the ground truth frame (original frame). Common cost functions include the sum of absolute differences (SAD) and mean square error (MSE). Next, the gradient of the cost with respect to each parameter is derived through the backpropagation algorithm. Using the gradient, the value of the parameter can be updated. The above process is repeated until the convergence criterion is met. After training is completed, the derived optimal parameters are saved and used in the inference phase.

[0323] 2.9.3.2 Convolution Process

[0324] Figure 12AAn example CNN is shown. During convolution, the filter moves across the image from left to right and top to bottom, changing one pixel column horizontally and one pixel row vertically. The amount of movement between filter applications to the input image is called the stride, and it is almost always symmetric across the height and width dimensions. For both height and width movement, the default stride or multiples of stride are (1,1) in both dimensions.

[0325] In most deep convolutional neural networks, residual blocks are used as basic modules and are stacked multiple times to build the final network, where in one example, Figure 12B As shown in Figure 3, the residual block is obtained by combining a convolutional layer, a ReLU / PReLU activation function, and a convolutional layer.

[0326] 2.9.3.3 Reasoning

[0327] During the inference phase, the distorted reconstructed frame is fed into the CNN and processed by the CNN model, whose parameters have been determined in the training phase. The input samples of the CNN can be the reconstructed samples before or after DB, or the reconstructed samples before or after SAO, or the reconstructed samples before or after ALF.

[0328] 3. Technical Problems Solved by the Disclosed Technical Solution

[0329] The example design for neural network based loop filtering has the following problems.

[0330] First, different models are trained for different types of stripes, resulting in large model storage.

[0331] Second, neural network (NN)-based filters do not consider the impact of different codec modes. For example, the distortion of the reconstructed intra frame may be related to the prediction angle. However, this information may not be used in the NN-based filtering process.

[0332] 4. List of solutions and implementation examples

[0333] The following detailed list should be considered as examples to explain the general concept. These examples should not be interpreted in a narrow sense. In addition, these examples can be combined in any way.

[0334] One or more neural network (NN) filter models are trained as part of a loop filtering technique or a filtering technique used in a post-processing stage to reduce distortion generated during compression. Samples with different characteristics are processed by different NN filter models. The present disclosure details how to design a unified NN filter model by feeding at least one indicator that can be related to quality level (e.g., QP or constant rate factor (CRF) value or bit rate) / slice type / codec mode / encoded information as input to the NN filter.

[0335] It should be noted that the concept of unifying NN models by feeding indicators as input to the NN process can also be extended to other NN-based codec tools, such as NN-based intra prediction, NN-based cross-component prediction, NN-based inter prediction, NN-based super-resolution, NN-based motion compensation, and NN-based transform design. In the following examples, NN-based filtering techniques are used as examples.

[0336] It should also be noted that the concept of feeding the post-codec information as input to the NN process can also be extended to non-NN based codec tools, such as non-NN based intra prediction, non-NN based cross-component prediction, non-NN based inter prediction, non-NN based super-resolution, non-NN based motion compensation and non-NN based transform design. For example, non-NN based codec tools can use the post-codec information to classify samples to be filtered into different categories.

[0337] In the present disclosure, the NN filter may be any type of NN filter, such as a convolutional neural network (CNN) filter, a fully connected neural network filter, a transformer-based filter, or a recurrent neural network-based filter. In the following discussion, the NN filter may also be referred to as a CNN filter.

[0338] In the following discussion, a video unit can be a sequence, a picture, a slice, a tile, a sub-picture, a codec tree unit (CTU) / codec tree block (CTB), a CTU / CTB row, one or more codec units (CU) / codec blocks (CB), one or more CTUs / CTBs, one or more virtual pipeline data units (VPDUs), a sub-region within a picture / slice / slice / tile, etc. A parent video unit represents a unit larger than a video unit. Typically, a parent unit contains several video units. For example, when the video unit is a CTU, the parent unit can be a slice, a CTU row, multiple CTUs, etc.

[0339] On Designing a Unified NN Filter Model

[0340] 1. The NN filter model may receive as input at least one indicator related to the slice type, which is called a slice type indicator (STI).

[0341] a. In one example, the stripe type indicator is represented by a value derived from the stripe type.

[0342] i. In one example, the stripe type is derived as follows, where a, b, and c are constants.

[0343]

[0344] 1) In one example, b=c.

[0345] 2) In one example, b=-a, c=-a.

[0346] 3) In one example, a=1, b=-1, c=-1.

[0347] 4) In one example, a=1, b=0.5, c=0.5.

[0348] b. In one example, the slice type indicator is first tiled or expanded into a two-dimensional array of the same size as the video unit to be filtered, and then treated as an additional input plane (see the tiling process for details). Figure 13 ). Figure 13 An example of tiling the slice type indicators (STI) into a two-dimensional array is shown.

[0349] i. In one example, the video unit is a CTU / CTB.

[0350] c. In one example, the STI may be signaled from the encoder to the decoder, for example in a sequence parameter set (SPS) / picture parameter set (PPS) / picture header / slice header / CTU / CU or at any region level.

[0351] d. In one example, f(STI) instead of STI can be fed into the NN, where f is an arbitrary function.

[0352] e. In one example, the use of STI may depend on the color component.

[0353] i. For example, a first STI may be used for a first component (such as luma), and a second STI may be used for a second component (such as chroma, e.g., Cb or Cr). The first STI may indicate the quality level of the first component, and the second STI may indicate the quality level of the second component.

[0354] ii. For example, a first STI may be used for a first component (such as luma), and a first STI may also be used for a second component (such as chroma, eg, Cb or Cr).The first STI may indicate a quality level of the first component.

[0355] iii. For example, STI may be used for NN filtering of a first component (such as luma), but not for NN filtering of a second component (such as chroma, eg, Cb or Cr). For example, STI may be used for NN filtering of all components.

[0356] f. “Slice type” can be replaced by “picture type”, “frame type”, “block type”, etc.

[0357] g. In one example, if a picture is divided into multiple slices, certain operations may be performed.

[0358] i. The slice type of one slice (such as the first slice) may be used by the NN for the entire picture.

[0359] ii. The stripe type of a stripe can be used by the NN for the samples in the stripe.

[0360] 2. The NN filter model can receive as input at least one indicator related to the codec mode, which is called a codec mode indicator (CMI).

[0361] a. In one example, the codec mode indicator is derived based on the codec mode.

[0362] i. In one example, the codec mode indicator may include intra prediction mode, inter prediction mode, etc.

[0363] ii. In one example, the codec mode may include IBC mode, palette mode, etc.

[0364] b. In one example, the codec mode indicator is a two-dimensional array with the same resolution as the video unit to be filtered. Each sample in the array is represented by a value that reflects the codec mode associated with the CU to which the sample belongs.

[0365] i. In one example, the codec mode indicator is derived as follows, where i and j are the vertical index and horizontal index inside the array, intra_pred_mode is the intra prediction direction associated with the CU to which the sample (i, j) belongs, and inter_pred_mode is the inter prediction mode associated with the CU to which the sample (i, j) belongs.

[0366] 1) In one example, possible inter prediction modes include Skip mode, Merge mode, and AMVP mode.

[0367]

[0368] c. In one example, CMI may be signaled from the encoder to the decoder, for example in SPS / PPS / picture header / slice header / CTU / CU or at any region level.

[0369] d. In one example, f(CMI) instead of CMI can be fed into the NN, where f is an arbitrary function.

[0370] e. In one example, the use of CMI may depend on the color component.

[0371] i. For example, a first CMI may be used for a first component (such as luma), and a second CMI may be used for a second component (such as chroma, e.g., Cb or Cr). The first CMI may indicate a quality level of the first component, and the second CMI may indicate a quality level of the second component.

[0372] ii. For example, a first CMI may be used for a first component (such as luma), and the first CMI may also be used for a second component (such as chroma, eg, Cb or Cr).The first CMI may indicate a quality level of the first component.

[0373] iii. For example, CMI may be used for NN filtering of a first component (such as luma), but not for NN filtering of a second component (such as chroma, eg, Cb or Cr).

[0374] iv. For example, CMI can be used for NN filtering of all components.

[0375] 3. The NN filter model may receive as input the encoded and / or decoded information and / or information reconstructed and / or derived from the encoded and decoded information.

[0376] a. In one example, the coded information may be a quantization parameter (QP) or a parameter derived based on the QP.

[0377] b. In one example, the coded information may be a prediction direction or may be derived based on the prediction direction.

[0378] c. In one example, the coded information may be a reference picture index or may be derived based on a reference picture index.

[0379] d. In one example, the encoded information may be a picture order count (POC) distance or a derivation based on the POC distance.

[0380] e. In one example, the coded information may be the number of reference pictures or may be derived based on the number of reference pictures.

[0381] f. In one example, the coded information may be a time domain layer ID or may be derived based on a time domain layer identifier (ID, or ID).

[0382] g. In one example, the coded information may be a motion vector or a signal derived based on a motion vector.

[0383] h. In one example, the encoded information may be a motion vector difference or a derivative based on a motion vector difference.

[0384] i. In one example, the coded information may be a transform type or may be derived based on a transform type.

[0385] j. In one example, the coded information may be residual information or derived based on residual information.

[0386] k. In one example, the coded information may be a coded block flag (CBF) or derived based on the CBF.

[0387] 1. In one example, the coded information can be derived based on whether other loop filters (eg, deblocking filter, SAO, ALF, CC-ALF, bilateral filter, etc.) have been turned on.

[0388] m. In one example, the coded information can be derived based on whether the sample is modified by other loop filters (eg, deblocking filter, SAO, ALF, CC-ALF, bilateral filter, etc.).

[0389] n. In one example, several encoded and decoded information mentioned above can be fed into the NN filter together.

[0390] 4. STI, CMI and the encoded and decoded information mentioned in the last item (item 3) can be used together by the NN filter.

[0391] a. Optionally, one of the STI, CMI and coded information mentioned in the last item (item 3) can be used by the NN filter, excluding the other two.

[0392] 5. At least one syntax element may be signaled to indicate whether the STI and / or CMI and / or coded information mentioned in item 3 may be used by the NN filter.

[0393] 6. At least one syntax element may be signaled to indicate how the STI and / or CMI and / or coded information mentioned in item 3 is used by the NN filter, for example, to indicate how to match the type / mode to the value input in the NN.

[0394] General Items

[0395] 7. The syntax elements disclosed above can be binarized as flags, fixed length codes, Exponential-Golomb codes (EG(x)), unary codes, truncated unary codes, truncated binary codes, etc. It can be signed or unsigned.

[0396] 8. The syntax elements disclosed above can be encoded or decoded using at least one context model, or they can be bypassed.

[0397] 9. The syntax elements (SE) disclosed above may be signaled in a conditional manner.

[0398] a. SE is transmitted via signal only when the corresponding function is applicable.

[0399] 10. The syntax elements disclosed above may be signaled at block level, sequence level, group of picture level, picture level, slice level, or slice group level, such as in the codec structure of a CTU, CU, transform unit (TU), prediction unit (PU), codec tree block (CTB), codec block (CB), transform block (TB), or prediction block (PB), or in a sequence header, picture header, SPS, VPS, DPS, DCI, PPS, APS, slice header, or slice group header.

[0400] 11. Whether and / or how to apply the method disclosed above can be transmitted by signaling at the block level / sequence level / picture group level / picture level / slice level / slice group level, such as in the codec structure of CTU, CU, TU, PU, ​​CTB, CB, TB, PB, or in a sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependency parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptation parameter set (APS), slice header or slice group header.

[0401] 12. Whether and / or how to apply the above disclosed methods may depend on coded information such as block size, color format, single / dual tree partitioning, color components, slice / picture type.

[0402] 13. The proposed method disclosed in this disclosure can be used in other codec tools that require chroma fusion.

[0403] 5. References

[0404] [1] Johannes Ballé, Valero Laparra, and Eero P Simoncelli, 2016, End-to-end optimization of nonlinear transform codes for perceptual quality, PCS. IEEE, 1-5.

[0405] [2] Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. 2017. Lossy image compression with compressive autoencoders. arXiv preprint arXiv:1703.00395 (2017).

[0406] [3] Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen Gao, 2018, Fully Connected Network-Based Intra Prediction for Image Coding, IEEE Transactions on Image Processing 27, 7 (2018), 3236-3247.

[0407] [4] Yuanying Dai, Dong Liu, and Feng Wu, 2017, A convolutional neural network approach for post-processing in HEVC intra coding, MMM. Springer, 28-39.

[0408] [5] Rui Song, Dong Liu, Houqiang Li, and Feng Wu, 2017, Neural network-based arithmetic coding of intra prediction modes in HEVC, VCIP. IEEE, 1-4.

[0409] [6] J Pfaff, P Helle, D Maniry, S Kaltenstadler, W Samek, H Schwarz, D Marpe, and T Wiegand, 2018, Neural network based intra prediction for video coding, in Applications of Digital Image Processing XLI, vol. 10752, International Society for Optics and Photonics, 1075213.

[0410] Figure 14 is a block diagram illustrating an example video processing system 4000 in which the various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 4000. System 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10 bit multi-component pixel values), or may be in a compressed or encoded format. Input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, a passive optical network (PON), etc.) and wireless interfaces, such as wireless fidelity (Wi-Fi) or a cellular interface.

[0411] System 4000 may include a codec component 4004 that can implement the various codecs or encoding methods described in this disclosure. Codec component 4004 can reduce the average bit rate of the video from input 4002 to the output of codec component 4004 to generate a codec representation of the video. Codec technology is therefore sometimes referred to as video compression or video transcoding technology. The output of codec component 4004 can be stored or transmitted via a communication connection such as represented by component 4006. The bitstream (or codec) representation of the video stored or communicated received at input 4002 can be used by component 4008 to generate pixel values ​​or displayable video that is sent to display interface 4010. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "encoding" operations or tools, it should be understood that the encoding tools or operations are used by the encoder, and the corresponding decoding tools or operations that reverse the encoding and decoding results will be performed by the decoder.

[0412] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High-Definition Multimedia Interface (HDMI) or DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interface, etc. The technology described in this disclosure may be embodied in various electronic devices, such as mobile phones, laptop computers, smartphones, or other devices capable of performing digital data processing and / or video display.

[0413] Figure 15 4 is a block diagram of an example video processing device 4100. Device 4100 can be used to implement one or more methods described herein. Device 4100 can be implemented as a smartphone, tablet computer, computer, Internet of Things (IoT) receiver, etc. Device 4100 may include one or more processors 4102, one or more memories 4104, and video processing circuitry 4106. Processor(s) 4102 can be configured to implement one or more methods described in this disclosure. Memory(s) 4104 can be used to store data and code used to implement the methods and techniques described herein. Video processing circuitry 4106 can be used to implement some of the techniques described in this disclosure in hardware circuitry. In some embodiments, video processing circuitry 4106 can be at least partially included in processor 4102, for example, a graphics coprocessor.

[0414] Figure 16 4 is a flow chart of an example method 4200 for video processing. The method 4200 includes, at step 4202, determining to apply a NN model to visual media data. The NN model receives as input a slice type indicator (STI) indicating a slice type. At step 4204, conversion between the visual media data and a bitstream is performed based on the NN model. According to an example, the conversion at step 4204 may include encoding at an encoder or decoding at a decoder.

[0415] It should be noted that method 4200 can be implemented in an apparatus for processing video data, the apparatus including a processor and non-transitory memory having instructions thereon, such as video encoder 4400, video decoder 4500, and / or encoder 4600. In this case, the instructions, when executed by the processor, cause the processor to perform method 4200. Furthermore, method 4200 can be performed by a non-transitory computer-readable medium comprising a computer program product for use by a video codec apparatus. The computer program product includes computer-executable instructions stored on a non-transitory computer-readable medium, such that when executed by the processor, the video codec apparatus performs method 4200.

[0416] Figure 17 4 is a block diagram illustrating an example video codec system 4300 that can utilize the techniques of this disclosure. Video codec system 4300 can include a source device 4310 and a destination device 4320. Source device 4310 generates encoded video data, where source device 4310 can be referred to as a video encoding device. Destination device 4320 can decode the encoded video data generated by source device 4310, where destination device 4320 can be referred to as a video decoding device.

[0417] Source device 4310 may include a video source 4312, a video encoder 4314, and an input / output (I / O) interface 4316. Video source 4312 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data may include one or more pictures. Video encoder 4314 encodes the video data from video source 4312 to generate a bitstream. The bitstream may include a sequence of bits that form a codec representation of the video data. The bitstream may include coded pictures and associated data. A coded picture is a codec representation of the picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 4316 may include a modulator / demodulator (modem) and / or a transmitter. The coded video data may be transmitted directly to target device 4320 via network 4330 via I / O interface 4316. The coded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0418] The target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. The I / O interface 4326 may include a receiver and / or a modem. The I / O interface 4326 may obtain encoded video data from the source device 4310 or the storage medium / server 4340. The video decoder 4324 may decode the encoded video data. The display device 4322 may display the decoded video data to a user. The display device 4322 may be integrated with the target device 4320, or may be external to the target device 4320, wherein the target device 4320 may be configured to interface with an external display device.

[0419] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other existing and / or further standards.

[0420] Figure 18 is a block diagram illustrating an example of a video encoder 4400, which may be Figure 17 Video encoder 4314 in system 4300 is shown. Video encoder 4400 can be configured to perform any or all of the techniques of this disclosure. Video encoder 4400 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 4400. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0421] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402, a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a cache 4413 and an entropy coding unit 4414. The prediction unit 4402 may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405 and an intra-frame prediction unit 4406.

[0422] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In one example, the prediction unit 4402 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in accordance with an IBC mode, where at least one reference picture is a picture in which the current video block is located.

[0423] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405 , may be highly integrated, but are represented separately in the example of the video encoder 4400 for purposes of explanation.

[0424] The segmentation unit 4401 may segment a picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0425] The mode selection unit 4403 can, for example, select one of a plurality of codec modes (intra-frame codec or inter-frame codec) based on the error result, and provide the generated intra-frame codec block or inter-frame codec block to the residual generation unit 4407 to generate residual block data, and to the reconstruction unit 4412 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 4403 can select a joint intra-frame and inter-frame prediction (CIIP) mode, in which prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 4403 can also select a resolution for the motion vector for the block (e.g., sub-pixel precision or integer pixel precision).

[0426] To perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 4413. The motion compensation unit 4405 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 4413 other than the picture associated with the current video block.

[0427] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations on the current video block, eg, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0428] In some examples, motion estimation unit 4404 may perform unidirectional prediction on the current video block, and motion estimation unit 4404 may search the reference pictures in list 0 or list 1 to find a reference video block for the current video block. Motion estimation unit 4404 may then generate a reference index indicating the reference picture in list 0 or list 1 containing the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 4404 may output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 4405 may generate a predicted video block for the current block based on the reference video block indicated by the motion information for the current video block.

[0429] In other examples, the motion estimation unit 4404 may perform bidirectional prediction on the current video block. The motion estimation unit 4404 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 4404 may then generate reference indexes indicating the reference pictures in list 0 and list 1 containing the reference video blocks, and a motion vector indicating the spatial displacement between the reference video blocks and the current video block. The motion estimation unit 4404 may output the reference index and motion vector for the current video block as motion information for the current video block. The motion compensation unit 4405 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0430] In some examples, motion estimation unit 4404 may output a complete set of motion information for use in the decoding process of a decoder. In some examples, motion estimation unit 4404 may not output a complete set of motion information for the current video. Instead, motion estimation unit 4404 may reference motion information of another video block to signal the motion information of the current video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0431] In one example, the motion estimation unit 4404 may indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0432] In another example, the motion estimation unit 4404 can identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0433] As discussed above, the video encoder 4400 can predictively signal motion vectors.Two examples of predictive signaling techniques that can be implemented by the video encoder 4400 include Advanced Motion Vector Prediction (AMVP) and Merge mode signaling.

[0434] Intra-frame prediction unit 4406 can perform intra-frame prediction on the current video block. When intra-frame prediction unit 4406 performs intra-frame prediction on the current video block, intra-frame prediction unit 4406 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0435] The residual generation unit 4407 may generate residual data for the current video block by subtracting the predicted video blocks of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0436] In other examples, such as in skip mode, there may be no residual data for the current video block and the residual generation unit 4407 may not perform a subtraction operation.

[0437] Transform processing unit 4408 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to a residual video block associated with the current video block.

[0438] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0439] Inverse quantization unit 4410 and inverse transform unit 4411 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 4412 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by prediction unit 4402 to generate a reconstructed video block associated with the current block for storage in buffer 4413.

[0440] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0441] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives data, the entropy coding unit 4414 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0442] Figure 19 is a block diagram illustrating an example of a video decoder 4500, which may be Figure 17 Video decoder 4324 in system 4300 is shown. Video decoder 4500 can be configured to perform any or all of the techniques of this disclosure. In the example shown, video decoder 4500 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 4500. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0443] In the example shown, video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra-prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, video decoder 4500 may perform a decoding process that is generally substantially the inverse of the encoding process described with respect to video encoder 4400.

[0444] The entropy decoding unit 4501 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded video data blocks). The entropy decoding unit 4501 can decode the entropy-encoded video data, and based on the entropy-encoded video data, the motion compensation unit 4502 can determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 4502 can determine this information, for example, by performing AMVP and Merge modes.

[0445] The motion compensation unit 4502 may generate a motion compensated block and may perform interpolation based on an interpolation filter. An identifier of an interpolation filter to be used with sub-pixel precision may be included in a syntax element.

[0446] The motion compensation unit 4502 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filters used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filters used by the video encoder 4400 based on received syntax information, and may use the interpolation filters to generate a prediction block.

[0447] The motion compensation unit 4502 can use some syntax information to determine the size of the blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partitioning information describing how each macroblock of the pictures of the encoded video sequence is partitioned, a mode indicating how to encode each partition, one or more reference frames (and reference frame lists) for each inter-frame codec block, and other information for decoding the encoded video sequence.

[0448] The intra-frame prediction unit 4503 can form a prediction block from spatially adjacent blocks using, for example, an intra-frame prediction mode received in the bitstream. The inverse quantization unit 4504 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0449] The reconstruction unit 4506 can sum the residual block with the corresponding prediction block generated by the motion compensation unit 4502 or the intra-frame prediction unit 4503 to form a decoded block. If necessary, a deblocking filter can also be used to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra-frame prediction and also produces the decoded video for presentation on a display device.

[0450] Figure 20 is a schematic diagram of an example encoder 4600. The encoder 4600 is suitable for implementing techniques for VVC. The encoder 4600 includes three loop filters, namely a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike the DF 4602, which uses a predefined filter, the SAO 4604 and the ALF 4606 use the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, and using the encoded side information to signal the offset and filter coefficients. The ALF 4606 is located at the last processing stage for each picture and can be seen as a tool that attempts to capture and repair artifacts caused by previous stages.

[0451] The encoder 4600 also includes an intra-frame prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive input video. The intra-frame prediction component 4608 is configured to perform intra-frame prediction, while the ME / MC component 4610 is configured to perform inter-frame prediction using reference pictures obtained from a reference picture cache 4612. The residual block from the inter-frame prediction or intra-frame prediction is fed into a transform (T) component 4614 and a quantization (Q) component 4616 to generate quantized residual transform coefficients, which are fed into an entropy codec component 4618. The entropy codec component 4618 entropy codes the prediction results and quantized transform coefficients and transmits them to a video decoder (not shown). The quantization component output from the quantization component 4616 can be fed into an inverse quantization (IQ) component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. The REC component 4624 can output images to the DF 4602 , SAO 4604 , and ALF 4606 for filtering before these images are stored in the reference picture cache 4612 .

[0452] A list of some example preferred solutions is provided below.

[0453] The following solutions illustrate examples of the techniques discussed herein.

[0454] 1. A method for processing video data, comprising: determining to apply a neural network (NN) model to visual media data, wherein the NN model receives a slice type indicator (STI), the STI indicating a slice type input into the NN model; and performing conversion between the visual media data and a bitstream based on the NN model.

[0455] 2. The method according to solution 1, wherein when the slice type is an intra-frame prediction (I) slice, the STI is set to 'a', when the slice type is a bidirectional inter-frame prediction (B) slice, the STI is set to 'b', and when the slice type is a unidirectional inter-frame prediction (P) slice, the STI is set to 'c'.

[0456] 3. The method according to solution 1 or 2, wherein the STI is included in a two-dimensional (2D) array having the same size as the video unit to be filtered by the NN model, and the STI is regarded as an input plane.

[0457] 4. A method according to any one of solutions 1-3, wherein the STI is included in a sequence parameter set (SPS), a picture parameter set (PPS), a picture header, a slice header, a codec tree unit (CTU), a codec unit (CU) or a combination thereof in the bitstream.

[0458] 5. The method according to any one of solutions 1-4, wherein the STI is encoded and decoded as f(STI), where f is a function.

[0459] 6. The method according to any one of solutions 1-5, wherein multiple STIs are used for different color components.

[0460] 7. The method according to any of solutions 1-6, wherein a single STI is used for multiple color components.

[0461] 8. The method according to any one of solutions 1-7, wherein the STI of a slice is used to indicate a slice type for the entire picture or a sample group thereof.

[0462] 9. A method according to any one of solutions 1-8, wherein the NN model receives a codec mode indicator (CMI), which indicates the codec mode of the video unit input into the NN model.

[0463] 10. A method according to any one of solutions 1-9, wherein the CMI indicates intra prediction mode, inter prediction mode, intra block copy mode, palette mode, skip mode, merge mode, advanced motion vector prediction (AMVP) or a combination thereof.

[0464] 11. A method according to any one of solutions 1-10, wherein the CMI is included in a 2D array having the same resolution as the video unit to be filtered by the NN model, and wherein each sample point in the 2D array indicates the coding mode of the associated coding unit (CU) in the video unit.

[0465] 12. The method according to any one of solutions 1-11, wherein the CMI is encoded and decoded as follows:

[0466]

[0467] 13. The method according to any one of solutions 1-12, wherein the CMI is included in an SPS, a PPS, a picture header, a slice header, a CTU, a CU, or a combination thereof in the bitstream.

[0468] 14. The method according to any one of solutions 1-13, wherein the CMI is encoded and decoded into f(CMI), where f is a function.

[0469] 15. The method according to any of solutions 1-14, wherein multiple CMIs are used for different color components.

[0470] 16. The method according to any of solutions 1-15, wherein a single CMI is used for multiple color components.

[0471] 17. A method according to any of solutions 1-16, wherein the NN model receives encoded and decoded information as input.

[0472] 18. A method according to any one of solutions 1-17, wherein the encoded information includes a quantization parameter (QP), a prediction direction, a reference picture index, a picture order count distance, the number of reference pictures, a temporal layer identifier, a motion vector, a motion vector difference, a transform type, residual information, a coded block flag (CBF), loop filter usage information, sample point filtering information, or a combination thereof.

[0473] 19. A method according to any one of solutions 1-18, wherein the STI, CMI and encoded and decoded information are used jointly by the NN model.

[0474] 20. The method of any one of solutions 1-19, wherein the bitstream includes at least one syntax element indicating whether the STI, CMI, and coded information are used by the NN model.

[0475] 21. A method according to any of solutions 1-20, wherein the bitstream includes at least one syntax element indicating how the STI, CMI and encoded information are used by the NN model.

[0476] 22. An apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of solutions 1-21.

[0477] 23. A non-transitory computer-readable medium comprising a computer program product for use by a video codec device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs the method of any one of Solutions 1-21.

[0478] 24. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method includes: determining to apply a neural network (NN) model to visual media data, wherein the NN model receives a slice type indicator (STI), the STI indicating a slice type input into the NN model; and generating the bitstream based on the determination.

[0479] 25. A method for storing a bitstream of a video, comprising: determining to apply a neural network (NN) model to visual media data, wherein the NN model receives a slice type indicator (STI), the STI indicating a slice type input into the NN model; generating the bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0480] 26. A method, apparatus or system as described in the present disclosure.

[0481] In the described solution, an encoder can conform to the format rules by generating a codec representation according to the format rules. In the described solution, a decoder can parse syntax elements in the codec representation according to the format rules using known information about the presence and absence of syntax elements to produce decoded video.

[0482] In the present disclosure, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. For example, the bitstream representation of a current video block may correspond to bits spread at the same or different positions in the bitstream defined by the syntax. For example, a macroblock may be encoded based on error residual values ​​after transformation and encoding and may also use bits in the header and other fields in the bitstream. Furthermore, during conversion, the decoder may parse the bitstream based on this determination, knowing that some fields may or may not be present, as described in the above solution. Similarly, the encoder may determine whether to include or not include specific syntax fields, and generate the codec representation accordingly by including the syntax fields or excluding the syntax fields from the codec representation.

[0483] The disclosed and other solutions, examples, embodiments, modules, and functional operations described in this disclosure may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this disclosure and their structural equivalents, or in one or more combinations thereof. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or to control the operation of a data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a composition of matter that effects a machine-readable propagated signal, or one or more combinations thereof. The term "data processing apparatus" includes all devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may also include code that creates an execution environment for the relevant computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[0484] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including stand-alone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that preserves other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0485] The processes and logic flows described in this disclosure may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0486] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal or removable hard disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated into, special-purpose logic circuitry.

[0487] Although this disclosure contains many details, these details should not be interpreted as limitations on any subject matter or the scope of what may be claimed, but rather as descriptions of features that are unique to particular embodiments of particular technologies. In this disclosure, certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable subcombination. In addition, although features may function in certain combinations as described above, and may even be initially claimed in this manner, in some cases, one or more features in a claimed combination may be omitted from that combination, and a claimed combination may be directed to a subcombination or a variant of a subcombination.

[0488] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed sequentially in the particular order or sequence shown, or that all illustrated operations be performed to achieve desired results. Furthermore, the partitioning of various system components in the embodiments described in this disclosure should not be understood as requiring such partitioning in all embodiments.

[0489] Only a few implementations and examples are described, and other implementations, improvements, and variations can be made based on what is described and illustrated in this disclosure.

[0490] A first component is directly coupled to a second component when there are no intervening components other than a line, trace, or other medium between the first and second components. A first component is indirectly coupled to a second component when there are intervening components other than a line, trace, or other medium between the first and second components. The term "coupled" and its variations encompass both direct and indirect couplings. The use of the term "about" is intended to encompass a range of ±10% of the subsequent figure unless otherwise indicated.

[0491] Although a number of embodiments are provided in this disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples should be considered illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0492] In addition, the techniques, systems, subsystems, and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items illustrated or discussed as coupled may be directly connected, or may be indirectly coupled or communicated through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and modifications may be determined by those skilled in the art and may be made without departing from the spirit and scope of the present disclosure.

Claims

1. A method for processing video data, comprising: determining to apply a neural network (NN) model to the visual media data, wherein the NN model is configured to receive as input a slice type indicator (STI) corresponding to a slice type; and Conversion between the visual media data and a bitstream is performed based on the NN model.

2. The method according to claim 1, further comprising: The STI is derived from the stripe type.

3. The method according to any one of claims 1 to 2, further comprising: When the slice type is an intra-frame predicted I slice, the STI is set to a, when the slice type is a bidirectional inter-frame predicted B slice, the STI is set to b, and when the slice type is a unidirectional inter-frame predicted P slice, the STI is set to c, where a, b and c are all constants.

4. The method according to claim 3, wherein: b equals c.

5. The method according to claim 3, wherein b equals -a, and c equals -a.

6. The method according to claim 3, wherein: a equals 1, b equals -1, and c equals -1.

7. The method according to claim 3, wherein: a equals 1, b equals 0.5, and c equals 0.

5.

8. The method according to any one of claims 1 to 7, further comprising: Tiling or expanding the STI into a two-dimensional (2D) array having the same size as the video unit of the visual media data, and After the tiling or the expansion, the STI is considered as an additional input.

9. The method according to claim 8, wherein The video unit includes a codec tree unit (CTU) or a codec tree block (CTB).

10. The method according to any one of claims 1 to 9, wherein The STI is included in a sequence parameter set SPS, a picture parameter set PPS, a picture header, a slice header, a CTU, a coding unit CU, or a region level of a bitstream generated by an encoder or received by a decoder.

11. The method according to any one of claims 1 to 10, wherein The STI includes f(STI), where f is an arbitrary function.

12. The method according to any one of claims 1 to 11, wherein The STI is applied to one or more color components.

13. The method according to any one of claims 1 to 12, wherein The STI includes a first STI and a second STI, wherein the first STI is used for a first color component and the second STI is used for a second color component.

14. The method according to claim 13, wherein The first color component comprises a luminance color component, and wherein the second color component comprises a chrominance color component, such as a blue difference color component Cb or a red difference color component Cr.

15. The method according to any one of claims 13-14, wherein The first STI indicates a quality level of the first color component.

16. The method according to any one of claims 13-14, wherein The second STI indicates a quality level of the second color component.

17. The method according to any one of claims 1 to 11, further comprising: The STI is used for NN filtering of a first color component but not for NN filtering of a second color component, wherein the first color component includes a luminance color component and wherein the second color component includes a chrominance color component, such as a blue difference color component Cb or a red difference color component Cr.

18. The method according to any one of claims 1 to 11, further comprising: The STI is used for NN filtering of all color components.

19. The method according to any one of claims 1 to 18, further comprising: A specific operation is performed when a picture is divided into a plurality of slices, wherein the specific operation includes the NN model using a slice type of a single slice among the plurality of slices for the entire picture.

20. The method according to claim 19, wherein The single slice is the first slice of the picture.

21. The method according to any one of claims 1 to 18, further comprising: A specific operation is performed when a picture is divided into a plurality of slices, wherein the specific operation includes the NN model using a slice type of a slice among the plurality of slices for samples in the slice.

22. The method according to any one of claims 1 to 21, wherein The NN model is configured to receive a codec mode indicator CMI corresponding to a codec mode as input.

23. The method according to claim 22, further comprising: The CMI is derived from the codec mode.

24. The method according to any one of claims 22-23, wherein The CMI indicates an intra prediction mode or an inter prediction mode.

25. The method according to any one of claims 22-23, wherein The CMI indicates an intra block copy mode or a palette mode.

26. The method according to any one of claims 22 to 25, wherein: The CMI comprises a two-dimensional 2D array having the same resolution as a video unit of the visual media data to be filtered, and wherein each sample in the 2D array is represented by a value reflecting a codec mode associated with a codec unit CU to which the sample belongs.

27. The method according to claim 26, wherein i and j are vertical and horizontal indices within the 2D array, wherein intra_pred_mode is the intra prediction mode associated with the coding unit CU to which the sample (i, j) belongs, wherein inter_pred_mode is the inter prediction mode associated with the CU to which the sample (i, j) belongs, and wherein the CMI(i, j) is derived as follows: When the sample point (i, j) belongs to an intra-coded CU, setting the CMI (i, j) to the intra_pred_mode; and When the sample point (i, j) belongs to an inter-coded CU, the CMI (i, j) is set to the inter_pred_mode.

28. The method according to claim 27, wherein The intra prediction mode includes a skip mode, a merge mode or an advanced motion vector prediction AMVP mode.

29. The method according to any one of claims 22 to 28, wherein The CMI is included in a sequence parameter set SPS, a picture parameter set PPS, a picture header, a slice header, a CTU, a coding unit CU, or a region level of a bitstream generated by an encoder or received by a decoder.

30. The method according to any one of claims 22 to 29, wherein The CMI comprises f(CMI), where f is an arbitrary function.

31. The method according to any one of claims 22 to 30, wherein: The CMI is applied to one or more color components.

32. The method according to any one of claims 22 to 31, wherein The CMI includes a first CMI and a second CMI, wherein the first CMI is used for a first color component, and wherein the second CMI is used for a second color component.

33. The method according to claim 32, wherein The first color component comprises a luminance color component, and wherein the second color component comprises a chrominance color component, such as a blue difference color component Cb or a red difference color component Cr.

34. The method according to any one of claims 32-33, wherein The first CMI indicates a quality level of the first color component.

35. The method according to any one of claims 32-33, wherein The second CMI indicates a quality level of the second color component.

36. The method according to any one of claims 22 to 35, further comprising: The CMI is used for NN filtering of a first color component but not for NN filtering of a second color component, wherein the first color component comprises a luma color component and wherein the second color component comprises a chroma color component, such as a blue difference color component Cb or a red difference color component Cr.

37. The method according to any one of claims 22 to 35, further comprising: The CMI is used for NN filtering of all color components.

38. The method according to any one of claims 1 to 37, wherein The NN model is configured to receive as input the encoded information, the reconstructed information, the information derived from the encoded information, or some combination thereof.

39. The method according to claim 38, wherein The coded information includes a quantization parameter QP, or the coded information is derived based on the QP.

40. The method of claim 38, wherein The coded information includes a prediction direction, or the coded information is derived based on the prediction direction.

41. The method of claim 38, wherein The coded information includes a reference picture index, or the coded information is derived based on the reference picture index.

42. The method of claim 38, wherein The coded information includes a picture order count (POC), or the coded information is derived based on the POC.

43. The method of claim 38, wherein The coded information includes the number of reference pictures, or the coded information is derived based on the number of reference pictures.

44. The method of claim 38, wherein The coded information includes a time domain layer identifier ID, or the coded information is derived based on the time domain layer ID.

45. The method of claim 38, wherein The coded information includes a motion vector, or the coded information is derived based on the motion vector.

46. ​​The method of claim 38, wherein The coded information includes a motion vector difference, or the coded information is derived based on the motion vector difference.

47. The method of claim 38, wherein The coded information includes a transform type, or the coded information is derived based on the transform type.

48. The method of claim 38, wherein The coded information includes residual information, or the coded information is derived based on the residual information.

49. The method of claim 38, wherein The coded information includes a coded block flag CBF, or the coded information is derived based on the CBF.

50. The method of claim 38, wherein The coded information is derived based on whether other loop filters have been enabled, and wherein the other loop filters include a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a cross-component ALF, or a bilateral filter.

51. The method of claim 38, wherein The coded information is derived based on whether the sample is modified by other loop filters, and wherein the other loop filters include a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a cross-component ALF, or a bilateral filter.

52. The method according to any one of claims 38 to 51, wherein Two or more types of the encoded and decoded information are input into the NN model together.

53. The method according to any one of claims 38 to 52, wherein: The STI, the CMI, and the encoded and decoded information are input into the NN model together.

54. The method according to any one of claims 38 to 52, wherein: Only one of the STI, the CMI, and the encoded and decoded information is used by the NN model.

55. The method according to any one of claims 38 to 54, wherein A syntax element is included in the bitstream to indicate whether the STI, the CMI, the coded information, or some combination thereof is used by the NN model.

56. The method according to any one of claims 38 to 54, wherein Syntax elements are included in the bitstream to indicate how the STI, the CMI, the coded information, or some combination thereof, are used by the NN model.

57. The method according to any one of claims 55-56, wherein The syntax element is binarized as a flag, a fixed length code, an Exponential-Golomb code, a unary code, or a truncated unary code, and wherein the syntax element is signed or unsigned.

58. The method according to any one of claims 55-56, wherein The syntax elements are encoded or decoded using at least one context model or are bypass encoded or decoded.

59. The method according to any one of claims 55-56, wherein The syntax elements are encoded and decoded in a conditional manner.

60. The method according to any one of claims 55-56, wherein The syntax element is included in the bitstream only if the STI, the CMI, or the coded information corresponding to the syntax element is applicable or enabled.

61. The method according to any one of claims 55-56, wherein The syntax element is included at the block level, sequence level, picture group level, picture level, slice level or slice group level of the bitstream, or is included in the codec tree unit CTU, codec unit CU, transform unit TU, prediction unit PU, codec tree block CTB, codec block CB, transform block TB or prediction block PB of the bitstream.

62. The method according to any one of claims 55-56, wherein The syntax element is included in a sequence header, a picture header, a sequence parameter set SPS, a video parameter set VPS, a dependency parameter set DPS, decoding capability information DCI, a picture parameter set PPS, an adaptation parameter set APS, a slice header or a slice group header of the bitstream.

63. The method according to any one of claims 55-56, wherein Indicate whether and / or how to apply the method at the block level, sequence level, picture group level, picture level, slice level or slice group level of the bitstream, or indicate whether and / or how to apply the method in the codec tree unit CTU, codec unit CU, transform unit TU, prediction unit PU, codec tree block CTB, codec block CB, transform block TB or prediction block PB of the bitstream.

64. The method according to any one of claims 55-56, wherein Whether and / or how to apply the method is indicated in a sequence header, picture header, sequence parameter set SPS, video parameter set VPS, dependency parameter set DPS, decoding capability information DCI, picture parameter set PPS, adaptation parameter set APS, slice header or slice group header of the bitstream.

65. The method according to any one of claims 1 to 64, wherein Whether and / or how the method is applied depends on the block size, color format, single-tree partitioning, dual-tree partitioning, color components, the slice type, picture type or other coded information.

66. The method according to any one of claims 1 to 65, wherein One or more of the described methods are used in another codec that performs chroma blending.

67. A method for processing video data, comprising: determining to apply a neural network (NN) model to the visual media data, wherein the NN model is configured to receive as input a type indicator corresponding to a type, wherein the type comprises a picture type, a frame type, or a block type; and Conversion between the visual media data and a bitstream is performed based on the NN model.

68. The method according to any one of claims 1 to 67, wherein The converting includes encoding the media data into the bitstream.

69. The method according to any one of claims 1 to 67, wherein The converting includes decoding the media data from the bitstream.

70. An apparatus for processing media data, comprising: processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-69.

71. A non-transitory computer-readable medium comprising a computer program product for use with a video codec device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium, such that when the computer-executable instructions are executed by a processor, the video codec device performs the method according to any one of claims 1-69.

72. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing apparatus, wherein: The method comprises the method according to any one of claims 1-69.

73. A method of storing a bitstream of a video, comprising the method according to any one of claims 1-69.

74. A method, apparatus or system as described in the present disclosure.

Images