System and method for signaling neural network post-filter parameter information in video coding

Abstract

A device may be configured to signal a neural network post-filter parameter information according to one or more of the techniques described herein.

Description

Technical Field

[0001] The present disclosure relates to video encoding, and more particularly to systems and methods for signaling neural network post-filter parameter information for encoding video. Background Art

[0002] Digital video functions can be incorporated into various devices, including digital televisions, laptops or desktop computers, tablet computers, digital recording devices, digital media players, video game devices, cellular phones (including so-called smart phones), medical imaging devices, etc. Digital video can be encoded according to video coding standards. Video coding standards define the format of compatible bit streams that encapsulate coded video data. A compatible bit stream is a data structure that can be received and decoded by a video decoding device to generate reconstructed video data. Video coding standards can be combined with video compression technology. Examples of video coding standards include ISO / IEC MPEG-4 Visual and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) and High Efficiency Video Coding (HEVC). HEVC is described in High Efficiency Video Coding (HEVC) of the ITU-T H.265 Recommendation of December 2016, which is incorporated herein by reference and is referred to herein as ITU-T H.265. ITU-TH.265 is being considered for expansion and improvement to develop the next generation of video coding standards. For example, the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG), collectively referred to as the Joint Video Study Group (JVET), have standardized video coding techniques with compression capabilities significantly exceeding the current HEVC standard. The Joint Exploration Model 7 (JEM 7), the algorithm description of the Joint Exploration Test Model 7 (JEM 7), ISO / IEC JTC1 / SC29 / WG11 document: JVET-G1001 (July 2017, Turin, Italy), incorporated herein by reference, describes the coding features studied by JVET under the Joint Test Model, which is a potential enhanced video coding technology that exceeds the capabilities of ITU-T H.265. It should be noted that the coding features of JEM 7 are implemented in the JEM reference software. As used herein, the term JEM may collectively refer to the algorithms included in JEM 7 and the specific implementation of the JEM reference software. In addition, in response to the "Joint Call for Proposals on Video Compression with Capabilities beyond HEVC" jointly issued by VCEG and MPEG, at the 10th meeting of ISO / IEC JTC1 / SC29 / WG11 held in San Diego, California (San Diego, CA) from April 16 to 20, 2018, various groups proposed multiple descriptions of video coding tools.Based on the various descriptions of the video coding tools, the final initial draft text of the video coding specification is described in "Versatile Video Coding (Draft 1)" at the 10th ISO / IEC JTC1 / SC29 / WG11 meeting held from April 16 to 20, 2018 in San Diego, California, document JVET-J1001-v2, which is incorporated herein by reference and is referred to as JVET-J1001. The development of the video coding standard by VCEG and MPEG is known as the Versatile Video Coding (VVC) project. "Versatile Video Coding (Draft 10)" at the 20th ISO / IEC JTC1 / SC29 / WG11 meeting held from October 7 to 16, 2020 (conference call, document JVET-T2001-v2, which is incorporated herein by reference and is referred to as JVET-T2001) represents the current iteration of the draft text of the video coding specification corresponding to the VVC project.

[0003] Video compression technology can reduce the data requirements for storing and transmitting video data. Video compression technology can reduce data requirements by exploiting the redundancy inherent in video sequences. Video compression technology can divide the video sequence into smaller and continuous parts (i.e., a group of pictures within a video sequence, pictures within a group of pictures, regions within pictures, sub-regions within regions, etc.). Intra-frame prediction coding techniques (e.g., spatial prediction techniques within pictures) and inter-frame prediction techniques (i.e., techniques (time) between pictures) can be used to generate the difference between the unit video data to be encoded and the reference unit of the video data. The difference can be called residual data. The residual data can be encoded as quantized transform coefficients. Syntax elements can involve residual data and reference coding units (e.g., intra-frame prediction mode indexes and motion information). The residual data and syntax elements can be entropy encoded. Entropy-encoded residual data and syntax elements can be included in the data structure that forms a compatible bitstream. Summary of the invention

[0004] In one example, a method for applying a neural network filter to reconstruct a video includes: receiving a neural network post filter characteristics message specifying a neural network; parsing a first syntax element identifying a neural network post filter (NNPF); parsing a second syntax element specifying that the NNPF associated with the first syntax element is a neural network identified by a uniform resource identifier; parsing a third syntax element containing the uniform resource identifier; and parsing a fourth syntax element identifying a format of the neural network data specified by the third syntax element.

[0005] In one example, a device includes one or more processors configured to: receive a neural network post filter characteristics message specifying a neural network; parse a first syntax element identifying a neural network post filter (NNPF); parse a second syntax element specifying that the NNPF associated with the first syntax element is a neural network identified by a uniform resource identifier; parse a third syntax element including the uniform resource identifier; and parse a fourth syntax element identifying a format of the neural network data specified by the third syntax element. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram illustrating an example of a system that may be configured to encode and decode video data in accordance with one or more techniques of this disclosure. Figure 2 is a conceptual diagram illustrating encoded video data and corresponding data structures in accordance with one or more techniques of this disclosure. Figure 3 is a conceptual diagram illustrating a data structure that encapsulates encoded video data and corresponding metadata, in accordance with one or more techniques of this disclosure. Figure 4 is a conceptual diagram illustrating an example of components that may be included in an implementation of a system that may be configured to encode and decode video data in accordance with one or more techniques of this disclosure. Figure 5 is a block diagram illustrating an example of a video encoder that may be configured to encode video data in accordance with one or more techniques of this disclosure. Figure 6 is a block diagram illustrating an example of a video decoder that may be configured to decode video data in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0007] In general, the present disclosure describes various techniques for encoding video data. Specifically, the present disclosure describes techniques for signaling neural network post-filter parameter information for encoding video data. It should be noted that although the techniques of the present disclosure are described with respect to ITU-TH.264, ITU-TH.265, JEM, and JVET-T2001, the techniques of the present disclosure are generally applicable to video encoding. For example, in addition to those techniques included in ITU-TH.265, JEM, and JVET-T2001, the encoding techniques described herein may be incorporated into video encoding systems (including video encoding systems based on future video encoding standards), including video block structures, intra-frame prediction techniques, inter-frame prediction techniques, transform techniques, filtering techniques, and / or other entropy coding techniques. Therefore, references to ITU-TH.264, ITU-TH.265, JEM, and / or JVET-T2001 are for descriptive purposes and should not be interpreted as limiting the scope of the techniques described herein. In addition, it should be noted that the incorporation of references herein is for descriptive purposes and should not be construed as limiting or creating ambiguity regarding the terms used herein. For example, where a definition of a term provided in an incorporated reference differs from that provided in another incorporated reference and / or as used herein, the term should be interpreted in a manner that broadly includes each corresponding definition and / or in a manner that includes each specific definition in alternatives.

[0008] In one example, a method of encoding video data includes signaling a neural network post filter characteristics message, and signaling syntax elements identifying a format and associated information of a neural network corresponding to the neural network post filter characteristics message.

[0009] In one example, a device includes one or more processors configured to signal a neural network post filter characteristics message and signal syntax elements identifying a format and associated information of a neural network corresponding to the neural network post filter characteristics message.

[0010] In one example, a non-transitory computer-readable storage medium includes instructions stored thereon that, when executed, cause one or more processors of a device to: signal a neural network post-filter characteristics message, and signal a syntax element identifying a format of a neural network and associated information corresponding to the neural network post-filter characteristics message.

[0011] In one example, an apparatus includes means for signaling a neural network post-filter characteristics message, and means for signaling a syntax element identifying a format and associated information of a neural network corresponding to the neural network post-filter characteristics message.

[0012] In one example, a method for decoding video data includes receiving a neural network post filter characteristics message, parsing syntax elements identifying a format and associated information of a neural network corresponding to the neural network post filter characteristics message, and determining one or more neural network parameters based on the parsed syntax elements.

[0013] In one example, a device includes one or more processors configured to receive a neural network post filter characteristics message, parse syntax elements identifying a format and associated information of a neural network corresponding to the neural network post filter characteristics message, and determine one or more neural network parameters based on the parsed syntax elements.

[0014] In one example, a non-transitory computer-readable storage medium includes instructions stored thereon that, when executed, cause one or more processors of a device to: receive an adaptive parameter set syntax structure, parse one or more syntax elements providing neural network in-loop filter information from the adaptive parameter set syntax structure, determine one or more neural network in-loop filter parameters based on the parsed syntax elements, and apply a neural network in-loop filter based on the determined neural network in-loop filter parameters.

[0015] In one example, an apparatus includes means for receiving a neural network post filter characteristics message, means for parsing syntax elements identifying a format and associated information of a neural network corresponding to the neural network post filter characteristics message, and means for determining one or more neural network parameters based on the parsed syntax elements.

[0016] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

[0017] The video content includes a video sequence consisting of a series of frames (or pictures). A series of frames may also be referred to as a group of pictures (GOP). Each video frame or picture may be divided into one or more regions. The region may be defined according to a set of rules for defining a basic unit (e.g., a video block) and a defined region. For example, the rule defining the region may be that the region must be an integer number of video blocks arranged in a rectangle. In addition, the video blocks in the region may be sorted according to a scanning mode (e.g., a raster scan). As used herein, the term "video block" may generally refer to a region of a picture, or may more specifically refer to a maximum array of sample values ​​that may be predictively encoded, its sub-partitions, and / or corresponding structures. In addition, the term "current video block" may refer to a region of a picture that is being encoded or decoded. A video block may be defined as an array of sample values. It should be noted that in some cases, a pixel value may be described as a sample value comprising a corresponding component of video data, which may also be referred to as a color component (e.g., brightness (Y) and chrominance (Cb and Cr) components or red, green, and blue components). It should be noted that in some cases, the terms "pixel value" and "sample value" may be used interchangeably. Additionally, in some cases, a pixel or sample may be referred to as a pel. A video sampling format (which may also be referred to as a chroma format) may define the number of chroma samples included in a video block relative to the number of luma samples included in the video block. For example, for a 4:2:0 sampling format, the sampling rate of the luma component is twice the sampling rate of the chroma components in both the horizontal and vertical directions.

[0018] The video encoder may perform predictive coding on a video block and its sub-partitions. A video block and its sub-partitions may be referred to as a node. ITU-T H.264 specifies a macroblock including 16×16 luminance samples. That is, in ITU-T H.264, a picture is segmented into macroblocks. ITU-T H.265 specifies a similar coding tree unit (CTU) structure (which may be referred to as a maximum coding unit (LCU)). In ITU-TH.265, a picture is segmented into CTUs. In ITU-T H.265, for a picture, the CTU size may be set to include 16×16, 32×32, or 64×64 luminance samples. In ITU-T H.265, a CTU is composed of a corresponding coding tree block (CTB) for each component of the video data (e.g., luminance (Y) and chrominance (Cb and Cr)). It should be noted that a video having a luminance component and two corresponding chrominance components may be described as having two channels, i.e., a luminance channel and a chrominance channel. In addition, in ITU-T H.265, CTUs can be divided according to a quadtree (QT) partitioning structure, which allows the CTBs of a CTU to be divided into coding blocks (CBs). That is, in ITU-T H.265, CTUs can be divided into quadtree leaf nodes. According to ITU-T H.265, a luma CB together with two corresponding chroma CBs and associated syntax elements is referred to as a coding unit (CU). In ITU-TH.265, a signal can be sent to notify the minimum allowed size of a CB. In ITU-T H.265, the minimum allowed minimum size of a luma CB is 8×8 luma samples. In ITU-T H.265, the decision to encode a picture area using intra prediction or inter prediction is made at the CU level.

[0019] In ITU-T H.265, a CU is associated with a prediction unit structure having its root at the CU. In ITU-T H.265, the prediction unit structure allows the partitioning of luma CB and chroma CB to generate corresponding reference samples. That is, in ITU-T H.265, the luma CB and chroma CB can be partitioned into corresponding luma prediction blocks and chroma prediction blocks (PBs), wherein the PBs include blocks of sample values ​​to which the same prediction is applied. In ITU-T H.265, the CB can be divided into 1, 2, or 4 PBs. ITU-T H.265 supports PB sizes from 64×64 samples down to 4×4 samples. In ITU-TH.265, square PBs are supported for intra prediction, where the CB can form a PB or the CB can be partitioned into four square PBs. In ITU-TH.265, in addition to square PBs, rectangular PBs are also supported for inter prediction, where the CB can be halved vertically or horizontally to form a PB. In addition, it should be noted that in ITU-T H.265, for inter prediction, four asymmetric PB partitions are supported, where a CB is divided into two PBs at one quarter of the height (top or bottom) or width (left or right) of the CB. Intra prediction data (e.g., intra prediction mode syntax elements) or inter prediction data (e.g., motion data syntax elements) corresponding to the PB are used to generate reference and / or prediction sample values ​​for the PB.

[0020] JEM specifies a CTU with a maximum size of 256×256 luma samples. JEM specifies a quadtree plus binary tree (QTBT) block structure. In JEM, the QTBT structure allows the quadtree leaf nodes to be further divided by the binary tree (BT) structure. That is, in JEM, the binary tree structure allows the quadtree leaf nodes to be recursively divided vertically or horizontally. In JVET-T2001, CTUs are divided according to a quadtree plus multi-type tree (QTMT or QT+MTT) structure. The QTMT in JVET-T2001 is similar to the QTBT in JEM. However, in JVET-T2001, in addition to indicating binary partitioning, the multi-type tree can also indicate a so-called ternary (or ternary tree (TT)) partitioning. Ternary partitioning divides a block vertically or horizontally into three blocks. In case of vertical TT splitting, the block is split at one quarter of its width from the left edge and at one quarter of its width from the right edge, and in case of horizontal TT splitting, the block is split at one quarter of its height from the top edge and at one quarter of its height from the bottom edge.

[0021] As described above, each video frame or picture may be divided into one or more regions. For example, according to ITU-T H.265, each video frame or picture may be divided into one or more slices, and further divided into one or more tiles, wherein each slice includes a CTU sequence (e.g., arranged in a raster scan order), and wherein the tile is a CTU sequence corresponding to a rectangular area of ​​the picture. It should be noted that in ITU-T H.265, a slice is a sequence of one or more slice segments starting from an independent slice segment and containing all subsequent subordinate slice segments (if any) before the next independent slice segment (if any). Slice segments (such as slices) are CTU sequences. Therefore, in some cases, the terms "slice" and "slice segment" may be used interchangeably to indicate a sequence of CTUs arranged in a raster scan order. In addition, it should be noted that in ITU-T H.265, a tile may be composed of CTUs contained in more than one slice, and a slice may be composed of CTUs contained in more than one tile. However, ITU-T H.265 specifies that one or both of the following conditions should be met: (1) all CTUs in a slice belong to the same tile; and (2) all CTUs in a tile belong to the same slice.

[0022] Regarding JVET-T2001, a slice needs to be composed of an integer number of complete tiles or an integer number of continuous complete CTU rows within a tile, rather than only an integer number of CTUs. It should be noted that in JVET-T2001, the slice design does not include a slice segment (ie, there is no independent / subordinate slice segment). Therefore, in JVET-T2001, a picture may include a single tile, wherein the single tile is contained in a single slice, or a picture may include multiple tiles, wherein the multiple tiles (or their CTU rows) may be contained in one or more slices. In JVET-T2001, the picture is divided into tiles by specifying the corresponding height of the tile row and the corresponding width of the tile column. Therefore, in JVET-T2001, a tile is a rectangular CTU area within a specific tile row and a specific tile column position. In addition, it should be noted that JVET-T2001 stipulates that a picture can be divided into sub-pictures, wherein the sub-picture is a rectangular CTU area within the picture. The upper left CTU of a sub-picture may be located at any CTU position within a picture, where the sub-picture is constrained to include one or more slices. Therefore, unlike tiles, sub-pictures do not have to be restricted to specific row and column positions. It should be noted that sub-pictures can be used to encapsulate regions of interest within a picture, and the sub-bitstream extraction process can be used to decode and display only specific regions of interest. That is, as described in further detail below, the bitstream of encoded video data includes a sequence of network abstraction layer (NAL) units, wherein the NAL unit encapsulates encoded video data (i.e., video data corresponding to a picture slice), or the NAL unit encapsulates metadata (e.g., parameter sets) for decoding the video data, and the sub-bitstream extraction process forms a new bitstream by removing one or more NAL units from the bitstream.

[0023] Figure 2 is a conceptual diagram showing examples of pictures within picture groups partitioned according to tiles, slices, and sub-pictures. It should be noted that the techniques described herein can be applicable to tiles, slices, sub-pictures, their sub-partitions, and / or their equivalent structures. That is, regardless of how the pictures are divided into regions, the techniques described herein are generally applicable. For example, in some cases, the techniques described herein can be applicable to cases where tiles can be partitioned into so-called bricks, where a brick is a rectangular CTU row region within a particular tile. In addition, for example, in some cases, the techniques described herein can be applicable to cases where one or more tiles can be included in a so-called tile group, where a tile group includes an integer number of adjacent tiles. Figure 2 In the example shown, Pic3 is shown to include 16 tiles (ie, tiles 0 to 15 ) and three slices (i.e., slice 0 to slice 2). Figure 2In the example shown, slice 0 includes four tiles (ie, tiles 0 to 3), and slice 1 includes eight tiles (ie, tiles 4 to 5). 11 ), and slice 2 includes four tiles (i.e., tiles 12 To Tile 15 ). In addition, Figure 2 As shown in the example of , Pic3 is shown to include two sub-pictures (i.e., sub-picture 0 and sub-picture 1), where sub-picture 0 includes slices 0 and slice 1 and where sub-picture 1 includes slice 2. As described above, sub-pictures can be used to encapsulate regions of interest within a picture, and a sub-bitstream extraction process can be used to selectively decode (and display) the regions of interest. For example, referring to Figure 2 , sub-picture 0 may correspond to the action portion of a sporting event presentation (e.g., a view of a field), and sub-picture 1 may correspond to a scrolling banner displayed during the sporting event presentation. By organizing pictures into sub-pictures in this manner, a viewer may be able to disable the display of a scrolling banner. That is, through the sub-bitstream extraction process, slice 2 NAL units may be removed from the bitstream (and therefore not decoded and / or displayed), and slice 0 NAL units and slice 1 NAL units may be decoded and displayed. How slices of a picture are encapsulated into corresponding NAL unit data structures and sub-bitstream extraction is described in further detail below.

[0024] For intra-frame prediction coding, the intra-frame prediction mode may specify the location of the reference sample within the picture. In ITU-TH.265, the possible intra-frame prediction modes defined include a plane (ie, surface fitting) prediction mode, a DC (ie, flat overall average) prediction mode, and 33 angular prediction modes (predMode:2-34). In JEM, the possible intra-frame prediction modes defined include a plane prediction mode, a DC prediction mode, and 65 angular prediction modes. It should be noted that the plane prediction mode and the DC prediction mode may be referred to as a non-directional prediction mode, and the angular prediction mode may be referred to as a directional prediction mode. It should be noted that regardless of the number of possible prediction modes defined, the techniques described herein may be generally applicable.

[0025] For inter-frame prediction coding, a reference picture is determined, and a motion vector (MV) identifies samples in the reference picture that are used to generate a prediction for the current video block. For example, the current video block may be predicted using reference sample values ​​located in one or more previously coded pictures, and a motion vector is used to indicate the position of the reference block relative to the current video block. The motion vector may describe, for example, the horizontal displacement component of the motion vector (i.e., the MV x ), the vertical displacement component of the motion vector (i.e., MV y) and the resolution of the motion vector (e.g., quarter pixel accuracy, half pixel accuracy, one pixel accuracy, two pixel accuracy, four pixel accuracy). Previously decoded pictures (which may include pictures output before or after the current picture) may be organized into one or more reference picture lists and identified using reference picture index values. In addition, in inter-frame prediction coding, uni-prediction refers to generating predictions using sample values ​​from a single reference picture, and bi-prediction refers to generating predictions using corresponding sample values ​​from two reference pictures. That is, in uni-prediction, a single reference picture and a corresponding motion vector are used to generate a prediction for the current video block, while in bi-prediction, a first reference picture and a corresponding first motion vector and a second reference picture and a corresponding second motion vector are used to generate a prediction for the current video block. In bi-prediction, corresponding sample values ​​are combined (e.g., added, rounded and cropped, or averaged according to weights) to generate a prediction. Pictures and their regions may be classified based on what types of prediction modes may be used to encode their video blocks. That is, for a region with a B type (e.g., a B slice), bi-prediction, uni-prediction, and intra-prediction modes can be utilized, for a region with a P type (e.g., a P slice), uni-prediction and intra-prediction modes can be utilized, and for a region with an I type (e.g., an I slice), only an intra-prediction mode can be utilized. As described above, a reference picture is identified by a reference index. For example, for a P slice, there may be a single reference picture list RefPicList0, and for a B slice, there may be a second independent reference picture list RefPicList1 in addition to RefPicList0. It should be noted that for a uni-prediction in a B slice, one of RefPicList0 or RefPicList1 may be used to generate a prediction. Furthermore, it should be noted that during the decoding process, when decoding of a picture begins, a reference picture list is generated from a previously decoded picture stored in a decoded picture buffer (DPB).

[0026] In addition, the coding standard may support various motion vector prediction modes. Motion vector prediction enables the value of the motion vector for the current video block to be derived based on another motion vector. For example, a set of candidate blocks with associated motion information can be derived from the spatial neighboring blocks and the temporal neighboring blocks of the current video block. In addition, the generated (or default) motion information can be used for motion vector prediction. Examples of motion vector prediction include advanced motion vector prediction (AMVP), temporal motion vector prediction (TMVP), so-called "merge" mode, and "skip" and "direct" motion reasoning. In addition, other examples of motion vector prediction include advanced temporal motion vector prediction (ATMVP) and space-time motion vector prediction (STMVP). For motion vector prediction, both the video encoder and the video decoder perform the same process to derive a set of candidates. Therefore, for the current video block, the same set of candidates is generated during encoding and decoding.

[0027] As described above, for inter-frame prediction coding, reference samples in previously encoded pictures are used to encode video blocks in the current picture. Previously encoded pictures that can be used as references when encoding the current picture are called reference pictures. It should be noted that the decoding order does not necessarily correspond to the picture output order, that is, the temporal order of pictures in a video sequence. In ITU-TH.265, when a picture is decoded, it is stored in a decoded picture buffer (DPB) (which may be called a frame buffer, a reference buffer, a reference picture buffer, etc.). In ITU-TH.265, pictures stored in the DPB are removed from the DPB when they are output and are no longer needed for encoding subsequent pictures. In ITU-TH.265, the determination of whether the picture should be removed from the DPB is called once for each picture after decoding the slice header, that is, at the beginning of decoding the picture. For example, the reference Figure 2 , Pic2 is shown as reference to Pic1. Similarly, Pic3 is shown as reference to Pic0. Figure 2, assuming that the number of pictures corresponds to the decoding order, the DPB will be filled as follows: after decoding Pic0, the DPB will include {Pic0}; at the beginning of decoding Pic1, the DPB will include {Pic0}; after decoding Pic1, the DPB will include {Pic0, Pic1}; at the beginning of decoding Pic2, the DPB will include {Pic0, Pic1}. Pic2 will then be decoded with reference to Pic1, and after decoding Pic2, the DPB will include {Pic0, Pic1, Pic2}. At the beginning of decoding Pic3, pictures Pic0 and Pic1 will be marked for removal from the DPB because they are not required for decoding Pic3 (or any subsequent pictures, not shown), and assuming that Pic1 and Pic2 have been output, the DPB will be updated to include {Pic0}. Pic3 will then be decoded with reference to Pic0. The process of marking pictures to remove them from the DPB can be called reference picture set (RPS) management.

[0028] As described above, intra prediction data or inter prediction data is used to generate reference sample values ​​for a block of sample values. The difference between the sample values ​​included in the current PB or another type of picture region structure and the associated reference samples (e.g., those generated using prediction) may be referred to as residual data. The residual data may include a corresponding difference array corresponding to each component of the video data. The residual data may be in the pixel domain. A transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), an integer transform, a wavelet transform, or a conceptually similar transform may be applied to the difference array to generate transform coefficients. It should be noted that in ITU-TH.265 and JVET-T2001, a CU is associated with a transform tree structure having its root at the CU level. The transform tree is divided into one or more transform units (TUs). That is, in order to generate transform coefficients, the array of difference values ​​may be divided (e.g., four 8×8 transforms may be applied to a 16×16 residual value array). For each component of the video data, such a subdivision of the difference value may be referred to as a transform block (TB). It should be noted that in some cases a core transform and a subsequent secondary transform may be applied (in a video encoder) to generate transform coefficients. For a video decoder, the order of transforms is reversed.

[0029] The quantization process may be performed directly on the transform coefficients or residual sample values ​​(e.g., in the case of palette coded quantization). Quantization approximates the transform coefficients by limiting the amplitude to a set of specified values. Quantization essentially scales the transform coefficients in order to change the amount of data required to represent a set of transform coefficients. Quantization may include dividing the transform coefficients (or the values ​​obtained by adding the offset values ​​to the transform coefficients) by a quantization scaling factor and any associated rounding function (e.g., rounding to the nearest integer). The quantized transform coefficients may be referred to as coefficient level values. Inverse quantization (or "dequantization") may include multiplying the coefficient level values ​​by the quantization scaling factor, as well as any reciprocal rounding or offset addition operations. It should be noted that, as used herein, the term quantization process may refer to dividing by a scaling factor in some cases to generate a level value, and may refer to multiplying by a scaling factor in some cases to recover the transform coefficients. That is, the quantization process may refer to quantization in some cases, and to inverse quantization in some cases. In addition, it should be noted that although the quantization process is described in some of the examples below with respect to arithmetic operations associated with decimal notation, such description is for illustrative purposes and should not be construed as limiting. For example, the techniques described herein can be implemented in devices that use binary operations, etc. For example, the multiplication and division operations described herein can be implemented using bit shift operations, etc.

[0030] The quantized transform coefficients and syntax elements (e.g., syntax elements indicating the coding structure of the video block) may be entropy encoded according to an entropy coding technique. The entropy coding process includes encoding the syntax element values ​​using a lossless data compression algorithm. Examples of entropy coding techniques include content adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), probability interval partitioning entropy coding (PIPE), etc. The entropy-coded quantized transform coefficients and the corresponding entropy-coded syntax elements may form a compatible bitstream that can be used to reproduce the video data at the video decoder. The entropy coding process, such as CABAC, may include binarizing the syntax elements. Binarization refers to the process of converting the value of the syntax element into a sequence of one or more bits. These bits may be referred to as "bins". Binarization may include one or a combination of the following coding techniques: fixed length coding, unary coding, truncated unary coding, truncated Rice coding, Golomb coding, k-order exponential Golomb coding, and Golomb-Rice coding. For example, binarization may include representing the integer value 5 of the syntax element as 00000101 using an 8-bit fixed-length binarization technique, or representing the integer value 5 as 11110 using a unary coding binarization technique. As used herein, each of the terms fixed-length coding, unary coding, truncated unary coding, truncated Rice coding, Golomb coding, k-order exponential Golomb coding, and Golomb-Rice coding may refer to general implementations of these techniques and / or more specific implementations of these coding techniques. For example, a Golomb-Rice coding implementation may be specifically defined according to a video coding standard. In the example of CABAC, for a particular bin, the context provides a maximum probability state (MPS) value of the bin (i.e., the MPS of the bin is one of 0 or 1), and a probability value that the bin is an MPS or a minimum probability state (LPS). For example, the context may indicate that the MPS of the bin is 0, and the probability that the bin is 1 is 0.3. It should be noted that the context may be determined based on the value of the previously encoded bin including the bin in the current syntax element and the previously encoded syntax element. For example, the values ​​of syntax elements associated with neighboring video blocks may be used to determine the context of the current bin.

[0031] As described above, the sample values ​​of the reconstructed block may be different from the sample values ​​of the current video block being encoded. In addition, it should be noted that in some cases, encoding video data block by block may result in artifacts (e.g., so-called block artifacts, band artifacts, etc.). For example, block artifacts may cause the encoding block boundaries of the reconstructed video data to be visually perceived by the user. In this way, the reconstructed sample values ​​may be modified to minimize the difference between the sample values ​​of the encoded current video block and the reconstructed block and / or minimize artifacts introduced by the video encoding process. Such modifications may generally be referred to as filtering. It should be noted that filtering may occur as part of an in-loop filtering process or a post-loop (or post-filtering) filtering process. For an in-loop filtering process, the resulting sample values ​​of the filtering process may be used to predict the video block (e.g., stored in a reference frame buffer for subsequent encoding at a video encoder and subsequent decoding at a video decoder). For a post-loop filtering process, the resulting sample values ​​of the filtering process are output only as part of the decoding process (e.g., not used for subsequent encoding). For example, in the case of a video decoder, for an in-loop filtering process, the sample values ​​generated by the filtered reconstruction block will be used for subsequent decoding (e.g., stored in a reference buffer) and will be output (e.g., output to a display). For a post-loop filtering process, the reconstruction block will be used for subsequent decoding, and the sample values ​​generated by the filtered reconstruction block will be output and will be used for subsequent decoding.

[0032] Deblocking (or deblocking), deblocking filtering or applying a deblocking filter refers to the process of smoothing the boundaries of adjacent reconstructed video blocks (i.e., making the boundaries less noticeable to the observer). Smoothing the boundaries of adjacent reconstructed video blocks may include modifying the sample values ​​included in the rows or columns adjacent to the boundaries. JVET-T2001 provides a scenario in which a deblocking filter is applied to reconstructed sample values ​​as part of an in-loop filtering process. In addition to applying a deblocking filter as part of an in-loop filtering process, JVET-T2001 also provides a scenario in which a sample adaptive offset (SAO) filter may be applied in the in-loop filtering process. Generally speaking, SAO is a process in which deblocked sample values ​​in a region are modified by conditionally adding an offset value. Another type of filtering process includes a so-called adaptive loop filter (ALF). ALF using block-based adaptation is specified in JEM. In JEM, ALF is applied after the SAO filter. It should be noted that ALF can be applied to reconstructed samples independently of other filtering techniques. The process of applying the ALF specified in JEM at a video encoder can be summarized as follows: (1) each 2×2 block of the luminance component used to reconstruct the image is classified according to a classification index; (2) a filter coefficient set for each classification index is derived; (3) a filtering decision is determined for the luminance component; (4) a filtering decision is determined for the chrominance components; and (5) filter parameters (e.g., coefficients and decisions) are signaled. JVET-T2001 specifies a deblocking filter, SAO filter, and ALF filter, which can be described as being generally based on the deblocking filter, SAO filter, and ALF filter provided in ITU-T H.265 and JEM.

[0033] It should be noted that JVET-T2001 is referred to as a pre-release version of ITU-TH.266, and therefore, it is an almost finalized draft of the video coding standard produced by the VVC project, and therefore, can be referred to as the first version of the VVC standard (or VVC or VVC version 1 or ITU-H.266). It should be noted that during the VVC project, convolutional neural network (CNN)-based techniques were studied, which showed the potential for removing artifacts and improving objective quality, but it was decided not to include such techniques in the VVC standard. However, CNN-based techniques are currently being considered for extensions and / or improvements to VVC. Some CNN-based techniques involve post-filtering. For example, "AHG11: Content-adaptive neural network post-filter" (teleconference, document JVET-Z0082-v2, referred to as JVET-Z0082 in this article) at the 26th meeting of ISO / IEC JTC1 / SC29 / WG11 held from April 20 to 29, 2022 describes a post-filter based on a content adaptive neural network. It should be noted that in JVET-Z0082, content adaptation is achieved by overfitting the NN post-filter to the test video. Furthermore, it should be noted that the result of the overfitting process in JVET-Z0082 is a weight update. JVET-Z0082 describes where the weight updates are encoded using ISO / IEC FDIS15938-17. Information Technology - Multimedia Content Description Interface - Part 17: Compression of Neural Networks for Multimedia Content Description and Analysis and Test Model for Incremental Compression of Neural Networks for Multimedia Content Description and Analysis (INCTM), N0179, February 2022, which may be collectively referred to as the MPEG NNR (Neural Network Representation) or Neural Network Coding (NNC) standard. JVET-Z0082 also describes where the coding weight updates are signaled within the video bitstream as NNR post-filter SEI messages. The 26th ISO / IEC JTC1 / SC29 / WG11 meeting “AHG9: NNR post-filter SEI message” (conference call, document JVET-Z0052-v1, referred to as JVET-Z0052 in this article), held from April 20 to 29, 2022, described the NNR post-filter SEI message utilized by JVET-Z0082.The elements of the NN post-filter described in JVET-Z0082 and the NNR post-filter SEI messages described in JVET-Z0052 are adopted in "Additional SEI messages for VSEI (Draft 1)" of the 26th ISO / IEC JTC1 / SC29 / WG11 meeting held from April 20 to 29, 2022 (conference call, document JVET-Z2006-v1, referred to herein as JVET-Z2006). JVET-Z2006 provides generic supplemental enhancement information messages for coded video bitstreams (VSEI). JVET-Z2006 also adopts elements of the NN post-filter supplemental enhancement information message described in "AHG9: NNR post-filter SEI" of the 26th ISO / IEC JTC1 / SC29 / WG11 meeting held from April 20 to 29, 2022 (conference call, document JVET-Z0244-v1, referred to herein as JVET-Z0244). JVET-Z2006 is described in more detail below. The techniques described herein provide techniques for signaling neural network parameters including, for example, NN post-filter parameters. For the formulas used in this article, the following arithmetic operators can be used: +Addition - Subtraction * Multiplication, including matrix multiplication x y Exponentiation. Raises x to the power of y. In other contexts, such symbols are used for superscripts and are not intended to be interpreted as exponentiation. IInteger division with truncation toward zero. For example, 7 / 4 and -7 / -4 are truncated to 1, and -7 / 4 and 7 / -4 are truncated to -1. ÷ is used to represent division in mathematical formulas when truncation or rounding is not intended. Used to represent division in mathematical formulas without the intent to truncate or round.

[0034] Additionally, the following math functions are available: Log2(x) is the logarithm of x with base 2; Ceil(x), the smallest integer greater than or equal to x.

[0035] With respect to the exemplary syntax used herein, the following definitions of logical operators may be applied: x&&y Boolean logical "AND" of x and y x||y Boolean logical OR of x and y ! Boolean logic "NO" x? y:z If x is TRUE or not equal to 0, it evaluates to y; otherwise, it evaluates to z.

[0036] Additionally, the following relational operators can be applied: > greater than ≥ greater than or equal to < ≤Less than or equal to == is equal to ! = not equal to

[0037] Furthermore, it should be noted that among the syntax descriptors used in this document, the following descriptors may be applied: -b(8): Byte (8 bits) with any bit string pattern. The parsing of this descriptor is specified by the return value of the function read_bit(8). -f(n): A fixed-pattern bit string written using n bits (from left to right) starting from the leftmost bit. The parsing process of this descriptor is specified by the return value of the function read_bits(n). -se(v): signed integer order 0 Exp-Golomb coded syntax element, starting from the leftmost bit. -tb(v): truncated binary code using at most maxVal bits, where maxVal is defined in the semantics of the syntax element. -tu(v): truncated unary code using at most maxVal bits, where maxVal is defined in the semantics of the syntax element. -u(n): Use n-bit unsigned integers. When n is "v" in the syntax table, the number of bits varies in a way that depends on the values ​​of other syntax elements. The parsing process of this descriptor is specified by the return value of the function read_bits(n), which is interpreted as the binary representation of the unsigned integer with the most significant bit written first. -ue(v): unsigned integer 0th order Exp-Golomb coded syntax element, starting from the leftmost bit.

[0038] As mentioned above, the video content includes a video sequence composed of a series of pictures, and each picture can be divided into one or more regions. In JVET-T2001, the coded representation of a picture includes a VCL NAL unit of a specific layer within an AU and contains all CTUs of the picture. For example, referring again to Figure 2, the coded representation of Pic3 is encapsulated in three coded slice NAL units (i.e., slice 0 NAL unit, slice 1 NAL unit, and slice 2 NAL unit). It should be noted that the term video coding layer (VCL) NAL unit is used as a general term for coded slice NAL units, that is, VCL NAL is a general term for slice NAL units of all types. As described above, and described in further detail below, NAL units can encapsulate metadata for decoding video data. NAL units that encapsulate metadata for decoding video sequences are generally referred to as non-VCL NAL units. Therefore, in JVET-T2001, NAL units can be VCL NAL units or non-VCL NAL units. It should be noted that the VCL NAL unit includes slice header data, which provides information for decoding a specific slice. Therefore, in JVET-T2001, information for decoding video data (which may be referred to as metadata in some cases) is not limited to being included in non-VCL NAL units. JVET-T2001 specifies that a picture unit (PU) is a group of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain exactly one coded picture, and an access unit (AU) is a group of PUs that belong to different layers and contain coded pictures associated with the same time output from the DPB. JVET-T2001 further specifies that a layer is a group of VCL NAL units and their associated non-VCL NAL units that all have a specific value of the layer identifier. In addition, in JVET-T2001, a PU consists of zero or one PH NAL unit, one coded picture (which consists of one or more VCL NAL units), and zero or more other non-VCL NAL units. Furthermore, in JVET-T2001, a coded video sequence (CVS) is an AU sequence consisting of a CVSS AU and zero or more subsequent AUs that are non-CVSS AUs arranged in decoding order (including all subsequent AUs up to (excluding) any subsequent AU that is a CVSS AU), wherein a coded video sequence start (CVSS) AU is an AU in which a PU exists for each layer in the CVS, and a coded picture in each existing picture unit is a coded layer video sequence start (CLVSS) picture. In JVET-T2001, a coded layer video sequence (CLVS) is a PU sequence within the same layer, consisting of a CLVSS PU and zero or more subsequent PUs that are non-CLVSS PUs arranged in decoding order (including all subsequent PUs up to (excluding) any subsequent PU that is a CLVSS PU). That is, in JVET-T2001, a bitstream may be described as including a sequence of AUs forming one or more CVSs.

[0039] Multi-layer video coding enables video presentation to be decoded / displayed as a presentation corresponding to a base layer of video data, and decoded / displayed as one or more additional presentations corresponding to an enhancement layer of video data. For example, a base layer can enable presentation of a video presentation with a basic quality level (e.g., high-definition presentation and / or a 30Hz frame rate), and an enhancement layer can enable presentation of a video presentation with an enhanced quality level (e.g., ultra-high-definition rendering and / or a 60Hz frame rate). The enhancement layer can be encoded by reference to the base layer. That is, for example, the picture in the enhancement layer can be encoded (e.g., using inter-layer prediction techniques) by reference to one or more pictures in the base layer (including a scaled version thereof). It should be noted that the layers can also be encoded independently of each other. In this case, there may be no inter-layer prediction between the two layers. Each NAL unit may include an identifier indicating the video data layer to which the NAL unit is associated. As described above, a sub-bitstream extraction process can be used to decode and display only a specific region of interest of a picture. In addition, a sub-bitstream extraction process can be used to decode and display only a specific video layer. Sub-bitstream extraction may refer to a process by which a device receiving a compatible or compliant bitstream forms a new compatible or compliant bitstream by discarding and / or modifying data in the received bitstream. For example, sub-bitstream extraction may be used to form a new compatible or compliant bitstream corresponding to a particular video representation (e.g., a high-quality representation).

[0040] In JVET-T2001, each of a video sequence, GOP, picture, slice, and CTU can be associated with metadata describing video coding attributes, and some types of metadata are encapsulated in non-VCL NAL units. JVET-T2001 defines parameter sets that can be used to describe video data and / or video coding attributes. Specifically, JVET-T2001 includes the following four parameter sets: video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), and adaptive parameter set (APS), wherein SPS is applied to zero or more integer CVSs, PPS is applicable to zero or more integer coded pictures, APS is applicable to zero or more slices, and VPS can be optionally referenced by SPS. PPS is applied to a single coded picture that references it. In JVET-T2001, parameter sets can be encapsulated as non-VCL NAL units, and / or can be sent as message signals. JVET-T2001 also includes a picture header (PH), which is encapsulated as a non-VCL NAL unit. In JVET-T2001, the picture header is applied to all slices of the coded picture. JVET-T2001 further enables the signaling of decoding capability information (DCI) and supplementary enhancement information (SEI) messages. In JVET-T2001, DCI and SEI messages assist processes related to decoding, display or other purposes, however, DCI and SEI messages may not be required to construct luminance or chrominance samples according to the decoding process. In JVET-T2001, non-VCL NAL units can be used to signal DCI and SEI messages in the bitstream. In addition, DCI and SEI messages can be transmitted by some mechanism rather than by being present in the bitstream (i.e., signaling out of band).

[0041] Figure 3 An example of a bitstream including a plurality of CVSs is shown, wherein the CVSs include AUs, and the AUs include picture units. Figure 3 The example shown in corresponds to encapsulating Figure 2 An example of a slice NAL unit is shown in the example of . Figure 3 In the example shown, the corresponding picture unit of Pic3 includes three VCL NAL coded slice NAL units, namely slice 0 NAL unit, slice 1 NAL unit and slice 2 NAL unit, and two non-VCL NAL units, namely PPS NAL unit and PH NAL unit. Figure 3 In , the header is the NAL unit header (i.e., not to be confused with the slice header). In addition, it should be noted that in Figure 3In the example, other non-VCL NAL units not shown may be included in the CVS, such as SPS NAL units, VPS NAL units, SEI message NAL units, etc. In addition, it should be noted that in other examples, the PPS NAL unit for decoding Pic3 may be included elsewhere in the bitstream, for example, in the picture unit corresponding to Pic0, or may be provided by an external agency. As described in further detail below, in JVET-T2001, the PH syntax structure may be present in the slice header of the VCL NAL unit or in the PH NAL unit of the current PU. JVET-T2001 defines the NAL unit header semantics, which specifies the type of raw byte sequence payload (RBSP) data structure included in the NAL unit. Table 1 shows the syntax of the NAL unit header provided in JVET-T2001. Table 1

[0042] JVET-T2001 provides the following definitions for the corresponding syntax elements shown in Table 1. forbidden_zero_bit should be equal to 0. nuh_reserved_zero_bit shall be equal to 0. A value of 1 for nuh_reserved_zero_bit may be specified by ITU-T|ISO / IEC in the future. Although in this version of this specification, the value of nuh_reserved_zero_bit is required to be equal to 0, decoders conforming to this version of this specification shall allow a value of nuh_reserved_zero_bit equal to 1 to appear in the syntax and shall ignore (i.e., remove and discard from the bitstream) NAL units with nuh_reserved_zero_bit equal to 1. nuh_layer_id specifies the identifier of the layer to which the VCL NAL unit belongs or the identifier of the layer to which the non-VCL NAL unit applies. The value of nuh_layer_id shall be in the range of 0 to 55, inclusive. Other values ​​of nuh_layer_id are reserved for future use by ITU-T | ISO / IEC. Although in this version of this specification, the value of nuh_layer_id is required to be in the range of 0 to 55, inclusive, a decoder conforming to this version of this specification shall allow values ​​of nuh_layer_id greater than 55 to appear in the syntax and shall ignore (i.e., remove and discard from the bitstream) NAL units with nuh_layer_id greater than 55. The value of nuh_layer_id of all VCL NAL units of a coded picture shall be the same. The value of nuh_layer_id of a coded picture or PU is the value of nuh_layer_id of the VCL NAL unit of the coded picture or PU. When nal_unit_type is equal to PH_NUT or FD_NUT, nuh_layer_id shall be equal to the nuh_layer_id of the associated VCL NAL unit. When nal_unit_type is equal to EOS_NUT, nuh_layer_id shall be equal to one of the nuh_layer_id values ​​of the layer present in the CVS.

[0043] NOTE—The value of nuh_layer_id for DCI, OPI, VPS, AUD, and EOB NAL units is not constrained. nuh_temporal_id_plus1 minus 1 specifies the temporal identifier of the NAL unit. The value of nuh_temporal_id_plus1 shall not be equal to 0. The variable TemporalId is derived as follows: TemporalId=nuh_temporal_id_plusl-1 When nal_unit_type is in the range of IDR_W_RADL to RSV_IRAP_11 (inclusive), TemporalId shall be equal to 0. When nal_unit_type is equal to STSA_NUT and When vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 1, TemporalId should be greater than 0. The value of TemporalId shall be the same for all VCL NAL units of an AU. The value of TemporalId of a coded picture, PU, ​​or AU is the value of TemporalId of the VCL NAL unit of the coded picture, PU, ​​or AU. The value of TemporalId of a sub-layer representation is the maximum value of the TemporalId of all VCL NAL units in the sub-layer representation. The value of TemporalId for non-VCL NAL units is constrained as follows: - If nal_unit_type is equal to DCI_NUT, OPI_NUT, VPS_NUT, or SPS_NUT, TemporalId Should be equal to 0, and the TemporalId of the AU containing the NAL unit should be equal to 0. - Otherwise, if nal_unit_type is equal to PH_NUT, TemporalId shall be equal to the TemporalId of the PU containing the NAL unit. - Otherwise, if nal_unit_type is equal to EOS_NUT or EOB_NUT, TemporalId shall be equal to 0. - Otherwise, if nal_unit_type is equal to AUD_NUT, FD_NUT, PREFIX_SEI_NUT, or SUFFIX_SEI_NUT, the TemporalId shall be equal to the TemporalId of the AU containing the NAL unit. - Otherwise, when nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, The TemporalId shall be greater than or equal to the TemporalId of the PU containing the NAL unit.

[0044] NOTE—When the NAL unit is a non-VCL NAL unit, the value of TemporalId is equal to the minimum of the TemporalId values ​​of all AUs to which the non-VCL NAL unit applies. When nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, TemporalId may be greater than or equal to the TemporalId of the containing AU, since all PPSs and APSs may be included in the beginning of the bitstream (e.g., when they are transported out-of-band and the receiver places them at the beginning of the bitstream), with the first coded picture having TemporalId equal to 0. nal_unit_type specifies the NAL unit type, ie, the type of RBSP data structure contained in the NAL unit as specified in Table 2. NAL units with a nal_unit_type in the range of UNSPEC28..UNSPEC31 (inclusive), whose semantics are unspecified, shall not affect the decoding process specified in this specification. NOTE—NAL unit types in the range of UNSPEC_28..UNSPEC_31 may be used as determined by the application. The decoding process for these values ​​of nal_unit_type is not specified in this specification. Since different applications may use these NAL unit types for different purposes, special attention is expected when designing encoders that generate NAL units with these nal_unit_type values ​​and when designing decoders that interpret the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values ​​may only be used in contexts where "conflicts" in use (i.e., the meaning of the NAL unit content for the same nal_unit_type value is defined differently) are not important, or are not possible or are managed, for example, defined or managed in a controlling application or transport specification, or managed by the environment that controls the distribution of the bitstream. For purposes other than determining the amount of data in a DU of the bitstream, a decoder should ignore (remove from the bitstream and discard) the contents of all NAL units using reserved values ​​of nal_unit_type. NOTE—This requirement allows for compatible extensions of this specification to be defined in the future.

[0045] Table 2

[0046] NOTE—A clean random access (CRA) picture may have associated RASL or RADL pictures present in the bitstream. NOTE—An Instantaneous Decoding Refresh (IDR) picture with nal_unit_type equal to IDR_N_LP does not have an associated leading picture present in the bitstream. An IDR picture with nal_unit_type equal to IDR_W_RADL does not have an associated RASL picture present in the bitstream, but may have an associated RADL picture in the bitstream. The value of nal_unit_type shall be the same for all VCL NAL units of a sub-picture. A sub-picture is said to have the same NAL unit type as the VCL NAL units of the sub-picture. For any particular picture's VCL NAL unit, the following applies: - If pps_mixed_nalu_types_in_pic_flag is equal to 0, then for all VCL NAL units of the picture, The value of nal_unit_type shall be the same, and a picture or PU is referred to as having the same NAL unit type as the coded slice NAL units of the picture or PU. - Otherwise (pps_mixed_nalu_types_in_pic_flag is equal to 1), all of the following constraints apply: -Picture should have at least two sub-pictures. - A VCL NAL unit of a picture shall have two or more different nal_unit_type values. - There shall be no VCL NAL units for pictures with nal_unit_type equal to GDR_NUT. - When the nal_unit_type of the VCL NAL unit of the picture is equal to the value of IDR_W_RADL, IDR_N_LP or When nalUnitTypeA is set to CRA_NUT, the nal_unit_type of other VCL NAL units of the picture should be equal to nalUnitTypeA or TRAIL_NUT. The value of nal_unit_type shall be the same for all pictures of an IRAP or GDR AU. When sps_video_parameter_set_id is greater than 0, vps_max_tid_il_ref_pics_plus1[i][j] is equal to 0 (for j equal to GeneralLayerIdx[nuh_layer_id] and any value of i in the range of j+1 to vps_max_layers_minus1, inclusive), and pps_mixed_nalu_types_in_pic_flag is equal to 1, the value of nal_unit_type shall not be equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT.

[0047] Bitstream conformance requires that the following constraints apply: - When the picture is the leading picture of an IRAP picture, the picture shall be a RADL or RASL picture. - When a sub-picture is a leading sub-picture of an IRAP sub-picture, the sub-picture shall be a RADL or RASL sub-picture. - When a picture is not the leading picture of an IRAP picture, the picture shall not be a RADL or RASL picture. - When a sub-picture is not a leading sub-picture of an IRAP sub-picture, the sub-picture shall not be a RADL or RASL sub-picture. - RASL pictures shall not be present in the bitstream, these RASL pictures are associated with IDR pictures. - RASL sub-pictures shall not be present in the bitstream, these RASL sub-pictures are associated with IDR sub-pictures. - RADL pictures shall not be present in the bitstream, these RADL pictures are associated with IDR pictures with nal_unit_type equal to IDR_N_LP. NOTE—Random access at the location of an IRAP PU can be performed by discarding all PUs preceding the IRAP AU (and correctly decoding non-RASL pictures in the IRAP AU and all subsequent AUs in decoding order), provided that each parameter set is available (in the bitstream or by external means not specified in this specification) when referenced. - RADL sub-pictures shall not be present in the bitstream, these RADL sub-pictures are associated with IDR sub-pictures with nal_unit_type equal to IDR_N_LP. - Any pictures with nuh_layer_id equal to a particular value layerId that precede the IRAP picture with nuh_layer_id equal to layerId in decoding order shall precede the IRAP picture in output order, and shall precede any RADL pictures associated with the IRAP picture in output order. - Any sub-picture with nuh_layer_id equal to a specific value layerId and sub-picture index equal to a specific value subpicIdx that precedes the IRAP sub-picture with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx in decoding order shall precede the IRAP sub-picture and all its associated RADL sub-pictures in output order. - Any pictures with nuh_layer_id equal to a particular value layerId that precede the recovery-point picture with nuh_layer_id equal to layerId in decoding order shall precede the recovery-point picture in output order. - Any sub-picture with nuh_layer_id equal to a particular value layerId and a sub-picture index equal to a particular value subpicIdx that precedes the sub-picture with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx in the recovery point picture in decoding order shall precede the sub-picture in the recovery point picture in output order. - Any RASL pictures associated with a CRA picture shall precede any RADL pictures associated with the CRA picture in output order. - Any RASL sub-picture associated with a CRA sub-picture shall precede any RADL sub-picture associated with the CRA sub-picture in output order. - Any RASL picture with nuh_layer_id equal to a particular value layerId associated with a CRA picture shall follow, in output order, any IRAP or GDR picture with nuh_layer_id equal to layerId that precedes the CRA picture in decoding order. - Any RASL sub-picture associated with a CRA sub-picture with nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx shall follow, in output order, any IRAP or GDR sub-picture with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx that precedes the CRA sub-picture in decoding order. - If sps_field_seq_flag is equal to 0, the following applies: when the current picture with nuh_layer_id equal to a specific value layerId is a leading picture associated with an IRAP picture, then the current picture shall precede all non-leading pictures associated with the same IRAP picture in decoding order. Otherwise (sps_field_seq_flag is equal to 1), let picA and picB be the first leading picture and the last leading picture associated with the IRAP picture in decoding order, respectively, there shall be at most one non-leading picture with nuh_layer_id equal to layerId before picA in decoding order, and there shall be no non-leading picture with nuh_layer_id equal to layerId between picA and picB in decoding order.

[0049] - If sps_field_seq_flag is equal to 0, the following applies: when the current sub-picture with nuh_layer_id equal to a specific value layerId and sub-picture index equal to a specific value subpicIdx is a leading sub-picture associated with an IRAP sub-picture, then the current sub-picture shall precede all non-leading sub-pictures associated with the same IRAP sub-picture in decoding order. Otherwise (sps_field_seq_flag is equal to 1), let subpicA and subpicB be the first leading sub-picture and the last leading sub-picture associated with the IRAP sub-picture, respectively, in decoding order, there shall be at most one non-leading sub-picture with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx before subpicA in decoding order, and there shall be no non-leading picture with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx between picA and picB in decoding order.

[0050] As provided in Table 2, the NAL unit may include a supplemental enhancement information (SEI) syntax structure, and Tables 3 and 4 show the supplemental enhancement information (SEI) syntax structure provided in JVET-T2001. Table 3 Table 4

[0051] For Tables 3 and 4, JVET-T2001 provides the following semantics: Each SEI message consists of variables that specify the type payloadType and size payloadSize of the SEI message payload. The derived SEI message payload size payloadSize is specified in bytes and shall be equal to the number of RBSP bytes in the SEI message payload. NOTE—The NAL unit byte sequence containing the SEI message may include one or more emulation prevention bytes (indicated by the emulation_prevention_three_byte syntax element). Since the payload size of the SEI message is specified in the RBSP bytes, the number of emulation prevention bytes is not included in the size of the SEI payload, payloadSize. payload_type_byte is a byte of the payload type of the SEI message. Payload_size_byte is the bytes of the payload size of the SEI message.

[0052] It should be noted that JVET-T2001 defines payload types, and the 25th meeting of ISO / IEC JTC1 / SC29 / WG11 held from January 12 to 21, 2022, "Additional SEI messages for VSEI (Draft6)" (teleconference, document JVET-Y2006-v1, which is incorporated herein by reference and referred to as JVET-Y2006) defines additional payload types. Table 5 generally shows the sei_payload() syntax structure. That is, Table 5 shows the sei_payload() syntax structure, but for the sake of brevity, not all possible types of payloads are included in Table 5. Table 5

[0053] For Table 5, JVET-T2001 provides the following semantics. sei_reserved_payload_extension_data shall not be present in a bitstream conforming to this version of this specification. However, decoders conforming to this version of this specification shall ignore the presence and value of sei_reserved_payload_extension_data. When present, the length in bits of sei_reserved_payload_extension_data is equal to 8*payloadSize-nEarlierBits-nPayloadZeroBits-1, where nEarlierBits is the number of bits in the sei_payload() syntax structure preceding the sei_reserved_payload_extension_data syntax element, and nPayloadZeroBits is the number of sei_payload_bit_equal_to_zero syntax elements at the end of the sei_payload() syntax structure. If more_data_in_payload() is true and nPayloadZeroBits is not equal to 7 after parsing the SEI message syntax structure (eg, buffering_period() syntax structure), PayloadBits is set equal to 8*payloadSize-nPayloadZeroBits-1; otherwise, PayloadBits is set equal to 8*payloadSize. Payload_bit_equal_to_one should be equal to 1. payload_bit_equal_to_zero shall be equal to 0. NOTE—SEI messages with the same payloadType value are conceptually the same SEI messages regardless of whether they are contained in prefix or suffix SEI NAL units. NOTE—For the SEI messages specified in this specification and the VSEI specification (ITU-T H.274 | ISO / IEC 23002-7), the payloadType values ​​are aligned with similar SEI messages specified in AVC (Rec. ITU-T H.264 | ISO / IEC 14496-10) and HEVC (Rec. ITU-T H.265 | ISO / IEC 23008-2).

[0054] The semantics and persistence scope of each SEI message is specified in the semantics specification of each specific SEI message. NOTE—The persistence information of the SEI message is summarized in Table 142.

[0055] JVET-T2001 also provides the following: The SEI message having the syntax structure identified in [Table 5] is specified in Rec. ITU-T H.274 ISO / IEC 23002-7 can be used with the bitstream specified in this specification. When any specific RecITU-T H.274 | ISO / IEC 23002-7 SEI message is included in a bitstream as specified in this specification, the SEI payload syntax shall be included in the sei_payload() syntax structure specified in [Table 5], the payloadType value specified in [Table 5] shall be used, and in addition, any SEI message specific constraints specified in this annex for the specific SEI message shall apply. The value of PayloadBits as specified above is passed to the parser of the SEI message syntax structure specified in Rec. ITU-T H.274 | ISO / IEC 23002-7.

[0056] As described above, JVET-Z2006 provides a NN post-filter supplemental enhancement information message. Specifically, JVET-Z2006 provides a neural network post-filter characteristics SEI message (payloadType==210) and a neural network post-filter activation SEI message (payloadType==211). Tables 6 and 7 show the syntax of the neural network post-filter characteristics SEI message provided in JVET-Z2006. Table 6

[0057] Table 7

[0058] For Table 6 and Table 7, JVET-Z2006 provides the following semantics. This SEI message specifies a neural network that can be used as a post-processing filter. The use of a specified post-processing filter for a particular picture is indicated by a Neural Network Post-Filter Activation SEI message. The use of this SEI message requires the definition of the following variables: - The cropped decoded output picture width and height in units of luma samples, denoted herein as InpPicWidthInLumaSamples and InpPicHeightInLumaSamples, respectively. - the luma sample array CroppedYPic[y][x] and the chroma sample arrays CroppedCbPic[y][x] and CroppedCrPic[y][x] (when present) of the cropped decoded output picture for vertical coordinate y and horizontal coordinate x, where the upper left corner of the sample array has coordinate y equal to 0 and coordinate x equal to 0. - The bit depth BitDepthY of the luma sample array of the cropped decoded output picture. - The bit depth BitDepthC of the chroma sample array (if any) of the cropped decoded output picture. - Chroma subsampling ratio relative to luma expressed as InpSubWidthC and InpSubHeightC. - When nnpfc_inp_order_idc is equal to 3, the initial luma quantization parameter value SliceQPY. When this SET message specifies a neural network that can be used as a post-processing filter, the semantics specify the derivation of the luma sample array FilteredYPic[y][x], the chroma sample arrays FilteredCbPic[y][x], and FilteredCrPi[y][x], as indicated by the value of nnpfc_out_order_idc, which contain the output of the post-processing filter. nnpfc_id contains an identification number that can be used to identify the post-processing filter. The value of nnpfc_id should be between 0 and 2. 32 -2 (including the end value).

[0059] nnpfc_id ranges from 256 to 511 (inclusive) and from 2 31 to 2 32 Values ​​of -2 (inclusive) are reserved for future use by ITU-T|ISO / IEC. 31 to 2 32 Decoders in the range -2 (inclusive) will ignore it. nnpfc_mode_idc equal to 0 specifies that the post-processing filter associated with the nnpfc_id value is determined by external means not specified in this specification. nnpfc_mode_idc equal to 1 specifies that the post-processing filter associated with the nnpfc_id value is a neural network represented by the ISO / IEC 15938-17 bitstream contained in this SEI message. The value of nnpfc_mode_idc shall be in the range of 0 to 255, inclusive. Values ​​of nnpfc_mode_idc greater than 1 are reserved for future ITU-T|ISO / IEC specifications and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnpfc_mode_idc. When the current CLVS contains, in decoding order, a previous neural network post filter characteristics SEI message with a nnpfc_id value equal to the same value of nnpfc_id in this SEI message, at least one of the following conditions shall apply: - This SEI message has nnpfc_mode_idc equal to 1 and the same content as the previous neural network post-filter characteristics SEI message, except that the values ​​of nnpfc_mode_idc and nnpfc_payload_byte[i] may be different in order to provide neural network updates. - This SEI message has the same content as the previous neural network post-filter characteristics SEI message.

[0060] When this SEI message is the first neural network post-filter characteristics SEI message with a particular nnpfc_id value within the current CLVS in decoding order, it specifies the basic post-processing filters that are relevant to the current decoded picture of the current layer and all subsequent decoded pictures in output order until the end of the current CLVS. When this SEI message is not the first neural network post-filter characteristics SEI message with a particular nnpfc_id value within the current CLVS in decoding order, this SEI message is relevant to the current decoded picture of the current layer and all subsequent decoded pictures in output order until the end of the current CLVS or the next neural network post-filter characteristics SEI message with the particular nnpfc_id value within the current CLVS in output order. nnpfc_purpose indicates the purpose of the post-processing filter as specified in Table 8. The value of nnpfc_purpose shall be between 0 and 2. 32 The nnpfc_purpose values ​​not appearing in Table 8 are reserved for future ITU-T|ISO / IEC specifications and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​for nnpfc_purpose.

[0061] Table 8

[0062] NOTE—When ITU-T | ISO / IEC uses the reserved value of nnpfc_purpose in the future, the syntax of this SEI message may be extended with syntax elements whose presence is conditional on nnpfc_purpose being equal to that value. nnpfc_out_sub_width_c_flag and nnpfc_out_sub_height_c_flag are used to derive variables outSubWidthC and outSubHeightC, respectively, which specify the chroma subsampling ratio relative to luma in the picture resulting from post-processing filtering. When not present, nnpfc_out_sub_width_c_flag and nnnpfc_out_sub_height_c_flag are both inferred to be equal to 0. When nnpfc_out_sub_width_c_flag and nnpfc_out_sub_height_c_flag are present, the sum of nnpfc_out_sub_width_c_flag and nnpfc_out_sub_height_c_flag shall be greater than 0. outSubWidthC=InpSubWidthC-nnpfc_out_sub_width_c_flag outSubHeightC=InpSubHeightC-nnpfc_out_sub_height_c_flag Bitstream conformance requires that both outSubWidthC and outSubHeightC should be greater than 0. nnpfc_pic_width_in_luma_samples and nnpfc_pic_height_in_luma_samples specify the width and height, respectively, of the luma sample array of the picture produced by applying the post-processing filter identified by nnpfc_id to the cropped decoded output picture. When nnpfc_pic_width_in_luma_samples and nnpfc_pic_height_in_luma_samples are not present, they are inferred to be equal to InpPicWidthInLumaSamples and InpPicHeightInLumaSamples, respectively. nnpfc_component_last_flag equal to 0 specifies that the second dimension of the input tensor inputTensor is input to the post-processing filter, and the output tensor outputTensor produced from the post-processing filter is used for this channel. nnpfc_component_last_flag equal to 1 specifies that the last dimension of the input tensor inputTensor is input to the post-processing filter, and the output tensor outputTensor produced from the post-processing filter is used for this channel.

[0063] Note - The specified values ​​of nnpfc_inp_sample_idc and nnpfc_out_sample_idc specify that the first dimension in the input and output tensors, respectively, is used for batch indexing, which is a practice in some neural network frameworks. Although the semantics of this SEI message uses a batch size equal to 1, it is up to the post-processing implementation to determine the batch size used as input for neural network inference. NOTE—Color components are examples of channels. nnpfc_inp_sample_idc indicates the method of converting the sample values ​​of the cropped decoded output picture into the input values ​​to the post-processing filter. When nnpfc_inp_sample_idc is equal to 0, 1, 2, or 3, the input values ​​to the post-processing filter are binary16, binary32, binary64, or binary128 floating point values, respectively, as specified in IEEE 754-2019, and the functions InpY and InPc are specified as follows: InpY(x)=x÷((1< <BitDepthY)-1) InpC(x)=x÷((1< <BitDepthC)-1}

[0064] When nnpfc_inp_sample_idc is equal to 4, the input values ​​to the post-processing filter are unsigned integers, and the functions InpY and InpC are specified as follows: The variable inpTensorBitDepth is derived from the syntax element nnpfc_inp_tensor_bitdepth_minus8 as specified below. The value of nnpfc_inp_sample_idc shall be in the range of 0 to 255, inclusive. Values ​​of nnpfc_inp_sample_idc greater than 4 are reserved for future ITU-T|ISO / IEC specifications and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnpfc_inp_sample_idc. nnpfc_inp_tensor_bitdepth_minus8 specifies the bit depth of the luma sample values ​​in the input integer tensor by adding 8. The value of inpTensorBitDepth is derived as follows: inpTensorBitDepth=nnpfc_inp_tensor_bitdepth_minus8+8 Bitstream conformance requires that the value of nnpfc_inp_tensor_bitdepth_minus8 should be in the range of 0 to 24 (inclusive).

[0065] nnpfc_inp_order_idc indicates a method for ordering the sample array of the cropped decoded output picture as input to the post-processing filter. Table 9 contains an informative description of the nnpfc_inp_order_idc value. The semantics of nnpfc_inp_order_idc in the range of 0 to 3 (including the end value) are specified in Table 11, which specifies the process for deriving the input tensor inputTensor for different values ​​of nnpfc_inp_order_idc, and specifies the given vertical sample coordinate cTop and horizontal sample coordinate cLeft of the top left sample position of the sample block included in the input tensor. When the chroma format of the cropped decoded output picture is not 4:2:0, nnpfc_inp_order_idc shall not be equal to 3. The value of nnpfc_inp_order_idc shall be in the range of 0 to 255 (including the end value). Values ​​of nnpfc_inp_order_idc greater than 3 are reserved for future specification by ITU-T|ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnpfc_inp_order_idc.

[0066] Table 9

[0067] A block is a rectangular array of samples of a component (eg, luma or chroma components) from a picture. Nnpfc_constant_patch_size_flag equal to 0 specifies that the post-processing filter accepts as input any block size that is a positive integer multiple of the block size indicated by nnpfc_patch_width_minus1 and nnpfc_patch_height_minus1. nnpfc_constant_patch_size_flag equal to 1 specifies that the post-processing filter accepts as input exactly the block size indicated by nnpfc_patch_width_minus1 and nnpfc_patch_height_minus1. nnpfc_patch_width_minus1+1, when nnpfc_constant_patch_size_flag is equal to 1, specifies the horizontal sample count of the block size required for input to the post-processing filter. When nnpfc_constant_patch_size_flag is equal to 0, any positive integer multiple of (nnpfc_patch_width_minus1+1) can be used as the horizontal sample count of the block size used for input to the post-processing filter. The value of nnpfc_patch_width_minus1 should be in the range of 0 to 32766, inclusive. nnpfc_patch_height_minus1+1, when nnpfc_constant_patch_size_flag is equal to 1, specifies the vertical sample count of the block size required for input to the post-processing filter. When nnpfc_constant_patch_size_flag is equal to 0, any positive integer multiple of (nnpfc_patch_height_minus1+1) can be used as the vertical sample count of the block size used for input to the post-processing filter. The value of nnpfc_patch_height_minus1 should be in the range of 0 to 32766, inclusive.

[0068] nnpfc_overlap specifies the overlapping horizontal and vertical sample counts of adjacent input tensors for the post-processing filter. The value of nnpfc_overlap should be in the range of 0 to 16383 (inclusive). The variables inpPatchWidth, inpPatchHeight, outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, outPatchCHeight, and overlapSize are derived as follows: Bitstream conformance requires that outPatchWidth*InpPicWidthInLumaSamples shall be equal to nnpfc_pic_width_in_luma_samples*inpPatchWidth, and outPatchHeight*InpPicHeightInLumaSamples shall be equal to nnpfc_pic_height_in_luma_samples*inpPatchHeight. nnpfc_padding_type specifies the padding process when referring to sample positions outside the boundaries of the cropped decoded output picture as described in Table 10. The value of nnpfc_padding_type shall be in the range of 0 to 15, inclusive.

[0069] nnpfc_padding_type describe 0 Zero padding 1 Copy Fill 2 Reflection Fill 3..15 reserve Table 10

[0070] The function InpSampleVal(y,x,picHeight,picWidth,croppedPic) whose input is the vertical sample position y, the horizontal sample position x, the picture height picHeight, the picture width picWidth and the sample array croppedPic returns the value of sampleVal derived as follows:

[0071] Table 11

[0072] nnpfc_complexity_idc greater than 0 specifies that there may be one or more syntax elements indicating the complexity of the post-processing filter associated with nnpfc_id. nnpfc_complexity_idc equal to 0 specifies that there are no syntax elements indicating the complexity of the post-processing filter associated with nnpfc_id. The value nnpfc_complexity_idc should be in the range of 0 to 255, inclusive. Values ​​of nnpfc_complexity_idc greater than 1 are reserved for use by future ITU-T|ISO / IEC specifications and should not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification should ignore SEI messages containing reserved values ​​of nnpfc_complexity_idc. nnpfc_out_sample_idc equal to 0, 1, 2, or 3 indicates that the sample values ​​output by the post-processing filter are binary16, binary32, binary64, or binary128 floating point values, respectively, as specified in IEEE 754-2019. The functions OutY and OutC for converting the luma sample values ​​and chroma sample values ​​output by the post-processing filter to integer values ​​with bit depths BitDepthY and BitDepthC, respectively, are specified as follows:

[0073] nnpfc_out_sample_idc equal to 4 indicates that the sample values ​​output by the post-processing filter are unsigned integers. The functions OutY and OutC are specified as follows: The variable outTensorBitDepth is derived from the syntax element nnpfc_out_tensor_bitdepth minus8 as described below. The value of nnpfc_out_sample_idc shall be in the range of 0 to 255, inclusive. Values ​​of nnpfc_out_sample_idc greater than 4 are reserved for future specifications by ITU-T | ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnpfc_out_sample_idc.

[0074] nnpfc_out_tensor_bitdepth_minus8 specifies the bit depth of the sample values ​​in the output integer tensor by adding 8. The value of outTensorBitDepth is derived as follows: outTensorBitDepth=nnpfc_out_tensor_bitdepth_minus8+8 Bitstream conformance requires that the value of nnpfc_out_tcnsor_bitdcpth_minus8 shall be in the range of 0 to 24, inclusive. nnpfc_out_order_idc indicates the output order of samples generated from the post-processing filter. Table 12 contains an informative description of nnpfc_out_order_idc values. The semantics of nnpfc_out_order_idc in the range of 0 to 3 (including the end value) are specified in Table 13, which specifies the process for deriving sample values ​​in the filtered output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic according to the output tensor outputTensor for different values ​​of nnpfc_out_order_idc, and specifies the given vertical sample coordinate cTop and horizontal sample coordinate cLeft of the top left sample position of the sample block included in the input tensor. When nnpfc_purpose is equal to 2 or 4, nnpfc_out_order_idc should not be equal to 3. The value of nnpfc_out_order_idc should be in the range of 0 to 255 (including the end value). Values ​​of nnpfc_out_order_idc greater than 3 are reserved for future specification by ITU-T|ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnpfc_out_order_idc.

[0075] Table 12

[0076] Table 13

[0077] The basic post-processing filters used for the cropped decoded output picture picA are the filters identified by the first neural network post-filter characteristics SEI message with a specific nnpfc_id value within CLVS in decoding order. If there is another neural network post-filter characteristics SEI message with the same nnpfc_id value, with nnpfc_mode_idc equal to 1, with different content than the neural network post-filter characteristics SEI message defining the basic post-processing filter and related to the picture picA, the basic post-processing filter is updated by decoding the ISO / IEC15938-17 bitstream in the neural network post-filter characteristics SEI message to obtain the post-processing filter PostProcessingFilter(). Otherwise, the post-processing filter PostProcessingFilter() is specified to be the same as the basic post-processing filter. The following process is used to filter the cropped decoded output picture using a post-processing filter PostProcessingFilter() to produce a filtered picture, which contains Y, Cb, Cr sample arrays FilteredYPic, FilteredCbPic, FilteredCrPic, respectively, as indicated by nnpfc_out_order_idc.

[0078] nnpfc_reserved_zero_bit should be equal to 0.

[0079] nnpfc_payload_byte[i] contains the i-th byte of a bitstream conforming to ISO / IEC 15938-17. The byte sequence nnpfc_payload_byte[i] for all current values ​​of i shall be a complete bitstream conforming to ISO / IEC 15938-17. nnpfc_parameter_type_flag equal to 0 indicates that the neural network uses only integer parameters. nnpfc_parameter_type_flag equal to 1 indicates that the neural network can use floating point or integer parameters. nnpfc_log2_parameter_bit_length_minus3 equal to 0, 1, 2, and 3 instructs the neural network to not use parameters with bit lengths greater than 8, 16, 32, and 64, respectively. nnpfc_num_parameters_idc indicates the maximum number of neural network parameters for the post-processing filters in powers of 2048. nnpfc_num_parameters_idc equal to 0 indicates that the maximum number of neural network parameters is not specified. If the value of nnpfc_num_parameters_idc is greater than zero, the variable maxNumParameters is derived as follows: maxNumParameters=(2048< <nnpfe_num_parameters_idc)-1 Bitstream conformance requires that the number of neural network parameters of the post-processing filter shall be less than or equal to maxNumParameters. nnpfc_num_kmac_operations_idc greater than 0 specifies that the maximum number of multiply-accumulate operations per sample of the post-processing filter is less than or equal to nnpfc_num_kmac_operations_idc * 1000. nnpfc_num_kmac_operations_idc equal to 0 specifies that the maximum number of multiply-accumulate operations is not specified.

[0080] Table 14 shows the syntax of the neural network post-filter activation SEI message provided in JVET-Z2006. Table 14

[0081] For Table 14, JVET-Z2006 provides the following semantics. This SEI message specifies the neural network post-processing filters that can be used as post-processing filters for the current picture. The neural network post-processing filter activation SEI message is only valid for the current picture. NOTE—There may be several neural network post-processing filter activation SEI messages for the same picture, for example, when the post-processing filters are intended for different purposes or filter different color components. nnpfa_id specifies that the neural network post-processing filters specified by one or more neural network post-processing filter characteristics SEI messages associated with the current picture and having nnpfc_id equal to nnfpa_id can be used for post-processing filtering of the current picture.

[0082] Furthermore, it should be noted that JVET-Z2006 stipulates the following for the use of the NN post-filter characteristics SEI message: For the purpose of interpreting the Neural Network Post-Filter Characteristics SEI message, the following variables are specified: - InpPicWidthInLumaSamples is set equal to pps_pic_width_in_luma_samples-SubWidthC*(pps_conf_win_left_offset+pps_conf_win_right_offset). -InpPicHeightInLumaSamples is set equal to pps_pic_height_in_luma_samples-SubHeightC*(pps_conf_win_top__offset+pps_conf_win_bottom_offset). - When present, the variables CroppedYPic[y][x], the chroma sample arrays CroppedCbPic[y][x], and CroppedCrPic[y][x] are set to 2-dimensional arrays of decoded sample values ​​of the 0th, 1st, and 2nd components, respectively, of the cropped decoded output picture to which the Neural Network Post-Filter Characteristics SEI message applies. -BitDepthY and BitDepthC are both set equal to BitDepth. -InpSubWidthC is set equal to SubWidthC. -InpSubHeightC is set equal to SubHeightC. -SliceQPY is set equal to SliceQpy. When neural network post-filter characteristics SEI messages with the same nnpfc_id and different contents are present in the same picture unit, the two neural network post-filter characteristics SEI messages shall be present in the same SEI NAL unit.

[0083] The neural network post-filter characteristics SEI message provided in JVET-Z2006 may not be ideal. Specifically, JVET-Z2006 does not include a method for specifying neural network post-filter data via an external URI (Uniform Resource Identifier). It should be noted that JVET-Z0052 includes the nnrpf_mode_idc syntax element. According to JVET-Z0052, nnrpf_mode_idc equal to 1 specifies that the post-processing filter associated with the nnrpf_id value is a neural network identified by a given Uniform Resource Identifier (URI) having the syntax and semantics as specified in IETF Internet Standard 66. In JVET-Z0052, the URI is signaled by the syntax element nnrpf_uri[i]. The semantics of nnrpf_mode_idc and nnrpf_uri[i] specified in JVET-Z0052 are as follows: nnrpf_mode_idc equal to 0 indicates that the post-processing filter associated with the nnrpf_id value is not specified in this SEI message and the persistence of the previous post-processing filter with the same nnrpf_id value is canceled, nnrpf_mode_idc equal to 1 specifies that the post-processing filter associated with the nnrpf_id value is a neural network identified by a given uniform resource identifier (URI) having syntax and semantics as specified in IETF Internet Standard 66. nnrpf_mode_idc equal to 2 specifies that the post-processing filter associated with the nnrpf_id value is a neural network represented by the ISO / IEC 15938-17 bitstream with general_profile_idc equal to 0 contained in this SEI message, and nnrpf_mode_idc equal to 3 specifies that the post-processing filter is a neural network identified by a previous NNR postfilter SEI message in decoding order with nnrpf_mode_idc equal to 1 or 2 and updated by the ISO / IEC 15938-17 bitstream with general_profile_idc equal to 1 contained in this SEI message. nnrpf_uri[i] contains the i-th byte of a NULL-terminated UTF-8 string as specified in IETF Internet Standard 63. This UTF-8 string contains a URI with syntax and semantics as specified in IETF Internet Standard 66 that identifies a neural network to be used as a post-processing filter.

[0084] It can be argued that the goal of specifying a URI to point to external neural network data is to allow external entities to define neural network post-processing filters in different formats based on their applications and / or preferences. However, merely signaling a URI with the nnrpf_uri[i] syntax element does not specify what the format of the neural network data in the specified URI is. Therefore, the URI specified by the nnrpf_uri[i] specified in JVET-Z0052 does not uniquely identify how to interpret the corresponding neural network data.

[0085] According to the techniques described herein, an additional URI may be signaled to uniquely identify the identity and format of the neural network data specified by the NN URI syntax element. In one example, according to the techniques described herein, the additional URI may be a tag URI. In one example, the tag URI may be signaled using the syntax element nnrpf_uri_tag[i], which is signaled only when nnrpf_mode_idc (or similar syntax element) specifies that the neural network data is specified via an external URI (i.e., for example, only when the nnrpf_uri[i] syntax element is signaled). Thus, according to the techniques described herein, the URI uniquely identifies the neural network post-processing filter pointed to by the other signaled URIs.

[0086] Figure 1 1 is a block diagram illustrating an example of a system that may be configured to encode (e.g., encode and / or decode) video data in accordance with one or more techniques of this disclosure. System 100 represents an example of a system that may encapsulate video data in accordance with one or more techniques of this disclosure. Figure 1 As shown, system 100 includes source device 102, communication medium 110, and target device 120. Figure 1 In the example shown, source device 102 may include any device configured to encode video data and transmit the encoded video data to communication medium 110. Target device 120 may include any device configured to receive the encoded video data via communication medium 110 and decode the encoded video data. Source device 102 and / or target device 120 may include computing devices equipped for wired and / or wireless communication, and may include, for example, set-top boxes, digital video recorders, televisions, desktop, laptop or tablet computers, game consoles, medical imaging devices, and mobile devices (including, for example, smart phones, cellular phones, personal gaming devices).

[0087] The communication medium 110 may include any combination of wireless and wired communication media and / or storage devices. The communication medium 110 may include coaxial cables, fiber optic cables, twisted pair cables, wireless transmitters and receivers, routers, switches, repeaters, base stations, or any other device that can be used to facilitate communication between various devices and sites. The communication medium 110 may include one or more networks. For example, the communication medium 110 may include a network configured to allow access to the World Wide Web, such as the Internet. The network may operate according to a combination of one or more telecommunication protocols. The telecommunication protocol may include proprietary aspects and / or may include standardized telecommunication protocols. Examples of standardized telecommunication protocols include digital video broadcasting (DVB) standards, advanced television system committee (ATSC) standards, integrated service digital broadcasting (ISDB) standards, cable data service interface specifications (DOCSIS) standards, global mobile communication systems (GSM) standards, code division multiple access (CDMA) standards, 3rd generation partnership project (3GPP) standards, European Telecommunications Standards Institute (ETSI) standards, Internet Protocol (IP) standards, wireless application protocol (WAP) standards, and Institute of Electrical and Electronics Engineers (IEEE) standards.

[0088] The storage device may include any type of device or storage medium capable of storing data. The storage medium may include a tangible or non-transitory computer-readable medium. The computer-readable medium may include an optical disk, a flash memory, a magnetic memory, or any other suitable digital storage medium. In some examples, the memory device or a portion thereof may be described as a non-volatile memory, and in other examples, a portion of the memory device may be described as a volatile memory. Examples of volatile memory may include random access memory (RAM), dynamic random access memory (DRAM), and static random access memory (SRAM). Examples of non-volatile memory may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable and programmable (EEPROM) memory. The storage device may include a memory card (e.g., a secure digital (SD) memory card), an internal / external hard disk drive, and / or an internal / external solid-state drive. Data may be stored on the storage device according to a defined file format.

[0089] Figure 4 is a conceptual diagram illustrating examples of components that may be included in a specific implementation of system 100. Figure 4 In the exemplary implementation shown, system 100 includes one or more computing devices 402A-402N, a television service network 404, a television service provider site 406, a wide area network 408, a local area network 410, and one or more content provider sites 412A-412N. Figure 4The specific implementations shown in represent examples of a system that can be configured to allow digital media content (such as movies, live sporting events, etc.) and data and applications associated therewith and media presentations to be distributed to and accessed by multiple computing devices (such as computing devices 402A to 402N). Figure 4 In the example shown, computing devices 402A to 402N may include any device configured to receive data from one or more of television service network 404, wide area network 408, and / or local area network 410. For example, computing devices 402A to 402N may be equipped for wired and / or wireless communications and may be configured to receive services via one or more data channels, and may include televisions, including so-called smart televisions, set-top boxes, and digital video recorders. In addition, computing devices 402A to 402N may include desktop computers, laptops or tablet computers, game consoles, mobile devices (including, for example, "smart" phones, cellular phones, and personal gaming devices).

[0090] The television service network 404 is an example of a network configured to enable distribution of digital media content that may include television services. For example, the television service network 404 may include a public air television network, a public or subscription-based satellite television service provider network, and a public or subscription-based cable television provider network and / or a cloud or Internet service provider. It should be noted that, although in some examples, the television service network 404 may be primarily used to allow the provision of television services, the television service network 404 may also allow the provision of other types of data and services according to any combination of the telecommunications protocols described herein. In addition, it should be noted that in some examples, the television service network 404 may allow two-way communication between the television service provider site 406 and one or more of the computing devices 402A to 402N. The television service network 404 may include any combination of wireless and / or wired communication media. The television service network 404 may include coaxial cables, fiber optic cables, twisted pair cables, wireless transmitters and receivers, routers, switches, repeaters, base stations, or any other device that can be used to facilitate communication between various devices and sites. The television service network 404 may operate according to a combination of one or more telecommunications protocols. The telecommunication protocol may include proprietary aspects and / or may include a standardized telecommunication protocol. Examples of standardized telecommunication protocols include the DVB standard, the ATSC standard, the ISDB standard, the DTMB standard, the DMB standard, the Data Over Cable Service Interface Specification (DOCSIS) standard, the HbbTV standard, the W3C standard, and the UPnP standard.

[0091] Reference again Figure 4, the television service provider site 406 may be configured to distribute television services via the television service network 404. For example, the television service provider site 406 may include one or more broadcast stations, cable television providers or satellite television providers, or Internet-based television providers. For example, the television service provider site 406 may be configured to receive transmissions (including television programs) via satellite uplink / downlink. In addition, as Figure 4 As shown, television service provider site 406 may be in communication with wide area network 408 and may be configured to receive data from content provider sites 412A through 412N. It should be noted that in some examples, television service provider site 406 may include a television studio and the content may originate from the television studio.

[0092] Wide area network 408 may include a packet-based network and operate according to a combination of one or more telecommunication protocols. Telecommunication protocols may include proprietary aspects and / or may include standardized telecommunication protocols. Examples of standardized telecommunication protocols include global system mobile communications (GSM) standards, code division multiple access (CDMA) standards, 3rd generation partnership project (3GPP) standards, European Telecommunications Standards Institute (ETSI) standards, European standards (EN), IP standards, wireless application protocols (WAP) standards, and Institute of Electrical and Electronics Engineers (IEEE) standards, such as, one or more IEEE 802 standards (e.g., Wi-Fi). Wide area network 408 may include any combination of wireless and / or wired communication media. Wide area network 408 may include coaxial cables, fiber optic cables, twisted pair cables, Ethernet cables, wireless transmitters and receivers, routers, switches, repeaters, base stations, or any other equipment that can be used to facilitate communication between various devices and sites. In one example, wide area network 408 may include the Internet. Local area network 410 may include a packet-based network and operate according to a combination of one or more telecommunication protocols. The local area network 410 may be distinguished from the wide area network 408 based on access levels and / or physical infrastructure. For example, the local area network 410 may include a secure home network.

[0093] Reference again Figure 4, content provider sites 412A to 412N represent examples of sites that can provide multimedia content to television service provider sites 406 and / or computing devices 402A to 402N. For example, a content provider site may include a studio with one or more studio content servers, which are configured to provide multimedia files and / or streams to television service provider sites 406. In one example, content provider sites 412A to 412N may be configured to provide multimedia content using an IP suite. For example, a content provider site may be configured to provide multimedia content to a receiver device according to a real-time streaming protocol (RTSP), HTTP, etc. In addition, content provider sites 412A to 412N may be configured to provide data including hypertext-based content, etc. to one or more of receiver devices 402A to 402N and / or television service provider sites 406 via a wide area network 408. Content provider sites 412A to 412N may include one or more web servers. The data provided by data provider sites 412A to 412N may be defined according to a data format.

[0094] Reference again Figure 1 , source device 102 includes video source 104, video encoder 106, data encapsulator 107 and interface 108. Video source 104 may include any device configured to capture and / or store video data. For example, video source 104 may include a camera and a storage device operably coupled thereto. Video encoder 106 may include any device configured to receive video data and generate a compatible bitstream representing the video data. A compatible bitstream may refer to a bitstream from which a video decoder can receive and reproduce video data. Various aspects of a compatible bitstream may be defined according to a video coding standard. When generating a compatible bitstream, video encoder 106 may compress the video data. The compression may be lossy (perceptible or imperceptible to the observer) or lossless. Figure 5 is a block diagram illustrating an example of a video encoder 500 that can implement the techniques described herein for encoding video data. It should be noted that although the exemplary video encoder 500 is illustrated as having different functional blocks, such illustration is intended for descriptive purposes and does not limit the video encoder 500 and / or its subcomponents to a particular hardware or software architecture. The functionality of the video encoder 500 may be implemented using any combination of hardware, firmware, and / or software implementations.

[0095] The video encoder 500 can perform intra-frame prediction encoding and inter-frame prediction encoding of a picture region and thus can be referred to as a hybrid video encoder. Figure 5In the example shown, the video encoder 500 receives a source video block. In some examples, the source video block may include picture regions that have been divided according to a coding structure. For example, the source video data may include macroblocks, CTUs, CBs, sub-partitions thereof, and / or additional equivalent coding units. In some examples, the video encoder 500 may be configured to perform additional subdivisions of the source video block. It should be noted that the techniques described herein are generally applicable to video encoding regardless of how the source video data is divided before and / or during encoding. Figure 5 In the example shown, the video encoder 500 includes an adder 502, a transform coefficient generator 504, a coefficient quantization unit 506, an inverse quantization and transform coefficient processing unit 508, an adder 510, an intra-frame prediction processing unit 512, an inter-frame prediction processing unit 514, a filter unit 516, and an entropy coding unit 518. Figure 5 As shown, video encoder 500 receives source video blocks and outputs a bitstream.

[0096] exist Figure 5 In the example shown, the video encoder 500 can generate residual data by subtracting the predicted video block from the source video block. The selection of the predicted video block is described in detail below. The summer 502 represents a component configured to perform the subtraction operation. In one example, the subtraction of the video block occurs in the pixel domain. The transform coefficient generator 504 applies a transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform (for example, four 8×8 transforms can be applied to a 16×16 residual value array) to the residual block or its sub-partition to generate a set of residual transform coefficients. The transform coefficient generator 504 can be configured to perform any and all combinations of transforms included in the discrete triangular transform series, including its approximation. The transform coefficient generator 504 can output the transform coefficients to the coefficient quantization unit 506. The coefficient quantization unit 506 can be configured to perform quantization of the transform coefficients. The quantization process can reduce the bit depth associated with some or all coefficients. The degree of quantization can change the rate distortion (i.e., the relationship between bit rate and video quality) of the encoded video data. The degree of quantization can be modified by adjusting a quantization parameter (QP). The quantization parameter can be determined based on a slice level value and / or a CU level value (e.g., a CU delta QP value). The QP data may include any data used to determine the QP used to quantize a particular set of transform coefficients. Figure 5 As shown, the quantized transform coefficients (which may be referred to as level values) are output to the inverse quantization and transform coefficient processing unit 508. The inverse quantization and transform coefficient processing unit 508 may be configured to apply inverse quantization and inverse transformation to generate reconstructed residual data. Figure 5As shown, at the summer 510, the reconstructed residual data can be added to the predicted video block. In this way, the encoded video block can be reconstructed, and the resulting reconstructed video block can be used to evaluate the coding quality of a given prediction, transformation and / or quantization. The video encoder 500 can be configured to perform multiple encoding rounds (e.g., performing encoding while changing one or more of the prediction, transformation parameters and quantization parameters). The rate distortion or other system parameters of the bitstream can be optimized based on the evaluation of the reconstructed video block. In addition, the reconstructed video block can be stored and used as a reference for predicting subsequent blocks.

[0097] Reference again Figure 5 , the intra-frame prediction processing unit 512 may be configured to select an intra-frame prediction mode for a video block to be encoded. The intra-frame prediction processing unit 512 may be configured to evaluate a frame and determine an intra-frame prediction mode for encoding a current block. As described above, possible intra-frame prediction modes may include a planar prediction mode, a DC prediction mode, and an angular prediction mode. Furthermore, it should be noted that in some examples, the prediction mode of the chrominance component may be inferred based on the prediction mode of the luma prediction mode. The intra-frame prediction processing unit 512 may select an intra-frame prediction mode after performing one or more encoding rounds. Furthermore, in one example, the intra-frame prediction processing unit 512 may select a prediction mode based on a rate-distortion analysis. As Figure 5 As shown, the intra prediction processing unit 512 outputs intra prediction data (e.g., syntax elements) to the entropy coding unit 518 and the transform coefficient generator 504. As described above, the transform performed on the residual data can be mode-dependent (e.g., the secondary transform matrix can be determined based on the prediction mode).

[0098] Reference again Figure 5 , the inter-frame prediction processing unit 514 may be configured to perform inter-frame prediction encoding for the current video block. The inter-frame prediction processing unit 514 may be configured to receive a source video block and calculate a motion vector of a PU of the video block. The motion vector may indicate the displacement of a prediction unit of a video block within a current video frame relative to a prediction block within a reference frame. Inter-frame prediction encoding may use one or more reference pictures. In addition, motion prediction may be unidirectional prediction (using one motion vector) or bidirectional prediction (using two motion vectors). The inter-frame prediction processing unit 514 may be configured to select a prediction block by calculating a pixel difference determined by, for example, the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. As described above, a motion vector may be determined and specified based on motion vector prediction. As described above, the inter-frame prediction processing unit 514 may be configured to perform motion vector prediction. The inter-frame prediction processing unit 514 may be configured to generate a prediction block using motion prediction data. For example, the inter-frame prediction processing unit 514 may locate a prediction video block ( Figure 5). It should be noted that the inter-frame prediction processing unit 514 can be further configured to apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for motion estimation. The inter-frame prediction processing unit 514 can output the motion prediction data of the calculated motion vector to the entropy coding unit 518.

[0099] like Figure 5 As shown, filter unit 516 receives the reconstructed video blocks and the encoding parameters and outputs the modified reconstructed video data. Filter unit 516 may be configured to perform deblocking and / or sample adaptive offset (SAO) filtering. SAO filtering is a nonlinear amplitude mapping that can be used to improve reconstruction by adding an offset to the reconstructed video data. It should be noted that, as Figure 5 As shown, the intra prediction processing unit 512 and the inter prediction processing unit 514 may receive the modified reconstructed video block via the filter unit 216. The entropy coding unit 518 receives the quantized transform coefficients and the prediction syntax data (i.e., the intra prediction data and the motion prediction data). It should be noted that in some examples, the coefficient quantization unit 506 may perform a scan of the matrix including the quantized transform coefficients before outputting the coefficients to the entropy coding unit 518. In other examples, the entropy coding unit 518 may perform the scan. The entropy coding unit 518 may be configured to perform entropy coding according to one or more of the techniques described herein. As such, the video encoder 500 represents an example of a device configured to generate encoded video data according to one or more techniques of the present disclosure.

[0100] Reference again Figure 1 , the data encapsulator 107 can receive the encoded video data and generate a compatible bitstream according to a defined data structure, for example, a sequence of NAL units. A device receiving the compatible bitstream can reproduce the video data therefrom. In addition, as described above, sub-bitstream extraction can refer to the process by which a device receiving a compatible bitstream forms a new compatible bitstream by discarding and / or modifying data in the received bitstream. It should be noted that the term conforming bitstream can be used instead of the term compatible bitstream. In one example, the data encapsulator 107 can be configured to generate syntax according to one or more techniques described herein. It should be noted that the data encapsulator 107 does not necessarily have to be located in the same physical device as the video encoder 106. For example, the functions described as being performed by the video encoder 106 and the data encapsulator 107 can be distributed in Figure 4 in the device shown.

[0101] As described above, in one example, according to the techniques described herein, an additional URI may be signaled to uniquely identify the identity and format of the neural network data specified by the NN URI syntax element. Table 15 shows an exemplary syntax of a neural network post-filter characteristics SEI message according to the techniques described herein. Table 15

[0102] With respect to Table 15, in one example, the semantics may be based on the semantics provided above and the following semantics: nnpfc_mode_idc equal to 0 specifies that the post-processing filter associated with the nnpfc_id value is determined by external means not specified in this specification. nnpfc_mode_idc equal to 1 specifies that the post-processing filter associated with the nnpfc_id value is a neural network represented by the ISO / IEC 15938-17 bitstream contained in this SEI message. nnrpf_mode_idc equal to 2 specifies that the post-processing filter associated with the nnpfc_id value is a neural network identified by the specified tag uniform resource identifier (URI) (nnrpf_uri_tag[i]) and the neural network information URI (nnrpf_uri[i]). The value of nnpfc_mode_idc shall be in the range of 0 to 255, inclusive. Values ​​of nnpfc_mode_idc greater than 1 are reserved for future ITU-T|ISO / IEC specifications and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnpfc_mode_idc. When the current CLVS contains, in decoding order, a previous neural network post filter characteristics SEI message with a nnpfc_id value equal to the same value of nnpfc_id in this SEI message, at least one of the following conditions shall apply: - This SEI message has nnpfc_mode_idc equal to 1 and the same content as the previous neural network post-filter characteristics SEI message, except that the values ​​of nnpfc_mode_idc and nnpfc_payload_byte[i] may be different in order to provide neural network updates. - This SEI message has the same content as the previous Neural Network Post-Filter Characteristics SEI message

[0103] When this SEI message is the first neural network post-filter characteristics SEI message with a particular nnpfc_id value within the current CLVS in decoding order, it specifies the basic post-processing filters that are relevant to the current decoded picture of the current layer and all subsequent decoded pictures in output order until the end of the current CLVS. When this SEI message is not the first neural network post-filter characteristics SEI message with a particular nnpfc_id value within the current CLVS in decoding order, this SEI message is relevant to the current decoded picture of the current layer and all subsequent decoded pictures in output order until the end of the current CLVS or the next neural network post-filter characteristics SEI message with the particular nnpfc_id value within the current CLVS in output order. nnrpf_uri_tag[i] contains the i-th byte of a NULL-terminated UTF-8 string that is a tag URI. This UTF-8 string contains a URI with syntax and semantics as specified in IETF RFC 4151 that uniquely identifies the format and associated information of the neural network specified by the nnrpf_uri[i] value used as a post-processing filter. NOTE—The nnrpf_uri_tag[i] element represents a "tag" URI that allows the format of the neural network data specified by the nnrpf_uri[i] value to be uniquely defined without the need for a central registration authority. or nnrpf_uri_tag[i] contains the i-th byte of a NULL-terminated UTF-8 string that is a tag URI, as specified in IETF RFC 4151. This UTF-8 string contains a URI with syntax and semantics as specified in IETF RFC 4151 that uniquely identifies the format and associated information of the neural network specified by the nnrpf_uri[i] value for use as a post-processing filter. NOTE—The nnrpf_uri_tag[i] element represents a "tag" URI that allows the format of the neural network data specified by the nnrpf_uri[i] value to be uniquely defined without the need for a central registration authority.

[0104] nnrpf_uri[i] contains the i-th byte of a NULL-terminated UTF-8 string as specified in IETF Internet Standard 63. This UTF-8 string contains a URI with syntax and semantics as specified in IETF Internet Standard 66 that identifies neural network information (e.g., data representation) used as a post-processing filter.

[0105] In one example, nnrpf_uri[i] may be allowed to be zero.

[0106] In one example, instead of nnrpf_mode_idc being equal to 2, some other value (eg, 3 or 5 or some other value) may be used to specify that the signaling external URI is to be sent. In one example, according to the techniques of this document, one or more URIs may be signaled by first signaling a syntax element that specifies the length of the URI and then signaling the bytes of the URI. Table 16 shows an example of the relevant syntax of a neural network post-filter characteristics SEI message set according to the techniques of this document. Table 16

[0107] With respect to Table 16, in one example, the semantics may be based on the semantics provided above and the following semantics: nnrpf_uri_tag_length specifies the number of nnrpf_uri_tag[i] elements that exist, and nnrpf_uri_tag_length equal to 0 specifies that there is no nnrpf_uri_tag[i]. In one example, the value of nnrf_uri_tag_length should not be equal to 0. nnrpf_uri_length specifies the number of nnrpf_uri[i] elements that exist, and nnrpf_uri_length equal to 0 specifies that there is no nnrpf_uri_tag[i]. The variant nnrpf_uri_tag length should not be equal to 0.

[0108] In one example, the syntax element nnrpf_uri_tag_length may be replaced with the syntax element nrpf_uri_tag_length_minus1 having the following syntax and semantics: nnrpf_uri_tag_length_minus1 plus 1 specifies the number of nnrpf_uri_tag[i] elements that exist. nnrpf_uri_tag_length equal to 0 specifies that there are no nnrpf_uri_tag[i].

[0109] Furthermore, in one example, the syntax element nnrpf_uri_length may be replaced with the syntax element nrpf_uri_length_minus1 having the following syntax and semantics: nnrpf_uri_length_minus1 plus 1 specifies the number of nnrpf_uri[i] elements that exist, and nnrpf_uri_length equal to 0 specifies that there is no nnrpf_uri_tag[i].

[0110] In one example, a signaling tag URI may be sent according to the techniques described herein. Table 17 shows an example syntax of a neural network post-filter characteristics SEI message according to the techniques described herein. Table 17

[0111] With respect to Table 17, in one example, the semantics may be based on the semantics provided above and the following semantics: nnrpf_uri_tag[i] contains the i-th byte of a NULL-terminated UTF-8 string as specified in IETF Internet Standard 63. This UTF-8 string contains a URI with syntax and semantics as specified in IETF RFC 4151 that uniquely identifies neural network information (e.g., data representation) used as a post-processing filter.

[0112] In one example, an external XYZ company may define a nnrpf_uri_tag[i] value that specifies a tag URI: tag:xyzcorp.com,2022:nnfilter1 In this case, the data format of the neural network post-processing filter information specified by the URI formed by the nnrpf_uri[i] value will be as recorded externally by XYZ Corporation according to their unique tags (tag:xyzcorp.com,2022:nnfilter1). The same external XYZ company may define additional filters that may be identified by exemplary tag URIs such as: tag:xyzcorp.com,2022:nnfilter2 tag:xyzcorp.com,2022:nnfilter3 tag:xyzcorp.com,2023:adv-nnfilterA

[0113] Similarly, a neural network post-filter defined by another GHJ company may use the following exemplary tag URI: tag:ghj.com,2022:nnf / pp1 tag:ghj.com,2022:nnf / pp2

[0114] As mentioned above, in one example, nnrpf_uri[i] is zero. This can be used when nnrpf_uri_tag[i] uses a tag URI that defines fixed neural network post-processing filter information. In this case, the information can be obtained without signaling the nnrpf_uri[i] URI. As provided in Table 6 above, in JVET-Z2006, when nnpfc_purpose is equal to 2 or 4 (i.e., when chroma upsampling neural network filter information is signaled), two syntax elements nnpfc_out_sub_width_c_flag and nnpfc_out_sub_height_c_flag are signaled. The syntax elements nnpfc_out_sub_width_c_flag and nnpfc_out_sub_height_c_flag are 1-bit flags that specify output chroma information. Table 18 shows how the chroma format is specified based on the value of the syntax element chroma_format_idc in JVET-T2001. In addition, Table 18 shows how the derived variables SubWidthC and SubHeightC are specified according to the chroma format. SubWidthC and SubHeightC are used, for example, for deblocking. With respect to Table 18, JVET-T2001 provides the following provisions: In monochrome sampling, there is only one sample array, which is nominally considered to be the brightness array. In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array. In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luma array. In 4:4:4 sampling, each of the two chroma arrays has the same height and the same width of the luma array. chroma_format_idc Chroma format SubWidthC SubHeightC 0 Monochrome 1 1 1 4:2:0 2 2 2 4:2:2 2 1 3 4:4:4 1 1 Table 18

[0115] In one example, according to the techniques herein, instead of signaling syntax elements nnpfc_out_sub_width_c_flag and nnpfc_out_sub_height_c_flag, a single syntax element may be signaled as a 1-bit flag specifying output chrominance information. This saves 1 bit when nnpfc_purpose is equal to 2 or 4. Table 19 shows an example of the relevant syntax of a neural network post-filter characteristics SEI message set according to the techniques herein. Table 19

[0116] With respect to Table 19, in one example, the semantics may be based on the semantics provided above and the following semantics: nnpfc_out_sub_c_flag equal to 1 specifies that outSubWidthC is equal to 1 and outSubHeightC is equal to 1 (ie, the output chroma format is 4:4:4). nnpfc_out_sub_c_flag equal to 0 specifies that outSubWidthC is equal to 2 and outSubHeightC is equal to 1 (ie, the output chroma format is 4:2:2). When nnpfc_out_sub_c_flag is not present, outSubWidthC is inferred to be equal to InpSubWidthC and outSubHeightC is inferred to be equal to InpSubHeightC. If InpSubWidthC is equal to 2 and InpSubHeightC is equal to 1, then nnpfc_out_sub_c_flag shall not be equal to 0.

[0117] In one example, the semantics may be based on the following description: nnpfc_out_sub_c_flag equal to 1 specifies that outSubWidthC is set equal to 1 and outSubHeightC is set equal to 1 (ie, the output chroma format is 4:4:4). nnpfc_out_sub_c_flag equal to 0 specifies that outSubWidthC is set equal to 2 and outSubHeightC is set equal to 1 (ie, the output chroma format is 4:2:2). When nnpfc_out_sub_c_flag is not present, outSubWidthC is inferred to be equal to InpSubWidthC and outSubHeightC is inferred to be equal to InpSubHeightC. If InpSubWidthC is equal to 2 and InpSubHeightC is equal to 1, then nnpfc_out_sub_c_flag shall not be equal to 0. In another example, the words "set to be equal to" above may be changed to "inferred to be equal to" or similar words.

[0118] It should be noted that in the semantics of nnpfc_out_sub_c_flag, the last constraint does not allow the case where the input chroma format is 4:2:2 and nnpfc_purpose indicates upsampling (i.e., nnpfc_purpose is equal to 2 or 4) and requires the output chroma format to be the same as the input chroma format of 4:2:2. According to the techniques herein, in some examples, the last constraint may be one of the following: If InpSubWidthC is equal to 2 and InpSubHeightC is equal to 1 (ie, sps_chroma_format_idc is equal to 1, ie, for 4:2:2), and nnpfc_purpose is equal to 2 or 4, then nnpfc_out_sub_c_flag shall not be equal to 0. or If InpSubWidthC is equal to 2 and InpSubHeightC is equal to 1 (ie, sps_chroma_formt_lidc is equal to 1, ie, for 4:2:2), and nnpfc_out_sub_c_flag is signaled, then nnpfc_out_sub_c_flag shall not be equal to 0. If InpSubWidthC is equal to 2 and InpSubHeightC is equal to 1, and nnpfc_purpose is equal to 2 or 4, then nnpfc_out_sub_c_flag shall be equal to 1. If InpSubWidthC is equal to 2 and InpSubHeightC is equal to 1, and nnpfc_out_sub_c_flag is signaled, then nnpfc_out_sub_c_flag shall be equal to 1. or If InpSubHeightC is equal to 1 and nnpfc_purpose is equal to 2 or 4, then nnpfc_out_sub_c_flag shall be equal to 1. If InpSubHeightC is equal to 1 (ie, sps_chroma_format_idc is equal to 1, ie, for 4:2:2), and nnpfc_out_sub_c_flag is signaled, then nnpfc_out_sub_c_flag shall not be equal to 0,

[0119] or If InpSubHeightC is equal to 1 and nnpfc_purpose is equal to 2 or 4, then nnpfc_out_sub_c_flag shall be equal to 1. If InpSubHeightC is equal to 1, and nnpfc_out_sub_c_flag is signaled, then nnpfc_out_sub_c_flag shall be equal to 1. or If InpSubHeightC is equal to 1, nnpfc_out_sub_c_flag shall not be equal to 0. If InpSubHeightC is equal to 1, then nnpfc_out_sub_c_flag shall be equal to 1.

[0120] In one example, according to the techniques herein, the semantics of nnpfc_out_sub_c_flag may be based on the following: nnpfc_out_sub_c_flag is used to derive the output chrominance information variables outSubWidthC, outSubHeightC as follows. When nnpfc_out_sub_c_flag is not present, outSubWidthC is inferred to be equal to InpSubWidthC and outSubHeightC is inferred to be equal to InpSubHeightC. If InpSubWidthC is equal to 2 and InpSubHeightC is equal to 1, then nnpfc_out_sub_c_flag shall not be equal to 0.

[0121] In one example, according to the techniques herein, if InpSubWidthC is equal to 2 and InpSubHeightC is equal to 1 and outSubWidthC and outSubHeightC are inferred instead, then a single bit flag is not signaled. Therefore, in this case (when nnpfc_purpose is equal to 2 or 4), a single bit flag is signaled when InpSubWidthC is equal to 2 and InpSubHeightC is equal to 2. This provides further bit savings. Table 20 shows an example of the relevant syntax of a neural network post filter characteristics SEI message set according to the techniques herein. Table 20

[0122] With respect to Table 20, in one example, the semantics may be based on the semantics provided above and the following semantics: nnpfc_out_sub_c_flag equal to 1 specifies that outSubWidthC is equal to 1 and outSubHeightC is equal to 1 (ie, the output chroma format is 4:4:4). nnpfc_out_sub_c_flag equal to 0 specifies that outSubWidthC is equal to 2 and outSubHeightC is equal to 1 (ie, the output chroma format is 4:2:2). When nnpfc_out_sub_c_flag does not exist, - infer outSubWidthC equal to 1 and infer outSubHeightC equal to 1, Otherwise, outSubWidthC is inferred to be equal to InpSubWidthC and outSubHeightC is inferred to be equal to InpSubHeightC. or nnpfc_out_sub_c_flag equal to 1 specifies that outSubWidthC is equal to 1 and outSubHeightC is equal to 1 (ie, the output chroma format is 4:4:4). nnpfc_out_sub_c_flag equal to 0 specifies that outSubWidthC is equal to 2 and outSubHeightC is equal to 1 (ie, the output chroma format is 4:2:2). When nnpfc_out_sub_c_flag is not present, outSubWidthC is inferred to be equal to 1 and outSubHeightC is inferred to be equal to 1. In another example, the word "equal to" above may be changed to "set to be equal to" or "inferred to be equal to" or similar words.

[0123] As provided in Table 6 above, in JVET-Z2006, it is allowed to signal nnpfc_purpose equal to 2 or 4 to indicate upsampling of the chroma format, even if the input is monochrome or 4:4:4. It can be considered that when the input chroma format is monochrome or 4:4:4, signaling chroma upsampling should not be allowed. In one example, according to the techniques of this article, constraints can be added to prevent this. That is, in one example, according to the techniques of this article, the semantics of the syntax element nnpfc_purpose can be as follows: nnpfc_purpose indicates the purpose of the post-processing filter as specified in Table 8. The value of nnpfc_purpose shall be between 0 and 2. 32 The nnpfc_purpose values ​​not appearing in Table 8 are reserved for future ITU-T|ISO / IEC specifications and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​for nnpfc_purpose. When InpSubWidthC is equal to 1 and InpSubHeightC is equal to 1, nnpfc_purpose shall not be equal to 2 or 4.

[0124] Thus, video encoder 500 represents an example of a device configured to: send a signal to notify a neural network post-filter characteristics message, and send a signal to notify a syntax element that identifies the format and associated information of the neural network corresponding to the neural network post-filter characteristics message.

[0125] Reference again Figure 1 , interface 108 may include any device configured to receive data generated by data encapsulator 107 and transmit and / or store the data to a communication medium. Interface 108 may include a network interface card such as an Ethernet card, and may include an optical transceiver, a radio frequency transceiver, or any other type of device that can send and / or receive information. In addition, interface 108 may include a computer system interface that enables files to be stored on a storage device. For example, interface 108 may include support for peripheral component interconnect (PCI) and high-speed peripheral component interconnect (PCIe) bus protocols, proprietary bus protocols, universal serial bus (USB) protocols, I 2 C's chipset or any other logical and physical structure that can be used to interconnect peer devices.

[0126] Reference again Figure 1 , target device 120 includes interface 122, data decapsulator 123, video decoder 124, and display 126. Interface 122 may include any device configured to receive data from a communication medium. Interface 122 may include a network interface card such as an Ethernet card, and may include an optical transceiver, a radio frequency transceiver, or any other type of device that can receive and / or send information. In addition, interface 122 may include a computer system interface that allows a compatible video bitstream to be retrieved from a storage device. For example, interface 122 may include a computer system that supports PCI and PCIe bus protocols, proprietary bus protocols, USB protocols, I 2 The data decapsulator 123 may be configured to receive and parse any of the example grammatical structures described herein.

[0127] Video decoder 124 may include any device configured to receive a bitstream (e.g., sub-bitstream extraction) and / or acceptable variants thereof and reproduce video data therefrom. Display 126 may include any device configured to display video data. Display 126 may include one of a variety of display devices such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display. Display 126 may include a high definition display or an ultra high definition display. It should be noted that although in Figure 1 In the example shown, the video decoder 124 is described as outputting data to a display 126, but the video decoder 124 can be configured to output video data to various types of devices and / or subcomponents thereof. For example, the video decoder 124 can be configured to output video data to any communication medium, as described herein.

[0128] Figure 6 600 is a block diagram illustrating an example of a video decoder that may be configured to decode video data in accordance with one or more techniques of this disclosure (e.g., a decoding process for reference picture list construction described above). In one example, the video decoder 600 may be configured to decode transform data and reconstruct residual data from transform coefficients based on the decoded transform data. The video decoder 600 may be configured to perform intra-frame prediction decoding and inter-frame prediction decoding, and may therefore be referred to as a hybrid decoder. The video decoder 600 may be configured to parse any combination of the syntax elements described above in Tables 1 to 20. The video decoder 600 may decode a picture based on or according to the above process and also based on the parsed values ​​in Tables 1 to 20.

[0129] exist Figure 6 In the example shown, the video decoder 600 includes an entropy decoding unit 602, an inverse quantization unit 604, an inverse transform coefficient processing unit 606, an intra-frame prediction processing unit 608, an inter-frame prediction processing unit 610, a summer 612, a post-filter unit 614, and a reference buffer 616. The video decoder 600 can be configured to decode video data in a manner consistent with a video encoding system. It should be noted that although the exemplary video decoder 600 shown has different functional blocks, such illustrations are intended for descriptive purposes and do not limit the video decoder 600 and / or its subcomponents to a specific hardware or software architecture. The functionality of the video decoder 600 can be implemented using any combination of hardware, firmware, and / or software implementations.

[0130] like Figure 6As shown, the entropy decoding unit 602 receives an entropy-encoded bitstream. The entropy decoding unit 602 may be configured to decode syntax elements and quantized coefficients from the bitstream according to a process that is inverse to the entropy encoding process. The entropy decoding unit 602 may be configured to perform entropy decoding according to any of the entropy encoding techniques described above. The entropy decoding unit 602 may determine the values ​​of syntax elements in the encoded bitstream in a manner consistent with the video coding standard. Figure 6 As shown, the entropy decoding unit 602 can determine the quantization parameter, the quantization coefficient value, the transformation data and the prediction data from the bit stream. Figure 6 As shown, the inverse quantization unit 604 and the inverse transform coefficient processing unit 606 receive the quantized coefficient values ​​from the entropy decoding unit 602 and output reconstructed residual data.

[0131] See again Figure 6 , the reconstructed residual data may be provided to the summer 612. The summer 612 may add the reconstructed residual data to the predicted video block and generate the reconstructed video data. The predicted video block may be determined according to a predictive video technique, namely intra prediction and inter prediction. The intra prediction processing unit 608 may be configured to receive an intra prediction syntax element and retrieve the predicted video block from a reference buffer 616. The reference buffer 616 may include a memory device configured to store one or more frames of video data. The intra prediction syntax element may identify an intra prediction mode, such as the intra prediction mode described above. The inter prediction processing unit 610 may receive the inter prediction syntax element and generate a motion vector to identify a predicted block in one or more reference frames stored in the reference buffer 616. The inter prediction processing unit 610 may generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter for motion estimation with sub-pixel precision may be included in the syntax element. The inter prediction processing unit 610 may use an interpolation filter to calculate an interpolated value for a sub-integer pixel of a reference block. The post filter unit 614 may be configured to perform filtering on the reconstructed video data. For example, the post filter unit 614 may be configured to perform deblocking and / or sample adaptive offset (SAO) filtering, for example based on parameters specified in the bitstream. Furthermore, it should be noted that in some examples, the post filter unit 614 may be configured to perform dedicated arbitrary filtering (e.g., visual enhancement, such as mosquito noise removal). Figure 6 As shown, the video decoder 600 may output a reconstructed video block. Thus, the video decoder 600 represents an example of a device configured to: receive a neural network post filter characteristics message, parse syntax elements identifying a format and associated information of a neural network corresponding to the neural network post filter characteristics message, and determine one or more neural network parameters based on the parsed syntax elements.

[0132] In one or more examples, the functions may be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media such as data storage media, or propagation media including any medium that facilitates the transmission of a computer program from one place to another, for example, according to a communication protocol. Thus, computer-readable media may generally correspond to: (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0133] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio and microwave are used to transmit instructions from a website, server or other remote source, then the coaxial cable, optical fiber cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other transient media, but are directed to non-transient tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical optical disks, digital versatile disks (DVDs), floppy disks and Blu-ray disks, wherein disks usually copy data magnetically, while optical disks use lasers to copy data optically. The above combination should also be included in the scope of computer-readable media.

[0134] Instructions may be executed by one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Moreover, these techniques may be implemented entirely in one or more circuits or logic elements.

[0135] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in a codec hardware unit, or provided by a collection of interoperating hardware units including one or more processors as described above, in combination with appropriate software and / or firmware.

[0136] In addition, each functional block or various features of the base station equipment and terminal equipment used in each of the above-mentioned embodiments can be realized or executed by a circuit (usually an integrated circuit or multiple integrated circuits). The circuit designed to perform the functions described in this specification may include a general-purpose processor, a digital signal processor (DSP), a dedicated or general-purpose integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices or discrete hardware components or a combination thereof. The general-purpose processor can be a microprocessor, or alternatively, the processor can be a conventional processor, a controller, a microcontroller or a state machine. The general-purpose processor or each of the above-mentioned circuits can be configured by a digital circuit, or can be configured by an analog circuit. In addition, when the technology of making an integrated circuit that replaces the current integrated circuit appears due to the progress of semiconductor technology, the integrated circuit produced by the technology can also be used.

[0137] Various examples have been described. These and other examples are within the scope of the following claims. <Cross Reference>

[0138] This nonprovisional application claims priority under 35 U.S.C. 119 to Provisional Application No. 63 / 356,340 filed on June 28, 2022 and Provisional Application No. 63 / 358,656 filed on July 6, 2022, the entire contents of which are hereby incorporated by reference.

Claims

1. A method for applying a neural network filter to reconstruct a video, the method comprising: receiving a neural network post-filter characteristics message for a specified neural network; Parsing a first syntax element identifying a neural network post filter (NNPF); parsing a second syntax element specifying that the NNPF associated with the first syntax element is a neural network identified by a uniform resource identifier; Parsing a third syntax element including the uniform resource identifier; as well as A fourth syntax element is parsed that identifies a format of the neural network data specified by the third syntax element.

2. A device comprising one or more processors, wherein the one or more processors are configured to: receiving a neural network post-filter characteristics message for a specified neural network; Parsing a first syntax element identifying a neural network post filter (NNPF); parsing a second syntax element specifying that the NNPF associated with the first syntax element is a neural network identified by a uniform resource identifier; Parsing a third syntax element including the uniform resource identifier; as well as A fourth syntax element is parsed that identifies a format of the neural network data specified by the third syntax element.

3. The device of claim 2, wherein the device comprises a video decoder.

Images