Video encoding / decoding method, recording medium storing a bitstream, and method for transmitting a bitstream.

Abstract

A video encoding / decoding method, a bitstream transmission method, and a computer-readable recording medium for storing a bitstream are provided. The video decoding method according to this disclosure may be a video decoding method in which an NNPF (neural-network post-filter) SEI (supplemental enhancement information) message is obtained, wherein the NNPF SEI message includes an NNPFC (neural-network post-filter characteristics) SEI message and an NNPFA (neural-network post-filter activation) SEI message, and the method includes the steps of determining a neural network that can be used as a post-processing filter based on the NNPFC SEI message; and determining whether or not to activate a target NNPF that can be applied to the current picture based on the NNPFA SEI message, wherein the target NNPF is used to perform post-processing filtering on the current picture and subsequent pictures in the current layer in the output order until a picture in the current layer associated with an NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order.

Description

[Technical Field]

[0001] This disclosure relates to a video encoding / decoding method, a recording medium storing a bitstream, and a method for transmitting a bitstream, and more particularly to a method for determining whether or not to perform persistence cancellation of a neural network post-filter. [Background technology]

[0002] Recently, demand for high-resolution, high-quality video / image data, such as 4K or 8K or higher UHD (Ultra High Definition) video / image data, has been increasing in various fields. As video / image data becomes higher resolution and higher quality, the amount of information or bits transmitted increases relative to existing video / image data. Therefore, when transmitting video / image data using existing wired or wireless broadband lines, or storing video / image data using existing storage media, transmission and storage costs increase.

[0003] This necessitates highly efficient image compression technology to effectively transmit, store, and reproduce high-resolution, high-quality image information. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] This disclosure aims to provide a video encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0005] Furthermore, this disclosure aims to provide a method for clearly distinguishing which NNPF persistence to revoke when two or more NNPF activations with the same identification information exist.

[0006] Furthermore, this disclosure aims to provide a method for preventing two or more NNPF activations having the same identification information from occurring simultaneously.

[0007] Furthermore, this disclosure aims to provide a non-temporary computer-readable recording medium for storing a bitstream generated by the video encoding method relating to this disclosure.

[0008] Furthermore, this disclosure aims to provide a method for transmitting a bitstream generated by the video encoding method relating to this disclosure.

[0009] Furthermore, this disclosure aims to provide a non-temporary computer-readable recording medium that stores a bitstream received and decoded by the video decoding device relating to this disclosure and used for video restoration.

[0010] The technical challenges that this disclosure seeks to address are not limited to those mentioned above, and any other technical challenges not mentioned can be clearly understood by a person with ordinary skill in the art to which this disclosure pertains from the following description. [Means for solving the problem]

[0011] A video decoding method according to one aspect of the present disclosure is a video decoding method performed by a video decoding device, wherein an NNPF (neural-network post-filter) SEI (supplemental enhancement information) message is obtained, wherein the NNPF SEI message includes (provides; constitutes; constructs; sets; inclusions; contains; includes) an NNPFC (neural-network post-filter characteristics) SEI message and an NNPFA (neural-network post-filter activation) SEI message, and the method includes the steps of determining a neural network that can be used as a post-processing filter based on the NNPFC SEI message; and determining whether or not to activate a target NNPF that can be applied to the current picture based on the NNPFA SEI message, wherein the target NNPF is used to perform post-processing filtering on the current picture and subsequent pictures in the current layer in the output order until a picture in the current layer associated with an NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order.

[0012] Other aspects of the present disclosure include a video encoding method performed by a video encoding device, comprising the steps of: encoding a neural network that can be used as a post-processing filter into an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message; and encoding whether or not a target NNPF that can be applied to the current picture is activated into an NNPFA (neural-network post-filter activation) SEI message, wherein the target NNPF is used to perform post-processing filtering on the current picture and subsequent pictures in the current layer in the output order until a picture in the current layer associated with an NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order.

[0013] Computer-readable recording media relating to other aspects of this disclosure can store bitstreams generated by the video encoding method or apparatus of this disclosure.

[0014] A transmission method relating to yet another aspect of the present disclosure may transmit a bitstream generated by the video encoding method or apparatus of the present disclosure.

[0015] The features of this disclosure, as briefly summarized above, are merely illustrative aspects of the detailed description of this disclosure described below and do not limit the scope of this disclosure. [Effects of the Invention]

[0016] According to this disclosure, a video encoding / decoding method and apparatus with improved encoding / decoding efficiency may be provided.

[0017] Also, according to the present disclosure, when there are two or more NNPF activations having the same identification information, the NNPF that cancels the persistence can be clearly identified.

[0018] Also, according to the present disclosure, it is possible to prevent two or more NNPF activations having the same identification information from being made simultaneously.

[0019] Also, according to the present disclosure, a non-transitory computer-readable recording medium for storing a bitstream generated by the video encoding method according to the present disclosure may be provided.

[0020] Also, according to the present disclosure, a non-transitory computer-readable recording medium for storing a bitstream that is received, decoded, and used for video restoration by the video decoding apparatus according to the present disclosure may be provided.

[0021] Also, according to the present disclosure, a method for transmitting a bitstream generated by a video encoding method may be provided.

[0022] The effects that can be obtained by the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those having ordinary knowledge in the technical field to which the present disclosure belongs from the following description.

Brief Description of Drawings

[0023] [Figure 1] A drawing schematically showing a video coding system to which an embodiment according to the present disclosure can be applied. [Figure 2] A drawing schematically showing a video encoding apparatus to which an embodiment according to the present disclosure can be applied. [Figure 3] A drawing schematically showing a video decoding apparatus to which an embodiment according to the present disclosure can be applied. [Figure 4] Exemplarily shows a hierarchical structure for coded video / video. [Figure 5]This is a diagram illustrating the interleaved method for guiding luma channels. [Figure 6] This is a diagram illustrating the video encoding method of a video encoding device. [Figure 7] This is a diagram illustrating the video decoding method of a video decoding device. [Figure 8] This diagram illustrates a video decoding method for an embodiment that determines whether or not the target NNPF is activated. [Figure 9-11] This diagram illustrates various examples of whether or not the persistence of the target NNPF is revoked. [Figure 12] This drawing illustrates a content streaming system to which the embodiments of this disclosure may be applied. [Modes for carrying out the invention]

[0024] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings, so that they can be easily implemented by a person with ordinary skill in the art to which the present disclosure pertains. However, the present disclosure can be implemented in a variety of different forms and is not limited to the embodiments described herein.

[0025] In describing embodiments of this disclosure, if it is determined that a specific description of a known configuration or function would obscure the gist of this disclosure, such detailed description will be omitted. In the drawings, parts unrelated to the description of this disclosure will be omitted, and similar parts will be denoted by the same reference numerals.

[0026] In this disclosure, when one component is described as being “connected,” “joined,” or “linked” to another component, this can include not only direct connections but also indirect connections where another component exists between them. Furthermore, when one component is described as “containing” or “having” another component, this means, unless otherwise stated to the contrary, that it may include another component rather than excluding it.

[0027] In this disclosure, terms such as "first," "second," etc., are used solely for the purpose of distinguishing one component from another, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be called a second component in another embodiment, and similarly, a second component in one embodiment may be called a first component in another embodiment.

[0028] In this disclosure, components that are distinguished from each other are used to clearly describe their respective characteristics and do not necessarily mean that the components are separate. In other words, multiple components may be integrated to constitute a single hardware or software unit, or a single component may be distributed to constitute multiple hardware or software units. Therefore, such integrated or distributed embodiments are also included in the scope of this disclosure, without needing to be specifically mentioned.

[0029] In this disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments consisting of a subset of the components described in one embodiment are also included in the scope of this disclosure. Furthermore, embodiments that include additional components in addition to the components described in various embodiments are also included in the scope of this disclosure.

[0030] This disclosure relates to the encoding and decoding of images, and the terms used in this disclosure may have their ordinary meanings in the art to which this disclosure pertains, unless otherwise defined herein.

[0031] This disclosure presents various embodiments relating to video / video coding, and unless otherwise noted, these embodiments may be implemented in combination with each other.

[0032] Unless otherwise defined herein, terms used in this disclosure may have their ordinary meanings as they are common in the art to which this disclosure pertains.

[0033] In this disclosure, "picture" generally means a unit representing any one image within a specific time period, and a slice / tile is an encoded unit that constitutes part of a picture, and a single picture can consist of one or more slices / tiles. A slice / tile can also contain one or more CTUs (coding tree units). A single picture may consist of one or more tile groups. A single tile group may contain one or more tiles. A brick can represent a square area of ​​a CTU row of tiles within a picture. In this document, tile groups and slices can be combined. For example, in this document, a tile group / tile group header may be referred to as a slice / slice head.

[0034] In this disclosure, “pixel” or “pel” may mean the smallest unit that constitutes a picture (or image). The term “sample” may also be used as a counterpart to pixel. A sample may generally represent a pixel or a pixel value, or it may represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component.

[0035] In this disclosure, “unit” can refer to a basic unit of image processing. A unit may include at least one of a specific region of a picture and information associated with that region. A unit may be used interchangeably with terms such as “sample array,” “block,” or “area,” as it may be used. Generally, an M×N block may include a set (or array) of samples (or sample arrays) or transform coefficients consisting of M columns and N rows.

[0036] In this disclosure, “current block” can mean any one of the following: “current coding block,” “current coding unit,” “block to encode,” “block to decode,” or “block to process.” If prediction is performed, “current block” can mean “current prediction block” or “block to predict.” If transformation (inverse transformation) / quantization (inverse quantization) is performed, “current block” can mean “current transformation block” or “block to transform.” If filtering is performed, “current block” can mean “block to filter.”

[0037] Furthermore, in this disclosure, "current block" may mean the block containing all luma component blocks and chroma component blocks, or the "luma block of the current block," unless there is an explicit mention of a chroma block. The chroma block of the current block may be expressed with an explicit mention of a chroma block, such as "chroma block" or "current chroma block."

[0038] In this disclosure, " / " and "," may be interpreted as "and / or." For example, "A / B" and "A, B" may be interpreted as "A and / or B." Also, "A / B / C" and "A, B, C" may mean "at least one of A, B and / or C."

[0039] In this disclosure, “or” may be interpreted as “and / or.” For example, “A or B” may mean 1) “A” only, 2) “B” only, or 3) “A and B.” Alternatively, in this disclosure, “or” may mean “additionally or alternatively.”

[0040] Overview of the video coding system Figure 1 shows the video coding system according to this disclosure.

[0041] A video coding system according to one embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 can transmit encoded video and / or image information or data to the decoding device 20 via a digital storage medium or network in file or streaming format.

[0042] An encoding device 10 according to one embodiment may include a video source generation unit 11, an encoding unit 12, and a transmission unit 13. A decoding device 20 according to one embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 may be called a video / image encoding unit, and the decoding unit 22 may be called a video / image decoding unit. The transmission unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may also include a display unit, which may be configured as a separate device or external component.

[0043] The video source generation unit 11 can acquire video / images through processes such as video / image capture, synthesis, or generation. The video source generation unit 11 may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, or a video / image archive containing previously captured video / images. The video / image generation device may include, for example, a computer, tablet, and smartphone, and may generate video / images (electronically). For example, virtual video / images may be generated via a computer, in which case the video / image capture process may be replaced by a process in which the relevant data is generated.

[0044] The encoding unit 12 can encode the input video / image. The encoding unit 12 can perform a series of steps such as prediction, transformation, and quantization for compression and encoding efficiency. The encoding unit 12 can output the encoded data (encoded video / image information) in bitstream format.

[0045] The transmission unit 13 can transmit encoded video / image information or data, output in bitstream format, to the receiving unit 21 of the decoding device 20 via a digital storage medium or network in file or streaming format. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray®, HDD, and SSD. The transmission unit 13 may include elements for generating media files via a predetermined file format and elements for transmission via a broadcast / communication network. The receiving unit 21 can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit 22.

[0046] The decoding unit 22 can decode the video / image by performing a series of steps such as inverse quantization, inverse transform, and prediction, corresponding to the operation of the encoding unit 12.

[0047] The rendering unit 23 can render the decoded video / image. The rendered video / image can be displayed via the display unit.

[0048] Video Encoding Device Overview Figure 2 is a schematic diagram illustrating the configuration of a video / image encoding device to which the embodiments described in this document may be applied.

[0049] As shown in Figure 2, the encoding device 200 can be configured to include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-predictor 221 and an intra-predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be called a reconstructor or a reconstructed block generator. The aforementioned image segmentation unit 210, prediction unit 220, residual processing unit 230, entropy encoding unit 240, addition unit 250, and filtering unit 260 can be configured by one or more hardware components (e.g., an encoder chipset or processor) depending on the embodiment. Furthermore, the memory 270 may include a DPB (decoded picture buffer) and may be configured by a digital recording medium. The hardware components may also further include the memory 270 as an internal / external component.

[0050] The image splitting unit 210 can split an input image (or picture, frame) input to the encoding device 200 into one or more processing units. For example, one of these processing units may be called a coding unit (CU). In this case, a coding unit can be recursively split from a coding tree unit (CTU) or the largest coding unit (LCU) using a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, one coding unit can be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, followed by the binary-tree structure and / or the ternary structure. Alternatively, the binary-tree structure may be applied first. The coding procedure described in this document can be executed based on the final coding unit that cannot be further split. In this case, based on coding efficiency according to image characteristics, the largest coding unit can be immediately used as the final coding unit, or, if necessary, the coding unit can be recursively divided into lower-depth coding units so that the optimally sized coding unit is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further comprise a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit can each be separated or partitioned from the final coding unit described above.The prediction unit is a unit of sample prediction, and the conversion unit is a unit that derives a conversion coefficient and / or a unit that derives a residual signal from the conversion coefficient.

[0051] The encoding device 200 can generate a residual signal (residual block, residual sample array) by subtracting the prediction signal (predicted block, predicted sample array) output from the inter-prediction unit 221 or intra-prediction unit 222 from the input image signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as shown in the figure, the unit that subtracts the prediction signal (predicted block, predicted sample array) from the input image signal (original block, original sample array) within the encoder 200 can be called the subtraction unit 231. The prediction unit can perform a prediction for the block to be processed (hereinafter referred to as the current block) and generate a predicted block that includes the predicted sample for the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied on a current block or CU basis. The prediction unit can generate various prediction-related information, such as prediction mode information, and transmit it to the entropy encoding unit 240, as will be described later in the explanation of each prediction mode. The prediction information can be encoded by the entropy encoding unit 240 and output in bitstream format.

[0052] The intra-prediction unit 222 can predict the current block by referring to a sample in the current picture. The referenced sample can be located in the vicinity (neighbor) of the current block or at a distance, depending on the prediction mode. In intra-prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC mode and Planar mode. Directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is merely an example, and more or fewer directional prediction modes can be used depending on the settings. The intra-prediction unit 222 can also determine the prediction mode to be applied to the current block using the prediction modes applied to adjacent blocks.

[0053] The interprediction unit 221 can derive a predicted block relative to the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, in order to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between adjacent blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, adjacent blocks may include spatially adjacent blocks existing in the current picture and temporally adjacent blocks existing in the reference picture. The reference picture containing the reference block and the reference picture containing the temporally adjacent block may be the same or different. The temporally adjacent block may be called a collocated reference block, colCU, etc., and the reference picture containing the temporally adjacent block may be called a collocated picture (colPic). For example, the interpretation unit 221 can construct a motion information candidate list based on adjacent blocks and generate information indicating which candidates are used to derive the motion vector and / or reference picture index of the current block. Interpretation can be performed based on various prediction modes; for example, in skip mode and merge mode, the interpretation unit 221 can use the motion information of adjacent blocks as the motion information of the current block. In skip mode, unlike merge mode, a residual signal may not be transmitted.In motion vector prediction (MVP) mode, the motion vector of an adjacent block is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0054] The prediction unit 220 can generate prediction signals based on various prediction methods described later. For example, the prediction unit can apply intra-prediction or inter-prediction for a prediction of a single block, and can also apply intra-prediction and inter-prediction simultaneously. This can be called combined inter and intra prediction (CIIP). The prediction unit can also be based on an intra-block copy (IBC) prediction mode or a palette mode for predictions of blocks. The IBC prediction mode or palette mode can be used for content image / video coding such as in games, for example, as in SCC (screen content coding). IBC basically performs predictions within the current picture, but can be performed similarly to inter-prediction in that it derives reference blocks within the current picture. That is, IBC can utilize at least one of the inter-prediction techniques described in this document.

[0055] The predicted signal generated via the prediction unit 220 may be used to generate a reconstructed signal or to generate a residual signal. The subtraction unit 231 subtracts the predicted signal (predicted block, predicted sample array) output from the prediction unit 220 from the input video signal (original block, original sample array) to generate a residual signal (residual block, residual sample array). The generated residual signal is transmitted to the conversion unit 232.

[0056] The transformation unit 232 can generate transformation coefficients by applying transformation techniques to the residual signal. For example, the transformation technique may include at least one of the following: DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is represented by this graph. CNT refers to a transformation obtained by generating a prediction signal using all previously reconstructed pixels. Furthermore, the transformation process can be applied to pixel blocks of the same size and are square, or to non-square, variable-sized blocks.

[0057] The quantization unit 233 quantizes the conversion coefficients and transmits them to the entropy encoding unit 240, which can encode the quantized signal (information about the quantized conversion coefficients) and output it as a bitstream. The information about the quantized conversion coefficients can be called residual information. The quantization unit 233 can rearrange the block-shaped quantized conversion coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized conversion coefficients based on the one-dimensional vector form of the quantized conversion coefficients.

[0058] The entropy encoding unit 240 can perform various encoding methods, such as exponential Golomb, CAVLC (context-adaptive variable length coding), and CABAC (context-adaptive binary arithmetic coding). In addition to the quantized conversion coefficients, the entropy encoding unit 240 can also encode information necessary for video / image reconstruction (e.g., the values ​​of syntax elements) together with or separately. The encoded information (e.g., encoded video / image information) can be transmitted or stored in bitstream form in units of NAL (network abstraction layer) units. The video / image information may further include information about various parameter sets, such as the adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information. In this document, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in video / image information. The video / image information may be encoded via the encoding procedure described above and included in the bitstream.

[0059] The bitstream can be transmitted over a network or stored on a digital recording medium. Here, the network may include broadcast networks and / or communication networks, and the digital recording medium may include various recording media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit 240 can be transmitted by a transmitting unit (not shown) and / or stored by a storage unit (not shown) which are configured as internal / external elements of the encoding device 200, or the transmitting unit may be included in the entropy encoding unit 240.

[0060] The quantized conversion coefficients output from the quantization unit 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying inverse quantization and inverse transformation to the quantized conversion coefficients via the inverse quantization unit 234 and the inverse transformation unit 235.

[0061] The adder 250 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the predicted signal output from the inter-prediction unit 221 or the intra-prediction unit 222. If there is no residual for the block to be processed, as in the case of skip mode, the predicted block can be used as the reconstructed block. The adder 250 can be called the reconstructor unit or reconstructed block generator. The generated reconstructed signal can be used for intra-prediction of the next block to be processed in the current picture, or, as described later, for inter-prediction of the next picture after filtering.

[0062] The filtering unit 260 can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 260 can apply various filtering methods to the restored picture to generate a modified restored picture, and the modified restored picture can be stored in the memory 270, specifically in the DPB of the memory 270. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter. The filtering unit 260 can generate various filtering-related information and transmit it to the entropy encoding unit 240, as will be described later in the explanation of each filtering method. The filtering-related information can be encoded by the entropy encoding unit 240 and output in bitstream format.

[0063] The corrected restored picture sent to memory 270 can be used as a reference picture in the interpretation unit 221. When interpretation is applied via this, the encoding device can avoid prediction mismatches between the encoding device 200 and the decoding device 300, and can also improve encoding efficiency.

[0064] Memory 270DPB can store the corrected restored picture for use as a reference picture in the inter-prediction unit 221. Memory 270 can store motion information of blocks from which motion information has been derived (or encoded) in the current picture and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 221 for use as motion information of spatially adjacent blocks or motion information of temporally adjacent blocks. Memory 270 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 222.

[0065] Overview of the image decoding device Figure 3 is a schematic diagram illustrating the configuration of a video / image decoding device to which the embodiments described in this document may be applied.

[0066] Referring to Figure 3, the decoding device 300 can be configured to include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an intra-predictor 331 and an inter-predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. The aforementioned entropy decoder 310, residual processor 320, predictor 330, adder 340, and filtering device 350 can be configured by a single hardware component (e.g., a decoder chipset or processor) depending on the embodiment. The memory 360 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware components may further include memory 360 as an internal / external component. When a bitstream containing video / image information is input, the decoding device 300 can reconstruct the image corresponding to the process by which the video / image information was processed in the encoding device of Figure 2. For example, the decoding device 300 can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device 300 can perform decoding using the processing units applied in the encoding device. Thus, the decoding processing units are, for example, coding units, which can be divided from a coding tree unit or a maximum coding unit according to a quad-tree structure, a binary tree structure, and / or a terminally tree structure. One or more conversion units can be derived from the coding unit. The reconstructed image signal decoded and output via the decoding device 300 can then be reproduced via a playback device.

[0067] The decoding device 300 can receive the signal output from the encoding device shown in Figure 2 in bitstream form, and the received signal can be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 can parse the bitstream to derive information necessary for image restoration (or picture restoration) (e.g., video / image information). The video / image information may further include information about various parameter sets, such as the adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information. The decoding device can further decode the picture based on the parameter set information and / or the general constraint information. The signaling / received information and / or syntax elements described later in this document can be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements necessary for image reconstruction, quantized values ​​of conversion coefficients related to residuals, etc. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using the information of the syntax element to be decoded, the decoded information of the surrounding and decoded blocks, or the symbol / bin information decoded in a previous step, predicts the probability of bin occurrence based on the determined context model, and performs arithmetic decoding of the bins to generate symbols corresponding to the values ​​of each syntax element. At this time, after determining the context model, the CABAC entropy decoding method can update the context model using the decoded symbol / bin information for the context model of the next symbol / bin.Of the information decoded by the entropy decoding unit 310, information related to prediction is provided to the prediction unit (inter-prediction unit 332 and intra-prediction unit 331), and the residual values ​​from which entropy decoding has been performed in the entropy decoding unit 310, i.e., quantized conversion coefficients and related parameter information, can be input to the residual processing unit 320. The residual processing unit 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). In addition, of the information decoded by the entropy decoding unit 310, information related to filtering can be provided to the filtering unit 350. On the other hand, a receiving unit (not shown) that receives signals output from the encoding device can be further configured as an internal / external element of the decoding device 300, or the receiving unit is a component of the entropy decoding unit 310.

[0068] On the other hand, the decoding device relating to this document may be called a video / image / picture decoding device, and the decoding device may also be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit 310, and the sample decoder may include at least one of the inverse quantization unit 321, inverse transformation unit 322, addition unit 340, filtering unit 350, memory 360, inter-prediction unit 332, and intra-prediction unit 331.

[0069] The inverse quantization unit 321 can inverse quantize the quantized transformation coefficients and output the transformation coefficients. The inverse quantization unit 321 can rearrange the quantized transformation coefficients in a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit 321 can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) to obtain the transformation coefficients.

[0070] In the inverse conversion unit 322, the conversion coefficients are inversely converted to obtain a residual signal (residual block, residual sample array).

[0071] The prediction unit can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit 310, the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block and can determine a specific intra / inter-prediction mode.

[0072] The prediction unit 320 can generate a prediction signal based on various prediction methods described later.

[0073] The intra-prediction unit 331 can predict the current block by referring to a sample in the current picture. The referenced sample can be located in the vicinity (neighbor) of the current block or at a distance from it, depending on the prediction mode. In intra-prediction, the prediction mode can include a plurality of non-directional modes and a plurality of directional modes. The intra-prediction unit 331 can also determine the prediction mode to be applied to the current block using the prediction modes applied to adjacent blocks.

[0074] The interprediction unit 332 can derive a predicted block for the current block based on a reference block (reference sample array) identified by motion vectors on a reference picture. In this case, in order to reduce the amount of motion information transmitted from the interprediction mode, motion information can be predicted in blocks, subblocks, or samples based on the correlation of motion information between adjacent blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, adjacent blocks may include spatially adjacent blocks that exist in the current picture and temporally adjacent blocks that exist in the reference picture. For example, the interprediction unit 332 can construct a motion information candidate list based on adjacent blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Interprediction can be performed based on various prediction modes, and the prediction information may include information indicating the mode of interprediction for the current block.

[0075] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the prediction unit (including the inter-prediction unit 332 and / or intra-prediction unit 331). If there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The adder 340 may be called the reconstruction unit or reconstructed block generation unit. The generated reconstructed signal can be used for intra-prediction of the next block to be processed in the current picture, and can be output after filtering as described later, or it can be used for inter-prediction of the next picture.

[0076] The filtering unit 350 can apply filtering to the restored signal to improve subjective / objective image quality. For example, the filtering unit 350 can apply various filtering methods to the restored picture to generate a modified restored picture, and can transmit the modified restored picture to the memory 360, specifically to the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter.

[0077] The (modified) restored picture stored in the DPB of memory 360 can be used as a reference picture by the inter-prediction unit 332. Memory 360 can store motion information of blocks from which motion information in the current picture has been derived (or decoded) and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 260 for use as motion information of spatially adjacent blocks or motion information of temporally adjacent blocks. Memory 360 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 331.

[0078] In this document, the embodiments described for the filtering unit 260, the inter-prediction unit 221, and the intra-prediction unit 222 of the encoding device 200 can also be applied identically or in a corresponding manner to the filtering unit 350, the inter-prediction unit 332, and the intra-prediction unit 331 of the decoding device 300, respectively.

[0079] Figure 4 illustrates the hierarchical structure for coded video / images.

[0080] Referring to Figure 4, coded video / images can be divided into the VCL (video coding layer), which handles the video / image decoding process and the video itself; a lower-level system that transmits and stores the coded information; and the NAL (network abstraction layer), which exists between the VCL and the lower-level system and is responsible for network adaptation functions.

[0081] For example, VCL can generate VCL data containing compressed image data (slice data), or it can generate a Picture Parameter Set (PPS), Sequence Parameter Set (SPS), Video Parameter Set (VPS), or a parameter set containing SEI (Supplemental Enhancement Information, SEI) messages additionally required during the video decoding process.

[0082] For example, in NAL, a NAL unit can be generated by adding header information (NAL unit header) to an RBSP (Raw Byte Sequence Payload) generated by VCL. In this case, the RBSP can refer to slice data, parameter sets, SEI messages, etc., generated by VCL. The NAL unit header can include NAL unit type information specified by the RBSP data contained in the NAL unit.

[0083] For example, as shown in Figure 4, NAL units can be classified into VCL NAL units and Non-VCL NAL units by the RBSP generated in VCL. VCL NAL units can represent NAL units containing information (slice data) for the image, while Non-VCL NAL units can represent NAL units containing information (parameter sets or SEI messages) necessary for image decoding.

[0084] The aforementioned VCL NAL units and Non-VCL NAL units can be transmitted over a network with header information attached according to the subsystem's data standard. For example, NAL units can be converted to predetermined standard data formats such as the H.266 / VVC file format, the Real-Time Transport Protocol (RTP), and the Transport Stream (TS), and transmitted over a variety of networks.

[0085] Furthermore, as mentioned above, the NAL unit type can be specified by the RBSP data structure contained within the NAL unit, and information regarding the NAL unit type can be stored in the NAL unit header and signaled.

[0086] For example, NAL units can be classified into VCL NAL unit types and Non-VCL NAL unit types depending on whether or not they contain information (slice data) related to the image. Furthermore, VCL NAL unit types can be classified by the characteristics and type of the picture contained within the VCL NAL unit, and Non-VCL NAL unit types can be classified by the type of parameter set.

[0087] The following is an example of a NAL unit type specified by the type of parameter set included in the Non-VCL NAL unit type.

[0088] APS (Adaptation Parameter Set) NAL Unit: Type for NAL units that include APS DPS (Decoding Parameter Set) NAL Unit: Type for NAL units that include DPS VPS (Video Parameter Set) NAL Unit: Type for NAL units that include VPS SPS (Sequence Parameter Set) NAL Unit: Type for NAL units that include SPS PPS (Picture Parameter Set) NAL Unit: Type for NAL units that include PPS The aforementioned NAL unit type may have syntax information for the NAL unit type, and this syntax information may be stored and signaled in the NAL unit header. For example, the syntax information may be nal_unit_type, and the NAL unit type may be specified in the nal_unit_type value.

[0089] On the other hand, a single picture can contain multiple slices, and a slice can contain a slice header and slice data. In this case, one picture header can be added to multiple slices (sets of slice headers and slice data). A picture header (picture header syntax) can contain information / parameters that can be commonly applied to pictures. A slice header (slice header syntax) can contain information / parameters that can be commonly applied to slices. An APS (APS syntax) or PPS (PPS syntax) can contain information / parameters that can be commonly applied to one or more slices or pictures. An SPS (SPS syntax) can contain information / parameters that can be commonly applied to one or more sequences. A VPS (VPS syntax) can contain information / parameters that can be commonly applied to multiple layers. A DPS (DPS syntax) can contain information / parameters that can be commonly applied to the entire video. A DPS can contain information / parameters related to the concatenation of coded video sequences (CVS).

[0090] In this document, the video information encoded from the encoding device to the decoding device and signaled in bitstream form may include not only partitioning-related information within the picture, intra / inter prediction information, interlayer prediction-related information, residual information, in-loop filtering information, etc., but may also include information contained in the slice header, the picture header, the APS, the PPS, the SPS, the VPS, and / or the DPS.

[0091] A common post-processing filtering procedure using a neural network post-filter. The input to this procedure is a bitstream, BitstreamToFilter, and the output is a list of NNPF output pictures, ListNnpfOutputPics. First, BitstreamToFilter is decoded, and the list CroppedDecodedPictures is set with the cropped and decoded pictures in output order as a result of decoding BitstreamToFilter. Secondly, the filtering process for each picture resides in CroppedDecodedPictures and is repeatedly applied in output order to each cropped and decoded picture (one or more NNPF activated pictures). The order of the pictures in ListNnpfOutputPics is in output order.

[0092] There can be no more than one picture associated with a particular time instance in ListNnpfOutputPics. If multiple NNPFs are activated for a particular picture in CroppedDecodedPictures, and any of the NNPFs can be selected, but only one selection is allowed to be applied, then the above constraint may apply regardless of which NNPF is selected to be applied to that particular picture.

[0093] The filtering procedures described later reside in CroppedDecodedPictures and can be applied to each cropped and decoded picture (referred to as the current picture) with one or more NNPF activated. When NNPF is applied to the current picture, the filtered and / or interpolated picture is generated by NNPF by applying the NNPF procedures explicitly stated in the semantics of the NNPFC SEI message to the current picture on a patch basis.

[0094] Currently, when applying NNPF to a picture, the order in which the pictures are generated by NNPF is the same as the output order, as the NNPF is stored in the NNPF output tensor. If the applied NNPF is the last NNPF applied to the picture, the pictures generated by NNPF and output by the NNPF procedure may be included in ListNnpfOutputPics in the same order in which the pictures are stored in the NNPF output tensor.

[0095] Neural network post-filter characteristics (NNPFC) The bonds in Tables 1 to 3 represent the NNPFC syntax structure.

[0096] [Table 1]

[0097] [Table 2]

[0098] [Table 3] The NNPFC syntax structures shown in Tables 1 to 3 can be signaled in the form of SEI (supplemental enhancement information) messages. SEI messages that signal the NNPFC syntax structures shown in Tables 1 to 3 can be referred to as NNPFC SEI messages.

[0099] NNPFC SEI messages can identify neural networks that can be used as post-processing filters. The use of identified post-processing filters (NNPFs) for a particular picture can be indicated using neural-network post-filter activation SEI messages. Here, "post-processing filter" and "post-filter" can have the same meaning.

[0100] Using SEI messages like this may require defining variables such as those shown below.

[0101] - The input picture width and height in luma samples can be indicated by CroppedWidth and CroppedHeight, respectively.

[0102] CroppedYPic[idx], a chroma sample array of input pictures with index idx in the range of -0 to numInputPics-1, and CroppedCbPic[idx] and CroppedCrPic[idx], which are chroma sample arrays, can be used as input to NNPF if they exist.

[0103] -BitDepthY can indicate the bit depth relative to the luma sample array of the input picture.

[0104] -BitDepthC can indicate the bit depth of the chroma sample array (if any) of the input picture.

[0105] -ChromaFormatIdc can indicate a chroma format identifier.

[0106] If the value of -nnpfc_auxiliary_inp_idc is 1, the filtering strength control value array StrengthControlVal[idx] must contain real numbers in the range of 0 to 1 for input pictures with index idx in the range of 0 to numInputPics-1.

[0107] An input picture with index 0 can correspond to a picture whose NNPF, defined by the NNPFC SEI message, is activated by the NNPFA SEI message. An input picture with index i in the range of 0 to numInputPics-1 can precede an input picture with index i-1 in the output order.

[0108] The variables SubWidthC and SubHeightC may be derived from ChromaFormatIdc. Two or more NNPFC SEI messages can exist for the same picture. If two or more NNPFC SEI messages with different nnpfc_id values ​​exist or are activated for the same picture, the two or more NNPFC SEI messages can have the same or different nnpfc_purpose and nnpfc_mode_idx values.

[0109] nnpfc_purpose can indicate the purpose of the NNPF as explicitly stated in Table 3. Here, a non-zero value for (nnpfc_purpose & bitMask) indicates that the NNPF has a purpose related to the bitMask value in Table 3. If nnpfc_purpose is greater than 0 and (nnpfc_purpose & bitMask) is 0, the purpose related to the bitMask value does not necessarily have to be applied to the NNPF. If nnpfc_purpose is 0, the NNPF may be used.

[0110] The nnpfc_purpose value must be in the range of 0 to 63 in the bitstream. Values ​​64 to 65535 for nnpfc_purpose may be reserved for future use and do not need to be present in the bitstream. The decoder must ignore NNPFC SEI messages containing npfc_purpose in the range of 64 to 65535.

[0111] [Table 4] The variables chromaUpsamplingFlag, resolutionResamplingFlag, pictureRateUpsamplingFlag, bitDepthUpsamplingFlag, and colourizationFlag, which determine whether nnpfc_purpose indicates an NNPF purpose including chroma upsampling, resolution resampling, picture rate upsampling, bit depth upsampling, and colourization, can be derived as shown in Table 5 below.

[0112] [Table 5] If a reserved value for nnpfc_purpose is to be used in the future, the syntax of the SEI message may be extended to include syntax elements whose existence is determined by the nnpfc_purpose that matches the reserved value.

[0113] If ChromaFormatIdc is 3, then chromaUpsamplingFlag must be the same as 0.

[0114] If ChromaFormatIdc or chromaUpsamplingFlag is not 0, then colourizationFlag must be the same as 0.

[0115] If pictureRateUpsamplingFlag is 1 and the input picture at index 0 is associated with a frame packing arrangement SEI message having the same fp_arrangement_type as 5, then all input pictures may be associated with a frame packing arrangement SEI message having the same fp_arrangement_type as 5 and the same value of fp_current_frame_is_frame0_flag.

[0116] nnpfc_id can contain an identification number that can be used to identify a post-processing filter. nnpfc_id values ​​are 0-2 32 It must be in the range of -2. The range is 256~511 and 2 31 ~2 32 -2 range nnpfc_id values ​​may be reserved for future use. Decoders are in the range of 256~511 or 2 31 ~2 32 NNPFC SEI messages with an nnpfc_id in the -2 range must be ignored.

[0117] The following may apply if an NNPFC SEI message is currently the first NNPFC SEI message in the decoding sequence with a specific nnpfc_id value within the CLVS (Coded Layer Video Sequence).

[0118] The aforementioned SEI message can indicate a base post-processing filter (NNPF).

[0119] The aforementioned SEI message may relate, in output order, to the currently decoded picture and all subsequent decoded pictures of the current layer until the CLVS terminates.

[0120] A value of 1 for nnpfc_base_flag indicates that the SEI message explicitly identifies the base NNPF. A value of 0 for nnpfc_base_flag indicates that the SEI message explicitly identifies updates related to the base NNPF.

[0121] The following constraints may apply to the nnpfc_base_flag value.

[0122] If an NNPFC SEI message is currently the first NNPFC SEI message in the decoding order that has a specific nnpfc_id value within CLVS, then the nnpfc_base_flag value must be the same as 1.

[0123] If an NNPFC SEI message nnpfcB is not the first NNPFC SEI message in the decoding order with a specific nnpfc_id value in CLVS, and has the same nnpfc_base_flag value as 1, then the NNPFC SEI message must be a repetition of the first NNPFC SEI message nnpfcA with the same nnpfc_id value in the decoding order. That is, the payload content of nnpfcB must be identical to the payload content of nnpfcA.

[0124] If nnpfc_base_flag is 0, the following may apply.

[0125] The aforementioned SEI message can define updates related to preceding base NNPFs in the decoding order that have the same nnpfc_id value. The updates are not cumulative; rather, each update can be applied to a base NNPF that currently has a specific nnpfc_id value in CLVS and is explicitly stated by the first NNPFC SEI message in the decoding order. The NNPF defined by the aforementioned SEI message can be obtained by applying the updates defined by the aforementioned SEI message to base NNPFs that have the same nnpfc_id value.

[0126] The aforementioned SEI message may relate to the currently decoded picture and all subsequent decoded pictures of the current layer in output order, until the current CLVS terminates or until the decoded picture that comes after the currently decoded picture in output order within the current CLVS is excluded, and may relate to subsequent NNPFC SEI messages in decoding order where nnpfc_base_flag is 0 and has an earlier specific nnpfc_id value within the current CLVS.

[0127] A value of 0 for nnpfc_mode_idc indicates that the SEI message explicitly identifies the base NNPF (if nnpfc_base_flag is 1), is an update related to the base NNPF with the same nnpfc_id value (if nnpfc_base_flag is 0), or contains a bitstream that complies with ISO / IEC 15938-17.

[0128] If nnpfc_mode_idc is 1, a value of 1 for nnpfc_mode_idc indicates that the base NNPF associated with the nnpfc_id value is a neural network, and that neural network may be a neural network identified by a URI represented by nnpfc_uri using the format identified by the tag URI nnpfc_tag_uri. If nnpfc_mode_idc is 0, a value of 1 for nnpfc_mode_idc indicates that an update to a base NNPF having the same nnpfc_id value is defined by a URI represented by nnpfc_uri using the format identified by the tag URI nnpfc_tag_uri.

[0129] The value of nnpfc_mode_idc must be in the range of 0 to 1 in the bitstream. Values ​​of nnpfc_mode_idc in the range of 2 to 255 may be reserved for future use and do not need to be present in the bitstream. The decoder must ignore NNPFC SEI messages with nnpfc_mode_idc in the range of 2 to 255. Values ​​of nnpfc_mode_idc greater than 255 do not exist in the bitstream and do not need to be reserved for future use.

[0130] nnpfc_reserved_zero_bit_a must be equivalent to 0 in the bitstream. The decoder must ignore NNPFC SEI messages where the value of nnpfc_reserved_zero_bit_a is not 0.

[0131] nnpfc_tag_uri can contain a tag URI with syntax and semantics specified in IETF RFC 4151 that identifies the format and related information for updates to a neural network used as the base NNPF or a base NNPF having the same nnpfc_id value identified by nnpfc_uri.

[0132] nnpfc_tag_uri can uniquely identify the format of neural network data explicitly specified by nnrpf_uri, even without a central registry.

[0133] The same nnpfc_tag_uri as "tag:iso.org, 2023:15938-17" can indicate that the neural network data identified by nnpfc_uri complies with ISO / IEC 15938-17.

[0134] The nnpfc_uri may contain a URI with syntax and semantics explicitly defined in IETF Internet Standard 66 that identifies a neural network used as the base NNPF or an update to a base NNPF having the same nnpfc_id value.

[0135] A value of 1 for nnpfc_formatting_and_purpose_flag indicates the presence of syntax elements related to the filter's purpose, input formatting, output formatting, and complexity. A value of 0 for nnpfc_formatting_and_purpose_flag indicates the absence of syntax elements related to the filter's purpose, input formatting, output formatting, and complexity.

[0136] If nnpfc_base_flag is 1, then nnpfc_property_present_flag must be the same as 1.

[0137] If nnpfc_property_present_flag is 0, the values ​​of all syntax elements that can only exist if nnpfc_property_present_flag is 1 can be inferred to be identical to each of those corresponding syntax elements in the NNPFC SEI message containing the base NNPF for which the SEI message provides the update.

[0138] If an NNPFC SEI message nnpfcCurr is not the first NNPFC SEI message in the decoding order that currently has a specific nnpfc_id value in CLVS, nor is it a repetition of the first NNPFC SEI message that has the said specific nnpfc_id value (i.e., the nnpfc_base_flag value is the same as 0), and the value of nnpfc_property_present_flag is the same as 1, then the following constraints may apply:

[0139] The nnpfc_purpose value of an NNPFC SEI message must be identical to the nnpfc_purpose value of the first NNPFC SEI message in the decoding order that currently has a specific nnpfc_id value within the CLVS.

[0140] In an NNPFC SEI message, the value of the syntax element that comes before nnpfc_complexity_info_present_flag and after nnpfc_property_present_flag in the decoding order must be identical to the value of the corresponding syntax element in the first NNPFC SEI message in the decoding order that currently has a specific nnpfc_id value in CLVS.

[0141] Currently, in CLVS, for the first NNPFC SEI message in the decoding order that has a specific nnpfc_id value, the nnpfc_complexity_info_present_flag (as shown in nnpfcBase below) must be identical to 0, or all nnpfc_complexity_info_present_flags must be identical to 1, and all of the following may apply:

[0142] The nnpfc_parameter_type_idc of nnpfcCurr must be identical to the nnpfc_parameter_type_idc of nnpfcBase.

[0143] If nnpfcCurr's nnpfc_log2_parameter_bit_length_minus3 exists, it must be less than or equal to nnpfcBase's nnpfc_log2_parameter_bit_length_minus3.

[0144] If nnpfc_num_parameters_idc of nnpfcBase is 0, then nnpfc_num_parameters_idc of nnpfcCurr must be the same as 0.

[0145] Otherwise (where nnpfc_num_parameters_idc of nnpfcBase is greater than 0), nnpfc_num_parameters_idc of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_num_parameters_idc of nnpfcBase.

[0146] If nnpfc_num_kmac_operations_idc of nnpfcBase is 0, then nnpfc_num_kmac_operations_idc of nnpfcCurr must be the same as 0.

[0147] Otherwise (where nnpfc_num_kmac_operations_idc of nnpfcBase is greater than 0), nnpfc_num_kmac_operations_idc of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_num_kmac_operations_idc of nnpfcBase.

[0148] If nnpfc_total_kilobyte_size of nnpfcBase is 0, then nnpfc_total_kilobyte_size of nnpfcCurr must be the same as 0.

[0149] Otherwise (where nnpfc_total_kilobyte_size of nnpfcBase is greater than 0), nnpfc_total_kilobyte_size of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_total_kilobyte_size of nnpfcBase.

[0150] nnpfc_num_input_pics_minus1+_1 can indicate the number of pictures used as input to NNPF. The value of nnpfc_num_input_pics_minus1 must be in the range of 0 to 63. If pictureRateUpsamplingFlag is the same as 1, the value of nnpfc_num_input_pics_minus1 must be greater than 0.

[0151] The variable numInputPics, which indicates the number of pictures used as input to NNPF, can be derived as shown in equation 1 below.

[0152]

number

[0153] A value of 1 for nnpfc_absent_input_pic_zero_flag indicates that NNPF expects input pictures that are not present in the bitstream to be represented by a sample array with a sample value of 0. A value of 0 for nnpfc_absent_input_pic_zero_flag indicates that NNPF expects input pictures that are not present in the bitstream to be represented by the closest input picture in the bitstream output order. nnpfc_out_sub_c_flag can indicate the values ​​of the variables outSubWidthC and outSubHeightC when chromaUpsamplingFlag is 1. A value of 1 for nnpfc_out_sub_c_flag indicates that the value of outSubWidthC is 1 and the value of outSubHeightC is 1. A value of 0 for nnpfc_out_sub_c_flag indicates that the value of outSubWidthC is 2 and the value of outSubHeightC is 1. If the value of ChromaFormatIdc is 2 and nnpfc_out_sub_c_flag exists, then the value of nnpfc_out_sub_c_flag must be the same as 1.

[0154] nnpfc_out_colour_format_idc can indicate the hue format of the NNPF and, consequently, the values ​​of the variables outSubWidthC and outSubHeightC, when colourizationFlag is 1. A value of 1 for nnpfc_out_colour_format_idc indicates that the hue format of the NNPF output is 4:2:0, and both outSubWidthC and outSubHeightC are 2. A value of 2 for nnpfc_out_colour_format_idc indicates that the hue format of the NNPF output is 4:2:2, outSubWidthC is 2, and outSubHeightC is 1. A value of 3 for nnpfc_out_colour_format_idc indicates that the hue format of the NNPF output is 4:4:4, and both outSubWidthC and outSubHeightC are 1. The value of nnpfc_out_colour_format_idc must not be 0.

[0155] If both chromaUpsamplingFlag and colourizationFlag are 0, outSubWidthC and outSubHeightC can be inferred to be identical to SubWidthC and SubHeightC, respectively. nnpfc_pic_width_num_minus1+1 and nnpfc_pic_width_denom_minus1+1 can represent the numerator and denominator, respectively, for the resampling ratio of the NNPF output picture width with respect to CroppedWidth. The value of (nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denom_minus1+1) must be in the range of 1÷16 to 16. If nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 do not exist, the values ​​of both nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 can be inferred to be identical to 0.

[0156] The variable nnpfcOutputPicWidth, which indicates the width of the luma sample array of the picture(s) resulting from applying the NNPF identified by nnpfc_id to the input picture(s), can be derived as shown in Equation 2 below.

[0157]

number

[0158] nnpfc_pic_height_num_minus1+1 and nnpfc_pic_height_denom_minus1+1 can represent the numerator and denominator, respectively, for the resampling ratio of the NNPF output picture height with respect to CroppedHeight. The value of (nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denom_minus1+1) must be in the range of 1÷16 to 16. If nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 do not exist, it can be inferred that the values ​​of both nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 are identical to 0.

[0159] The variable nnpfcOutputPicHeight, which indicates the height of the luma sample array of the picture(s) resulting from applying the NNPF identified by nnpfc_id to the input picture(s), can be derived as shown in the following equation 3.

[0160]

number

[0161] If nnpfc_pic_width_num_minus1, nnpfc_pic_width_denom_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denom_minus1 exist, then at least one of the following must be true:

[0162] The value of nnpfcOutputPicWidth is not the same as CroppedWidth.

[0163] The value of nnpfcOutputPicHeight is not the same as CroppedHeight.

[0164] nnpfc_interpolated_pics[i] can indicate the number of interpolated pictures generated by NNPF between the i-th picture and the (i+1)th picture used as input to NNPF.

[0165] The value of nnpfc_interpolated_pics[i] must be in the range of 0 to 63. The value of nnpfc_interpolated_pics[i] must be greater than 0 for at least one value of i, in the range of 0 to nnpfc_num_input_pics_minus1-1.

[0166] The variables NumInpPicsInOutputTensor, which indicates the number of pictures present in the NNPF output tensor with the corresponding input picture, InpIdx[idx], which indicates the input picture index of the idx-th picture present in the NNPF output tensor with the corresponding input picture, and numOutputPics, which indicates the total number of pictures present in the NNPF output tensor, can be derived as shown in Table 6 below.

[0167] [Table 6] A value of 1 for nnpfc_component_last_flag indicates that the last dimension of the input tensor `inputTensor` for NNPF and the output tensor `outputTensor`, which is the result of NNPF, are currently used for the channel. A value of 0 for nnpfc_component_last_flag indicates that the third dimension of the input tensor `inputTensor` for NNPF and the output tensor `outputTensor`, which is the result of NNPF, are currently used for the channel.

[0168] The first dimension of the input and output tensors can be used as a batch index in some neural network frameworks. While the formula within the semantics of this SEI message uses the size of the batch corresponding to a batch index of 0, the size of the batch used as input for neural network inference can be determined by the implementation of post-processing.

[0169] For example, when the value of nnpfc_inp_order_idc is the same as 3 and the value of nnpfc_auxiliary_inp_idc is the same as 1, the input tensor can have 7 channels, including 4 luma matrices, 2 chroma matrices, and 1 auxiliary input matrix. In this case, the DeriveInputTensors() process can induce each of the 7 channels of the input tensor one by one, and when a particular channel is processed, that channel can be referred to as the current channel in the process.

[0170] nnpfc_inp_format_idc indicates how to convert the sample values ​​of the input picture into NNPF input values. If nnpfc_inp_format_idc is 0, the input values ​​for NNPF are real numbers and the functions InpY() and InpC() can be specified as shown in equation 4 below.

[0171]

number

[0172] [Table 7] The variable inpTensorBitDepthy can be derived from the syntax element nnpfc_inp_tensor_bitlength_minus8 described below. inpTensorBitDepthC can be derived from the syntax element nnpfc_inp_tensor_chroma_bitdepth_minus8 described below.

[0173] A value of nnpfc_inp_format_idc greater than 1 may be reserved for future use and may not exist in the bitstream. The decoder must ignore NNPFC SEI messages containing reserved values ​​for nnpfc_inp_format_idc.

[0174] A value of nnpfc_auxiliary_inp_idc greater than 0 indicates that auxiliary input data exists in the NNPF input tensor. A value of nnpfc_auxiliary_inp_idc of 0 indicates that auxiliary input data does not exist in the input tensor. A value of nnpfc_auxiliary_inp_idc of 1 indicates that auxiliary input data is induced, as explained in equation 5 below.

[0175] The value of nnpfc_auxiliary_inp_idc must be in the range of 0 to 1 in the bitstream. Values ​​of nnpfc_auxiliary_inp_idc between 2 and 255 may be reserved for future use and do not need to be present in the bitstream. The decoder must ignore NNPFC SEI messages containing nnpfc_auxiliary_inp_idc values ​​in the range of 2 to 255. Values ​​of nnpfc_auxiliary_inp_idc greater than 255 do not exist in the bitstream and are not reserved for future use.

[0176] nnpfc_inp_order_idc can show how to align the sample array of input pictures to form the input tensor for NNPF.

[0177] The value of nnpfc_inp_order_idc must be in the range of 0 to 3 in the bitstream. Values ​​of nnpfc_inp_order_idc between 4 and 255 may be reserved for future use and do not exist in the bitstream. The decoder must ignore NNPFC SEI messages with nnpfc_inp_order_idc in the range of 4 to 255. Values ​​of nnpfc_inp_order_idc greater than 255 do not exist in the bitstream and are not reserved for future use.

[0178] If the value of ChromaFormatIdc is not 1, then the value of nnpfc_inp_order_idc must not be 3.

[0179] If ChromaFormatIdc is 0, then nnpfc_inp_order_idc must be the same as 0.

[0180] If chromaUpsamplingFlag is 1, then nnpfc_inp_order_idc must not be 0.

[0181] Table 8 includes explanations for the nnpfc_inp_order_idc values.

[0182] [Table 8] nnpfc_inp_tensor_bitlength_minus8+8 is an input constant tensor that can indicate the bit depth of the luma sample value. The value of inpTensorBitDepthY can be derived as shown in equation 5.

[0183]

number

[0184] nnpfc_inp_tensor_chroma_bitdepth_minus8+8 is an input constant tensor that can indicate the bit depth of the chroma sample values. The value of inpTensorBitDepthC can be derived as shown in equation 6.

[0185]

number

[0186] If nnpfc_auxiliary_inp_idc is 1, the variable strengthControlScaledVal may be induced as shown in Table 9 below.

[0187] [Table 9] A patch can be a rectangular array of samples from the components of a picture (e.g., luma or chroma components).

[0188] The process DeriveInputTensors() for deriving the input tensor inputTensor with respect to the horizontal sample coordinate cLeft that specifies the top-left sample position for the given vertical sample coordinate cTop and the patch of samples included in the input tensor can be shown as a combination of Tables 10 to 12.

[0189]

Table 10

Table 11

[0193] The value of nnpfc_out_order_idc must be in the range of 0 to 3 in the bitstream. Values ​​of nnpfc_out_order_idc between 4 and 255 may be reserved for future use and do not exist in the bitstream. The decoder must ignore NNPFC SEI messages containing nnpfc_out_order_idc values ​​in the range of 4 to 255. Values ​​of nnpfc_out_order_idc greater than 255 do not exist in the bitstream and are not reserved for future use.

[0194] If chromaUpsamplingFlag is 1, then nnpfc_out_order_idc must not be 0 or 3.

[0195] If colourizationFlag is 1, then nnpfc_out_order_idc must not be 0.

[0196] Table 13 includes explanations for the nnpfc_out_order_idc values.

[0197] [Table 13] nnpfc_out_tensor_luma_bitdepth_minus8+8 can indicate the bit depth of the luma sample value of the output constant tensor. The value of nnpfc_out_tensor_luma_bitdepth_minus8 must be in the range of 0 to 24. The value of outTensorBitDepthY can be derived as shown in equation 7.

[0198]

number

[0199]

number

[0200] nnpfc_out_tensor_luma_bitdepth_minus8 exists, and outTensorBitDepthY is greater than BitDepthY.

[0201] nnpfc_out_tensor_chroma_bitdepth_minus8 exists, and outTensorBitDepthC is greater than BitDepthC.

[0202] If nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepthY is greater than inpTensorBitDepthY, then outTensorBitDepthC must be less than inpTensorBitDepthC.

[0203] If nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepthC is greater than inpTensorBitDepthC, then outTensorBitDepthY must be less than inpTensorBitDepthY.

[0204] The StoreOutputTensors() process for deriving sample values ​​in the output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic, filtered from the output tensor outputTensor, which has been filtered for the given vertical sample coordinate cTop and the horizontal sample coordinate cLeft indicating the top-left sample position for the patch of samples contained in the input tensor, can be represented as the joins in Tables 14 and 15.

[0205] [Table 14] [Table 15] A value of 1 for nnpfc_separate_colour_description_present_flag indicates that a unique combination of color primaries, transformation properties, matrix coefficients, and scaling and offset values ​​applied in relation to the matrix coefficients for the picture by NNPF is specified in the SEI message syntax structure. A value of 0 for nnpfc_separate_colour_description_present_flag indicates that the combination of color primaries, transformation properties, matrix coefficients, and scaling and offset values ​​applied in relation to the matrix coefficients for the picture by the post-processing filter is identical to that displayed in the CLVS VUI parameters.

[0206] nnpfc_colour_primaries can have the same semantics as those defined for the vui_colour_primaries syntax element, except for the following:

[0207] nnpfc_colour_primaries can indicate the primary hues of a picture that are not used for CLVS, but rather the primary hues shown as a result of applying the NNPF specified in the SEI message.

[0208] If nnpfc_colour_primaries does not exist in the NNPFC SEI message, it can be inferred that the value of nnpfc_colour_primaries is the same as the value of vui_colour_primaries.

[0209] nnpfc_transfer_characteristics can have the same semantics as those defined for the vui_transfer_characteristics syntax elements, except for the following:

[0210] nnpfc_transfer_characteristics can indicate the picture transformation characteristics that are not used in CLVS, but rather the characteristics of the picture that are shown as a result of applying the NNPF specified in the SEI message.

[0211] If nnpfc_transfer_characteristics is not present in the NNPFC SEI message, it can be inferred that the value of nnpfc_transfer_characteristics is the same as the value of vui_transfer_characteristics.

[0212] nnpfc_matrix_coeffs can describe the formulas used to induce lumens and chroma signals in green, blue, red, or the Y, Z, and X primary colors. Its semantics apply to the picture shown as a result of applying the NNPF specified in the SEI message and may be identical to those specified for MatrixCoefficients containing the same outTensorBitDepthC and BitDepthC as outTensorBitDepthY and outTensorBitDepthC, respectively.

[0213] If nnpfc_matrix_coeffs is not present in the NNPFC SEI message, it can be inferred that the value of nnpfc_matrix_coeffs is the same as the value of vui_matrix_coeffs.

[0214] nnpfc_matrix_coeffs cannot be identical to 0 unless all of the following conditions are true.

[0215] nnpfc_out_tensor_chroma_bitdepth_minus8 is identical to nnpfc_out_tensor_luma_bitdepth_minus8.

[0216] nnpfc_out_order_idc is the same as 2, outSubHeightC is the same as 1, and outSubWidthC is the same as 1.

[0217] nnpfc_matrix_coeffs cannot be identical to 8 unless one of the following conditions is true.

[0218] nnpfc_out_tensor_chroma_bitdepth_minus8 is identical to nnpfc_out_tensor_luma_bitdepth_minus8.

[0219] nnpfc_out_tensor_chroma_bitdepth_minus8 is the same as nnpfc_out_tensor_luma_bitdepth_minus8+1, nnpfc_out_order_idc is the same as 2, outSubHeightC is the same as 1, and outSubWidthC is the same as 1.

[0220] nnpfc_full_range_flag can indicate the scaling and offset values ​​applied in relation to the matrix coefficients, as specified by nnpfc_matrix_coeffs. Its semantics may be identical to those specified for the VideoFullRangeFlag parameter. If it is not present, the value of nnpfc_full_range_flag can be inferred to be equivalent to 0.

[0221] A value of 1 for nnpfc_chroma_loc_info_present_flag indicates the presence of the nnpfc_chroma_sample_loc_type_frame syntax element in an NNPFC SEI message. A value of 0 for nnpfc_chroma_loc_info_present_flag indicates the absence of the nnpfc_chroma_sample_loc_type_frame syntax element in an NNPFC SEI message. If colourizationFlag is 0 or nnpfc_out_colour_format_idc is not 1, the value of nnpfc_chroma_loc_info_present_flag must be the same as 0.

[0222] nnpfc_chroma_sample_loc_type_frame can indicate the chroma sample location in the output picture if nnpfc_chroma_sample_loc_type_frame is not 6 and nnpfc_out_colour_format_idc is 1. If nnpfc_chroma_sample_loc_type_frame is 6 and nnpfc_out_colour_format_idc is 1, it can indicate that the chroma sample location is not known, not explicitly stated, or will be stated by other means. The value of nnpfc_chroma_sample_loc_type_frame must be in the range of 0 to 6.

[0223] nnpfc_overlap can indicate the number of overlapping horizontal and vertical samples of adjacent input tensors in NNPF. The value of nnpfc_overlap must be in the range of 0 to 16383. A value of 1 for nnpfc_constant_patch_size_flag indicates that NNPF will accept the exact patch size indicated by nnpfc_patch_width_minus1 and nnpfc_patch_height_minus1 as input. A value of 0 for nnpfc_constant_patch_size_flag indicates that NNPF will accept any patch size as input, where the width is inpPatchWidth and the height is inpPatchHeight. This allows the width of an extended patch (i.e., patch + overlapping region) to be a positive integer multiple of nnpfc_extended_patch_width_cd_delta_minus1+1+2*nnpfc_overlap, and the height of an extended patch to be a positive integer multiple of nnpfc_extended_patch_height_cd_delta_minus1+1+2*nnpfc_overlap, which is the same as inpPatchHeight+2*nnpfc_overlap.

[0224] npfc_patch_width_minus1+1 can indicate the number of horizontal samples required for the patch size of the NNPF input when the value of nnpfc_constant_patch_size_flag is 1. The value of nnpfc_patch_width_minus1 must be in the range of 0 to Min(32766, CroppedWidth-1).

[0225] npfc_patch_height_minus1+1 can indicate the number of vertical samples required for the patch size of the NNPF input when the value of nnpfc_constant_patch_size_flag is 1. The value of nnpfc_patch_height_minus1 must be in the range of 0 to Min(32766, CroppedHeight-1).

[0226] nnpfc_extended_patch_width_cd_delta_minus1+1+2*nnpfc_overlap can represent the common divisor of all allowed values ​​for the width of the extended patch required as input to NNPF when nnpfc_constant_patch_size_flag is 0. The value of nnpfc_extended_patch_width_cd_delta_minus1 must be in the range of 0 to Min(32766, CroppedWidth-1).

[0227] nnpfc_extended_patch_height_cd_delta_minus1+1+2*nnpfc_overlap can represent the common divisor of all allowed values ​​for the extended patch height required as input to NNPF when nnpfc_constant_patch_size_flag is 0. The value of nnpfc_extended_patch_height_cd_delta_minus1 must be in the range of 0 to Min(32766, CroppedHeight-1).

[0228] The variables inpPatchWidth and inpPatchHeight can be set to the width and height of the patch, respectively.

[0229] If the value of nnpfc_constant_patch_size_flag is 0, the following may apply.

[0230] The values ​​of inpPatchWidth and inpPatchHeight can be provided by external means or set by the post-processor itself.

[0231] The value of inpPatchWidth + 2 * nnpfc_overlap must be a positive integer multiple of ofnnpfc_extended_patch_width_cd_delta_minus1 + 1 + 2 * nnpfc_overlap, and inpPatchWidth must be less than or equal to CroppedWidth. The value of inpPatchHeight + 2 * nnpfc_overlap must be a positive integer multiple of ofnnpfc_extended_patch_height_cd_delta_minus1 + 1 + 2 * nnpfc_overlap, and inpPatchHeight must be less than or equal to CroppedHeight.

[0232] Otherwise (if the value of nnpfc_constant_patch_size_flag is the same as 1), the value of inpPatchWidth may be set to the same as nnpfc_patch_width_minus1+1, and the value of inpPatchHeight may be set to the same as nnpfc_patch_height_minus1+1.

[0233] The variables outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, outPatchCHeight, and overlapSize can be derived as shown in Table 16.

[0234] [Table 16] It is a bitstream compliance requirement that outPatchWidth * CroppedWidth must be the same as nnpfc_pic_width_in_luma_samples * inpPatchWidth, and outPatchHeight * CroppedHeight must be the same as nnpfc_pic_height_in_luma_samples * inpPatchHeight.

[0235] As described in Table 17, nnpfc_padding_type can indicate the padding process when referring to sample positions outside the boundaries of the input picture. The value of nnpfc_padding_type must exist in the range of 0 to 4 in the bitstream. Values from 5 to 15 for nnpfc_padding_type can be reserved for future use and do not exist in the bitstream. The decoder must ignore NNPFC SEI messages containing nnpfc_padding_type in the range of 5 to 15. Values of nnpfc_padding_type greater than 15 do not exist in the bitstream and are not reserved for future use.

[0236]

Table 17

[0237] When the value of nnpfc_padding_type is 4, nnpfc_cb_padding_val can indicate the Cb value used for padding. The value of nnpfc_cb_padding_val must exist in the range of 0 to (1 << BitDepthC) - 1.

[0238] When the value of nnpfc_padding_type is 4, nnpfc_cr_padding_val can indicate the Cr value used for padding. The value of nnpfc_cr_padding_val must exist within the range of 0 to (1<<BitDepthC)-1.

[0239] The function InpSampleVal(y, x, picHeight, picWidth, CroppedPic) with inputs of vertical sample position y, horizontal sample position x, picture height picHeight, picture width picWidth, sample array CroppedPic, and component index cIdx (the same as 0 for luma, the same as 1 for Cb, and the same as 2 for Cr) can return the value of SampleVal derived as shown in Table 18.

[0240] For the inputs to the function InpSampleVal(), the vertical position can be listed before the horizontal position for compatibility with the input tensor rules of some inference engines.

[0241]

Table 18

[0242]

Table 19

[0243] The NNPF process can output the NNPF-generated pictures in their increasing index order to constitute the process defined by Table 19. Here, all NNPF-generated pictures interpolated by NNPF are output, and the NNPF-generated pictures corresponding to all input pictures for NNPF can be output as specified in the semantics of the NNPFA SEI message.

[0244] A value of 1 for the nnpfc_complexity_info_present_flag can indicate that there is one or more syntax elements indicating the complexity of the NNPF related to nnpfc_id. A value of 0 for the nnpfc_complexity_info_present_flag can indicate that there are no syntax elements indicating the complexity of the NNPF related to nnpfc_id.

[0245] A value of 0 for nnpfc_parameter_type_idc can indicate that the neural network uses only integer parameters. A value of 1 for nnpfc_parameter_type_flag can indicate that the neural network can use floating-point or integer parameters. A value of 2 for nnpfc_parameter_type_idc can indicate that the neural network uses only binary parameters. A value of 3 for nnpfc_parameter_type_idc can be reserved for future use and does not exist in the bitstream. The decoder must ignore the NNPFC SEI message with a value of 3 for nnpfc_parameter_type_idc.

[0246] The values ​​0, 1, 2, and 3 for nnpfc_log2_parameter_bit_length_minus3 indicate that the neural network will not use parameters with bit lengths greater than 8, 16, 32, and 64, respectively. If nnpfc_parameter_type_idc exists and nnpfc_log2_parameter_bit_length_minus3 does not exist, the neural network may not use parameters with bit lengths greater than 1.

[0247] nnpfc_num_parameters_idc can indicate the maximum number of neural network parameters for NNPF in powers of 2048. A value of nnpfc_num_parameters_idc of 0 indicates that the maximum number of neural network parameters is unknown. The value of nnpfc_num_parameters_idc must be in the range of 0 to 52. Values ​​of nnpfc_num_parameters_idc greater than 52 may be reserved for future use and do not exist in the bitstream. The decoder must ignore NNPFC SEI messages with nnpfc_num_parameters_idc greater than 52.

[0248] If the value of nnpfc_num_parameters_idc is greater than 0, the variable maxNumParameters may be derived as shown in equation 9.

[0249]

number

[0250] A value of nnpfc_num_kmac_operations_idc greater than 0 indicates that the maximum number of multiply-accumulate operations per sample in NNPF is less than or equal to nnpfc_num_kmac_operations_idc * 1000. A value of nnpfc_num_kmac_operations_idc of 0 indicates that the network does not know the maximum number of multiply-accumulate operations. The value of nnpfc_num_kmac_operations_idc is between 0 and 2. 32 It must exist within the range of -2.

[0251] A value of nnpfc_total_kilobyte_size greater than 0 can indicate the total size (in kilobytes) required to store the uncompressed parameters of a neural network. The total size in bits can be a number that is greater than or equal to the sum of the bits used to store each parameter. nnpfc_total_kilobyte_size can be the total size (in bits) divided by 8000 and rounded. A value of nnpfc_total_kilobyte_size of 0 indicates that the total size required to store the parameters for the neural network is unknown. The value of nnpfc_total_kilobyte_size is between 0 and 2. 32 It must exist within the range of -2.

[0252] A value of 0 for nnpfc_metadata_extension_num_bits can indicate that nnpfc_reserved_metadata_extension does not exist. A value of nnpfc_metadata_extension_num_bits greater than 0 can indicate the length of nnpfc_reserved_metadata_extension in bits. nnpfc_metadata_extension_num_bits must be identical to 0. Values ​​in the range of 1 to 2048 for nnpfc_metadata_extension_num_bits may be reserved for future use and do not exist in the bitstream. The decoder must accept all values ​​of nnpfc_metadata_extension_num_bits in the range of 0 to 2048. Values ​​of nnpfc_metadata_extension_num_bits greater than 2048 do not exist in the bitstream and are not reserved for future use.

[0253] nnpfc_reserved_metadata_extension does not exist in the bitstream. However, the decoder must ignore the existence and value of nnpfc_reserved_metadata_extension. If it exists, the bitwise length of nnpfc_reserved_metadata_extension may be identical to nnpfc_metadata_extension_num_bits.

[0254] nnpfc_reserved_zero_bit_b must be equivalent to 0 in the bitstream. The decoder must ignore NNPFC SEI messages where nnpfc_reserved_zero_bit_b is not 0.

[0255] nnpfc_payload_byte[i] can contain the i-th byte of the bitstream. The byte sequence nnpfc_payload_byte[i] for all existing values ​​of i must be a complete bitstream compliant with ISO / IEC 15938-17.

[0256] Neural network post-filter activation (NNPFA) The syntax structure for NNPFA is shown in Table 20.

[0257] [Table 20] The NNPFA syntax structure in Table 20 can be signaled in the form of an SEI message. An SEI message that signals the NNPFA syntax structure in Table 20 can be referred to as an NNPFA SEI message.

[0258] The NNPFA SEI message can activate or deactivate the possible use of the target neural network post-processing filter, identified by nnpfa_target_id and nnpfc_base_flag, for post-processing filtering of a picture set. For a specific picture with an activated NNPF, the target NNPF may be derived as follows:

[0259] If nnpfa_target_base_flag is 1, the target NNPF is the base NNPF with the same nnpfc_id as nnpfa_target_id.

[0260] Otherwise (when nnpfa_target_base_flag is 0), the target NNPF is identified by the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id, and is not a repetition of NNPFC SEI messages containing the base NNPF that precedes the first VCL NAL unit of the current picture in the decoding order.

[0261] Multiple NNPFA SEI messages can exist for the same picture when the post - processing filter is used for other purposes or filters other hue components.

[0262] The nnpfa_target_id can indicate the target NNPF specified by one or more NNPFC SEI messages related to the current picture and having the same nnpfc_id as the nnpfa_target_id.

[0263] The value of nnpfa_target_id must exist in the range of 0 to 2 32 -2. It must not be in the range of 256 to 511 and 2 31 ~2 32 The nnpfa_target_id values within the range of 256 to 511 or 2 31 ~2 32 -2 may be reserved for future use. The decoder must ignore NNPFA SEI messages having nnpfa_target_id in the range of 256 to 511 or 2

[0264] An NNPFA SEI message having a specific value of nnpfa_target_id must not exist in the current PU (Picture Unit) unless one or both of the following conditions are all true.

[0265] There exists an NNPFC SEI message within the current CLVS having the same nnpfc_id as the specific value of nnpfa_target_id that exists in a PU preceding the current PU in decoding order There exists an NNPFC SEI message in the current PU having the same nnpfc_id as the specific value of nnpfa_target_id If the PU contains all NNPFC SEI messages having a specific value of nnpfc_id and NNPFA SEI messages having the same nnpfa_target_id as the specific value of nnpfc_id, the NNPFC SEI message must precede the NNPFA SEI message in decoding order.

[0266] A value of 1 for nnpfa_cancel_flag indicates that the persistence of the target neural network post-processing filter, set by any previous NNPFA SEI message having the same nnpfa_target_id as the current SEI message, is cancelled. In other words, the target neural network post-processing filter will not be used anymore unless activated by another NNPFA SEI message having the same nnpfa_target_id and the same nnpfa_cancel_flag (0) as the current SEI message. A value of 0 for nnpfa_cancel_flag indicates that nnpfa_target_base_flag, nnpfa_persistence_flag, and nnpfa_num_output_entries will follow.

[0267] A value of 1 for nnpfa_target_base_flag indicates that the target NNPF is a base NNPF with the same nnpfc_id as nnpfa_target_id. A value of 0 for nnpfa_target_base_flag indicates that the target NNPF is identified by the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id and precedes the first VCL NAL unit of the current picture in the decoding order.

[0268] The nnpfa_persistence_flag can indicate the persistence of the target neural network post-processing filter for the current layer. A value of nnpfa_persistence_flag of 0 indicates that the target neural network post-processing filter can only be used for post-processing filtering on the current picture. A value of nnpfa_persistence_flag of 1 indicates that the target neural network post-processing filter can be used for post-processing filtering on the current picture and all subsequent pictures in the current layer in output order until one or more of the following conditions are true.

[0269] A new CLVS for the current layer is initiated. The bitstream is ending. Currently, pictures in the current layer associated with NNPFA SEI messages that have the same nnpfa_target_id and nnpfa_cancel_flag as the SEI message and have the same value as 1 will be output after the current picture in the output order. The target neural network post-processing filter is not applied to subsequent pictures in the current layer related to NNPFA SEI messages that have the same nnpfa_target_id and nnpfa_cancel_flag as the current SEI message (nnpfa_target_id and nnpfa_cancel_flag are both 1).

[0270] Let nnpfcTargetPictures be the set of pictures associated with the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id while currently preceding the NNPFA SEI message in the decoding order. Let nnpfaTargetPictures be the set of pictures whose target NNPF is currently activated by the NNPFA SEI message. The bitstream compatibility requirement is that all pictures included in nnpfaTargetPictures must also be included in nnpfcTargetPictures.

[0271] nnpfa_num_output_entries can indicate the number of nnpfa_output_flag[i] syntax elements present in an NNPFA SEI message. The value of nnpfa_num_output_entries falls within the range of 0 to NumInpPicsInOutputTensor.

[0272] A value of 1 for nnpfa_output_flag[i] indicates that the NNPF-generated picture corresponding to the input picture with index InpIdx[i] is output by the NNPF process activated by this NNPFA SEI message, where the NNPF process can be identified within the semantics of the NNPFC SEI message. A value of 0 for nnpfa_output_flag[i] can indicate that the NNPF-generated picture corresponding to the input picture with index InpIdx[i] is not output by the NNPF process activated by this NNPFA SEI message. If nnpfa_num_output_entries is less than NumInpPicsInOutputTensor, then nnpfa_output_flag[i] can be inferred to be 1 for each value of i in the range from nnpfa_num_output_entries to NumInpPicsInOutputTensor-1.

[0273] Post-filter hint The syntax structure for post-filter hints is shown in Table 21.

[0274] [Table 21] The post-filter hint syntax structures in Table 21 can be signaled in the form of SEI messages. SEI messages that signal the post-filter hint syntax structures in Table 21 can be referred to as post-filter hint SEI messages.

[0275] Post-filter hint SEI messages can provide post-filter coefficients or relational information for post-filter design, potentially allowing the decoded and output picture set to be used for post-processing to obtain improved display quality.

[0276] A value of 1 for `filter_hint_cancel_flag` indicates that the SEI message cancels the persistence of a previous post-filter hint SEI message in the output order applied to the current layer. A value of 0 for `filter_hint_cancel_flag` indicates that post-filter hint information follows.

[0277] The `filter_hint_persistence_flag` can indicate the persistence of a post-filter hint SEI message for the current layer. A value of 0 for `filter_hint_persistence_flag` indicates that the post-filter hint applies only to the currently decoded picture. A value of 1 for `filter_hint_persistence_flag` indicates that the post-filter hint SEI message applies to the currently decoded picture and persists for all subsequent pictures in the current layer in the output order until one or more of the following conditions are true.

[0278] A new CLVS for the current layer is initiated. The bitstream is ending. Post-filter hints related to SEI messages: Pictures in the AU's current layer are output after the current picture in output order. `filter_hint_size_y` can represent the filter coefficient or the vertical size of the relation array. The value of `filter_hint_size_y` must be in the range of 1 to 15.

[0279] `filter_hint_size_x` can represent the filter coefficient or the horizontal size of the relation array. The value of `filter_hint_size_x` must be in the range of 1 to 15.

[0280] As shown in Table 19, `filter_hint_type` can indicate the type of filter hint transmitted. The value of `filter_hint_type` must be in the range of 0 to 2. A `filter_hint_type` value identical to 3 does not exist in the bitstream. The decoder must ignore post-filter hint SEI messages where `filter_hint_type` is 3.

[0281] [Table 22] A value of 1 for `filter_hint_chroma_coeff_present_flag` indicates that a filter coefficient exists for the chroma. A value of 0 for `filter_hint_chroma_coeff_present_flag` indicates that no filter coefficient exists for the chroma.

[0282] `filter_hint_value[cIdx][cy][cx]` can represent the filter coefficients, or the cross-correlation matrix elements between the original signal and the decoded signal, with 16-bit precision. The value of `filter_hint_value[cIdx][cy][cx]` is -2 31 +1~2 31 It must be within the -1 range. cIdx can indicate the associated hue element, cy can indicate the vertical counter, and cx can indicate the horizontal counter. Depending on the value of filter_hint_type, the following may apply:

[0283] If the value of filter_hint_type is 0, coefficients of a 2D FIR (Finite Impulse Response) filter with size filter_hint_size_y * filter_hint_size_x may be transmitted.

[0284] Otherwise, if the value of filter_hint_type is 1, the filter coefficients of two 1D FIR filters may be transmitted. In this case, the value of filter_hint_size_y must be 2. The index cy that is 0 can indicate the filter coefficients of the horizontal filter, and cy that is 1 can indicate the filter coefficients of the vertical filter. In the filtering process, the horizontal filter is applied first, and the result can be filtered by the vertical filter.

[0285] Otherwise (if the value of filter_hint_type is 2), the transmitted hint can indicate the cross-correlation matrix between the original signal s and the decoded signal s'.

[0286] The normalized cross-correlation matrix for the correlation color component identified by cIdx of size filter_hint_size_y * filter_hint_size_x can be defined as in Equation 10.

[0287]

Equation

[0288] The decoder can derive the Wiener post-filter from the cross-correlation matrix of the original signal and the decoded signal and the auto-cross-correlation matrix of the decoded signal.

[0289] Problems with conventional technology In the current configuration of NNPFC and NNPFA SEI messages, NNPFA SEI messages can activate either a base filter (i.e., NNPFA SEI messages containing the same nnpfc_base_flag as 1) or an update filter (e.g., NNPFA SEI messages containing the same nnpfc_base_flag as 0) based on the value of nnpfa_target_base_flag.

[0290] The nnpfa_target_base_flag can only be signaled when the value of nnpfa_cancel_flag is 0. This can cause problems in identifying which NNPF activation should be canceled if there are two NNPF activations with the same nnpfa_target_id but different nnpfa_target_base_flag values.

[0291] In other words, if both the base NNPF and update NNPF with the same nnpfc_id are activated for the same picture, it is not clear how to distinguish which of the two persistences will be canceled when a new NNPFA SEI message with the same nnpfa_target_id and the same nnpfa_cancel_flag exists later, unless such a case is permitted.

[0292] [Implementation Method] Examples This application provides solutions to the aforementioned problems. Each embodiment can be carried out individually or two or more embodiments can be carried out in combination.

[0293] The embodiments proposed by this application can be summarized as follows:

[0294] The nnpfa_target_base_flag can be signaled without any conditions on the nnpfa_cancel_flag value.

[0295] If the nnpfa_cancel_flag value is the same as 1, the persistence of a previously activated target NNPF containing the same values ​​for nnpfa_target_id and nnpfa_target_base_flag can be revoked.

[0296] As an alternative, if an NNPFA containing a specific nnpfa_target_id and nnpfa_target_base_flag exists, the persistence of previous NNPFA SEI messages with the same nnpfa_target_id and nnpfa_target_base_flag values ​​in line with the output order can be revoked.

[0297] Another alternative is to restrict the activation of two (or more) NNPFA SEI messages (i.e., those containing nnpfa_cancel_flag identical to 0) for the same picture, regardless of the value of nnpfa_target_base_flag.

[0298] In other words, regardless of the value of nnpfa_target_base_flag, two NNPFA SEI messages with the same nnpfa_target_id and an nnpfa_cancel_flag of 0 will not be activated.

[0299] Example 1 Example 1 provides an explanation for the aforementioned Abstracts 1 and 2. The table may be updated as follows:

[0300] [Table 23] The NNPFA SEI message can activate or deactivate the possible use of the target NNPF, identified by nnpfa_target_id and nnpfa_target_base_flag, for post-processing filtering of a set of pictures. nnpfa_target_base_flag can be parsed without the condition of the nnpfa_cancel_flag value, as shown in Table 23. Therefore, for a particular picture in which the NNPF is activated, the target NNPF may be induced as follows:

[0301] If nnpfa_target_base_flag is the same as 1, the target NNPF can be the base NNPF where nnpfc_id is the same as nnpfa_target_id.

[0302] Otherwise (where nnpfa_target_base_flag is the same as 0), the target NNPF is the NNPF specified by the last NNPF SEI message that has the same nnpfc_id as nnpfa_target_id and precedes the first VCL NAL unit of the current picture in the decoding order, and does not have to be a repetition of NNPFC SEI messages that include the base NNPF.

[0303] For example, if NNPF means filtering for other purposes or other hue components, there may be several NNPFA SEI messages present for the same picture.

[0304] nnpfa_target_id can currently indicate the target NNPF, which is explicitly shown by one or more NNPFC SEI messages that have the same nnpfc_id as nnpfa_target_id in relation to the picture. The nnpfa_target_id value is 0 to 2 32 It must exist within the range of -2.

[0305] An NNPFA SEI message with a specific value for nnpfa_target_id should not currently exist in the PU unless one or two of the following conditions are all true:

[0306] Currently, within the CLVS, there may be NNPFC SEI messages that have the same nnpfc_id as a specific value of nnpfa_target_id in a PU that precedes the current PU in the decoding order.

[0307] Currently, it is possible for an NNPFC SEI message to exist in the PU that has an nnpfc_id that is identical to a specific value of nnpfa_target_id.

[0308] If the PU includes an NNPFC SEI message having a specific value for nnpfc_id and an NNPFA SEI message having the same nnpfa_target_id as the aforementioned specific value for nnpfc_id, the NNPFC SEI message must precede the NNPFA SEI message in the decoding order.

[0309] A value of 1 for nnpfa_target_base_flag indicates that the target NNPF is a base NNPF with the same nnpfc_id as nnpfa_target_id. A value of 0 for nnpfa_target_base_flag indicates that the target NNPF is an NNPF specified by the last NNPFC SEI message with the same nnpfc_id as nnpfa_target_id that precedes the first VCL NAL unit of the current picture in the decoding order, and is not a repetition of an NNPFC SEI message containing a base NNPF.

[0310] A value of 1 for nnpfa_cancel_flag can indicate that the persistence of the target NNPF, set by any previous NNPFA SEI message containing the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message, is cancelled. That is, the target NNPF now has the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message and does not need to be used anymore unless activated by another NNPFA SEI message with nnpfa_cancel_flag set to 0. A value of 0 for nnpfa_cancel_flag can indicate that nnpfa_persistence_flag and nnpfa_num_output_entries will follow.

[0311] The nnpfa_persistence_flag can currently indicate the persistence of the target NNPF for a given layer.

[0312] A value of 0 for nnpfa_persistence_flag indicates that the target NNPF can currently be used for post-processing filtering of only pictures.

[0313] A value of nnpfa_persistence_flag 1 can indicate that the target NNPF may be used for post-processing filtering of the current picture and all subsequent pictures of the current layer in the output order until one or more of the following conditions are true.

[0314] A new CLVS for the current layer is initiated. The bitstream is ending. Currently, the picture of the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message and whose nnpfa_cancel_flag is 1 will be output after the current picture in the output order.

[0315] The target NNPF does not have to be applied to subsequent pictures in the current layer associated with an NNPFA SEI message that currently has the same nnpfa_target_id and nnpfa_target_base_flag as the NNPFA SEI message and whose nnpfa_cancel_flag is 1. In other words, if the nnpfa_persistence_flag value is 1, the target NNPF may be used until the picture in the current layer is output immediately after the current picture, based on the fact that the nnpfa_cancel_flag value in the NNPFA SEI message that currently has the same nnpfa_target_id and nnpfa_target_base_flag as the SEI message is 1.

[0316] nnpfcTargetPictures can be set as the set of pictures associated with the last NNPFC SEI message that has the same nnpfc_id as the nnpfa_target_id that currently precedes the NNPFA SEI message in the decoding order. nnpfaTargetPictures can be set as the set of pictures whose target NNPF is currently activated by the NNPFA SEI message. A bitstream conformance requirement is that all pictures included in nnpfaTargetPictures should also be included in nnpfcTargetPictures.

[0317] Example 2 Example 2 provides an explanation for the summaries 1 and 3 described above. The table may be updated as follows:

[0318] [Table 24] The NNPFA SEI message can activate or deactivate the possible use of the target NNPF, identified by nnpfa_target_id and nnpfa_target_base_flag, for post-processing filtering of the picture set. Similar to Example 1, nnpfa_target_base_flag can be parsed without the condition of the nnpfa_cancel_flag value, as shown in Table 24. Thus, for a particular picture in which the NNPF is activated, the target NNPF may be induced as follows:

[0319] If nnpfa_target_base_flag is the same as 1, the target NNPF can be the base NNPF where nnpfc_id is the same as nnpfa_target_id.

[0320] Otherwise (where nnpfa_target_base_flag is the same as 0), the target NNPF is the NNPF specified by the last NNPF SEI message that has the same nnpfc_id as nnpfa_target_id preceding the first VCL NAL unit of the current picture in the decoding order, and does not have to be a repetition of NNPFC SEI messages containing the base NNPF.

[0321] For example, if NNPF means filtering for other purposes or other hue components, then there may be several NNPFA SEI messages present for the same picture.

[0322] nnpfa_target_id can currently indicate a target NNPF indicated by one or more NNPFC SEI messages that have the same nnpfc_id as nnpfa_target_id in relation to the picture. nnpfa_target_id values ​​are 0-2 32 It must exist within the range of -2.

[0323] An NNPFA SEI message with a specific value for nnpfa_target_id should not currently exist in the PU unless one or two of the following conditions are all true:

[0324] Currently, within the CLVS, there may be NNPFC SEI messages that have the same nnpfc_id as a specific value of nnpfa_target_id in a PU that precedes the current PU in the decoding order.

[0325] Currently, it is possible for an NNPFC SEI message to exist in the PU that has an nnpfc_id that is identical to a specific value of nnpfa_target_id.

[0326] If the PU includes an NNPFC SEI message having a specific value for nnpfc_id and an NNPFA SEI message having the same nnpfa_target_id as the aforementioned specific value for nnpfc_id, the NNPFC SEI message must precede the NNPFA SEI message in the decoding order.

[0327] A value of 1 for nnpfa_target_base_flag indicates that the target NNPF is a base NNPF with the same nnpfc_id as nnpfa_target_id. A value of 0 for nnpfa_target_base_flag indicates that the target NNPF is an NNPF specified by the last NNPFC SEI message with the same nnpfc_id as nnpfa_target_id that precedes the first VCL NAL unit of the current picture in the decoding order, and is not a repetition of an NNPFC SEI message containing a base NNPF.

[0328] A value of 1 for nnpfa_cancel_flag can indicate that the persistence of the target NNPF, set by any previous NNPFA SEI message containing the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message, is cancelled. That is, the target NNPF now has the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message and does not need to be used anymore unless activated by another NNPFA SEI message with nnpfa_cancel_flag set to 0. A value of 0 for nnpfa_cancel_flag can indicate that nnpfa_persistence_flag and nnpfa_num_output_entries will follow.

[0329] The nnpfa_persistence_flag can currently indicate the persistence of the target NNPF for a given layer.

[0330] A value of 0 for nnpfa_persistence_flag indicates that the target NNPF can currently be used for post-processing filtering of only pictures.

[0331] A value of nnpfa_persistence_flag 1 can indicate that the target NNPF may be used for post-processing filtering of the current picture and all subsequent pictures of the current layer in the output order until one or more of the following conditions are true.

[0332] A new CLVS for the current layer is initiated. The bitstream is ending. Currently, the picture of the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message is output after the current picture in the output order.

[0333] The target NNPF does not have to be applied to subsequent pictures in the current layer related to an NNPFA SEI message that currently has the same nnpfa_target_id and nnpfa_target_base_flag as the NNPFA SEI message. In other words, if the nnpfa_persistence_flag value is 1, the target NNPF can be used until the picture in the current layer is output after the current picture in the output order, regardless of the value of nnpfa_cancel_flag included in the NNPFA SEI message that currently has the same nnpfa_target_id and nnpfa_target_base_flag as the SEI message.

[0334] nnpfcTargetPictures can be configured to be the set of pictures associated with the last NNPFC SEI message that has the same nnpfc_id as the nnpfa_target_id that currently precedes the NNPFA SEI message in the decoding order. nnpfaTargetPictures can be configured to be the set of pictures whose target NNPF is currently activated by the NNPFA SEI message. A bitstream conformance requirement is that all pictures included in nnpfaTargetPictures should also be included in nnpfcTargetPictures.

[0335] Example 3 Example 3 provides an explanation for the summary 4 described above. The table may be updated as follows:

[0336] [Table 25] The NNPFA SEI message can activate or deactivate the possible use of the target NNPF, identified by nnpfa_target_id and nnpfa_target_base_flag, for post-processing filtering of a set of pictures. For a specific picture in which the NNPF has been activated, the target NNPF may be induced as follows:

[0337] If nnpfa_target_base_flag is the same as 1, the target NNPF can be the base NNPF where nnpfc_id is the same as nnpfa_target_id.

[0338] Otherwise (where nnpfa_target_base_flag is the same as 0), the target NNPF is the NNPF specified by the last NNPF SEI message that has the same nnpfc_id as nnpfa_target_id preceding the first VCL NAL unit of the current picture in the decoding order, and does not have to be a repetition of NNPFC SEI messages containing the base NNPF.

[0339] For example, if NNPF means filtering for other purposes or other hue components, there may be several NNPFA SEI messages present for the same picture.

[0340] nnpfa_target_id can currently indicate the target NNPF, which is explicitly shown by one or more NNPFC SEI messages that have the same nnpfc_id as nnpfa_target_id in relation to the picture. The nnpfa_target_id value is 0 to 2 32 It must exist within the range of -2.

[0341] An NNPFA SEI message with a specific value for nnpfa_target_id should not currently exist in the PU unless one or two of the following conditions are all true:

[0342] Currently, within the CLVS, there may be an NNPFC SEI message with an nnpfc_id that is identical to a specific value of nnpfa_target_id in a PU that precedes the current PU in the decoding order.

[0343] Currently, there may be NNPFC SEI messages in the PU that have an nnpfc_id that is identical to a specific value of nnpfa_target_id.

[0344] If the PU includes an NNPFC SEI message having a specific value for nnpfc_id and an NNPFA SEI message having the same nnpfa_target_id as the aforementioned specific value for nnpfc_id, the NNPFC SEI message must precede the NNPFA SEI message in the decoding order.

[0345] A value of 1 for nnpfa_cancel_flag can indicate that the persistence of the target NNPF, set by any previous NNPFA SEI message containing the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message, is cancelled. That is, the target NNPF now has the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message and does not need to be used anymore unless activated by another NNPFA SEI message with nnpfa_cancel_flag set to 0. A value of 0 for nnpfa_cancel_flag can indicate that nnpfa_persistence_flag, nnpfa_persistence_flag, and nnpfa_num_output_entries will follow.

[0346] A value of 1 for nnpfa_target_base_flag indicates that the target NNPF is a base NNPF with the same nnpfc_id as nnpfa_target_id. A value of 0 for nnpfa_target_base_flag indicates that the target NNPF is an NNPF specified by the last NNPFC SEI message with the same nnpfc_id as nnpfa_target_id that precedes the first VCL NAL unit of the current picture in the decoding order, and is not a repetition of an NNPFC SEI message containing a base NNPF.

[0347] Regardless of the value of nnpfa_target_base_flag, there is a constraint that there cannot be two NNPFA SEI message activations for the same picture (i.e., one containing the same nnpfa_cancel_flag as 0).

[0348] The nnpfa_persistence_flag can currently indicate the persistence of the target NNPF for a given layer.

[0349] A value of 0 for nnpfa_persistence_flag indicates that the target NNPF can currently be used for post-processing filtering of only pictures.

[0350] A value of nnpfa_persistence_flag 1 can indicate that the target NNPF may be used for post-processing filtering of the current picture and all subsequent pictures of the current layer in the output order until one or more of the following conditions are true.

[0351] A new CLVS for the current layer is initiated. The bitstream is ending. Currently, the picture of the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id and nnpfa_target_base_flag as the current SEI message and whose nnpfa_cancel_flag is 1 will be output after the current picture in the output order.

[0352] The target NNPF does not need to be applied to subsequent pictures in the current layer associated with an NNPFA SEI message that currently has the same nnpfa_target_id and nnpfa_target_base_flag as the NNPFA SEI message and whose nnpfa_cancel_flag is 1.

[0353] nnpfcTargetPictures can be set as the set of pictures associated with the last NNPFC SEI message that has the same nnpfc_id as the nnpfa_target_id that currently precedes the NNPFA SEI message in the decoding order. nnpfaTargetPictures can be set as the set of pictures whose target NNPF is currently activated by the NNPFA SEI message. A bitstream conformance requirement is that all pictures included in nnpfaTargetPictures should also be included in nnpfcTargetPictures.

[0354] Video encoding method and video decoding method The following describes the video encoding method and video decoding method according to various embodiments of the present invention.

[0355] Figure 6 is a diagram illustrating the video encoding method of the video encoding device, and Figure 7 is a diagram illustrating the video decoding method of the video decoding device.

[0356] Referring to Figure 6, the video encoding device 200 can encode information for at least one neural network that can be used as a post-processing filter into at least one NNPFC SEI message (S610). In other words, the video encoding device 200 can determine at least one neural network that can be used as a post-processing filter, and encode information for the determined at least one neural network into at least one NNPFC SEI message.

[0357] The video encoding device 200 can encode information about the target NNPF, for which activation has been determined, into an NNPFA SEI message (S620). In other words, the video encoding device 200 can determine whether or not activation is required for the target NNPF that can be applied to the picture, and encode information about the determined target NNPF into an NNPFA SEI message.

[0358] The process for determining whether or not to activate a target NNPF may include a process for determining a target NNPF that can be applied to the picture at the moment, a process for determining whether or not to revoke the persistence of the target NNPF, and a process for determining the persistence of the target NNPF. For example, the video encoding device 200 can determine a target NNPF that can be applied to the picture at the moment and determine whether or not to revoke the persistence of the target NNPF based on the existence of a previous NNPFA SEI message having the same target NNPF identifier. If a previous NNPFA SEI message having the same target NNPF identifier exists, the video encoding device 200 may decide to revoke the persistence of the target NNPF specified by that NNPFA SEI message. If it is decided not to revoke the persistence of the target NNPF, the video encoding device 200 can determine the persistence of the target NNPF. Information about the target NNPF determined in this way may be encoded in an NNPFA SEI message.

[0359] The video encoding device 200 can encode post-filter coefficients or related information for the design of the post-filter into a post-filter hint SEI message (S630). NNPFC SEI messages, NNPFA SEI messages, and / or post-filter hint SEI messages may be included in NNPF SEI messages.

[0360] Referring to Figure 7, the video decoding device 300 can obtain SEI messages for the NNPF currently applied to the picture from the bitstream. SEI messages for the NNPF may include NNPFC SEI messages, NNPFA SEI messages, and / or post-filter hint SEI messages.

[0361] The video decoding device 300 can determine at least one neural network that can be used as a post-processing filter based on at least one NNPFC SEI message included in the SEI message for NNPF when the SEI message for NNPF is currently applied to a picture (S710).

[0362] The video decoding device 300 can determine whether or not to activate the target NNPF that can now be applied to the picture, based on at least one NNPFA SEI message obtained from the bitstream (S720).

[0363] The process for determining whether or not a target NNPF is activated may include processes for determining the target NNPF, determining whether or not the persistence of the target NNPF is revoked, and determining the persistence of the target NNPF. For example, the process for determining whether or not a target NNPF is activated may include processes for determining the target NNPF based on the NNPFA SEI message, determining whether or not the persistence of the target NNPF is revoked, and determining the persistence of the target NNPF. Here, the target NNPF identification information may include nnpfa_target_id, or it may include nnpfa_target_id and nnpfa_target_base_flag.

[0364] The target NNPF may be determined based on nnpfa_target_id, which is an identifier indicating the target NNPF. In various embodiments, the target NNPF may be determined based on nnpfa_target_id and nnpfa_target_base_flag.

[0365] nnpfa_target_id can now indicate a target NNPF that is explicitly identified by one or more NNPFC SEI messages that have the same filter identifier (i.e., nnpfc_id) as the target NNPF identifier associated with the picture.

[0366] The nnpfa_target_base_flag can indicate whether the target NNPF is a base NNPF having the same filter identifier as the target NNPF identifier, or an NNPF specified by the last NNPF SEI message with the same identifier preceding the first VCL NAL unit of the current picture in the decoding order. In other words, the target NNPF may be determined to be a base NNPF having the same filter identifier as the target NNPF identifier based on the target filter flag. Alternatively, the target NNPF may be determined to be an NNPF specified by the last NNPFC SEI message with the same filter identifier as the target NNPF identifier, preceding the first VCL NAL unit of the current picture in the decoding order, based on the target filter flag. In this case, the target NNPF does not have to be a repetition of the NNPFC SEI message containing the base NNPF.

[0367] The video decoding device 300 determines whether there is a previous NNPFA SEI message that has the same target NNPF identification information as the current NNPFA SEI message. If there is a previous NNPFA SEI message that has the same target NNPF identification information as the current NNPFA SEI message, it can determine whether or not the persistence of the target NNPF has been revoked. In this case, the video decoding device 300 can determine whether or not the persistence of the target NNPF has been revoked based on the persistence revocation flag, nnpfa_cancel_flag, which indicates whether or not the persistence of the target NNPF has been revoked.

[0368] The nnpfa_cancel_flag flag can indicate whether the persistence of a target NNPF, set by a previous NNPFA SEI message having the same target NNPF identifier as the current SEI message (e.g., the current NNPFA SEI message), can be revoked. For example, if the nnpfa_cancel_flag value is 1, the persistence of a target NNPF, set by a previous NNPFA SEI message containing the same target NNPF identifier and target filter flags as the current SEI message, may be revoked. In this case, the target NNPF does not need to be used again unless it is activated by another NNPFA SEI message that has the same target NNPF identifier and target filter flags as the current SEI message and has an nnpfa_cancel_flag value of 0. For example, if the nnpfa_cancel_flag value is 0, the nnpfa_persistence_flag and nnpfa_target_base_flag flags, which indicate the persistence of the target NNPF for the current layer, may be induced.

[0369] In various embodiments, if the nnpfa_cancel_flag value is 0, nnpfa_target_base_flag, nnpfa_persistence_flag, and nnpfa_num_output_entries may be induced.

[0370] The video decoding device 300 can determine the persistence of the target NNPF based on the nnpfa_persistence_flag value. For example, if nnpfa_persistence_flag is 0, the target NNPF may be used only for the current picture. For example, if nnpfa_persistence_flag is 1, the target NNPF may be used for the current picture and all subsequent pictures of the current layer in the output order. In this case, the target NNPF may be used for the current picture and all subsequent pictures of the current layer in the output order until one or more of the following conditions are met: a new CLVS of the current layer begins, the bitstream ends, or a picture of the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id and nnpfa_target_base_flag as the current NNPFA SEI message and whose nnpfa_cancel_flag is 1 is output after the current picture in the output order. In other words, the persistence of the target NNPF can be determined based on whether the picture of the current layer associated with the NNPFA SEI message, which currently has the same nnpfa_target_id and nnpfa_target_base_flag as the NNPFA SEI message and whose nnpfa_cancel_flag is 1, is output after the current picture in the output order.

[0371] In various embodiments, if nnpfa_persistence_flag is 1, the target NNPF can be used for the current picture and all subsequent pictures of the current layer until the picture of the current layer associated with the NNPFA SEI message that has the same nnpfa_target_id and nnpfa_target_base_flag as the current NNPFA SEI message is output after the current picture in the output order. In other words, the persistence of the target NNPF can be determined based on whether the picture of the current layer associated with the NNPFA SEI message that has the same nnpfa_target_id and nnpfa_target_base_flag as the current NNPFA SEI message is output after the current picture in the output order, regardless of the nnpfa_cancel_flag value included in the NNPFA SEI message associated with the current layer picture.

[0372] According to various implementations, regardless of the nnpfa_target_base_flag value, two or more NNPFA SEI messages for the same picture do not need to be activated.

[0373] According to various embodiments, the target NNPF does not necessarily have to be applied to subsequent pictures of the current layer associated with an NNPFA SEI message that currently has the same nnpfa_target_id and nnpfa_cancel_flag as the SEI message.

[0374] Figure 8 is a diagram illustrating a video decoding method for an example in which the activation status of the target NNPF is determined.

[0375] Referring to Figure 8, the video decoding device 300 can determine the target NNPF based on the NNPFA SEI message (S800). For example, the video decoding device 300 can determine the target NNPF based on the target NNPF identification information (i.e., nnpfa_target_id) of the NNPFA SEI message. For example, the video decoding device 300 can determine the target NNPF specified by one or more NNPFC SEI messages that currently have the same nnpfc_id value as the nnpfa_target_id value in relation to the picture.

[0376] The video decoding device 300 can determine whether or not to cancel the persistence of the target NNPF (S810). For example, the video decoding device 300 can determine whether or not to cancel the persistence of the target NNPF based on a persistence cancellation flag (i.e., nnpfa_cancel_flag) that indicates whether or not to cancel the persistence of the target NNPF.

[0377] For example, if the persistence cancellation flag value is 1, the video decoding device 300 can decide to cancel the persistence of the target NNPF set by any previous NNPFA SEI message that has the same target NNPF identification information as the current SEI message. In other words, the target NNPF does not need to be used anymore unless it is activated by another NNPFA SEI message with the same target NNPF identification information as the current SEI message but with a persistence cancellation flag value of 0. For example, if the persistence cancellation flag value is 0, the video decoding device 300 can decide not to cancel the persistence of the target NNPF.

[0378] The video decoding device 300 can determine the persistence of the target NNPF based on whether or not to cancel the persistence of the target NNPF (S830). For example, if the persistence of the target NNPF is not canceled, the video decoding device 300 can decide to use the target NNPF only for post-processing filtering on the current picture, or it can decide to use the target NNPF for post-processing filtering on the current picture and all subsequent pictures in the current layer in output order until the aforementioned conditions are met.

[0379] Thus, once the persistence of the target NNPF is determined, the video decoding device 300 can decide to activate or maintain the target NNPF.

[0380] Figures 9 to 11 are diagrams illustrating various examples of whether or not the persistence of the target NNPF is revoked.

[0381] As shown in Figure 9, if the current NNPFA SEI message has an nnpfa_cancel_flag value of 1, then if there is a previous NNPFA SEI message with the same target NNPF identification information as the current NNPFA SEI message (i.e., an nnpfa_target_id value of 1 and an nnpfa_target_base_flag value of 1), the persistence for the target NNPF specified by the NNPFC SEI message with an nnpfc_id value of 1 and an nnpfc_base_flag value of 1 may be revoked.

[0382] As shown in Figure 10, if there is a previous NNPFA SEI message that has the same target NNPF identification information as the current NNPFA SEI message (i.e., the nnpfa_target_id value is 1 and the nnpfa_target_base_flag value is 1), the persistence for the target NNPF may be revoked regardless of the nnpfa_cancel_flag value included in the current NNPFA SEI message.

[0383] As shown in Figure 11, if there is a previous NNPFA SEI message that has the same target NNPF identification information as the current NNPFA SEI message (i.e., the nnpfa_target_id value is 1), the persistence for the target NNPF may be revoked.

[0384] Figure 12 illustrates a content streaming system structure to which the embodiments described in this document apply.

[0385] The content streaming system to which the embodiments described herein apply may broadly include an encoding server, a streaming server, a web server, a media storage facility, user equipment, and multimedia input devices.

[0386] The encoding server is responsible for compressing content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and then transmitting this bitstream to the streaming server. In other cases, if a multimedia input device such as a smartphone, camera, or camcorder directly generates the bitstream, the encoding server can be omitted.

[0387] The bitstream can be generated by an encoding method or bitstream generation method to which an embodiment of this document applies, and the streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0388] The streaming server transmits multimedia data to user devices based on user requests via a web server, and the web server acts as an intermediary to inform users about available services. When a user requests a desired service from the web server, the web server transmits this to the streaming server, and the streaming server transmits multimedia data to the user. In this case, the content streaming system may include a separate control server, in which case the control server controls the commands / responses between each device within the content streaming system.

[0389] The streaming server can receive content from a media storage and / or encoding server. For example, if it starts receiving content from the encoding server, it can receive the content in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0390] Examples of user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (such as smartwatches, smart glasses, and HMDs), digital TVs, desktop computers, and digital signage.

[0391] Each server within the aforementioned content streaming system can be operated as a distributed server, in which case the data received by each server can be processed in a distributed manner.

[0392] The scope of this disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable operation by various embodiments to be performed on a device or computer, and non-transitory computer-readable medium on which such software or instructions etc. are stored and executable on a device or computer.

[0393] [Industrial applicability] The embodiments described herein can be used for encoding / decoding images.

[0394] [Claims when filing an international application] [Claim 1] A video decoding method performed by a video decoding device, The NNPF (neural-network post-filter) stage involves acquiring SEI (supplemental enhancement information) messages; The aforementioned NNPF SEI message includes NNPFC (neural-network post-filter characteristics) SEI messages and NNPFA (neural-network post-filter activation) SEI messages. The step of determining a neural network that can be used as a post-processing filter based on the aforementioned NNPFC SEI message; and The step of determining whether or not to activate the target NNPF that can be applied to the current picture based on the NNPFA SEI message; The aforementioned target NNPF is used to perform post-processing filtering on the current picture and subsequent pictures of the current layer in the output order until the picture of the current layer associated with an NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order. [Claim 2] The step of determining whether or not the target NNPF is activated is: The step of determining the target NNPF; The step of determining whether or not to revoke the persistence of the aforementioned NNPF; and The video decoding method according to claim 1, further comprising the step of determining the persistence of the target NNPF based on not canceling the persistence of the target NNPF. [Claim 3] The video decoding method according to claim 1, wherein the target NNPF is determined based on target NNPF identification information indicating a target NNPF specified by one or more NNPFC SEI messages having the same NNPF identification information as the NNPFA SEI message in relation to the current picture. [Claim 4] The video decoding method according to claim 2, wherein the presence or absence of persistence cancellation of the target NNPF is determined based on a persistence cancellation flag indicating the presence or absence of persistence cancellation of the target NNPF, which is set by a previous NNPFA SEI message having the same target NNPF identification information as the current SEI message. [Claim 5] The video decoding method according to claim 2, wherein the persistence of the target NNPF is determined based on whether the picture of the current layer is output next to the current picture in the output order, regardless of the value of the persistence cancellation flag in the NNPFA SEI message associated with the picture of the current layer. [Claim 6] A video encoding method performed by a video encoding device, The step of encoding a neural network that can be used as a post-processing filter into an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message; and The process currently includes the step of encoding whether or not the target NNPF that can be applied to the picture is activated into an NNPFA (neural-network post-filter activation) SEI message; The aforementioned target NNPF is used to perform post-processing filtering on the current picture and subsequent pictures within the current layer in the output order until the picture in the current layer associated with the NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order. [Claim 7] Whether or not the target NNPF is activated is, Determine the target NNPF as described above, Determine whether the persistence of the aforementioned NNPF is revoked. The video encoding method according to claim 6, wherein the persistence of the target NNPF is determined by determining the persistence of the target NNPF on the basis that the persistence of the target NNPF is not revoked. [Claim 8] The video coding method according to claim 6, wherein the target NNPF is determined based on target NNPF identification information indicating a target NNPF specified by one or more NNPFC SEI messages having the same NNPF identification information as the NNPFA SEI message in relation to the current picture. [Claim 9] The video encoding method according to claim 7, wherein the presence or absence of persistence cancellation of the target NNPF is determined based on a persistence cancellation flag indicating the presence or absence of persistence cancellation of the target NNPF, which is set by a previous NNPFA SEI message having the same target NNPF identification information as the current SEI message. [Claim 10] The video encoding method according to claim 7, wherein the persistence of the target NNPF is determined based on whether the picture of the current layer is output after the current picture in the output order, regardless of the value of the persistence cancellation flag in the NNPFA SEI message associated with the picture of the current layer. [Claim 11] A computer-readable recording medium for storing a bitstream generated by a video encoding method, The aforementioned bitstream is At least one NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message indicating a neural network that can be used as a post-processing filter, and It includes an NNPFA (neural-network post-filter activation) SEI message indicating whether or not the target neural network post-processing filter that can be applied to the picture is activated, The target NNPF is a computer-readable recording medium used to perform post-processing filtering on the current picture and subsequent pictures within the current layer in the output order until the picture in the current layer associated with the NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order. [Claim 12] A transmission method for transmitting data for video, A step of generating a bitstream for the aforementioned video; The bitstream is generated by encoding a neural network that can be used as a post-processing filter into an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message, and by encoding whether or not the target neural network post-processing filter that can be applied to the picture is activated into an NNPFA (neural-network post-filter activation) SEI message. The step of transmitting the data including the bitstream; A transmission method in which the target NNPF is used to perform post-processing filtering on the current picture and subsequent pictures in the current layer in the output order until the picture in the current layer associated with the NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order.

Claims

1. A video decoding method performed by a video decoding device, The stage of acquiring NNPF (Neural Network Post-Filter) SEI (Supplemental Enhancement Information) messages; The aforementioned NNPF SEI message includes an NNPFC (neutral-network post-filter characteristics) SEI message and an NNPFA (neutral-network post-filter activation) SEI message. The step of determining a neural network that can be used as a post-processing filter based on the aforementioned NNPFC SEI message; and The step of determining whether or not to activate the target NNPF that can be applied to the current picture based on the NNPFA SEI message; The aforementioned target NNPF is used for post-processing filtering of the current picture and subsequent pictures of the current layer in the output order until the picture of the current layer associated with the NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order.

2. The step of determining whether or not the target NNPF is activated is: The step of determining the target NNPF; The step of determining whether or not the persistence of the aforementioned NNPF is revoked; and The video decoding method according to claim 1, further comprising the step of determining the persistence of the target NNPF based on not canceling the persistence of the target NNPF.

3. The video decoding method according to claim 1, wherein the target NNPF is determined based on target NNPF identification information indicating a target NNPF specified by one or more NNPFC SEI messages having the same NNPF identification information as the NNPFA SEI message in relation to the current picture.

4. The video decoding method according to claim 2, wherein whether or not the persistence of the target NNPF is revoked is determined based on a persistence revocation flag indicating whether or not the persistence of the target NNPF is revoked, which is set by a previous NNPFA SEI message having the same target NNPF identification information as the current SEI message.

5. The video decoding method according to claim 2, wherein the persistence of the target NNPF is determined based on whether the picture of the current layer is output after the current picture in the output order, regardless of the value of the persistence cancellation flag in the NNPFA SEI message associated with the picture of the current layer.

6. A video encoding method performed by a video encoding device, The step of encoding a neural network that can be used as a post-processing filter into an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message; and The process includes the step of encoding the activation status of the target NNPF that can be applied to the picture into an NNPFA (neural-network post-filter activation) SEI message; The aforementioned target NNPF is used for post-processing filtering of the current picture and subsequent pictures in the current layer in the output order until the picture in the current layer associated with the NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order.

7. Whether or not the aforementioned target NNPF is activated is, Determine the target NNPF mentioned above, Determine whether the persistence of the aforementioned NNPF is revoked. The video encoding method according to claim 6, wherein the persistence of the target NNPF is determined by determining the persistence of the target NNPF on the basis that the persistence of the target NNPF is not revoked.

8. The video encoding method according to claim 6, wherein the target NNPF is determined based on target NNPF identification information indicating a target NNPF specified by one or more NNPFC SEI messages having the same NNPF identification information as the NNPFA SEI message in relation to the current picture.

9. The video encoding method according to claim 7, wherein whether or not the persistence of the target NNPF is revoked is determined based on a persistence revocation flag indicating whether or not the persistence of the target NNPF is revoked, which is set by a previous NNPFA SEI message having the same target NNPF identification information as the current SEI message.

10. The video encoding method according to claim 7, wherein the persistence of the target NNPF is determined based on whether the picture of the current layer is output next to the current picture in the output order, regardless of the value of the persistence cancellation flag in the NNPFA SEI message associated with the picture of the current layer.

11. A computer-readable recording medium for storing a bitstream generated by a video encoding method, The aforementioned bitstream is At least one NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message that shows a neural network that can be used as a post-processing filter, and It includes an NNPFA (neural-network post-filter activation) SEI message indicating whether or not the target neural network post-processing filter that can be applied to the picture is activated, The aforementioned target NNPF is a computer-readable recording medium used to perform post-processing filtering on the current picture and subsequent pictures within the current layer in the output order until the picture of the current layer associated with the NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order.

12. A transmission method for transmitting data for video, A step of generating a bitstream for the aforementioned video; The bitstream is generated based on encoding a neural network that can be used as a post-processing filter into an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message, and encoding whether or not the target neural network post-processing filter that can be applied to the picture is activated into an NNPFA (neural-network post-filter activation) SEI message. The step of transmitting the data, including the bitstream; A transmission method in which the target NNPF is used to perform post-processing filtering on the current picture and subsequent pictures in the current layer in the output order until the picture in the current layer associated with the NNPFA SEI message having the same target NNPF identification information as the current SEI message is output after the current picture in the output order.

Images