Video encoding / decoding method, recording medium storing a bitstream, and method for transmitting a bitstream.

Abstract

A video decoding method relating to one aspect of this disclosure is a video decoding method performed by a video decoding device, comprising the steps of: acquiring an NNPFC (neural-network post-filter characteristics) SEI message and an NNPFA (neural-network post-filter activation) SEI message; determining at least one neural network that can be used as a neural-network post-processing filter based on the NNPFC SEI message; and determining whether or not to activate a target neural network post-processing filter that can now be applied to a picture based on the NNPFA SEI message, wherein the NNPFA SEI message includes target identification information and target basic flag information for the target neural network post-processing filter, the target neural network post-processing filter is determined based on the target identification information and the target basic flag information, and based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information is in the decoding order of the NNPFA This could be a video decoding method that existed before SEI.

Description

[Technical Field]

[0001] This disclosure relates to a video encoding / decoding method, a recording medium storing a bitstream, and a method for transmitting a bitstream, and more particularly to a method for determining whether or not to perform persistence cancellation of a neural network post-filter. [Background technology]

[0002] Recently, demand for high-resolution, high-quality video / image data, such as 4K or 8K or higher UHD (Ultra High Definition) video / image data, has been increasing in various fields. As video / image data becomes higher resolution and higher quality, the amount of information or bits transmitted increases relative to existing video / image data. Therefore, when transmitting video / image data using existing wired or wireless broadband lines, or storing video / image data using existing storage media, transmission and storage costs increase.

[0003] This necessitates highly efficient image compression technology to effectively transmit, store, and reproduce high-resolution, high-quality image information. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The purpose of this disclosure is to provide a video encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0005] Furthermore, this disclosure aims to provide a method for clearly determining which filters among the basic and updated filters are to be activated.

[0006] Furthermore, this disclosure aims to provide a method for preventing situations where it is unclear whether to activate the basic filter or the updated filter.

[0007] Furthermore, this disclosure aims to provide a non-temporary computer-readable recording medium for storing a bitstream generated by the video encoding method relating to this disclosure.

[0008] Furthermore, this disclosure aims to provide a non-temporary computer-readable recording medium that stores a bitstream received and decoded by the video decoding device relating to this disclosure and used for video restoration.

[0009] Furthermore, this disclosure aims to provide a method for transmitting a bitstream generated by the video encoding method relating to this disclosure.

[0010] The technical challenges that this disclosure seeks to address are not limited to those mentioned above, and any other technical challenges not mentioned can be clearly understood by a person with ordinary skill in the art to which this disclosure pertains from the following description. [Means for solving the problem]

[0011] A video decoding method relating to one aspect of the present disclosure is a video decoding method performed by a video decoding device, comprising the steps of: acquiring NNPFC (neural-network post-filter characteristics) SEI messages and NNPFA (neural-network post-filter activation) SEI messages; determining at least one neural network that can be used as a neural-network post-processing filter based on the NNPFC SEI messages; and determining whether or not to activate a target neural-network post-processing filter that can be applied to the picture based on the NNPFA SEI messages. The SEI message includes target identification information and target basic flag information for the target neural network post-processing filter, the target neural network post-processing filter is determined based on the target identification information and the target basic flag information, and the at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI in the decoding order, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, which may be a video decoding method.

[0012] A video encoding method relating to another aspect of this disclosure is a video encoding method performed by a video encoding device, comprising the steps of: encoding at least one neural network that can be used as a post-processing filter into an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message; and encoding whether or not a target neural network post-processing filter that can be applied to the picture now is activated into an NNPFA (neural-network post-filter activation) SEI message, wherein the NNPFA SEI message includes target identification information and target basic flag information for the target neural network post-processing filter, and based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI in the decoding order.

[0013] Computer-readable recording media relating to other aspects of this disclosure can store bitstreams generated by the video encoding method or apparatus of this disclosure.

[0014] A transmission method relating to yet another aspect of the present disclosure may transmit a bitstream generated by the video encoding method or apparatus of the present disclosure.

[0015] The features of this disclosure briefly summarized above are merely illustrative aspects of the detailed description of this disclosure described below and do not limit the scope of this disclosure. [Effects of the Invention]

[0016] According to the present disclosure, a video encoding / decoding method and apparatus with improved encoding / decoding efficiency may be provided.

[0017] Also, according to the present disclosure, the neural network post-filter applied among the neural network post-filters of many preceding NNPFC SEI messages may be clarified.

[0018] Also, according to the present disclosure, a non-transitory computer-readable recording medium for storing a bitstream generated by the video encoding method according to the present disclosure may be provided.

[0019] Also, according to the present disclosure, a non-transitory computer-readable recording medium for storing a bitstream received and decoded by the video decoding apparatus according to the present disclosure and used for video restoration may be provided.

[0020] Also, according to the present disclosure, a method for transmitting a bitstream generated by a video encoding method may be provided.

[0021] The effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present disclosure pertains from the following description.

Brief Description of the Drawings

[0022] [Figure 1] A drawing schematically showing a video coding system to which an embodiment according to the present disclosure may be applied. [Figure 2] A drawing schematically showing a video encoding apparatus to which an embodiment according to the present disclosure may be applied. [Figure 3] A drawing schematically showing a video decoding apparatus to which an embodiment according to the present disclosure may be applied. [Figure 4] Exemplarily showing a hierarchical structure for coded video / video. [Figure 5]This is a diagram illustrating the interleaved method for guiding luma channels. [Figure 6] This diagram illustrates potential problem scenarios that may arise when identifying the target NNPF to activate based on the NNPFA SEI message. [Figure 7] This is an example of how the embodiments described herein may be applied. [Figure 8] This is another example of how the embodiments described herein may be applied. [Figure 9] An example of a video encoding method to which the embodiments described herein may be applied is shown. [Figure 10] An example of a video decoding method to which the embodiments described herein may be applied is shown. [Figure 11] The embodiments of this disclosure demonstrate the process of determining the target NNPF. [Figure 12] The embodiments of this disclosure demonstrate the process of determining the target NNPF. [Figure 13] This drawing illustrates a content streaming system to which the embodiments of this disclosure may be applied. [Modes for carrying out the invention]

[0023] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings, so that they can be easily implemented by a person with ordinary skill in the art to which the present disclosure pertains. However, the present disclosure can be implemented in a variety of different forms and is not limited to the embodiments described herein.

[0024] In describing embodiments of this disclosure, if it is determined that a specific description of a known configuration or function would obscure the gist of this disclosure, such detailed description will be omitted. In the drawings, parts unrelated to the description of this disclosure will be omitted, and similar parts will be denoted by the same reference numerals.

[0025] In this disclosure, when one component is described as being “connected,” “joined,” or “linked” to another component, this can include not only direct connections but also indirect connections where another component exists between them. Furthermore, when one component is described as “containing” or “having” another component, this means, unless otherwise stated to the contrary, that it may include another component rather than excluding it.

[0026] In this disclosure, terms such as "first," "second," etc., are used solely for the purpose of distinguishing one component from another, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be called a second component in another embodiment, and similarly, a second component in one embodiment may be called a first component in another embodiment.

[0027] In this disclosure, components that are distinguished from each other are used to clearly describe their respective characteristics and do not necessarily mean that the components are separate. In other words, multiple components may be integrated to constitute a single hardware or software unit, or a single component may be distributed to constitute multiple hardware or software units. Therefore, such integrated or distributed embodiments are also included in the scope of this disclosure, without needing to be specifically mentioned.

[0028] In this disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments consisting of a subset of the components described in one embodiment are also included in the scope of this disclosure. Furthermore, embodiments that include additional components in addition to the components described in various embodiments are also included in the scope of this disclosure.

[0029] This disclosure relates to the encoding and decoding of images, and the terms used in this disclosure may have their ordinary meanings in the art to which this disclosure pertains, unless otherwise defined herein.

[0030] This disclosure presents various embodiments relating to video / video coding, and unless otherwise noted, these embodiments may be implemented in combination with each other.

[0031] Unless otherwise defined herein, terms used in this disclosure may have their ordinary meanings as they are common in the art to which this disclosure pertains.

[0032] In this disclosure, "picture" generally means a unit representing any one image within a specific time period, and a slice / tile is an encoded unit that constitutes part of a picture, and a single picture may consist of one or more slices / tiles. A slice / tile may also contain one or more CTUs (coding tree units). A single picture may consist of one or more tile groups. A single tile group may contain one or more tiles. A brick can represent a square area of ​​a CTU row of tiles within a picture. In this document, tile groups and slices can be combined. For example, in this document, a tile group / tile group header may be referred to as a slice / slice head.

[0033] In this disclosure, “pixel” or “pel” may mean the smallest unit that constitutes a picture (or image). The term “sample” may also be used as a counterpart to pixel. A sample may generally represent a pixel or a pixel value, or it may represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component.

[0034] In this disclosure, “unit” can refer to a basic unit of image processing. A unit may include at least one of a specific region of a picture and information associated with that region. A unit may be used interchangeably with terms such as “sample array,” “block,” or “area,” as it may be used. Generally, an M×N block may include a set (or array) of samples (or sample arrays) or transform coefficients consisting of M columns and N rows.

[0035] In this disclosure, “current block” can mean any one of the following: “current coding block,” “current coding unit,” “block to encode,” “block to decode,” or “block to process.” If prediction is performed, “current block” can mean “current prediction block” or “block to predict.” If transformation (inverse transformation) / quantization (inverse quantization) is performed, “current block” can mean “current transformation block” or “block to transform.” If filtering is performed, “current block” can mean “block to filter.”

[0036] Furthermore, in this disclosure, "current block" may mean the block containing all luma component blocks and chroma component blocks, or the "luma block of the current block," unless there is an explicit mention of a chroma block. The chroma block of the current block may be expressed with an explicit mention of a chroma block, such as "chroma block" or "current chroma block."

[0037] In this disclosure, " / " and "," may be interpreted as "and / or." For example, "A / B" and "A, B" may be interpreted as "A and / or B." Also, "A / B / C" and "A, B, C" may mean "at least one of A, B and / or C."

[0038] In this disclosure, “or” may be interpreted as “and / or.” For example, “A or B” may mean 1) “A” only, 2) “B” only, or 3) “A and B.” Alternatively, in this disclosure, “or” may mean “additionally or alternatively.”

[0039] Overview of the video coding system Figure 1 shows the video coding system according to this disclosure.

[0040] A video coding system according to one embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 can transmit encoded video and / or image information or data to the decoding device 20 via a digital storage medium or network in file or streaming format.

[0041] An encoding device 10 according to one embodiment may include a video source generation unit 11, an encoding unit 12, and a transmission unit 13. A decoding device 20 according to one embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 may be called a video / image encoding unit, and the decoding unit 22 may be called a video / image decoding unit. The transmission unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may also include a display unit, which may be configured as a separate device or external component.

[0042] The video source generation unit 11 can acquire video / images through processes such as video / image capture, synthesis, or generation. The video source generation unit 11 may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, or a video / image archive containing previously captured video / images. The video / image generation device may include, for example, a computer, tablet, and smartphone, and may generate video / images (electronically). For example, virtual video / images may be generated via a computer, in which case the video / image capture process may be replaced by a process in which the relevant data is generated.

[0043] The encoding unit 12 can encode the input video / image. The encoding unit 12 can perform a series of steps such as prediction, transformation, and quantization for compression and encoding efficiency. The encoding unit 12 can output the encoded data (encoded video / image information) in bitstream format.

[0044] The transmission unit 13 can transmit encoded video / image information or data, output in bitstream format, to the receiving unit 21 of the decoding device 20 via a digital storage medium or network in file or streaming format. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray®, HDD, and SSD. The transmission unit 13 may include elements for generating media files via a predetermined file format and elements for transmission via a broadcast / communication network. The receiving unit 21 can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit 22.

[0045] The decoding unit 22 can decode the video / image by performing a series of steps such as inverse quantization, inverse transform, and prediction, corresponding to the operation of the encoding unit 12.

[0046] The rendering unit 23 can render the decoded video / image. The rendered video / image can be displayed via the display unit.

[0047] Video Encoding Device Overview Figure 2 is a schematic diagram illustrating the configuration of a video / image encoding device to which the embodiments described in this document may be applied.

[0048] As shown in Figure 2, the encoding device 200 can be configured to include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-predictor 221 and an intra-predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be called a reconstructor or a reconstructed block generator. The aforementioned image segmentation unit 210, prediction unit 220, residual processing unit 230, entropy encoding unit 240, addition unit 250, and filtering unit 260 can be configured by one or more hardware components (e.g., an encoder chipset or processor) depending on the embodiment. Furthermore, the memory 270 may include a DPB (decoded picture buffer) and may be configured by a digital recording medium. The hardware components may also further include the memory 270 as an internal / external component.

[0049] The image splitting unit 210 can split an input image (or picture, frame) input to the encoding device 200 into one or more processing units. For example, one of these processing units may be called a coding unit (CU). In this case, a coding unit can be recursively split from a coding tree unit (CTU) or the largest coding unit (LCU) using a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, one coding unit can be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, followed by the binary-tree structure and / or the ternary structure. Alternatively, the binary-tree structure may be applied first. The coding procedure described in this document can be executed based on the final coding unit that cannot be further split. In this case, based on coding efficiency according to image characteristics, the largest coding unit can be immediately used as the final coding unit, or, if necessary, the coding unit can be recursively divided into lower-depth coding units so that the optimally sized coding unit is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further comprise a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit can each be separated or partitioned from the final coding unit described above.The prediction unit is a unit of sample prediction, and the conversion unit is a unit that derives a conversion coefficient and / or a unit that derives a residual signal from the conversion coefficient.

[0050] The encoding device 200 can generate a residual signal (residual block, residual sample array) by subtracting the prediction signal (predicted block, predicted sample array) output from the inter-prediction unit 221 or intra-prediction unit 222 from the input image signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as shown in the figure, the unit that subtracts the prediction signal (predicted block, predicted sample array) from the input image signal (original block, original sample array) within the encoder 200 can be called the subtraction unit 231. The prediction unit can perform a prediction for the block to be processed (hereinafter referred to as the current block) and generate a predicted block that includes the predicted sample for the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied on a current block or CU basis. The prediction unit can generate various prediction-related information, such as prediction mode information, and transmit it to the entropy encoding unit 240, as will be described later in the explanation of each prediction mode. The prediction information can be encoded by the entropy encoding unit 240 and output in bitstream format.

[0051] The intra-prediction unit 222 can predict the current block by referring to a sample in the current picture. The referenced sample can be located in the vicinity (neighbor) of the current block or at a distance, depending on the prediction mode. In intra-prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC mode and Planar mode. Directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is merely an example, and more or fewer directional prediction modes can be used depending on the settings. The intra-prediction unit 222 can also determine the prediction mode to be applied to the current block using the prediction modes applied to adjacent blocks.

[0052] The interprediction unit 221 can derive a predicted block relative to the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, in order to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between adjacent blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, adjacent blocks may include spatially adjacent blocks existing in the current picture and temporally adjacent blocks existing in the reference picture. The reference picture containing the reference block and the reference picture containing the temporally adjacent block may be the same or different. The temporally adjacent block may be called a collocated reference block, colCU, etc., and the reference picture containing the temporally adjacent block may be called a collocated picture (colPic). For example, the interpretation unit 221 can construct a motion information candidate list based on adjacent blocks and generate information indicating which candidates are used to derive the motion vector and / or reference picture index of the current block. Interpretation can be performed based on various prediction modes; for example, in skip mode and merge mode, the interpretation unit 221 can use the motion information of adjacent blocks as the motion information of the current block. In skip mode, unlike merge mode, a residual signal may not be transmitted.In motion vector prediction (MVP) mode, the motion vector of an adjacent block is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0053] The prediction unit 220 can generate prediction signals based on various prediction methods described later. For example, the prediction unit can apply intra-prediction or inter-prediction for a prediction of a single block, and can also apply intra-prediction and inter-prediction simultaneously. This can be called combined inter and intra prediction (CIIP). The prediction unit can also be based on an intra-block copy (IBC) prediction mode or a palette mode for predictions of blocks. The IBC prediction mode or palette mode can be used for content image / video coding such as in games, for example, as in SCC (screen content coding). IBC basically performs predictions within the current picture, but can be performed similarly to inter-prediction in that it derives reference blocks within the current picture. That is, IBC can utilize at least one of the inter-prediction techniques described in this document.

[0054] The predicted signal generated via the prediction unit 220 may be used to generate a reconstructed signal or to generate a residual signal. The subtraction unit 231 subtracts the predicted signal (predicted block, predicted sample array) output from the prediction unit 220 from the input video signal (original block, original sample array) to generate a residual signal (residual block, residual sample array). The generated residual signal is transmitted to the conversion unit 232.

[0055] The transformation unit 232 can generate transformation coefficients by applying transformation techniques to the residual signal. For example, the transformation technique may include at least one of the following: DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is represented by this graph. CNT refers to a transformation obtained by generating a prediction signal using all previously reconstructed pixels. Furthermore, the transformation process can be applied to pixel blocks of the same size and are square, or to non-square, variable-sized blocks.

[0056] The quantization unit 233 quantizes the conversion coefficients and transmits them to the entropy encoding unit 240, which can encode the quantized signal (information about the quantized conversion coefficients) and output it as a bitstream. The information about the quantized conversion coefficients can be called residual information. The quantization unit 233 can rearrange the block-shaped quantized conversion coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized conversion coefficients based on the one-dimensional vector form of the quantized conversion coefficients.

[0057] The entropy encoding unit 240 can perform various encoding methods, such as exponential Golomb, CAVLC (context-adaptive variable length coding), and CABAC (context-adaptive binary arithmetic coding). In addition to the quantized conversion coefficients, the entropy encoding unit 240 can also encode information necessary for video / image reconstruction (e.g., the values ​​of syntax elements) together with or separately. The encoded information (e.g., encoded video / image information) can be transmitted or stored in bitstream form in units of NAL (network abstraction layer) units. The video / image information may further include information about various parameter sets, such as the adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information. In this document, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in video / image information. The video / image information may be encoded via the encoding procedure described above and included in the bitstream.

[0058] The bitstream can be transmitted over a network or stored on a digital recording medium. Here, the network may include broadcast networks and / or communication networks, and the digital recording medium may include various recording media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit 240 can be transmitted by a transmitting unit (not shown) and / or stored by a storage unit (not shown) which are configured as internal / external elements of the encoding device 200, or the transmitting unit may be included in the entropy encoding unit 240.

[0059] The quantized conversion coefficients output from the quantization unit 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying inverse quantization and inverse transformation to the quantized conversion coefficients via the inverse quantization unit 234 and the inverse transformation unit 235.

[0060] The adder 250 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the predicted signal output from the inter-prediction unit 221 or the intra-prediction unit 222. If there is no residual for the block to be processed, as in the case of skip mode, the predicted block can be used as the reconstructed block. The adder 250 can be called the reconstructor unit or reconstructed block generator. The generated reconstructed signal can be used for intra-prediction of the next block to be processed in the current picture, or, as described later, for inter-prediction of the next picture after filtering.

[0061] The filtering unit 260 can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 260 can apply various filtering methods to the restored picture to generate a modified restored picture, and the modified restored picture can be stored in the memory 270, specifically in the DPB of the memory 270. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter. The filtering unit 260 can generate various filtering-related information and transmit it to the entropy encoding unit 240, as will be described later in the explanation of each filtering method. The filtering-related information can be encoded by the entropy encoding unit 240 and output in bitstream format.

[0062] The corrected restored picture sent to memory 270 can be used as a reference picture in the interpretation unit 221. When interpretation is applied via this, the encoding device can avoid prediction mismatches between the encoding device 200 and the decoding device 300, and can also improve encoding efficiency.

[0063] Memory 270DPB can store the corrected restored picture for use as a reference picture in the inter-prediction unit 221. Memory 270 can store motion information of blocks from which motion information has been derived (or encoded) in the current picture and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 221 for use as motion information of spatially adjacent blocks or motion information of temporally adjacent blocks. Memory 270 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 222.

[0064] Overview of the image decoding device Figure 3 is a schematic diagram illustrating the configuration of a video / image decoding device to which the embodiments described in this document may be applied.

[0065] Referring to Figure 3, the decoding device 300 can be configured to include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an intra-predictor 331 and an inter-predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. The aforementioned entropy decoder 310, residual processor 320, predictor 330, adder 340, and filtering device 350 can be configured by a single hardware component (e.g., a decoder chipset or processor) depending on the embodiment.

[0066] Furthermore, the memory 360 may include a DPB (decoded picture buffer) and may be composed of a digital storage medium. The hardware component may further include the memory 360 as an internal / external component. When a bitstream containing video / image information is input, the decoding device 300 can reconstruct the image in accordance with the process by which the video / image information was processed in the encoding device of Figure 2.

[0067] For example, the decoding device 300 can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device 300 can perform decoding using processing units applied in the encoding device. Thus, the decoding processing unit is, for example, a coding unit, which can be divided from a coding tree unit or a maximum coding unit according to a quad-tree structure, a binary tree structure, and / or a terminally tree structure. One or more conversion units can be derived from the coding unit. The decoded and output reconstructed image signal via the decoding device 300 can then be reproduced via a playback device.

[0068] When a bitstream containing video / image information is input, the decoding device 300 can restore the image by executing a process corresponding to the process executed in the encoding device 200 in Figure 2. For example, the decoding device 300 can perform decoding using the processing unit applied in the encoding device 200.

[0069] Therefore, the decoding processing unit may be, for example, a coding unit.

[0070] The coding unit may be a coding tree unit, or it may be obtained by dividing the maximum coding unit according to a quad-tree structure, a binary tree structure, and / or a ternary tree structure. One or more conversion units may be derived from the coding unit. The restored video signal decoded and output by the decoding device 300 can then be reproduced through a playback device (not shown).

[0071] The decoding device 300 can receive the signal output from the encoding device shown in Figure 2 in bitstream form, and the received signal can be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 can parse the bitstream to derive information necessary for image restoration (or picture restoration) (e.g., video / image information). The video / image information may further include information about various parameter sets, such as the adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information.

[0072] The decoding device can further decode the picture based on the information regarding the parameter set and / or the general restriction information. The signaling / receiving information and / or syntax elements described later in this document can be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 can decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements necessary for image restoration, quantized values ​​of conversion coefficients related to resistivity, and so on.

[0073] More specifically, the CABAC entropy decoding method receives bins corresponding to each syntactic element in a bitstream, determines a context model using the information of the syntactic element to be decoded, the decoded information of the surrounding and decoded blocks, or the symbol / bin information decoded in a previous step, predicts the probability of bin occurrence based on the determined context model, and performs arithmetic decoding of the bins to generate symbols corresponding to the values ​​of each syntactic element. At this time, after determining the context model, the CABAC entropy decoding method can update the context model using the decoded symbol / bin information for the context model of the next symbol / bin. Of the information decoded by the entropy decoding unit 310, information related to predictions is provided to the prediction unit (inter-prediction unit 332 and intra-prediction unit 331), and the residual values ​​for which entropy decoding was performed by the entropy decoding unit 310, i.e., quantized conversion coefficients and related parameter information, can be input to the residual processing unit 320.

[0074] The residual processing unit 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). Furthermore, information related to filtering from the information decoded by the entropy decoding unit 310 can be provided to the filtering unit 350. Meanwhile, a receiving unit (not shown) that receives signals output from the encoding device can be further configured as an internal / external element of the decoding device 300, or the receiving unit may be a component of the entropy decoding unit 310.

[0075] On the other hand, the decoding device relating to this document may be called a video / image / picture decoding device, and the decoding device may also be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit 310, and the sample decoder may include at least one of the inverse quantization unit 321, inverse transformation unit 322, addition unit 340, filtering unit 350, memory 360, inter-prediction unit 332, and intra-prediction unit 331.

[0076] The inverse quantization unit 321 can inverse quantize the quantized transformation coefficients and output the transformation coefficients. The inverse quantization unit 321 can rearrange the quantized transformation coefficients in a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit 321 can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) to obtain the transformation coefficients.

[0077] In the inverse conversion unit 322, the conversion coefficients are inversely converted to obtain a residual signal (residual block, residual sample array).

[0078] The prediction unit can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit 310, the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block and can determine a specific intra / inter-prediction mode.

[0079] The prediction unit 320 can generate a prediction signal based on various prediction methods described later.

[0080] The intra-prediction unit 331 can predict the current block by referring to a sample in the current picture. The referenced sample can be located in the vicinity (neighbor) of the current block or at a distance from it, depending on the prediction mode. In intra-prediction, the prediction mode can include a plurality of non-directional modes and a plurality of directional modes. The intra-prediction unit 331 can also determine the prediction mode to be applied to the current block using the prediction modes applied to adjacent blocks.

[0081] The interprediction unit 332 can derive a predicted block relative to the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, in order to reduce the amount of motion information transmitted from the interprediction mode, motion information can be predicted on a block, subblock, or sample basis based on the correlation of motion information between adjacent blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, adjacent blocks may include spatially adjacent blocks existing in the current picture and temporally adjacent blocks existing in the reference picture.

[0082] For example, the interpretation unit 332 can construct a motion information candidate list based on adjacent blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Interpretation can be performed based on various prediction modes, and the prediction information may include information indicating the mode of interpretation for the current block.

[0083] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the prediction unit (including the inter-prediction unit 332 and / or intra-prediction unit 331). If there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The adder 340 may be called the reconstruction unit or reconstructed block generation unit. The generated reconstructed signal can be used for intra-prediction of the next block to be processed in the current picture, and can be output after filtering as described later, or it can be used for inter-prediction of the next picture.

[0084] The filtering unit 350 can apply filtering to the restored signal to improve subjective / objective image quality. For example, the filtering unit 350 can apply various filtering methods to the restored picture to generate a modified restored picture, and can transmit the modified restored picture to the memory 360, specifically to the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter.

[0085] The (modified) restored picture stored in the DPB of memory 360 can be used as a reference picture by the inter-prediction unit 332. Memory 360 can store motion information of blocks from which motion information in the current picture has been derived (or decoded) and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 260 for use as motion information of spatially adjacent blocks or motion information of temporally adjacent blocks. Memory 360 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 331.

[0086] In this document, the embodiments described for the filtering unit 260, the inter-prediction unit 221, and the intra-prediction unit 222 of the encoding device 200 can also be applied identically or in a corresponding manner to the filtering unit 350, the inter-prediction unit 332, and the intra-prediction unit 331 of the decoding device 300, respectively.

[0087] Figure 4 illustrates the hierarchical structure for coded video / images.

[0088] Referring to Figure 4, coded video / images can be divided into the VCL (video coding layer), which handles the video / image decoding process and the video itself; a lower-level system that transmits and stores the coded information; and the NAL (network abstraction layer), which exists between the VCL and the lower-level system and is responsible for network adaptation functions.

[0089] For example, VCL can generate VCL data containing compressed image data (slice data), or it can generate a Picture Parameter Set (PPS), Sequence Parameter Set (SPS), Video Parameter Set (VPS), or a parameter set containing SEI (Supplemental Enhancement Information, SEI) messages additionally required during the video decoding process.

[0090] For example, in NAL, a NAL unit can be generated by adding header information (NAL unit header) to an RBSP (Raw Byte Sequence Payload) generated by VCL. In this case, the RBSP can refer to slice data, parameter sets, SEI messages, etc., generated by VCL. The NAL unit header can include NAL unit type information specified by the RBSP data contained in the NAL unit.

[0091] For example, as shown in Figure 4, NAL units can be classified into VCL NAL units and Non-VCL NAL units by the RBSP generated in VCL. VCL NAL units can represent NAL units containing information (slice data) for the image, while Non-VCL NAL units can represent NAL units containing information (parameter sets or SEI messages) necessary for image decoding.

[0092] The aforementioned VCL NAL units and Non-VCL NAL units can be transmitted over a network with header information attached according to the subsystem's data standard. For example, NAL units can be converted to predetermined standard data formats such as the H.266 / VVC file format, the Real-Time Transport Protocol (RTP), and the Transport Stream (TS), and transmitted over a variety of networks.

[0093] Furthermore, as mentioned above, the NAL unit type can be specified by the RBSP data structure contained within the NAL unit, and information regarding the NAL unit type can be stored in the NAL unit header and signaled.

[0094] For example, NAL units can be classified into VCL NAL unit types and Non-VCL NAL unit types depending on whether or not they contain information (slice data) related to the image. Furthermore, VCL NAL unit types can be classified by the characteristics and type of the picture contained within the VCL NAL unit, and Non-VCL NAL unit types can be classified by the type of parameter set.

[0095] The following is an example of a NAL unit type specified by the type of parameter set included in the Non-VCL NAL unit type.

[0096] APS (Adaptation Parameter Set) NAL Unit: Type for NAL units that include APS DPS (Decoding Parameter Set) NAL Unit: Type for NAL units that include DPS VPS (Video Parameter Set) NAL Unit: Type for NAL units that include VPS SPS (Sequence Parameter Set) NAL Unit: Type for NAL units that include SPS PPS (Picture Parameter Set) NAL Unit: Type for NAL units that include PPS The aforementioned NAL unit type may have syntax information for the NAL unit type, and this syntax information may be stored and signaled in the NAL unit header. For example, the syntax information may be nal_unit_type, and the NAL unit type may be specified in the nal_unit_type value.

[0097] On the other hand, a single picture can contain multiple slices, and a slice can contain a slice header and slice data. In this case, one picture header can be added to multiple slices (sets of slice headers and slice data). A picture header (picture header syntax) can contain information / parameters that can be commonly applied to pictures. A slice header (slice header syntax) can contain information / parameters that can be commonly applied to slices. An APS (APS syntax) or PPS (PPS syntax) can contain information / parameters that can be commonly applied to one or more slices or pictures. An SPS (SPS syntax) can contain information / parameters that can be commonly applied to one or more sequences. A VPS (VPS syntax) can contain information / parameters that can be commonly applied to multiple layers. A DPS (DPS syntax) can contain information / parameters that can be commonly applied to the entire video. A DPS can contain information / parameters related to the concatenation of coded video sequences (CVS).

[0098] In this document, the video information encoded from the encoding device to the decoding device and signaled in bitstream form may include not only partitioning-related information within the picture, intra / inter prediction information, interlayer prediction-related information, residual information, in-loop filtering information, etc., but may also include information contained in the slice header, the picture header, the APS, the PPS, the SPS, the VPS, and / or the DPS.

[0099] A common post-processing filtering procedure using a neural network post-filter. The input to this procedure is a bitstream, BitstreamToFilter, and the output is a list of NNPF output pictures, ListNnpfOutputPics. First, BitstreamToFilter is decoded, and the list CroppedDecodedPictures is set as a list of cropped and decoded pictures in output order as a result of decoding BitstreamToFilter. Secondly, the filtering process for a single picture resides in CroppedDecodedPictures and is repeatedly applied in output order to each cropped and decoded picture (one or more NNPF activated pictures). The order of the pictures in ListNnpfOutputPics is in the same order as the output.

[0100] There can be no more than one picture associated with a particular time instance in ListNnpfOutputPics. If multiple NNPFs are activated for a particular picture in CroppedDecodedPictures, and any of the NNPFs can be selected, but only one selection is allowed to be applied, then the above constraint may apply regardless of which NNPF is selected to be applied to that particular picture.

[0101] The filtering procedures described later reside in CroppedDecodedPictures and can be applied to each cropped and decoded picture (referred to as the current picture) with one or more NNPF activated. When NNPF is applied to the current picture, the filtered and / or interpolated picture is generated by NNPF by applying the NNPF procedures explicitly stated in the semantics of the NNPFC SEI message to the current picture on a patch-by-patch basis.

[0102] Currently, when applying NNPF to a picture, the order in which the pictures are generated by NNPF is the same as the output order, as the NNPF is stored in the NNPF output tensor. If the applied NNPF is the last NNPF applied to the picture, the pictures generated by NNPF and output by the NNPF procedure may be included in ListNnpfOutputPics in the same order in which the pictures are stored in the NNPF output tensor.

[0103] Neural network post-filter characteristics (NNPFC) The bonds in Tables 1 to 3 represent the NNPFC syntax structure.

[0104] [Table 1]

[0105] [Table 2]

[0106] [Table 3]

[0107] The NNPFC syntax structures shown in Tables 1 to 3 can be signaled in the form of SEI (supplemental enhancement information) messages. SEI messages that signal the NNPFC syntax structures shown in Tables 1 to 3 can be referred to as NNPFC SEI messages.

[0108] NNPFC SEI messages can identify neural networks that can be used as post-processing filters. The use of identified post-processing filters (NNPFs) for a particular picture can be indicated using neural-network post-filter activation SEI messages. Here, "post-processing filter" and "post-filter" can have the same meaning.

[0109] Using SEI messages like this may require defining variables such as those shown below.

[0110] - The input picture width and height in luma samples can be indicated by CroppedWidth and CroppedHeight, respectively.

[0111] CroppedYPic[idx], a chroma sample array of input pictures with index idx in the range of -0 to numInputPics-1, and CroppedCbPic[idx] and CroppedCrPic[idx], which are chroma sample arrays, can be used as input to NNPF if they exist.

[0112] -BitDepthY can indicate the bit depth relative to the luma sample array of the input picture.

[0113] -BitDepthC can indicate the bit depth of the chroma sample array (if any) of the input picture.

[0114] -ChromaFormatIdc can indicate a chroma format identifier.

[0115] If the value of -nnpfc_auxiliary_inp_idc is 1, the filtering strength control value array StrengthControlVal[idx] must contain real numbers in the range of 0 to 1 for input pictures with index idx in the range of 0 to numInputPics-1.

[0116] An input picture with index 0 can correspond to a picture whose NNPF, defined by the NNPFC SEI message, is activated by the NNPFA SEI message. An input picture with index i in the range of 0 to numInputPics-1 can precede an input picture with index i-1 in the output order.

[0117] The variables SubWidthC and SubHeightC may be derived from ChromaFormatIdc. Two or more NNPFC SEI messages can exist for the same picture. If two or more NNPFC SEI messages with different nnpfc_id values ​​exist for or are activated for the same picture, the two or more NNPFC SEI messages can have the same or different nnpfc_purpose and nnpfc_mode_idx values.

[0118] nnpfc_purpose can indicate the purpose of the NNPF as explicitly stated in Table 3. Here, a non-zero value for (nnpfc_purpose & bitMask) indicates that the NNPF has a purpose related to the bitMask value in Table 3. If nnpfc_purpose is greater than 0 and (nnpfc_purpose & bitMask) is 0, the purpose related to the bitMask value does not necessarily have to be applied to the NNPF. If nnpfc_purpose is 0, the NNPF may be used.

[0119] The nnpfc_purpose value must be in the range of 0 to 63 in the bitstream. Values ​​64 to 65535 for nnpfc_purpose may be reserved for future use and do not need to be present in the bitstream. The decoder must ignore NNPFC SEI messages containing npfc_purpose in the range of 64 to 65535.

[0120] [Table 4]

[0121] The variables chromaUpsamplingFlag, resolutionResamplingFlag, pictureRateUpsamplingFlag, bitDepthUpsamplingFlag, and colourizationFlag, which indicate whether nnpfc_purpose indicates an NNPF purpose including chroma upsampling, resolution resampling, picture rate upsampling, bit depth upsampling, and colourization, can be derived as shown in Table 5 below.

[0122] [Table 5]

[0123] If a reserved value for nnpfc_purpose is to be used in the future, the syntax of the SEI message may be extended to include syntax elements whose existence is determined by the nnpfc_purpose that matches the reserved value.

[0124] If ChromaFormatIdc is 3, then chromaUpsamplingFlag must be the same as 0.

[0125] If ChromaFormatIdc or chromaUpsamplingFlag is not 0, then colourizationFlag must be the same as 0.

[0126] If pictureRateUpsamplingFlag is 1 and the input picture at index 0 is associated with a frame packing arrangement SEI message having the same fp_arrangement_type as 5, then all input pictures may be associated with a frame packing arrangement SEI message having the same fp_arrangement_type as 5 and the same value of fp_current_frame_is_frame0_flag.

[0127] nnpfc_id can contain an identification number that can be used to identify a post-processing filter. nnpfc_id values ​​are 0-2 32 It must be within the range of -2. The range is 256~511 and 2 31 ~2 32 -2 range nnpfc_id values ​​may be reserved for future use. Decoders are in the range of 256~511 or 2 31 ~2 32 NNPFC SEI messages with an nnpfc_id in the -2 range must be ignored.

[0128] The following may apply if an NNPFC SEI message is currently the first NNPFC SEI message in the decoding sequence with a specific nnpfc_id value within the CLVS (Coded Layer Video Sequence).

[0129] The aforementioned SEI message can indicate a base post-processing filter (NNPF).

[0130] The aforementioned SEI message may relate, in output order, to the currently decoded picture and all subsequent decoded pictures of the current layer until the CLVS terminates.

[0131] A value of 1 for nnpfc_base_flag indicates that the SEI message explicitly identifies the base NNPF. A value of 0 for nnpfc_base_flag indicates that the SEI message explicitly identifies updates related to the base NNPF.

[0132] The following constraints may apply to the nnpfc_base_flag value.

[0133] If an NNPFC SEI message is currently the first NNPFC SEI message in the decoding order that has a specific nnpfc_id value within CLVS, then the nnpfc_base_flag value must be the same as 1.

[0134] If an NNPFC SEI message nnpfcB is not the first NNPFC SEI message in the decoding order with a specific nnpfc_id value in CLVS, and has the same nnpfc_base_flag value as 1, then the NNPFC SEI message must be a repetition of the first NNPFC SEI message nnpfcA with the same nnpfc_id value in the decoding order. That is, the payload content of nnpfcB must be identical to the payload content of nnpfcA.

[0135] If nnpfc_base_flag is 0, the following may apply.

[0136] The aforementioned SEI message can define updates related to preceding base NNPFs in the decoding order that have the same nnpfc_id value. The updates are not cumulative; rather, each update can be applied to a base NNPF that currently has a specific nnpfc_id value in CLVS and is explicitly stated by the first NNPFC SEI message in the decoding order. The NNPF defined by the aforementioned SEI message can be obtained by applying the updates defined by the aforementioned SEI message to base NNPFs that have the same nnpfc_id value.

[0137] The aforementioned SEI message may relate to the currently decoded picture and all subsequent decoded pictures of the current layer in output order, until the current CLVS terminates or until the decoded picture that comes after the currently decoded picture in output order within the current CLVS is excluded, and may relate to subsequent NNPFC SEI messages in decoding order where nnpfc_base_flag is 0 and has an earlier specific nnpfc_id value within the current CLVS.

[0138] A value of 0 for nnpfc_mode_idc indicates that the SEI message explicitly identifies the base NNPF (if nnpfc_base_flag is 1), is an update related to the base NNPF with the same nnpfc_id value (if nnpfc_base_flag is 0), or contains a bitstream that complies with ISO / IEC 15938-17.

[0139] If nnpfc_mode_idc is 1, a value of 1 for nnpfc_mode_idc indicates that the base NNPF associated with the nnpfc_id value is a neural network, and that neural network may be a neural network identified by a URI represented by nnpfc_uri using the format identified by the tag URI nnpfc_tag_uri. If nnpfc_mode_idc is 0, a value of 1 for nnpfc_mode_idc indicates that an update to a base NNPF having the same nnpfc_id value is defined by a URI represented by nnpfc_uri using the format identified by the tag URI nnpfc_tag_uri.

[0140] The value of nnpfc_mode_idc must be in the range of 0 to 1 in the bitstream. Values ​​of nnpfc_mode_idc in the range of 2 to 255 may be reserved for future use and do not need to be present in the bitstream. The decoder must ignore NNPFC SEI messages with nnpfc_mode_idc in the range of 2 to 255. Values ​​of nnpfc_mode_idc greater than 255 do not exist in the bitstream and do not need to be reserved for future use.

[0141] nnpfc_reserved_zero_bit_a must be equivalent to 0 in the bitstream. The decoder must ignore NNPFC SEI messages where the value of nnpfc_reserved_zero_bit_a is not 0.

[0142] nnpfc_tag_uri may contain a tag URI having syntax and semantics specified in IETF RFC 4151 that identifies the format and related information for updates to a neural network used as the base NNPF or a base NNPF having the same nnpfc_id value identified by nnpfc_uri.

[0143] nnpfc_tag_uri can uniquely identify the format of neural network data explicitly specified by nnrpf_uri, even without a central registry.

[0144] The same nnpfc_tag_uri as "tag:iso.org, 2023:15938-17" can indicate that the neural network data identified by nnpfc_uri complies with ISO / IEC 15938-17.

[0145] The nnpfc_uri may contain a URI with syntax and semantics explicitly defined in IETF Internet Standard 66 that identifies a neural network used as the base NNPF or an update to a base NNPF having the same nnpfc_id value.

[0146] A value of 1 for nnpfc_formatting_and_purpose_flag indicates the presence of syntax elements related to the filter's purpose, input formatting, output formatting, and complexity. A value of 0 for nnpfc_formatting_and_purpose_flag indicates the absence of syntax elements related to the filter's purpose, input formatting, output formatting, and complexity.

[0147] If nnpfc_base_flag is 1, then nnpfc_property_present_flag must be the same as 1.

[0148] If nnpfc_property_present_flag is 0, the values ​​of all syntax elements that can only exist if nnpfc_property_present_flag is 1 can be inferred to be identical to each of those corresponding syntax elements in the NNPFC SEI message containing the base NNPF for which the SEI message provides the update.

[0149] If an NNPFC SEI message nnpfcCurr is not the first NNPFC SEI message in the decoding order that currently has a specific nnpfc_id value in CLVS, nor is it a repetition of the first NNPFC SEI message that has the said specific nnpfc_id value (i.e., the nnpfc_base_flag value is the same as 0), and the value of nnpfc_property_present_flag is the same as 1, then the following constraints may apply:

[0150] The nnpfc_purpose value of an NNPFC SEI message must be identical to the nnpfc_purpose value of the first NNPFC SEI message in the decoding order that currently has a specific nnpfc_id value within the CLVS.

[0151] In an NNPFC SEI message, the value of the syntax element that comes before nnpfc_complexity_info_present_flag and after nnpfc_property_present_flag in the decoding order must be identical to the value of the corresponding syntax element in the first NNPFC SEI message in the decoding order that currently has a specific nnpfc_id value in CLVS.

[0152] Currently, in CLVS, for the first NNPFC SEI message in the decoding order that has a specific nnpfc_id value, the nnpfc_complexity_info_present_flag (as shown in nnpfcBase below) must be identical to 0, or all nnpfc_complexity_info_present_flags must be identical to 1, and all of the following may apply:

[0153] The nnpfc_parameter_type_idc of nnpfcCurr must be identical to the nnpfc_parameter_type_idc of nnpfcBase.

[0154] If nnpfcCurr's nnpfc_log2_parameter_bit_length_minus3 exists, it must be less than or equal to nnpfcBase's nnpfc_log2_parameter_bit_length_minus3.

[0155] If nnpfc_num_parameters_idc of nnpfcBase is 0, then nnpfc_num_parameters_idc of nnpfcCurr must be the same as 0.

[0156] Otherwise (where nnpfc_num_parameters_idc of nnpfcBase is greater than 0), nnpfc_num_parameters_idc of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_num_parameters_idc of nnpfcBase.

[0157] If nnpfc_num_kmac_operations_idc of nnpfcBase is 0, then nnpfc_num_kmac_operations_idc of nnpfcCurr must be the same as 0.

[0158] Otherwise (where nnpfc_num_kmac_operations_idc of nnpfcBase is greater than 0), nnpfc_num_kmac_operations_idc of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_num_kmac_operations_idc of nnpfcBase.

[0159] If nnpfc_total_kilobyte_size of nnpfcBase is 0, then nnpfc_total_kilobyte_size of nnpfcCurr must be the same as 0.

[0160] Otherwise (where nnpfc_total_kilobyte_size of nnpfcBase is greater than 0), nnpfc_total_kilobyte_size of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_total_kilobyte_size of nnpfcBase.

[0161] nnpfc_num_input_pics_minus1+_1 can indicate the number of pictures used as input to NNPF. The value of nnpfc_num_input_pics_minus1 must be in the range of 0 to 63. If pictureRateUpsamplingFlag is the same as 1, the value of nnpfc_num_input_pics_minus1 must be greater than 0.

[0162] The variable numInputPics, which indicates the number of pictures used as input to NNPF, can be derived as shown in equation 1 below.

[0163]

number

[0164] A value of 1 for nnpfc_input_pic_output_flag[i] indicates that NNPF will generate an output picture for the i-th input picture. A value of 0 for nnpfc_input_pic_output_flag[i] indicates that NNPF will not generate an output picture for the i-th input picture. If nnpfc_num_input_pics_minus1 is equal to 0, then nnpfc_input_pic_output_flag[0] can be inferred to be 1. If pictureRateUpsamplingFlag is equal to 0 and nnpfc_num_input_pics_minus1 is greater than 0, then nnpfc_input_pic_output_flag[i] must be equal to 1 for at least one value of i in the range 0 to nnpfc_num_input_pics_minus1.

[0165] A value of 1 for nnpfc_absent_input_pic_zero_flag indicates that NNPF expects input pictures not present in the bitstream to be represented by a sample array with a sample value of 0. A value of 0 for nnpfc_absent_input_pic_zero_flag indicates that NNPF expects input pictures not present in the bitstream to be represented by the closest input picture in the bitstream output order. nnpfc_out_sub_c_flag can indicate the values ​​of the variables outSubWidthC and outSubHeightC when chromaUpsamplingFlag is 1. A value of 1 for nnpfc_out_sub_c_flag indicates that the value of outSubWidthC is 1 and the value of outSubHeightC is 1. A value of 0 for nnpfc_out_sub_c_flag indicates that the value of outSubWidthC is 2 and the value of outSubHeightC is 1. If the value of ChromaFormatIdc is 2 and nnpfc_out_sub_c_flag exists, then the value of nnpfc_out_sub_c_flag must be the same as 1.

[0166] nnpfc_out_colour_format_idc can indicate the hue format of the NNPF and, consequently, the values ​​of the variables outSubWidthC and outSubHeightC, when colourizationFlag is 1. A value of 1 for nnpfc_out_colour_format_idc indicates that the hue format of the NNPF output is 4:2:0, and both outSubWidthC and outSubHeightC are 2. A value of 2 for nnpfc_out_colour_format_idc indicates that the hue format of the NNPF output is 4:2:2, outSubWidthC is 2, and outSubHeightC is 1. A value of 3 for nnpfc_out_colour_format_idc indicates that the hue format of the NNPF output is 4:4:4, and both outSubWidthC and outSubHeightC are 1. The value of nnpfc_out_colour_format_idc must not be 0.

[0167] If both chromaUpsamplingFlag and colourizationFlag are 0, outSubWidthC and outSubHeightC can be inferred to be the same as SubWidthC and SubHeightC, respectively. nnpfc_pic_width_num_minus1+1 and nnpfc_pic_width_denom_minus1+1 can represent the numerator and denominator, respectively, for the resampling ratio of the NNPF output picture width with respect to CroppedWidth. The value of (nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denom_minus1+1) must be in the range of 1÷16 to 16. If nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 do not exist, the values ​​of both nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 can be inferred to be the same as 0.

[0168] The variable nnpfcOutputPicWidth, which indicates the width of the luma sample array of the picture(s) resulting from applying the NNPF identified by nnpfc_id to the input picture(s), can be derived as shown in Equation 2 below.

[0169]

number

[0170] The requirement that the value of nnpfcOutputPicWidth%outSubWidthC be equal to 0 is a requirement for bitstream compatibility. nnpfc_pic_height_num_minus1+1 and nnpfc_pic_height_denom_minus1+1 can represent the numerator and denominator, respectively, for the resampling ratio of the NNPF output picture height with respect to CroppedHeight. The value of (nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denom_minus1+1) must be in the range of 1÷16 to 16. If nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 do not exist, it can be inferred that the values ​​of both nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 are identical to 0.

[0171] The variable nnpfcOutputPicHeight, which indicates the height of the luma sample array of the picture(s) resulting from applying the NNPF identified by nnpfc_id to the input picture(s), can be derived as shown in the following equation 3.

[0172]

number

[0173] The requirement that the value of nnpfcOutputPicHeight%outSubHeightC be equal to 0 is a requirement for bitstream compatibility. If nnpfc_pic_width_num_minus1, nnpfc_pic_width_denom_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denom_minus1 exist, then at least one of the following must be true:

[0174] The value of nnpfcOutputPicWidth is not the same as CroppedWidth.

[0175] The value of nnpfcOutputPicHeight is not the same as CroppedHeight.

[0176] nnpfc_interpolated_pics[i] can indicate the number of interpolated pictures generated by NNPF between the i-th picture and the (i+1)th picture used as input to NNPF.

[0177] The value of nnpfc_interpolated_pics[i] must be in the range of 0 to 63. The value of nnpfc_interpolated_pics[i] must be greater than 0 for at least one value of i, in the range of 0 to nnpfc_num_input_pics_minus1-1.

[0178] The variables NumInpPicsInOutputTensor, which indicates the number of pictures in the NNPF output tensor with the corresponding input picture, InpIdx[idx], which indicates the input picture index of the idx-th picture in the NNPF output tensor with the corresponding input picture, and numOutputPics, which indicates the total number of pictures in the NNPF output tensor, can be derived as shown in Table 6 below.

[0179] [Table 6]

[0180] A value of 1 for nnpfc_component_last_flag indicates that the last dimension of the input tensor `inputTensor` for NNPF and the output tensor `outputTensor`, which is the result of NNPF, are currently used for the channel. A value of 0 for nnpfc_component_last_flag indicates that the third dimension of the input tensor `inputTensor` for NNPF and the output tensor `outputTensor`, which is the result of NNPF, are currently used for the channel.

[0181] The first dimension of the input and output tensors can be used as a batch index in some neural network frameworks. While the formula within the semantics of this SEI message uses the size of the batch corresponding to a batch index of 0, the size of the batch used as input for neural network inference can be determined by the implementation of post-processing.

[0182] For example, when the value of nnpfc_inp_order_idc is the same as 3 and the value of nnpfc_auxiliary_inp_idc is the same as 1, the input tensor can have 7 channels, including 4 luma matrices, 2 chroma matrices, and 1 auxiliary input matrix. In this case, the DeriveInputTensors() process can induce each of the 7 channels of the input tensor one by one, and when a particular channel is processed, that channel can be referred to as the current channel in the process.

[0183] nnpfc_inp_format_idc indicates how to convert the sample values ​​of the input picture into NNPF input values. If nnpfc_inp_format_idc is 0, the input values ​​for NNPF are real numbers and the functions InpY() and InpC() can be specified as shown in equation 4 below.

[0184]

number

[0185] If the value of nnpfc_inp_format_idc is 1, the input values ​​for NNPF are unsigned integer numbers, and the functions InpY() and InpC() can be derived as shown in Table 7.

[0186] [Table 7]

[0187] The variable inpTensorBitDepthy can be derived from the syntax element nnpfc_inp_tensor_bitlength_minus8 described below. inpTensorBitDepthC can be derived from the syntax element nnpfc_inp_tensor_chroma_bitdepth_minus8 described below.

[0188] A value of nnpfc_inp_format_idc greater than 1 may be reserved for future use and may not exist in the bitstream. The decoder must ignore NNPFC SEI messages containing reserved values ​​for nnpfc_inp_format_idc.

[0189] A value of nnpfc_auxiliary_inp_idc greater than 0 indicates that auxiliary input data exists in the NNPF input tensor. A value of nnpfc_auxiliary_inp_idc of 0 indicates that auxiliary input data does not exist in the input tensor. A value of nnpfc_auxiliary_inp_idc of 1 indicates that auxiliary input data is induced, as explained in equation 5 below.

[0190] The value of nnpfc_auxiliary_inp_idc must be in the range of 0 to 1 in the bitstream. Values ​​of nnpfc_auxiliary_inp_idc between 2 and 255 may be reserved for future use and do not need to be present in the bitstream. The decoder must ignore NNPFC SEI messages containing nnpfc_auxiliary_inp_idc values ​​in the range of 2 to 255. Values ​​of nnpfc_auxiliary_inp_idc greater than 255 do not exist in the bitstream and are not reserved for future use.

[0191] nnpfc_inp_order_idc can show how to align the sample array of input pictures to form the input tensor for NNPF.

[0192] The value of nnpfc_inp_order_idc must be in the range of 0 to 3 in the bitstream. Values ​​of nnpfc_inp_order_idc between 4 and 255 may be reserved for future use and do not exist in the bitstream. The decoder must ignore NNPFC SEI messages with nnpfc_inp_order_idc in the range of 4 to 255. Values ​​of nnpfc_inp_order_idc greater than 255 do not exist in the bitstream and are not reserved for future use.

[0193] If the value of ChromaFormatIdc is not 1, then the value of nnpfc_inp_order_idc must not be 3.

[0194] If ChromaFormatIdc is 0, then nnpfc_inp_order_idc must be the same as 0.

[0195] If chromaUpsamplingFlag is 1, then nnpfc_inp_order_idc must not be 0.

[0196] Table 8 includes explanations for the nnpfc_inp_order_idc values.

[0197] [Table 8]

[0198] nnpfc_inp_tensor_bitlength_minus8 + 8 is an input constant tensor that can represent the bit depth of the luma sample value. The value of inpTensorBitDepthY can be derived as shown in equation 5.

[0199]

number

[0200] The requirement that the value of nnpfc_inp_tensor_luma_bitdepth_minus8 must be within the range of 0 to 24 is a bitstream compatibility requirement. nnpfc_inp_tensor_chroma_bitdepth_minus8+8 is an input constant tensor that can indicate the bit depth of the chroma sample values. The value of inpTensorBitDepthC can be derived as shown in equation 6.

[0201]

number

[0202] The requirement that the value of nnpfc_inp_tensor_chroma_bitdepth_minus8 must be within the range of 0 to 24 is a bitstream compatibility requirement.

[0203] If nnpfc_auxiliary_inp_idc is 1, the variable strengthControlScaledVal may be induced as shown in Table 9 below.

[0204] [Table 9]

[0205] The patch can be a rectangular array of samples from the components of the picture (e.g., luma or chroma components). The process DeriveInputTensors() for deriving the input tensor inputTensor with respect to the given vertical sample coordinate cTop and the horizontal sample coordinate cLeft that indicates the upper left sample position for the patch of samples included in the input tensor can be shown as a combination of Tables 10 to 12.

[0206] [Table 10]

[0207] [Table 11]

[0208] [Table 12]

[0209] The value 0 of nnpfc_out_format_id can indicate that the sample values output to the NNPF are real numbers linearly mapped to the unsigned integer value range of 0 to (1<<bitDepth)-1 for the desired bit depth bitDepth for subsequent post-processing or display. Here, the real number can be in the range of 0 to 1 value.

[0210] The value 1 of nnpfc_out_format_flag can indicate that the luma sample values output by the NNPF are unsigned integers in the range of 0 to (1<<outTensorBitDepthY)-1, and the chroma sample values output by the NNPF are unsigned integers in the range of (1<<outTensorBitDepthC)-1.

[0211] Values ​​of nnpfc_out_format_idc greater than 1 may be reserved for future specifications and will not exist in the bitstream. The decoder must ignore NNPFC SEI messages containing reserved values ​​for nnpfc_out_format_idc.

[0212] nnpfc_out_order_idc can indicate the output order of samples by NNPF.

[0213] The value of nnpfc_out_order_idc must be in the range of 0 to 3 in the bitstream. Values ​​of nnpfc_out_order_idc between 4 and 255 may be reserved for future use and do not exist in the bitstream. The decoder must ignore NNPFC SEI messages containing nnpfc_out_order_idc values ​​in the range of 4 to 255. Values ​​of nnpfc_out_order_idc greater than 255 do not exist in the bitstream and are not reserved for future use.

[0214] If chromaUpsamplingFlag is 1, then nnpfc_out_order_idc must not be 0 or 3.

[0215] If colourizationFlag is 1, then nnpfc_out_order_idc must not be 0.

[0216] Table 13 includes explanations for the nnpfc_out_order_idc values.

[0217] [Table 13]

[0218] nnpfc_out_tensor_luma_bitdepth_minus8+8 can indicate the bit depth of the luma sample value of the output constant tensor. The value of nnpfc_out_tensor_luma_bitdepth_minus8 must be in the range of 0 to 24. The value of outTensorBitDepthY can be derived as shown in equation 7.

[0219]

number

[0220] nnpfc_out_tensor_chroma_bitdepth_minus8+8 can indicate the bit depth of the chroma sample values ​​of the output constant tensor. The value of nnpfc_out_tensor_chroma_bitdepth_minus8 must be in the range of 0 to 24. The value of outTensorBitDepthC can be derived as shown in equation 8.

[0221]

number

[0222] If bitDepthUpsamplingFlag is 1, then the value of nnpfc_out_format_idc must be the same as 1, and at least one of the following conditions must be met.

[0223] nnpfc_out_tensor_luma_bitdepth_minus8 exists, and outTensorBitDepthY is greater than BitDepthY.

[0224] nnpfc_out_tensor_chroma_bitdepth_minus8 exists, and outTensorBitDepthC is greater than BitDepthC.

[0225] If nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepthY is greater than inpTensorBitDepthY, then outTensorBitDepthC must be less than inpTensorBitDepthC.

[0226] If nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepthC is greater than inpTensorBitDepthC, then outTensorBitDepthY must be less than inpTensorBitDepthY.

[0227] The StoreOutputTensors() process for deriving sample values ​​in the output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic, filtered from the output tensor outputTensor, which is based on the given vertical sample coordinate cTop and the horizontal sample coordinate cLeft, which indicates the upper-left sample position for the patch of samples contained in the input tensor, can be represented as the joins in Tables 14 and 15.

[0228] [Table 14]

[0229] [Table 15]

[0230] A value of 1 for nnpfc_separate_colour_description_present_flag indicates that a unique combination of color primaries, transformation properties, matrix coefficients, and scaling and offset values ​​applied in relation to the matrix coefficients for the picture by NNPF is specified in the SEI message syntax structure. A value of 0 for nnpfc_separate_colour_description_present_flag indicates that the combination of color primaries, transformation properties, matrix coefficients, and scaling and offset values ​​applied in relation to the matrix coefficients for the picture by the post-processing filter is identical to that displayed in the CLVS VUI parameters.

[0231] nnpfc_colour_primaries can have the same semantics as those defined for the vui_colour_primaries syntax element, except for the following:

[0232] nnpfc_colour_primaries can indicate the primary hues of a picture that are not used for CLVS, but rather the primary hues shown as a result of applying the NNPF specified in the SEI message.

[0233] If nnpfc_colour_primaries does not exist in the NNPFC SEI message, it can be inferred that the value of nnpfc_colour_primaries is the same as the value of vui_colour_primaries.

[0234] nnpfc_transfer_characteristics can have the same semantics as those defined for the vui_transfer_characteristics syntax elements, except for the following:

[0235] nnpfc_transfer_characteristics can indicate the picture transformation characteristics that are not used in CLVS, but rather the characteristics of the picture that are shown as a result of applying the NNPF specified in the SEI message.

[0236] If nnpfc_transfer_characteristics is not present in the NNPFC SEI message, it can be inferred that the value of nnpfc_transfer_characteristics is the same as the value of vui_transfer_characteristics.

[0237] nnpfc_matrix_coeffs can describe the formulas used to induce lumens and chromens signals in green, blue, red, or the Y, Z, and X primary colors. Its semantics apply to the picture shown as a result of applying the NNPF specified in the SEI message and may be identical to those specified for MatrixCoefficients containing the same outTensorBitDepthC and BitDepthC as outTensorBitDepthY and outTensorBitDepthC, respectively.

[0238] If nnpfc_matrix_coeffs is not present in the NNPFC SEI message, it can be inferred that the value of nnpfc_matrix_coeffs is the same as the value of vui_matrix_coeffs.

[0239] nnpfc_matrix_coeffs cannot be identical to 0 unless all of the following conditions are true.

[0240] nnpfc_out_tensor_chroma_bitdepth_minus8 is identical to nnpfc_out_tensor_luma_bitdepth_minus8.

[0241] nnpfc_out_order_idc is the same as 2, outSubHeightC is the same as 1, and outSubWidthC is the same as 1.

[0242] nnpfc_matrix_coeffs cannot be the same as 8 unless one of the following conditions is true.

[0243] nnpfc_out_tensor_chroma_bitdepth_minus8 is the same as nnpfc_out_tensor_luma_bitdepth_minus8.

[0244] nnpfc_out_tensor_chroma_bitdepth_minus8 is the same as nnpfc_out_tensor_luma_bitdepth_minus8 + 1, nnpfc_out_order_idc is the same as 2, outSubHeightC is the same as 1, and outSubWidthC is the same as 1.

[0245] nnpfc_full_range_flag can indicate the scaling and offset values applied in relation to the matrix coefficients as specified by nnpfc_matrix_coeffs. Its semantics can be the same as those specified for the VideoFullRangeFlag parameter. If it does not exist, it can be inferred that the value of nnpfc_full_range_flag is the same as 0.

[0246] The value 1 of the nnpfc_chroma_loc_info_present_flag can indicate the presence of the nnpfc_chroma_sample_loc_type_frame syntax element in the NNPFC SEI message. The value 0 of the nnpfc_chroma_loc_info_present_flag can indicate the absence of the nnpfc_chroma_sample_loc_type_frame syntax element in the NNPFC SEI message. When the colourizationFlag is 0 or the nnpfc_out_colour_format_idc is not 1, the value of the nnpfc_chroma_loc_info_present_flag must be the same as 0.

[0247] If the nnpfc_chroma_sample_loc_type_frame is not 6 and the nnpfc_out_colour_format_idc is the same as 1, the nnpfc_chroma_sample_loc_type_frame can indicate the position of the chroma samples in the output picture. If the nnpfc_chroma_sample_loc_type_frame is 6 and the nnpfc_out_colour_format_idc is the same as 1, it can indicate that the position of the chroma samples is not known or not explicitly stated or is indicated by other means. The value of the nnpfc_chroma_sample_loc_type_frame must exist in the range of 0 to 6.

[0248] nnpfc_overlap can indicate the number of horizontal and vertical overlapping samples of adjacent input tensors in NNPF. The value of nnpfc_overlap must be in the range of 0 to 16383. A value of 1 for nnpfc_constant_patch_size_flag indicates that NNPF will accept the exact patch size indicated by nnpfc_patch_width_minus1 and nnpfc_patch_height_minus1 as input. A value of 0 for nnpfc_constant_patch_size_flag indicates that NNPF will accept any patch size as input, where the width is inpPatchWidth and the height is inpPatchHeight. This allows the width of an extended patch (i.e., patch + overlapping region) to be a positive integer multiple of nnpfc_extended_patch_width_cd_delta_minus1+1+2*nnpfc_overlap, and the height of an extended patch to be a positive integer multiple of nnpfc_extended_patch_height_cd_delta_minus1+1+2*nnpfc_overlap, which is the same as inpPatchHeight+2*nnpfc_overlap.

[0249] npfc_patch_width_minus1+1 can indicate the number of horizontal samples required for the patch size of the NNPF input when the value of nnpfc_constant_patch_size_flag is 1. The value of nnpfc_patch_width_minus1 must be in the range of 0 to Min(32766, CroppedWidth-1).

[0250] npfc_patch_height_minus1+1 can indicate the number of vertical samples required for the patch size of the NNPF input when the value of nnpfc_constant_patch_size_flag is 1. The value of nnpfc_patch_height_minus1 must be in the range of 0 to Min(32766, CroppedHeight-1).

[0251] nnpfc_extended_patch_width_cd_delta_minus1+1+2*nnpfc_overlap can represent the common divisor of all allowed values ​​for the width of the extended patch required as input to NNPF when nnpfc_constant_patch_size_flag is 0. The value of nnpfc_extended_patch_width_cd_delta_minus1 must be in the range of 0 to Min(32766, CroppedWidth-1).

[0252] nnpfc_extended_patch_height_cd_delta_minus1+1+2*nnpfc_overlap can represent the common divisor of all allowed values ​​for the extended patch height required as input to NNPF when nnpfc_constant_patch_size_flag is 0. The value of nnpfc_extended_patch_height_cd_delta_minus1 must be in the range of 0 to Min(32766, CroppedHeight-1).

[0253] The variables inpPatchWidth and inpPatchHeight can be set to the width and height of the patch, respectively.

[0254] If the value of nnpfc_constant_patch_size_flag is 0, the following may apply.

[0255] The values ​​of inpPatchWidth and inpPatchHeight can be provided by external means or set by the post-processor itself.

[0256] The value of inpPatchWidth + 2 * nnpfc_overlap must be a positive integer multiple of ofnnpfc_extended_patch_width_cd_delta_minus1 + 1 + 2 * nnpfc_overlap, and inpPatchWidth must be less than or equal to CroppedWidth. The value of inpPatchHeight + 2 * nnpfc_overlap must be a positive integer multiple of ofnnpfc_extended_patch_height_cd_delta_minus1 + 1 + 2 * nnpfc_overlap, and inpPatchHeight must be less than or equal to CroppedHeight.

[0257] Otherwise (if the value of nnpfc_constant_patch_size_flag is the same as 1), the value of inpPatchWidth may be set to the same as nnpfc_patch_width_minus1+1, and the value of inpPatchHeight may be set to the same as nnpfc_patch_height_minus1+1.

[0258] The variables outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, outPatchCHeight, and overlapSize can be derived as shown in Table 16.

[0259] [Table 16]

[0260] It is a requirement for bitstream compliance that outPatchWidth * CroppedWidth must be the same as nnpfc_pic_width_in_luma_samples * inpPatchWidth, and outPatchHeight * CroppedHeight must be the same as nnpfc_pic_height_in_luma_samples * inpPatchHeight.

[0261] As described in Table 17, nnpfc_padding_type can indicate the padding process when referring to sample positions outside the boundaries of the input picture. The value of nnpfc_padding_type must exist in the range of 0 to 4 in the bitstream. Values from 5 to 15 for nnpfc_padding_type can be reserved for future use and do not exist in the bitstream. The decoder must ignore NNPFC SEI messages containing nnpfc_padding_type in the range of 5 to 15. Values of nnpfc_padding_type greater than 15 do not exist in the bitstream and are not reserved for future use.

[0262] [Table 17]

[0263] When the value of nnpfc_padding_type is 4, nnpfc_luma_padding_val can indicate the luma value used for padding. The value of nnpfc_luma_padding_val must exist in the range of 0 to (1 << BitDepthY)-1.

[0264] When the value of nnpfc_padding_type is 4, nnpfc_cb_padding_val can indicate the Cb value used for padding. The value of nnpfc_cb_padding_val must exist in the range of 0 to (1 << BitDepthC)-1.

[0265] When the value of nnpfc_padding_type is 4, nnpfc_cr_padding_val can indicate the Cr value used for padding. The value of nnpfc_cr_padding_val must be in the range of 0 to (1<<BitDepthC)-1.

[0266] The function InpSampleVal(y, x, picHeight, picWidth, CroppedPic), which has inputs of vertical sample position y, horizontal sample position x, picture height picHeight, picture width picWidth, sample array CroppedPic, and component index cIdx (the same as 0 for luma, the same as 1 for Cb, and the same as 2 for Cr), can return the value of SampleVal derived as shown in Table 18.

[0267] For the inputs to the function InpSampleVal(), the vertical position can be listed before the horizontal position for compatibility with the input tensor rules of some inference engines.

[0268]

Table 18

[0269] NNPF PostProcessingFilter() can be the target NNPF derived from the semantics of the NNPFA SEI message. The following example process can be used to generate filtered and / or interpolated picture(s) in a patch-wise manner. The filtered and / or interpolated picture(s) can include the Y sample array FilteredYPic, the Cb sample array FilteredCbPic, and the Cr sample array FilteredCrPic as specified by nnpfc_out_order_idc.

[0270]

Table 19

[0271] The NNPF-generated picture containing index i may include the sample arrays FilteredYPic[i], FilteredCbPic[i], and FilteredCrPic[i], if present, as induced by Table 19. The NNPF-generated picture does not need to contain overlapping regions.

[0272] The NNPF process can constitute the process defined by Table 19 by outputting NNPF-generated pictures in their increasing index order. Here, all NNPF-generated pictures interpolated by NNPF are output, and NNPF-generated pictures corresponding to all input pictures to NNPF may be output as specified in the semantics of the NNPFA SEI message.

[0273] A value of 1 for nnpfc_complexity_info_present_flag indicates that there is one or more syntax elements indicating the complexity of the NNPF associated with nnpfc_id. A value of 0 for nnpfc_complexity_info_present_flag indicates that there are no syntax elements indicating the complexity of the NNPF associated with nnpfc_id.

[0274] A value of 0 for nnpfc_parameter_type_idc indicates that the neural network will only use integer parameters. A value of 1 for nnpfc_parameter_type_flag indicates that the neural network can use floating-point or integer parameters. A value of 2 for nnpfc_parameter_type_idc indicates that the neural network will only use binary parameters. A value of 3 for nnpfc_parameter_type_idc may be reserved for future use and is not present in the bitstream. The decoder must ignore NNPFC SEI messages where the value of nnpfc_parameter_type_idc is 3.

[0275] The values ​​0, 1, 2, and 3 for nnpfc_log2_parameter_bit_length_minus3 indicate that the neural network will not use parameters with bit lengths greater than 8, 16, 32, and 64, respectively. If nnpfc_parameter_type_idc exists and nnpfc_log2_parameter_bit_length_minus3 does not exist, the neural network may not use parameters with bit lengths greater than 1.

[0276] nnpfc_num_parameters_idc can indicate the maximum number of neural network parameters for NNPF in powers of 2048. A value of nnpfc_num_parameters_idc of 0 indicates that the maximum number of neural network parameters is unknown. The value of nnpfc_num_parameters_idc must be in the range of 0 to 52. Values ​​of nnpfc_num_parameters_idc greater than 52 may be reserved for future use and do not exist in the bitstream. The decoder must ignore NNPFC SEI messages with nnpfc_num_parameters_idc greater than 52.

[0277] If the value of nnpfc_num_parameters_idc is greater than 0, the variable maxNumParameters may be derived as shown in equation 9.

[0278]

number

[0279] The requirement for bitstream compatibility is that the number of neural network parameters in an NNPF must be less than or equal to maxNumParameters. A value of nnpfc_num_kmac_operations_idc greater than 0 indicates that the maximum number of multiply-accumulate operations per sample in NNPF is less than or equal to nnpfc_num_kmac_operations_idc * 1000. A value of nnpfc_num_kmac_operations_idc of 0 indicates that the network does not know the maximum number of multiply-accumulate operations. The value of nnpfc_num_kmac_operations_idc is between 0 and 2. 32 It must exist within the range of -2.

[0280] A value of nnpfc_total_kilobyte_size greater than 0 can indicate the total size (in kilobytes) required to store the uncompressed parameters of a neural network. The total size in bits can be a number that is greater than or equal to the sum of the bits used to store each parameter. nnpfc_total_kilobyte_size can be the total size (in bits) divided by 8000 and rounded. A value of nnpfc_total_kilobyte_size of 0 indicates that the total size required to store the parameters for the neural network is unknown. The value of nnpfc_total_kilobyte_size is between 0 and 2. 32 It must exist within the range of -2.

[0281] A value of 0 for nnpfc_metadata_extension_num_bits can indicate that nnpfc_reserved_metadata_extension does not exist. A value of nnpfc_metadata_extension_num_bits greater than 0 can indicate the length of nnpfc_reserved_metadata_extension in bits. nnpfc_metadata_extension_num_bits must be identical to 0. Values ​​in the range of 1 to 2048 for nnpfc_metadata_extension_num_bits may be reserved for future use and do not exist in the bitstream. The decoder must accept all values ​​of nnpfc_metadata_extension_num_bits in the range of 0 to 2048. Values ​​of nnpfc_metadata_extension_num_bits greater than 2048 do not exist in the bitstream and are not reserved for future use.

[0282] nnpfc_reserved_metadata_extension does not exist in the bitstream. However, the decoder must ignore the existence and value of nnpfc_reserved_metadata_extension. If it exists, the bitwise length of nnpfc_reserved_metadata_extension may be the same as nnpfc_metadata_extension_num_bits.

[0283] nnpfc_reserved_zero_bit_b must be equivalent to 0 in the bitstream. The decoder must ignore NNPFC SEI messages where nnpfc_reserved_zero_bit_b is not 0.

[0284] nnpfc_payload_byte[i] can contain the i-th byte of the bitstream. The byte sequence nnpfc_payload_byte[i] for all existing values ​​of i must be a complete bitstream compliant with ISO / IEC 15938-17.

[0285] Neural network post-filter activation (NNPFA) The syntax structure for NNPFA is shown in Table 20.

[0286] [Table 20]

[0287] The NNPFA syntax structure in Table 20 can be signaled in the form of an SEI message. An SEI message that signals the NNPFA syntax structure in Table 20 can be referred to as an NNPFA SEI message.

[0288] The NNPFA SEI message can activate or deactivate the possible use of the target neural network post-processing filter, identified by nnpfa_target_id and nnpfc_base_flag, for post-processing filtering of a picture set. For a specific picture with an activated NNPF, the target NNPF may be derived as follows:

[0289] If nnpfa_target_base_flag is 1, the target NNPF is the base NNPF with the same nnpfc_id as nnpfa_target_id.

[0290] Otherwise (when nnpfa_target_base_flag is 0), the target NNPF is identified by the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id, and is not a repetition of NNPFC SEI messages containing the base NNPF that precedes the first VCL NAL unit of the current picture in the decoding order.

[0291] When the post - processing filter is used for other purposes or filters other hue components, multiple NNPFA SEI messages can exist for the same picture.

[0292] nnpfa_target_id can indicate the target NNPF specified by one or more NNPFC SEI messages that are related to the current picture and have the same nnpfc_id as nnpfa_target_id.

[0293] The value of nnpfa_target_id must exist in the range of 0 to 2 32 -2. It must not exist in the ranges of 256 to 511 and 2 31 ~2 32 The nnpfa_target_id values within the range of 2 to -2 can be reserved for future use. The decoder must ignore NNPFA SEI messages having nnpfa_target_id in the ranges of 256 to 511 or 2 31 ~2 32 -2.

[0294] An NNPFA SEI message having a specific value of nnpfa_target_id must not exist in the current PU (Picture Unit) unless one or both of the following conditions are all true.

[0295] There exists an NNPFC SEI message within the current CLVS that has the same nnpfc_id as the specific value of nnpfa_target_id existing in a PU preceding the current PU in the decoding order There exists an NNPFC SEI message within the current PU that has the same nnpfc_id as the specific value of nnpfa_target_id If a PU contains all NNPFC SEI messages with a specific value for nnpfc_id and all NNPFA SEI messages with the same nnpfa_target_id as the specific value for nnpfc_id, the NNPFC SEI messages must precede the NNPFA SEI messages in the decoding order.

[0296] A value of 1 for nnpfa_cancel_flag can indicate that the persistence of the target neural network post-processing filter, set by any previous NNPFA SEI message having the same nnpfa_target_id as the current SEI message, is cancelled. In other words, the target neural network post-processing filter will not be used anymore unless activated by another NNPFA SEI message having the same nnpfa_target_id and the same nnpfa_cancel_flag (0) as the current SEI message. A value of 0 for nnpfa_cancel_flag can indicate that nnpfa_target_base_flag, nnpfa_persistence_flag, and nnpfa_num_output_entries will follow.

[0297] A value of 1 for nnpfa_target_base_flag indicates that the target NNPF is a base NNPF with the same nnpfc_id as nnpfa_target_id. A value of 0 for nnpfa_target_base_flag indicates that the target NNPF is identified by the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id and precedes the first VCL NAL unit of the current picture in the decoding order.

[0298] The nnpfa_persistence_flag can indicate the persistence of the target neural network post-processing filter for the current layer. A value of nnpfa_persistence_flag of 0 indicates that the target neural network post-processing filter can only be used for post-processing filtering on the current picture. A value of nnpfa_persistence_flag of 1 indicates that the target neural network post-processing filter can be used for post-processing filtering on the current picture and all subsequent pictures in the current layer in output order until one or more of the following conditions are true.

[0299] A new CLVS for the current layer is initiated. The bitstream is ending. Currently, pictures in the current layer related to NNPFA SEI messages that have the same nnpfa_target_id and nnpfa_cancel_flag as the SEI message are output after the current picture in the output order. The target neural network post-processing filter is not applied to subsequent pictures in the current layer related to NNPFA SEI messages that have the same nnpfa_target_id and nnpfa_cancel_flag as the current SEI message.

[0300] Let nnpfcTargetPictures be the set of pictures associated with the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id while currently preceding the NNPFA SEI message in the decoding order. Let nnpfaTargetPictures be the set of pictures whose target NNPF is currently activated by the NNPFA SEI message. The bitstream compatibility requirement is that all pictures included in nnpfaTargetPictures must also be included in nnpfcTargetPictures.

[0301] nnpfa_num_output_entries can indicate the number of nnpfa_output_flag[i] syntax elements present in an NNPFA SEI message. The value of nnpfa_num_output_entries falls within the range of 0 to NumInpPicsInOutputTensor.

[0302] A value of 1 for nnpfa_output_flag[i] indicates that the NNPF-generated picture corresponding to the input picture with index InpIdx[i] is output by the NNPF process activated by this NNPFA SEI message, where the NNPF process can be identified within the semantics of the NNPFC SEI message. A value of 0 for nnpfa_output_flag[i] can indicate that the NNPF-generated picture corresponding to the input picture with index InpIdx[i] is not output by the NNPF process activated by this NNPFA SEI message. If nnpfa_num_output_entries is less than NumInpPicsInOutputTensor, then nnpfa_output_flag[i] can be inferred to be 1 for each value of i in the range from nnpfa_num_output_entries to NumInpPicsInOutputTensor-1.

[0303] Post-filter hint The syntax structure for post-filter hints is shown in Table 21.

[0304] [Table 21]

[0305] The post-filter hint syntax structures in Table 21 can be signaled in the form of SEI messages. SEI messages that signal the post-filter hint syntax structures in Table 21 can be referred to as post-filter hint SEI messages.

[0306] Post-filter hint SEI messages can provide post-filter coefficients or relational information for post-filter design, potentially allowing the decoded and output picture set to be used for post-processing to obtain improved display quality.

[0307] A value of 1 for `filter_hint_cancel_flag` indicates that the SEI message cancels the persistence of a previous post-filter hint SEI message in the output order applied to the current layer. A value of 0 for `filter_hint_cancel_flag` indicates that post-filter hint information follows.

[0308] The `filter_hint_persistence_flag` can indicate the persistence of a post-filter hint SEI message for the current layer. A value of 0 for `filter_hint_persistence_flag` indicates that the post-filter hint applies only to the currently decoded picture. A value of 1 for `filter_hint_persistence_flag` indicates that the post-filter hint SEI message applies to the currently decoded picture and persists for all subsequent pictures in the current layer in the output order until one or more of the following conditions are true.

[0309] A new CLVS for the current layer is initiated. The bitstream is ending. Post-filter hints related to SEI messages: Pictures in the AU's current layer are output after the current picture in output order. `filter_hint_size_y` can represent the filter coefficient or the vertical size of the relation array. The value of `filter_hint_size_y` must be in the range of 1 to 15.

[0310] `filter_hint_size_x` can represent the filter coefficient or the horizontal size of the relation array. The value of `filter_hint_size_x` must be in the range of 1 to 15.

[0311] As shown in Table 19, `filter_hint_type` can indicate the type of filter hint transmitted. The value of `filter_hint_type` must be in the range of 0 to 2. A `filter_hint_type` value identical to 3 does not exist in the bitstream. The decoder must ignore post-filter hint SEI messages where `filter_hint_type` is 3.

[0312] [Table 22]

[0313] A value of 1 for `filter_hint_chroma_coeff_present_flag` indicates that a filter coefficient exists for the chroma. A value of 0 for `filter_hint_chroma_coeff_present_flag` indicates that no filter coefficient exists for the chroma.

[0314] `filter_hint_value[cIdx][cy][cx]` can represent the filter coefficients, or the cross-correlation matrix elements between the original signal and the decoded signal, with 16-bit precision. The value of `filter_hint_value[cIdx][cy][cx]` is -2 31 +1~2 31 It must be within the -1 range. cIdx can indicate the associated hue element, cy can indicate the vertical counter, and cx can indicate the horizontal counter. Depending on the value of filter_hint_type, the following may apply:

[0315] If the value of filter_hint_type is 0, coefficients of a 2D FIR (Finite Impulse Response) filter with size filter_hint_size_y * filter_hint_size_x may be transmitted.

[0316] Otherwise, if the value of filter_hint_type is 1, the filter coefficients of two one-dimensional FIR filters can be transmitted. In this case, the value of filter_hint_size_y must be 2. An index cy of 0 can indicate the filter coefficient of a horizontal filter, and a cy of 1 can indicate the filter coefficient of a vertical filter. In the filtering process, the horizontal filter is applied first, and the result can then be filtered by the vertical filter.

[0317] Otherwise (if the value of filter_hint_type is 2), the transmitted hint can represent the cross-correlation matrix between the original signal s and the decoded signal s'.

[0318] A normalized cross-correlation matrix for related hue components identified by cIdx of size filter_hint_size_y * filter_hint_size_x can be defined as shown in Equation 10.

[0319]

number

[0320] In Equation 10, s represents the sample array of the hue component cIdx of the original picture, s’ represents the array of the decoded picture corresponding thereto, h represents the vertical height of the relevant hue component, w represents the horizontal width of the relevant hue component, and bitDepth represents the bit depth of the hue component. Also, OffsetY is the same as (filter_hint_size_y>>1), OffsetX is the same as (filter_hint_size_x>>1), the range of cy is 0 <= cy < filter_hint_size_y, and the range of cx is 0 <= cx < filter_hint_size_x.

[0321] The decoder can derive a Wiener post-filter from the cross-correlation matrix of the original signal and the decoded signal and the auto-cross-correlation matrix of the decoded signal.

[0322] Problems with conventional technology In the current design of the Neural-Network Post-Filter Characteristic (NNPFC) and the Neural-Network Post-Filter Activation SEI message, the NNPFA SEI message can activate a basic filter (i.e., the NNPFC SEI message having nnpfc_base_flag equal to 1) or an update filter (i.e., the NNPFC SEI message having nnpfc_base_flag equal to 0) based on nnpfa_target_base_flag.

[0323] In identifying the target NNPF, there are the following problems. FIG. 6 illustrates a problem situation that may occur when identifying the target NNPF to be activated based on the NNPFA SEI message.

[0324] When the value of nnpfa_target_base_flag is 0, there are two conflicting specifications defining which NNPFC is activated:

[0325] As shown in Table 23, the start of semantics for NNPFA is defined as follows: or (nnpfa_target_base_flag is 0), the target NNPF is the NNPF that precedes the first VCL NAL unit of the current picture in the decoding order, is not a repetition of NNPFC SEI messages containing the base NNPF, and is identified by the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id.

[0326] [Table 23]

[0327] -As shown in Table 24, the latter part of the semantics of nnpfa_target_base_flag itself is defined as follows: An nnpfa_target_base_flag identical to 0 indicates that the target NNPF is the NNPF identified by the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id, and is not a repetition of NNPFC SEI messages containing the base NNPF.

[0328] [Table 24]

[0329] When nnpfa_target_base_flag is 0, but there are no updates to the NNPFC SEI with the same nnpfc_id as the target NNPF nnpfa_target_id, it is unclear whether the decoder should activate the base filter. This situation can occur as illustrated in Figure 6. In the example in Figure 6, an NNPFA SEI message exists in an AU (Access Unit) containing a PU with POC3, the NNPFA SEI message has a specific nnpfa_target_id value, and the value of nnpfa_target_base_flag is 0. An nnpfa_target_base_flag value of 0 indicates that the target NNPF is not a base NNPF, in other words, it is an updated NNPF.

[0330] Specifically, the target NNPF activated by the NNPFA SEI message is the NNPF identified by the last NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id, and is not a repetition of an NNPFC SEI message containing the base NNPF, in terms of the decoding order, and is not nnpfa_target_id. However, as illustrated in Figure 6, if there is no update to the NNPFC SEI message with the same nnpfc_id as nnpfa_target_id, it is unclear whether the base NNPF should be activated even if the value of nnpfa_target_base_flag is 0.

[0331] [Implementation Method] Examples The embodiments described below provide solutions to the aforementioned problems. The following is a summary of each embodiment.

[0332] 1. If the value of nnpfa_target_base_flag is 0 in an NNPFA SEI message, a constraint is imposed that there must be at least one NNPFC SEI message that precedes the NNPFA SEI message in the decoding order, has the same nnpfc_id as nnpfa_target_id, and has nnpfc_base_flag 0.

[0333] 2. When nnpfa_target_base_flag is 0, if there is no NNPFC SEI message that precedes an NNPFA SEI message in the decoding order, has the same nnpfc_id as nnpfa_target_id, and has nnpfc_base_flag 0, the target NNPF can be explicitly or defined as a base NNPF.

[0334] This application proposes various embodiments to solve the aforementioned problems. The embodiments proposed by this application may be carried out individually or in combination of two or more.

[0335] In the following, NNPFC is the NNPFC syntax structure shown in Tables 1 to 3, and can be signaled in SEI message form; in this case, NNPFC can be an NNPFC SEI message. NNPFA is the NNPFA syntax structure shown in Table 20, and can be signaled in SEI message form; in this case, NNPFA can be an NNPFA SEI message. Post-filter hints are the post-filter hint syntax structure shown in Table 21, and can be signaled in SEI message form; in this case, post-filter hints can be post-filter hint SEI messages.

[0336] Example 1 Example 1 relates to Summary 1 described above. According to Example 1, if the value of nnpfa_target_base_flag is 0 in an NNPFA SEI message, then at least one NNPFC SEI message whose nnpfc_id is the same as nnpfa_target_id and whose nnpfc_base_flag is 0 must precede the NNPFA SEI message in the decoding order. For example, based on the fact that the value of nnpfa_target_base_flag is 0 in an NNPFA SEI message, at least one NNPFC SEI message whose nnpfc_id is the same as nnpfa_target_id and whose nnpfc_base_flag is 0 is encoded so that it precedes the NNPFA SEI message in the decoding order. In Example 1, for a particular picture in which NNPF is activated, the target NNPF may be derived as follows:

[0337] A value of nnpfa_target_base_flag of 1 indicates that the target NNPF is a base NNPF with the same nnpfc_id as nnpfa_target_id. A value of nnpfa_target_base_flag of 0 does not indicate that the target NNPF is a base NNPF, but rather that the target NNPF precedes the first VCL NAL unit of the current picture in the decoding order and is identified by the last NNPFC SEI message with the same nnpfa_id as nnpfa_target_id, rather than being a repetition of an NNPFC SEI message containing a base NNPF. If nnpfa_target_base_flag is 0, then there exists at least one NNPFC SEI that precedes an NNPFA SEI message in the decoding order and has the same nnpfc_id as nnpfa_target_id and an nnpfc_base_flag of 0. Therefore, when nnpfa_target_base_flag is 0, there is an updated NNPF identified by an NNPFC SEI message that precedes the NNPFA SEI message (in terms of decoding order), and the updated NNPF can be activated without any ambiguity as to whether the base NNPF needs to be activated.

[0338] Figure 7 shows an example of how the embodiments described herein may be applied.

[0339] In the example in Figure 7, we consider a case where an NNPFA SEI message exists in an AU containing a PU with POC3, and the NNPFA SEI message has a specific nnpfa_target_id value and the value of nnpfa_target_base_flag is 0. An nnpfa_target_base_flag value of 0 indicates that the target NNPF is not a base NNPF, in other words, it is an updated NNPF.

[0340] In this case, according to Example 1, as shown in Figure 7, in terms of decoding order, there is at least one NNPFC SEI message that has the same nnpfc_id as nnpfa_target_id and an nnpfc_base_flag of 0 before the NNPFA SEI message. Therefore, since there is an updated NNPF identified by the NNPFC SEI message that is before the NNPFA SEI message (in terms of decoding order), the target NNPF to be activated by the NNPFA SEI message can be determined to be the updated NNPF.

[0341] Example 2 Example 2 relates to Summary 2 described above. According to Example 2, when nnpfa_target_base_flag is 0, if there is no NNPFC SEI message that precedes an NNPFA SEI message in the decoding order, has the same nnpfc_id as nnpfa_target_id, and has nnpfc_base_flag 0, the target NNPF may be the base NNPF. For example, when nnpfa_target_base_flag is 0, if there is no NNPFC SEI message that precedes an NNPFA SEI message in the decoding order, has the same nnpfc_id as nnpfa_target_id, and has nnpfc_base_flag 0, the target NNPF may be explicitly determined to be the base NNPF. In Example 2, for a specific picture in which the NNPF is activated, the target NNPF may be derived as follows:

[0342] A flag of 1 for nnpfa_target_base_flag indicates that the target NNPF is a base NNPF with the same nnpfc_id as nnpfa_target_id. A flag of 0 for nnpfa_target_base_flag indicates that the target NNPF is not a base NNPF, but rather that the target NNPF precedes the first VCL NAL unit of the current picture in the decoding order and is identified by the last NNPFC SEI message with the same nnpfa_id as nnpfa_target_id, rather than being a repetition of NNPFC SEI messages containing a base NNPF.

[0343] Here, when nnpfa_target_base_flag is 0, if there is no NNPFC SEI message that precedes the NNPFA SEI message in the decoding order, has the same nnpfc_id as nnpfa_target_id, and has nnpfc_base_flag 0, the target NNPF can be determined to be the base NNPF. Therefore, when nnpfa_target_base_flag has a value of 0 but there is no updated NNPF to activate, the base NNPF can be determined to be the target NNPF without any ambiguity regarding the target to activate.

[0344] Figure 8 shows another example of how the embodiments of this disclosure may be applied.

[0345] In the example in Figure 8, we consider a case where an NNPFA SEI message exists in a PU with POC3, and that NNPFA SEI message has a specific nnpfa_target_id value and the value of nnpfa_target_base_flag is 0. An nnpfa_target_base_flag value of 0 indicates that the target NNPF is not a base NNPF, in other words, it is an updated NNPF.

[0346] At this time, as shown in Figure 8, if there is no NNPFC SEI message that precedes the NNPFA SEI message in the decoding order, has the same nnpfc_id as nnpfa_target_id, and has nnpfc_base_flag as 0, then according to Example 2, the basic NNPF can be determined to be the target NNPF activated by the NNPFA SEI message.

[0347] Video encoding method and video decoding method The following describes the video encoding method and video decoding method according to the embodiment of the present application.

[0348] Figure 9 shows an example of a video encoding method to which the embodiments of this disclosure may be applied, and Figure 10 shows an example of a video decoding method to which the embodiments of this disclosure may be applied.

[0349] Referring to Figure 9, at least one neural network that can be used as a post-processing filter may be determined, and information about the determined neural network may be encoded in at least one NNPFC SEI message (S610).

[0350] The activation status of the target NNPF that can be applied to the picture can be determined, and information about the determined target NNPF can be encoded in the NNPFA SEI message (S620). The S620 process for determining whether or not to activate the target NNPF may include a process for determining the target NNPF, a process for determining whether or not to revoke the persistence of the target NNPF, a process for determining whether or not the target NNPF has persistence, and a process for determining whether or not to apply the basic NNPF.

[0351] Post-filter coefficients or correlation information for post-filter design may be encoded in the post-filter hint SEI message (S630). NNPFC SEI messages, NNPFA SEI messages, and / or post-filter hint SEI messages may be included in the NNPF SEI message.

[0352] The target NNPF can be determined or activated by various embodiments of the present invention when an NNPF SEI message is currently applied to a picture in the video decoding device 300.

[0353] For example, the decision of which NNPF to activate may be based on the NNPFA SEI message. Here, the target NNPF to be activated can be identified by the target identification information and target basic flag information contained in the NNPFA SEI message. For example, the decision of whether to activate a basic NNPF or an updated NNPF may be based on the NNPFA SEI message. Specifically, the decision of whether to activate a basic NNPF or an updated NNPF may be based on the target basic flag information contained in the NNPFA SEI message.

[0354] For example, the target identification information may correspond to nnpfa_target_id, which indicates a target neural network post-processing filter specified by one or more NNPFC SEI messages that have the same identification information (nnpfc_id) as this target identification information and are currently associated with a picture. The base flag information may correspond to nnpfc_base_flag, which indicates whether the NNPF specified by the NNPFC SEI message is a base NNPF, and the target base flag information may correspond to nnpfa_target_base_flag, which indicates whether the target NNPF is a base NNPF that has the same identification information as the target identification information.

[0355] Referring to Figure 10, the SEI message for NNPF currently applied to the picture can be obtained from the bitstream. The SEI message for NNPF can include an NNPFC SEI message, an NNPFA SEI message, and / or a post-filter hint SEI message.

[0356] When an SEI message for an NNPF is now applied to a picture, at least one neural network that can be used as a post-processing filter can be determined based on at least one NNPFC SEI message included in the SEI message for the NNPF (S710).

[0357] Based on at least one NNPFA SEI message obtained from the bitstream, it can be determined whether or not the target NNPF that can be applied to the picture is activated (S720). The S720 process for determining whether or not the target NNPF is activated may include a process for determining the target NNPF, a process for determining whether or not to revoke the persistence of the target NNPF, a process for determining whether or not the target NNPF has persistence, and a process for determining whether or not to apply the basic NNPF.

[0358] If the target NNPF is activated (decided or persisted), the target neural network post-processing filter may be applied to the current picture (S730).

[0359] The various embodiments of this application can be used to determine or activate the target NNPF.

[0360] For example, it is possible to determine which NNPF to activate based on the NNPFA SEI message. Here, the target NNPF to be activated can be identified by the target identification information and target basic flag information contained in the NNPFA SEI message. For example, it is possible to determine whether to activate a basic NNPF or an updated NNPF based on the NNPFA SEI message. Specifically, it is possible to determine whether to activate a basic NNPF or an updated NNPF based on the target basic flag information contained in the NNPFA SEI message.

[0361] Figure 11 shows the operation for determining the target NNPF according to an embodiment of this disclosure. For example, the operation shown in Figure 11 can be performed by a video decoding device.

[0362] Referring to Figure 11, the target identification information and target basic flag information of the NNPFA SEI message are obtained (S1110). For example, the target identification information and target basic flag information of the NNPFA SEI message can be obtained from the bitstream.

[0363] For example, the target identification information of the NNPFA SEI message may be information indicating a target neural network post-processing filter specified by one or more NNPFC SEI messages that are currently associated with the picture and have the same identification information as the target identification information. The target basic flag information may be information indicating whether the target NNPF is a basic NNPF that has the same identification information as the target identification information.

[0364] Furthermore, for example, if the value of the target basic flag information in the NNPFA SEI message is 1, it can indicate that the target NNPF is a basic NNPF. If the value of the target basic flag information in the NNPFA SEI message is 0, it can indicate that the target NNPF is not a basic NNPF. For example, if the value of the target basic flag information is 0, it can indicate that the target NNPF is an updated NNPF.

[0365] Based on the fact that the target basic flag information indicates the basic NNPF (yes in S1120), that is, based on the value of the target basic flag information being 1, the basic NNPF can be determined to be the target NNPF (S1130).

[0366] Based on the fact that the target basic flag information does not indicate a basic NNPF (no in S1120), that is, based on the value of the target basic flag information being 0, the target NNPF can be determined by an NNPFC SEI message that precedes the NNPFA SEI message in the decoding order, has the same identification information as the NNPFA SEI message, and whose basic flag information does not indicate a basic NNPF (S1140).

[0367] For this reason, based on the fact that the target basic flag information of an NNPFA SEI message does not indicate a basic NNPF, there exists an NNPFC SEI message that has the same identification information as an NNPFPA SEI message and whose basic flag information does not indicate a basic NNPF, i.e., the value of the basic flag information is 0, before the NNPFA SEI message in the decoding order. For example, the video encoding device 200 can encode an NNPFC SEI message that has the same identification information as an NNPFA SEI message (in the decoding order) and whose basic flag information does not indicate a basic NNPF, i.e., the value of the basic flag information is 0, before an NNPFA SEI message whose target basic flag information value is 0.

[0368] Figure 12 shows the operation for determining the target NNPF according to an embodiment of this disclosure. For example, the operation shown in Figure 12 can be performed by a video decoding device.

[0369] Referring to Figure 12, the target identification information and target basic flag information of the NNPFA SEI message are obtained (S1210). For example, the target identification information and target basic flag information of the NNPFA SEI message can be obtained from the bitstream.

[0370] For example, the target identification information of the NNPFA SEI message may be information indicating a target neural network post-processing filter specified by one or more NNPFC SEI messages that are currently associated with the picture and have the same identification information as the target identification information. The target basic flag information may be information indicating whether the target NNPF is a basic NNPF that has the same identification information as the target identification information.

[0371] Furthermore, for example, if the value of the target basic flag information in the NNPFA SEI message is 1, it can indicate that the target NNPF is a basic NNPF. If the value of the target basic flag information in the NNPFA SEI message is 0, it can indicate that the target NNPF is not a basic NNPF. Specifically, this can indicate that the target NNPF is an updated NNPF.

[0372] Based on the fact that the target basic flag information indicates the basic NNPF (yes in S1220), that is, based on the value of the target basic flag information being 1, the basic NNPF can be determined to be the target NNPF (S1240).

[0373] Based on the fact that the target basic flag information does not indicate a basic NNPF (no in S1120), that is, based on the value of the target basic flag information being 0, it is determined whether there exists an NNPFC SEI message that precedes the NNPFA SEI message in the decoding order, has the same identification information as the NNPFA SEI message, and whose basic flag information does not indicate a basic NNPF (S1230).

[0374] Based on the existence of an NNPFC SEI message meeting the above conditions (Yes in S1230), the NNPF identified by the NNPFC SEI message can be determined as the target NNPF (S1230).

[0375] Based on the absence of an NNPFC SEI message meeting the above conditions (No in S1230), the base NNPF can be determined as the target NNPF (S1240).

[0376] According to the example described above, it is possible to clearly determine which filter among the basic filter and the updated filter will be activated.

[0377] Furthermore, according to the aforementioned embodiment, it is possible to prevent situations where it is unclear whether to activate the basic filter or the updated filter.

[0378] Figure 12 illustrates a content streaming system structure to which the embodiments described in this document apply.

[0379] The content streaming system to which the embodiments described herein apply may broadly include an encoding server, a streaming server, a web server, a media storage facility, user equipment, and multimedia input devices.

[0380] The encoding server is responsible for compressing content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and then transmitting this bitstream to the streaming server. In other cases, if a multimedia input device such as a smartphone, camera, or camcorder directly generates the bitstream, the encoding server can be omitted.

[0381] The bitstream can be generated by an encoding method or bitstream generation method to which an embodiment of this document applies, and the streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0382] The streaming server transmits multimedia data to user devices based on user requests via a web server, and the web server acts as an intermediary to inform users about available services. When a user requests a desired service from the web server, the web server transmits this to the streaming server, and the streaming server transmits multimedia data to the user. In this case, the content streaming system may include a separate control server, in which case the control server controls the commands / responses between each device within the content streaming system.

[0383] The streaming server can receive content from a media storage and / or encoding server. For example, if it starts receiving content from the encoding server, it can receive the content in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0384] Examples of user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (such as smartwatches, smart glasses, and HMDs), digital TVs, desktop computers, and digital signage.

[0385] Each server within the aforementioned content streaming system can be operated as a distributed server, in which case the data received by each server can be processed in a distributed manner.

[0386] The scope of this disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable operation by various embodiments to be performed on a device or computer, and non-transitory computer-readable medium on which such software or instructions etc. are stored and executable on a device or computer.

[0387] [Industrial applicability] The embodiments described herein can be used for encoding / decoding images.

[0388] [Claims when filing an international application] [Claim 1] A video decoding method performed by a video decoding device, The stage for acquiring NNPFC (neural-network post-filter characteristics) SEI messages and NNPFA (neural-network post-filter activation) SEI messages; A step of determining at least one neural network that can be used as a neural network post-processing filter based on the aforementioned NNPFC SEI message; and The process includes the step of determining whether or not to activate the target neural network post-processing filter that can be applied to the picture at present, based on the aforementioned NNPFA SEI message; The NNPFA SEI message includes the target identification information and target basic flag information of the target neural network post-processing filter. The post-processing filter for the target neural network is determined based on the target identification information and the target basic flag information. A video decoding method in which, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI message in the decoding order. [Claim 2] The video decoding method according to claim 1, wherein the value of the target basic flag information being 1 indicates that the target neural network post-processing filter is a basic neural network post-processing filter having the same identification information as the target identification information. [Claim 3] The fact that the value of the target basic flag information is 0 indicates that the target neural network post-processing filter is a neural network post-processing filter identified by an NNPFC SEI message having the same identification information as the target identification information. The video decoding method according to claim 1, wherein the NNPFC SEI message having the same identification information as the target identification information is not a repetition of an NNPFC SEI message including a basic neural network post-processing filter, and is the last NNPFC SEI message preceding the first VCL NAL unit of the current picture. [Claim 4] A video encoding method performed by a video encoding device, The step of encoding at least one neural network that can be used as a post-processing filter into an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message; and The process includes a step of encoding the activation status of the target neural network post-processing filter that can be applied to the picture into an NNPFA (neural-network post-filter activation) SEI message; The NNPFA SEI message includes the target identification information and target basic flag information of the target neural network post-processing filter. A video encoding method in which, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI message in the decoding order. [Claim 5] The video coding method according to claim 4, wherein the value of the target basic flag information being 1 indicates that the target neural network post-processing filter is a basic neural network post-processing filter having the same identification information as the target identification information. [Claim 6] The fact that the value of the target basic flag information is 0 indicates that the target neural network post-processing filter is a neural network post-processing filter identified by an NNPFC SEI message having the same identification information as the target identification information. The video coding method according to claim 4, wherein the NNPFC SEI message having the same identification information as the target identification information is not a repetition of an NNPFC SEI message including a basic neural network post-processing filter, and is the last NNPFC SEI message preceding the first VCL NAL unit of the current picture. [Claim 7] A computer-readable recording medium for storing a bitstream generated by a video encoding method, The aforementioned bitstream is NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) messages indicating at least one neural network that can be used as a post-processing filter; and It includes an NNPFA (neural-network post-filter activation) SEI message indicating whether or not the target neural network post-processing filter that can be applied to the picture is activated; The NNPFA SEI message includes the target identification information and target basic flag information of the target neural network post-processing filter. A computer-readable recording medium wherein, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI message in the decoding order. [Claim 8] A method for transmitting data to video, A step in which a bitstream is generated for the aforementioned video, the bitstream includes an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message indicating at least one neural network that can be used as a post-processing filter, and an NNPFA (neural-network post-filter activation) SEI message indicating whether or not the target neural network post-processing filter that can be applied to the picture is activated; and The step of transmitting data including the generated bitstream; The NNPFA SEI message includes the target identification information and target basic flag information of the target neural network post-processing filter. A transmission method wherein, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI message in the decoding order.

Claims

1. A video decoding method performed by a video decoding device, NNPFC (Neural-Network Post-Filter Characteristics) is the stage for acquiring SEI messages and NNPFA (Neural-Network Post-Filter Activation) is the stage for acquiring SEI messages; A step of determining at least one neural network that can be used as a neural network post-processing filter based on the aforementioned NNPFC SEI message; and The step of determining whether or not to activate the target neural network post-processing filter that can be applied to the picture at present, based on the NNPFA SEI message; The NNPFA SEI message includes the target identification information and target basic flag information of the target neural network post-processing filter. The post-processing filter for the target neural network is determined based on the target identification information and the target basic flag information. A video decoding method in which, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI message in the decoding order.

2. The video decoding method according to claim 1, wherein the value of the target basic flag information being 1 indicates that the target neural network post-processing filter is a basic neural network post-processing filter having the same identification information as the target identification information.

3. The fact that the value of the target basic flag information is 0 indicates that the target neural network post-processing filter is a neural network post-processing filter identified by an NNPFC SEI message having the same identification information as the target identification information. The video decoding method according to claim 1, wherein the NNPFC SEI message having the same identification information as the aforementioned target identification information is not a repetition of an NNPFC SEI message including a basic neural network post-processing filter, and is the last NNPFC SEI message preceding the first VCL NAL unit of the current picture.

4. A video encoding method performed by a video encoding device, The step of encoding an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message into at least one neural network that can be used as a post-processing filter; and The process includes the step of encoding the activation status of the target neural network post-processing filter that can be applied to the picture into an NNPFA (neural-network post-filter activation) SEI message; The NNPFA SEI message includes the target identification information and target basic flag information of the target neural network post-processing filter. A video encoding method in which, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI message in the decoding order.

5. The video coding method according to claim 4, wherein the value of the target basic flag information being 1 indicates that the target neural network post-processing filter is a basic neural network post-processing filter having the same identification information as the target identification information.

6. The fact that the value of the target basic flag information is 0 indicates that the target neural network post-processing filter is a neural network post-processing filter identified by an NNPFC SEI message having the same identification information as the target identification information. The video coding method according to claim 4, wherein the NNPFC SEI message having the same identification information as the aforementioned target identification information is not a repetition of an NNPFC SEI message including a basic neural network post-processing filter, and is the last NNPFC SEI message preceding the first VCL NAL unit of the current picture.

7. A computer-readable recording medium for storing a bitstream generated by a video encoding method, The aforementioned bitstream is An NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message indicating at least one neural network that can be used as a post-processing filter; and It includes an NNPFA (neural-network post-filter activation) SEI message indicating whether or not the target neural network post-processing filter that can be applied to the picture is activated; The NNPFA SEI message includes the target identification information and target basic flag information of the target neural network post-processing filter. A computer-readable recording medium wherein, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI message in the decoding order.

8. A method for transmitting data to video, A step that generates a bitstream for the aforementioned video, the bitstream including an NNPFC (neural-network post-filter characteristics) SEI (supplemental enhancement information) message indicating at least one neural network that can be used as a post-processing filter, and an NNPFA (neural-network post-filter activation) SEI message indicating whether or not the target neural network post-processing filter that can now be applied to the picture is activated; and The step of transmitting data including the generated bitstream; The NNPFA SEI message includes the target identification information and target basic flag information of the target neural network post-processing filter. A transmission method wherein, based on the fact that the target basic flag information does not indicate that the target neural network post-processing filter is a basic neural network post-processing filter, at least one NNPFC SEI message having the same identification information as the target identification information and basic flag information with the same value as the target basic flag information exists before the NNPFA SEI message in the decoding order.

Images