Method for decoding image information, method for encoding image information, method for storing a bitstream of image information, and method for transmitting a bitstream of image information

Abstract

Methods for decoding image information, methods for encoding image information, methods for storing image information, and methods for transmitting image information are provided. A method for decoding image information according to the present disclosure can include obtaining image information including supplemental enhancement information (SEI) related information including an SEI message, and obtaining an SEI payload from the SEI message. The SEI message can include SEI payload type information indicating a payload type of the SEI message and an SEI payload. The SEI related information can include only one SEI message.

Description

Technical Field

[0001] This disclosure relates to methods for decoding image information, methods for encoding image information, methods for storing bit streams of image information, and methods for transmitting bit streams of image information. Background Technology

[0002] Recently, the demand for high-resolution and high-quality images, such as HD (High Definition) and UHD (Ultra High Definition) images, has been increasing across various fields. As image data becomes higher in resolution and quality, the amount of information or bits transmitted increases relatively compared to existing image data. This increase in the amount of information or bits transmitted leads to increased transmission and storage costs.

[0003] Therefore, there is a need for an efficient image compression technology for effectively transmitting, storing, and reproducing high-resolution and high-quality image information. Summary of the Invention

[0004] Technical issues

[0005] This disclosure aims to provide an encoding / decoding method and / or apparatus with improved encoding / decoding efficiency.

[0006] The present disclosure is to propose a new RBSP (Raw Byte Sequence Payload) for delivering SEI (Supplemental Enhancement Information) of large payload size.

[0007] This disclosure is intended to define one of the large SEI RBSPs as a versatile SEI RBSP comprising a large SEI message.

[0008] This disclosure is intended to provide a method and / or a computer-readable recording medium for storing a bitstream generated by an encoding method according to this disclosure.

[0009] This disclosure aims to provide a method and / or apparatus for transmitting a bit stream generated by an encoding method according to this disclosure.

[0010] The technical problems to be solved by this disclosure are not limited to those mentioned above, and those skilled in the art to which this disclosure pertains will clearly understand from the following description other technical problems not mentioned.

[0011] Technical solution

[0012] According to one aspect of the disclosure, a method for decoding image information may include: obtaining image information including SEI (Supplemental Enhancement Information) related information, wherein the SEI related information includes an SEI message; and obtaining an SEI payload from the SEI message. The SEI message may include SEI payload type information indicating the payload type of the SEI message and the SEI payload itself. The SEI related information may include only one SEI message.

[0013] According to one aspect of this disclosure, an apparatus for decoding image information includes: a memory; and a processor electrically connected to the memory, wherein the processor can obtain image information including SEI (Supplemental Enhancement Information) related information, wherein the SEI related information includes SEI messages, and the processor can obtain SEI payloads from the SEI messages. The SEI messages may include SEI payload type information indicating the payload type of the SEI message and the SEI payload. The SEI related information may include only one SEI message.

[0014] According to one aspect of this disclosure, a method for encoding image information may include: generating an SEI (Supplemental Enhancement Information) payload; generating an SEI message including SEI payload type information indicating the type of the SEI payload and the SEI payload; generating SEI-related information including the SEI message; and encoding image information including the SEI-related information. The SEI-related information may include only one SEI message. The SEI message may include SEI payload type information and the SEI payload, but not information about the payload size of the SEI message.

[0015] According to one aspect of this disclosure, an apparatus for encoding image information includes: a memory; and a processor electrically connected to the memory, wherein the processor can generate an SEI (Supplemental Enhancement Information) payload; generate SEI payload type information including SEI payload type information indicating the type of the SEI payload and an SEI payload message; generate SEI-related information including the SEI message; and encode image information including the SEI-related information.

[0016] According to one aspect of this disclosure, a method for storing a bitstream of image information in a computer-readable storage medium may include: obtaining a bitstream of image information, and storing data including the bitstream in the computer-readable storage medium. The SEI-related information may include only one SEI message. The SEI message may include an SEI payload and SEI payload type information indicating the payload type of the SEI message.

[0017] According to one aspect of this disclosure, a computer-readable storage medium can store a bitstream of image information. The SEI-related information may consist of only one SEI message. The SEI message may include an SEI payload and SEI payload type information indicating the payload type of the SEI message.

[0018] According to one aspect of this disclosure, a method for transmitting a bitstream of image information may include: obtaining a bitstream of image information, and transmitting data including the bitstream. The image information may include SEI (Supplemental Enhancement Information) related information. The SEI related information may include only one SEI message. The SEI message may include an SEI payload and SEI payload type information indicating the payload type of the SEI message.

[0019] According to one aspect of this disclosure, an apparatus for transmitting a bitstream of image information may include: a processor that acquires the bitstream of image information; and a transmitter that transmits data including the bitstream. The image information may include SEI (Supplemental Enhancement Information) related information. The SEI related information may include only one SEI message. The SEI message may include an SEI payload and SEI payload type information indicating the payload type of the SEI message.

[0020] Regarding methods / apparatus for decoding image information, methods / apparatus for encoding image information, methods / storage media for storing bitstreams of image information, and methods for transmitting bitstreams of image information, SEI messages may include SEI payload type information and SEI payload, but not information about the payload size of the SEI message.

[0021] Methods / apparatus for decoding image information, methods / apparatus for encoding image information, methods / storage media for storing bit streams of image information, and methods / apparatus for transmitting bit streams of image information can be obtained through the NAL (Network Abstraction Layer) unit.

[0022] Regarding methods / apparatus for decoding image information, methods / apparatus for encoding image information, methods / storage media for storing bit streams of image information, and methods / apparatus for transmitting bit streams of image information, the NAL unit may include a prefix SEI NAL unit and a suffix SEI NAL unit.

[0023] Regarding methods / apparatus for decoding image information, methods / apparatus for encoding image information, methods / storage media for storing bitstreams of image information, and methods / apparatus for transmitting bitstreams of image information, the SEI payload may include at least one SEI payload segment.

[0024] Regarding methods / apparatus for decoding image information, methods / apparatus for encoding image information, methods / storage media for storing bitstreams of image information, and methods / apparatus for transmitting bitstreams of image information, the SEI message may include SEI payload size information indicating the size of each of at least one SEI payload segment.

[0025] The features briefly summarized above are merely exemplary aspects of the detailed description of the present disclosure described below, and do not limit the scope of the present disclosure.

[0026] Beneficial effects

[0027] According to this disclosure, encoding / decoding methods and / or apparatuses with improved encoding / decoding efficiency can be provided.

[0028] According to this disclosure, by allowing a large SEI RBSP to include a large SEI message, the potential complexity / delay in SEI generation can be avoided without requiring a large signaling payload size.

[0029] According to this disclosure, a method and / or a computer-readable recording medium for storing a bit stream generated by an encoding method according to this disclosure can be provided.

[0030] According to this disclosure, a method and / or apparatus for transmitting a bit stream generated by an encoding method according to this disclosure can be provided.

[0031] The effects that can be obtained from this disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains from the following description. Attached Figure Description

[0032] Figure 1 This is a view schematically illustrating a video encoding system to which embodiments of the present disclosure are applicable.

[0033] Figure 2 This is a schematic diagram illustrating an encoding apparatus that can be applied according to embodiments of the present disclosure.

[0034] Figure 3 This is a schematic diagram illustrating a decoding apparatus that can be applied according to embodiments of the present disclosure.

[0035] Figure 4 An exemplary illustration shows the hierarchical structure of a video / image used for encoding.

[0036] Figure 5 This is a diagram used to explain the interleaving method used to derive the luminance channel.

[0037] Figure 6This is a diagram illustrating a method for decoding image information according to an embodiment of the present disclosure.

[0038] Figure 7 This is a diagram illustrating a method for encoding image information according to an embodiment of the present disclosure.

[0039] Figure 8 This is an exemplary illustration of a content streaming system that can be applied to embodiments of the present disclosure. Detailed Implementation

[0040] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to facilitate implementation by those skilled in the art. However, the present disclosure can be implemented in various different forms and is not limited to the embodiments described herein.

[0041] In describing this disclosure, detailed descriptions of relevant known functions or constructions will be omitted if they unnecessarily obscure the scope of this disclosure. In the accompanying drawings, portions irrelevant to the description of this disclosure are omitted, and similar reference numerals are attached to similar portions.

[0042] In this disclosure, when a component is “connected,” “coupled,” or “linked” to another component, it can include not only direct connections but also indirect connections with intermediate components. Furthermore, when a component “includes” or “has” other components, unless otherwise stated, this means that other components may be further included, rather than excluded.

[0043] In this disclosure, unless otherwise stated, the terms first, second, etc., may be used only for the purpose of distinguishing one component from other components, without limiting the order or importance of the components. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0044] In this disclosure, the components distinguished from each other are intended to clearly describe each feature and do not imply that these components must be separate. That is, multiple components may be integrated and implemented in a single hardware or software unit, or a single component may be distributed and implemented in multiple hardware or software units. Therefore, such embodiments in which components are integrated or distributed are included within the scope of this disclosure, even if not otherwise stated.

[0045] In this disclosure, the components described in the various embodiments are not necessarily essential components, and some components may be optional. Therefore, embodiments comprising a subset of the components described in the embodiments are also included within the scope of this disclosure. Furthermore, embodiments including components other than those described in the various embodiments are included within the scope of this invention.

[0046] This disclosure relates to the encoding and decoding of images. For example, the methods and embodiments disclosed in this document can be applied to the manner disclosed in VVC (Multi-Functional Video Coding), EVC (Essential Video Coding), AV1 (AOMedia Video 1), AVS2 (Audio Video 2nd Generation), or next-generation video / image coding standards (e.g., H.267 or H.268).

[0047] This disclosure presents various embodiments of video / image coding, and unless otherwise stated, these embodiments may be implemented in combination with each other.

[0048] Unless defined in this disclosure, the terms used in this disclosure may have their usual meaning in the technical field to which this disclosure pertains.

[0049] In this disclosure, "picture" generally refers to a unit representing an image over a specific time period, and a slice / tile is a coding unit that constitutes part of a picture, and a picture can be composed of one or more slices / tiles. Additionally, a slice / tile may include one or more CTUs (Code Tree Units). A picture can be composed of one or more tile groups. A tile group may include one or more tiles. A tile can represent a rectangular area of ​​a CTU row in a picture. In this document, tile groups and slices are used interchangeably. For example, in this document, a tile group / tile group header may be referred to as a slice / slice header.

[0050] In this disclosure, "pixel" or "cell" can refer to the smallest unit that constitutes a picture (or image). Additionally, "sample" can be used as a term corresponding to a pixel. A sample can typically represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0051] In this disclosure, "unit" can refer to a basic unit of image processing. A unit may include a specific region of an image and at least one of the information associated with that region. A unit may include a luminance block and two chrominance (e.g., Cb, Cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "sample array," "block," or "region." In general, an M-column block may include a sample (or sample array) or set (or array) of transform coefficients in M ​​columns and N rows.

[0052] In this invention, "current block" can refer to one of "current coding block," "current coding unit," "coding target block," "decoding target block," or "processing target block." When performing prediction, "current block" can refer to "current prediction block" or "prediction target block." When performing transform (inverse transform) / quantization (dequantization), "current block" can represent "current transform block" or "transform target block." When performing filtering, "current block" can refer to "filter target block."

[0053] In this disclosure, "current block" can mean a block that includes both luma component blocks and chroma component blocks, or "the luma block of the current block," unless it is an explicit description of a chroma block. The luma component block of the current block can be represented by an explicit description that includes it as a luma component block such as "luma block" or "current luma block." Similarly, the chroma component block of the current block can be expressed by an explicit description that includes it as a chroma component block such as "chroma block" or "current chroma block."

[0054] In this disclosure, the terms “,” and “,” should be interpreted as indicating “and / or”. For example, the expression “e.g., table” and “, B” can mean “with and / or B with”. In addition, “,” and “, B, C” can mean “at least one of B and / or C”.

[0055] In this disclosure, the term "or" should be interpreted as indicating "and / or". For example, the expression "Example or B" may include 1) only "" package, 2) only "" package, and / or 3) "and" and B and both. In other words, in this disclosure, the term "or" should be interpreted as indicating "additionally or alternatively".

[0056] Figure 1 The illustration shows an example of a video / image encoding system that can be applied to the files in this document.

[0057] refer to Figure 1 A video / image encoding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device in the form of a file or stream via digital storage media or a network.

[0058] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0059] Video sources can acquire video / images through processes of capturing, compositing, or generating video / images. Video sources can include video / image capture devices and / or video / image generation devices. Video / image capture devices can include, for example, one or more cameras, video / image archives including previously captured video / images, etc. Video / image generation devices can include, for example, computers, tablets, and smartphones, and can generate video / images (electronically). For example, virtual video / images can be generated by computers, etc. In this case, the video / image capture process can be replaced by a process of generating related data.

[0060] The encoding device can encode the input video / image. It can perform a series of processes, such as prediction, transformation, and quantization for compression and encoding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.

[0061] A transmitter can send encoded images / image information or data, output as a bitstream, to a receiver in a receiving device via digital storage media or a network, either as a file or a stream. Digital storage media can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter can include elements for generating media files according to a predetermined file format and may include elements for transmission via a broadcast / communication network. The receiver can receive / extract the bitstream and send the received bitstream to a decoding device.

[0062] Decoding devices can decode video / images by performing a series of processes such as dequantization, inverse transform, and prediction, which correspond to the operations of encoding devices.

[0063] The renderer can render the decoded video / image. The rendered video / image can be displayed on a monitor.

[0064] Figure 2 This is a schematic diagram illustrating an image encoding apparatus that can be applied according to embodiments of the present disclosure.

[0065] refer to Figure 2The encoding apparatus 200 includes an image segmenter 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstruction block generator. According to embodiments, the image segmenter 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). Additionally, the memory 270 may include a decoded picture buffer (DPB) or may be configured by digital storage media. The hardware component may also include the memory 270 as an internal / external component.

[0066] Image segmenter 210 can segment an input image (or picture or frame) input to encoding device 200 into one or more processors. For example, a processor may be referred to as a coding unit (CU). In this case, coding units can be recursively segmented from coding tree units (CTUs) or maximum coding units (LCUs) according to a quadtree-binary-tritree (QTBTTT) structure. For example, a coding unit can be divided into multiple deeper coding units based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure can be applied first, and a binary tree structure and / or a ternary structure can be applied later. Alternatively, a binary tree structure can be applied first. The encoding process according to this document can be performed based on the final coding unit that is no longer divided. In this case, the maximum coding unit can be used as the final coding unit based on the coding efficiency according to the image characteristics, or, if necessary, the coding unit can be recursively segmented into deeper coding units, and the coding unit with the optimal size can be used as the final coding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes, which will be described later. As another example, the processor may also include a predictor (PU) or a transform unit (TU). In this case, the predictor and transform unit can be split or divided from the final encoding unit described above. The predictor may be a unit for predicting samples, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving residual signals from transform coefficients.

[0067] The terminology units can be used interchangeably with terms such as blocks or regions, depending on the context. Typically, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. Samples can typically represent pixels or pixel values, and can represent pixel / pixel values ​​of only the luminance component, or pixel / pixel values ​​of only the chrominance component. Samples can be used as a term corresponding to pixels or cells in a picture (or image).

[0068] In the encoding apparatus 200, the predicted signal (prediction block, prediction sample array) output from the inter-frame predictor 221 or the intra-frame predictor 222 is subtracted from the input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is sent to the converter 232. In this case, the unit used to subtract the predicted signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) in the encoder 200, as shown, can be called the subtractor 231. The predictor can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a prediction block that includes the prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction based on the current block or CU. As described later in the description of each prediction mode, the predictor can generate various information related to the prediction, such as prediction mode information, and send the generated information to the entropy encoder 240. The information about the prediction can be encoded in the entropy encoder 240 and output in the form of a bitstream.

[0069] Intra-predictor 222 can predict the current block by referencing samples in the current image. The reference samples can be located in the neighborhood of the current block or can be positioned separately according to the prediction mode. In intra-prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC mode and planar mode. Depending on the level of detail in the prediction direction, the directional modes can include, for example, 33 or 65 directional prediction modes. However, this is merely an example, and more or fewer directional prediction modes can be used depending on the settings. Intra-predictor 222 can determine the prediction mode applied to the current block by using the prediction modes applied to neighboring blocks.

[0070] Inter-frame predictor 221 can derive the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. Here, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially adjacent blocks existing in the current image and temporally adjacent blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally adjacent block may be the same or different. The temporally adjacent block may be referred to as a co-located reference block, a co-located CU (colCU), etc., and the reference image including the temporally adjacent block may be referred to as a co-located image (colPic). For example, inter-frame predictor 221 can configure a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 221 can use the motion information of neighboring blocks as the motion information of the current block. In skip mode, unlike merge mode, residual signals may not be sent. In motion vector prediction (MVP) mode, the motion vectors of neighboring blocks can be used as motion vector prediction values, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0071] Predictor 220 can generate prediction signals based on various prediction methods described below. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but can also apply both intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined intra-frame and inter-frame prediction (CIIP). Alternatively, the predictor can be based on an intra-frame block copy (IBC) prediction mode or a palette mode for predicting blocks. The IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction in the current frame, but can be performed similarly to inter-frame prediction because a reference block is derived in the current frame. That is, IBC can use at least one of the inter-frame prediction techniques described in this document. A palette mode can be considered as an example of intra-frame coding or intra-frame prediction. When a palette mode is applied, sample values ​​within the frame can be signaled based on information about the palette table and palette index.

[0072] The prediction signal generated by the predictor (including inter-frame predictor 221 and / or intra-frame predictor 222) can be used to generate a reconstructed signal or a residual signal. The subtraction unit 115 can subtract the prediction signal (predicted block, predicted sample array) output from the predictor 200 from the input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array). The generated residual signal can be sent to the conversion unit 232.

[0073] Transformer 232 can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Lou transform (KLT), Graph-based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is represented by a graph. CNT refers to a transform generated based on a prediction signal generated using all previously reconstructed pixels. Furthermore, the transform process can be applied to square pixel blocks of the same size, or it can be applied to blocks of variable size that are not square.

[0074] Quantizer 233 can quantize the transform coefficients and send them to entropy encoder 240, which can encode the quantized signal (information about the quantized transform coefficients) and output a bitstream. This information about the quantized transform coefficients can be called residual information. Quantizer 233 can rearrange the block-type quantized transform coefficients into a one-dimensional vector based on the coefficient scan order, and generate information about the quantized transform coefficients based on this one-dimensional vector form. Information about the transform coefficients can be generated.

[0075] The entropy encoder 240 can perform various encoding methods, such as exponential Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 240 can encode, either together or separately, the information required for video / image reconstruction, excluding quantization transform coefficients (e.g., values ​​of syntax elements). The encoded information (e.g., encoded video / image information) can be transmitted or stored in units of the NAL (Network Abstraction Layer) in the form of a bitstream. The video / image information may further include information about various parameter sets, such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). Furthermore, the video / image information may also include general constraint information. In this document, information transmitted from the encoding device / signed to the decoding device and / or syntax elements can be included in the video / image information. The video / image information can be encoded and included in the bitstream through the encoding process described above.

[0076] The bitstream can be transmitted over a network or stored in a digital storage medium. The network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that sends the signal output from the entropy encoder 240 and / or a storage unit (not shown) that stores the signal may be included as internal / external components of the encoding device 200, and alternatively, the transmitter may be included within the entropy encoder 240.

[0077] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients through dequantizer 234 and inverse transformer 235.

[0078] Additionally, LMCS (Luminosity Mapping and Chroma Scaling) can be applied during the image encoding and / or restoration process.

[0079] Adder 250 adds the reconstructed residual signal to the predicted signal output from inter-frame predictor 221 or intra-frame predictor 222 to generate a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array). If no residual exists for the block to be processed, such as in the case of applying a skip mode, the predicted block can be used as a reconstructed block. Adder 250 may be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image, and can be used for inter-frame prediction of the next image after filtering as described below.

[0080] Filter 260 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 260 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and store the modified reconstructed image in memory 270 (specifically, the DPB of memory 270). Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. Filter 260 can generate various filtering-related information and send the generated information to entropy encoder 240, as described later in the description of each filtering method. The filtering-related information can be encoded by entropy encoder 240 and output as a bitstream.

[0081] The modified reconstructed image sent to memory 270 can be used as a reference image in inter-frame predictor 221. When inter-frame prediction is applied through the encoding device, prediction mismatch between the encoding device 200 and the decoding device can be avoided, and encoding efficiency can be improved.

[0082] The DPB of memory 270 can store modified reconstructed images for use as reference images in inter-frame predictor 221. Memory 270 can store motion information of blocks from which motion information in the current image is derived (or encoded) and / or motion information of blocks in already reconstructed images. The stored motion information can be sent to inter-frame predictor 221 and used as motion information for spatially adjacent blocks or temporally adjacent blocks. Memory 270 can store reconstructed samples of reconstructed blocks in the current image and can transmit these reconstructed samples to intra-frame predictor 222.

[0083] Figure 3 This is a schematic diagram illustrating an image decoding apparatus that can be applied according to embodiments of the present disclosure.

[0084] refer to Figure 3 The decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor 331 and an intra-frame predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. According to embodiments, the entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 may be configured by hardware components (e.g., a decoder chipset or processor). Additionally, the memory 360 may include a decoded picture buffer (DPB) or may be configured by digital storage media. The hardware components may also include the memory 360 as an internal / external component.

[0085] When the input includes a bitstream containing video / image information, the decoding device 300 can reconstruct the data from the input bitstream. Figure 2 The image corresponds to the process of processing video / image information in the encoding device. For example, the decoding device 300 can derive units / blocks based on block partitioning information obtained from the bitstream. The decoding device 300 can use a processor applied in the encoding device to perform decoding. Therefore, the decoding processor can be, for example, an encoding unit, and the encoding unit can be partitioned according to a quadtree structure, binary tree structure, and / or ternary tree structure from the encoding tree unit or the maximum encoding unit. One or more transform units can be derived from the encoding unit. The reconstructed image signal decoded and output by the decoding device 300 can be reproduced by a reproduction device.

[0086] Decoding device 300 is capable of receiving data from a source in the form of a bitstream. Figure 2The signal output by the encoding device can be decoded by the entropy decoder 310. For example, the entropy decoder 310 can parse the bitstream to derive the information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information may further include information about various parameter sets such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). In addition, the video / image information may also include general constraint information. The decoding device can also decode the picture based on the information about the parameter sets and / or general constraint information. The transmitted / received information and / or syntax elements described later in this document can be obtained from the bitstream by decoding the picture. For example, the entropy decoder 310 decodes the information in the bitstream based on encoding methods such as Exponential Golomb coding, CAVLC, or CABAC, and outputs the quantized values ​​of the syntax elements and transform coefficients of the residuals required for image reconstruction. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, determine a context model using information about the target syntax element, the target block, or the symbols / bins decoded in the previous stage, and perform arithmetic decoding on the bins by predicting the occurrence probability of the bins based on the determined context model, generating symbols corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method can update the context model after determining it by using information about the decoded symbols / bins of the context model of the next symbol / bin. Information related to the prediction in the information decoded by the entropy decoder 310 can be provided to the predictors (inter-frame predictor 332 and intra-frame predictor 331), and the residual values ​​(i.e., quantization transform coefficients and related parameter information) from the entropy decoding performed in the entropy decoder 310 can be input to the residual processor 320. The residual processor 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). Furthermore, information about the filtering in the information decoded by the entropy decoder 310 can be provided to the filter 350. Meanwhile, the receiver (not shown) for receiving the signal output from the encoding device can be further configured as an internal / external element of the decoding device 300, or the receiver can be a component of the entropy decoder 310. Furthermore, the decoding device according to this document can be referred to as a video / image / picture decoding device, and the decoding device can be classified as an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverse transformer 322, an adder 340, a filter 350, a memory 360, an inter-frame predictor 332, and an intra-frame predictor 331.

[0087] Dequantizer 321 can dequantize the quantized transform coefficients and output the transform coefficients. Dequantizer 321 can rearrange the quantized transform coefficients in the form of two-dimensional blocks. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device. Dequantizer 321 can perform dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0088] The inverse transformer 322 inversely transforms the transformation coefficients to obtain the residual signal (residual block, residual sample array).

[0089] Predictor 330 can generate a prediction signal based on various prediction methods described below. For example, the predictor can apply intra-frame prediction or inter-frame prediction to predict a block, and can also apply intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined intra-frame and inter-frame prediction (CIIP). Alternatively, the predictor can predict a block based on an intra-frame block copy (IBC) prediction mode or a palette mode. The IBC prediction mode or palette mode can be used for content image / video encoding such as in games, such as screen content encoding (SCC). IBC essentially performs prediction within the current frame, but can be performed similarly to inter-frame prediction because it derives a reference block within the current frame. That is, IBC can use at least one of the inter-frame prediction techniques described in this document. The palette mode can be considered as an example of intra-frame coding or intra-frame prediction. When a palette mode is applied, information about the palette table and palette index can be sent by signaling and included in the video / image information.

[0090] Intra-predictor 332 can predict the current block by referencing samples in the current image. The reference samples can be located in the neighborhood of the current block or can be located separately based on the prediction mode. In intra-prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. Intra-predictor 331 can determine the prediction mode applied to the current block by using the prediction modes applied to neighboring blocks.

[0091] Inter-frame predictor 331 can derive the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially adjacent blocks existing in the current image and temporally adjacent blocks existing in the reference image. For example, inter-frame predictor 332 can configure a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference image index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and the information about the prediction may include information indicating the inter-frame prediction mode of the current block.

[0092] Adder 340 generates a reconstruction signal (reconstructed image, reconstruction block, reconstruction sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (including inter-frame predictor 332 and / or intra-frame predictor 331). If no residual exists for the block to be processed, such as when a skip mode is applied, the prediction block can be used as the reconstruction block. Adder 340 may be referred to as a reconstructor or reconstruction block generator. The generated reconstruction signal can be used for intra-frame prediction of the next block to be processed in the current image, by filtering the output as described below, or it can be used for inter-frame prediction of the next image.

[0093] At the same time, LMCS (Luminosity Mapping and Chromaticity Scaling) can be applied during image decoding.

[0094] Filter 350 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 360 (specifically, the DPB of memory 360). Various filtering methods may include, for example, unblocking filtering, sample adaptive shifting, adaptive loop filtering, bilateral filtering, etc.

[0095] The (modified) reconstructed image stored in the DPB of memory 360 can be used as a reference image in inter-frame predictor 332. Memory 360 can store motion information of blocks from which motion information in the current image is derived (or decoded) and / or motion information of blocks in reconstructed images. The stored motion information can be sent to inter-frame predictor 260 to be used as motion information for spatially adjacent blocks or temporally adjacent blocks. Memory 360 can store reconstructed samples of reconstructed blocks in the current image and transmit the reconstructed samples to intra-frame predictor 331.

[0096] In this disclosure, the embodiments described in the encoding apparatus 100, including filter 260, inter-frame predictor 221, and intra-frame predictor 222, may be the same as or correspond to the decoding apparatus 300, including filter 350, inter-frame predictor 332, and intra-frame predictor 331, respectively. This can also be applied to unit 332 and intra-frame predictor 331.

[0097] Figure 4 An exemplary illustration shows a hierarchical structure of video / images encoded according to embodiments of this disclosure.

[0098] refer to Figure 4 The encoded image / video is divided into VCL (Encoding Layer), which processes the image / video and its own decoding process; subsystems that transmit and store encoded information; and NAL (Network Abstraction Layer), which is responsible for functions and exists between VCL and subsystems.

[0099] In VCL, VCL data is generated that includes compressed image data (slice data), or parameter sets that include additional picture parameter sets (PSP), sequence parameter sets (SPS), and video parameter sets (VPS) or supplementary enhancement information (SEI) messages required for the image decoding process.

[0100] In NAL, NAL units can be generated by adding header information (NAL unit header) to the raw byte sequence payload (RBSP) generated in VCL. In this case, RBSP refers to slice data, parameter sets, SEI messages, etc., generated in VCL. The NAL unit header can include NAL unit type information specified according to the RBSP data included in the corresponding NAL unit.

[0101] As shown in the figure, NAL units can be classified into VCL NAL units and non-VCL NAL units based on the RBSP generated in the VCL. VCL NAL units can represent NAL units that include information about the image (slice data), while non-VCL NAL units can represent NAL units that include information required for decoding the image (parameter set or SEI message).

[0102] The aforementioned VCL NAL units and non-VCL NAL units can be transmitted over a network by adding header information according to the subsystem's data standard. For example, NAL units can be transformed into predetermined standard data formats, such as H.266 / VVC file format, Real-time Transport Protocol (RTP), Transport Stream (TS), etc., and transmitted over various networks.

[0103] As described above, a NAL cell can be specified using the NAL cell type according to the RBSP data structure included in the corresponding NAL cell, and information about the NAL cell type can be stored and signaled in the NAL cell header.

[0104] For example, NAL units can be classified into VCL NAL unit types and non-VCL NAL unit types based on whether they include information about images (slice data). VCL NAL unit types can be classified based on the nature and type of the images included in the VCL NAL unit, and non-VCL NAL unit types can be classified based on the type of parameter set.

[0105] The following is an example of a NAL cell type specified based on the type of the parameter set included in a non-VCL NAL cell type.

[0106] APS (Adaptive Parameter Set) NAL Unit: A type of NAL unit that includes APS.

[0107] DPS (Decoding Parameter Set) NAL Unit: The type of NAL unit that includes the DPS.

[0108] VPS (Video Parameter Set) NAL Unit: Includes the type of NAL unit for the VPS.

[0109] SPS (Sequence Parameter Set) NAL Unit: A type of NAL unit that includes SPS.

[0110] PPS (Picture Parameter Set) NAL Unit: Includes the type of NAL unit for PPS.

[0111] The aforementioned NAL unit type may have syntax information for the NAL unit type, and this syntax information may be stored in the NAL unit header and signaled within the NAL unit header. For example, the syntax information may be nal_unit_type, and the NAL unit type may be specified by the nal_unit_type value.

[0112] A slice header (slice header syntax) may include information / parameters that can be applied collectively to a slice. APS (APS syntax) or PPS (PPS syntax) may include information / parameters that can be applied collectively to one or more slices or images. SPS (SPS syntax) may include information / parameters that can be applied collectively to one or more sequences. VPS (VPS syntax) may include information / parameters that can be applied collectively to multiple layers. DPS (DPS syntax) may include information / parameters that can typically be applied to the entire video. DPS may include information / parameters related to the concatenation of encoded video sequences (CVS). The High-Level Syntax (HLS) in this document may include at least one of the following: APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, image header syntax, and slice header syntax.

[0113] In this document, the image / video information encoded by the encoding device and signaled to the decoding device in the form of a bitstream includes not only partition-related information, intra / inter-frame prediction information, residual information, and in-loop filtering information in the image, but also information in the slice header, the image header, the APS, the PPS, the SPS, the VPS, and / or the DPS.

[0114] The SEI message related to this invention is described.

[0115] Large supplemental enhancement messages, large SEI messages

[0116] Table 1 shows an example of the syntax for large SEI messages.

[0117] [Table 1]

[0118] Each large SEI message consists of variables specifying the type (payloadType) and size (payloadSize) of the large SEI message payload. The derived large SEI message payload size (payloadSize) is specified in bytes and should be equal to the number of RBSP bytes in the large SEI message payload. The NAL unit byte sequence containing the large SEI message may include one or more emulation prevention bytes (indicated by the emulation_prevention_three_byte syntax element). Because the large SEI message payload size is specified in RBSP bytes, the number of emulation prevention bytes is not included in the large SEI payload size (payloadSize).

[0119] `lsei_position` indicates whether a SEI message corresponds to either `PREFIX_SEI_NUT` or `SUFFIX_SEI_NUT`. `lsei_position` equal to 0 indicates that the SEI message is treated as `PREFIX_SEI_NUT`. `lsei_position` equal to 1 indicates that the SEI message is treated as `SUFFIX_SEI_NUT`. Values ​​3 and 4 of `lsei_position` are reserved for future use and should be ignored.

[0120] `lsei_relevance` indicates the relevance of the SEI message to the target application. `lsei_relevance` ranges from 0 to 3, with 0 being the lowest relevance and 3 the highest. The relevance of the SEI message is arbitrarily determined, and its purpose is specified by the target application.

[0121] lsei_reserved is reserved for future use and should be ignored.

[0122] `lsei_payload_type_byte` is a byte representation of the large SEI message payload type. `payloadType = lsei_payload_type_byte`.

[0123] payload_size_16bits is the payload size in bits for a large SEI message. payloadSize = payload_size_16bits.

[0124] The general post-processing filtering procedure using NNPF (Neural Network Post-Filter SEI Message)

[0125] The input to this process is a bitstream, BitstreamToFilter. The output of this process is a list of NNPF output images, ListNnpfOutputPics.

[0126] First, decode BitstreamToFilter and set the list CroppedDecodedPictures to a list of cropped and decoded images generated from decoding BitstreamToFilter in output order.

[0127] Second, for each cropped and decoded picture in CroppedDecodedPictures and for which one or more NNPFs are activated, the filtering process for one picture is repeated in the output order, as specified in sub-clause 2.5.2.1.2.

[0128] The order of the images in ListNnpfOutputPics is the output order.

[0129] Within ListNnpfOutputPics, there should be no more than one image associated with any particular output time instance. When multiple NNPFs are active for any particular image in CroppedDecodedPictures, and although any NNPF can be selected, only one NNPF is allowed to be applied, the above constraints will apply regardless of which NNPF is chosen for the particular image.

[0130] The process of filtering an image using NNPF

[0131] The filtering process specified in this sub-clause applies to each cropped and decoded picture (referred to as the current picture) in CroppedDecodedPictures and for which one or more NNPFs are activated.

[0132] When NNPF is applied to the current image, NNPF generates filtered and / or interpolated images by applying the NNPF procedure specified in the semantics of the NNPFC SEI message to the current image in a patchwise manner.

[0133] When NNPF is applied to the current image, the order in which the images generated by NNPF through the application of the NNPF process stored in the NNPF output tensor is the output order.

[0134] When the applied NNPF is the last NNPF applied to the current image, the images generated by the NNPF and output by the NNPF process are included in ListNnpfOutputPics in the same order as when the images are stored in the output tensor of the NNPF.

[0135] Post-filter characteristics of neural networks (SEI message)

[0136] Table 2 shows an example of the NNPFC SEI message syntax.

[0137] [Table 2]

[0138] The Neural Network Post-Filter Characteristics (NNPFC) SEI message specifies the neural network that can be used as a post-processing filter. The Neural Network Post-Filter Activation (NNPFA) SEI message indicates the use of a specific Neural Network Post-Processing Filter (NNPF) for a particular image.

[0139] To use this SEI message, the following variables need to be defined: - The width and height of the input image, expressed in units of brightness samples, are represented in this paper as CroppedWidth and CroppedHeight, respectively.

[0140] - A luminance sample array CroppedYPic[idx] and chrominance sample arrays CroppedCbPic[idx] and CroppedCrPic[idx] (if present) of an input image with an index idx in the range of 0 to numInputPics-1 (inclusive), which are used as inputs for NNPF.

[0141] - BitDepthY, the bit depth of the luminance sample array used for the input image.

[0142] - BitDepthC, if any, of the chroma sample array used for the input image.

[0143] - Chroma format indicator, referred to as ChromaFormatIdc in this document.

[0144] - When nnpfc_auxiliary_inp_idc equals 1, the filter strength control value array StrengthControlVal[idx] should contain real numbers in the range 0 to 1 (inclusive) for the input image with an index idx in the range 0 to numInputPics-1 (inclusive).

[0145] The input image with index 0 corresponds to the image activated by the NNPFA SEI message for the NNPF defined by this NNPFC SEI message. The input image with index i in the range of 1 to numInputPics-1 (inclusive) precedes the input image with index i-1 in the output order.

[0146] The variables SubWidthC and SubHeightC are derived from ChromaFormatIdc, as specified in Table 2.

[0147] For the same image, there can be more than one NNPFC SEI message. When more than one NNPFC SEI message with different nnpfc_id values ​​exists or is activated for the same image, they can have the same or different nnpfc_purpose and nnpfc_mode_idc values.

[0148] The nnpfc_purpose indicates the purpose of NNPF as specified in Table 3.

[0149] [Table 3]

[0150] Here, nnpfc_purpose indicates the purpose of the NNPF as specified in Table 20, where (nnpfc_purpose & bitMask) not equal to 0 indicates that the NNPF has a purpose associated with the bitMask value in Table 3. When nnpfc_purpose is greater than 0 and (nnpfc_purpose & bitMask) is equal to 0, the purpose associated with the bitMask value is not applicable to the NNPF. When nnpfc_pupose is equal to 0, the NNPF determined by the application can be used.

[0151] In this version of the bitstream conforming to this document, the value of nnpfc_purpose should be in the range of 0 to 63 (inclusive). Values ​​of 64 to 65535 (inclusive) for nnpfc_purpose are reserved for future use by ITU-T|ISO / IEC and should not exist in this version of the bitstream conforming to this document. Decoders conforming to this version of this document will ignore NNPFC SEI messages with nnfc_purpose in the range of 64 to 65535 (inclusive).

[0152] As shown in Table 4, the derived variables chromaUpsamplingFlag, resolutionResamplingFlag, pictureRateUpsamplingFlag, bitDepthUpsamplingFlag, and coulourizationFlag specify whether nnpfc_purpose indicates the purpose of NNPF, which includes chroma upsampling, resolution resampling, picture rate upsampling, bit depth upsampling, and coulourizationFlag, respectively. [Table 4]

[0153] When the reserved value of nnpfc_purpose is used in the future, the syntax of this SEI message can be extended with the existence of a syntax element that conditions nnpfc_purpose equal to the value.

[0154] When ChromaFormatIdc equals 3, chromaUpsamplingFlag should equal 0.

[0155] When ChromaFormatIdc or chromaUpsamplingFlag is not equal to 0, colourizationFlag should be equal to 0.

[0156] When the pictureRateUpsamplingFlag is equal to 1 and the input picture with index 0 is associated with a frame encapsulation arrangement SEI message with fp_arrangement_type equal to 5, all input pictures are associated with a frame encapsulation arrangement SEI message with the same values ​​of fp_arrangement_type and fp_current_frame_is_frame0_flag equal to 5.

[0157] nnpfc_id contains an identifier that can be used to identify NNPF. The value of nnpfc_id should be between 0 and 2. 32 -2 (including 0 and 2) 32 The range is from 256 to 511 (inclusive) and from 2... 31 to 2 32 -2 (including 2) 31 and 2 32 The value of nnpfc_id in -2) is reserved for future use by ITU-T|ISO / IEC. When nnpfc_id is in the range of 256 to 511 (inclusive) or in the range of 2... 31 to 2 32 -2 (including 2) 31 and 2 32 For NNPFC SEI messages within the range of -2), the decoder conforming to this version of this document should ignore the SEI messages.

[0158] The following applies when the NNPFC SEI message is the first NNPFC SEI message with a specific nnpfc_id value in the current CLVS in decoding order: - This SEI message specifies the underlying NNPF.

[0159] - This SEI message is output in the order of the currently decoded image and all subsequently decoded images in the current layer, until the current CLVS ends.

[0160] An nnpfc_base_flag value of 1 specifies that the SEI message specifies the underlying NNPF. An nnpfc_base_flag value of 1 specifies that the SEI message specifies updates related to the underlying NNPF.

[0161] The following constraints apply to the value of nnpfc_base_flag: - When the NNPFC SEI message is the first NNPFC SEI message in the decoding order that has a specific nnpfc_id value within the current CLVS, the value of nnpfc_base_flag should be equal to 1.

[0162] - When NNPFC SEI message nnpfcB is not the first NNPFC SEI message in the decoding order that has a specific nnpfc_id value within the current CLVS and the value of nnpfc_base_flag is equal to 1, the NNPFC SEI message should be a duplicate of the first NNPFC SEI message nnnpfcA that has the same nnpfc_id value in the decoding order. That is, the payload content of nnpfcB should be the same as the payload content of nnpfcA.

[0163] When nnpfc_base_flag equals 0, the following applies: This SEI message defines an update relative to a previous base NNPF with the same nnpfc_id value in decoding order. Updates are not cumulative; instead, each update is applied to a base NNPF, which is the NNPF specified by the first NNPFC SEI message in decoding order and has a specific npfc_id value within the current CLVS. The NNPF defined by this SEI message is obtained by applying the update defined by this SEI message relative to the base NNPF with the same nnpfc_id value.

[0164] - This SEI message relates to the currently decoded image and all subsequently decoded images in the current layer (in output order), up to the end of the current CLVS or up to, but not including, the decoded images that follow the currently decoded image in output order within the current CLVS and are associated with subsequent NNPFC SEI messages (in decoding order), which have an nnpfc_base_flag equal to 0 and the specific nnpfc_id value within the current CLVS (whichever is earlier).

[0165] An nnpfc_mode_idc value of 0 indicates that this SEI message contains an ISO / IEC 15938-17 bitstream that specifies a base NNPF (when nnpfc_base_flag equals 1) or is an update relative to a base NNPF with the same nnpfc_id value (when nnpfc_base_flag equals 0).

[0166] When nnpfc_base_flag equals 1, nnpfc_mode_idc equals 1, indicating that the underlying NNPF associated with the nnpfc_id value is a neural network identified by a URI in the format indicated by nnpfc_uri, which is identified by the tag URI nnpfc_tag_uri.

[0167] When nnpfc_base_flag equals 0, nnpfc_mode_idc equals 1, indicating that the update relative to the base NNPF with the same nnpfc_id value is defined by the URI indicated by nnpfc_uri, which has a format identified by the tag URI nnpfc_tag_uri.

[0168] In this version of the bitstream conforming to this document, the value of nnpfc_mode_idc should be in the range of 0 to 1 (inclusive). Values ​​of nnpfc_mode_idc from 2 to 255 (inclusive) are reserved for future use and should not exist in this version of the bitstream conforming to this document. Decoders conforming to this version of this document should ignore NNPFC SEI messages with nnpfc_mode_idc values ​​in the range of 2 to 255 (inclusive). Values ​​of nnpfc_mode_idc greater than 255 should not exist in this version of the bitstream conforming to this document and are not reserved for future use.

[0169] In this version of the bitstream conforming to this document, nnpfc_reserved_zero_bit_a should be equal to 0. The decoder should ignore NNPFC SEI messages where nnpfc_reserved_zero_bit_a is not equal to 0.

[0170] The nnpfc_tag_uri includes a tag URI with the syntax and semantics specified in IETF RFC4151, which identifies information about the format and associated information of the neural network used as the underlying NNPF or an update relative to the underlying NNPF with the same nnpfc_id value specified by the nnpfc_uri.

[0171] Note that nnpfc_tag_uri enables the unique identification of the format of neural network data specified by nnrpf_uri without requiring a central registry.

[0172] The nnpfc_tag_uri is equal to "g:iso.org,2023:15938-17o" indicating that neural network data identified by nnpfc_uri conforms to ISO / IEC 15938-17.

[0173] nnpfc_uri includes URIs with syntax and semantics as specified in IETF Internet Standard 66, which identify the neural network used as the underlying NNPF or an update relative to the underlying NNPF with the same nnpfc_id value.

[0174] The value of nnpfc_property_present_flag equal to 1 indicates that syntax elements related to filter purpose, input formatting, output formatting, and complexity exist. The value of nnpfc_property_present_flag equal to 0 indicates that syntax elements related to filter purpose, input formatting, output formatting, and complexity do not exist.

[0175] When nnpfc_base_flag equals 1, nnpfc_property_present_flag should equal 1.

[0176] When nnpfc_property_present_flag equals 0, the values ​​of all syntax elements that could exist only when nnpfc_property_present_flag equals 1 are inferred to be equal to their corresponding syntax elements in the NNPFC SEI message that includes the underlying NNPF that provides the update for this SEI message.

[0177] When the NNPFC SEI message nnpfcCurr is not the first NNPFC SEI message in decoding order with a specific nnpfc_id value within the current CLVS, is not a duplicate of the first NNPFC SEI message with that specific npfc_id (i.e., the value of nnpfc_base_flag is equal to 0), and the value of nnpfc_property_present_flag is equal to 1, the following constraints apply: The value of nnpfc_purpose in the -NNPFC SEI message will be the same as the value of nnpfc_purpose in the first NNPFC SEI message. In the decoding order, the first NNPFC SEI message has a specific nnpfc_id value within the current CLVS.

[0178] The value of the syntax element in the -NNPFC SEI message that follows nnpfc_property_present_flag in decoding order and precedes nnpfc_complexity_info_present_flag will be the same as the value of the corresponding syntax element in the first NNPFC SEI message that has the specified npfc_id value in the current CLVS in decoding order.

[0179] - In the first NNPFC SEI message, nnpfc_complexity_info_present_flag should be equal to 0 or both nnpfc_complexity_info_present_flag should be equal to 1, in decoding order, the first NNPFC SEI message has the specific nnpfc_id value (hereinafter referred to as nnpfcBase) within the current CLVS and all of the following apply: The nnpfc_parameter_type_idc in -npfcCurr should be equal to the nnpfc_parameter_type_idc in nnpfcBase.

[0180] The nnpfc_log2_parameter_bit_length_minus3 in -npfcCurr, if present, should be less than or equal to the nnpfc_log2_parameter_bit_length_minus3 in nnpfcBase.

[0181] - If nnpfc_num_parameters_idc in nnpfcBase is equal to 0, then nnpfc_num_parameters_idc in nnpfcCurr should be equal to 0.

[0182] Otherwise (nnpfc_num_parameters_idc is greater than 0 in nnpfcBase), nnpfc_num_parameters_idc in nnpfcCurr should be greater than 0 and less than or equal to nnpfc_num_parameters_idc in nnpfcBase.

[0183] - If nnpfc_num_kmac_operations_idc in nnpfcBase is equal to 0, then nnpfc_num_kmac_operations_idc should be equal to 0.

[0184] Otherwise (nnpfc_num_kmac_operations_idc is greater than 0 in nnpfcBase), nnpfc_num_kmac_operations_idc in nnpfcCurr should be greater than 0 and less than or equal to nnpfc_num_kmac_operations_idc in nnpfcBase.

[0185] - If nnpfc_total_kilobyte_size in nnpfcBase is equal to 0, then nnpfc_total_kilobyte_size in nnpfcCurr should be equal to 0.

[0186] Otherwise (nnpfc_total_kilobyte_size in nnpfcBase is greater than 0), nnpfc_total_kilobyte_size in nnpfcCurr should be greater than 0 and less than or equal to nnpfc_total_kilobyte_size in nnpfcBase.

[0187] `nnpfc_num_input_pics_minus1` incremented by 1 specifies the number of pictures used as input for NNPF. The value of `nnpfc_num_input_pics_minus1` should be in the range of 0 to 63 (inclusive). When `pictureRateUpsamplingFlag` is equal to 1, the value of `nnpfc_num_input_pics_minus1` should be greater than 0.

[0188] The variable `numInputPics`, which specifies the number of images used as input for NNPF, is exported as shown in Table 5. [Table 5]

[0189] `nnpfc_input_pic_output_flag[i]` equal to 1 indicates that NNPF generates the corresponding output image for the i-th input image. `nnpfc_input_pic_output_flag[i]` equal to 0 indicates that NNPF will not generate the corresponding output image for the i-th input image. When `nnpfc_num_input_pics_minus1` equals 0, it is inferred that `nnpfc_input_pic_output_flag[0]` equals 1. When `pictureRateUpsamplingFlag` equals 0 and `nnpfc_num_input_pics_minus1` is greater than 0, for at least one value of `i` in the range of 0 to `nnpfc_num_input_pics_minus1` (inclusive), `nnpfc_input_pic_output_flag[i]` should equal 1.

[0190] An nnpfc_absent_input_pic_zero_flag value of 1 indicates that NNPF expects input images that are not present in the bitstream to be represented by an array of samples with a value of 0. An nnpfc_absent_input_pic_flag value of 0 indicates that NNPF expects input images that are not present in the bitstream to be represented by the nearest neighbor input image in the bitstream in the output order.

[0191] When `chromaUpsamplingFlag` equals 1, `nnpfc_out_sub_c_flag` specifies the values ​​of variables `outSubWidthC` and `outSubHeightC`. `nnpfc_out_sub_c_flag` equal to 1 specifies that `outSubWidthC` and `outSubHeightC` are both 1. `nnpfc_out_sub_c_flag` equal to 0 specifies that `outSubWidthC` is 2 and `outSubHeightC` is 1. When `ChromaFormatIdc` equals 2 and `nnpfc_out_sub_c_flag` exists, its value should be 1.

[0192] When `colourizationFlag` equals 1, `nnpfc_out_colour_format_idc` specifies the color format of the NNPF output, and thus specifies the values ​​of the variables `outSubWidthC` and `outSubHeightC`. `nnpfc_out_colour_format_idc` equal to 1 specifies that the NNPF output color format is 4:2:0, and both `outSubWidthC` and `outSubHeightC` are equal to 2. `nnpfc_out_colour_format_idc` equal to 2 specifies that the NNPF output color format is 4:2:2, and both `outSubWidthC` and `outSubHeightC` are equal to 1. `nnpfc_out_colour_format_idc` equal to 3 indicates that the NNPF color format output is 4:4:4, and both `outSubWidthC` and `outSubHeightC` are equal to 1. The value of `nnpfc_out_colour_format_idc` should not be equal to 0.

[0193] When both chromaUpsamplingFlag and colourizationFlag are equal to 0, it is inferred that outSubWidthC and outSubHeightC are equal to SubWidthC and SubHeightC, respectively.

[0194] `nnpfc_pic_width_num_minus1+1` and `nnpfc_pic_width_denom_minus1+1` specify the numerator and denominator of the resampling rate of the NNPF output image width relative to `CroppedWidth`, respectively. The value of `(nnpfc_pic_width_num_minus1+1) ÷ (nnpfc_pic_width_denom_minus1+1)` should be in the range of `l÷16` to `l16` (inclusive). When `nnpfc_pic_width_num_minus1` and `nnpfc_pic_width_denom_minus1` do not exist, their values ​​are inferred to be equal to 0.

[0195] The variable nnpfcOutputPicWidth represents the width of the brightness sample array of the image generated by applying the NNPF identified by nnpfc_id to the input image, as derived in Table 6: [Table 6]

[0196] The requirement for bitstream consistency is that the value of nnpfcOutputPicHeight%outSubHeightC should be equal to 0.

[0197] `nnpfc_pic_height_num_minus1 + 1` and `nnpfc_pic_height_denom_minus1 + 1` specify the numerator and denominator of the NNPF output image height resampling rate relative to `CroppedHeight`, respectively. The value of `(nnpfc_pic_height_num_minus1 + 1) ÷ (nnpfc_pic_height_denom_minus1 + 1)` should be in the range of `l ÷ 16` to `l 16` (inclusive). When `nnpfc_pic_height_num_minus1` and `nnpfc_pic_height_denom_minus1` do not exist, their values ​​are inferred to be equal to 0.

[0198] The variable nnpfcOutputPicHeight represents the height of the luminance sample array of the image generated by applying the NNPF identified by nnpfc_id to the input image, as derived from Table 7: [Table 7]

[0199] The requirement for bitstream consistency is that the value of nnpfcOutputPicHeight%outSubHeightC should be equal to 0.

[0200] When nnpfc_pic_width_num_minus1, nnpfc_pic_width_denom_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denom_minus1 exist, at least one of the following should be true: The value of -nnpfcOutputPicWidth is not equal to CroppedWidth.

[0201] The value of -nnpfcOutputPicHeight is not equal to CroppedHeight.

[0202] `nnpfc_interpolated_pics[i]` specifies the number of interpolated pictures generated by NNPF between the i-th picture and the (i+1)-th picture used as input for NNPF. The value of `nnpfc_interpolated_pics[i]` should be in the range of 0 to 63 (inclusive). For at least one value of `i` in the range of 0 to `nnpfc_num_input_pics_minus-1-1` (inclusive), the value of `nnpfc_interpolated_pics[i]` should be greater than 0.

[0203] The variable `NumInpPicsInOutputTensor` specifies the number of images that have corresponding input images and exist in the NNPF output tensor. `InpIdx[idx]` specifies the index of the `idx`-th input image that exists in the NNPF output tensor and has corresponding input images. `numOutputPics` specifies the total number of images that exist in the NNPF output tensor, as shown in Table 8. [Table 8]

[0204] A value of 1 for nnpfc_component_last_flag indicates that the last dimension of both the input tensor and the output tensor generated from NNPF is used for the current channel. A value of 0 for nnpfc_component_last_flag indicates that the third dimension of both the input tensor and the output tensor generated from NNPF is used for the current channel.

[0205] Note that the first dimension in both the input and output tensors is used for the batch index, which is a practice in some neural network frameworks. While the formula in the semantics of this SEI message uses the batch size corresponding to a batch index equal to 0, this depends on the post-processing implementation to determine the batch size used as input for neural network inference.

[0206] Note that, for example, when nnpfc_inp_order_idc equals 3 and nnpfc_auxiliary_inp_idc equals 1, the input tensor has 7 channels, including four luminance matrices, two chrominance matrices, and one auxiliary input matrix. In this case, the procedure DerivedInputTensors() will derive each of these 7 channels of the input tensor one by one, and when processing a particular channel among these channels, that channel is referred to as the current channel during the procedure.

[0207] `nnpfc_inp_format_idc` indicates the method for converting sample values ​​from the input image into input values ​​for NNPF. When `nnpfc_inp_format_idc` equals 0, the input values ​​for NNPF are real numbers, and the functions `InpY()` and `InpC()` are specified as shown in Table 9. [Table 9]

[0208] When nnpfc_inp_format_idc equals 1, the input values ​​for NNPF are unsigned integers, and the functions InpY() and InpC() are specified as shown in Tables 10 and 11: [Table 10]

[0209] [Table 11]

[0210] The variable inpTensorBitDepthY is derived from the syntax element nnpfc_inp_tensor_luma_bitdepth_minus8 specified below. The variable inpTensorBitDepthC is derived from the syntax element nnpfc_inp_tensor_chroma_bitdepth_minus8 specified below.

[0211] Values ​​of nnpfc_inp_format_idc greater than 1 are reserved for future ITU-T|ISO / IEC specifications and should not exist in bitstreams conforming to this version of the document. Decoders conforming to this version of the document should ignore NNPFC SEI messages containing reserved values ​​of nnpfc_inp_format_idc.

[0212] A value greater than 0 for nnpfc_auxiliary_inp_idc indicates that the auxiliary input data exists in the input tensor of NNPF, a value equal to 0 for nnpfc_auxiliary_inp_idc indicates that the auxiliary input data does not exist in the input tensor, and a value equal to 1 for nnpfc_auxiliary_inp_idc specifies that the auxiliary input data is derived as specified in Equation 85.

[0213] In this version of the bitstream that conforms to this document, the value of nnpfc_auxiliary_inp_idc should be in the range of 0 to 1 (inclusive).

[0214] Values ​​of nnpfc_auxiliary_inp_idc ranging from 2 to 255 (inclusive) are reserved for future use and should not exist in bitstreams conforming to this version of the document. Decoders conforming to this version of the document should ignore NNPFC SEI messages with nnpfc_auxiliary_inp_idc values ​​in the range of 2 to 255 (inclusive). Values ​​of nnpfc_auxiliary_inp_idc greater than 255 should not exist in bitstreams conforming to this version of the document and are not reserved for future use.

[0215] nnpfc_inp_order_idc indicates the method for sorting the sample array of input images to form the input tensor of NNPF.

[0216] In this version of the bitstream conforming to this document, the value of nnpfc_inp_order_idc should be in the range of 0 to 3 (inclusive). Values ​​of nnpfc_inp_order_idc from 4 to 255 (inclusive) are reserved for future use by ITU-T|ISO / IEC and should not exist in this version of the bitstream conforming to this document. Decoders conforming to this version of this document should ignore NNPFC SEI messages with nnpfc_inp_order_idc values ​​in the range of 4 to 255 (inclusive). Values ​​of nnpfc_inp_order_idc greater than 255 should not exist in this version of the bitstream conforming to this document and are not reserved for future use.

[0217] When ChromaFormatIdc is not equal to 1, nnpfc_inp_order_idc should not be equal to 3.

[0218] When ChromaFormatIdc equals 0, nnpfc_inp_order_idc should equal 0.

[0219] When chromaUpsamplingFlag equals 1, nnpfc_inp_order_idc should not equal 0.

[0220] Table 12 contains information describing the nnpfc_inp_order_idc value.

[0221] [Table 12]

[0222] The table mentioned above Figure 5 This is a diagram used to explain the interleaving method used to derive the luminance channel.

[0223] `nnpfc_inp_tensor_luma_bitdepth_minus8` plus 8 specifies the bit depth of the luminance sample values ​​in the input integer tensor. The values ​​of `inpTensorBitDepthY` are derived as shown in Table 13. [Table 13]

[0224] The requirement for bitstream consistency is that the value of nnpfc_inp_tensor_luma_bitdepth_minus8 should be in the range of 0 to 24, inclusive (0 and 24).

[0225] nnpfc_inp_tensor_chroma_bitdepth_minus8 plus 8 specifies the bit depth of chroma sample values in the input integer tensor. The value of inpTensorBitDepthC is derived as follows: [Table 14]

[0226] The requirement for bitstream consistency is that the value of nnpfc_inp_tensor_chroma_bitdepth_minus8 shall be in the range of 0 to 24, inclusive (of 0 and 24).

[0227] When nnpfc_auxiliary_inp_idc is equal to 1, the variable strengthControlScaledVal is derived as follows: [Table 15]

[0228] A patch is a rectangular array of samples from a component (e.g., a luma or chroma component) of a picture.

[0229] The process DerivedInputTensors() for deriving an input tensor inputTensor for a given vertical sample coordinate cTop and a horizontal sample coordinate cLeft, which specifies the top-left sample position of the patch of samples included in the input tensor, is specified in Table 16 as follows: [Table 16]

[0230] nnpfc_out_format_idc equal to 0 indicates that sample values output by NNPF are real numbers, wherein the value range from 0 to 1 (inclusive of 0 and 1) is linearly mapped to the unsigned integer value range from 0 to (1<<bitDepth)-1 (inclusive of 0 and (1<<bitDepth)-1) for subsequent post-processing or display.

[0231] When nnpfc_out_format_idc is equal to 1, it indicates that the luminance sample values output by NNPF are unsigned integers in the range from 0 to (1 << outTensorBitDepthY) - 1 (inclusive of 0 and (1 << outTensorBitDepthY) - 1), and the chrominance sample values output by NNPF are unsigned integers in the range from 0 to (1 << outTensorBitDepthC) - 1 (inclusive of 0 and (1 << outTensorBitDepthC) - 1).

[0232] Values of nnpfc_out_format_idc greater than 1 are reserved for future use by ITU-T | ISO / IEC, and shall not be present in a bitstream conforming to this version of the present document. A decoder conforming to this version of the present document shall ignore NNPFC SEI messages that contain a reserved value of nnpfc_out_format_idc.

[0233] nnpfc_out_order_idc indicates the output order of samples generated by NNPF.

[0234] In a bitstream conforming to this version of the present document, the value of nnpfc_out_order_idc shall be in the range from 0 to 3 (inclusive of 0 and 3). Values of nnpfc_out_order_idc from 4 to 255 (inclusive of 4 and 255) are reserved for future use by ITU-T | ISO / IEC, and shall not be present in a bitstream conforming to this version of the present document. A decoder conforming to this version of the present document shall ignore NNPFC SEI messages that contain nnfc_out_order_idc in the range from 4 to 255 (inclusive of 4 and 255). Values of nnpfc_out_order_idc greater than 255 shall not be present in a bitstream conforming to this version of the present document and are not reserved for future use.

[0235] When chromaUpsamplingFlag is equal to 1, nnpfc_out_order_idc shall not be equal to 0 or 3.

[0236] When colourizationFlag is equal to 1, nnpfc_out_order_idc shall not be equal to 0.

[0237] Table 17 contains a description of the information for values of nnpfc_out_order_idc.

[0238] [Table 17]

[0239] `nnpfc_out_tensor_luma_bitdepth_minus8` plus 8 specifies the bit depth of the luminance sample values ​​in the output integer tensor. The value of `nnpfc_out_tensor_luma_bitdepth_minus8` should be in the range of 0 to 24 (inclusive). The values ​​of `outTensorBitDepthY` are derived as shown in Table 18. [Table 18]

[0240] `nnpfc_out_tensor_chroma_bitdepth_minus8` plus 8 specifies the bit depth of the chroma sample values ​​in the output integer tensor. The value of `nnpfc_out_tensor_chroma_bitdepth_minus8` should be in the range of 0 to 24 (inclusive). The values ​​of `outTensorBitDepthC` are derived as shown in Table 19. [Table 19]

[0241] When bitDepthUpsamplingFlag equals 1, the value of nnpfc_out_format_idc will be equal to 1 and at least one of the following conditions will be true: There exists nnpfc_out_tensor_luma_bitdepth_minus8, and outTensorBitDepthY is greater than BitDepthY.

[0242] -nnpfc_out_tensor_chroma_bitdepth_minus8 exists, and outTensorBitDepthC is greater than BitDepthC.

[0243] When nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepthY is greater than inpTensorBitDepthY, outTensorBitDepthC will not be less than inpTensorBitDepthC.

[0244] When nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepthY is greater than inpTensorBitDepthY, outTensorBitDepthC must not be less than inpTensorBitDepthC. When nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepthC is greater than inpTensorBitDepthC, outTensorBitDepthY must not be less than inpTensorBitDepthY.

[0245] The StoreOutputTensors() procedure, for a given vertical sample coordinate cTop and horizontal sample coordinate cLeft of the top-left sample position of the patch of samples included in the input tensor, derives the sample values ​​from the filtered output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic from the output tensor outputTensor, as specified in Table 20: [Table 20]

[0246] `nnpfc_separate_colour_description_present_flag` equal to 1 indicates that a different combination of primary colors, transport properties, matrix coefficients, and scaling and offset values ​​related to the matrix coefficients of the image generated by NNPF is specified in the SEI message syntax structure. `nnpfc_separate_colour_description_present_flag` equal to 0 indicates that the combination of primary colors, transport properties, matrix coefficients, and scaling and offset values ​​related to the matrix coefficients of the image generated by NNPF is the same as indicated in the CLVS VUI parameters.

[0247] nnpfc_colour_primaries has the same semantics as specified for the vui_colour_primaries syntax element in sub-clause 7.3, with the following differences: - nnpfc_colour_primaries specifies the color primaries of the image produced by NNPF as specified in the application SEI message, instead of the color primaries used for CLVS.

[0248] - When nnpfc_colour_primaries does not exist in the NNPFC SEI message, it is inferred that the value of nnpfc_colour_primaries is equal to vui_colour_primaries.

[0249] nnpfc_transfer_characteristics has the same semantics as specified in sub-clause 7.3 for the vui_transfer_characteristics syntax element, with the following differences: - nnpfc_transfer_characteristics specifies the transfer characteristics of the image generated by NNPF as specified in the application SEI message, instead of the transfer characteristics used for CLVS.

[0250] - When nnpfc_transfer_characteristics does not exist in the NNPFC SEI message, it is inferred that the value of nnpfc_transfer_characteristics is equal to vui_transfer_characteristics.

[0251] The `nnpfc_matrix_coeffs` parameter describes the equations used to derive luminance and chrominance signals from the green, blue, and red, or Y, Z, and X primary colors. Its semantics apply to images generated using the NNPF specified in this SEI message and are identical to those specified for `MatrixCoefficients` in Rec.ITU-TH.273|ISO / IEC 23091-2, where `BitDepthY` and `BitDepthC` are equal to `outTensorBitDepthY` and `outTensorBitDepthC`, respectively.

[0252] When nnpfc_matrix_coeffs does not exist in the NNPFC SEI message, it is inferred that the value of nnpfc_matrix_coeffs is equal to vui_matrix_coeffs.

[0253] nnpfc_matrix_coeffs should not be equal to 0 unless both of the following conditions are true: -nnpfc_out_tensor_chroma_bitdepth_minus8 is equal to nnpfc_out_tensor_luma_bitdepth_minus8.

[0254] -nnpfc_out_order_idc equals 2, outSubHeightC equals 1, and outSubWidthC equals 1.

[0255] nnpfc_matrix_coeffs should not be equal to 8 unless one of the following conditions is true: -nnpfc_out_tensor_chroma_bitdepth_minus8 is equal to nnpfc_out_tensor_luma_bitdepth_minus8.

[0256] -npfc_out_tensor_chroma_bitdepth_minus8 equals nnpfc_out_tensor_luma_bitdepth_minus8+1, nnpfc_out_order_idc equals 2, outSubHeightC equals 1, and outSubWidthC equals 1.

[0257] `nnpfc_full_range_flag` indicates the scaling and offset values ​​applied in association with the matrix coefficients specified by `nnpfc_matrix_coeffs`. Its semantics are the same as those specified for the `VideoFullRangeFlag` parameter in Rec.ITU-TH.273|ISO / IEC23091-2. When it does not exist, the value of `nnpfc_full_range_flag` is presumed to be 0.

[0258] A value of 1 for `npfc_chroma_loc_info_present_flag` indicates the presence of the `nnpfc_chroma_sample_loc_type_frame` syntax element in the NNPFC SEI message; a value of 0 for `npfc_chroma_loc_info_present_flag` indicates its absence. When `colourizationFlag` is 0 or `nnpfc_out_colour_format_idc` is not 1, the value of `nnpfc_chroma_loc_info_present_flag` should be 0.

[0259] `nnpfc_chroma_sample_loc_type_frame` specifies the position of the output image chroma samples when it is not equal to 6 and `nnpfc_out_colour_format_idc` is equal to 1. `nnpfc_chroma_sample_loc_type_frame` being equal to 6 and `nnpfc_out_colour_format_idc` being equal to 1 indicates that the chroma sample position is unknown, unspecified, or specified in a manner not specified in this document. The value of `nnpfc_chroma_sample_loc_type_frame` should be in the range of 0 to 6 (inclusive).

[0260] nnpfc_overlap indicates the horizontal and vertical sample counts of overlap between adjacent input tensors in NNPF. The value of nnpfc_overlap should be in the range of 0 to 16383 (inclusive).

[0261] `nnpfc_constant_patch_size_flag` equal to 1 indicates that NNPF accepts the exact patch size indicated by `nnpfc_patch_width_minus1` and `nnpfc_patch_height_minus1` as input. `nnpfc_constant_patch_size_flag` equal to 0 indicates that NNPF accepts any patch size with width `inpPatchWidth` and height `inpPatchHeight` as input, such that it equals `inpPatchWidth + 2`. The width of the extended patch of nnpfc_overlap (i.e., the patch plus the overlapping area) is equal to inpPatchWidth + 2. nnpfc_overlap) is nnpfc_extended_patch_width_cd_delta_minus1+2 The value is a positive integer multiple of npfc_overlap and equal to inpPatchHeight + 2. The height of the extended patch in npfc_overlap (which is equal to inpPatchHeight + 2). nnpfc_overlap) is nnpfc_extended_patch_height_cd_delta_minus1+2 A positive integer multiple of npfc_overlap.

[0262] When nnpfc_constant_patch_size_flag equals 1, nnpfc_patch_width_minus1 incremented by 1 indicates the horizontal sample count of the patch size required for the NNPF input. The value of nnpfc_patch_width_minus1 should be in the range of 0 to Min(32766, CroppedWidth-1) (inclusive).

[0263] When nnpfc_constant_patch_size_flag equals 1, nnpfc_patch_height_minus1 incremented by 1 indicates the vertical sample count of the patch size required for the NNPF input. The value of nnpfc_patch_height_minus1 should be in the range of 0 to Min(32766, CroppedHeight-1) (inclusive).

[0264] When nnpfc_constant_patch_size_flag equals 0, nnpfc_extended_patch_width_cd_delta_minus1 is incremented by 1 and 2. `npfc_overlap` indicates the common divisor of all allowed values ​​for the width of the extended patch required for the NNPF input. The value of `nnpfc_extended_patch_width_cd_delta_minus1` should be in the range of 0 to Min(32766, CroppedWidth-1) (inclusive).

[0265] When nnpfc_constant_patch_size_flag equals 0, nnpfc_extended_patch_height_cd_delta_minus1 is incremented by 1 and 2. `npfc_overlap` indicates the common divisor of all allowed values ​​for the height of the extended patch required for the NNPF input. The value of `nnpfc_extended_patch_height_cd_delta_minus1` should be in the range of 0 to Min(32766, CroppedHeight-1) (inclusive).

[0266] Let the variables inpPatchWidth and inpPatchHeight represent the patch width and patch height, respectively.

[0267] If nnpfc_constant_patch_size_flag equals 0, then the following applies: The values ​​of -inpPatchWidth and inpPatchHeight are provided by external means not specified in this document, or set by the post-processor itself.

[0268] -inpPatchWidth+2 The value of nnpfc_overlap should be nnpfc_extended_patch_width_cd_delta_minus1+1+2 The value should be a positive integer multiple of nnpfc_overlap, and inpPatchWidth should be less than or equal to CroppedWidth. inpPatchHeight + 2 The value of nnpfc_overlap should be nnpfc_extended_patch_height_cd_delta_minus1+1+2 The value must be a positive integer multiple of npfc_overlap, and inpPatchHeight should be less than or equal to CroppedHeight.

[0269] Otherwise (npfc_constant_patch_size_flag equals 1), the value of inpPatchWidth is set to equal to nnpfc_patch_width_minus1+1, and the value of inpPatchHeight is set to equal to nnpfc_patch_height_minus1+1.

[0270] The variables outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, and outPatchCHeight are derived as shown in Table 21. [Table 21]

[0271] The requirement for bitstream consistency is outPatchWidth CroppedWidth should be equal to nnpfcOutputPicWidth inpPatchWidth and outPatchHeight CroppedHeight should be equal to nnpfcOutputPicHeight inpPatchHeight.

[0272] `nnpfc_padding_type` indicates the padding process when referencing sample locations outside the boundaries of the input image as described in Table 17. In this version of the bitstream conforming to this document, the value of `nnpfc_padding_type` should be in the range of 0 to 4 (inclusive). Values ​​of `nnpfc_padding_type` from 5 to 15 (inclusive) are reserved for future use and should not exist in this version of the bitstream conforming to this document. The decoder in this version of the document will ignore NNPFC SEI messages with `nnpfc_padding_type` in the range of 5 to 15 (inclusive). Values ​​of `nnpfc_padding_type` greater than 15 should not exist in this version of the bitstream conforming to this document and are not reserved for future use.

[0273] [Table 22]

[0274] nnpfc_padding_type specifies the padding process when referencing sample locations outside the boundary of an input picture as described in Table 17. In a bitstream conforming to this document, the value of nnpfc_padding_type shall be in the range of 0 to 4, inclusive. The values 5 to 15 of nnpfc_padding_type, inclusive, are reserved for future use and shall not be present in bitstreams conforming to this version of this document. A decoder conforming to this version of this document shall ignore an NNPFC SEI message with nnfc_padding_type in the range of 5 to 15, inclusive. Values of nnpfc_padding_type greater than 15 shall not be present in bitstreams conforming to this version of this document and are not reserved for future use.

[0275] nnpfc_luma_padding_val specifies the luma value to be used for padding when nnpfc_padding_type is equal to 4. The value of nnpfc_luma_padding_val shall be in the range of 0 to (1<<BitDepthY)-1, inclusive.

[0276] nnpfc_cb_padding_val specifies the Cb value to be used for padding when nnpfc_padding_type is equal to 4. The value of nnpfc_cb_padding_val shall be in the range of 0 to (1<<BitDepthC)-1, inclusive.

[0277] nnpfc_cr_padding_val specifies the Cr value to be used for padding when nnpfc_padding_type is equal to 4. The value of nnpfc_cr_padding_val shall be in the range of 0 to (1<<BitDepthC)-1, inclusive.

[0278] The function InpSampleVal(y, x, picHeight, picWidth, croppedPic, cIdx) returns the following derived sampleVal value, where the inputs are the vertical sample position y, the horizontal sample position x, the image height picHeight, the image width picWidth, the sample array croppedPic, and the component index cIdx (for luminance equal to 0, for Cb equal to 1, and for Cr equal to 2): [Table 23]

[0279] Note that for the input to the function InpSampleVal(), the vertical position is listed before the horizontal position to be compatible with the input tensor conventions of some inference engines.

[0280] The NNPF PostProcessingFilter() is the target NNPF as derived in the semantics of the NNPFA SEI message. The example procedures in Table 24 can be used with the NNPF postprocessing filter to generate filtered and / or interpolated images in a patch-by-patch manner, containing Y, Cb, and Cr sample arrays: FilteredYPic, FilteredCbPic, and FilteredCrPic, as indicated by nnpfc_out_order_idc. [Table 24]

[0281] The image generated by NNPF with index i contains the sample arrays FilteredYPic[i], FilteredCbPic[i], and FilteredCrPic[i] (if present) derived from Equation 99. The image generated by NNPF does not include overlapping regions.

[0282] The NNPF process consists of the procedures defined in Table 24, and then outputs the NNPF-generated images in ascending index order. Here, all NNPF-generated images interpolated by NNPF are output, as well as those NNPF-generated images corresponding to any input image of NNPF, as specified in the semantics of the NNPFA SEI message.

[0283] A value of 1 for nnpfc_complexity_info_present_flag indicates the presence of one or more syntax elements that indicate the complexity of the NNPF associated with nnpfc_id. A value of 0 for nnpfc_complexity_info_present_flag indicates the absence of syntax elements that indicate the complexity of the NNPF associated with nnpfc_id.

[0284] An nnpfc_parameter_type_idc value of 0 indicates that the neural network uses only integer parameters; an nnpfc_parameter_type_flag value of 1 indicates that the neural network can use either floating-point or integer parameters; an nnpfc_parameter_type_idc value of 2 indicates that the neural network uses only binary parameters; and an nnpfc_parameter_type_idc value of 3 is reserved for future use by ITU-T|ISO / IEC and should not exist in bitstreams conforming to this version of the document. Decoders conforming to this version of the document should ignore NNPFC SEI messages with an nnpfc_parameter_type_idc value of 3.

[0285] `nnfc_log2_parameter_bit_length_minus3` equals 0, 1, and 2, with 3 indicating that the neural network does not use parameters with bit lengths greater than 8, 16, 32, and 64, respectively. When `nnpfc_parameter_type_idc` exists and `nnpfc_log2_parameter_bit_length_minus3` does not exist, the neural network does not use parameters with a bit length greater than 1.

[0286] `nnpfc_num_parameters_idc` represents the maximum number of parameters in an NNPF neural network, in powers of 2048. `nnpfc_num_parameters_idc` equal to 0 indicates that the maximum number of neural network parameters is unknown. The value of `nnpfc_num_parameters_idc` should be in the range of 0 to 52 (inclusive). Values ​​of `nnpfc_num_parameters_idc` greater than 52 are reserved for future use by ITU-T|ISO / IEC and should not exist in bitstreams conforming to this version of this document. Decoders conforming to this version of this document should ignore NNPFC SEI messages with `nnpfc_num_parameters_idc` greater than 52.

[0287] If the value of nnpfc_num_parameters_idc is greater than zero, then the variable maxNumParameters is exported as shown in Table 25: [Table 25]

[0288] The requirement for bitstream consistency is that the number of neural network parameters in NNPF should be less than or equal to maxNumParameters.

[0289] A value greater than 0 for nnpfc_num_kmac_operations_idc indicates that the maximum number of multiplication-accumulation operations per sample in NNPF is less than or equal to nnpfc_num_kmac_operations_idc. 1000. `nnpfc_num_kmac_operations_idc` equal to 0 indicates that the maximum number of multiply-accumulate operations in the network is unknown. The value of `nnpfc_num_kmac_operations_idc` should be between 0 and 2. 32 -2 (including 0 and 2) 32 Within the range of -2).

[0290] A `nnpfc_total_kilobyte_size` greater than 0 indicates the total size, in kilobytes, required to store the uncompressed parameters of the neural network. The total size in bits is a number equal to or greater than the sum of the bits used to store each parameter. `nnpfc_total_kilobyte_size` is the total size in bits divided by 8000 and rounded up. A `nnpfc_total_kilobyte_size` of 0 indicates that the total size required to store the neural network parameters is unknown. The value of `nnpfc_total_kilobyte_size` should be between 0 and 2. 32 -2 (0 and 2) 32 Within the range of -2).

[0291] A value of 0 for nnpfc_metadata_extension_num_bits indicates that nnpfc_reserved_metadata_extension does not exist. A value greater than 0 for nnpfc_metadata_extension_num_bits indicates the length of nnpfc_reserved_metadata_extension in bits. In this version of this document, nnpfc_metadata_extension_num_bits should be equal to 0. Values ​​in the range of 1 to 2048 (inclusive) for nnpfc_metadata_extension_num_bits are reserved for future use by ITU-T|ISO / IEC and should not exist in bitstreams conforming to this version of this document. Decoders conforming to this version of this document will allow any value for nnpfc_metadata_extension_num_bits in the range of 0 to 2048 (inclusive). Values ​​greater than 2048 for nnpfc_metadata_extension_num_bits should not exist in the bitstream conforming to this version of the document and are not reserved for future use.

[0292] The nnpfc_reserved_metadata_extension should not exist in the bitstream conforming to this version of the document. However, decoders conforming to this version of the document will ignore the presence and value of nnpfc_reserved_metadata_extension. When present, the bit length of nnpfc_reserved_metadata_extension is equal to nnpfc_metadata_extension_num_bits.

[0293] In this version of the bitstream conforming to this document, nnpfc_reserved_zero_bit_b should be equal to 0. The decoder should ignore NNPFC SEI messages where nnpfc_reserved_zero_bit_b is not equal to 0.

[0294] nnpfc_payload_byte[i] contains the i-th byte of a bitstream conforming to ISO / IEC 15938-17. The sequence of bytes for all current i values ​​nnpfc_payload_byte[i] should be a complete bitstream conforming to ISO / IEC 15938-17.

[0295] Neural Network Post-Filter Activation (NNPFA) SEI Message

[0296] Table 26 shows the syntax structure of NNFPA.

[0297] [Table 26]

[0298] The Neural Network Post-Filter Activation (NNPFA) SEI message activates or deactivates a target Neural Network Post-Processing Filter (NNPF) identified by nnpfa_target_id and nnpfa_target_base_flag, which can be used for post-processing filtering of a set of images. For a specific image to activate the NNPF, the target NNPF is derived as follows: - If nnpfa_target_base_flag equals 1, then the target NNPF is the base NNPF whose nnpfc_id equals nnpfa_target_id.

[0299] - Otherwise (nnpfa_target_base_flag equals 0), the target NNPF is the NNPF specified by the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id, which is in the decoding order before the first VCL NAL unit of the current image, and is not a duplicate of the NNPFC SEI message that includes the base NNPF.

[0300] Note that, for example, when NNPF is intended for different purposes or for filtering different color components, there may be several NNPFA SEI messages for the same image.

[0301] `nnpfa_target_id` indicates the target NNPF, which is specified by one or more NNPFC SEI messages that are associated with the current image and have an `nnpfc_id` equal to `nnpfa_target_id`. The value of `nnpfa_target_id` should be between 0 and 2. 32 -2 (including 0 and 2) 32 Within the range of -2).

[0302] NNPFA SEI messages with a specific value of nnpfa_target_id will not exist in the current PU unless one or both of the following conditions are true: - Within the current CLVS, there exists an NNPFC SEI message whose nnpfc_id is equal to a specific value of nnpfa_target_id that exists in the PU preceding the current PU in the decoding order.

[0303] - There exists an NNPFC SEI message with an nnpfc_id that has a specific value equal to the nnpfa_target_id in the current PU.

[0304] When a PU contains both an NNPFC SEI message with a specific value of nnpfc_id and an NNPFA SEI message with a specific value of nnpfa_target_id equal to nnpfc_id, the NNPFC SEI message will precede the NNPFA SEI message in the decoding order.

[0305] A `nnpfa_cancel_flag` value of 1 cancels the persistence of any target NNPF established by a previous NNPFASEI message with the same `nnpfa_target_id` as the current SEI message. In other words, the target NNPF is no longer used unless activated by another NNPFASEI message with the same `nnpfa_target_id` as the current SEI message and a `nnpfa_cancel_flag` value of 0. A `nnpfa_cancel_flag` value of 0 indicates that `nnpfa_target_base_flag`, `nnpfa_persistence_flag`, and `nnpfa_num_output_entries` follow.

[0306] An `nnpfa_target_base_flag` of 1 indicates that the target NNPF is the base NNPF with an `nnpfc_id` equal to `nnpfa_target_id`. An `nnpfa_target_base_flag` of 0 indicates that the target NNPF is the NNPF specified by the last NNPFC SEI message with an `nnpfc_id` equal to `nnpfa_target_id`, which is before the first VCL NAL unit of the current image in the decoding order, and is not a duplicate of the NNPFC SEI message containing the base NNPF.

[0307] nnpfa_persistence_flag specifies the persistence of the target NNPF for the current layer.

[0308] Setting nnpfa_persistence_flag to 0 specifies that the target NNPF can be used for post-processing filtering of the current image.

[0309] nnpfa_persistence_flag equal to 1 specifies that the target NNPF can be used for post-processing filtering of the current image and all subsequent images of the current layer in output order, until one or more of the following conditions are true: - A new CLVS begins for the current layer.

[0310] -End of bitstream.

[0311] - The image in the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag equal to 1 is output, following the current image in the output order.

[0312] The target NNPF should not be applied to this subsequent picture in the current layer that is associated with an NNPFA SEI message that has the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag equal to 1.

[0313] Make nnpfcTargetPictures the set of pictures associated with the last NNPFCSEI message that has an nnpfc_id equal to nnpfa_target_id, in the decoding order preceding the current NNPFA SEI message. Make nnpfaTargetPictures the set of pictures whose target NNPF is activated by the current NNPFA SEI message. The requirement for bitstream consistency is that any pictures included in nnpfTargetPictures should also be included in nnpfcTargetPictures.

[0314] nnpfa_num_output_entries specifies the number of nnpfa_output_flag[i] syntax elements present in the NNPFA SEI message. The value of nnpfa_num_output_entries should be in the range of 0 to NumInpPicsInOutputTensor (inclusive).

[0315] `nnpfa_output_flag[i]` equal to 1 indicates that the NNPF process activated by the `NNPFASEI` message outputs the NNPF-generated image corresponding to the input image with index `InpIdx[i]`, where the NNPF process is specified in the semantics of the `NNPFCSEI` message. `nnpfa_output_flag[i]` equal to 0 indicates that the NNPF process activated by the `NNPFASEI` message does not output the NNPF-generated image corresponding to the input image with index `InpIdx[i]`. When `nnpfa_num_output_entries` is less than `NumInpPicsInOutputTensor`, for each value of `i` in the range `nnpfa_num_output_entries` to `NumInpPicsInOutputTensor-1` (inclusive), it is inferred that `nnpfa_output_flag[i]` equals 1.

[0316] Neural Network Post-Filter Bank Characteristics (NNPFGC) SEI Message

[0317] Table 27 shows an example of the NNPFGC SEI message syntax.

[0318] [Table 27]

[0319] The Neural Network Post-Filter Bank Properties (NNPFGC) SEI message specifies a set of Neural Network Post-Filter Banks (NNPFs). The SEI message indicates whether this NNPF set defines a cascade of NNPFs, or defines mutually interchangeable NNPFs, or a cascade of NNPFs. The Neural Network Post-Filter Bank Activation (NNPFGA) SEI message indicates whether to use a cascade of NNPFs for a specific image.

[0320] nnpfgc_id contains an identifier number that can be used to identify NNPF groups. The value of nnpfgc_id should be between 0 and 2. 32 - The range of 2 (inclusive). From 256 to 511 (inclusive) and from 2 31 Up to 2 32 - Values ​​of nnpfgc_id in the range of 2 (inclusive) are reserved for future use by ITU-T|ISO / IEC. Values ​​with an nnpfgc_id in the range of 256 to 511 (inclusive) or 2... 31 to 2 32 - For NNPFGC SEI messages with nnpfgc_id in the range of -2 (inclusive), decoders conforming to this document for this version should ignore the SEI message. The value of nnpfgc_id must not be equal to the nnpfgc_id value of any NNPFGC SEI message existing in the same CLVS. When the nnpfgc_id value of NNPFGC SEI message nnpfgcSeiA is equal to the nnpfgc_id value of another NNPFGC SEI message nnpfgcSeiB existing in the same CLVS, nnpfgcSeiA and nnpfgcSeiB should be the same.

[0321] An nnpfgc_grouping_type value of 0 indicates that this SEI message specifies a set of cascaded neural network post-filters.

[0322] An nnpfgc_grouping_type of 1 indicates that the NNPF or NNPF group identified by nnpfgc_member_id[i] is interchangeable with each other, and the postprocessor should select only one of them to apply.

[0323] The nnpfgc_grouping_type being equal to 2 indicates that this SEI message specifies a set of NNPFs intended to be used in combination and activated in an alternating manner, such that for any image, at most one of these NNPFs is activated.

[0324] The nnpfgc_grouping_type equals 3, indicating that the NNPF or NNPF group identified by nnpfgc_member_id[i] is intended for use in parallel.

[0325] The nnpfgc_grouping_type being equal to 4 indicates that the NNPF or NNPF group identified by nnpfgc_member_id[i] is optional, that is, the postprocessor may or may not be applied.

[0326] The value of nnpfgc_grouping_type should be in the range of 0 to 255 (inclusive). Values ​​of nnpfgc_grouping_type in the range of 5 to 255 (inclusive) are reserved for future use by ITU-T|ISO / IEC and should not appear in bitstreams conforming to this document. Decoders conforming to this document should ignore NNPFGC SEI messages with nnpfgc_grouping_type in the range of 5 to 255 (inclusive).

[0327] nnpfgc_purpose has the semantics of nnpfc_purpose, but the difference is that these semantics are specified for the NNPF group defined by this SEI message, rather than for the NNPF defined by the NNPFC SEI message.

[0328] The value nnpfgc_num_members_minus2 plus 2 indicates the number of NNPFs or NNPF groups in the NNPF group defined by this SEI message.

[0329] nnpfgc_member_id[i] indicates the i-th member in the NNPF group defined by this SEI message, as follows: – If there exists an NNPF defined in CLVS with an nnpfc_id equal to nnpfgc_member_id[i], then the i-th member in the NNPF group defined by this SEI message is an NNPF with an nnpfc_id equal to nnpfgc_member_id[i].

[0330] – Otherwise (there is no NNPF defined in CLVS with nnpfc_id equal to nnpfgc_member_id[i]), then the i-th member in the NNPF group defined by this SEI message is the NNPF group with nnpfgc_id equal to nnpfgc_member_id[i].

[0331] When the value of nnpfgc_member_id[i] references the nnpfgc_id value of the NNPFGC SEI message nnpfgcSei, the bitstream consistency requirement is that the NNPFGC SEI message nnpfgcSei should have nnpfgc_grouping_type equal to 0. When nnpfgc_grouping_type equals 0 or 2, the bitstream consistency requirement is that there exists an NNPF defined in CLVS with an nnpfc_id value equal to nnpfgc_member_id[i]. When nnpfgc_grouping_type equals 1, 3, or 4, the bitstream consistency requirement is that there exists an NNPF defined in CLVS with an nnpfc_id value equal to nnpfgc_member_id[i] or an NNPF group with an nnpfgc_id value equal to nnpfgc_member_id[i].

[0332] When nnpfgc_grouping_type equals 0, NNPFs with nnpfc_id equal to nnpfgc_member_id[i] will be executed in ascending order of i, as activated by an NNPFGA SEI message with nnpfga_target_id equal to nnpfgc_id.

[0333] The semantics of nnpfgc_complexity_info_present_flag, nnpfgc_parameter_type_idc, nnpfgc_log2_parameter_bit_length_minus3, nnpfgc_num_parameters_idc, nnpfgc_num_kmac_operations_idc, and nnpfgc_total_kilobyte_size are respectively nnpfc_complexity_info_present_flag, nnpfc_parameter_type_idc, nnpfc_log2_parameter_bit_length_minus3, nnpfc_num_parameters_idc, nnpfc_num_kmac_operations_idc, and nnpfc_total_kilobyte_size. However, the difference is that these semantics are specified for the NNPF group defined by this SEI message, rather than for the NNPF defined by the NNPFC SEI message. When nnpfgc_grouping_type equals 1, nnpfgc_complexity_info_present_flag should equal 0.

[0334] Neural Network Post-Filter Bank Activation (NNPFGA) SEI Message

[0335] Table 28 shows an example of the NNPFGA SEI message syntax.

[0336] [Table 28]

[0337] The Neural Network Post-Filter Bank Activation (NNPFGA) SEI message activates or deactivates the possible use of the target Neural Network Post-Processing Filter Bank (NNPFG) identified by nnpfga_target_id for post-processing filtering of a set of images. The nnpfgc_grouping_type used for the identified NNPF group should be equal to 0 (cascaded) or 1 (alternate). When nnpfgc_grouping_type equals 1, each member of the group should have the same number of input images and NNPF output images. For a specific image where an NNPFG is activated, the target NNPFG is the NNPFG specified by the last NNPFGC SEI message with nnpfgc_id equal to nnpfga_target_id, which is located before the first VCL NAL unit of the current image in decoding order, and the NNPF of the target NNPFG is defined by the NNPFC SEI message, which has an nnpfc_id equal to any nnpfgc_member_id[i] value of the target NNPFG and exists in the current image unit or is located before the current image in decoding order.

[0338] To use this SEI message, the following variables need to be defined: – The input image width and height, in units of brightness samples, of the candidate input image with an index idx in the range of 0 to numCandInputPics - 1 (inclusive), are represented in this paper by InitCroppedWidth[idx] and InitCroppedHeight[idx], respectively, and can be used as input to NNPPFG.

[0339] – A luminance sample array InitCroppedYPic[idx] and chrominance sample arrays InitCroppedCbPic[idx] and InitCroppedCrPic[idx] (if present) of candidate input images with indices idx in the range of 0 to numCandInputPics - 1 (inclusive) can be used as input to NNPPFG.

[0340] – BitDepthY, the bit depth of the luminance sample array used for candidate input images.

[0341] – BitDepthC for the chroma sample array (if any) used for candidate input images.

[0342] – Chroma format indicator, referred to as ChromaFormatIdc in this document.

[0343] – When nnpfc_auxiliary_inp_idc equals 1, the filter strength control value array StrengthControlVal[idx] should contain real numbers in the range of 0 to 1 (inclusive) for candidate input images with indices idx from 0 to numCandInputPics - 1 (inclusive).

[0344] The candidate input image with index 0 corresponds to the image for which the NPFG is activated by the NNPFGA SEI message. The candidate input image with index i in the range of 1 to numCandInputPics – 1 (inclusive) precedes the candidate input image with index i - 1 in output order. Let candInputPicList[0] be a list of candidate input images in reverse output order.

[0345] nnpfga_target_id indicates the target NNPFG, which is specified by the NNPFGC SEI message associated with the current image, and has nnpfgc_id equal to nnpfga_target_id.

[0346] The value of nnpfga_target_id should be between 0 and 2. 32 – 2 (inclusive) range.

[0347] Unless there is an NNPFGCSEI message with a specific value of nnpfga_target_id in the current PU or in a PU preceding the current PU in the current CLVS in the decoding order, the current PU should not contain an NNPFGA SEI message with a specific value of nnpfga_target_id equal to nnpfgc_id and nnpfgc_grouping_type equal to 0.

[0348] When a PU contains both an NNPFGC SEI message with a specific value of nnpfgc_id and an NNPFGA SEI message with a specific value of nnpfga_target_id equal to nnpfgc_id, the NNPFGC SEI message should be placed before the NNPFGA SEI message in the decoding order.

[0349] A `nnpfga_cancel_flag` equal to 1 indicates that the persistence of a target NNPPG established by any previous NNPFGA SEI message with the same `nnpfga_target_id` as the current SEI message is cancelled. That is, the target NNPPG will no longer be used unless activated by another NNPFGA SEI message with the same `nnpfga_target_id` as the current SEI message and `nnpfga_cancel_flag` equal to 0. A `nnpfga_cancel_flag` equal to 0 indicates that the target NNPPG is activated for use.

[0350] nnpfga_persistence_flag specifies the persistence of the target NNPG for the current layer.

[0351] Setting nnpfga_persistence_flag to 0 specifies that the target NNPGF can only be used for post-processing filtering of the current image.

[0352] nnpfga_persistence_flag equal to 1 specifies that the target NPFG can be used to perform post-processing filtering on the current image and all subsequent images of the current layer in output order until one or more of the following conditions are true: – A new CLVS begins for the current layer.

[0353] – End of bitstream.

[0354] – The image in the current layer associated with an NNPFGA SEI message that has the same nnpfga_target_id as the current SEI message that follows the current image in the output order.

[0355] For this subsequent image in the current layer that is associated with an NNPFGA SEI message that has the same nnpfga_target_id as the current SEI message, no target NNPFG will be applied.

[0356] Let nnpfgcTargetPictures be the set of pictures belonging to the last NNPFGA SEI message with nnpfgc_id equal to nnpfga_target_id, which precedes the current NNPFGA SEI message in decoding order. Let nnpfgaTargetPictures be the set of pictures that activate the target NNPFG in the current NNPFGA SEI message. The bitstream consistency requirement is that any pictures included in nnpfgaTargetPictures should also be included in nnpfgcTargetPictures.

[0357] The increment of 2 in nnpfga_num_filters_minus2 indicates the number of NNPFs in the NNPFG activated by this SEI message. The value of nnpfga_num_filters_minus2 should be equal to the value of nnpfgc_num_members_minus2 in the NNPFGCSEI message with nnpfgc_id equal to nnpfga_target_id.

[0358] An nnpfga_target_base_flag[i] equal to 1 indicates that the i-th NNPF in the target NNPFG is the base NNPF with nnpfgc_id equal to nnpfgc_member_id[i] in the NNPFGC SEI message with nnpfgc_id equal to nnpfga_target_id. An nnpfga_target_base_flag[i] equal to 0 indicates that the i-th NNPF in the target NNPFG is the NNPF specified by the last NNPFC SEI message, which has an nnpfgc_id equal to nnpfgc_member_id[i] in the NNPFGC SEI message with nnpfgc_id equal to nnpfga_target_id, is located before the first VCL NAL unit of the current image in the decoding order, and is not a duplicate of the NNPFCSEI message containing the base NNPF.

[0359] `nnpfga_input_all_pics_flag[i]` equal to 1 specifies that the i-th NNPF input image is selected from the list of candidate input images `candInputPicList[i]` without skipping any. `nnpfga_input_all_pics_flag[i]` equal to 0 specifies that the i-th NNPF input image is selected from the list of candidate input images `candInputPicList[i]` with some candidate images skipped.

[0360] `nnpfga_num_input_pics_minus1[i]` specifies the number of input images for the i-th NNPF in the target NNPFG. When present, for an NNPF with an NNPGC SEI message containing an NNPGC_id equal to `nnpfgc_target_id` and an NNPGC_member_id equal to `nnpfgc_member_id[i]`, `nnpfga_num_input_pics_minus1[i]` should be equal to `nnpfc_num_input_pics_minus1`. When absent, for an NNPF with an NNPGC SEI message containing an NNPGC_member_id equal to `nnpfgc_target_id` and an NNPGC_member_id equal to `nnpfgc_target_id`, `nnpfga_num_input_pics_minus1[i]` is inferred to be equal to `nnpfc_num_input_pics_minus1`.

[0361] `nnpfga_input_pic_skip_count[i][j]` specifies the count of the j-th picture skipped in the list of candidate input pictures `candInputPicList[i]` when selecting an input picture for the NNPF activated for the i-th loop entry. When `nnpfga_input_pic_skip_count[i][j]` does not exist, all j values ​​in the range 0 to `nnpfga_num_input_pics_minus1[i]` (inclusive) are inferred to be equal to 0. The variable `numCandInputPics` indicates the number of candidate input pictures to `NNPFG`, derived as shown in Table 29. [Table 29]

[0362] Let the list of images in the range of 1 to nnpfga_num_filters_minus2 + 1 (inclusive) for m be the reverse output order of the candInputPicList[m]. It is initially empty and includes each image output by the NNPF process that does not have the nth loop entry of the corresponding image that already exists in candInputPicList[m] and finally includes each image existing in candInputPicList[0] that does not have the corresponding image that already exists in candInputPicList[m]. The images are formed in descending order of n in the range of 0 to m – 1 (inclusive).

[0363] When the candidate input image candInputPicList[m][idx] is the NNPF output image of the nth NNPF process with an n value less than m, for any value of m in the range of 1 to nnpfga_num_filters_minus2 + 1 (inclusive), the width and height of the candidate input image are equal to the nnpfcOutputPicWidth and nnpfcOutputPicHeight of the NNPF output image, respectively.

[0364] The list of input images for the NNPF up to the m-th loop entry, inputPicList[m], is exported as shown in Table 30: [Table 30]

[0365] The requirement for bitstream consistency is that candIdx should not exceed the number of images in candInputPicList[m].

[0366] The requirement for bitstream consistency is that for any value of m in the range of 1 to nnpfga_num_filters_minus2 + 1 (inclusive), the images present in inputPicList[m] should have the same width, height, bit depth, and chroma format.

[0367] To interpret the NNPFC SEI message with nnpfc_id equal to nnpfgc_member_id[i] in the NNPFGC SEI message with nnpfgc_id equal to nnpfga_target_id, the following variables need to be specified for the i-th cycle entry: – The variables BitDepthY, BitDepthC, and ChromaFormatIdc are used, as provided for interpreting this SEI message.

[0368] – Set CroppedWidth and CroppedHeight to be equal to the width and height of the image in inputPicList[i], respectively, in brightness samples.

[0369] – For each input image k in the range from 0 to nnpfga_num_input_pics_minus1[i], the following applies: – CroppedYPic[k], CroppedCbPic[k], and CroppedCrPic[k] (if they exist) are set to their respective sample arrays equal to inputPicList[i][k]. – When nnpfc_auxiliary_inp_idc equals 1 in an NNPF GC SEI message with nnpfc_id equal to nnpfga_target_id and nnpfc_id equal to nnpfgc_member_id[i], the following applies: – The requirement for bitstream consistency is that for any value of idx in the range of 0 to numCandInputPics - 1 (inclusive), inputPicList[i][k] is the same as candInputPicList[0][idx].

[0370] – StrengthControlVal[k] is set to be equal to InitStrengthControlVal[idx].

[0371] `nnpfga_num_output_entries[i]` specifies the number of `nnpfga_output_flag[i][j]` syntax elements present in the NNPFGA SEI message. For an NNPF GC SEI message with `nnpfc_id` equal to `nnpfga_target_id` and an NNPF with `nnpfc_member_id[i]`, the value of `nnpfga_num_output_entries[i]` should be in the range of 0 to `NumInpPicsInOutputTensor` (inclusive).

[0372] `nnpfga_output_flag[i][j]` equal to 1 indicates that the NNPF process activated by this loop entry outputs the NNPF-generated image corresponding to the input image with the index `InpIdx[j]` derived for the i-th NNPF of the target NNPFG, where the NNPF process is specified in the semantics of the NNPFC SEI message. `nnpfga_output_flag[i][j]` equal to 0 indicates that the NNPF-generated image corresponding to the input image with the index `InpIdx[j]` derived for the i-th NNPF of the target NNPFG will not be output by the NNPF process activated by this loop entry. When `nnpfga_num_output_entries[i]` is less than `NumInpPicsInOutputTensor` derived for the i-th NNPF of the target NNPFG, `nnpfga_output_flag[i][j]` is inferred to be equal to 1 for each `i` value in the range from `nnpfga_num_output_entries[i]` to `NumInpPicsInOutputTensor - 1` (inclusive).

[0373] Let NnpfgaOutputPicList, which is a list of images output by the NNPF process through NNPFG in output order, be initially empty, and form each image output by the NNPF process, including the nth loop entry of the corresponding image that does not already exist in NnpfgaOutputPicList, in descending order within the range of 0 to nnpfga_num_filters_minus2 + 1 (inclusive).

[0374] Source Image Timing Information (SPTI)

[0375] Table 31 shows an example of SPTI syntax.

[0376] [Table 31]

[0377] The Source Image Timing Information (SPTI) SEI message indicates the temporal distance between the source images associated with the corresponding decoded output image before encoding. For example, for content captured by a camera, the temporal distance between source images is the difference between the time when the image sensor was exposed in output order to produce the source image associated with the currently decoded image and the time when the image sensor was exposed to produce the source image associated with the previously decoded image. The information provided by the SPTI SEI message applies only to images starting from the current layer in the access unit containing the SPTI SEI message, and all subsequent images of the current layer in output order based on their persistence.

[0378] `spti_cancel_flag` equal to 1 indicates the persistence of any previous SPTI SEI messages canceled to apply to the output order of the current layer. `spti_cancel_flag` equal to 0 indicates that source image timing information follows.

[0379] spti_persistence_flag specifies the persistence of SPTI SEI messages used for the current layer.

[0380] A value of spti_persistence_flag equal to 0 indicates that the SPTI SEI message applies only to the currently decoded image.

[0381] `spti_persistence_flag` equal to 1 specifies that the SPTI SEI message applies to the currently decoded image and persists for all subsequent images of the current layer in output order until one or more of the following conditions are true: – A new CLVS begins for the current layer.

[0382] – End of bitstream.

[0383] – The image of the current layer in the AU associated with the SPTI SEI message is output, following the current image in the output order.

[0384] `spti_source_timing_equals_output_timing_flag` equal to 1 indicates that the timing of the source image is the same as the timing of the corresponding decoded output image. `spti_source_timing_equals_output_timing_flag` equal to 0 indicates that the timing of the source image may be different from the timing of the corresponding decoded output image.

[0385] When spti_source_timing_equals_output_timing_flag equals 1 and an image timing SEI message exists for the current image, the source image timing can be determined based on the information conveyed in the image timing SEI message.

[0386] A value of 1 for spti_source_type_present_flag indicates that the syntax element spti_source_type exists in the SEI message. A value of 0 for spti_source_type_present_flag indicates that the syntax element spti_source_type does not exist in the SEI message.

[0387] spti_source_type indicates the timing relationship between the source image and the corresponding decoded output image, as specified in Table 32.

[0388] [Table 32]

[0389] Where (spti_source_type & bitMask) is not equal to 0, it indicates that the timing relationship has an interpretation associated with the bitMask value in Table 32. When spti_source_type is greater than 0 and (spti_source_type & bitMask) is equal to 0, the interpretation associated with the bitMask value does not apply to SPTI SEI messages. When spti_source_type is equal to 0, the timing relationship can be specified by the application.

[0390] In bitstreams conforming to this document, the value of spti_source_type should be in the range of 0 to 127 (inclusive). Values ​​of 128 to 255 (inclusive) for spti_source_type are reserved for future use by ITU-T|ISO / IEC and should not appear in bitstreams conforming to this document. Decoders of this version conforming to this document should ignore SPTI SEI messages with spti_source_type in the range of 128 to 255 (inclusive).

[0391] The values ​​of (spti_source_type & 0x04) and (spti_source_type & 0x08) should be zero (i.e., spti_source_type should not indicate both high-speed imaging and time-lapse imaging at the same time).

[0392] spti_time_scale specifies the number of time units that elapse in one second. The value of spti_time_scale should not be equal to 0. For example, a time coordinate system that measures time using a 27 MHz clock has a spti_time_scale of 27,000,000.

[0393] spti_num_units_in_elemental_interval specifies the number of time units of a clock operating at a frequency of spti_time_scale Hz, which corresponds to the indicated element source image interval of consecutive images in the output order in CLVS.

[0394] The indicated element source picture interval, also represented by the variable ElementalSourcePictureInterval (in seconds), is equal to the quotient of spti_num_units_in_elemental_interval divided by spti_time_scale. For example, to represent a basic source picture interval of 0.04 seconds, spti_time_scale could be equal to 27,000,000, and spti_num_units_in_elemental_interval could be equal to 1,080,000.

[0395] Incrementing spti_max_sublayers_minus_1 by 1 specifies the maximum number of temporal sublayers used to signal the image interval scaling factor (spti_sublayer_interval_scale_factor[i]) and synthesis flag (spti_sublayer_synthesized_picture_flag[i]). If spti_max_sublayers_minus_1 does not exist, it is inferred to be equal to TemporalId.

[0396] spti_sublayer_interval_scale_factor[i] (if present) specifies the scaling factor used to determine the source image interval in output order for consecutive images in a CLVS with TemporalId less than or equal to i. A value of 0 can be used to indicate that the source image corresponding to the currently decoded output image is the same as the source image corresponding to the previously decoded output image.

[0397] The indicated source image interval associated with the output image having a TemporalId less than or equal to i, represented by the variable SourcePictureInterval[i] in seconds, is derived as shown in Table 33: [Table 33]

[0398] The variable temporaryReversalFlag is equal to (spti_source_type & 0x10) ? 1 : 0.

[0399] A value of 1 for spti_sublayer_synthesized_picture_flag[i] (if present) indicates that the decoded output image belonging to the i-th temporal sublayer is synthesized and does not correspond to the unmodified original source image. A value of 0 for spti_sublayer_synthesized_picture_flag[i] does not provide such an indication. When it does not exist, the value of spti_sublayer_synthesized_picture_flag[i] is inferred to be 0.

[0400] Note that when the TemporalId of the SPTI SEI message is greater than 0, and the SPTI SEI message persists with one or more images with a lower TemporalId, the encoder can repeat the information of the SPTI SEI message by including the information of the SPTI SEI message in one or more SPTI SEI messages with a lower TemporalId, so as to avoid information loss when images in the temporal sublayer are lost or removed.

[0401] question

[0402] The current design of SEI messages has been asserted to be inefficient for carrying large payloads. It was pointed out that the number of bits required to signal the payload size increases proportionally with the payload size; that is, 8 bits are needed for every 256 bytes of payload. Since SEI messages typically have only a few bytes of payload, this was previously asserted not to be a problem.

[0403] Discussions on solutions to this problem include designing new data structures for SEI messages, in which a fixed 16 bits are used to signal the payload size, which is asserted to be sufficient for payloads up to 64 kilobytes as shown in Table 34.

[0404] [Table 34]

[0405] As can be seen, the idea behind this is simply to use a specific fixed number of bits (i.e., 16 bits so far) to fix the signaling size. In our assertion, this may not be the optimal solution, considering the potential complexity of the encoder having to write down the size first when the payload size is large.

[0406] Compared to any other RBSP, only for SEI messages do we need to signal the size of the payload beforehand. This design choice was made because SEI messages were expected to be small in the past, thus allowing us to carry multiple SEI messages together with a single SEI RBSP, which we asserted for efficiency reasons. The result of this design is that we need to signal the size of the payload of each SEI message included in the RBSP beforehand.

[0407] For large SEIs, having multiple SEI messages within a single SEI RBSP is no longer very meaningful. If a large SEI RBSP contains a large SEI message, there is no need to signal the payload size. This design will also help the encoder avoid potential complexity / latency when processing the generation of such SEIs.

[0408] Example

[0409] The following embodiments provide solutions to the above problems. Each embodiment may be applied individually or in combination.

[0410] The embodiments presented in this disclosure are summarized below.

[0411] Define a new RBSP for SEI, which may be called a multi-functional SEI RBSP. A multi-functional SEI RBSP contains only one multi-functional SEI message data structure.

[0412] The multi-functional SEI message data structure is designed as follows: - Payload type - Payload data.

[0413] Multifunctional SEI RBSPs can be carried in two different NAL cell types: prefix multifunctional SEI NAL cells and suffix multifunctional SEI NAL cells.

[0414] Alternatively, in order to reduce the latency of generating an SEI with a large payload size, partial payload writing is allowed.

[0415] Example 1

[0416] Example 1 provides a description of the above overviews 1 and 2. Table 35 may be presented as follows.

[0417] [Table 35]

[0418] The Multifunctional Supplemental Enhancement Message (RBSP) contains information unnecessary for decoding samples of the encoded image from the VCL NAL unit. The VSEI RBSP contains a VSEI message.

[0419] In addition, Table 36 can be proposed as follows.

[0420] [Table 36]

[0421] The Multifunctional Supplemental Enhancement Message (RBSP) contains information unnecessary for decoding samples of the encoded image from the VCL NAL unit. The VSEI RBSP contains a VSEI message.

[0422] vsei_payload_type_byte is a byte representing the payload type of a versatile SEI message.

[0423] Example 2

[0424] Example 2 provides a description of the above overview 3. Table 37 can be presented as follows.

[0425] [Table 37]

[0426] Example 3

[0427] Example 3 provides a description of the above overview 4. Table 38 can be presented as follows.

[0428] [Table 38]

[0429] The Multifunctional Supplemental Enhancement Message (RBSP) contains information unnecessary for decoding samples of the encoded image from the VCL NAL unit. The VSEI RBSP contains a VSEI message.

[0430] vsei_payload_type_byte is a byte representing the payload type of a versatile SEI message.

[0431] vsei_payload_size_len specifies the number of bits used in the signaling syntax element vsei_payload_size.

[0432] Let the variable MaxSegmentSize equal to 2 vsei_payload_size -1.

[0433] vsei_payload_size[i] specifies the payload size of the i-th segment of the SEI message.

[0434] Figure 6 This is a diagram illustrating a method for decoding image information according to an embodiment of the present disclosure.

[0435] Figure 6 The terms or names described herein (e.g., names of grammatical elements or variable names, etc.) are merely examples, and the technical features of this disclosure are not limited to the terms described in Figure 21. For example, Figure 6 The image information described herein may include various information according to the embodiments described herein, and may include information described in at least one of the tables above.

[0436] The decoding method (S600) may include the following operations. These operations do not correspond to essential components of the decoding method according to the embodiment, and at least some of these operations may be omitted or additional operations may be added. Furthermore, these operations may be performed by a decoding apparatus including a memory and a processor electrically connected to the memory, and may be performed, for example, by the processor.

[0437] The decoding device can obtain SEI (Supplemental Enhancement Information) related information (S610).

[0438] As an example, a processor included in a decoding device can obtain image information including SEI-related information.

[0439] SEI-related information can take various forms and be expressed by various names. For example, SEI-related information can be a syntax element or a syntax structure that includes one or more syntax elements. Alternatively, SEI-related information can be a raw byte sequence payload (RBSP) that includes one or more syntax elements or one or more syntax structures. For example, SEI-related information can be expressed differently as sei_rbsp(), vsei_rbsp(), or versatile_sei_rbsp(), but is not limited to these.

[0440] SEI-related information can be delivered in NAL (Network Abstraction Layer) units. Specifically, SEI-related information can be delivered in two different NAL unit types. For example, SEI-related information can be delivered in both prefix SEI NAL unit types (prefix_sei_nut or prefix_vsei_nut) and suffix SEI NAL unit types (suffix_sei_nut or suffix_vsei_nut). However, this disclosure is not limited thereto.

[0441] SEI-related information may include SEI messages. Specifically, SEI-related information may consist of only one SEI message. For example, SEI-related information may include information unnecessary for decoding samples of an encoded image from a VCL NAL unit.

[0442] SEI messages can be syntax elements or syntax structures that include one or more syntax elements. For example, SEI messages can be expressed differently as sei_message(), vsei_message(), or versatile_sei_message(), but are not limited to these.

[0443] The decoding device can obtain the SEI payload (S620).

[0444] As an example, the processor of the decoding device can obtain the SEI payload from the SEI message. The SEI message may include the SEI payload and SEI payload type information.

[0445] SEI payloads may include information accompanying the image (e.g., personal information, copyright information, etc.), information related to image preprocessing, information related to image postprocessing, or information about the purpose of using the image.

[0446] The SEI payload can be a syntax element or a syntax structure that includes one or more syntax elements. For example, the SEI payload can be expressed differently as sei_payload() or vsei_payload(), but is not limited to these.

[0447] SEI payload type information can indicate the type of SEI payload.

[0448] SEI payload type information can be a syntax element or a syntax structure that includes one or more syntax elements. For example, SEI payload type information can be expressed as sei_payload_type, sei_payload_type_byte, vsei_payload_type, or vsei_payload_type_byte, but is not limited to these.

[0449] For example, the SEI message may not include information indicating the size of the SEI payload. Therefore, potential complexity / latency can be avoided when the encoder processes the generation of the corresponding SEI.

[0450] As another example, the SEI payload may include one or two or more SEI payload fragments, and the SEI message may include SEI payload size information indicating the size of each of the one or two or more SEI payload fragments. Partial payloads can be used to reduce latency in generating large payload sizes. The SEI payload size information may be a syntax element or a syntax structure including one or more syntax elements. For example, the SEI payload size information may be expressed as `sei_payload_size`, `sei_payload_size_len`, `vsei_payload_size`, or `vsei_payload_size_len`, but is not limited to these.

[0451] The decoding device can process the SEI payload (S630).

[0452] As an example, the processor of the decoding device can process the SEI payload of the SEI message.

[0453] As mentioned above, SEI-related information consists of only one SEI message, and the SEI message may not include information indicating the size of the SEI payload. Therefore, potential complexity / latency can be avoided when the encoder processes the generation of the corresponding SEI.

[0454] Figure 7 This is a diagram illustrating a method for encoding image information according to an embodiment of the present invention.

[0455] Figure 7 The terms or names described herein (e.g., names of grammatical elements or variable names, etc.) are merely examples, and the technical features of this disclosure are not limited to those described herein. Figure 7 The terms described in the text. For example... Figure 7 The image information described herein may include various information according to the embodiments described herein, and may include information described in at least one of the tables above.

[0456] The encoding method (S700) may include the following operations. These operations do not correspond to essential components of the encoding method according to the embodiment, and at least some of these operations may be omitted or additional operations may be added. Furthermore, these operations may be performed by an encoding device including a memory and a processor electrically connected to the memory, and may be performed, for example, by the processor.

[0457] The encoding device can generate the SEI payload (S710).

[0458] As an example, the processor of the encoding device can generate the SEI payload.

[0459] SEI payloads may include information accompanying the image (e.g., personal information, copyright information, etc.), information related to image preprocessing, information related to image postprocessing, or information about the purpose of using the image.

[0460] The SEI payload can be a syntax element or a syntax structure that includes one or more syntax elements. For example, the SEI payload can be expressed differently as sei_payload() or vsei_payload(), but is not limited to these.

[0461] The encoding device can generate SEI messages (S720).

[0462] As an example, the processor of the encoding device can generate an SEI message that includes the SEI payload.

[0463] SEI messages can be syntax elements or syntax structures that include one or more syntax elements. For example, SEI messages can be expressed differently as sei_message(), vsei_message(), or versatile_sei_message(), but are not limited to these.

[0464] SEI messages can further include SEI payload type information. SEI payload type information can indicate the type of SEI payload.

[0465] SEI payload type information can be a syntax element or a syntax structure that includes one or more syntax elements. For example, SEI payload type information can be expressed as sei_payload_type, sei_payload_type_byte, vsei_payload_type, or vsei_payload_type_byte, but is not limited to these.

[0466] For example, the SEI message may not include information indicating the size of the SEI payload. Therefore, potential complexity / latency can be avoided when the encoder processes the generation of the corresponding SEI.

[0467] As another example, the SEI payload may include one or two or more SEI payload fragments, and the SEI message may include SEI payload size information indicating the size of each of the one or two or more SEI payload fragments. Partial payloads can be used to reduce latency in generating large payload sizes. The SEI payload size information may be a syntax element or a syntax structure including one or more syntax elements. For example, the SEI payload size information may be expressed as `sei_payload_size`, `sei_payload_size_len`, `vsei_payload_size`, or `vsei_payload_size_len`, but is not limited to these.

[0468] The encoding device can generate SEI-related information (S730).

[0469] As an example, the processor of the encoding device can generate SEI-related information, including SEI messages.

[0470] SEI-related information can be in various forms or under various names.

[0471] For example, SEI-related information can be a syntax element or a syntax structure that includes one or more syntax elements. Alternatively, SEI-related information can be a raw byte sequence payload (RBSP) that includes one or more syntax elements or one or more syntax structures. For example, SEI-related information can be expressed differently as sei_rbsp(), vsei_rbsp(), or versatile_sei_rbsp(), but is not limited to these.

[0472] SEI-related information can be delivered in NAL (Network Abstraction Layer) units. Specifically, SEI-related information can be delivered in two different NAL unit types. For example, SEI-related information can be delivered in both prefix SEI NAL unit types (prefix_sei_nut or prefix_vsei_nut) and suffix SEI NAL unit types (suffix_sei_nut or suffix_vsei_nut). However, this disclosure is not limited thereto.

[0473] SEI-related information may include SEI messages. Specifically, SEI-related information may consist of only one SEI message. For example, SEI-related information may include information unnecessary for decoding samples of an encoded image from a VCL NAL unit.

[0474] The encoding device can encode image information including SEI-related information (S740).

[0475] As an example, the processor of the encoding device can encode image information including SEI-related information.

[0476] As mentioned above, SEI-related information consists of only one SEI message, and the SEI message may not include information indicating the size of the SEI payload. Therefore, potential complexity / latency can be avoided when the encoder processes the generation of the corresponding SEI.

[0477] Image information including SEI-related information encoded according to the above encoding method (S700) can be stored in a computer-readable storage medium. Image information including SEI-related information can be transmitted via a transmitter and / or a transmission medium.

[0478] As described above, according to the traditional large SEI message syntax (see Table 1), each large SEI message (lsei_message) includes a syntax element lsei_payload_type_byte indicating the type of the large SEI message payload and a syntax element lsei_payload_size_16bit indicating the size of the payload. Therefore, in an SEI message, the size of the payload is signaled before the payload.

[0479] This is because, as shown in Table 39 below, the existing SEI-related information (sei_rbsp) allows multiple SEI messages (sei_message or lsei_message), such that each SEI message (sei_message or lsei_message) includes a syntax element payload_size_byte or lsei_payload_type_byte that indicates the size of the payload.

[0480] [Table 39]

[0481] In this disclosure, since SEI-related information (sei_rbsp or vsei_rbsp) includes only one SEI message (sei_message or vsei_message), the efficiency of data transmission can be improved.

[0482] In other words, because SEI-related information (sei_rbsp or vsei_rbsp) consists of only one SEI message (sei_message or vsei_message), the syntax element indicating the payload size can be omitted from the SEI message (sei_message or vsei_message). Therefore, the SEI message (sei_message or vsei_message) can include only the syntax element (vsei_payload_type_byte) indicating the payload type and payload data (vsei_payload).

[0483] In this way, because the syntax element indicating the size of the payload is omitted from the SEI message (sei_message or vsei_message), the number of bits of data sent can be reduced and the efficiency of data transmission of SEI messages can be improved.

[0484] Furthermore, in this disclosure, SEI-related information (sei_rbsp or vsei_rbsp) can be delivered in different NAL unit types (prefix VSEI NAL unit and suffix VSEI NAL unit). This further improves the efficiency of SEI message data transmission.

[0485] Figure 8 This is a diagram illustrating a content streaming system that can be applied to embodiments of the present disclosure.

[0486] refer to Figure 8 The content streaming system using the embodiments of this document may mainly include an encoding server, a streaming server, a web server, a media storage device, a user device, and a multimedia input device.

[0487] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream and then sends the bitstream to the streaming server. As another example, when multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server can be omitted.

[0488] Bitstreams can be generated using the encoding methods or bitstream generation methods described in the embodiments of this document, and the stream server can temporarily store the bitstreams during the sending or receiving of the bitstreams.

[0489] A streaming server sends multimedia data to a user's device based on a user's request via a web server, and the web server acts as a medium for notifying the user of services. When a user requests a desired service from the web server, the web server delivers it to the streaming server, and the streaming server transmits the multimedia data to the user. In this scenario, the content streaming system may include a separate control server. In this case, the control server is used to control the commands / responses between devices in the content streaming system.

[0490] A streaming server can receive content from media storage and / or encoding servers. For example, when content is received from an encoding server, it can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a predetermined period of time.

[0491] Examples of user equipment may include mobile phones, smartphones, laptops, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays), digital TVs, desktop computers, digital signage, etc.

[0492] In a content streaming system, each server can operate as a distributed server, in which case the data received from each server can be distributed.

[0493] The scope of this disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable operation according to various embodiments of this disclosure to be performed on a device or computer, as well as non-transitory computer-readable media having such software or instructions stored thereon and executable on a device or computer.

[0494] [Industrial Applicability]

[0495] The embodiments of this disclosure can be used to encode / decode image information.

Claims

1. A method for decoding image information, the method comprising: Obtain the image information including SEI (Supplemental Enhancement Information) related information, wherein the SEI related information includes SEI messages; and Obtain the SEI payload from the SEI message. The SEI message includes SEI payload type information indicating the payload type of the SEI message and the SEI payload, as well as... The SEI-related information includes only one SEI message.

2. The method according to claim 1, wherein, The SEI message includes the SEI payload type information and the SEI payload, but does not contain information about the payload size of the SEI message.

3. The method according to claim 1, wherein, The SEI-related information is obtained through NAL (Network Abstraction Layer) units, and The NAL unit includes a prefix SEI NAL unit and a suffix SEI NAL unit.

4. The method according to claim 1, wherein, The SEI payload includes at least one SEI payload segment, and The SEI message includes SEI payload size information indicating the size of each of the at least one SEI payload segment.

5. A method for encoding image information, the method comprising: Generate SEI (Supplemental Enhancement Information) payload; Generate SEI payload type information that indicates the type of the SEI payload and an SEI message for the SEI payload; Generate SEI-related information including the SEI message; and The encoding includes image information related to the SEI. The SEI-related information includes only one SEI message.

6. The method according to claim 5, wherein, The SEI message includes the SEI payload type information and the SEI payload, but does not contain information about the payload size of the SEI message.

7. A method for storing a bitstream of image information, the method comprising: Obtain the bitstream of the image information; as well as Storing data including the bit stream in a computer-readable storage medium. The image information includes SEI (Supplemental Enhancement Information) related information. The SEI-related information includes only one SEI message, and The SEI message includes the SEI payload and SEI payload type information indicating the payload type of the SEI message.

8. The method according to claim 7, wherein, The SEI message includes the SEI payload type information and the SEI payload, but does not contain information about the payload size of the SEI message.

9. A method for transmitting a bitstream of image information, the method comprising: Obtain the bitstream of the image information; as well as Send data including the bit stream. The image information includes SEI (Supplemental Enhancement Information) related information. The SEI-related information includes only one SEI message, and The SEI message includes the SEI payload and SEI payload type information indicating the payload type of the SEI message.

10. The method according to claim 9, wherein, The SEI message includes the SEI payload type information and the SEI payload, but does not contain information about the payload size of the SEI message.

Images