Methods for decoding image information, methods for encoding image information, and methods for transmitting bit streams of image information.

Abstract

The invention relates to a method for decoding image information comprising steps for acquiring image information including supplemental enhancement information (SEI) indicating the processing order for groups of SEI message types, and determining the processing order based on SEI processing order information, wherein the SEI processing order information includes useful payload information indicating the type of SEI message, prefix presence information indicating whether the prefix information of this SEI message is present, and processing order information indicating the processing order for this type of SEI message, and wherein the SEI processing order information also includes parallel processing information indicating whether at least two SEI messages with the same processing order in the SEI processing order information are called in parallel. The invention also relates to a method for encoding image information and a method for transmitting the bit stream of image information.

Description

A method for decoding image information, a method for encoding image information, a method for storing a bitstream of image information, and a method for transmitting a bitstream of image information

[0001] The present disclosure relates to a method for decoding image information, a method for encoding image information, a method for storing a bitstream of image information, and / or a method for transmitting a bitstream of image information.

[0002] Recently, demand for high-resolution, high-quality images, such as HD (High Definition) and UHD (Ultra High Definition) images, has been increasing across various fields. As image data becomes higher resolution and higher quality, the amount of information transmitted, or bits, increases relative to conventional image data. This increase in information or bits transmitted leads to increased transmission and storage costs.

[0003] Accordingly, a highly efficient image compression technology is required to effectively transmit, store, and play high-resolution, high-quality image information.

[0004] The present disclosure seeks to provide an encoding / decoding method and / or device with improved encoding / decoding efficiency.

[0005] The present disclosure seeks to provide a method for supporting parallel grouping of SEIs of SEI processing order SEI messages for a coded video bitstream.

[0006] The present disclosure seeks to utilize a flag indicating whether parallel invocation of SEI messages with the same processing order is preferred to support parallel grouping of SEIs.

[0007] The present disclosure seeks to provide a method and / or a computer-readable recording medium for storing a bitstream generated by an encoding method according to the present disclosure.

[0008] The present disclosure seeks to provide a method and / or device for transmitting a bitstream generated by an encoding method according to the present disclosure.

[0009] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0010] According to one aspect of the present disclosure, a method for decoding video information may include obtaining video information including SEI processing order information including a processing order for a group of types of SEI messages, and determining a processing order for the group of types of SEI messages based on the SEI processing order information. The SEI processing order information may include payload type information indicating a type of SEI message, prefix presence information indicating whether prefix information of the SEI message exists, and processing order information indicating a processing order according to the type of the SEI message. The SEI processing order information may further include parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

[0011] According to one aspect of the present disclosure, a device for decoding image information includes a memory and a processor connected to the memory, wherein the processor obtains image information including SEI processing order information including a processing order for a group of types of SEI messages, and determines a processing order for the group of types of SEI messages based on the SEI processing order information. The SEI processing order information may include payload type information indicating a type of SEI message, prefix presence information indicating whether prefix information of the SEI message exists, and processing order information indicating a processing order according to the type of the SEI message. The SEI processing order information may further include parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

[0012] According to one aspect of the present disclosure, a method for encoding video information may include determining a processing order for a group of types of supplemental enhancement information (SEI) messages, generating SEI processing order information including the processing order for the group of types of SEI messages based on the processing order, and encoding the video information including the SEI processing order information. The SEI processing order information may include payload type information indicating a type of SEI message, prefix presence information indicating whether prefix information of the SEI message exists, and processing order information indicating a processing order according to the type of the SEI message. The SEI processing order information may further include parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

[0013] According to one aspect of the present disclosure, a device for encoding image information includes a memory and a processor connected to the memory, wherein the processor determines a processing order for a group of types of supplemental enhancement information (SEI) messages, generates SEI processing order information including the processing order for the group of types of SEI messages based on the processing order, and encodes the image information including the SEI processing order information. The SEI processing order information may include payload type information indicating a type of SEI message, prefix presence information indicating whether prefix information of the SEI message exists, and processing order information indicating a processing order according to the type of the SEI message. The SEI processing order information may further include parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

[0014] According to one aspect of the present disclosure, a method for storing a bitstream for image information may include obtaining image information including SEI processing order information including a processing order for a group of types of SEI messages, and storing data including a bitstream of the image information in a computer-readable storage medium. The SEI processing order information may include payload type information indicating a type of SEI message, prefix presence information indicating whether prefix information of the SEI message exists, and processing order information indicating a processing order according to the type of the SEI message. The SEI processing order information may further include parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

[0015] According to one aspect of the present disclosure, a computer-readable storage medium stores data including a bitstream of image information, wherein the image information may include a processing order for a group of types of supplemental enhancement information (SEI) messages. The SEI processing order information may include payload type information indicating a type of SEI message, prefix presence information indicating whether prefix information of the SEI message exists, and processing order information indicating a processing order according to the type of the SEI message. The SEI processing order information may further include parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

[0016] According to one aspect of the present disclosure, a method for transmitting a bitstream of image information may include obtaining image information including SEI processing order information including a processing order for a group of types of SEI messages, and transmitting data including a bitstream of the image information. The SEI processing order information may include payload type information indicating a type of SEI message, prefix presence information indicating whether prefix information of the SEI message exists, and processing order information indicating a processing order according to the type of the SEI message. The SEI processing order information may further include parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

[0017] According to one aspect of the present disclosure, a device for transmitting a bitstream of image information may include a processor for obtaining image information including SEI processing order information including a processing order for a group of types of SEI (supplemental enhancement information) messages, and a transmission unit for transmitting data including the bitstream of the image information. The SEI processing order information may include payload type information indicating a type of SEI message, prefix presence information indicating whether prefix information of the SEI message exists, and processing order information indicating a processing order according to the type of the SEI message. The SEI processing order information may further include parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

[0018] In relation to the method / device for decoding / encoding the above image information, the method / storage medium for storing the bitstream of the image information, and the method / device for transmitting the bitstream of the image information, a value of 1 for the parallel processing possible information may indicate that at least two SEI messages having the same processing order are called in parallel, and a value of 0 for the parallel processing possible information may indicate that SEI messages are not called in parallel.

[0019] In relation to the method / device for decoding / encoding the above image information, the method / storage medium for storing the bitstream of the image information, and the method / device for transmitting the bitstream of the image information, at least two SEI messages of the same type and having the same prefix information may have the same processing order. In addition, at least two SEI messages of the same type and having no prefix information may have the same processing order.

[0020] In relation to the method / device for decoding / encoding the above image information, the method / storage medium for storing the bitstream of the image information, and the method / device for transmitting the bitstream of the image information, the image information may further include processing order superposition information including position information of a specific SEI message within a processing order defined by the SEI processing order information. At least two SEI messages of the same type, having the same prefix information, and not included in the processing order superposition information may have the same processing order. In addition, at least two SEI messages of the same type, having no prefix information, and not included in the processing order superposition information may have the same processing order.

[0021] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0022] According to the present disclosure, an encoding / decoding method and / or device with improved encoding / decoding efficiency can be provided.

[0023] According to the present disclosure, parallel grouping can be supported in which SEI messages having the same processing order are called in parallel.

[0024] According to the present disclosure, a method and / or a computer-readable recording medium for storing a bitstream generated by an encoding method according to the present disclosure can be provided.

[0025] According to the present disclosure, a method and / or device for transmitting a bitstream generated by an encoding method according to the present disclosure can be provided.

[0026] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0027] FIG. 1 is a diagram schematically illustrating a video coding system to which an embodiment according to the present disclosure can be applied.

[0028] FIG. 2 is a schematic diagram of an encoding device to which an embodiment according to the present disclosure can be applied.

[0029] FIG. 3 is a schematic diagram showing a decoding device to which an embodiment according to the present disclosure can be applied.

[0030] Figure 4 illustrates an example of a hierarchical structure for coded video / images.

[0031] Figure 5 is a diagram for explaining an interleaved method for deriving a luma channel.

[0032] FIG. 6 is a diagram illustrating a method for decoding image information according to one embodiment of the present disclosure.

[0033] FIG. 7 is a diagram illustrating a method for encoding image information according to one embodiment of the present disclosure.

[0034] FIG. 8 is a diagram exemplifying a content streaming system to which an embodiment according to the present disclosure can be applied.

[0035] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0036] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.

[0037] In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.

[0038] In this disclosure, terms such as first, second, etc. are used solely to distinguish one component from another, and do not limit the order or importance of components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0039] In this disclosure, distinct components are used to clearly illustrate their respective characteristics, and do not necessarily imply that the components are separated. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0040] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also within the scope of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also within the scope of the present disclosure.

[0041] The present disclosure relates to encoding and decoding of video. For example, the methods and embodiments disclosed in this document can be applied to methods disclosed in the versatile video coding (VVC) standard, the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second generation of audio video coding standard (AVS2), or the next generation of video / image coding standards (e.g., H.267 or H.268).

[0042] The present disclosure presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0043] Terms used in this disclosure may have their usual meanings commonly used in the technical field to which this disclosure belongs, unless newly defined in this disclosure.

[0044] In the present disclosure, a "picture" generally means a unit representing one image of a specific time period, and a slice / tile is a coding unit constituting a part of a picture, and one picture may be composed of one or more slices / tiles. In addition, a slice / tile may include one or more CTUs (coding tree units). One picture may be composed of one or more tile groups. One tile group may include one or more tiles. A brick may represent a rectangular area of ​​CTU rows of tiles in a picture. In this document, tile group and slice may be used interchangeably. For example, in this document, a tile group / tile group header may be called a slice / slice header.

[0045] In the present disclosure, "pixel" or "pel" may refer to the smallest unit that constitutes a picture (or image). Additionally, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luma component or only a pixel / pixel value of a chroma component.

[0046] In the present disclosure, a "unit" may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "sample array," "block," or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0047] In the present disclosure, the "current block" may mean one of the following: a "current coding block," a "current coding unit," a "block to be encoded," a "block to be decoded," or a "block to be processed." When prediction is performed, the "current block" may mean a "current prediction block" or a "block to be predicted." When transformation (inverse transformation) / quantization (inverse quantization) is performed, the "current block" may mean a "current transformation block" or a "block to be transformed." When filtering is performed, the "current block" may mean a "block to be filtered."

[0048] In the present disclosure, a "current block" may mean a block that includes both a luma component block and a chroma component block, or a "luma block of the current block," unless explicitly described as a chroma block. The luma component block of the current block may be explicitly expressed by including an explicit description of the luma component block, such as "luma block" or "current luma block." Additionally, the chroma component block of the current block may be explicitly expressed by including an explicit description of the chroma component block, such as "chroma block" or "current chroma block."

[0049] In this disclosure, " / " and "," can be interpreted as "and / or". For example, "A / B" and "A, B" can be interpreted as "A and / or B". Additionally, "A / B / C" and "A, B, C" can mean "at least one of A, B, and / or C."

[0050] In this disclosure, "or" may be interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." Alternatively, "or" in this disclosure may mean "additionally or alternatively."

[0051] FIG. 1 is a schematic diagram illustrating a video / image coding system to which an embodiment according to the present disclosure can be applied.

[0052] Referring to FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0053] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / video encoding device, and the decoding device may be referred to as a video / video decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0054] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0055] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0056] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file via a predetermined file format and an element for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0057] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0058] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0059] FIG. 2 is a schematic diagram illustrating an encoding device to which an embodiment according to the present disclosure can be applied.

[0060] As illustrated in FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a prediction unit (predictor) 220, a residual processor (residual processor) 230, an entropy encoder (entropy encoder) 240, an adder (adder) 250, a filter (filter) 260, and a memory (memory) 270. The prediction unit (220) may include an inter prediction unit (221) and an intra prediction unit (222). The residual processor (230) may include a transformer (transformer) 232, a quantizer (quantizer) 233, a dequantizer (dequantizer) 234, and an inverse transformer (inverse transformer) 235. The residual processing unit (230) may further include a subtractor (231). The addition unit (250) may be called a reconstructor or a recontructed block generator. The image segmentation unit (210), the prediction unit (220), the residual processing unit (230), the entropy encoding unit (240), the addition unit (250), and the filtering unit (260) described above may be configured by one or more hardware components (e.g., an encoder chipset or a processor) depending on the embodiment. In addition, the memory (270) may include a DPB (Decoded Picture Buffer) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0061] The image segmentation unit (210) can segment an input image (or picture, frame) input to the encoding device (200) into one or more processing units. For example, the processing units may be referred to as coding units (CUs). The coding units can be obtained by recursively segmenting a coding tree unit (CTU) or a largest coding unit (LCU) according to a QT / BT / TT (Quad-tree / binary-tree / ternary-tree) structure. For example, one coding unit can be segmented into a plurality of coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary-tree structure. For segmenting a coding unit, the quad-tree structure may be applied first, and the binary-tree structure and / or the ternary-tree structure may be applied later. The coding procedure according to the present disclosure can be performed based on a final coding unit that is no longer segmented. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit may be used as the final coding unit, or, if necessary, the maximum coding unit may be recursively divided into coding units of lower depths, and the coding unit of the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and / or restoration, which will be described later. As another example, the processing unit of the coding procedure may be a prediction unit (PU) or a transformation unit (TU). The prediction unit and the transformation unit may each be divided or partitioned from the final coding unit. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0062] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0063] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input video signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, as illustrated, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input video signal (original block, original sample array) within the encoding device (200) may be called a subtraction unit (231). The prediction unit (220) can perform prediction on a block to be processed (current block) and generate a predicted block including prediction samples for the current block. The prediction unit (220) can determine whether intra prediction or inter prediction is applied on a current block or CU basis. The prediction unit (220) can generate various information regarding the prediction of the current block and transmit it to the entropy encoding unit (240). The information regarding the prediction can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0064] The intra prediction unit (222) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the intra prediction mode and / or intra prediction technique. The intra prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of detail in the prediction direction. However, this is merely an example, and a greater or lesser number of directional prediction modes may be used depending on the settings. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0065] The inter prediction unit (221) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), etc. A reference picture including the above temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (221) may construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the current block can be signaled by using the motion vector of the surrounding blocks as the motion vector predictor and encoding the motion vector difference and an indicator for the motion vector predictor. The motion vector difference can mean the difference between the motion vector of the current block and the motion vector predictor.

[0066] The prediction unit (220) can generate a prediction signal based on various prediction methods and / or prediction techniques described below. For example, the prediction unit (220) can apply intra prediction or inter prediction to predict the current block, and can also apply intra prediction and inter prediction simultaneously. A prediction method that simultaneously applies intra prediction and inter prediction to predict the current block may be called combined inter and intra prediction (CIIP). In addition, the prediction unit (220) may be based on an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described in this document. Palette mode can be viewed as an example of intracoding or intraprediction. When applied, palette mode can signal sample values ​​within a picture based on information about the palette table and palette index.

[0067] The prediction signal generated through the prediction unit (220) can be used to generate a restoration signal or a residual signal. The subtraction unit (231) can generate a residual signal (residual block, residual sample array) by subtracting the prediction signal (predicted block, predicted sample array) output from the prediction unit (220) from the input image signal (original block, original sample array). The generated residual signal can be transmitted to the conversion unit (232).

[0068] The transform unit (232) can apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. The transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0069] The quantization unit (233) can quantize the transform coefficients and transmit them to the entropy encoding unit (240). The entropy encoding unit (240) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information. The quantization unit (233) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0070] The entropy encoding unit (240) can perform various encoding methods such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit (190) can also encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information on various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaled information, transmitted information, and / or syntax elements mentioned in the present disclosure may be included in video / image information. The video / image information may be encoded through the encoding procedure described above and included in the bitstream.

[0071] The above bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) for transmitting the signal output from the entropy encoding unit (240) and / or a storage unit (not shown) for storing the signal may be provided as an internal / external element of the encoding device (200), or the transmission unit may be provided as a component of the entropy encoding unit (240).

[0072] The quantized transform coefficients output from the quantization unit (233) can be used to generate a residual signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (234) and inverse transformation unit (235), a residual signal (residual block or residual samples) can be restored.

[0073] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0074] The addition unit (250) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (221) or the intra prediction unit (222). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The addition unit (250) can be called a reconstructed unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0075] The filtering unit (260) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (260) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (270), specifically, the DPB of the memory (170). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240), as described later in the description of each filtering method. The information regarding filtering can be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0076] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter prediction unit (221). Through this, when inter prediction is applied, the encoding device (200) can avoid prediction mismatch between the encoding device (200) and the decoding device, and can also improve encoding efficiency.

[0077] The DPB in the memory (270) can store a modified reconstructed picture to be used as a reference picture in the inter prediction unit (221). The memory (270) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been reconstructed. The stored motion information can be transferred to the inter prediction unit (221) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (270) can store reconstructed samples of reconstructed blocks in the current picture and transfer them to the intra prediction unit (222).

[0078] FIG. 3 is a schematic diagram illustrating a decoding device to which an embodiment according to the present disclosure can be applied.

[0079] As illustrated in FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-prediction unit (332) and an intra-prediction unit (331). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321). The entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) described above may be configured by a single hardware component (e.g., decoder chipset or processor) depending on the embodiment. In addition, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0080] When a bitstream including video / image information is input, the decoding device (300) can restore the image by performing a process corresponding to the process performed in the encoding device (200) of FIG. 2. For example, the decoding device (300) can perform decoding using a processing unit applied in the encoding device (200). Therefore, the processing unit for decoding may be, for example, a coding unit. The coding unit may be a coding tree unit or may be obtained by dividing the maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. In addition, the restored image signal decoded and output by the decoding device (300) can be reproduced through a reproduction device (not shown).

[0081] The decoding device (300) can receive a signal output from the encoding device (200) of FIG. 2 in the form of a bitstream. The received signal can be decoded through the entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The decoding device (300) can decode a picture further based on the information on the parameter set and / or the general constraint information. The signaling / received information and / or syntax elements described below can be decoded through the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit (310) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements required for image restoration and the quantized values ​​of transform coefficients for residuals. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using information of the syntax element to be decoded and decoding information of the surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (310) is provided to the prediction unit (330), and residual values ​​on which entropy decoding is performed by the entropy decoding unit (310), i.e., quantized transform coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive a residual signal (residual block, residual samples, residual sample array). In addition, information regarding filtering among the information decoded by the entropy decoding unit (310) can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (300), or the receiving unit may be a component of an entropy decoding unit (310). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The information decoder may include the entropy decoding unit (310), and the sample decoder may include at least one of the inverse quantization unit (321), the inverse transformation unit (322), the addition unit (340), the filtering unit (350), the memory (360), the inter prediction unit (332), and the intra prediction unit (331).

[0082] The inverse quantization unit (321) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (321) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device (200). The inverse quantization unit (321) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0083] In the inverse transform unit (322), the transform coefficients can be inversely transformed to obtain a residual signal (residual block, residual sample array).

[0084] The prediction unit (330) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. Palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information about the palette table and palette index may be signaled and included in the video / image information.

[0085] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The description of the intra prediction unit (222) can be equally applied to the intra prediction unit (331). The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (331) can also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0086] The inter prediction unit (332) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes (techniques), and the information about the prediction can include information indicating the mode (technique) of inter prediction for the current block.

[0087] The addition unit (340) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (330) (including the inter prediction unit (332) and / or the intra prediction unit (331)). When there is no residual for the block to be processed, such as when the skip mode is applied, the predicted block can be used as the restoration block. The description of the addition unit (250) can be equally applied to the addition unit (340). The addition unit (340) can be called a restoration unit or a restoration block generation unit. The generated restoration signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after going through filtering as described below.

[0088] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0089] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can apply various filtering methods to the restored picture to generate a modified restored picture, and store the modified restored picture in the memory (360), specifically, in the DPB of the memory (360). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0090] The (modified) reconstructed picture stored in the DPB of the memory (360) can be used as a reference picture in the inter prediction unit (332). The memory (360) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (332) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (360) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (331).

[0091] In this specification, the embodiments described in the filtering unit (260), the inter prediction unit (221), and the intra prediction unit (222) of the encoding device (200) can be applied to the filtering unit (350), the inter prediction unit (332), and the intra prediction unit (331) of the decoding device (300) in the same or corresponding manner, respectively.

[0092] Figure 4 illustrates an example of a hierarchical structure for coded video / images.

[0093] Referring to FIG. 4, the coded image is divided into a VCL (Video Coding Layer) that handles the decoding processing of the image and the image itself, a subsystem that transmits and stores the coded information, and a NAL (Network Abstraction Layer) that exists between the VCL and the subsystem and is responsible for network adaptation functions.

[0094] In VCL, VCL data containing compressed image data (slice data) can be generated, or a parameter set containing information such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), etc., or an SEI (Supplemental Enhancement Information) message additionally required for the image decoding process can be generated.

[0095] In NAL, a NAL unit can be created by adding header information (NAL unit header) to an RBSP (Raw Byte Sequence Payload) generated from a VCL. At this time, RBSP refers to slice data, parameter sets, SEI messages, etc. generated from a VCL. The NAL unit header can include NAL unit type information that is specific to the RBSP data included in the NAL unit.

[0096] As illustrated in Fig. 4, NAL units can be divided into VCL NAL units and non-VCL NAL units according to the RBSP generated from VCL. A VCL NAL unit can refer to a NAL unit that contains information about a video (slice data), and a non-VCL NAL unit can refer to a NAL unit that contains information necessary for decoding a video (parameter set or SEI message).

[0097] The above-described VCL NAL units and non-VCL NAL units can be transmitted over a network by attaching header information according to the data specifications of the lower system. For example, NAL units can be transformed into data formats of a certain standard, such as the H.266 / VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted over various networks.

[0098] As described above, a NAL unit can be specified as a NAL unit type according to the RBSP data structure included in the NAL unit, and information about the NAL unit type can be stored and signaled in the NAL unit header.

[0099] For example, depending on whether a NAL unit contains information about a picture (slice data), it can be broadly classified into VCL NAL unit types and Non-VCL NAL unit types. The VCL NAL unit type can be classified according to the nature and type of the picture contained in the VCL NAL unit, and the Non-VCL NAL unit type can be classified according to the type of parameter set, etc.

[0100] Below are examples of NAL unit types, specified by the type of parameter set included in the Non-VCL NAL unit type.

[0101] - APS (Adaptation Parameter Set) NAL unit: Type for NAL units containing APS

[0102] - DPS (Decoding Parameter Set) NAL unit: Type for NAL unit containing DPS

[0103] - VPS (Video Parameter Set) NAL unit: Type for NAL units containing VPS

[0104] - SPS (Sequence Parameter Set) NAL unit: Type for NAL units containing SPS

[0105] - PPS (Picture Parameter Set) NAL unit: Type for NAL units containing PPS

[0106] The above-described NAL unit types have syntax information for the NAL unit type, and the syntax information can be stored and signaled in the NAL unit header. For example, the syntax information can be nal_unit_type, and NAL unit types can be specified by the nal_unit_type value.

[0107] The slice header (slice header syntax, slice header information) may include information / parameters that are commonly applicable to the slices. The APS (APS syntax) or PPS (PPS syntax) may include information / parameters that are commonly applicable to one or more slices or pictures. The SPS (SPS syntax) may include information / parameters that are commonly applicable to one or more sequences. The VPS (VPS syntax) may include information / parameters that are commonly applicable to multiple layers. The DPS (DPS syntax) may include information / parameters that are commonly applicable to the entire video. The DPS may include information / parameters related to the concatenation of CVS (coded video sequence). In the present disclosure, the High Level Syntax (HLS) may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, or slice header syntax.

[0108] In the present disclosure, image / video information encoded in an encoding device and signaled in the form of a bitstream may include information related to partitioning within a picture, intra / inter prediction information, residual information, in-loop filtering information, etc., and may also include information included in the slice header, information included in the APS, information included in the PPS, information included in the SPS, information included in the VPS, and / or information included in the DPS.

[0109] Below, the SEI message related to the present invention will be described.

[0110] Large supplemental enhancement information message (Large SEI message)

[0111] Table 1 shows an example of the large SEI message syntax.

[0112] [Table 1]

[0113]

[0114] Each large SEI message contains variables that specify the type payloadType and the size payloadSize of the large SEI message payload. The derived large SEI message payload size payloadSize is specified in bytes and must be equal to the number of RBSP bytes of the large SEI message payload. Note that the NAL unit byte sequence containing the large SEI message contains one or more emulation prevention bytes (represented by emulation_prevention_three_byte syntax elements). Since the payload size of a large SEI message is specified in RBSP bytes, the amount of emulation prevention bytes is not included in the size payloadSize of the large SEI payload.

[0115] lsei_position indicates whether the SEI message corresponds to PREFIX_SEI_NUT and SUFFIX_SEI_NUT. If lsei_position is 0, the SEI message is treated as PREFIX_SEI_NUT. If lsei_position is 1, the SEI message is treated as SUFFIX_SEI_NUT. lsei_position values ​​3 and 4 are reserved for future use and should be ignored.

[0116] lsei_relevance indicates the relevance of an SEI message to the target application. lsei_relevance ranges from 0 to 3, with 0 being the least relevant and 3 being the most relevant. Note that the relevance of an SEI message is arbitrary and its use must be specified by the target application.

[0117] lsei_reserved is reserved for future use and should be ignored.

[0118] lsei_payload_type_byte is the byte of the payload type of the large SEI message. payloadType = lsei_payload_type_byte.

[0119] payload_size_16bits is the payload size in bits of the large SEI message. payloadSize = payload_size_16bits.

[0120] A general post-processing filtering process using NNPF SEI messages (Neural-network post-filter SEI messages).

[0121] The input to this process is the bitstream BitstreamToFilter. The output of this process is a list of NNPF output pictures, ListNnpfOutputPics.

[0122] First, BitstreamToFilter is decoded, and the list CroppedDecodedPictures is set to a list of cropped decoded pictures in the output order resulting from decoding BitstreamToFilter.

[0123] Second, the filtering process for a single picture specified below is called recursively for each cropped decoded picture in CroppedDecodedPictures in output order, and one or more NNPFs are activated.

[0124] The order of the pictures in ListNnpfOutputPics is the output order.

[0125] Within ListNnpfOutputPics there must not be more than one picture associated with any particular output time instance. If for any particular picture in CroppedDecodedPictures there are multiple NNPFs enabled, and although any of the NNPFs can be selected, only one of the NNFPs can be selected and applied, the above constraints must be applied regardless of which NNPF is selected to apply to a particular picture.

[0126] Filtering process for a single picture using NNPF

[0127] This filtering process is applied to each cropped decoded picture that is referenced as the current picture within CroppedDecodedPictures and for which one or more NNPFs are activated.

[0128] When applying NNPF to the current picture, filtered and / or interpolated pictures are generated by NNPF by applying the NNPF process specified in the semantics of the NNPF SEI message to the current picture in a patch-wise manner.

[0129] When applying NNPF to the current picture, the order in which the pictures generated by NNPF are stored in the output tensor of NNPF by applying the NNPF process is the output order.

[0130] When the applied NNPF is the last NNPF applied to the current picture, the pictures generated and output by the NNPF process are included in ListNnpfOutputPics in the same order in which the pictures were stored in the output tensor of the NNPF.

[0131] NNPFC SEI message (Neural-network post-filter characteristics SEI message)

[0132] Table 2 shows an example of the NNPFC SEI message syntax.

[0133] [Table 2]

[0134]

[0135]

[0136]

[0137] The NNPFC SEI message specifies which neural networks can be used as post-processing filters. The use of specified NNPFs for a particular picture is indicated by the NNPFA SEI message (neural-network post-filter activation SEI message).

[0138] Use of this SEI message requires defining the following variables:

[0139] - The input picture width and height in Luna sample units, represented here as CroppedWidth and CroppedHeight, respectively.

[0140] - (If present) Luna sample array CroppedYPic[idx] and chroma sample arrays CroppedCbPic[idx] and CroppedCrPic[idx] of input pictures to be used as input to NNPF. Input pictures have indices idx in the range 0 to numInputPics - 1.

[0141] - Bit depth for the Luna sample array of input pictures BitDepthY

[0142] - BitDepthC, the bit depth for the chroma sample array of the input pictures (if any).

[0143] - Chroma format indicator, represented here as ChromaFormatIdc.

[0144] - When nnpfc_auxiliary_inp_idc is 1, the filtering strength control value array StrengthControlVal[idx] must contain real numbers ranging from 0 to 1 for the input pictures. The input pictures contain indices idx ranging from 0 to numInputPics - 1.

[0145] The input picture with index 0 corresponds to the picture whose NNPF defined by this NNPFC SEI message is activated by the NNPFA SEI message. The input picture with index i in the range 1 to numInputPics - 1 precedes the input picture with index i - 1 in the output order.

[0146] The variables SubWidthC and SubHeightC are derived from ChromaFormatIdc.

[0147] Note that multiple NNPFC SEI messages can exist for the same picture. When more than one NNPFC SEI message with different nnpfc_id values ​​exists or is enabled for the same picture, they can have the same or different nnpfc_purpose and nnpfc_mode_idc values.

[0148] nnpfc_purpose represents the purpose of NNPF as described in Table 3 below.

[0149] [Table 3]

[0150]

[0151] Here, (nnpfc_purpose & bitMask) indicates that NNPF has the purpose associated with the bitmask value in Table 3 if its value is not 0. If nnpfc_purpose is greater than 0 and (nnpfc_purpose & bitMask) is 0, the purpose associated with the bitmask value may not be applied to NNPF. If nnpfc_purpose is 0, NNPF may be used as determined by the application.

[0152] The nnpfc_purpose value MUST be in the range 0 to 63 inclusive for bitstreams conforming to this document. Values ​​in the range 64 to 65,535 for nnpfc_purpose are reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in bitstreams conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages with nnpfc_purpose in the range 64 to 65,535.

[0153] The variables chromaUpsamplingFlag, resolutionResamplingFlag, pictureRateUpsamplingFlag, bitDepthUpsamplingFlag, and colorizationFlag, which specify whether nnpfc_purpose indicates the purpose of NNPF, including chroma upsampling, resolution resampling, picture rate upsampling, bit depth upsampling, and colorization, are derived as shown in Table 4 below.

[0154] [Table 4]

[0155]

[0156] Note that when reserved values ​​for nnpfc_purpose are used in the future in ITU-T | ISO / IEC, the syntax of this SEI message may be extended with syntax elements whose presence is conditioned by nnpfc_purpose equal to that value.

[0157] If ChromaFormatIdc is 3, chromaUpsamplingFlag must be 0.

[0158] If ChromaFormatIdc or chromaUpsamplingFlag is non-zero, colorizationFlag must be 0.

[0159] If an input picture with pictureRateUpsamplingFlag equal to 1 and index 0 is associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5, then all input pictures are associated with frame packing arrangement SEI messages with fp_arrangement_type equal to 5 and the same fp_current_frame_is_frame0_flag value.

[0160] nnpfc_id contains an identification number that can be used to identify the NNPF. The nnpfc_id value is between 0 and 2. 32 - It must be in the range of 256 to 511 and 2 31 In 2 32 - nnpfc_id values ​​in the range 2 to 511 are reserved for future use by ITU-T | ISO / IEC. The range 256 to 511 and 2 31 In 2 32 Decoders conforming to this document that encounter an NNPFC SEI message with an nnpfc_id in the range 2 to 100 must ignore this SEI message.

[0161] If the NNPFC SEI message is the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS, then the following applies:

[0162] - This SEI message specifies the base NNPF.

[0163] - This SEI message applies to the currently decoded picture and all subsequent decoded pictures of the current layer, in output order, up to the end of the current CLVS.

[0164] nnpfc_base_flag, if its value is 1, indicates that the SEI message specifies the base NNPF. nnpfc_base_flag, if its value is 0, indicates that the SEI message specifies updates related to the base NNPF.

[0165] The following constraints apply to the nnpfc_base_flag value:

[0166] - If the NNPFC SEI message is the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS, the nnpfc_base_flag value must be 1.

[0167] - If NNPFC SEI message nnpfcB is not the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS and an nnpfc_base_flag value of 1, then the NNPFC SEI message shall be a repetition of the first NNPFC SEI message nnpfcA in decoding order with the same nnpfc_id value. That is, the payload content of nnpfcB shall be identical to that of nnpfcA.

[0168] If nnpfc_base_flag is 0, the following applies:

[0169] - This SEI message defines an update relative to the preceding base NNPF in decoding order with the same nnpfc_id value. Updates are not cumulative, but rather each update is applied to the base NNPF, which is the NNPF specified by the first NNPFC SEI message with the same nnpfc_id value in decoding order within the current CLVS. The NNPF defined by this SEI message is obtained by applying the update defined by this SEI message relative to the base NNPF with the same nnpfc_id value.

[0170] - This SEI message applies to the current decoded picture and all subsequent decoded pictures of the current layer in output order up to and including the end of the current CLVS or a decoded picture that follows the current decoded picture in output order within the current CLVS, and relates to subsequent NNPFC SEI messages in decoding order that have an nnpfc_base_flag whose value is 0 and a specific nnpfc_id value within the current CLVS, whichever is earlier.

[0171] nnpfc_mode_idc, if its value is 0, indicates that this SEI message contains an ISO / IEC 15938-17 bitstream that specifies a base NNPF (if nnpfc_base_flag is 1) or is an update relative to a base NNPF with the same nnpfc_id value (if nnpfc_base_flag is 0).

[0172] If nnpfc_base_flag is 1, nnpfc_mode_idc specifies that the base NNPF associated with the nnpfc_id value is the neural network identified by the URI indicated by nnpfc_uri, with a format identified by the tag URI nnpfc_tag_uri.

[0173] If nnpfc_base_flag is 0, nnpfc_mode_idc specifies that updates relative to the base NNPF with the same nnpfc_id value as its value is 1 are defined by the URI indicated by nnpfc_uri with the format identified by the tag URI nnpfc_tag_uri.

[0174] The nnpfc_mode_idc value MUST be in the range 0 to 1, inclusive, for a bitstream conforming to this document. Values ​​2 to 255 for nnpfc_mode_idc are reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages containing nnpfc_mode_idc in the range 2 to 255, inclusive. nnpfc_mode_idc values ​​greater than 255 MUST NOT be present in a bitstream conforming to this document and MUST NOT be present in a bitstream conforming to this document and MUST NOT be present in a bitstream conforming to this document and MUST NOT be present in a bitstream conforming to this document.

[0175] nnpfc_reserved_zero_bit_a MUST be 0 in a bitstream conforming to this document. Decoders MUST ignore NNPFC SEI messages with nnpfc_reserved_zero_bit_a that is not 0.

[0176] nnpfc_tag_uri contains a URI with syntax and semantics specified in IETF RFC 4151 that identifies relevant information and format for updates related to the neural network used as the base NNPF or to a base NNPF with the same nnpfc_id value specified by nnpfc_uri.

[0177] Note that nnpfc_tag_uri can be used to uniquely identify the format of neural network data specified by nnrpf_uri without a central registry authority.

[0178] nnpfc_tag_uri indicates that the neural network data identified by nnpfc_uri is ISO / IEC 15938-17 compliant if its value is "tag:iso.org,2023:15938-17".

[0179] nnpfc_uri contains a URI with the syntax and semantics specified in IETF Internet Standard 66 that identifies a base NNPF or an update related to a base NNPF with the same nnpfc_id value.

[0180] nnpfc_property_present_flag, if its value is 1, specifies that syntax elements related to filter purpose, input formatting, output formatting, and complexity are present. nnpfc_property_present_flag, if its value is 0, specifies that syntax elements related to filter purpose, input formatting, output formatting, and complexity are not present.

[0181] If nnpfc_base_flag is 1, nnpfc_property_present_flag must be 1.

[0182] If nnpfc_property_present_flag is 0, then all syntax element values ​​that can only be present when nnpfc_property_present_flag is 1 are inferred to match each corresponding syntax element of the NNPFC SEI message containing the base NNPF to which this SEI message provides updates.

[0183] If the NNPFC SEI message nnpfcCurr is not the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS, is not a repeat of the first NNPFC SEI message with a particular nnpfc_id value (i.e., has nnpfc_base_flag equal to 0), and has nnpfc_property_present_flag equal to 0, the following constraints apply:

[0184] - The nnpfc_purpose value of an NNPFC SEI message must be identical to the nnpfc_purpose value of the first NNPFC SEI message in decoding order with a specific nnpfc_id value within the current CLVS.

[0185] - The syntax element values ​​preceding nnpfc_complexity_info_present_flag and following nnpfc_property_present_flag in the decoding order of the NNPFC SEI message must be identical to the corresponding syntax element values ​​of the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS.

[0186] - In the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS (denoted as nnpfcBase below), either nnpfc_complexity_info_present_flag must be 0 or both nnpfc_complexity_info_present_flag must be 1, and the following applies:

[0187] - nnpfc_parameter_type_idc of nnpfcCurr must be the same as nnpfc_parameter_type_idc of nnpfcBase.

[0188] - (If present) nnpfc_log2_parameter_bit_length_minus3 of nnpfcCurr must be less than or equal to nnpfc_log2_parameter_bit_length_minus3 of nnpfcBase.

[0189] - If nnpfc_num_parameters_idc of nnpfcBase is 0, nnpfc_num_parameters_idc of nnpfcCurr must be 0.

[0190] - Otherwise (if nnpfc_num_parameters_idc of nnpfcBase is greater than 0), nnpfc_num_parameters_idc of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_num_parameters_idc of nnpfcBase.

[0191] - If nnpfc_num_kmac_operations_idc of nnpfcBase is 0, nnpfc_num_kmac_operations_idc of nnpfcCurr must be 0.

[0192] - Otherwise (if nnpfc_num_kmac_operations_idc of nnpfcBase is greater than 0), nnpfc_num_kmac_operations_idc of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_num_kmac_operations_idc of nnpfcBase.

[0193] - If nnpfc_total_kilobyte_size of nnpfcBase is 0, nnpfc_total_kilobyte_size of nnpfcCurr must be 0.

[0194] - Otherwise (if nnpfc_total_kilobyte_size of nnpfcBase is greater than 0), nnpfc_total_kilobyte_size of nnpfcCurr must be greater than 0 and less than or equal to nnpfc_total_kilobyte_size of nnpfcBase.

[0195] Meanwhile, nnpfc_num_input_pics_minus1 plus 1 specifies the number of pictures used as input to NNPF. The value of nnpfc_num_input_pics_minus1 must be in the range of 0 to 63. If pictureRateUpsamplingFlag is 1, the value of nnpfc_num_input_pics_minus1 must be greater than 0.

[0196] The variable numInputPics, which specifies the number of pictures used as input to NNPF, is derived as shown in Table 5 below.

[0197] [Table 5]

[0198]

[0199] nnpfc_input_pic_output_flag[i], if its value is 1, indicates that NNPF generates the corresponding output picture for the i-th input picture. nnpfc_input_pic_output_flag[i], if its value is 0, indicates that NNPF does not generate the corresponding output picture for the i-th input picture. If nnpfc_num_input_pics_minus1 is 0, nnpfc_input_pic_output_flag[0] is inferred to be equal to 1. If pictureRateUpsamplingFlag is 0 and nnpfc_num_input_pics_minus1 is greater than 0, nnpfc_input_pic_output_flag[i] must be 1 for at least one value of i in the range from 0 to nnpfc_num_input_pics_minus1.

[0200] nnpfc_absent_input_pic_zero_flag, when its value is 1, indicates that NNPF expects an input picture that is not present in the bitstream represented by a sample array whose sample values ​​are 0. nnpfc_absent_input_pic_zero_flag, when its value is 0, indicates that NNPF expects an input picture that is not present in the bitstream represented by the input picture that is closest in output order within the bitstream.

[0201] nnpfc_out_sub_c_flag specifies the values ​​of the variables outSubWidthC and outSubHeightC when chromaUpsamplingFlag is 1. nnpfc_out_sub_c_flag specifies that outSubWidthC is 1 and outSubHeightC is 1 when its value is 1. nnpfc_out_sub_c_flag specifies that outSubWidthC is 2 and outSubHeightC is 1 when its value is 0. If ChromaFormatIdc is 2 and nnpfc_out_sub_c_flag is present, the value of nnpfc_out_sub_c_flag must be 1.

[0202] If colorizationFlag is 1, nnpfc_out_colour_format_idc specifies the colour format of the NNPF output and the corresponding values ​​of the variables outSubWidthC and outSubHeightC. If nnpfc_out_colour_format_idc is 1, the colour format of the NNPF output is 4:2:0 format, and both outSubWidthC and outSubHeightC are 2. If nnpfc_out_colour_format_idc is 2, the colour format of the NNPF output is 4:2:2 format, outSubWidthC is 2, and outSubHeightC is 1. If nnpfc_out_colour_format_idc is 3, the colour format of the NNPF output is 4:4:4 format, and both outSubWidthC and outSubHeightC are 1. The nnpfc_out_colour_format_idc value must not be 0.

[0203] If both chromaUpsamplingFlag and colorizationFlag are 0, then outSubWidthC and outSubHeightC are inferred to be equal to SubWidthC and SubHeightC, respectively.

[0204] nnpfc_pic_width_num_minus1 plus 1 and nnpfc_pic_width_denom_minus1 plus 1 specify the numerator and denominator, respectively, for the resampling ratio of the NNPF output picture width relative to CroppedWidth. The value (nnpfc_pic_width_num_minus1 + 1) ÷ (nnpfc_pic_width_denom_minus1 + 1) must be in the range 1 ÷ 16 through 16, inclusive. If nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 are not present, the values ​​of nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 are both inferred to be equal to 0.

[0205] The variable nnpfcOutputPicWidth, which represents the width of the luma sample arrays of the picture(s) resulting from applying the NNFP identified by nnpfc_id to the input picture(s), is derived as shown in Table 6 below.

[0206] [Table 6]

[0207]

[0208] It is a bitstream conformance requirement that the value of nnpfcOutputPicWidth % outSubWidthC must be 0.

[0209] nnpfc_pic_height_num_minus1 plus 1 and nnpfc_pic_height_denom_minus1 plus 1 specify the numerator and denominator, respectively, for the resampling ratio of the NNPF output picture height relative to CroppedHeight. The value (nnpfc_pic_height_num_minus1 + 1) ÷ (nnpfc_pic_height_denom_minus1 + 1) must be in the range 1 ÷ 16 through 16, inclusive. If nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 are not present, the values ​​of nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 are both inferred to be equal to 0.

[0210] The variable nnpfcOutputPicHeight, which represents the height of the luma sample arrays of the picture(s) resulting from applying the NNPF identified by nnpfc_id to the input picture(s), is derived as shown in Table 7 below.

[0211] [Table 7]

[0212]

[0213] It is a bitstream conformance requirement that nnpfcOutputPicHeight % outSubHeightC value must be 0.

[0214] If nnpfc_pic_width_num_minus1, nnpfc_pic_width_denom_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denom_minus1 exist, then at least one of the following must be true:

[0215] - nnpfcOutputPicWidth value is not equal to CroppedWidth.

[0216] - nnpfcOutputPicHeight value is not equal to CroppedHeight.

[0217] nnpfc_interpolated_pics[i] specifies the number of interpolated pictures used by NNPF between the i-th and (i + 1)-th pictures used as input to NNPF. The value of nnpfc_interpolated_pics[i] must be in the range of 0 to 63, inclusive. The value of nnpfc_interpolated_pics[i] must be greater than 0 for at least one value of i in the range of 0 to nnpfc_num_input_pics_minus1 - 1.

[0218] The variables NumInpPicsInOutputTensor, which specifies the number of pictures existing in the output tensor of NNPF with the corresponding input picture, InpIdx[idx], which specifies the input picture index of the idxth picture existing in the output tensor of NNPF and having the corresponding input picture, and numOutputPics, which specifies the total number of pictures existing in the output tensor of NNPF, are derived as shown in Table 8 below.

[0219] [Table 8]

[0220]

[0221] nnpfc_component_last_flag, if its value is 1, indicates that the last dimension of the input tensor inputTensor for NNPF and the output tensor outputTensor generated from NNPF are used for the current channel. nnpfc_component_last_flag, if its value is 0, indicates that the third dimension of the input tensor inputTensor for NNPF and the output tensor outputTensor generated from NNPF are used for the current channel.

[0222] Note that the first dimension of the input and output tensors is used for batch indices, a common practice in some neural network frameworks. While the semantics of this SEI message use a batch size corresponding to a batch index whose value is 1, it is up to the postprocessing implementation to determine the batch size used as input for neural network inference.

[0223] Note that for example, if nnpfc_inp_order_idc is 3 and nnpfc_auxiliary_inp_idc is 1, there are 7 channels of the input tensor, including 4 luma matrices, 2 chroma matrices, and 1 auxiliary input matrix. In this case, the process DeriveInputTensors() can derive one for each of the 7 channels of the input tensor, and when a particular channel among these channels is processed, that channel will be referred to as the current channel during processing.

[0224] nnpfc_inp_format_idc indicates how to convert the sample values ​​of the input picture into input values ​​for NNPF. If nnpfc_inp_format_idc is 0, the input values ​​for NNPF are real numbers, and the functions InpY( ) and InpC( ) are specified as shown in Table 9 below.

[0225] [Table 9]

[0226]

[0227] If nnpfc_inp_format_idc is 1, the input values ​​to NNPF are unsigned integers, and the functions InpY( ) and InpC( ) are specified as shown in Tables 10 and 11 below.

[0228] [Table 10]

[0229]

[0230] [Table 11]

[0231]

[0232] The variable inpTensorBitDepthY is derived from the syntax element nnpfc_inp_tensor_luma_bitdepth_minus8 specified below. The variable inpTensorBitDepthC is derived from the syntax element nnpfc_inp_tensor_chroma_bitdepth_minus8 specified below.

[0233] nnpfc_inp_format_idc values ​​greater than 1 are reserved for future specification by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages containing reserved values ​​for nnpfc_inp_format_idc.

[0234] nnpfc_auxiliary_inp_idc indicates that auxiliary input data exists in the input tensor of NNPF if its value is greater than 0. nnpfc_auxiliary_inp_idc indicates that auxiliary input data does not exist in the input tensor if its value is 0. nnpfc_auxiliary_inp_idc indicates that the data is derived as specified in Table 12 below if its value is 1.

[0235] The nnpfc_auxiliary_inp_idc value must be in the range 0 to 1 in a bitstream that conforms to this document.

[0236] For nnpfc_auxiliary_inp_idc, values ​​in the range 2 through 255 are reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages with nnpfc_auxiliary_inp_idc in the range 2 through 255. nnpfc_auxiliary_inp_idc values ​​greater than 255 MUST NOT be present in a bitstream conforming to this document and ...

[0237] nnpfc_inp_order_idc indicates how to order the sample arrays of input pictures to form the input tensor for NNPF.

[0238] The nnpfc_inp_order_idc value MUST be in the range 0 to 3, inclusive, for a bitstream conforming to this document. Values ​​4 to 255 for nnpfc_inp_order_idc are reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages with nnpfc_inp_order_idc in the range 4 to 255, inclusive. Values ​​nnpfc_inp_order_idc greater than 255 MUST NOT be present in a bitstream conforming to this document and MUST NOT be present in a bitstream conforming to this document and MUST NOT be present in a bitstream conforming to this document.

[0239] If ChromaFormatIdc is not 1, nnpfc_inp_order_idc must be 3.

[0240] If ChromaFormatIdc is 0, nnpfc_inp_order_idc must be 0.

[0241] If chromaUpsamplingFlag is 1, nnpfc_inp_order_idc must not be 0.

[0242] Table 12 below contains information descriptions of the nnpfc_inp_order_idc values.

[0243] [Table 12]

[0244]

[0245] FIG. 5, mentioned in Table 12 above, is a drawing for explaining an interleaved method for deriving a luma channel.

[0246] nnpfc_inp_tensor_luma_bitdepth_minus8 plus 8 specifies the bit depth of the luma sample values ​​of the input integer tensor. inpTensorBitDepth Y The values ​​are derived as shown in Table 13 below.

[0247] [Table 13]

[0248]

[0249] It is a bitstream conformance requirement that the nnpfc_inp_tensor_luma_bitdepth_minus8 value be in the range 0 to 24.

[0250] nnpfc_inp_tensor_chroma_bitdepth_minus8 plus 8 specifies the bit depth of the chroma sample values ​​of the input integer tensor. inpTensorBitDepth C The values ​​are derived as shown in Table 14 below.

[0251] [Table 14]

[0252]

[0253] It is a bitstream conformance requirement that the nnpfc_inp_tensor_chroma_bitdepth_minus8 value be in the range 0 to 24.

[0254] If nnpfc_auxiliary_inp_idc is 1, the variable strengthControlScaledVal is derived as shown in Table 15 below.

[0255] [Table 15]

[0256]

[0257] A patch is a rectangular array of samples from a component of a picture (e.g., a luma or chroma component).

[0258] The process DeriveInputTensors( ) for deriving the input tensor inputTensor for the given vertical sample coordinates cTop and horizontal sample coordinates cLeft is derived as shown in Table 16 below. Here, the given vertical sample coordinates cTop and horizontal sample coordinates cLeft specify the upper-left sample location for the patch of samples contained in the input tensor.

[0259] [Table 16]

[0260]

[0261]

[0262]

[0263]

[0264] nnpfc_out_format_idc, if its value is 0, indicates that the sample values ​​output by NNPF are real numbers. Here, the value range from 0 to 1 is linearly mapped to the unsigned integer value range from 0 to (1 << bitDepth) - 1 for any desired bit depth bitDepth for subsequent post-processing or display.

[0265] nnpfc_out_format_idc, if its value is 1, indicates that the luma sample values ​​output by NNPF are unsigned integers in the range 0 to (1 << outTensorBitDepthY) - 1, and that the chroma sample values ​​output by NNPF are unsigned integers in the range 0 to (1 << outTensorBitDepthC) - 1.

[0266] nnpfc_out_format_idc values ​​greater than 1 are reserved for future specification by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages containing reserved values ​​for nnpfc_out_format_idc.

[0267] nnpfc_out_order_idc indicates the output order of samples generated as NNPF results.

[0268] The nnpfc_out_order_idc value MUST be in the range 0 to 3, inclusive, in a bitstream conforming to this document. Values ​​4 to 255 for nnpfc_out_order_idc are reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages with nnpfc_out_order_idc in the range 4 to 255, inclusive. Values ​​nnpfc_out_order_idc greater than 255 MUST NOT be present in a bitstream conforming to this document and MUST NOT be present in the future.

[0269] If chromaUpsamplingFlag is 1, nnpfc_out_order_idc must not be 0 or 3.

[0270] If colorizationFlag is 1, nnpfc_out_order_idc must not be 0.

[0271] Table 17 contains information descriptions of the nnpfc_out_order_idc values.

[0272] [Table 17]

[0273]

[0274] nnpfc_out_tensor_luma_bitdepth_minus8 plus 8 specifies the bit depth of the luma sample values ​​of the output integer tensor. The value of nnpfc_out_tensor_luma_bitdepth_minus8 must be in the range 0 to 24. outTensorBitDepth Y The value is derived as shown in Table 18 below.

[0275] [Table 18]

[0276]

[0277] nnpfc_out_tensor_chroma_bitdepth_minus8 plus 8 specifies the bit depth of the chroma sample values ​​of the output integer tensor. The value of nnpfc_out_tensor_chroma_bitdepth_minus8 must be in the range 0 to 24. outTensorBitDepth C The values ​​are derived as shown in Table 19 below.

[0278] [Table 19]

[0279]

[0280] If bitDepthUpsamplingFlag is 1, the nnpfc_out_format_idc value must be 1 and at least one of the following conditions must be true:

[0281] - nnpfc_out_tensor_luma_bitdepth_minus8 exists and outTensorBitDepth Y BitDepth Y Bigger than.

[0282] - nnpfc_out_tensor_chroma_bitdepth_minus8 exists and outTensorBitDepth C BitDepth C Bigger than.

[0283] nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepth Y inpTensorBitDepth Y If greater than outTensorBitDepth C is inpTensorBitDepth C It must not be less than.

[0284] nnpfc_inp_tensor_luma_bitdepth_minus8, nnpfc_inp_tensor_chroma_bitdepth_minus8, nnpfc_out_tensor_luma_bitdepth_minus8, and nnpfc_out_tensor_chroma_bitdepth_minus8 exist and outTensorBitDepth C inpTensorBitDepth C If greater than outTensorBitDepth Y is inpTensorBitDepth Y It must not be less than.

[0285] The process StoreOutputTensors( ) for deriving sample values ​​of filtered output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic from the output tensor outputTensor, for a given vertical sample coordinate cTop and horizontal sample coordinate cLeft, which specify the top-left sample location for a patch of samples contained in the input tensor, is specified as shown in Table 20 below.

[0286] [Table 20]

[0287]

[0288]

[0289]

[0290] nnpfc_separate_colour_description_present_flag, if its value is 1, indicates that a unique combination of colour primaries, transfer characteristics, matrix coefficients, and scaling and offset values ​​applied with respect to matrix coefficients for pictures generated in NNPF is specified in the SEI message syntax structure. nnpfc_separate_colour_description_present_flag, if its value is 0, indicates that the combination of colour primaries, transfer characteristics, matrix coefficients, and scaling and offset values ​​applied with respect to matrix coefficients for pictures generated in NNPF are the same as those indicated in the VUI parameters for CLVS.

[0291] nnpfc_colour_primaries has the same semantics as the vui_colour_primaries syntax element, except that:

[0292] - nnpfc_colour_primaries specifies the color primaries of the picture generated by applying NNPF specified in the SEI message rather than the color primaries used in CLVS.

[0293] - If nnpfc_colour_primaries is not present in the NNPFC SEI message, the value of nnpfc_colour_primaries is inferred to be equal to vui_colour_primaries.

[0294] nnpfc_transfer_characteristics has the same semantics as the vui_transfer_characteristics syntax element, except that:

[0295] - nnpfc_transfer_characteristics specifies the transfer characteristics of the picture generated by applying the NNPF specified in the SEI message rather than the transfer characteristics used in CLVS.

[0296] - If nnpfc_transfer_characteristics is not present in the NNPFC SEI message, the nnpfc_transfer_characteristics value is inferred to be equal to vui_transfer_characteristics.

[0297] nnpfc_matrix_coeffs describes the equations used to derive luma and chroma signals from green, blue, and red or Y, Z, and X sources. Its semantics are applied to pictures generated by applying the NNPF specified in this SEI message and outTensorBitDepth Y and outTensorBitDepth C BitDepth same as each Y and BitDepth C As specified for MatrixCoefficients in Rec. ITU-T H.273 | ISO / IEC 23091-2.

[0298] If nnpfc_matrix_coeffs is not present in the NNPFC SEI message, the nnpfc_matrix_coeffs value is inferred to be equal to vui_matrix_coeffs.

[0299] nnpfc_matrix_coeffs must not be zero unless all of the following conditions are true:

[0300] - nnpfc_out_tensor_chroma_bitdepth_minus8 is identical to nnpfc_out_tensor_luma_bitdepth_minus8.

[0301] - nnpfc_out_order_idc is 2, outSubHeightC is 1, and outSubWidthC is 1.

[0302] nnpfc_matrix_coeffs must not be 8 unless one of the following conditions is true:

[0303] - nnpfc_out_tensor_chroma_bitdepth_minus8 is identical to nnpfc_out_tensor_luma_bitdepth_minus8.

[0304] - nnpfc_out_tensor_chroma_bitdepth_minus8 is equivalent to nnpfc_out_tensor_luma_bitdepth_minus8 + 1, nnpfc_out_order_idc is 2, outSubHeightC is 1, and outSubWidthC is 1.

[0305] nnpfc_full_range_flag indicates the scaling and offset values ​​applied to the matrix coefficients as specified by nnpfc_matrix_coeffs. Its semantics are the same as those specified for the VideoFullRangeFlag parameter in Rec. ITU-T H.273 | ISO / IEC 23091-2. If it is not present, nnpfc_full_range_flag is assumed to be equal to 0.

[0306] nnpfc_chroma_loc_info_present_flag, if its value is 1, indicates the presence of the nnpfc_chroma_sample_loc_type_frame syntax element in the NNPFC SEI message. nnpfc_chroma_loc_info_present_flag, if its value is 0, indicates the absence of the nnpfc_chroma_sample_loc_type_frame syntax element in the NNPFC SEI message. nnpfc_chroma_loc_info_present_flag must be 0 if colorizationFlag is 0 or nnpfc_out_colour_format_idc is not 1.

[0307] If nnpfc_chroma_sample_loc_type_frame is not 6 and nnpfc_out_colour_format_idc is 1, nnpfc_chroma_sample_loc_type_frame specifies the locations of the chroma samples of the output pictures. nnpfc_chroma_sample_loc_type_frame being 6 and nnpfc_out_colour_format_idc being 1 indicates that the locations of the chroma samples are unknown, unspecified, or specified in some other sense not specified in this document. The value of nnpfc_chroma_sample_loc_type_frame must be in the range 0 to 6, inclusive.

[0308] nnpfc_overlap represents the counts of overlapping horizontal and vertical samples in adjacent input tensors of NNPF. The nnpfc_overlap value must be in the range of 0 to 16,383.

[0309] nnpfc_constant_patch_size_flag, when its value is 1, indicates that NNPF accepts as input exactly the patch size indicated by nnpfc_patch_width_minus1 and nnpfc_patch_height_minus1. nnpfc_constant_patch_size_flag, if its value is 0, indicates that NNPF accepts as input any patch size with width inpPatchWidth and height inpPatchHeight such that the width of the extended patch (i.e., patch + overlap region) equal to inpPatchWidth + 2 * nnpfc_overlap is a positive integer multiple of nnpfc_extended_patch_width_cd_delta_minus1 + 1 + 2 * nnpfc_overlap, and the height of the extended patch equal to inpPatchWidth + 2 * nnpfc_overlap is a positive integer multiple of nnpfc_extended_patch_height_cd_delta_minus1 + 1 + 2 * nnpfc_overlap.

[0310] If nnpfc_constant_patch_size_flag is 1, nnpfc_patch_width_minus1 plus 1 represents the horizontal sample counts of the patch size required as input to NNPF. The value of nnpfc_patch_width_minus1 must be in the range 0 to Min(32,766, CroppedWidth - 1).

[0311] If nnpfc_constant_patch_size_flag is 1, nnpfc_patch_height_minus1 plus 1 represents the vertical sample counts of the patch size required as input to NNPF. The value of nnpfc_patch_height_minus1 must be in the range 0 to Min(32,766, CroppedHeight - 1).

[0312] If nnpfc_constant_patch_size_flag is 0, then nnpfc_extended_patch_width_cd_delta_minus1 plus 1 plus 2 * nnpfc_overlap represents the common divisor of all allowed values ​​of extended patch width required as input to NNPF. The value of nnpfc_extended_patch_width_cd_delta_minus1 must be in the range 0 to Min(32,766, CroppedWidth - 1).

[0313] If nnpfc_constant_patch_size_flag is 0, then nnpfc_extended_patch_height_cd_delta_minus1 plus 1 plus 2 * nnpfc_overlap represents the common divisor of all allowed values ​​of extended patch height required as input to NNPF. The value of nnpfc_extended_patch_height_cd_delta_minus1 must be in the range 0 to Min(32,766, CroppedHeight - 1).

[0314] The variables inpPatchWidth and inpPatchHeight are the patch size width and patch size height, respectively.

[0315] If nnpfc_constant_patch_size_flag is 0, the following applies:

[0316] - The inpPatchWidth and inpPatchHeight values ​​are provided by external means not specified in this document or are set by the post-processor itself.

[0317] - The value inpPatchWidth + 2 * nnpfc_overlap must be a positive integer multiple of nnpfc_extended_patch_width_cd_delta_minus1 + 1 + 2 * nnpfc_overlap, and inpPatchWidth must be less than or equal to CroppedWidth. The value inpPatchHeight + 2 * nnpfc_overlap must be a positive integer multiple of nnpfc_extended_patch_height_cd_delta_minus1 + 1 + 2 * nnpfc_overlap, and inpPatchHeight must be less than or equal to CroppedHeight.

[0318] Otherwise (nnpfc_constant_patch_size_flag is 1), the inpPatchWidth value is set equal to nnpfc_patch_width_minus1 + 1, and the inpPatchHeight value is set equal to nnpfc_patch_height_minus1 + 1.

[0319] The variables outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, and outPatchCHeight are derived as shown in Table 21 below.

[0320] [Table 21]

[0321]

[0322] It is a bitstream conformance requirement that outPatchWidth * CroppedWidth must equal nnpfcOutputPicWidth * inpPatchWidth and outPatchHeight * CroppedHeight must equal nnpfcOutputPicHeight * inpPatchHeight.

[0323] nnpfc_padding_type indicates the padding process when referencing sample locations outside the boundaries of the input picture, as shown in Table 22 below.

[0324] [Table 22]

[0325]

[0326] The nnpfc_padding_type value MUST be in the range 0 to 4, inclusive, for a bitstream conforming to this document. The values ​​5 to 15 for nnpfc_padding_type are reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages with nnpfc_padding_type in the range 5 to 15, inclusive. The values ​​nnpfc_padding_type greater than 15 MUST NOT be present in a bitstream conforming to this document and MUST NOT be present in a bitstream conforming to this document and MUST NOT be present in a bitstream conforming to this document.

[0327] nnpfc_luma_padding_val represents the luma value used for padding when nnpfc_padding_type is 4. The nnpfc_luma_padding_val value must be in the range 0 to (1 << BitDepthY) - 1.

[0328] nnpfc_cb_padding_val represents the Cb value used for padding when nnpfc_padding_type is 4. The nnpfc_cb_padding_val value must be in the range of 0 to (1 << BitDepthC) - 1.

[0329] nnpfc_cr_padding_val represents the Cr value used for padding when nnpfc_padding_type is 4. The nnpfc_cr_padding_val value must be in the range of 0 to (1 << BitDepthC) - 1.

[0330] The function InpSampleVal(y, x, picHeight, picWidth, croppedPic, cIdx) that takes as input a vertical sample position y, a horizontal sample position x, a picture height picHeight, a picture width picWidth, a sample array croppedPic, and a component index cIdx (0 for luma, 1 for Cb, and 2 for Cr) returns a sampleVal value derived as shown in Table 23 below.

[0331] [Table 23]

[0332]

[0333] Note that for the input to the function InpSampleVal() , the vertical positions are listed before the horizontal positions for compatibility with the input tensor conventions of some inference engines.

[0334] NNPF PostProcessingFilter( ) is a target NNPF derived from the semantics of the NNPFA SEI message. To generate filtered and / or interpolated pictures in a patch-wise manner, the example process in Table 24 below can be used with NNPF PostProcessingFilter( ), where the Y, Cb, and Cr sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic are included, respectively, as indicated in nnpfc_out_order_idc.

[0335] [Table 24]

[0336]

[0337] An NNPF-generated picture containing index i, if any, contains the sample arrays FilteredYPic[i], FilteredCbPic[i], and FilteredCrPic[i] derived from Table 24. An NNPF-generated picture does not contain overlapping regions.

[0338] The NNPF process consists of the processes defined in Table 24 performed by outputting NNPF-generated pictures in their increasing index order. Here, all NNPF-generated pictures interpolated by the NNPF are output, and their corresponding NNPF-generated pictures for all input pictures to the NNPF are output as specified in the semantics of the NNPFA SEI message.

[0339] nnpfc_complexity_info_present_flag, if its value is 1, specifies that one or more syntax elements indicating the complexity of the NNPF associated with the nnpfc_id are present. nnpfc_complexity_info_present_flag, if its value is 0, specifies that one or more syntax elements indicating the complexity of the NNPF associated with the nnpfc_id are not present.

[0340] nnpfc_parameter_type_idc, if its value is 0, indicates that the network uses only integer parameters. nnpfc_parameter_type_idc, if its value is 1, indicates that the network can use either floating point or integer parameters. nnpfc_parameter_type_idc, if its value is 2, indicates that the network uses only binary parameters. nnpfc_parameter_type_idc, if its value is 3, is reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST ignore NNPFC SEI messages containing nnpfc_parameter_type_idc with a value of 3.

[0341] nnpfc_log2_parameter_bit_length_minus3, when its value is 0, 1, 2, and 3, indicates that the network will not use parameters with bit lengths greater than 8, 16, 32, and 64, respectively. If nnpfc_parameter_type_idc is present and nnpfc_log2_parameter_bit_length_minus3 is not present, the network will not use parameters with bit lengths greater than 1.

[0342] nnpfc_num_parameters_idc represents the maximum number of neural network parameters for NNPF in powers of 2,048. A value of nnpfc_num_parameters_idc of 0 indicates that the maximum number of neural network parameters is unknown. The value of nnpfc_num_parameters_idc must be in the range 0 to 52, inclusive. Values ​​of nnpfc_num_parameters_idc greater than 52 are reserved for future use by ITU-T | ISO / IEC and must not be present in a bitstream conforming to this document. Decoders conforming to this document must ignore NNPFC SEI messages with values ​​of nnpfc_num_parameters_idc greater than 52.

[0343] If the value of nnpfc_num_parameters_idc is greater than zero, the variable maxNumParameters is derived as shown in Table 25 below.

[0344] [Table 25]

[0345]

[0346] It is a requirement of bitstream conformance that the number of neural network parameters of NNPF is less than or equal to maxNumParameters.

[0347] nnpfc_num_kmac_operations_idc, if its value is greater than 0, indicates that the maximum number of multiply-accumulate operations per sample in NNPF is less than or equal to nnpfc_num_kmac_operations_idc * 1,000. nnpfc_num_kmac_operations_idc, if its value is 0, indicates that the maximum number of multiply-accumulate operations in the network is unknown. The value of nnpfc_num_kmac_operations_idc can be between 0 and 2. 32 - Must be in the range of 2.

[0348] nnpfc_total_kilobyte_size, if greater than 0, indicates the total size in kilobytes required to store the uncompressed parameters for the neural network. The total size in bits is greater than or equal to the sum of the bits used to store each parameter. nnpfc_total_kilobyte_size is the total size in bits divided by 8,000, rounded up. nnpfc_total_kilobyte_size, if 0, indicates that the total size required to store the parameters of the neural network is unknown. The value of nnpfc_total_kilobyte_size can be between 0 and 2. 32 - Must be in the range of 2.

[0349] nnpfc_metadata_extension_num_bits specifies that nnpfc_reserved_metadata_extension does not exist if its value is 0. nnpfc_metadata_extension_num_bits specifies the length of nnpfc_reserved_metadata_extension in bits if its value is greater than 0. In this document, nnpfc_metadata_extension_num_bits MUST be 0. Values ​​in the range 1 to 2,048 for nnpfc_metadata_extension_num_bits are reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in a bitstream conforming to this document. Decoders conforming to this document MUST accept all values ​​in the range 0 to 2,048, inclusive. nnpfc_metadata_extension_num_bits values ​​greater than 2,048 MUST NOT be present in bitstreams conforming to this document and are not reserved for future use.

[0350] nnpfc_reserved_metadata_extension MUST NOT be present in a bitstream conforming to this document. However, decoders conforming to this document MUST ignore the values ​​and presence of nnpfc_reserved_metadata_extension. If present, the length in bits of nnpfc_reserved_metadata_extension is equal to nnpfc_metadata_extension_num_bits.

[0351] nnpfc_reserved_zero_bit_b MUST be 0 in a bitstream conforming to this document. Decoders MUST ignore NNPFC SEI messages where nnpfc_reserved_zero_bit_b is not 0.

[0352] nnpfc_payload_byte[i] contains the ith byte of a bitstream conforming to ISO / IEC 15938-17. The byte sequence nnpfc_payload_byte[i] for all present values ​​of i must be a complete bitstream conforming to ISO / IEC 15938-17.

[0353] NNPFA SEI message (Neural-network post-filter activation SEI message)

[0354] Table 26 shows an example of the NNPFA SEI message syntax.

[0355] [Table 26]

[0356]

[0357] The NNPFA SEI message enables or disables the possible use of a target NNPF identified by nnpfa_target_id and nnpfa_target_base_flag for post-processing filtering on a set of pictures. For a particular picture for which NNPF is enabled, the target NNPF is derived as follows.

[0358] - If nnpfa_target_base_flag is 1, the target NNPF is the base NNPF with the same nnpfc_id as nnpfa_target_id.

[0359] - Otherwise (nnpfa_target_base_flag is 0), the target NNPF is the NNPF specified by the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id that precedes the first VCL NAL unit of the current picture in decoding order and is not a repeat of the NNPFC SEI message containing the base NNPF.

[0360] Note that there may be several NNPFA SEI messages for the same picture, for example, if NNPFs are used for different purposes or to filter different color components.

[0361] nnpfa_target_id indicates the target NNPF associated with the current picture and specified by one or more NNPFC SEI messages with nnpfc_id equal to nnpfa_target_id. nnpfa_target_id values ​​are in the range 0 to 2. 32 - Must be in the range of 2.

[0362] An NNPFA SEI message with a particular value of nnpfa_target_id must not exist in the current PU unless one or both of the following conditions are true:

[0363] - Currently, within the CLVS, there exists an NNPFC SEI message with an nnpfc_id that is identical to a specific value of nnpfa_target_id that exists in the PU preceding the current PU in decoding order.

[0364] - Currently, the PU has an NNPFC SEI message with an nnpfc_id equal to a specific value of nnpfa_target_id.

[0365] If a PU contains both an NNPFC SEI message with a specific value of nnpfc_id and an NNPFA SEI message with an nnpfa_target_id equal to a specific value of nnpfc_id, the NNPFC SEI message must precede the NNPFA SEI message in decoding order.

[0366] nnpfa_cancel_flag, if its value is 1, indicates that the persistence of the target NNPF established by all previous NNPFA SEI messages with the same nnpfa_target_id as the current SEI message is canceled. That is, the target NNPF is no longer used unless it is activated by another NNPFA SEI message with the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag is 0. nnpfa_cancel_flag, if its value is 0, indicates that nnpfa_target_base_flag, nnpfa_persistence_flag, and nnpfa_num_output_entries follow.

[0367] nnpfa_target_base_flag, if its value is 1, specifies that the target NNPF is the base NNPF with nnpfc_id equal to nnpfa_target_id. nnpfa_target_base_flag, if its value is 0, specifies that the target NNPF is the NNPF specified by the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id that precedes the first VCL NAL unit of the current picture in decoding order and is not a repeat of the NNPFC SEI message containing the base NNPF.

[0368] nnpfa_persistence_flag specifies the persistence of the target NNPF for the current layer.

[0369] nnpfa_persistence_flag, if its value is 0, specifies that the target NNPF can be used for post-processing filtering only on the current picture.

[0370] nnpfa_persistence_flag, if its value is 1, specifies that the target NNPF can be used for post-processing filtering for the current picture and all subsequent pictures of the current layer in output order until one or more of the following conditions are true:

[0371] - A new CLVS for the current layer starts.

[0372] - Bitstream ends.

[0373] - The picture of the current layer associated with the NNPFA SEI message with the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag set to 1 is output after the current picture in output order.

[0374] Note that the target NNPF is not applied to subsequent pictures of the current layer associated with an NNPFA SEI message with the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag set to 1.

[0375] nnpfcTargetPictures is the set of pictures associated with the last NNPFC SEI message with the same nnpfc_id as the nnpfa_target_id that precedes the current NNPFA SEI message in decoding order. nnpfaTargetPictures is the set of pictures whose target NNPF is activated by the NNPFA SEI message. It is a bitstream conformance requirement that all pictures included in nnpfaTargetPictures must also be included in nnpfcTargetPictures.

[0376] nnpfa_num_output_entries specifies the number of nnpfa_output_flag[i] syntax elements present in the NNPFA SEI message. The nnpfa_num_output_entries value must be in the range 0 to NumInpPicsInOutputTensor, inclusive.

[0377] nnpfa_output_flag[i], if its value is 1, specifies that the NNPF-generated picture corresponding to the input picture with index InpIdx[i] is output by the NNPF process activated by this NNPFA SEI message, where the NNPF process is specified in the semantics of the NNPFC SEI message. nnpfa_output_flag[i], if its value is 0, specifies that the NNPF-generated picture corresponding to the input picture with index InpIdx[i] is not output by the NNPF process activated by this NNPFA SEI message. If nnpfa_num_output_entries is less than NumInpPicsInOutputTensor, then nnpfa_output_flag[i] is inferred to be equal to 1 for each value in the range from nnpfa_num_output_entries to NumInpPicsInOutputTensor - 1.

[0378] NNPFGC SEI message (Neural-network post-filter group characteristics SEI message)

[0379] Table 27 shows an example of the NNPFGC SEI message syntax.

[0380] [Table 27]

[0381]

[0382] The NNPFGC SEI message specifies an NNPF group. The SEI message indicates whether the NNPF group defines an NNPF cascade, or whether it defines NNPFs or NNPF groups in an NNPF cascade that replace each other. The use of NNPF groups in an NNPF cascade for a particular picture is indicated by the NNPFGA SEI message.

[0383] nnpfgc_id contains an identifier that can be used to identify an NNPF group. nnpfgc_id values ​​range from 0 to 2. 32 - Must be in the range of 256 to 511 and 2 31 In 2 32 - nnpfgc_id values ​​up to 2 are reserved for future use by ITU-T | ISO / IEC. The range is 256 to 511 or 2 31 In 2 32 - Decoders conforming to this document that encounter an NNPFGC SEI message with an nnpfgc_id in the range 2 to 1 must ignore that SEI message. The nnpfgc_id value must not be equal to any nnpfc_id value of any NNPFGC SEI message present in the same CLVS. If the nnpfgc_id value of NNPFGC SEI message nnpfgcSeiA is equal to the nnpfgc_id value of another NNPFGC SEI message nnpfgcSeiB present in the same CLVS, then nnpfgcSeiA and nnpfgcSeiB must match.

[0384] nnpfgc_grouping_type, if its value is 0, indicates that this SEI message specifies a group of cascaded neural-network post-filters.

[0385] nnpfgc_grouping_type, if its value is 1, indicates that the NNPFs or NNPF groups identified by nnpfgc_member_id[i] are mutually selectable, meaning that the post-processor must choose only one of them to apply.

[0386] nnpfgc_grouping_type, if its value is 2, specifies the NNPF groups that this SEI message is intended to be used with, and indicates that these NNPFs are activated in rotation so that at most one of them is active for every picture.

[0387] nnpfgc_grouping_type, if its value is 3, indicates that the NNPFs or NNPF groups identified by nnpfgc_member_id[i] are intended to be used in parallel.

[0388] nnpfgc_grouping_type, if its value is 4, indicates that the NNPFs or NNPF groups identified by nnpfgc_member_id[i] are optional, i.e., may or may not be applied by the post processor.

[0389] The nnpfgc_grouping_type value must be in the range 0 to 255, inclusive. nnpfgc_grouping_type values ​​in the range 5 to 255 are reserved for future specification by ITU-T | ISO / IEC and must not be present in a bitstream conforming to this document. Decoders conforming to this document must ignore NNPFGC SEI messages with nnpfgc_grouping_type in the range 5 to 255.

[0390] nnpfgc_purpose has the semantics of nnpfgc_purpose, except that the semantics are specified for NNPF groups defined by this SEI message rather than for NNPFs defined by the NNPFC SEI message.

[0391] nnpfgc_num_members_minus2 plus 2 represents the number of NNPFs or NNPF groups in the NNPF group defined by this SEI message.

[0392] nnpfgc_member_id[i] represents the ith member of the NNPF group defined by this SEI message, as follows.

[0393] - If there is an NNPF with an nnpfc_id equal to nnpfgc_member_id[i] defined in CLVS, the ith member of the NNPF group defined by this SEI message is the NNPF with an nnpfc_id equal to nnpfgc_member_id[i].

[0394] - Otherwise (if there is no NNPF with nnpfc_id equal to nnpfgc_member_id[i] defined in CLVS), the ith member of the NNPF group defined by this SEI message is the NNPF group with nnpfgc_id equal to nnpfgc_member_id[i].

[0395] If the value of nnpfgc_member_id[i] refers to the nnpfgc_id value of an NNPFGC SEI message nnpfgcSei, it is a bitstream conformance requirement that the NNPFGC SEI message nnpfgcSei have an nnpfgc_grouping_type equal to 0. If nnpfgc_grouping_type is 0 or 2, it is a bitstream conformance requirement that there exists an NNPF with an nnpfc_id equal to nnpfgc_member_id[i] defined in CLVS. If nnpfgc_grouping_type is 1, 3, or 4, it is a bitstream conformance requirement that there exists an NNPF with an nnpfc_id value equal to nnpfgc_member_id[i] or an NNPF group with an nnpfc_id value equal to nnpfgc_member_id[i] defined in CLVS.

[0396] If nnpfgc_grouping_type is 0, NNPFs with nnpfc_id equal to nnpfgc_member_id[i] are executed in the cascade in increasing order of i as activated by NNPFGA SEI messages with nnpfga_target_id equal to nnpfgc_id.

[0397] nnpfgc_complexity_info_present_flag, nnpfgc_parameter_type_idc, nnpfgc_log2_parameter_bit_length_minus3, nnpfgc_num_parameters_idc, nnpfgc_num_kmac_operations_idc, and nnpfgc_total_kilobyte_size have the semantics of nnpfc_complexity_info_present_flag, nnpfc_parameter_type_idc, nnpfc_log2_parameter_bit_length_minus3, nnpfc_num_parameters_idc, nnpfc_num_kmac_operations_idc, and nnpfc_total_kilobyte_size, respectively, but the semantics are specified for the NNPF group defined by this SEI message rather than for the NNPF defined by the NNPFC SEI message. Except that if nnpfgc_grouping_type is 1, nnpfgc_complexity_info_present_flag must be 0.

[0398] NNPFGA SEI message (Neural-network post-filter group activation SEI message)

[0399] Table 28 shows an example of the NNPFGA SEI message syntax.

[0400] [Table 28]

[0401]

[0402] The NNPFGA SEI message enables or disables the possible use of target NNPFGs of NNPF groups identified by nnpfga_target_id for post-processing filtering on a set of pictures. nnpfgc_grouping_type for the identified NNPF group must be 0 (cascade) or 1 (replacement). If nnpfgc_grouping_type is 1, each member of the group must have the same number of input pictures and NNPF output pictures. For a particular picture with NNPFG enabled, the target NNPFG is the NNPFG specified by the last NNPFGC SEI message with nnpfgc_id equal to nnpfga_target_id that precedes the first VCL NAL unit of the current picture in decoding order, and the NNPFs of the target NNPFG are defined by the NNPFC SEI message that exists in the current picture unit and has nnpfc_id equal to all nnpfgc_member_id[i] values ​​of the target NNPFG and precedes the current picture in decoding order.

[0403] The use of this SEI message requires the definition of the following variables:

[0404] - Input picture width and height in units of luma samples, represented here as InitCroppedWidth[idx] and InitCroppedHeight[idx] of candidate input pictures with indices idx ranging from 0 to numCandInputPics - 1, which can be used as input to NNPFG.

[0405] - an array of luma samples, InitCroppedYPic[idx], and chroma sample arrays, InitCroppedCbPic[idx] and InitCroppedCrPic[idx], of candidate input pictures with indices idx ranging from 0 to numCandInputPics - 1, which can be used as input to NNPFG (if any);

[0406] - Bit depth for the luma sample array of candidate input pictures BitDepth Y

[0407] - BitDepth, the bit depth for the chroma sample arrays of the candidate input pictures (if any). C

[0408] - Chroma format indicator, represented here as ChromaFormatIdc.

[0409] - An array of filtering strength control values ​​StrengthControlVal[idx] that must contain real numbers in the range 0 to 1 for candidate input pictures with indices idx in the range 0 to numCandInputPics - 1 when nnpfc_auxiliary_inp_idc is 1.

[0410] The candidate input picture with index 0 corresponds to the picture for which NNPFG is activated by this NNPFGA SEI message. The candidate input picture with index i in the range of 1 to numCandInputPics - 1 precedes the candidate input picture with index i-1 in the output order. Let candInputPicList[0] be the list of candidate input pictures in the reverse output order.

[0411] nnpfga_target_id represents the target NNPFG associated with the current picture and specified by an NNPFGC SEI message with an nnpfgc_id equal to nnpfga_target_id.

[0412] The nnpfga_target_id value is from 0 to 2 32 - Must be in the range of 2.

[0413] An NNPFGA SEI message with a feature value of nnpfga_target_id shall not be present in the current PU unless there is an NNPFGC SEI with an nnpfgc_id equal to a specific value of nnpfga_target_id present in the current PU or in a PU preceding the current PU in decoding order within the current CLVS and with an nnpfgc_grouping_type of 0.

[0414] If a PU contains both an NNPFGC SEI message with a specific value of nnpfgc_id and an NNPFGA SEI message with an nnpfga_target_id that is equal to a specific value of nnpfgc_id, the NNPFGC SEI message must precede the NNPFGA SEI message in decoding order.

[0415] nnpfga_cancel_flag, if its value is 1, indicates that the persistence of the target NNPFG established by all previous NNPFGA SEI messages with the same nnpfga_target_id as the current SEI message is canceled. That is, the target NNPFG is no longer in use unless it is activated by another NNPFGA SEI message with the same nnpfga_target_id as the current SEI message and nnpfga_cancel_flag is 0. nnpfga_cancel_flag, if its value is 0, indicates that the target NNPFG is activated for use.

[0416] nnpfga_persistence_flag specifies the persistence of the target NNPFG for the current layer.

[0417] nnpfga_persistence_flag, if its value is 0, specifies that the target NNPFG can be used for post-processing filtering only for the current picture.

[0418] nnpfga_persistence_flag, if its value is 1, specifies that the target NNPFG can be used for post-processing filtering for the current picture and all subsequent pictures in the current layer in output order until one or more of the following conditions are true:

[0419] - A new CLVS for the current layer starts.

[0420] - Bitstream ends.

[0421] - The picture of the current layer associated with the NNPFGA SEI message with the same nnpfga_target_id as the current SEI message that is output after the current picture in output order.

[0422] Note that the target NNPFG does not apply to subsequent pictures of the current layer associated with an NNPFGA SEI message with the same nnpfga_target_id as the current SEI message.

[0423] nnpfgcTargetPictures is the set of pictures associated with the last NNPFGC SEI message with the same nnpfgc_id as the nnpfga_target_id that precedes the current NNPFGA SEI message in decoding order. nnpfgaTargetPictures is the set of pictures whose target NNPFG is activated by the current NNPFGA SEI message. It is a bitstream conformance requirement that all pictures included in nnpfgaTargetPictures must also be included in nnpfgcTargetPictures.

[0424] nnpfga_num_filters_minus2 plus 2 represents the number of NNPFs in the NNPFG that activate this SEI message. The value of nnpfga_num_filters_minus2 shall be equal to the value of nnpfgc_num_members_minus2 of the NNPFGC SEI message with the same nnpfgc_id as the nnpfga_target_id.

[0425] nnpfga_target_base_flag[i], if its value is 1, specifies that the ith NNPF of the target NNPFG is the base NNPF having nnpfc_id equal to nnpfgc_member_id[i] of an NNPFGC SEI message having nnpfgc_id equal to nnpfga_target_id. nnpfga_target_base_flag[i], if its value is 0, indicates that the ith NNPF of the target NNPFG is the NNPF specified by the last NNPFC SEI message having nnpfc_id equal to nnpfgc_member_id[i] of an NNPFGC SEI message having nnpfgc_id equal to nnpfga_target_id that precedes the first VCL NAL unit of the current picture in decoding order and is not a repeat of the NNPFC SEI message containing the base NNPF.

[0426] nnpfga_input_all_pics_flag[i] specifies that the input pictures for the ith NNPF are selected from the list of candidate input pictures candInputPicList[i] without skipping if its value is 1. nnpfga_input_all_pics_flag[i] specifies that the input pictures for the ith NNPF are selected from the list of candidate input pictures candInputPicList[i] in a way that some of the candidate input pictures are skipped if its value is 0.

[0427] nnpfga_num_input_pics_minus1[i] specifies the number of input pictures for the ith NNPF of the target NNPFG. If present, nnpfga_num_input_pics_minus1[i] must be equal to nnpfc_num_input_pics_minus1 for the NNPF with nnpfc_id equal to nnpfgc_member_id[i] of the NNPFGC SEI message with nnpfgc_id equal to nnpfga_target_id. If it does not exist, nnpfga_num_input_pics_minus1[i] is inferred to be equal to nnpfc_num_input_pics_minus1 for the NNPF with nnpfc_id equal to nnpfgc_member_id[i] of the NNPFGC SEI message with nnpfgc_id equal to nnpfga_target_id.

[0428] nnpfga_input_pic_skip_count[i][j] specifies the j-th picture count to be skipped in the list of candidate input pictures candInputPicList[i] when selecting input pictures for the NNPF activated by the ith loop entry. If nnpfga_input_pic_skip_count[i][j] does not exist, it is inferred to be equal to 0 for all values ​​of j in the range from 0 to nnpfga_num_input_pics_minus1[i]. numCandInputPics, which represents the number of candidate input pictures for the NNPFG, is derived as shown in Table 29 below.

[0429] [Table 29]

[0430]

[0431] For each picture output by the NNPF process of the nth loop entry that does not have a corresponding picture already present in candInputPicList[m], and for each picture in candInputPicList[0] that does not have a corresponding picture already present in candInputPicList[m], for m in the range 1 to nnpfga_num_filters_minus2 + 1, let candInputPicList[m] be initially empty and be a list of pictures in reverse output order formed in decreasing order of n in the range 0 to m - 1.

[0432] For all values ​​of m in the range from 1 to nnpfga_num_filters_minus2 + 1, if the candidate input picture candInputPicList[m][idx] is the NNPF output picture of the nth NNPF process with a value of n less than m, then the width and height of the candidate input picture are equal to nnpfcOutputPicWidth and nnpfcOutputPicHeight of the NNPF output picture, respectively.

[0433] The list of input pictures for the NNPF of the mth loop entry, inputPicList[m], is derived as shown in Table 30 below.

[0434] [Table 30]

[0435]

[0436] It is a bitstream conformance requirement that candIdx must not exceed the number of pictures in candInputPicList[m].

[0437] It is a bitstream conformance requirement that for all values ​​in the range 1 to nnpfga_num_filters_minus2 + 1, the pictures present in inputPicList[m] must have the same width, height, bit depth, and chroma format.

[0438] For the purpose of interpreting an NNPFC SEI message with an nnpfc_id equal to nnpfgc_member_id[i] of an NNPFGC SEI message with an nnpfgc_id equal to nnpfga_target_id, the following variables are specified for the ith loop entry.

[0439] - The variables BitDepthY, BitDepthC, and ChromaFormatIdc are used as provided for interpreting this message.

[0440] - CroppedWidth and CroppedHeight are set equal to the width and height of the pictures in inputPicList[i] in units of luma samples, respectively.

[0441] - For each input picture k in the range 0 to nnpfga_num_input_pics_minus1[i], the following applies:

[0442] - (If present) CroppedYPic[k], CroppedCbPic[k], and CroppedCrPic[k] are set equal to the corresponding sample arrays in inputPicList[i][k].

[0443] - For an NNPF with nnpfc_id equal to nnpfgc_member_id[i] of an NNPFGC SEI message with nnpfgc_id equal to nnpfga_target_id, if nnpfc_auxiliary_inp_idc is 1, the following applies:

[0444] - It is a bitstream conformance requirement that inputPicList[i][k] is equal to candInputPicList[0][idx] for all idx values ​​in the range 0 to numCandInputPics - 1.

[0445] - StrengthControlVal[k] is set to the same as InitStrengthControlVal[idx].

[0446] nnpfga_num_output_entries[i] specifies the number of nnpfga_output_flag[i][j] syntax elements present in the NNPFGA SEI message. For an NNPF with an nnpfc_id equal to nnpfgc_member_id[i] of an NNPFGC SEI message with an nnpfgc_id equal to nnpfga_target_id, the value of nnpfga_num_output_entries[i] must be in the range 0 to NumInpPicsInOutputTensor, inclusive.

[0447] nnpfga_output_flag[i][j], if its value is 1, indicates that the NNPF-generated picture corresponding to the input picture having the derived index InpIdx[j] for the ith NNPFG is output by the NNPF process activated by this loop entry, where the NNPF process is specified in the semantics of the NNPFC SEI message. nnpfga_output_flag[i][j], if its value is 0, indicates that the NNPF-generated picture corresponding to the input picture having the derived index InpIdx[j] for the ith NNPFG is not output by the NNPF process activated by this loop entry. If nnpfga_num_output_entries[i] is less than NumInpPicsInOutputTensor derived for the i-th NNPF of the target NNPFG, then nnpfga_output_flag[i][j] is inferred to be equal to 1 for each value of i in the range from nnpfga_num_output_entries[i] to NumInpPicsInOutputTensor - 1.

[0448] NnpfgaOutputPicList, which is a list of pictures output by the NNPF process of NNPFG in output order, is initially empty and is formed in decreasing order of n ranging from 0 to nnpfga_num_filters_minus2 + 1, including each picture output by the NNPF process of the nth loop entry that does not already have a corresponding picture in NnpfgaOutputPicListd.

[0449] Source Picture Timing Information (SPTI)

[0450] Table 31 shows an example of SPTI syntax.

[0451] [Table 31]

[0452]

[0453] The SPTI SEI message indicates the temporal distance between source pictures associated with the corresponding decoded output pictures before encoding. For example, for camera-captured content, the temporal distance between source pictures is the difference between the time the image sensor is exposed to generate the source picture associated with the currently decoded picture and the time the image sensor is exposed to generate the source picture associated with the previously decoded picture in output order. The information provided by the SPTI SEI message is relevant only to the picture(s) starting from the picture of the current layer of the access unit that contains the SPTI SEI message and all subsequent pictures of the current layer in output order based on its continuity.

[0454] spti_cancel_flag, if its value is 1, indicates that the SPTI SEI message cancels the persistence of all previous SPTI SEI messages in the output order applied to the current layer. spti_cancel_flag, if its value is 0, indicates that source picture timing information follows.

[0455] spti_persistence_flag specifies the persistence of SPTI SEI messages for the current layer.

[0456] spti_persistence_flag, if its value is 0, specifies that the SPTI SEI message applies only to the currently decoded picture.

[0457] spti_persistence_flag, if its value is 1, specifies that the SPTI SEI message is applied to the currently decoded picture and persists for all subsequent pictures in the current layer in output order until one or more of the following conditions are true:

[0458] - A new CLVS for the current layer is started.

[0459] - Bitstream terminated.

[0460] - The picture of the current layer of the AU associated with the SPTI SEI message is output after the current picture in the output order.

[0461] spti_source_timing_equals_output_timing_flag, if its value is 1, indicates that the timing of the source picture is the same as the timing of the corresponding decoded output pictures. spti_source_timing_equals_output_timing_flag, if its value is 0, indicates that the timing of the source picture may not be the same as the timing of the corresponding decoded output pictures.

[0462] If spti_source_timing_equals_output_timing_flag is 1 and a picture timing SEI message exists for the current picture, the source picture timing can be determined from the information passed in the picture timing SEI message.

[0463] spti_source_type_present_flag, if its value is 1, indicates that the syntax element spti_source_type is present in the SEI message. spti_source_type_present_flag, if its value is 0, indicates that the syntax element spti_source_type is not present in the SEI message.

[0464] spti_source_type indicates the timing relationship between source pictures and their corresponding decoded output pictures as specified in Table 32 below.

[0465] [Table 32]

[0466]

[0467] Here, (spti_source_type & bitMask) indicates that the timing relationship has the interpretation associated with the bitmask value in Table 32 if its value is not 0. If spti_source_type is greater than 0 and (spti_source_type & bitMask) is 0, it indicates that the interpretation associated with the bitmask value is not applicable to the SPTI SEI message. If spti_source_type is 0, the timing relationship can be specified by the application.

[0468] The spti_source_type value MUST be in the range 0 to 127 inclusive for bitstreams conforming to this document. The values ​​128 to 255 are reserved for future use by ITU-T | ISO / IEC and MUST NOT be present in bitstreams conforming to this document. Decoders conforming to this document MUST ignore SPTI SEI messages with spti_source_type in the range 128 to 255.

[0469] (spti_source_type & 0x04) & (spti_source_type & 0x08) must be zero (i.e. spti_source_type must not indicate both high-speed imaging and time-lapse imaging).

[0470] spti_time_scale specifies the number of time units that pass in one second. The spti_time_scale value must not be zero. For example, a time coordinate system that measures time using a 27 MHz clock would have an spti_time_scale of 27,000,000.

[0471] spti_num_units_in_elemental_interval specifies the number of time units of the clock operating at frequency spti_time_scale Hz corresponding to the indicated elemental source picture interval of consecutive pictures in the output order of the CLVS.

[0472] The indicated elemental source picture interval, also denoted by the variable ElementalSourcePictureInterval, is equal to the quotient of spti_num_units_in_elemental_interval divided by spti_time_scale in seconds. For example, to indicate an elemental source picture interval equal to 0.04 seconds, spti_time_scale could be equal to 27,000,000 and spti_num_units_in_elemental_interval could be equal to 1,080,000.

[0473] spti_max_sublayers_minus_1 plus 1 specifies the maximum number of temporal sublayers for which the picture interval scale factor (spti_sublayer_interval_scale_factor[i]) and synthesized flag (spti_sublayer_synthesized_picture_flag[i]) information is signaled. If spti_max_sublayers_minus_1 is not present, it is inferred to be equal to TemporalId.

[0474] spti_sublayer_interval_scale_factor[i], if present, specifies the scale factor used to determine the source picture spacing of corresponding consecutive pictures in the output order of the CLVS with TemporalId less than or equal to i. A value of 0 can be used to indicate that the source picture corresponding to the currently decoded output picture is identical to the source picture corresponding to the previously decoded output picture.

[0475] The specified source picture interval associated with output pictures with TemporalId less than or equal to i, represented by the variable SourcePictureInterval[i], is derived in seconds as shown in Table 33 below.

[0476] [Table 33]

[0477]

[0478] The variable temporalReversalFlag is ' ' is the same as.

[0479] spti_sublayer_synthesized_picture_flag[i], if present, indicates that the decoded output pictures belonging to the ith temporal sublayer are synthesized and do not correspond to the unmodified original source pictures if its value is 1. spti_sublayer_synthesized_picture_flag[i] does not provide such an indication if its value is 0. If it is not present, the value of spti_sublayer_synthesized_picture_flag[i] is inferred to be equal to 0.

[0480] Note that if the TemporalId of an SPTI SEI message is greater than 0 and the SPTI SEI message persists for one or more pictures with lower TemporalId, the encoder may repeat the information of the SPTI SEI message by including it in one or more SPTI SEI messages with lower TemporalId to prevent information loss when pictures of the temporal sublayer(s) are lost or removed.

[0481] SEI processing order SEI message

[0482] Table 34 shows an example of the SEI processing order SEI message syntax.

[0483] [Table 34]

[0484]

[0485] SEI Processing Order SEI messages convey information indicating the preferred processing order, as determined by the encoder (i.e., content producer), for a group of types of SEI messages that may exist in the CVS.

[0486] The semantics of SEI processing order SEI messages use the notion of types of SEI messages. SEI messages with different payloadType values ​​are considered different types of SEI messages. Furthermore, different SEI messages with the same payloadType value but distinguished by the values ​​of syntax elements in the SEI payload are considered different types of SEI messages. This distinction by the values ​​of syntax elements in the SEI payload is made by comparing the values ​​passed using the po_sei_prefix_data_bit[i][j] syntax elements (if present) or the values ​​passed within an SEI message within a processing order nested SEI message (if present). For example, NNPFC SEI messages can be distinguished by having different nnpfc_id values.

[0487] If an SEI Processing Order SEI message with a specific po_id value exists in all access units of the CVS, then the SEI Processing Order SEI message with a specific po_id value must exist in the first access unit of the CVS in decoding order. The number of SEI messages specified in each SEI Processing Order SEI message with the same po_id value and the payloadType codes of the SEI messages continue in decoding order in the current access unit until the end of the CVS in output order.

[0488] SEI Processing Order An SEI message may carry one or more SEI prefix indications of a particular payloadType. When present, each SEI prefix indication is a bitstring that follows the SEI payload syntax of the payloadType value and contains the number of complete syntax elements starting from the first syntax element of the SEI payload. These SEI prefix indications should provide sufficient information to determine a particular processing order for SEI message types that have the same payloadType value but different preferred processing orders.

[0489] po_id contains an identification number to identify the SEI processing sequence SEI message.

[0490] Each SEI message in an SEI message group for which preferred processing order information is provided in the SEI processing order SEI message is identified by the syntax elements po_sei_payload_type[i], po_sei_wrapping_flag[i], po_sei_processing_order[i] and, if present, po_num_bits_in_prefix_indication_minus1[i] and po_prefix_data_bit[i][j].

[0491] For each picture, there may be multiple ongoing or active SEI messages belonging to one or more groups of SEI messages.

[0492] Note that SEI message groups are interchangeable. That is, at most one group can be selected for application, or they can be complementary. That is, multiple groups, each producing a single output, are selected and applied individually.

[0493] Adding 2 to po_num_sei_messages_minus2 indicates the number of types of SEI messages whose preferred order of processing appears in the SEI processing order SEI messages.

[0494] po_sei_wrapping_flag[i], if its value is 1, specifies that there must be at least one processing order overlapping SEI message that has both of the following constraints:

[0495] - pon_target_po_id[j] with all j values ​​is equal to po_id.

[0496] - There exists a kth loop entry of the SEI processing order nested SEI message such that the payload of the kth nested SEI message is equal to po_sei_payload_type[i] and pon_processing_order[k] is equal to po_sei_processing_order[i].

[0497] If po_sei_wrapping_flag[i] is 0, an SEI message with payloadType equal to po_sei_payload_type[i] (and, if po_sei_wrapping_flag[i] is 1, with prefix data matching the value of po_sei_prefix_data_bit[i][j]) must exist outside the processing order nested SEI message. However, if there is no SEI message with payloadType equal to po_sei_wrapping_flag[i] equal to 0 and po_sei_payload_type[i], or if there is no SEI message with payloadType equal to po_sei_payload_type[i] equal to po_sei_wrapping_flag[i] equal to 0 and po_sei_prefix_flag[i] equal to 1 and with prefix data matching the value of po_sei_prefix_data_bit[i][j], then the following applies.

[0498] - If po_sei_importance_flag[i] is 1, the decoder must ignore the entire SEI processing order SEI message.

[0499] - Otherwise, the decoder must ignore all data associated with the loop variable value i.

[0500] Note that po_sei_wrapping_flag[i], when its value is 1, allows SEI messages to be passed within a processing order nested SEI message to prevent decoders that do not process SEI processing order SEI messages from misinterpreting these SEI messages. Therefore, po_sei_wrapping_flag[i], when its value is 1, is intended to be used in cases where a po_sei_wrapping_flag[i] of 0 could cause unintended results by such decoders.

[0501] po_sei_importance_flag[i] indicates the importance determined by the encoder for the SEI message type with index i.

[0502] If a decoding system cannot interpret or does not support the functionality indicated by any given SEI message with po_sei_importance_flag[i] equal to 1, it must ignore the entire SEI processing sequence SEI message.

[0503] po_sei_payload_type[i] specifies the payloadType value of the i-th type of the SEI message.

[0504] po_sei_prefix_flag[i], if its value is 1, specifies that the syntax elements po_num_bits_in_prefix_indication_minus1[i] and some syntax elements po_sei_prefix_data_bit[i][j] are present. po_sei_prefix_flag[i], if its value is 0, specifies that these syntax elements are not present.

[0505] SeiProcessingOrderSeiList is set to contain payloadType values ​​3, 4, 5, 19, 137, 142, 144, 147, 148, 149, 165, 177, 210, and 211. For each i in the range 0 to po_num_sei_messages_minus2, the value of po_sei_payload_type[i] must be equal to a value in SeiProcessingOrderSeiList.

[0506] po_sei_processing_order[i] represents the preferred order of processing of the ith type of SEI message for which preferred processing order information is provided in the SEI processing order SEI message. For two different integer values ​​m and n, po_sei_processing_order[m] represents that the SEI message type associated with index m should be processed before the SEI message type associated with index n if its value is less than po_sei_processing_order[n]. And po_sei_processing_order[m] represents that there is no preferred order of processing between the SEI message types associated with indices m and n if its value is po_sei_processing_order[n] (e.g., they may both represent different attributes that can be applied at that stage or alternative processes that can be applied, or one may represent an attribute and the other a process).

[0507] For i greater than 0, po_sei_processing_order[i] must be greater than or equal to po_sei_processing_order[i - 1].

[0508] po_num_bits_in_prefix_indication_minus1[i] and po_sei_prefix_data_bit[i][j], if present, have the same semantics as the num_bits_in_prefix_indication_minus1[i] and sei_prefix_data_bit[i][j] syntax elements of an SEI prefix indication SEI message, with prefix_sei_payload_type replaced by po_sei_payload_type[i].

[0509] If one or more SEI Processing Order SEI messages with a particular po_id value exist in the CVS, then for each value of po_num_sei_messages_minus2 and for each value of i, the values ​​of po_sei_wrapping_flag[i], po_sei_prefix_flag[i], po_sei_importance_flag[i], po_sei_payload_type[i], po_sei_processing_order[i] must be identical to those of other SEI Processing Order SEI messages in the CVS with the same po_id value.

[0510] po_byte_alignment_bit_equal_to_one must be 1.

[0511] Processing order nesting SEI message

[0512] Table 35 shows an example of the processing order nested SEI message syntax.

[0513] [Table 35]

[0514]

[0515] A Processing Order Overlapping SEI message contains one or more SEI messages that must apply only to a portion of the processing chain identified by the associated SEI Processing Order SEI message and must not be applied in a manner that contradicts the processing chain identified by the associated SEI Processing Order SEI message.

[0516] SEI messages included in a processing-order-nested SEI message are also referred to as processing-order-nested messages.

[0517] All SEI messages contained in a nested SEI message with the same processing order must have the same persistence.

[0518] pon_num_po_ids_minus1 plus 1 specifies the number of SEI processing order SEI messages associated with this processing order nested SEI message.

[0519] pon_target_po_id[i] represents the po_id of the SEI message of the i-th associated SEI processing order.

[0520] pon_num_seis_minus1 plus 1 specifies the number of processing-order-nested SEI messages contained in this SEI message.

[0521] pon_processing_order[i] specifies the position of the processing-order-nested SEI message contained in this SEI message within the processing order defined by the associated SEI processing-order SEI message. If i is greater than 0, pon_processing_order[i] must be greater than or equal to pon_processing_order[i - 1].

[0522] For each associated SEI processing order SEI message, there must be at least one i value in the range 0 to pon_num_seis_minus1 of a processing order nested SEI message with some entry k for which all of the following are true:

[0523] - po_sei_processing_order[k] is the same as pon_processing_order[i].

[0524] - po_sei_payload_type[k] is equal to the payloadType value of the ith processing-order-nested SEI message.

[0525] - If po_sei_prefix_flag[k] is 1, then for i in the range 0 to po_num_bits_in_prefix_indication_minus1[k], inclusive, po_sei_prefix_data_bit[k][j] contains content equal to po_num_bits_in_prefix_indication_minus1[k] plus 1 initial bits of the SEI message payload of the ith processing-order-nested SEI message.

[0526] The ith processing-order-nested SEI message shall be applied as the kth loop entry of the associated SEI processing-order SEI message.

[0527] Problems with prior art

[0528] It is argued that the grouping mechanism should support at least the following grouping types: cascading grouping, alternative grouping, and parallel grouping. While the mechanism works for the first two grouping types, it lacks support for enabling SEI processing order SEI messages to account for parallel grouping.

[0529] Parallel grouping is a grouping in which SEI messages within a group are not called / executed in a cascading / sequential order, but rather are called in parallel. When SEI messages are called in a cascading order, the output from the first SEI is used as input for the second SEI message call. On the other hand, when two SEIs are called in parallel, they both use the same inputs and can have independent outputs.

[0530] Example

[0531] The present invention provides solutions to the problems described above. Each embodiment may be performed individually or in combination with two or more embodiments.

[0532] The embodiments proposed by the present invention are summarized as follows.

[0533] 1. Add a flag to the SEI Processing Order SEI message to indicate whether SEI messages with the same processing order values ​​are preferred to be called in parallel or whether there is no preference for them.

[0534] Example 1

[0535] Example 1 provides an explanation of Summary 1 described above. The following Table 36 may be proposed.

[0536] [Table 36]

[0537]

[0538] Adding 2 to po_num_sei_messages_minus2 indicates the number of types of SEI messages whose preferred order of processing is indicated in the SEI processing order SEI message.

[0539] po_parallel_processing_enabled_flag, if its value is 1, specifies that SEI messages contained in this SEI processing order SEI message with the same processing order are preferred to be called in parallel. po_parallel_processing_enabled_flag, if its value is 0, specifies that there is no preference for the processing order of SEI messages contained in this SEI processing order SEI message with the same processing order.

[0540] FIG. 6 is a diagram illustrating a method for decoding image information according to one embodiment of the present disclosure.

[0541] The terms or names described in FIG. 6 (e.g., names of syntax elements or names of variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms described in FIG. 21, etc. For example, the image information described in FIG. 6 may include various information according to the embodiments described in the present disclosure, and may include information described in at least one of the tables described above.

[0542] The decoding method (S600) may include the following operations. The following operations are not essential components of the decoding method according to one embodiment, and at least some of the following operations may be omitted or other operations may be added. In addition, the following operations may be executed by a decoding device including a memory and a processor electrically connected to the memory, for example, by a processor.

[0543] The decoding device can obtain SEI (supplemental enhancement information) processing order information (S610).

[0544] For example, a processor of a decoding device may obtain image information including SEI processing order information. Image information obtained by the processor may include SEI processing order information.

[0545] SEI processing order information may include information about the processing order for a group of types of SEI messages that may exist in a coded video sequence (CVS) or a coded layer video sequence (CLVS).

[0546] SEI processing order information can take various forms and be expressed by various names. For example, SEI processing order information can be a syntax element or a syntax structure including one or more syntax elements. In addition, SEI processing order information can be a raw byte sequence payload (RBSP) including one or more syntax elements or one or more syntax structures. For example, SEI processing order information can be expressed as sei_processing_order( ), but is not limited thereto.

[0547] SEI processing order information may include wrapping information, payload type information, prefix information, prefix existence information, processing order information, and / or parallel processing capability information.

[0548] The wrapping information may indicate whether the processing order nesting information includes information indicating a processing order for the SEI message type corresponding to the wrapping information. For example, a value of 1 in the wrapping information may specify that the processing order nesting information includes information indicating a processing order for the SEI message type corresponding to the wrapping information. In addition, a value of 0 in the wrapping information may specify that the processing order nesting information includes information indicating a processing order for the SEI message type corresponding to the wrapping information. However, this is not limited thereto, and alternatively, what the value of 1 in the wrapping information specifies may be interchangeable with what the value of 0 in the wrapping information specifies.

[0549] Wrapping information can take various forms and be expressed by various names. For example, wrapping information can be a syntax element or a syntax structure containing one or more syntax elements. For example, wrapping information as a syntax element can be a one-bit wrapping flag or a two-bit wrapping indicator. Wrapping information as a syntax element can be expressed as, but is not limited to, po_sei_wrapping_flag[i].

[0550] Payload type information can indicate the type of an SEI message. For example, payload type information can specify the payloadType value of an SEI message.

[0551] Payload type information can take various forms and be expressed by various names. For example, payload type information can be a syntax element or a syntax structure containing one or more syntax elements. For example, payload type information can be expressed as po_sei_payload_type[i], but is not limited thereto.

[0552] The prefix information may include one or more SEI prefix markers for the type of the corresponding SEI message. Each SEI prefix marker is a bit string that follows the SEI payload syntax of the payloadType value and contains a number of complete syntax elements starting from the first syntax element of the SEI payload.

[0553] Prefix information may include bit information of a prefix indication and prefix data bits. The bit information of the prefix indication and the prefix data bits may each have various forms and may be expressed by various names. For example, the bit information of the prefix indication and the prefix data bits may each be a syntax element or a syntax structure including one or more syntax elements. For example, the bit information of the prefix indication may be expressed as po_num_bits_in_prefix_indication_minus1[i], and the prefix data bits may be expressed as po_sei_prefix_data_bit[i], but the present invention is not limited thereto.

[0554] The prefix presence information can indicate whether prefix information corresponding to the type of the corresponding SEI message exists. In other words, the prefix presence information can indicate the number of bits of the prefix mark corresponding to the type of the corresponding SEI message and whether prefix data bits exist. For example, a value of 1 for the prefix presence information can indicate that prefix information corresponding to the type of the corresponding SEI message exists. In addition, a value of 0 for the prefix presence information can indicate that prefix information corresponding to the type of the corresponding SEI message does not exist. However, the present invention is not limited thereto, and alternatively, what is indicated by a value of 1 for the prefix presence information can be interchanged with what is indicated by a value of 0 for the prefix presence information.

[0555] Prefix presence information can take various forms and be expressed by various names. For example, prefix presence information can be a syntax element or a syntax structure including one or more syntax elements. For example, prefix presence information as a syntax element can be a one-bit prefix presence flag or a two or more-bit prefix presence indicator. Prefix presence information as a syntax element can be expressed as, but is not limited to, po_sei_prefix_flag[i].

[0556] The processing order information may indicate the processing order corresponding to the type of the corresponding SEI message. The smaller the value of the processing order information for a specific type, the earlier the SEI message of that specific type may be processed. In other words, if the value of the processing order information for the first type is smaller than the value of the processing order information for the second type, the SEI message of the first type may be processed earlier than the SEI message of the second type.

[0557] Processing order information can take various forms and be expressed by various names. For example, processing order information can be a syntax element or a syntax structure containing one or more syntax elements. For example, processing order information can be expressed as po_sei_processing_order[i], but is not limited thereto.

[0558] The parallel processing capability information may indicate whether at least two SEI messages having the same processing order are called in parallel. For example, a value of 1 for the parallel processing capability information may indicate that at least two SEI messages having the same processing order are called in parallel. Additionally, a value of 0 for the parallel processing capability information may indicate that the SEI messages are not called in parallel. However, this is not limited thereto, and alternatively, the indication of a value of 1 for the parallel processing capability flag may be interchanged with the indication of a value of 0 for the parallel processing capability flag.

[0559] Here, the SEI processing order can be determined according to the group of types of SEI messages. SEI messages with different payloadType values ​​are considered as different types of SEI messages. Additionally, different SEI messages that have the same payloadType value but are distinguished by the values ​​of syntax elements of the SEI payload are considered as different types of SEI messages. This distinction by the values ​​of syntax elements of the SEI payload can be performed based on prefix information and / or processing order overlap information.

[0560] Therefore, parallel processing capability information can indicate whether SEI messages of the same type are called in parallel. Here, SEI messages with the same payload type and the same prefix information may be SEI messages of the same type. Furthermore, SEI messages with the same payload type and no prefix information may also be SEI messages of the same type.

[0561] Parallel processing enabled information can take various forms and be expressed by various names. For example, parallel processing enabled information can be a syntax element or a syntax structure including one or more syntax elements. For example, the syntax element parallel processing enabled information can include a one-bit parallel processing enabled flag or a two-bit parallel processing enabled indicator. The syntax element parallel processing enabled information can be expressed as, but is not limited to, po_parallel_processing_enabled_flag.

[0562] Image information acquired by the processor may further include processing order overlap information.

[0563] Processing order overlap information includes information about the processing order for types of SEI messages and may be associated with specific SEI processing order information. Furthermore, processing order overlap information may only apply to a portion of the processing chain identified by the associated SEI processing order information.

[0564] The processing order nesting information can take various forms and be expressed by various names. For example, the processing order nesting information can be a syntax element or a syntax structure including one or more syntax elements. In addition, the processing order nesting information can be a raw byte sequence payload (RBSP) including one or more syntax elements or one or more syntax structures. For example, the processing order nesting information can be expressed as processing_order_nesting( ), but is not limited thereto.

[0565] The processing order overlap information may include overlap order information.

[0566] Nesting order information can specify the position of an SEI message within the processing order defined by the associated processing order information.

[0567] Nesting order information can take various forms and be expressed by various names. For example, nesting order information can be a syntax element or a syntax structure containing one or more syntax elements. For example, nesting order information can be expressed as pon_processing_order[i], but is not limited thereto.

[0568] As previously described, parallel processing capability information can indicate whether SEI messages of the same type are called in parallel. Here, SEI messages that have the same payload type, the same prefix information, and are not included in the processing order overlap information may be SEI messages of the same type. In addition, SEI messages that have the same payload type, do not have prefix information, and are not included in the processing order overlap information may also be SEI messages of the same type.

[0569] The decoding device can determine the processing order for the SEI message (S620).

[0570] For example, a processor of a decoding device may determine a processing order for an SEI message based on SEI processing order information. The processor may determine a processing order for a group of SEI message types based on processing order information included in the SEI processing order information. Furthermore, the processor may process SEI messages according to the determined processing order.

[0571] As described above, SEI messages with the same processing order are allowed to be processed in parallel, and parallel processing-enabled information may be provided to allow parallel processing of SEI messages. By processing SEI messages with the same processing order in parallel, SEI messages can be processed efficiently, and further, there is the technical effect of reducing the delay in processing SEI messages.

[0572] FIG. 7 is a diagram illustrating a method for encoding image information according to one embodiment of the present disclosure.

[0573] The terms or names described in FIG. 7 (e.g., names of syntax elements or names of variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms described in FIG. 7. For example, the image information described in FIG. 7 may include various information according to the embodiments described in the present disclosure, and may include information described in at least one of the tables described above.

[0574] The encoding method (S700) may include the following operations. The following operations are not essential components of the encoding method according to one embodiment, and at least some of the following operations may be omitted or other operations may be added. In addition, the following operations may be executed by a decoding device including a memory and a processor electrically connected to the memory, and may be executed by, for example, a processor.

[0575] The encoding device can determine the processing order for the SEI (supplemental enhancement information) message (S710).

[0576] For example, a processor of an encoding device can determine the processing order for a group of types of SEI messages.

[0577] The encoding device can generate SEI processing order information (S720).

[0578] For example, a processor of an encoding device can generate SEI processing order information based on a processing order for a group of types of SEI messages.

[0579] SEI processing order information may include information about the processing order for a group of types of SEI messages that may exist in a coded video sequence (CVS) or a coded layer video sequence (CLVS).

[0580] SEI processing order information can take various forms and be expressed by various names. For example, SEI processing order information can be a syntax element or a syntax structure including one or more syntax elements. In addition, SEI processing order information can be a raw byte sequence payload (RBSP) including one or more syntax elements or one or more syntax structures. For example, SEI processing order information can be expressed as sei_processing_order( ), but is not limited thereto.

[0581] SEI processing order information may include wrapping information, payload type information, prefix information, prefix existence information, processing order information, and / or parallel processing capability information.

[0582] The wrapping information may indicate whether the processing order nesting information includes information indicating a processing order for the SEI message type corresponding to the wrapping information. For example, a value of 1 in the wrapping information may specify that the processing order nesting information includes information indicating a processing order for the SEI message type corresponding to the wrapping information. In addition, a value of 0 in the wrapping information may specify that the processing order nesting information includes information indicating a processing order for the SEI message type corresponding to the wrapping information. However, this is not limited thereto, and alternatively, what the value of 1 in the wrapping information specifies may be interchangeable with what the value of 0 in the wrapping information specifies.

[0583] Wrapping information can take various forms and be expressed by various names. For example, wrapping information can be a syntax element or a syntax structure containing one or more syntax elements. For example, wrapping information as a syntax element can be a one-bit wrapping flag or a two-bit wrapping indicator. Wrapping information as a syntax element can be expressed as, but is not limited to, po_sei_wrapping_flag[i].

[0584] Payload type information can indicate the type of an SEI message. For example, payload type information can specify the payloadType value of an SEI message.

[0585] Payload type information can take various forms and be expressed by various names. For example, payload type information can be a syntax element or a syntax structure containing one or more syntax elements. For example, payload type information can be expressed as po_sei_payload_type[i], but is not limited thereto.

[0586] The prefix information may include one or more SEI prefix markers for the type of the corresponding SEI message. Each SEI prefix marker is a bit string that follows the SEI payload syntax of the payloadType value and contains a number of complete syntax elements starting from the first syntax element of the SEI payload.

[0587] Prefix information may include bit information of a prefix indication and prefix data bits. The bit information of the prefix indication and the prefix data bits may each have various forms and may be expressed by various names. For example, the bit information of the prefix indication and the prefix data bits may each be a syntax element or a syntax structure including one or more syntax elements. For example, the bit information of the prefix indication may be expressed as po_num_bits_in_prefix_indication_minus1[i], and the prefix data bits may be expressed as po_sei_prefix_data_bit[i], but the present invention is not limited thereto.

[0588] The prefix presence information can indicate whether prefix information corresponding to the type of the corresponding SEI message exists. In other words, the prefix presence information can indicate the number of bits of the prefix mark corresponding to the type of the corresponding SEI message and whether prefix data bits exist. For example, a value of 1 for the prefix presence information can indicate that prefix information corresponding to the type of the corresponding SEI message exists. In addition, a value of 0 for the prefix presence information can indicate that prefix information corresponding to the type of the corresponding SEI message does not exist. However, the present invention is not limited thereto, and alternatively, what is indicated by a value of 1 for the prefix presence information can be interchanged with what is indicated by a value of 0 for the prefix presence information.

[0589] Prefix presence information can take various forms and be expressed by various names. For example, prefix presence information can be a syntax element or a syntax structure including one or more syntax elements. For example, prefix presence information as a syntax element can be a one-bit prefix presence flag or a two or more-bit prefix presence indicator. Prefix presence information as a syntax element can be expressed as, but is not limited to, po_sei_prefix_flag[i].

[0590] The processing order information may indicate the processing order corresponding to the type of the corresponding SEI message. The smaller the value of the processing order information for a specific type, the earlier the SEI message of that specific type may be processed. In other words, if the value of the processing order information for the first type is smaller than the value of the processing order information for the second type, the SEI message of the first type may be processed earlier than the SEI message of the second type.

[0591] Processing order information can take various forms and be expressed by various names. For example, processing order information can be a syntax element or a syntax structure containing one or more syntax elements. For example, processing order information can be expressed as po_sei_processing_order[i], but is not limited thereto.

[0592] The parallel processing capability information may indicate whether at least two SEI messages having the same processing order are called in parallel. For example, a value of 1 for the parallel processing capability information may indicate that at least two SEI messages having the same processing order are called in parallel. Additionally, a value of 0 for the parallel processing capability information may indicate that the SEI messages are not called in parallel. However, this is not limited thereto, and alternatively, the indication of a value of 1 for the parallel processing capability flag may be interchanged with the indication of a value of 0 for the parallel processing capability flag.

[0593] Here, the SEI processing order can be determined according to the group of types of SEI messages. SEI messages with different payloadType values ​​are considered as different types of SEI messages. Additionally, different SEI messages that have the same payloadType value but are distinguished by the values ​​of syntax elements of the SEI payload are considered as different types of SEI messages. This distinction by the values ​​of syntax elements of the SEI payload can be performed based on prefix information and / or processing order overlap information.

[0594] Therefore, parallel processing capability information can indicate whether SEI messages of the same type are called in parallel. Here, SEI messages with the same payload type and the same prefix information may be SEI messages of the same type. Furthermore, SEI messages with the same payload type and no prefix information may also be SEI messages of the same type.

[0595] Parallel processing enabled information can take various forms and be expressed by various names. For example, parallel processing enabled information can be a syntax element or a syntax structure including one or more syntax elements. For example, the syntax element parallel processing enabled information can include a one-bit parallel processing enabled flag or a two-bit parallel processing enabled indicator. The syntax element parallel processing enabled information can be expressed as, but is not limited to, po_parallel_processing_enabled_flag.

[0596] The processor of the encoding device may additionally generate processing order overlap information based on the processing order for the group of types of SEI messages.

[0597] Processing order overlap information includes information about the processing order for types of SEI messages and may be associated with specific SEI processing order information. Furthermore, processing order overlap information may only apply to a portion of the processing chain identified by the associated SEI processing order information.

[0598] The processing order nesting information can take various forms and be expressed by various names. For example, the processing order nesting information can be a syntax element or a syntax structure including one or more syntax elements. In addition, the processing order nesting information can be a raw byte sequence payload (RBSP) including one or more syntax elements or one or more syntax structures. For example, the processing order nesting information can be expressed as processing_order_nesting( ), but is not limited thereto.

[0599] The processing order overlap information may include overlap order information.

[0600] Nesting order information can specify the position of an SEI message within the processing order defined by the associated processing order information.

[0601] Nesting order information can take various forms and be expressed by various names. For example, nesting order information can be a syntax element or a syntax structure containing one or more syntax elements. For example, nesting order information can be expressed as pon_processing_order[i], but is not limited thereto.

[0602] As previously described, parallel processing capability information can indicate whether SEI messages of the same type are called in parallel. Here, SEI messages that have the same payload type, the same prefix information, and are not included in the processing order overlap information may be SEI messages of the same type. In addition, SEI messages that have the same payload type, do not have prefix information, and are not included in the processing order overlap information may also be SEI messages of the same type.

[0603] The encoding device can encode image information including SEI processing order information (S730).

[0604] For example, a processor of an encoding device can encode image information including SEI processing order information.

[0605] As described above, SEI messages with the same processing order are allowed to be processed in parallel, and parallel processing-enabled information may be provided to allow parallel processing of SEI messages. By processing SEI messages with the same processing order in parallel, SEI messages can be processed efficiently, and further, there is the technical effect of reducing the delay in processing SEI messages.

[0606] Image information including SEI processing order information encoded according to the encoding method (S700) described above can be stored in a computer-readable storage medium. Image information including SEI processing order information can be transmitted via a transmission unit and / or a transmission medium.

[0607] FIG. 8 is a diagram illustrating an example of a content streaming system to which an embodiment according to the present disclosure can be applied.

[0608] As illustrated in FIG. 8, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0609] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, generates a bitstream, and transmits it to the streaming server. Alternatively, if multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server may be omitted.

[0610] The above bitstream can be generated by a video encoding method and / or encoding device to which an embodiment of the present disclosure is applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0611] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server may control commands / responses between each device within the content streaming system.

[0612] The streaming server can receive content from a media repository and / or encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0613] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0614] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0615] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0616] Embodiments according to the present disclosure can be used to encode / decode images.

Claims

1. As a method of decoding image information, Obtaining image information including SEI processing order information including a processing order for a group of types of SEI (supplemental enhancement information) messages; Including determining the processing order for the type group of the SEI messages based on the SEI processing order information, The above SEI processing order information includes payload type information indicating the type of the SEI message, prefix existence information indicating whether prefix information of the SEI message exists, and processing order information indicating the processing order according to the type of the SEI message. A method wherein the SEI processing order information further includes parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

2. A method in the first paragraph, wherein a value of the parallel processing possible information of 1 indicates that at least two SEI messages having the same processing order are called in parallel, and a value of the parallel processing possible information of 0 indicates that SEI messages are not called in parallel.

3. In the first paragraph, a method in which at least two SEI messages having the same type and the same prefix information have the same processing order.

4. In the first paragraph, a method in which at least two SEI messages having the same type and no prefix information have the same processing order.

5. A method in accordance with claim 1, wherein the image information further includes processing order overlapping information including position information of a specific SEI message within a processing order defined by the SEI processing order information.

6. In the fifth paragraph, a method in which at least two SEI messages having the same type and the same prefix information and not included in the processing order overlapping information have the same processing order.

7. In the fifth paragraph, a method in which at least two SEI messages having the same type, no prefix information, and not included in the processing order overlapping information have the same processing order.

8. In a method of encoding image information, Determines the processing order for a group of types of SEI (supplemental enhancement information) messages; Generate SEI processing order information including a processing order for a group of types of SEI messages based on the above processing order; Encoding the image information including the SEI processing order information, The above SEI processing order information includes payload type information indicating the type of the SEI message, prefix existence information indicating whether prefix information of the SEI message exists, and processing order information indicating the processing order according to the type of the SEI message. A method wherein the SEI processing order information further includes parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

9. A method for storing a bitstream of video information, Obtaining image information including SEI processing order information including a processing order for a group of types of SEI (supplemental enhancement information) messages; Including storing data including a bitstream of the above image information in a computer-readable storage medium, The above SEI processing order information includes payload type information indicating the type of the SEI message, prefix existence information indicating whether prefix information of the SEI message exists, and processing order information indicating the processing order according to the type of the SEI message. A method wherein the SEI processing order information further includes parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

10. A method for transmitting a bitstream of video information, Obtaining image information including SEI processing order information including a processing order for a group of types of SEI (supplemental enhancement information) messages; Including transmitting data including a bitstream of the above image information, The above SEI processing order information includes payload type information indicating the type of the SEI message, prefix existence information indicating whether prefix information of the SEI message exists, and processing order information indicating the processing order according to the type of the SEI message. A method wherein the SEI processing order information further includes parallel processing capability information indicating whether at least two SEI messages having the same processing order are called in parallel.

Images