Method for decoding image information, method for encoding image information, method for bitstream, and computer-readable storage medium for storing bitstream

Abstract

A method according to the present disclosure comprises: acquiring image information comprising at least one supplemental enhancement information (SEI) message, an SPO SEI message, and a PON SEI message; and on the basis of the SPO SEI message, deriving a processing order for a PON nested SEI message of the PON SEI message and at least one SEI message other than the PON nested SEI message, wherein a first SEI message is included in the at least one SEI message, not in the PON nested SEI message, and a second SEI message having the same payload type as the first SEI message is included in the at least one SEI message, not in the PON nested SEI message, on the basis of identifier information and / or persistence information of the first SEI message.

Description

A method for decoding image information, a method for encoding image information, a method relating to a bitstream, and a computer-readable storage medium for storing a bitstream

[0001] The present disclosure relates to a method for decoding image information, a method for encoding image information, a method for bitstreams, and a computer-readable storage medium for storing bitstreams.

[0002] Recently, the demand for high-resolution, high-quality video, such as HD (High Definition) and UHD (Ultra High Definition), has been increasing across various fields. As video data becomes higher in resolution and quality, the relative amount of information or bits transmitted increases compared to conventional video data. This increase in transmitted information or bits leads to higher transmission and storage costs.

[0003] Accordingly, high-efficiency video compression technology is required to effectively transmit, store, and play back high-resolution, high-quality video information.

[0004] The present disclosure aims to improve the reliability of a coding system including an encoding device and a decoding device.

[0005] The present disclosure aims to improve the coding efficiency of a coding system including an encoding device and a decoding device.

[0006] The present disclosure aims to improve the data transmission efficiency of a coding system including an encoding device and a decoding device.

[0007] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0008] A method for decoding image information according to one aspect of the present disclosure comprises: acquiring image information including at least one SEI (supplemental enhancement information) message, an SPO SEI message, and a PON SEI message; and deriving a processing order for the PON-nested SEI message of the PON SEI message and the at least one SEI message that is not the PON-nested SEI message based on the SPO SEI message, wherein the first SEI message is included in the at least one SEI message that is not the PON-nested SEI message, and the second SEI message having the same payload type as the first SEI message is included in the at least one SEI message that is not the PON-nested SEI message based on identifier information and / or persistence information of the first SEI message.

[0009] According to one aspect of the present disclosure, an apparatus for decoding image information comprises a memory and a processor connected to the memory, wherein the processor acquires the image information including at least one SEI (supplemental enhancement information) message, an SPO SEI message, and a PON SEI message; and derives a processing order for the PON-nested SEI message of the PON SEI message and the at least one SEI message that is not the PON-nested SEI message based on the SPO SEI message, wherein the first SEI message is included in the at least one SEI message that is not the PON-nested SEI message, and the second SEI message having the same payload type as the first SEI message is included in the at least one SEI message that is not the PON-nested SEI message based on the identifier information and / or persistence information of the first SEI message.

[0010] In a method or device for decoding the above image information, a second SEI message having the same payload type as the first SEI message may be included in at least one SEI message that is not the PON-nested SEI message, based on the fact that the first SEI message continues beyond the current picture and does not include identifier information.

[0011] In a method or device for decoding the above image information, a second SEI message having the same payload type as the first SEI message may be included in at least one SEI message that is not the PON-nested SEI message, based on the fact that the first SEI message continues beyond the current picture and includes identifier information identical to the second SEI message.

[0012] In a method or device for decoding the above image information, a second SEI message having the same payload type as the first SEI message may be included in at least one SEI message that is not the PON-nested SEI message, based on the fact that the first SEI message has persistence information such as a value of 1 and does not include identifier information.

[0013] In a method or device for decoding the above image information, a second SEI message having the same payload type as the first SEI message may be included in at least one SEI message that is not the PON-nested SEI message, based on the fact that the first SEI message has persistence information with a value of 1 and includes identifier information identical to the second SEI message.

[0014] In a method or device for decoding the above image information, a second SEI message having the same payload type as the first SEI message may be included in the at least one SEI message that is not the PON-nested SEI message, based on the fact that the first SEI message does not include identifier information and persistence information.

[0015] A method for encoding image information according to one aspect of the present disclosure comprises determining a processing order for a nested SEI (supplemental enhancement information) message and at least one SEI message that is not a PON nested SEI message; and encoding the image information including a PON SEI message generated based on the PON nested SEI message and a SPO SEI message generated based on the processing order, wherein the first SEI message is generated from the at least one SEI message that is not a PON nested SEI message, and the second SEI message having the same payload type as the first SEI message is generated from the at least one SEI message that is not a PON nested SEI message based on identifier information and / or persistence information of the first SEI message.

[0016] According to one aspect of the present disclosure, an apparatus for encoding image information comprises a memory and a processor connected to the memory, wherein the processor determines a processing order for a nested SEI (supplemental enhancement information) message and at least one SEI message that is not a PON nested SEI message; and encodes the image information, which includes a PON SEI message generated based on the PON nested SEI message and a SPO SEI message generated based on the processing order, wherein the first SEI message is generated as the at least one SEI message that is not a PON nested SEI message, and the second SEI message having the same payload type as the first SEI message is generated as the at least one SEI message that is not a PON nested SEI message based on identifier information and / or persistence information of the first SEI message.

[0017] In a method or device for encoding the above-mentioned image information, a second SEI message having the same payload type as the first SEI message may be generated as at least one SEI message that is not the PON-nested SEI message, based on the fact that the first SEI message persists beyond the current picture and does not include identifier information.

[0018] In a method or device for encoding the above-mentioned image information, a second SEI message having the same payload type as the first SEI message may be generated as at least one SEI message that is not the PON-nested SEI message, based on the fact that the first SEI message continues beyond the current picture and includes identifier information identical to the second SEI message.

[0019] In a method or device for encoding the above video information, a second SEI message having the same payload type as the first SEI message may be generated as at least one SEI message that is not the PON nested SEI message, based on the fact that the first SEI message has persistence information such as a value of 1 and does not include identifier information.

[0020] In a method or device for encoding the above video information, a second SEI message having the same payload type as the first SEI message may be generated as at least one SEI message that is not the PON nested SEI message, based on the fact that the first SEI message has persistence information with a value of 1 and includes identifier information identical to the second SEI message.

[0021] In a method or device for encoding the above-mentioned image information, a second SEI message having the same payload type as the first SEI message may be generated as at least one SEI message that is not the PON-nested SEI message, based on the fact that the first SEI message does not include identifier information and persistence information.

[0022] A method for a bitstream according to one aspect of the present disclosure comprises determining a processing order for a nested SEI (supplemental enhancement information) message and at least one SEI message that is not a PON nested SEI message; generating a bitstream based on image information, comprising a PON SEI message generated based on the PON nested SEI message and a SPO SEI message generated based on the processing order; and transmitting data for the bitstream, wherein the first SEI message is generated from the at least one SEI message that is not a PON nested SEI message, and the second SEI message having the same payload type as the first SEI message is generated from the at least one SEI message that is not a PON nested SEI message, based on identifier information and / or persistence information of the first SEI message.

[0023] According to one aspect of the present disclosure, an apparatus for a bitstream comprises: at least one processor for generating a bitstream based on image information, wherein the processor determines a processing order for a nested SEI (supplemental enhancement information) message and at least one SEI message that is not a PON nested SEI message, and the processor generates a PON SEI message based on the PON nested SEI message and a SPO SEI message based on the processing order; and a transmission unit for transmitting data for the bitstream, wherein the first SEI message is generated from the at least one SEI message that is not a PON nested SEI message, and the second SEI message having the same payload type as the first SEI message is generated from the at least one SEI message that is not a PON nested SEI message based on identifier information and / or persistence information of the first SEI message.

[0024] According to one aspect of the present disclosure, a computer-readable storage medium for storing a bitstream, wherein the storage medium stores the bitstream generated based on image information, the bitstream comprising a PON SEI message generated based on a nested SEI (supplemental enhancement information) message and an SPO SEI message generated based on a processing order for at least one SEI message not a PON nested SEI message, wherein the first SEI message is generated from the at least one SEI message not a PON nested SEI message, and the second SEI message having the same payload type as the first SEI message is generated from the at least one SEI message not a PON nested SEI message based on identifier information and / or persistence information of the first SEI message.

[0025] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0026] According to the present disclosure, independence between PON-nested SEI messages and general SEI messages can be ensured.

[0027] According to the present disclosure, by ensuring independence between PON-nested SEI messages and general SEI messages, the reliability of a coding system including an encoding device and a decoding device can be improved.

[0028] According to the present disclosure, by ensuring independence between PON-nested SEI messages and general SEI messages, the coding efficiency of a coding system including an encoding device and a decoding device can be improved.

[0029] According to the present disclosure, by ensuring independence between PON-nested SEI messages and general SEI messages, the data transmission efficiency of a coding system including an encoding device and a decoding device can be improved.

[0030] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below.

[0031] FIG. 1 is a schematic diagram illustrating a video coding system to which an embodiment according to the present disclosure can be applied.

[0032] FIG. 2 is a schematic diagram showing an encoding device to which an embodiment according to the present disclosure can be applied.

[0033] FIG. 3 is a schematic diagram showing a decoding device to which an embodiment according to the present disclosure can be applied.

[0034] Figure 4 illustrates an exemplary hierarchical structure for a coded video / image.

[0035] FIG. 5 is a diagram illustrating a method for decoding image information according to one embodiment of the present disclosure.

[0036] FIG. 6 is a diagram illustrating a method for encoding image information according to one embodiment of the present disclosure.

[0037] FIG. 7 is a drawing illustrating an exemplary content streaming system to which an embodiment according to the present disclosure can be applied.

[0038] Hereinafter, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0039] In describing the embodiments of the present disclosure, detailed descriptions of known configurations or functions are omitted if it is determined that such descriptions could obscure the essence of the present disclosure. Additionally, parts of the drawings unrelated to the description of the present disclosure have been omitted, and similar parts are denoted by similar reference numerals.

[0040] In the present disclosure, when a component is described as being "connected," "combined," or "joined" with another component, this may include not only a direct connection but also an indirect connection in which another component exists in between. Furthermore, when a component is described as "comprising" or "having" another component, this means that, unless specifically stated otherwise, it does not exclude the other component but may include an additional component.

[0041] In the present disclosure, terms such as first, second, etc. are used solely for the purpose of distinguishing one component from another and do not limit the order or importance of the components unless specifically stated otherwise. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and likewise, a second component in one embodiment may be referred to as a first component in another embodiment.

[0042] In this disclosure, distinct components are intended to clearly describe their respective features and do not imply that the components are separate. That is, multiple components may be integrated to form a single hardware or software unit, or a single component may be distributed to form multiple hardware or software units. Accordingly, such integrated or distributed embodiments are included within the scope of this disclosure, unless otherwise noted.

[0043] In the present disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Furthermore, embodiments including additional components in addition to the components described in various embodiments are also included within the scope of the present disclosure.

[0044] The present disclosure relates to the encoding and decoding of images. For example, the methods and embodiments disclosed in this document may be applied to methods disclosed in the VVC (versatile video coding) standard, EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard) or next-generation video / image coding standards (e.g., H.267 or H.268).

[0045] The present disclosure presents various embodiments relating to video / image coding, and unless otherwise stated, said embodiments may be performed in combination with one another.

[0046] Unless newly defined in this disclosure, the terms used herein may have the ordinary meanings commonly used in the technical field to which this disclosure belongs.

[0047] In this disclosure, "video" may refer to a set of images over time. In this disclosure, "picture" generally refers to a unit representing a single image at a specific time, and a slice / tile is a unit that constitutes a part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A picture may be composed of one or more tile groups. A tile group may include one or more tiles. A brick may represent a rectangular area of ​​rows of CTUs within a tile in a picture. In this document, tile groups and slices may be used interchangeably. For example, in this document, a tile group / tile group header may be referred to as a slice / slice header.

[0048] In the present disclosure, "pixel" or "pel" may refer to the smallest unit constituting a picture (or image). Additionally, "sample" may be used as a term corresponding to pixel. A sample may generally represent a pixel or a pixel value, may represent only the pixel / pixel value of the luminance component, or may represent only the pixel / pixel value of the chroma component.

[0049] In this disclosure, "unit" may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. A unit may include one luminance block and two chroma (e.g., cb, cr) blocks. Depending on the case, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0050] In the present disclosure, "current block" may mean one of "current coding block," "current coding unit," "block to be encoded," "block to be decoded," or "block to be processed." When prediction is performed, "current block" may mean "current prediction block" or "block to be predicted." When transformation (inverse transformation) / quantization (inverse quantization) is performed, "current block" may mean "current transformation block" or "block to be transformed." When filtering is performed, "current block" may mean "block to be filtered."

[0051] In the present disclosure, "current block" may mean a block comprising both a luminous component block and a chroma component block, or "luma block of the current block," unless explicitly stated as a chroma block. The luminous component block of the current block may be expressed by including an explicit description of a luminous component block, such as "luma block" or "current luminous block." Additionally, the chroma component block of the current block may be expressed by including an explicit description of a chroma component block, such as "chroma block" or "current chroma block."

[0052] In the present disclosure, " / " and "," may be interpreted as "and / or." For example, "A / B" and "A, B" may be interpreted as "A and / or B." Additionally, "A / B / C" and "A, B, C" may mean "at least one of A, B and / or C."

[0053] In the present disclosure, "or" may be interpreted as "and / or". For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B". Alternatively, in the present disclosure, "or" may mean "additionally or alternatively".

[0054] FIG. 1 is a schematic diagram illustrating a video / image coding system to which an embodiment according to the present disclosure can be applied.

[0055] Referring to FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image or data in the form of a file or streaming to the receiving device via a digital storage medium or a network.

[0056] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be called a video / image encoding device, and the decoding device may be called a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be composed of a separate device or an external component.

[0057] A video source may acquire video / images through processes such as video / image capture, synthesis, or generation. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device may include, for example, a computer, a tablet, and a smartphone, etc., and may generate video / images (electronically). For example, virtual video / images may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process in which related data is generated.

[0058] The encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0059] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device in the form of a file or streaming via a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0060] The decoding device can decode video / images by performing a series of procedures such as inverse quantization, inverse transform, and prediction corresponding to the operation of the encoding device.

[0061] The renderer can render the decoded video / image. The rendered video / image can be displayed through the display unit.

[0062] FIG. 2 is a schematic diagram illustrating an encoding device to which an embodiment according to the present disclosure can be applied.

[0063] Referring to FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a predictor (220), a residual processor (230), an entropy encoder (240), an adder (250), a filter (260), and a memory (270). The predictor (220) may include an inter-predictor (221) and an intra-predictor (222). The residual processor (230) may include a transformer (232), a quantizer (233), a dequantizer (234), and an inverse transformer (235). The residual processor (230) may further include a subtractor (231). The addition unit (250) may be referred to as a reconstructor or a reconstructed block generator. The above-described image segmentation unit (210), prediction unit (220), residual processing unit (230), entropy encoding unit (240), addition unit (250), and filtering unit (260) may be configured by one or more hardware components (e.g., an encoder chipset or processor) according to the embodiment. Additionally, the memory (270) may include a DPB (Decoded Picture Buffer) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0064] The image segmentation unit (210) can divide an input image (or picture, frame) input to an encoding device (200) into one or more processing units. For example, the processing unit may be called a coding unit (CU). A coding unit may be recursively divided into a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, a single coding unit may be divided into multiple coding units of a deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary-tree structure. For example, a quad-tree structure may be applied first, and a binary-tree structure and / or a ternary-tree structure may be applied later. Alternatively, a binary-tree structure may be applied first. A coding procedure according to the present disclosure may be performed based on the final coding unit that is no longer divided. In this case, based on coding efficiency according to image characteristics, the maximum coding unit may be used directly as the final coding unit, or, if necessary, the maximum coding unit may be recursively divided into lower-depth coding units so that a coding unit of the optimal size is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). The prediction unit and the transformation unit may each be divided or partitioned from the final coding unit.The above prediction unit may be a unit of sample prediction, and the above transformation unit may be a unit that derives transformation coefficients and / or a unit that derives a residual signal from transformation coefficients.

[0065] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the context. In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component. A sample may be used to refer to a single picture (or image) as a term corresponding to a pixel or pel.

[0066] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, as illustrated, the unit that subtracts the prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) within the encoding device (200) may be called a subtraction unit (231). The prediction unit (220) can perform a prediction for a block to be processed (hereinafter, current block) and generate a predicted block (predicted block) containing prediction samples for said current block. The prediction unit (220) can determine whether intra prediction is applied or inter prediction is applied in units of the current block or CU. The prediction unit (220) can generate various information regarding prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit it to the entropy encoding unit (240). The information regarding prediction can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0067] The intra prediction unit (222) can predict the current block by referring to samples within the current picture. The referenced samples may be located near the current block or away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a Planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is merely an example, and depending on the settings, more or fewer directional prediction modes may be used. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.

[0068] The inter prediction unit (221) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference blocks and the reference picture containing the temporal neighboring blocks may be the same or different from each other. The temporal neighboring blocks may be referred to by names such as collocated reference block, collocated CU (colCU), etc. A reference picture containing the aforementioned temporal surrounding blocks may be called a collocated picture (colPic). For example, the inter prediction unit (221) may construct a list of motion information candidates based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of surrounding blocks as motion information of the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0069] The prediction unit (220) may generate a prediction signal based on various prediction methods and / or prediction techniques described below. For example, the prediction unit (220) may apply intra prediction or inter prediction for the prediction of the current block, as well as apply intra prediction and inter prediction simultaneously. A prediction method that applies intra prediction and inter prediction simultaneously for the prediction of the current block may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit (220) may be based on an intra block copy (IBC) prediction mode or a palette mode for the prediction of the block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. Palette mode can be viewed as an example of intra-coding or intra-prediction. When palette mode is applied, sample values ​​within a picture can be signaled based on information regarding palette tables and palette indices.

[0070] The prediction signal generated through the prediction unit (220) can be used to generate a restoration signal or to generate a residual signal. The subtraction unit (231) can generate a residual signal (residual signal, residual block, residual sample array) by subtracting the prediction signal (predicted block, prediction sample array) output from the prediction unit (220) from the input image signal (original block, original sample array). The generated residual signal can be transmitted to the conversion unit (232).

[0071] The transformation unit (232) can generate transform coefficients by applying a transformation technique to a residual signal. For example, the transformation technique may include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is represented as a graph. CNT refers to a transformation obtained based on a prediction signal generated using all previously reconstructed pixels. The transformation process may be applied to a block of pixels of the same size in a square, or to a block of variable size that is not square.

[0072] The quantization unit (233) can quantize the transformation coefficients and transmit them to the entropy encoding unit (240). The entropy encoding unit (240) can encode the quantized signal (information regarding the quantized transformation coefficients) and output it as a bitstream. The information regarding the quantized transformation coefficients may be called residual information. The quantization unit (233) can rearrange the block-shaped quantized transformation coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information regarding the quantized transformation coefficients based on the one-dimensional vector-shaped quantized transformation coefficients.

[0073] The entropy encoding unit (240) can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit (190) may encode information required for video / image restoration (e.g., values ​​of syntax elements) together or separately, in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The signaling information, transmitted information, and / or syntax elements mentioned in the present disclosure may be included in the video / image information. The video / image information may be encoded through the encoding procedure described above and included in the bitstream.

[0074] The above bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) for transmitting a signal output from the entropy encoding unit (240) and / or a storage unit (not shown) for storing it may be provided as an internal / external element of the encoding device (200), or the transmission unit may be provided as a component of the entropy encoding unit (240).

[0075] The quantized transformation coefficients output from the quantization unit (233) can be used to generate a residual signal. For example, a residual signal (residual block or residual samples) can be restored by applying inverse quantization and inverse transformation to the quantized transformation coefficients through the inverse quantization unit (234) and the inverse transformation unit (235).

[0076] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0077] The adder (250) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (221) or the intra prediction unit (222). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block. The adder (250) may be called a reconstructed unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after undergoing filtering as described below.

[0078] The filtering unit (260) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (260) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (270), specifically in the DPB of memory (170). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240), as described below in the description of each filtering method. The information regarding filtering can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0079] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter-prediction unit (221). Through this, the encoding device (200) can avoid prediction mismatches between the encoding device (200) and the decoding device when inter-prediction is applied, and can also improve encoding efficiency.

[0080] The DPB in memory (270) can store a modified restored picture to be used as a reference picture in the inter prediction unit (221). Memory (270) can store motion information of blocks from which motion information is derived (or encoded) in the current picture and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter prediction unit (221) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory (270) can store restoration samples of restored blocks in the current picture and transmit them to the intra prediction unit (222).

[0081] FIG. 3 is a schematic diagram illustrating a decoding device to which an embodiment according to the present disclosure can be applied.

[0082] As illustrated in FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-predictor (332) and an intra-predictor (331). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321). The aforementioned entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) may be configured by a single hardware component (e.g., a decoder chipset or a processor) according to an embodiment. Additionally, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0083] When a bitstream containing video / image information is input, the decoding device (300) can restore the image by performing a process corresponding to the process performed by the encoding device (200) of FIG. 2. For example, the decoding device (300) can perform decoding using a processing unit applied in the encoding device (200). Thus, the processing unit for decoding may be, for example, a coding unit. The coding unit may be a coding tree unit, or a maximum coding unit may be obtained by dividing it according to a quad tree structure, a binary tree structure, and / or a binary tree structure. And, the restored image signal decoded and output through the decoding device (300) can be played back through a playback device (not shown).

[0084] The decoding device (300) can receive a signal output from the encoding device (200) of FIG. 2 in the form of a bitstream. The received signal can be decoded through an entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information necessary for image restoration (or picture restoration) (e.g., video / image information). The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The decoding device (300) can decode the picture based on the information regarding the parameter sets and / or the general constraint information. The signaling / received information and / or syntax elements described below can be obtained from the bitstream by decoding through the decoding procedure. For example, the entropy decoding unit (310) can decode information within the bitstream based on coding methods such as exponential coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transformation coefficients regarding residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, determine a context model using information on the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information on symbols / bins decoded in the previous step, predict the probability of occurrence of the bin according to the determined context model, and perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the decoded symbol / bin information for the context model of the next symbol / bin after determining the context model. Among the information decoded in the entropy decoding unit (310), information regarding prediction is provided to the prediction unit (330), and residual values ​​for which entropy decoding was performed in the entropy decoding unit (310), i.e., quantized transformation coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive residual signals (residual blocks, residual samples, residual sample array). Additionally, among the information decoded in the entropy decoding unit (310), information regarding filtering can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (300), or the receiving unit may be a component of the entropy decoding unit (310). Meanwhile, the decoding device according to the present document may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit (310), and the sample decoder may include at least one of the inverse quantization unit (321), inverse transform unit (322), adder (340), filtering unit (350), memory (360), inter prediction unit (332), and intra prediction unit (331).

[0085] In the inverse quantization unit (321), the quantized transformation coefficients can be inversely quantized to output transformation coefficients. The inverse quantization unit (321) can rearrange the quantized transformation coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device (200). The inverse quantization unit (321) can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) and obtain transformation coefficients.

[0086] In the inverse conversion unit (322), the conversion coefficients can be inversely converted to obtain a residual signal (residual block, residual sample array).

[0087] The prediction unit (330) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but it can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information regarding the palette table and palette index can be included in the above video / image information and signaled.

[0088] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The description of the intra prediction unit (222) may be applied equally to the intra prediction unit (331). The referenced samples may be located in the neighborhood of the current block or located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit (331) may determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0089] The inter prediction unit (332) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) may construct a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-prediction can be performed based on various prediction modes (techniques), and information regarding the prediction may include information indicating the mode (technique) of inter-prediction for the current block.

[0090] The adder (340) can generate a restoration signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the prediction signal (predicted block, predicted sample array) output from the prediction unit (330) (including the inter prediction unit (332) and / or intra prediction unit (331)). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the restoration block. The description of the adder (250) can be applied equally to the adder (340). The adder (340) may be called a restoration unit or a restoration block generation unit. The generated restoration signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after undergoing filtering as described below.

[0091] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0092] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (360), specifically in the DPB of memory (360). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0093] The (modified) restored picture stored in the DPB of the memory (360) can be used as a reference picture in the inter-prediction unit (332). The memory (360) can store motion information of blocks from which motion information within the current picture has been derived (or decoded) and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit (332) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (360) can store restoration samples of blocks restored within the current picture and transmit them to the intra-prediction unit (331).

[0094] In this specification, the embodiments described in the filtering unit (260), inter prediction unit (221), and intra prediction unit (222) of the encoding device (200) may be applied to the filtering unit (350), inter prediction unit (332), and intra prediction unit (331) of the decoding device (300) in the same or corresponding manner.

[0095] Figure 4 illustrates an exemplary hierarchical structure for a coded video / image.

[0096] Referring to Figure 4, the coded image is divided into a Video Coding Layer (VCL) that handles the decoding processing of the image and the image itself, a subsystem that transmits and stores the encoded information, and a Network Abstraction Layer (NAL) that exists between the VCL and the subsystem and is responsible for network adaptation functions.

[0097] In VCL, VCL data containing compressed image data (slice data) can be generated, or parameter sets containing information such as Picture Parameter Set (PPS), Sequence Parameter Set (SPS), and Video Parameter Set (VPS), or SEI (Supplemental Enhancement Information) messages that are additionally required in the decoding process of the image can be generated.

[0098] In NAL, a NAL unit can be created by adding header information (NAL unit header) to the Raw Byte Sequence Payload (RBSP) generated in VCL. In this case, the RBSP refers to slice data, parameter sets, SEI messages, etc. generated in VCL. The NAL unit header may include NAL unit type information specified according to the RBSP data included in the NAL unit.

[0099] As shown in FIG. 4, NAL units can be classified into VCL NAL units and Non-VCL NAL units depending on the RBSP generated in VCL. A VCL NAL unit may refer to a NAL unit containing information about an image (slice data), and a Non-VCL NAL unit may refer to a NAL unit containing information necessary to decode an image (parameter set or SEI message).

[0100] The aforementioned VCL NAL unit and Non-VCL NAL unit can be transmitted over a network by attaching header information according to the data specifications of the underlying system. For example, the NAL unit can be transformed into a data format of a specified specification, such as H.266 / VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted over various networks.

[0101] As described above, the NAL unit type can be determined according to the RBSP data structure included in the NAL unit, and information about this NAL unit type can be stored in the NAL unit header and signaled.

[0102] For example, NAL units can be broadly classified into VCL NAL unit types and Non-VCL NAL unit types depending on whether they contain information about the image (slice data). VCL NAL unit types can be classified according to the properties and types of the picture included in the VCL NAL unit, while Non-VCL NAL unit types can be classified according to the types of parameter sets.

[0103] The following is an example of a NAL unit type specified according to the type of parameter set included in the Non-VCL NAL unit type.

[0104] - APS (Adaptation Parameter Set) NAL unit: Type for the NAL unit containing the APS

[0105] - DPS(Decoding Parameter Set) NAL unit: Type for the NAL unit containing the DPS

[0106] - VPS (Video Parameter Set) NAL unit: Type for the NAL unit containing the VPS

[0107] - SPS (Sequence Parameter Set) NAL unit: Type for the NAL unit containing the SPS

[0108] - PPS(Picture Parameter Set) NAL unit: Type for the NAL unit containing the PPS

[0109] The above-described NAL unit types have syntax information for the NAL unit type, and said syntax information can be stored in the NAL unit header and signaled. For example, said syntax information may be nal_unit_type, and NAL unit types may be specified by the nal_unit_type value.

[0110] A slice header (slice header syntax, slice header information) may include information / parameters that can be commonly applied to the slice. The APS (APS syntax) or PPS (PPS syntax) may include information / parameters that can be commonly applied to one or more slices or pictures. The SPS (SPS syntax) may include information / parameters that can be commonly applied to one or more sequences. The VPS (VPS syntax) may include information / parameters that can be commonly applied to multiple layers. The DPS (DPS syntax) may include information / parameters that can be commonly applied to the entire video. The DPS may include information / parameters related to the concatenation of a CVS (coded video sequence). In the present disclosure, High Level Syntax (HLS) may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, or slice header syntax.

[0111] In the present disclosure, image / video information encoded by an encoding device and signaled in the form of a bitstream includes not only information related to picture partitioning, intra / inter prediction information, residual information, in-loop filtering information, etc., but may also include information included in the slice header, information included in the APS, information included in the PPS, information included in the SPS, information included in the VPS, and / or information included in the DPS.

[0112] The following descriptor of the present disclosure specifies the parsing process for each syntax element.

[0113] - ae(v): Context-adaptive arithmetic entropy encoding syntax element.

[0114] - b(8): A byte (8 bits) with an arbitrary bit sequence pattern. The parsing process for this descriptor is specified by the return value of the function read_bits(8).

[0115] - f(n): An n-bit fixed pattern bit sequence written starting from the left bit (left to right). The parsing process for this descriptor is specified by the return value of the function read_bits(n).

[0116] - i(n): A signed integer using n bits. If n is "v" in the syntax table, the number of bits depends on the value of other syntax elements. The parsing process of this descriptor is specified by the return value of the read_bits(n) function, which is interpreted as a 2-complement integer representation with the most significant bit written first.

[0117] - se(v): A zero-order Exp-Golomb-coded signed integer syntax element starting from the left bit. The parsing process of this descriptor is specified as the case where order k is 0.

[0118] - st(v): A null-terminated string encoded in Universal Code Character Set (UCS) Transport Format-8 (UTF-8) characters. The syntax resolution process is specified as follows: st(v) reads a sequence of bytes from the bitstream starting at the byte alignment position within the bitstream, from the current position to the next byte alignment position (excluding bytes equal to 0x00), and returns the bitstream pointer by (stringLength + 1) * 8 bit positions, where stringLength is equal to the number of bytes returned.

[0119] For reference, the st(v) syntax descriptor is used in this disclosure only when the current position in the bitstream is a byte-aligned position.

[0120] - tu(v): A truncated one-way operator using up to maxVal bits, where maxVal is defined in the semantics of the syntax element.

[0121] - u(n): An unsigned integer using n bits. If n is "v" in the syntax table, the number of bits depends on the values ​​of other syntax elements. The parsing process of this descriptor is determined by interpreting the return value of the read_bits(n) function as a binary representation of an unsigned integer with the most significant bit written first.

[0122] - ue(v): An unsigned integer Exp-Golomb coded syntax element of order 0 with the left bit first. The parsing process for this descriptor is specified as the case where order k is 0.

[0123] The SEI message related to the present disclosure will be described below.

[0124] Table 1 shows an example of SEI message syntax for SEI processing order according to one embodiment.

[0125] [Table 1]

[0126]

[0127] An example of SEI message semantics for an SEI processing sequence according to one embodiment is described.

[0128] SEI Processing Order (SPO): SEI messages convey information indicating the preferred processing order determined by the encoder (i.e., the content creator) for the groups of SEI message types that may exist in CVS.

[0129] To use this SEI message, you must define the following:

[0130] - Two payload type value lists, SeiProcessingOrderSeiList and SpoProcessSeiList.

[0131] The semantics of SPO SEI messages utilize the concept of SEI message types. SEI messages with different payloadType values ​​are considered to be of different types of SEI messages. Additionally, different SEI messages that have the same payloadType value but are distinguished by the syntax element values ​​within the SEI payload are also considered to be of different types. This distinction based on syntax element values ​​within the SEI payload is performed by comparing the transmitted values ​​using the po_sei_prefix_data_bit[ i ][ j ] syntax element (if present) or by comparing the transmitted values ​​within SEI messages in processing order nesting SEI messages (if present). For example, Neural Network Post-processing Filter (NNPFC) SEI messages can be distinguished because they have different nnpfc_id values.

[0132] If the i-th SEI message seiA within a certain SPO SEI message has both po_sei_wrapping_flag[ i ] and po_sei_prefix_flag[ i ] 0, then other SEI messages seiB contained in the same SPO SEI message or another SPO SEI message within the current CVS must not satisfy all of the following conditions:

[0133] - The value of po_sei_payload_type[i] in seiB is the same as the value in seiA.

[0134] - The value of po_sei_wrapping_flag[i] in seiB is 0.

[0135] - The value of po_sei_prefix_flag[i] in seiB is 1.

[0136] If an SPO SEI message with a specific po_id value exists in any access unit of the CVS, that SPO SEI message with the specific po_id value must exist in the first access unit of the CVS in terms of decoding order. The number of SEI messages and the payloadType code of the SEI messages indicated within each SPO SEI message with the same po_id value are maintained in the decoding order from the current access unit to the end of the CVS and are provided in the output order.

[0137] An SPO SEI message may include one or more SEI prefix marks of a specific payloadType. If present, each SEI prefix mark is a bit string following the SEI payload syntax of the corresponding payloadType value, containing multiple complete syntax elements starting from the first syntax element of the SEI payload. These SEI prefix marks must provide sufficient information to determine a specific processing order for SEI message types that have the same payloadType value but different preferred processing orders.

[0138] po_id contains an identification number that identifies the SPO SEI message.

[0139] A processing chain consists of a list of SEI message types identified by the SPO SEI message, in the order of priority processing indicated in the SPO SEI message. If multiple processing chains are indicated by the SPO SEI message for the same access unit, the decoder must operate only one of the indicated processing chains.

[0140] Each SEI message type within the processing chain specified by the SPO SEI message is identified by the syntax elements po_sei_payload_type[ i ], po_sei_wrapping_flag[ i ], po_sei_processing_order[ i ], and if present, po_num_bits_in_prefix_indication_minus1[ i ] and po_prefix_data_bit[ i ][ j ].

[0141] SEI message types do not necessarily have to belong to a specific processing chain and can belong to multiple processing chains identified by SPO SEI messages with different po_id values.

[0142] Each SEI message of the SEI message type identified within the SPO SEI message has the same persistence range as when the corresponding SEI message is transmitted outside the SPO SEI message and is not identified within the SPO SEI message.

[0143] For reference, if an SEI message specifies a process and is not associated with a processing chain specified in any SPO SEI message, that SEI message itself implicitly becomes the processing chain. Some standards specify an implicit processing chain where another NNPF is executed following an ultra-high resolution NNPF. The implicitly specified processing chain is treated the same as the processing chain specified by the SPO SEI message when selecting the SEI message to apply.

[0144] If po_for_human_viewing_idc is 3, it indicates that human viewing is included in the optimal use purpose of the image generated by the processing chain specified by this SPO SEI message. If po_for_human_viewing_idc is 2, it indicates that the image generated by the processing chain specified by this SPO SEI message is suitable for human viewing but is not specifically optimized. If po_for_human_viewing_idc is 1, it indicates that the image generated by the processing chain specified by this SPO SEI message is unsuitable for human viewing. If po_for_human_viewing_idc is 0, it indicates that it is unknown whether the image generated by the processing chain specified by this SPO SEI message is suitable for human viewing.

[0145] If the value of po_for_machine_analysis_idc is 3, it indicates that machine analysis is included in the intended optimal use case of the image generated by the processing chain specified by this SPO SEI message. If the value of po_for_machine_analysis_idc is 2, it indicates that the image generated by the processing chain specified by this SPO SEI message is suitable for machine analysis but is not specifically optimized. If po_for_machine_analysis_idc is 1, it indicates that the image generated by the processing chain specified by this SPO SEI message is unsuitable for machine analysis. If po_for_machine_analysis_idc is 0, it indicates that it is unknown whether the image generated by the processing chain specified by this SPO SEI message is suitable for machine analysis.

[0146] As a requirement for bitstream conformity, the values ​​of po_for_human_viewing_idc and po_for_machine_analysis_idc must not both be 1.

[0147] For reference, the values ​​of po_for_human_viewing_idc and po_for_machine_analysis_idc must be the same as the values ​​of the encoder optimization information (EOI) SEI messages (eoi_for_human_viewing_idc and eoi_for_machine_analysis_idc) or the neural network post-processing filter characteristic (NNPFC) SEI messages (nnpfc_for_human_viewing_idc and nnpfc_for_machine_analysis_idc).

[0148] po_reserved_zero_4bits must be equal to 0. Values ​​greater than 0 for po_reserved_zero_4bits are reserved for future use and must not exist in a bit stream compliant with the present disclosure. A decoder compliant with the present disclosure must accept all values ​​of po_reserved_zero_4bits within the range (inclusive) from 0 to 15.

[0149] The value of po_num_sei_messages_minus2 plus 2 represents the number of SEI message types with processing priority assigned in SPO SEI messages.

[0150] If po_breadth_first_flag is 1, breadth-first handling of the processing chain is applied to determine the picture used to interpret the semantics of the SEI message applied as part of the processing chain specified by this SPO SEI message. If po_breadth_first_flag is 0, breadth-first handling of the processing chain or depth-first handling of the processing chain is applied to determine the picture used to interpret the semantics of the SEI message applied as part of the processing chain specified by this SPO SEI message.

[0151] For reference, when po_breadth_first_flag is 0, the processing chain can be performed on the cropped decoded picture without processing SEI messages applied to subsequent picture units in the output order.

[0152] If po_sei_wrapping_flag[ i ] is 1, the SEI message (if any) applied as the i-th SEI message type within the processing chain specified in this SPO SEI message indicates that it is a SEI message included in a PON SEI message where both of the following two conditions are true:

[0153] - pon_target_po_id[ j ] exists where j is the same as po_id.

[0154] - Processing order nesting: The k-th loop entry exists within the SEI message, and the payloadType of the k-th nested SEI message is the same as po_sei_payload_type[ i ] and pon_processing_order[ k ] is the same as po_sei_processing_order[ i ].

[0155] If po_sei_wrapping_flag[ i ] is 0, the SEI message (if any) applied as the i-th SEI message type in the processing chain specified in this SPO SEI message is a SEI message not included in the PON SEI message, and this means that both of the following two conditions are true:

[0156] - The payload type of the SEI message is the same as po_sei_payload_type[ i ].

[0157] - If po_sei_prefix_flag[ i ] is 0 or po_sei_prefix_flag[ i ] is 1, the payload of the SEI message starts with the value of po_sei_prefix_data_bit[ i ][ j ].

[0158] For reference, when po_sei_wrapping_flag[ i ] is 1, it ensures that the SEI message is included within the processing order nesting SEI message, preventing decoders that do not process SPO SEI messages from misinterpreting the SEI message. Therefore, when po_sei_wrapping_flag[ i ] is 1, it is intended for use in situations where unintended results may occur in the corresponding decoder when po_sei_wrapping_flag[ i ] is 0.

[0159] If po_sei_importance_flag[ i ] is 1, it affects the derivation of PoSeiList, which is a list of SEI messages that the decoding system must process for a specific picture picA, as specified below.

[0160] If po_sei_importance_flag[ i ] is 0, it specifies that when the decoding system cannot interpret or does not support the function represented by the i-th SEI message type, all data associated with the i loop variable value should be ignored and the i-th SEI message type should be excluded from the processing chain performed by the decoding system.

[0161] po_sei_processing_degree_flag[ i ] affects the derivation of PoSeiList as specified below.

[0162] PoSeiList is derived as follows:

[0163] - PoSeiList is initially empty.

[0164] - The following applies to all i values ​​in the range (inclusive) from 0 to po_num_sei_messages_minus2 + 1 in the non-decreasing order of po_sei_processing_order[ i ] values ​​and continues unless terminated early as specified below:

[0165] - If the SEI message seiA associated with the i-th SEI message type persists for picA, the following applies:

[0166] - If the decoding system can interpret and support the function represented by seiA, seiA is added to the end of PoSeiList.

[0167] - Otherwise, if po_sei_importance_flag[ i ] is 1 and po_sei_processing_degree_flag[ i ] is 0, the derivation of PoSeiList ends.

[0168] - Otherwise, if po_sei_importance_flag[ i ] is 1 and po_sei_processing_degree_flag[ i ] is 1, the processing chain specified by this SPO SEI message must not be performed for picA, PoSeiList is set to empty and the derivation of PoSeiList is terminated.

[0169] po_sei_payload_type[ i ] specifies the payloadType value of the i-th type of SEI message.

[0170] If po_sei_prefix_flag[ i ] is 1, it indicates that po_num_bits_in_prefix_indication_minus1[ i ] and some po_sei_prefix_data_bit[ i ][ j ] syntax elements exist. If po_sei_prefix_flag[ i ] is 0, it indicates that these syntax elements do not exist.

[0171] For each i in the range from 0 to po_num_sei_messages_minus2 + 1 (inclusive), the value of po_sei_payload_type[ i ] must be equal to the value of SeiProcessingOrderSeiList.

[0172] When po_sei_payload_type[ i ] is equal to the value of SpoProcessSeiList, the i-th SEI message type represents a process.

[0173] spoPropertySeiList is configured to consist of the payloadType values ​​included in SeiProcessingOrderSeiList, excluding the payloadType values ​​included in SpoProcessSeiList. If po_sei_payload_type[ i ] is the same as a value in spoPropertySeiList, the i-th SEI message type represents an attribute.

[0174] po_sei_processing_order[ i ] indicates the preferred processing order of the i-th SEI message type for which preferred processing order information is provided in the SPO SEI message. If m and n are two distinct integer values, if po_sei_processing_order[ m ] is less than po_sei_processing_order[ n ], it indicates that the SEI message type associated with index m must be processed before the SEI message type associated with index n, and if po_sei_processing_order[ m ] is equal to po_sei_processing_order[ n ], it indicates that there is no processing priority between the SEI message types associated with indices m and n (e.g., both may represent different attributes applicable at that stage, or one may represent an attribute and the other a process).

[0175] When i is greater than 0, po_sei_processing_order[ i ] must be greater than or equal to po_sei_processing_order[ i - 1 ].

[0176] Let seiMsgA be an SEI message that is applied as the i-th SEI message type in the processing chain specified in this SPO SEI message, persists for a specific picture picA, and has po_sei_processing_order[ i ] equal to poValA.

[0177] Let seiMsgSet be a set of SEI messages consisting of each SEI message satisfying all of the following conditions:

[0178] - The corresponding SEI message is applied as the k-th SEI message type in the processing chain specified in this SPO SEI message, where the value of k is less than i.

[0179] - The corresponding SEI message persists for picA.

[0180] - po_sei_processing_order[ k ] is smaller than poValA.

[0181] - The payloadType value of the corresponding SEI message is one of the values ​​included in SpoProcessSeiList.

[0182] Pictures to which the semantics of seiMsgA apply are designated as follows:

[0183] - If seiMsgSet is not empty, the semantics of seiMsgA apply to all pictures generated by the process implied by the SEI message with the largest po_sei_processing_order[ k ] value among the SEI messages in seiMsgSet.

[0184] - Otherwise, the semantics of seiMsgA apply to picA.

[0185] For reference, when an NNPF process outputs one or more NNPF-generated pictures, the semantics of the SEI message following NNPF in the processing order apply to all of these NNPF-generated pictures.

[0186] If po_num_bits_in_prefix_indication_minus1[ i ] and po_sei_prefix_data_bit[ i ][ j ] exist, they have the same meaning as the syntax elements num_bits_in_prefix_indication_minus1[ i ] and sei_prefix_data_bit[ i ][ j ], and prefix_sei_payload_type is replaced with po_sei_payload_type[ i ].

[0187] If there is one or more SPO SEI messages with a specific po_id value in CVS, the value of po_num_sei_messages_minus2 and for each i value, the values ​​of po_sei_wrapping_flag[ i ], po_sei_prefix_flag[ i ], po_sei_importance_flag[ i ], po_sei_payload_type[ i ], and po_sei_processing_order[ i ] must be the same as other SPO SEI messages with the same po_id value in CVS.

[0188] Table 2 shows an example of a processing order nesting SEI message syntax according to one embodiment.

[0189] [Table 2]

[0190]

[0191] An example of processing order nesting SEI message semantics according to one embodiment is described.

[0192] A Process Order Nesting (PON) SEI message contains one or more SEI messages, which must be applied only as part of the processing chain identified by the associated SEI processing order SEI message, and must not be applied in a manner that conflicts with the processing chain identified by the associated SEI processing order SEI message.

[0193] To use this SEI message, you must define the following:

[0194] - Syntax structure of SEI message container, sei_pon_nested_message()

[0195] SEI messages included in a PON SEI message are called PON nested SEI messages.

[0196] For reference, the encoder may contain multiple PON SEI messages within the same access unit. For example, the first PON SEI message within the access unit may contain a PON nested SEI message applicable to multiple processing chains and one or more other PON SEI messages applicable to only a single processing chain within the same access unit.

[0197] As a requirement of bitstream conformity, the semantics and effects of a SEI message that is not a PON-nested SEI message must not depend on any PON-nested SEI message. As a result of this constraint, the following specific constraints arise, wherein the associated SEI message is considered to be a SEI message that influences the semantics or effects of a specific SEI message:

[0198] - If a neural network post-processing filter characteristic SEI message has a specific nnpfc_id value and is a PON-nested SEI message, then the associated neural network post-processing filter activation SEI message, where nnpfa_target_id is the same as that specific nnpfc_id value, must also be a PON-nested SEI message.

[0199] - If nnpfa_persistence_flag is 1 and there is a neural network post-processing filter activation (NNPFA) SEI message that is not a PON nested SEI message with a specific nnpfa_target_id value, and there is an NNPFA SEI message with the same nnpfa_target_id value in the next picture in the same CLVS in the output order (if applicable), then there must not be an associated NNPFA SEI message that is a PON nested SEI message in that picture.

[0200] - If a film grain characteristic SEI message exists and fg_characteristics_persistence_flag is 1 and is not a PON-nested SEI message, there must not be an associated film grain characteristic SEI message that is a PON-nested SEI message within the same CLVS.

[0201] - If a picture packing array SEI message exists and fp_arrangement_persistence_flag is 1 and is not a PON nested SEI message, there must not be an associated picture packing array SEI message within the same CLVS that has fp_arrangement_cancel_flag 1 or has the same fp_arrangement_id value and is a PON nested SEI message.

[0202] - If a Content Color Volume (CCV) SEI message exists and ccv_persistence_flag is 1 and it is not a PON-nested SEI message, there must not be an associated Picture Packing Array SEI message that is a PON-nested SEI message within the same CLVS.

[0203] - If there is an conformal SEI message with an ERP persistence flag of 1 but it is not a PON-nested SEI message, there must not be an associated conformal SEI message that is a PON-nested SEI message within the same CLVS.

[0204] - If a sphere rotation SEI message with sphere_rotation_persistence_flag 1 exists but is not a PON-nested SEI message, there must not be any PON-nested sphere rotation SEI messages associated with that message within the same CLVS.

[0205] - If there is a region-specific packing SEI message with rwp_persistence_flag 1 that is not a PON-nested SEI message, there must not be an associated region-specific packing SEI message that is a PON-nested SEI message within the same CLVS.

[0206] - If there is a forward viewport SEI message with omni_viewport_persistence_flag of 1 and that message is not a PON-nested SEI message, there must not be an associated forward viewport SEI message that is a PON-nested SEI message within the same CLVS.

[0207] - If there is a sample aspect ratio SEI message with sari_persistence_flag 1 but it is not a PON-nested SEI message, there must not be an associated sample aspect ratio SEI message that is a PON-nested SEI message within the same CLVS.

[0208] - If an annotated area SEI message exists and it is not a PON-nested SEI message, there must not be an associated annotated area SEI message within the same CLVS that is a PON-nested SEI message with that message.

[0209] - If an alpha channel information SEI message exists and it is not a PON-nested SEI message, there must not be an associated alpha channel information SEI message that is a PON-nested SEI message within the same CLVS.

[0210] - If a display direction SEI message exists that is not a PON nested SEI message, there must not be an associated display direction SEI message that is a PON nested SEI message within the same CLVS.

[0211] - If a color transformation indicator SEI message exists and colour_transform_persistence_flag is 1 and is not a PON-nested SEI message, there must not be an associated color transformation indicator SEI message within the same CLVS that colour_transform_cancel_flag is 1 or has the same colour_transform_id value and is a PON-nested SEI message.

[0212] pon_num_po_ids_minus1 plus 1 specifies the number of SEI processing sequence SEI messages associated with this PON SEI message.

[0213] pon_target_po_id[ i ] represents the po_id of the i-th SEI processing sequence SEI message associated with this PON SEI message.

[0214] pon_num_seis_minus1 plus 1 specifies the number of PON nested SEI messages included in this PON SEI message.

[0215] pon_processing_order[ i ] specifies the position of the i-th PON-nested SEI message within the processing order defined by the associated SEI processing order SEI message. When i is greater than 0, pon_processing_order[ i ] must be greater than or equal to pon_processing_order[ i - 1 ].

[0216] For the i-th PON-nested SEI message, the associated SEI processing sequence SEI message is a SEI processing sequence SEI message with item k satisfying all of the following conditions:

[0217] - po_sei_processing_order[ k ] is the same as pon_processing_order[ i ].

[0218] - po_sei_payload_type[ k ] is the same as the payloadType value of the i-th PON-nested SEI message.

[0219] - When po_sei_prefix_flag[ k ] is 1, po_sei_prefix_data_bit [ k ][ j ] (where j ranges from 0 to po_num_bits_in_prefix_indication_minus1[ k ]) contains the same content as the initial bit of po_num_bits_in_prefix_indication_minus1[ k ] plus 1 of the SEI message payload of the i-th PON-nested SEI message.

[0220] The i-th PON nested SEI message can have any number of associated SEI processing sequence SEI messages in the range from 0 to pon_num_po_ids_minus1 + 1 (inclusive).

[0221] If there is an associated SEI processing sequence SEI message for the i-th PON nested SEI message, the i-th PON nested SEI message must be applied as the k-th loop item of the associated SEI processing sequence SEI message.

[0222] The semantics of the i-th PON-nested SEI message applied as the k-th loop entry of an associated SEI processing sequence SEI message with a specific po_id value are applied without considering any PON-nested SEI messages not associated with that SEI processing sequence SEI message with the specific po_id value.

[0223] For all SEI processing sequence SEI messages that exist in CVS and are equal to pon_target_po_id[m] for any value m in the range from 0 to pon_num_po_ids_minus1 (inclusive), at least one value n in the range from 0 to pon_num_seis_minus1 (inclusive) must exist, and the SEI processing sequence SEI message must be the SEI processing sequence SEI message associated with the nth PON nested SEI message.

[0224] general details

[0225] The processing chains are interchangeable, meaning that the decoding system can apply at most one processing chain at a time.

[0226] A special case of NNPF cascading is defined as when two NNPFs are both enabled for a single picture: both PPFs are NNPFs, one of the two NNPFs has nnpfc_purpose of 4 and the other has multiple input pictures, and neither NNPF is associated with an SPO SEI message. In this case, the two NNPFs are implicitly considered to belong to a single processing chain, and the NNPF with nnpfc_purpose of 4 is applied first.

[0227] Except for special NNPF cascading cases, each processing chain containing multiple SEI message types is represented by an SPO SEI message with a specific po_id value. Except for special NNPF cascading cases, all SEI messages that have payloadType in SpoProcessSeiList but are not represented by an SPO SEI message belong to their own processing chain.

[0228] Width-first processing of the processing chain

[0229] The decoding system can select and apply processing chains as follows:

[0230] First, decode the bitstream, set the PoPicList list to the cropped decoded picture list in the output order generated as a result of the bitstream decoding, and select the processing chain.

[0231] - For each SEI message type in the selected processing chain, the following is applied in the non-decreasing order of the corresponding po_sei_processing_order[ i ] values:

[0232] - If there is a SEI message associated with the i-th SEI message type in the PoSeiList derived for a picture with an NNPF enabled for picA generated for picA by a preceding process in the processing chain, the following applies to each picture picA in PoPicList in the order of output:

[0233] - If picA is not a cropped decoded picture, the following exception applies to SEI message interpretation:

[0234] Interface variables for interpreting SEI messages are derived from picA instead of syntax elements representing the attributes of the cropped and decoded image.

[0235] - The semantics of the SEI message, or (if the SEI message is an NNPFA SEI message) the semantics of the associated NNPFC SEI message and SEI message, are applied to the images in PoPicList instead of the cropped decoded images.

[0236] - If the i-th SEI message type exists in SpoProcessingList, the processing implied by the corresponding SEI message is performed; the picture is replaced with the corresponding processed picture (if any) generated as a result of the processing, and PoPicList is updated by inserting another picture (if any) generated as a result of the processing into PoPicList in accordance with the output order.

[0237] Depth-first processing of the processing chain

[0238] The decoding system can select and apply processing chains as follows:

[0239] First, decode the bitstream, set the PoPicList list to the cropped decoded picture list in the output order generated as a result of the bitstream decoding, and select the processing chain.

[0240] - The following is applied iteratively to each picture picA in PoPicList in the order of output:

[0241] - The following applies to each SEI message in PoSeiList and is derived for picA in increasing order of the list index of PoSeiList:

[0242] - If the current SEI message is not the first in the set of SEI messages, the following exceptions apply to SEI message interpretation:

[0243] - Interface variables for interpreting SEI messages are derived from the images in the updated PoPicList instead of syntax elements representing the attributes of the cropped and decoded images.

[0244] - The semantics of the SEI message, or (if the SEI message is an NNPFA SEI message) the semantics of the associated NNPFC SEI message and SEI message, are applied to the images in PoPicList instead of the cropped decoded images.

[0245] - The process implied by the SEI message is called repeatedly in the output order. This is performed for each picture in picA and PoPicList, and these pictures are the interpolated / extrapolated pictures or equivalent pictures generated by applying the process implied by the previous SEI message (if any) to picA. After each process call, PoPicList is updated by replacing the picture with the corresponding processed picture (if any) generated as a result of the process, and inserting another picture (if any) generated as a result of the process into PoPicList to adhere to the output order.

[0246] SEI messages that are part of a processing chain directed by SEI Sequence Order (SPO) messages can exist inside or outside of Process Sequence Nesting (PON) SEI messages. The design of PON SEI messages includes several specific constraints for each SEI message that can be included in a PON SEI message. However, additional constraints must be defined for new SEI messages that may be included in future or currently existing SEI messages. Instead of specific constraints for each PON nesting SEI message, general forms of constraints are required.

[0247] In one embodiment, the following items may be applied individually or in combination.

[0248] 1. Specify a constraint such that if a non-PON nested SEI message seiA exists, when one of the following conditions is true, an associated SEI message seiB, which is a PON nested SEI message with the same payloadType in the same CLVS, does not exist, as a requirement for bitstream conformance.

[0249] - No persistence information in seiA.

[0250] - seiA persists beyond the current picture and has no identifier or target identifier.

[0251] - seiA has an identifier or target identifier identical to the identifier or target identifier of seiB that persists beyond the current picture.

[0252] 2. Specify a list (e.g., PonSeiList) configured to consist of payloadType values.

[0253] - The list may consist of some or all of the following values: 3, 4, 5, 19, 45, 47, 137, 142, 144, 147, 148, 149, 150, 153, 154, 155, 156, 165, 177, 202, 204, 210, 211, 215, 216, 217, 218, 219.

[0254] 3. Specify a list consisting of payloadType values ​​(e.g., PonSeiList).

[0255] - The list PonSeiList is set to be the same as the list SeiProcessingOrderSeiList.

[0256] - The SeiProcessingOrderSeiList list may consist of some or all of the following values: 3, 4, 5, 19, 45, 47, 137, 142, 144, 147, 148, 149, 150, 153, 154, 155, 156, 165, 177, 202, 204, 210, 211, 215, 216, 217, 218, 219.

[0257] 4. Specify additional constraints on the existence of SEI messages within PON SEI messages based on the existence of identifiers / target identifiers or persistences.

[0258] An example of PON SEI message semantics according to one embodiment is described. Semantics different from the previously described PON SEI message semantics are described, and descriptions of semantics identical to the previously described PON SEI message semantics are replaced with descriptions of the previously described PON SEI message semantics. Specifically, among the previously described PON SEI message semantics, semantics regarding bitstream conformity requirements are described, and descriptions of other semantics are replaced with descriptions of the previously described PON SEI message semantics.

[0259] The bitstream conformance requirement is that if a non-PON nested SEI message seiA exists, there must not be an associated SEI message seiB with the same payloadType in the same CLVS that is a PON nested SEI message, provided that one of the following conditions is true.

[0260] - No persistence information in seiA.

[0261] - seiA persists beyond the current picture and has no identifier or target identifier.

[0262] - seiA has an identifier or target identifier identical to the identifier or target identifier of seiB that persists beyond the current picture.

[0263] An example of PON SEI message semantics according to one embodiment is described. Semantics different from the previously described PON SEI message semantics are described, and descriptions of semantics identical to the previously described PON SEI message semantics are replaced with descriptions of the previously described PON SEI message semantics. Specifically, among the previously described PON SEI message semantics, semantics regarding bitstream conformity requirements are described, and descriptions of other semantics are replaced with descriptions of the previously described PON SEI message semantics.

[0264] Regarding the use of SEI processing order and nested processing order SEI messages, the following is specified for interpreting SEI processing order and nested processing order (PON) SEI messages.

[0265] - The SeiProcessingOrderSeiList list is configured to consist of payloadType values ​​3, 4, 5, 19, 137, 142, 144, 147, 148, 149, 150, 153, 155, 165, 177, 210, and 211.

[0266] - The SpoProcessSeiList list is configured to consist of payloadType values ​​19, 142, 155, 210, and 211.

[0267] - The PonSeiList list is set to consist of payloadType values ​​3, 4, 5, 19, 45, 47, 137, 142, 144, 147, 148, 149, 150, 153, 154, 155, 156, 165, 177, 202, 204, 210, 211, 215, 216, 217, 218, and 219.

[0268] - The syntax structure sei_pon_nested_message(), which is the container for SEI messages, is set to be identical to the syntax structure sei_message().

[0269] The bitstream conformance requirement is that if there is a SEI message seiA with a payloadType value in PonSeiList that is not a PON-nested SEI message, then there must not be an associated SEI message seiB in the same CLVS that is a PON-nested SEI message with the same payloadType, provided that one of the following conditions is true.

[0270] - No persistence information in seiA.

[0271] - seiA persists beyond the current picture and has no identifier or target identifier.

[0272] - seiA has an identifier or target identifier identical to the identifier or target identifier of seiB that persists beyond the current picture.

[0273] An example of PON SEI message semantics according to one embodiment is described. Semantics different from the previously described PON SEI message semantics are described, and descriptions of semantics identical to the previously described PON SEI message semantics are replaced with descriptions of the previously described PON SEI message semantics. Specifically, among the previously described PON SEI message semantics, semantics regarding bitstream conformity requirements are described, and descriptions of other semantics are replaced with descriptions of the previously described PON SEI message semantics.

[0274] Regarding the use of SEI processing order and nested processing order SEI messages, the following is specified for interpreting SEI processing order and nested processing order (PON) SEI messages.

[0275] - The SeiProcessingOrderSeiList list is configured to consist of payloadType values ​​3, 4, 5, 19, 45, 47, 137, 142, 144, 147, 148, 149, 150, 153, 154, 155, 156, 165, 177, 202, 204, 210, 211, 215, 216, 217, 218, and 219.

[0276] - The SpoProcessSeiList list is configured to consist of payloadType values ​​19, 142, 155, 210, and 211.

[0277] - The syntax structure sei_pon_nested_message(), which is the container for SEI messages, is set to be identical to the syntax structure sei_message().

[0278] The bitstream conformity requirement is that if there is a SEI message seiA in SeiProcessingOrderSeiList that is not a PON-nested SEI message with a payloadType value, then there must not be an associated SEI message seiB in the same CLVS that is a PON-nested SEI message with the same payloadType value, provided that one of the following conditions is true.

[0279] - No persistence information in seiA.

[0280] - seiA has persistence on more pictures than the current picture and has no identifier or target identifier.

[0281] - seiA has persistence on more pictures than the current picture, and has an identifier or target identifier identical to the identifier or target identifier of seiB.

[0282] An example of PON SEI message semantics according to one embodiment is described. Semantics different from the previously described PON SEI message semantics are described, and descriptions of semantics identical to the previously described PON SEI message semantics are replaced with descriptions of the previously described PON SEI message semantics. Specifically, among the previously described PON SEI message semantics, semantics regarding bitstream conformity requirements are described, and descriptions of other semantics are replaced with descriptions of the previously described PON SEI message semantics.

[0283] The bitstream conformity requirement is that the semantics and effects of SEI messages other than PON-nested SEI messages must not depend on PON-nested SEI messages. The result of this constraint includes the following specific constraints, in which case the associated SEI message is considered to be an SEI message that affects the semantics or effects of the specific SEI message.

[0284] - If a neural network post-filter characteristic SEI message is a PON-nested SEI message with a specific nnpfc_id value, the associated neural network post-filter activation SEI message with nnpfa_target_id that is the same as the nnpfc_id value must also be a PON-nested SEI message.

[0285] The bitstream conformity requirement states that if there exists an SEI message seiA of a specific payload type that is not a PON-nested SEI message, has a specific identifier or target identifier, and persists to more pictures than the current picture, and if there is an SEI message seiB in the next picture in output order within the same CLVS, and seiA and seiB have the same payload type and satisfy one or more of the following conditions, then seiB must not be a PON-nested SEI message.

[0286] - seiA and seiB have the same identifier or target identifier.

[0287] - seiB has a cancellation flag with a value equal to 1.

[0288] The bitstream conformity requirement is that if there is no identifier or target identifier other than a PON-nested SEI message, and if there is an SEI message that persists for more pictures than the current screen, then there must not be any associated SEI messages of the same payload type that are PON-nested SEI messages in the same CLVS.

[0289] An example of PON SEI message semantics according to one embodiment is described. Semantics different from the previously described PON SEI message semantics are described, and descriptions of semantics identical to the previously described PON SEI message semantics are replaced with descriptions of the previously described PON SEI message semantics. Specifically, among the previously described PON SEI message semantics, semantics regarding bitstream conformity requirements are described, and descriptions of other semantics are replaced with descriptions of the previously described PON SEI message semantics.

[0290] The bitstream conformity requirement is that the semantics and effects of SEI messages other than PON-nested SEI messages must not depend on PON-nested SEI messages. The result of this constraint includes the following specific constraints, in which case the associated SEI message is considered to be an SEI message that affects the semantics or effects of the specific SEI message.

[0291] - If a neural network post-filter characteristic SEI message is a PON-nested SEI message with a specific nnpfc_id value, the associated neural network post-filter activation SEI message with nnpfa_target_id that is the same as the nnpfc_id value must also be a PON-nested SEI message.

[0292] The bitstream conformity requirement is that if there exists an SEI message seiA of a specific payload type that is not a PON-nested SEI message, has a specific identifier or target identifier, and persists over more pictures than the current picture, and if there is an SEI message seiB in the next picture in output order within the same CLVS, and seiA and seiB have the same payload type and the same identifier or target identifier, then seiB must not be a PON-nested SEI message.

[0293] The bitstream conformance requirement is that if there is no identifier or target identifier that is not a PON-nested SEI message, and if there is an SEI message that persists for more pictures than the current picture, then no associated SEI message of the same payload type that is a PON-nested SEI message exists in the same CLVS.

[0294] An example of PON SEI message semantics according to one embodiment is described. Semantics different from the previously described PON SEI message semantics are described, and descriptions of semantics identical to the previously described PON SEI message semantics are replaced with descriptions of the previously described PON SEI message semantics. Specifically, among the previously described PON SEI message semantics, semantics regarding bitstream conformity requirements are described, and descriptions of other semantics are replaced with descriptions of the previously described PON SEI message semantics.

[0295] The bitstream conformity requirement is that the semantics and effects of SEI messages other than PON-nested SEI messages must not depend on PON-nested SEI messages. The result of this constraint includes the following specific constraints, in which case the associated SEI message is considered to be an SEI message that affects the semantics or effects of the specific SEI message.

[0296] - If a neural network post-filter characteristic SEI message is a PON-nested SEI message with a specific nnpfc_id value, the associated neural network post-filter activation SEI message with nnpfa_target_id that is the same as the nnpfc_id value must also be a PON-nested SEI message.

[0297] The bitstream conformance requirement is that if there is a SEI message seiA of a specific payload type with a persistence flag of 1 and a specific identifier or target identifier that is not a PON nested SEI message, then in the same CLVS, the next picture in the output order is a SEI message seiB, and if seiA and seiB have the same payload type and the same identifier or target identifier, then seiB must not be a PON nested SEI message.

[0298] The bitstream conformance requirement is that if there is a specific payload type SEI message with a persistence flag of 1 and no identifier or target identifier that is not a PON-nested SEI message, then there must not be any associated SEI message of the same payload type that is a PON-nested SEI message in the same CLVS.

[0299] An example of PON SEI message semantics according to one embodiment is described. Semantics different from the previously described PON SEI message semantics are described, and descriptions of semantics identical to the previously described PON SEI message semantics are replaced with descriptions of the previously described PON SEI message semantics. Specifically, among the previously described PON SEI message semantics, semantics regarding bitstream conformity requirements are described, and descriptions of other semantics are replaced with descriptions of the previously described PON SEI message semantics.

[0300] The bitstream conformity requirement is that the semantics and effects of SEI messages other than PON-nested SEI messages must not depend on PON-nested SEI messages. The result of this constraint includes the following specific constraints, in which case the associated SEI message is considered to be an SEI message that affects the semantics or effects of the specific SEI message.

[0301] - If a neural network post-filter characteristic SEI message is a PON-nested SEI message with a specific nnpfc_id value, the associated neural network post-filter activation SEI message with nnpfa_target_id that is the same as the nnpfc_id value must also be a PON-nested SEI message.

[0302] The bitstream conformance requirement is that if there is a SEI message seiA of a specific payload type with a persistence flag of 1 and a specific identifier or target identifier that is not a PON nested SEI message, then in the same CLVS, the next picture in the output order is a SEI message seiB, and if seiA and seiB have the same payload type and the same identifier or target identifier, then seiB must not be a PON nested SEI message.

[0303] The bitstream conformance requirement is that if there is a specific payload type SEI message with a persistence flag of 1 and no identifier or target identifier that is not a PON-nested SEI message, then there must not be any associated SEI message of the same payload type that is a PON-nested SEI message in the same CLVS.

[0304] The bitstream conformance requirement is that if there exists an SEI message that is not a PON-nested SEI message, has no identifier or target identifier, and lacks a persistence flag, then there must not be any associated SEI message of the same payload type that is a PON-nested SEI message in the same CLVS.

[0305] The terms or names described below (e.g., names of syntax elements or variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms, etc. described below. For example, the image information described below may include various information according to the embodiments described in the present disclosure and may include information described in at least one of the tables described above.

[0306] The operations described below do not constitute an essential component of one embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not constitute a sufficient component of one embodiment, and previously described operations may be added. Moreover, unless they contradict previously described operations, the operations described below form one embodiment integrally with previously described operations and do not form a separate embodiment distinct from previously described operations.

[0307] FIG. 5 is a diagram illustrating a method for decoding image information according to one embodiment of the present disclosure.

[0308] Terms or names (e.g., names of syntax elements or names of variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms, etc. described in FIG. 5. For example, the image information described in FIG. 5 may include various information according to the embodiments described in the present disclosure and may include information described in at least one of the tables described above.

[0309] The decoding method (S500) may include operations described below. The operations described below do not constitute an essential component of the decoding method according to one embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not constitute a sufficient component of the decoding method according to one embodiment, and the previously described operations may be added. Moreover, unless the operations described below contradict the previously described operations, they form an embodiment integrally with the previously described operations and do not form a separate embodiment distinguished from the previously described operations.

[0310] The decoding method (S500) can be executed by a decoding device including a memory and a processor electrically connected to the memory, for example, by a processor.

[0311] The decoding device can acquire image information (S510).

[0312] For example, a processor of a decoding device may acquire image information. The image information may include SEI (supplemental enhancement information) messages. SEI messages may convey specific types of information that assist in processes related to the decoding, display, or other purposes of the image information. Here, SEI messages may not be necessary for the decoding process to determine the sample values ​​of the decoded picture.

[0313] For example, an SEI message may include an SEI processing order (SPO) SEI message and a processing order nesting (PON) SEI message.

[0314] The SPO SEI message can provide information regarding the processing order of at least one other SEI message included in the image information (e.g., at least one SEI message other than the SPO SEI message and a PON nested SEI message included in the PON SEI message).

[0315] At least one SEI message other than the SPO SEI message and the netted SEI message included in the PON SEI message may have various payload types. At least one SEI message and the netted SEI message may include SEI messages having different payload types and / or SEI messages having the same payload type.

[0316] The SPO SEI message may have various names, such as SEI processing order SEI message, SEI processing order related message, SEI processing order related information, processing order SEI message, processing order message, processing order related message, processing order related information, SPO message, SPO related message, and SPO related information, and such names are not limited.

[0317] SPO SEI messages can take various forms. For example, an SPO SEI message may be a syntax element or a syntax structure containing one or more syntax elements. Additionally, an SPO SEI message may be a raw byte sequence payload (RBSP) containing one or more syntax elements or one or more syntax structures. For example, an SPO SEI message may be represented as sei_processing_order( payloadSize ), but is not limited thereto.

[0318] The SPO SEI message may include identifier information of the SPO SEI message, human viewing information of the SPO SEI message, machine analysis information of the SPO SEI message, width priority information of the SPO SEI message, message count information of at least one SEI message, wrapping information of at least one SEI message, importance information of at least one SEI message, processing level information of at least one SEI message, payload type information of at least one SEI message, prefix information of at least one SEI message, and processing order information of at least one SEI message.

[0319] Identifier information may include an identifier for identifying SPO SEI messages. Image information may include multiple SPO SEI messages. Each of the multiple SPO SEI messages may include an identifier for identifying SPO SEI messages.

[0320] Identifier information may take various forms and may be represented by various names. For example, identifier information may be a syntax element or a syntax structure containing one or more syntax elements. For example, identifier information that is a syntax element may be an 8-bit integer. Identifier information that is a syntax element may be represented as po_id, etc., but is not limited thereto.

[0321] Human viewing information can indicate whether the intended use of the image generated in the processing chain by the SPO SEI message involves human viewing. In other words, human viewing information can indicate whether the image generated in the processing chain by the SPO SEI message is optimized for, suitable for, unsuitable for, or unknown for human viewing.

[0322] Human viewing information may take various forms and may be expressed by various names. For example, human viewing information may be a syntax element or a syntax structure containing one or more syntax elements. For example, human viewing information that is a syntax element may be a one-bit flag or an indicator with two or more bits. Human viewing information that is a syntax element may be expressed as po_for_human_viewing_idc or po_for_human_viewing_flag, but is not limited thereto.

[0323] Machine analysis information can indicate whether the intended use of the image generated in the processing chain by the SPO SEI message involves machine analysis. In other words, machine analysis information can indicate whether the image generated in the processing chain by the SPO SEI message is optimized for, suitable for, unsuitable for, or unknown for machine analysis.

[0324] Machine analysis information may take various forms and may be represented by various names. For example, machine analysis information may be a syntax element or a syntax structure containing one or more syntax elements. For example, machine analysis information that is a syntax element may be a one-bit flag or an indicator with two or more bits. Machine analysis information that is a syntax element may be represented as po_for_machine_analysis_idc or po_for_machine_analysis_flag, but is not limited thereto.

[0325] Breadth-first information may indicate whether breadth-first handling or depth-first handling is applied to the processing chain by the SPO SEI message. For example, based on breadth-first information with a value of 1, breadth-first handling may be fixedly applied to the processing chain by the SPO SEI message. Additionally, based on breadth-first information with a value of 0, breadth-first handling or depth-first handling may be selectively applied to the processing chain by the SPO SEI message. However, this is not limited thereto, and what is indicated by breadth-first information with a value of 1 may be interchangeable with what is indicated by breadth-first information with a value of 0.

[0326] Breadth-first information may take various forms and may be represented by various names. For example, breadth-first information may be a syntax element or a syntax structure containing one or more syntax elements. For example, breadth-first information that is a syntax element may be a one-bit flag or an indicator with two or more bits. Breadth-first information that is a syntax element may be represented as po_breadth_first_flag or po_breadth_first_idc, but is not limited thereto.

[0327] Message count information may indicate the number of payload types of at least one SEI message whose processing order is specified by the SPO SEI message. Based on the value of the message count information, the number of wrapping information, importance information, processing level information, payload type information, prefix information, and processing order information included in the SPO SEI message may be specified.

[0328] Message count information may take various forms and may be represented by various names. For example, message count information may be a syntax element or a syntax structure containing one or more syntax elements. For example, message count information that is a syntax element may be a 7-bit integer. Message count information that is a syntax element may be represented as po_num_sei_messages_minus2 or po_num_sei_msg_types_minus1, but is not limited thereto.

[0329] Wrapping information may indicate whether the SEI message whose processing order is specified by the SPO SEI message is a general SEI message or a PON nested SEI message included in the processing order PON SEI message. For example, wrapping information with a value of 1 may indicate that the SEI message whose processing order is specified by the SPO SEI message is a PON nested SEI message included in the processing order PON SEI message. Additionally, wrapping information with a value of 0 may indicate that the SEI message whose processing order is specified by the SPO SEI message is a general SEI message. However, this is not limited thereto, and what is indicated by wrapping information with a value of 1 may be interchangeable with what is indicated by wrapping information with a value of 0.

[0330] Wrapping information may take various forms and may be represented by various names. For example, wrapping information may be a syntax element or a syntax structure containing one or more syntax elements. For example, wrapping information that is a syntax element may be a one-bit flag or an indicator with two or more bits. Wrapping information that is a syntax element may be represented as po_sei_wrapping_flag[ i ] or po_sei_wrapping_idc[ i ], but is not limited thereto.

[0331] Importance information and processing level information may influence the derivation of a list of SEI messages that the decoding device must process for the current picture. For example, if the decoding device cannot process an SEI message associated with importance information with a value of 0, the payload type of the said SEI message may be excluded from the processing chain by the SPO SEI message. Importance information with a value of 1 and processing level information with a value of 0 may indicate the termination of deriving a list of SEI messages that the decoding device must process. If the decoding device cannot process an SEI message associated with importance information with a value of 0, processing by the processing chain by the SPO SEI message may not be performed for the current picture. However, this is not limited to the following, and what is indicated by importance information with a value of 1 and processing level information with a value of 1 may be interchangeable with what is indicated by importance information with a value of 0 and processing level information with a value of 0.

[0332] Importance information and processing degree information may each take various forms and be represented by various names. For example, importance information and processing degree information may each be a syntax element or a syntax structure containing one or more syntax elements. For example, the importance information and processing degree information that are syntax elements may be a one-bit flag or an indicator with two or more bits. The importance information that is a syntax element may be represented as po_sei_importance_flag[ i ] or po_sei_importance_idc[ i ], and the processing degree information that is a syntax element may be represented as po_sei_processing_degree_flag[ i ] or po_sei_processing_degree_idc[ i ], but is not limited thereto. Additionally, the importance information and processing degree information may be replaced as a single unit with an indicator of two or more bits.

[0333] Payload type information can indicate the payload type of at least one SEI message or at least one PON-nested SEI message. In other words, payload type information can specify the payloadType value of the SEI message.

[0334] Payload type information can take various forms and can be represented by various names. For example, payload type information can be a syntax element or a syntax structure containing one or more syntax elements. For example, payload type information that is a syntax element can be a 12-bit integer. Payload type information that is a syntax element can be represented as po_sei_payload_type[ i ], but is not limited thereto.

[0335] Prefix information may indicate whether an SPO SEI message contains prefix data for at least one SEI message. For example, prefix information with a value of 1 may indicate that an SPO SEI message contains prefix data for at least one SEI message. Additionally, prefix information with a value of 0 may indicate that an SPO SEI message does not contain prefix data for at least one SEI message. However, this is not limited thereto, and what prefix information with a value of 1 indicates may be interchangeable with what prefix information with a value of 0 indicates.

[0336] Prefix information can take various forms and can be represented by various names. For example, prefix information may be a syntax element or a syntax structure containing one or more syntax elements. For example, prefix information that is a syntax element may be a one-bit flag or an indicator with two or more bits. Prefix information that is a syntax element may be represented as po_sei_prefix_flag[ i ] or po_sei_prefix_idc[ i ], but is not limited thereto.

[0337] Processing order information may indicate the processing order for at least one payload type of SEI message. The smaller the value of the processing order information for a specific payload type, the earlier the SEI message of that payload type may be processed. In other words, if the value of the processing order information for a first payload type is smaller than the value of the processing order information for a second payload type, the SEI message of the first payload type may be processed before the SEI message of the second payload type.

[0338] Processing order information can take various forms and can be represented by various names. For example, processing order information can be a syntax element or a syntax structure containing one or more syntax elements. For example, processing order information that is a syntax element can be an 8-bit integer. Processing order information that is a syntax element can be represented as po_sei_processing_order[ i ], but is not limited thereto.

[0339] The PON SEI message may include at least one PON nested SEI message in which information regarding the processing order is provided by the SPO SEI message.

[0340] The PON SEI message may have various names, such as Processing order nesting SEI message, Processing order nesting related message, Processing order nesting related information, Processing order nesting message, Processing order nesting information, PON SEI message, PON related message, PON related information, PON message, PON information, Nesting SEI message, Nesting related message, Nesting related information, Nesting message, Nesting information, etc., and such names are not limited.

[0341] PON processing order information can take various forms and can be represented by various names. For example, PON processing order information can be a syntax element or a syntax structure containing one or more syntax elements. For example, PON processing order information that is a syntax element can be an 8-bit integer. PON processing order information that is a syntax element can be represented as pon_processing_order[ i ], but is not limited thereto.

[0342] A PON nested SEI message may include target identifier information, PON processing order information, and a PON nested SEI message.

[0343] Target identifier information may include the identifier of an SPO SEI message associated with a PON-nested SEI message. A PON-nested SEI message may be associated with multiple SPO SEI messages, and the target identifier information may include multiple target identifiers.

[0344] Target identifier information may take various forms and may be represented by various names. For example, target identifier information may be a syntax element or a syntax structure containing one or more syntax elements. For example, target identifier information that is a syntax element may be an 8-bit integer. Target identifier information that is a syntax element may be represented as pon_target_po_id[ i ], but is not limited thereto.

[0345] PON processing order information can indicate the processing order for payload types of PON-nested SEI messages. For example, PON processing order information can indicate the position of PON-nested SEI messages within the processing order defined by the associated SPO SEI message.

[0346] PON nested SEI messages are merely SEI messages contained within PON SEI messages, and like general SEI messages, they can convey specific types of information that assist in processes related to the decoding, display, or other purposes of image information. If a specific SEI message is not defined in some of the multiple decoding devices, that specific SEI message may be included in the PON SEI message so that undefined SEI messages are ignored by the decoding devices.

[0347] PON-nested SEI messages are contained within PON SEI messages and are strictly separated from regular SEI messages. In other words, there is no association or dependency between PON-nested SEI messages and regular SEI messages.

[0348] For example, the semantics and effects of an SEI message that is not a PON-nested SEI message must not depend on the PON-nested SEI message. For instance, if a neural network post-filter feature SEI message is a PON-nested SEI message with a specific identifier value, then a neural network post-filter activation SEI message with target identifier information identical to that specific identifier value must also be a PON-nested SEI message.

[0349] If an SEI message of a specific payload type is not a PON-nested SEI message, then an SEI message of the same payload type as said SEI message must not be a PON-nested SEI message. If an SEI message of a specific payload type is not a PON-nested SEI message, then an SEI message of the same payload type as said SEI message must not be a PON-nested SEI message based on the identifier information and / or persistence information of said SEI message.

[0350] If a SEI message of a specific payload type having a specific identifier or target identifier and persistence for more pictures than the current picture is not a PON-nested SEI message, then a SEI message of the same payload type having the same specific identifier or target identifier as said SEI message must not be a PON-nested SEI message. Additionally, if a SEI message of a specific payload type having a specific identifier or target identifier and persistence information with a value of 1 is not a PON-nested SEI message, then a SEI message of the same payload type having the same specific identifier or target identifier as said SEI message must not be a PON-nested SEI message.

[0351] If a SEI message of a specific payload type that does not have identifier information and has persistence for more pictures than the current picture is not a PON-nested SEI message, then a SEI message of the same payload type as said SEI message must not be a PON-nested SEI message. Additionally, if a SEI message of a specific payload type that does not have identifier information and has persistence information such as a value of 1 is not a PON-nested SEI message, then a SEI message of the same payload type as said SEI message must not be a PON-nested SEI message.

[0352] If an SEI message of a specific payload type that does not have identifier information and / or persistence information is not a PON-nested SEI message, then an SEI message of the same payload type as said SEI message must not be a PON-nested SEI message.

[0353] In this way, by utilizing the identifier information and / or persistence information of SEI messages, it is possible to ensure that no association or dependency exists between PON-nested SEI messages and general SEI messages.

[0354] In other words, based on the identifier information and / or persistence information of a first SEI message that is not a PON-nested SEI message, it can be derived that a second SEI message having the same payload type as the first SEI message is not a PON-nested SEI message.

[0355] Based on the fact that a first SEI message, which is not a PON-nested SEI message, persists beyond the current picture and does not contain identifier information, it can be derived that a second SEI message having the same payload type as the first SEI message is not a PON-nested SEI message. Additionally, based on the fact that a first SEI message, which is not a PON-nested SEI message, has persistence information with a value of 1 and does not contain identifier information, it can be derived that a second SEI message having the same payload type as the first SEI message is not a PON-nested SEI message.

[0356] Based on the fact that a first SEI message, which is not a PON-nested SEI message, persists beyond the current picture and contains identifier information, it can be derived that a second SEI message, which has the same identifier information as the first SEI message and the same payload type as the first SEI message, is not a PON-nested SEI message. Additionally, based on the fact that a first SEI message, which is not a PON-nested SEI message, has persistence information with a value of 1 and contains identifier information, it can be derived that a second SEI message, which has the same identifier information as the first SEI message and the same payload type as the first SEI message, is not a PON-nested SEI message.

[0357] Based on the fact that it does not contain persistence information and identifier information that is not a PON-nested SEI message, it can be derived that a second SEI message having the same payload type as the first SEI message is not a PON-nested SEI message.

[0358] In this way, by using identifier information and / or persistence information of SEI messages, there is no association or dependency between PON-nested SEI messages and general SEI messages, thereby ensuring independence between PON-nested SEI messages and general SEI messages.

[0359] The decoding device can derive the processing order for the SEI message (S520).

[0360] For example, the processor of the decoding device can derive a processing order for the payload type of the SEI message based on the SPO SEI message, and furthermore, can derive a processing order for the SEI message and / or PON-nested SEI message.

[0361] The decoding device can process PON-nested SEI messages and / or non-PON-nested SEI messages for the current picture according to the derived processing order.

[0362] As previously explained, by using identifier information and / or persistence information of SEI messages, there is no association or dependency between PON-nested SEI messages and general SEI messages, thereby ensuring independence between PON-nested SEI messages and general SEI messages.

[0363] As a result, malfunctions in some decoding devices caused by association or dependency between PON-nested SEI messages and general SEI messages can be suppressed, prevented, or minimized.

[0364] As a result, the reliability of a coding system including an encoding device and a decoding device can be improved.

[0365] In addition, the coding efficiency of a coding system including an encoding device and a decoding device can be improved.

[0366] In addition, the data transmission efficiency of a coding system including an encoding device and a decoding device can be improved.

[0367] FIG. 6 is a diagram illustrating a method for encoding image information according to one embodiment of the present disclosure.

[0368] The terms or names described in FIG. 6 (e.g., names of syntax elements or variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms, etc. described in FIG. 6. For example, the image information described in FIG. 6 may include various information according to the embodiments described in the present disclosure and may include information described in at least one of the tables described above.

[0369] The encoding method (S600) may include operations described below. The operations described below do not constitute an essential component of the decoding method according to one embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not constitute a sufficient component of the encoding method according to one embodiment, and the previously described operations may be added. Moreover, unless the operations described below contradict the previously described operations, they form an embodiment integrally with the previously described operations and do not form a separate embodiment distinct from the previously described operations.

[0370] The encoding method (S600) can be executed by an encoding device including a memory and a processor electrically connected to the memory, for example, by a processor.

[0371] The encoding device can determine the processing order of SEI (supplemental enhancement information) messages (S610).

[0372] For example, a processor of an encoding device may generate an SEI message. The SEI message may convey a specific type of information that assists in processes related to the decoding, display, or other purposes of image information. Here, the SEI message may not be necessary for the decoding process to determine the sample values ​​of the decoded picture.

[0373] SEI messages can have various payload types. For example, an SEI message may include SEI messages having different payload types and / or SEI messages having the same payload type.

[0374] Additionally, SEI messages may include general SEI messages and PON-nested SEI messages. PON-nested SEI messages may refer to SEI messages included in processing order nesting (PON) SEI messages. General SEI messages may refer to SEI messages that are not PON-nested SEI messages.

[0375] The processor of the encoding device can determine the processing order for SEI messages including PON-nested SEI messages and SEI messages that are not PON-nested SEI messages. Specifically, the processor of the encoding device can determine the processing order for the payload types of SEI messages.

[0376] The encoding device can encode video information including SPO SEI messages and PON SEI messages (S620).

[0377] For example, the processor of the encoding device can generate an SPO SEI message based on the processing order of the payload types of the SEI message. Additionally, the processor of the encoding device can generate a PON SEI message based on a PON nested SEI message.

[0378] The SPO SEI message can provide information regarding the processing order of at least one SEI message (e.g., other SEI messages other than the SPO SEI message and the processing order PON SEI message) including the PON SEI message and a SEI message that is not a PON SEI message.

[0379] The SPO SEI message may have various names, such as SEI processing order SEI message, SEI processing order related message, SEI processing order related information, processing order SEI message, processing order message, processing order related message, processing order related information, SPO message, SPO related message, and SPO related information, and such names are not limited.

[0380] SPO SEI messages can take various forms. For example, an SPO SEI message may be a syntax element or a syntax structure containing one or more syntax elements. Additionally, an SPO SEI message may be a raw byte sequence payload (RBSP) containing one or more syntax elements or one or more syntax structures. For example, an SPO SEI message may be represented as sei_processing_order( payloadSize ), but is not limited thereto.

[0381] The SPO SEI message may include identifier information of the SPO SEI message, human viewing information of the SPO SEI message, machine analysis information of the SPO SEI message, width priority information of the SPO SEI message, message count information of at least one SEI message, wrapping information of at least one SEI message, importance information of at least one SEI message, processing level information of at least one SEI message, payload type information of at least one SEI message, prefix information of at least one SEI message, and processing order information of at least one SEI message.

[0382] The identifier information, human viewing information, machine analysis information, width priority information, message count information, wrapping information, importance information, processing level information, payload type information, prefix information, and processing order information included in the SPO SEI message may be the same as the identifier information, human viewing information, machine analysis information, width priority information, message count information, wrapping information, importance information, processing level information, payload type information, prefix information, and processing order information described above in operation 510 of FIG. 5.

[0383] The description of the identifier information, human viewing information, machine analysis information, width priority information, message count information, wrapping information, importance information, processing level information, payload type information, prefix information, and processing order information included in the SPO SEI message is replaced with the description of the identifier information, human viewing information, machine analysis information, width priority information, message count information, wrapping information, importance information, processing level information, payload type information, prefix information, and processing order information described above in operation 510 of FIG. 5.

[0384] The PON SEI message may include at least one PON nested SEI message in which information regarding the processing order is provided by the SPO SEI message.

[0385] The PON SEI message may have various names, such as Processing order nesting SEI message, Processing order nesting related message, Processing order nesting related information, Processing order nesting message, Processing order nesting information, PON SEI message, PON related message, PON related information, PON message, PON information, Nesting SEI message, Nesting related message, Nesting related information, Nesting message, Nesting information, etc., and such names are not limited.

[0386] PON processing order information can take various forms and can be represented by various names. For example, PON processing order information can be a syntax element or a syntax structure containing one or more syntax elements. For example, PON processing order information that is a syntax element can be an 8-bit integer. PON processing order information that is a syntax element can be represented as pon_processing_order[ i ], but is not limited thereto.

[0387] A PON nested SEI message may include target identifier information, PON processing order information, and a PON nested SEI message.

[0388] The target identifier information, PON processing order information, and PON nested SEI message may be the same as the target identifier information, PON processing order information, and PON nested SEI message described in operation 510 of FIG. 5.

[0389] The description of the target identifier information, PON processing order information, and PON nested SEI message is replaced with the description of the target identifier information, PON processing order information, and PON nested SEI message described in operation 510 of FIG. 5.

[0390] PON-nested SEI messages are contained within PON SEI messages and are strictly separated from regular SEI messages. In other words, there is no association or dependency between PON-nested SEI messages and regular SEI messages.

[0391] For example, the semantics and effects of an SEI message that is not a PON-nested SEI message must not depend on the PON-nested SEI message. For instance, if a neural network post-filter feature SEI message is a PON-nested SEI message with a specific identifier value, then a neural network post-filter activation SEI message with target identifier information identical to that specific identifier value must also be a PON-nested SEI message.

[0392] If an SEI message of a specific payload type is not a PON-nested SEI message, then an SEI message of the same payload type as said SEI message must not be a PON-nested SEI message. If an SEI message of a specific payload type is not a PON-nested SEI message, then an SEI message of the same payload type as said SEI message must not be a PON-nested SEI message based on the identifier information and / or persistence information of said SEI message.

[0393] If a SEI message of a specific payload type having a specific identifier or target identifier and persistence for more pictures than the current picture is not a PON-nested SEI message, then a SEI message of the same payload type having the same specific identifier or target identifier as said SEI message must not be a PON-nested SEI message. Additionally, if a SEI message of a specific payload type having a specific identifier or target identifier and persistence information with a value of 1 is not a PON-nested SEI message, then a SEI message of the same payload type having the same specific identifier or target identifier as said SEI message must not be a PON-nested SEI message.

[0394] If a SEI message of a specific payload type that does not have identifier information and has persistence for more pictures than the current picture is not a PON-nested SEI message, then a SEI message of the same payload type as said SEI message must not be a PON-nested SEI message. Additionally, if a SEI message of a specific payload type that does not have identifier information and has persistence information such as a value of 1 is not a PON-nested SEI message, then a SEI message of the same payload type as said SEI message must not be a PON-nested SEI message.

[0395] If an SEI message of a specific payload type that does not have identifier information and / or persistence information is not a PON-nested SEI message, then an SEI message of the same payload type as said SEI message must not be a PON-nested SEI message.

[0396] In this way, by utilizing the identifier information and / or persistence information of SEI messages, it is possible to ensure that no association or dependency exists between PON-nested SEI messages and general SEI messages.

[0397] In other words, based on the identifier information and / or persistence information of the first SEI message that is not a PON-nested SEI message, a second SEI message having the same payload type as the first SEI message can be generated as not being a PON-nested SEI message.

[0398] Based on the fact that a first SEI message, which is not a PON-nested SEI message, persists beyond the current picture and does not contain identifier information, it can be derived that a second SEI message having the same payload type as the first SEI message is not a PON-nested SEI message. Additionally, based on the fact that a first SEI message, which is not a PON-nested SEI message, has persistence information with a value of 1 and does not contain identifier information, a second SEI message having the same payload type as the first SEI message can be generated as not a PON-nested SEI message.

[0399] Based on the fact that a first SEI message, which is not a PON-nested SEI message, persists beyond the current picture and contains identifier information, a second SEI message having the same identifier information as the first SEI message and the same payload type as the first SEI message can be derived as not being a PON-nested SEI message. Additionally, based on the fact that a first SEI message, which is not a PON-nested SEI message, has persistence information with a value of 1 and contains identifier information, a second SEI message having the same identifier information as the first SEI message and the same payload type as the first SEI message can be generated as not being a PON-nested SEI message.

[0400] Based on the fact that it does not contain persistence information and identifier information that is not a PON-nested SEI message, a second SEI message having the same payload type as the first SEI message can be generated as not being a PON-nested SEI message.

[0401] In this way, by using identifier information and / or persistence information of SEI messages, there is no association or dependency between PON-nested SEI messages and general SEI messages, thereby ensuring independence between PON-nested SEI messages and general SEI messages.

[0402] As a result, malfunctions in some decoding devices caused by association or dependency between PON-nested SEI messages and general SEI messages can be suppressed, prevented, or minimized.

[0403] As a result, the reliability of a coding system including an encoding device and a decoding device can be improved.

[0404] In addition, the coding efficiency of a coding system including an encoding device and a decoding device can be improved.

[0405] In addition, the data transmission efficiency of a coding system including an encoding device and a decoding device can be improved.

[0406] A bitstream is generated based on video information encoded according to the encoding method (S600) described above, and the bitstream can be stored on a computer-readable storage medium.

[0407] In addition, a bitstream is generated based on video information encoded according to the encoding method (S600) described above, and the bitstream can be transmitted through a transmission unit and / or a transmission medium.

[0408] FIG. 7 is a drawing illustrating an exemplary content streaming system to which an embodiment according to the present disclosure can be applied.

[0409] As illustrated in FIG. 7, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0410] The above encoding server compresses content input from multimedia input devices, such as smartphones, cameras, and camcorders, into digital data to generate a bitstream and transmits it to the streaming server. As another example, if multimedia input devices, such as smartphones, cameras, and camcorders, generate the bitstream directly, the encoding server may be omitted.

[0411] The bitstream may be generated by a video encoding method and / or encoding device to which an embodiment of the present disclosure is applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0412] The streaming server transmits multimedia data to a user device based on a user request through a web server, and the web server can act as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server can perform the role of controlling commands and responses between each device within the content streaming system.

[0413] The streaming server can receive content from a media storage and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a seamless streaming service, the streaming server can store the bitstream for a certain period of time.

[0414] Examples of the above user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs (head-mounted displays)), digital TVs, desktop computers, digital signage, etc.

[0415] Each server within the above-mentioned content streaming system can be operated as a distributed server, and in this case, data received from each server can be processed in a distributed manner.

[0416] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating system, application, firmware, program, etc.) that enable an operation according to a method of various embodiments to be executed on a device or computer, and a non-transitory computer-readable medium on which such software or instructions, etc. are stored and executable on a device or computer.

[0417] An embodiment according to the present disclosure can be used to encode / decode images.

Claims

1. Regarding the method of decoding video information, Acquiring the image information including at least one SEI (supplemental enhancement information) message, an SPO (SEI processing order) SEI message, and a PON (processing order nesting) SEI message; Deriving a processing order for the PON nested SEI message of the PON SEI message and the at least one SEI message that is not the PON nested SEI message based on the above SPO SEI message, wherein A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a method that is not the PON-nested SEI message.

2. In Paragraph 1, A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a method that is not the PON-nested SEI message based on the persistence information or identifier information of the first SEI message.

3. In Paragraph 2, A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a method that is not the PON-nested SEI message, based on the fact that the first SEI message continues beyond the current picture and does not contain identifier information.

4. In Paragraph 2, A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a method that is not the PON-nested SEI message, based on the first SEI message continuing beyond the current picture and containing the same identifier information as the second SEI message.

5. In Paragraph 2, A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a method that is not the PON-nested SEI message, based on the fact that the first SEI message has persistence information such as a value of 1 and does not include identifier information.

6. In Paragraph 2, A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is based on the fact that the first SEI message has persistence information with a value of 1 and includes identifier information identical to the second SEI message, wherein the PON-nested SEI message is a method.

7. In Paragraph 2, A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a method that is not the PON-nested SEI message, based on the fact that the first SEI message does not include identifier information and persistence information.

8. Regarding the method of encoding video information, Determining the processing order for PON (processing order nesting) nested SEI (supplemental enhancement information) messages and at least one SEI message that is not a PON nested SEI message; Encoding the image information, which includes a PON SEI message generated based on the above PON nested SEI message and an SPO (SEI processing order) SEI message generated based on the above processing order. A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a method that is not the PON-nested SEI message.

9. In Paragraph 8, A second SEI message having the same payload type as the first SEI message that is not the PON-nested SEI message is a method that is not the PON-nested SEI message, based on the identifier information and / or persistence information of the first SEI message.

10. In Paragraph 9, A second SEI message having the same payload type as the first SEI message is not a PON nested SEI message, based on the fact that the first SEI message persists beyond the current picture and does not contain identifier information.

11. In Paragraph 9, A second SEI message having the same payload type as the first SEI message is not a PON nested SEI message, based on the fact that the first SEI message persists beyond the current picture and includes the same identifier information as the second SEI message.

12. In Paragraph 9 A second SEI message having the same payload type as the first SEI message is not a PON-nested SEI message, based on the fact that the first SEI message has persistence information such as a value of 1 and does not include identifier information.

13. In Paragraph 9, A second SEI message having the same payload type as the first SEI message is not a PON-nested SEI message, based on the fact that the first SEI message has persistence information with a value of 1 and includes identifier information identical to the second SEI message.

14. In Paragraph 9, A second SEI message having the same payload type as the first SEI message is not a PON-nested SEI message, based on the fact that the first SEI message does not include identifier information and persistence information.

15. Regarding methods concerning bitstreams, Determining the processing order for PON (processing order nesting) nested SEI (supplemental enhancement information) messages and at least one SEI message that is not a PON nested SEI message; Generating a bitstream based on image information, comprising a PON (SEI processing order) SEI message generated based on the above PON nested SEI message and an SPO SEI message generated based on the above processing order; It includes transmitting data regarding the above bitstream, and A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a method that is not the PON-nested SEI message.

16. In a computer-readable storage medium for storing a bitstream, The above storage medium stores the bitstream generated based on image information, comprising a PON SEI message generated based on a PON (processing order nesting) nested SEI (supplemental enhancement information) message and an SPO (SEI processing order) SEI message generated based on a processing order for at least one SEI message other than the PON nested SEI message. A second SEI message having the same payload type as the first SEI message, which is not the PON-nested SEI message, is a storage medium that is not the PON-nested SEI message.

Images