Image encoding / decoding methods, and bitstream transmission methods.

Abstract

The invention relates to a method and apparatus for encoding and decoding images. The method for decoding images performed by the apparatus for decoding images under this invention may include the following steps: obtaining, from the bit stream, the maximum time identifier information indicating the maximum time identifier of the reference class to which the interclass prediction of the current class is referenced; and performing, based on the maximum time identifier information, a procedure for extracting a sub-bit stream from the bit stream, such that, in the procedure for extracting the sub-bit stream, among the parameter sets of the reference class, the parameter set not referenced for the interclass prediction of the current class can be removed from the bit stream.

Description

A video encoding / decoding method and device for performing a sub-bitstream extraction process based on a maximum time identifier, and a computer-readable recording medium storing a bitstream

[0001] The present disclosure relates to a video encoding / decoding method and device, and more particularly, to a video encoding / decoding method and device capable of efficiently removing a non-referenced parameter set from a parameter set of a reference layer based on a time identifier, and a method for transmitting a bitstream generated by the video encoding method / device of the present disclosure.

[0002] Recently, demand for high-resolution, high-quality images, such as HD (High Definition) and UHD (Ultra High Definition) images, has been increasing across various fields. As image data becomes higher resolution and higher quality, the amount of information transmitted, or bits, increases relative to conventional image data. This increase in information or bits transmitted leads to increased transmission and storage costs.

[0003] Accordingly, a highly efficient image compression technology is required to effectively transmit, store, and play high-resolution, high-quality image information.

[0004] The present disclosure aims to provide a video encoding / decoding method and device with improved encoding / decoding efficiency.

[0005] In addition, the present disclosure aims to provide an image encoding / decoding method and device capable of efficiently removing an unreferenced parameter set from among a parameter set of a reference layer based on a time identifier.

[0006] In addition, the present disclosure aims to provide a method for transmitting a bitstream generated by an image encoding method or device according to the present disclosure.

[0007] In addition, the present disclosure aims to provide a recording medium storing a bitstream generated by an image encoding method or device according to the present disclosure.

[0008] In addition, the present disclosure aims to provide a recording medium storing a bitstream received and decoded by an image decoding device according to the present disclosure and used for image restoration. For example, the recording medium may store a bitstream that causes the decoding device according to the present disclosure to perform an image decoding method according to the present disclosure.

[0009] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0010] An image decoding method performed by an image decoding device according to one aspect of the present disclosure includes the steps of obtaining maximum time identifier information indicating a maximum time identifier of a reference layer referenced for inter-layer prediction of a current layer from a bitstream, and performing a sub-bitstream extraction process from the bitstream based on the maximum time identifier information, wherein, in the sub-bitstream extraction process, among parameter sets of the reference layer, a parameter set that is not referenced for inter-layer prediction of the current layer can be removed from the bitstream.

[0011] In the image decoding method of the present disclosure, the parameter set may be an Adaptive Parameter Set (APS).

[0012] In the image decoding method of the present disclosure, a parameter set having a time identifier greater than the maximum time identifier among the parameter sets of the reference layer may not be referenced for inter-layer prediction of the current layer.

[0013] In the video decoding method of the present disclosure, a picture having a time identifier greater than the maximum time identifier among the pictures of the reference layer may not be referenced for inter-layer prediction of the current layer.

[0014] In the image decoding method of the present disclosure, the reference layer may be a non-output layer.

[0015] According to another aspect of the present disclosure, an image decoding device includes a memory and at least one processor, wherein the at least one processor obtains maximum temporal identifier information indicating a maximum temporal identifier of a reference layer referenced for inter-layer prediction of a current layer from a bitstream, and performs a sub-bitstream extraction process from the bitstream based on the maximum temporal identifier information, and in the sub-bitstream extraction process, among parameter sets of the reference layer, a parameter set that is not referenced for inter-layer prediction of the current layer may be removed from the bitstream.

[0016] According to another aspect of the present disclosure, a video encoding method performed by a video encoding device includes the steps of determining maximum temporal identifier information indicating a maximum temporal identifier of a reference layer referenced for inter-layer prediction of a current layer, and encoding the maximum temporal identifier information into a bitstream, wherein the maximum temporal identifier information is used to perform a sub-bitstream extraction process from the bitstream, and in the sub-bitstream extraction process, a parameter set not referenced for inter-layer prediction of the current layer among parameter sets of the reference layer can be removed from the bitstream.

[0017] In the image encoding method of the present disclosure, the parameter set may be an Adaptive Parameter Set (APS).

[0018] In the image encoding method of the present disclosure, a parameter set having a time identifier greater than the maximum time identifier among the parameter sets of the reference layer may not be referenced for inter-layer prediction of the current layer.

[0019] In the video encoding method of the present disclosure, a picture having a time identifier greater than the maximum time identifier among the pictures of the reference layer may not be referenced for inter-layer prediction of the current layer.

[0020] In the image encoding method of the present disclosure, the reference layer may be a non-output layer.

[0021] In addition, a transmission method according to another aspect of the present disclosure can transmit a bitstream generated by an image encoding device or an image encoding method of the present disclosure.

[0022] In addition, a computer-readable recording medium according to another aspect of the present disclosure can store a bitstream generated by an image encoding method or an image encoding device of the present disclosure.

[0023] Additionally, a computer-readable recording medium according to another aspect of the present disclosure can store a bitstream that causes a decoding device to perform the image decoding method of the present disclosure.

[0024] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0025] According to the present disclosure, a video encoding / decoding method and device with improved encoding / decoding efficiency can be provided.

[0026] In addition, according to the present disclosure, a method and device for encoding / decoding an image can be provided that can efficiently remove a non-referenced parameter set among a parameter set of a reference layer based on a time identifier.

[0027] Additionally, according to the present disclosure, a method for transmitting a bitstream generated by an image encoding method or device according to the present disclosure can be provided.

[0028] In addition, according to the present disclosure, a recording medium storing a bitstream generated by an image encoding method or device according to the present disclosure can be provided.

[0029] In addition, according to the present disclosure, a recording medium storing a bitstream received and decoded by an image decoding device according to the present disclosure and used for image restoration can be provided.

[0030] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0031] FIG. 1 is a schematic diagram illustrating a video coding system to which an embodiment according to the present disclosure can be applied.

[0032] FIG. 2 is a schematic diagram illustrating an image encoding device to which an embodiment according to the present disclosure can be applied.

[0033] FIG. 3 is a schematic diagram illustrating an image decoding device to which an embodiment according to the present disclosure can be applied.

[0034] FIGS. 4 and 5 are diagrams illustrating examples of picture decoding and encoding procedures to which embodiments according to the present disclosure may be applied.

[0035] FIG. 6 is a diagram illustrating a hierarchical structure for a coded image to which an embodiment according to the present disclosure can be applied.

[0036] Figures 7 and 8 are diagrams explaining multi-layer based encoding and decoding.

[0037] FIGS. 9 to 17 are drawings illustrating a VPS to which an embodiment according to the present disclosure can be applied.

[0038] FIGS. 18 and 19 are drawings explaining a method for extracting (acquiring) a sub-bitstream to which an embodiment according to the present disclosure can be applied.

[0039] FIGS. 20 and 21 are diagrams illustrating a decryption method and / or an encoding method to which an embodiment according to the present disclosure can be applied.

[0040] FIG. 22 is a diagram illustrating a content streaming system to which an embodiment of the present disclosure can be applied.

[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0042] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.

[0043] In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.

[0044] In this disclosure, terms such as first, second, etc. are used solely to distinguish one component from another, and do not limit the order or importance of components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0045] In this disclosure, distinct components are used to clearly illustrate their respective characteristics, and do not necessarily imply that the components are separated. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0046] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also within the scope of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also within the scope of the present disclosure.

[0047] The present disclosure relates to encoding and decoding of images, and terms used in the present disclosure may have their usual meanings commonly used in the technical field to which the present disclosure belongs, unless newly defined in the present disclosure.

[0048] The method / embodiment disclosed in the present disclosure can be applied to a method disclosed in the versatile video coding (VVC) standard. In addition, the method / embodiment disclosed in the present disclosure can be applied to a method disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd generation of audio video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).

[0049] The present disclosure presents various embodiments of video / image coding, and unless otherwise stated, the embodiments in the present disclosure may be performed in combination with each other.

[0050] In the present disclosure, "video" may refer to a set of images over time. "Picture" generally refers to a unit representing one image at a specific time point, and a slice / tile is a coding unit that constitutes a part of a picture in encoding. A slice / tile may include one or more CTUs (coding tree units). The CTUs may be divided into one or more CUs.

[0051] A picture may be composed of one or more slices / tiles. A tile is a rectangular area existing within a specific tile row and a specific tile column within a picture, and may be composed of multiple CTUs. A tile row may be defined as a rectangular area of ​​CTUs, may have a height equal to the height of the picture, and may have a width specified by a syntax element signaled from a bitstream part such as a picture parameter set. A tile row may be defined as a rectangular area of ​​CTUs, may have a width equal to the width of the picture, and may have a height specified by a syntax element signaled from a bitstream part such as a picture parameter set.

[0052] Tile scan is a method for specifying a predetermined sequential order of CTUs that divide a picture. Here, CTUs can be sequentially ordered within a tile according to the CTU raster scan, and tiles within a picture can be sequentially ordered according to the raster scan order of the tiles in the picture. A slice can contain an integer number of complete tiles, or a contiguous integer number of complete CTU rows within a tile of a picture. A slice can be contained exclusively within a single NAL unit.

[0053] A picture may be divided into two or more subpictures. A subpicture may be a rectangular region of one or more slices within the picture.

[0054] A picture may be composed of one or more tile groups. A tile group may contain one or more tiles. A brick may represent a rectangular area of ​​CTU rows within a tile within a picture. A tile may contain one or more bricks. A brick may represent a rectangular area of ​​CTU rows within the tile. A tile may be divided into multiple bricks, and each brick may contain one or more CTU rows belonging to the tile. A tile that is not divided into multiple bricks may also be treated as a brick.

[0055] In the present disclosure, "pixel" or "pel" may refer to the smallest unit that constitutes a picture (or image). Additionally, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luma component or only a pixel / pixel value of a chroma component.

[0056] In the present disclosure, a "unit" may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., Cb, Cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "sample array," "block," or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0057] In the present disclosure, the "current block" may mean one of the following: a "current coding block," a "current coding unit," a "block to be encoded," a "block to be decoded," or a "block to be processed." When prediction is performed, the "current block" may mean a "current prediction block" or a "block to be predicted." When transformation (inverse transformation) / quantization (inverse quantization) is performed, the "current block" may mean a "current transformation block" or a "block to be transformed." When filtering is performed, the "current block" may mean a "block to be filtered."

[0058] Additionally, in the present disclosure, "current block" may mean "luma block of the current block" unless explicitly described as a chroma block. "Chroma block of the current block" may be expressed by explicitly including explicit description of chroma block, such as "chroma block" or "current chroma block."

[0059] In this disclosure, " / " and "," can be interpreted as "and / or". For example, "A / B" and "A, B" can be interpreted as "A and / or B". Additionally, "A / B / C" and "A, B, C" can mean "at least one of A, B, and / or C."

[0060] In this disclosure, "or" may be interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." Alternatively, "or" in this disclosure may mean "additionally or alternatively."

[0061] In the present disclosure, “at least one of A and B” may mean “only A,” “only B,” or “both A and B.” Additionally, in the present disclosure, the expressions “at least one of A or B” or “at least one of A and / or B” may be interpreted identically to “at least one of A and B.”

[0062] Additionally, in the present disclosure, “at least one of A, B and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B and C.” Additionally, “at least one of A, B or C” or “at least one of A, B and / or C” can mean “at least one of A, B and C.”

[0063] Additionally, parentheses used in this disclosure may mean "for example." Specifically, when "prediction (intra-prediction)" is indicated, "intra-prediction" may be suggested as an example of "prediction." In other words, "prediction" in this disclosure is not limited to "intra-prediction," and "intra-prediction" may be suggested as an example of "prediction." Furthermore, even when indicated as "prediction (i.e., intra-prediction)", "intra-prediction" may be suggested as an example of "prediction."

[0064] Technical features individually described in one drawing in this disclosure may be implemented individually or simultaneously.

[0065] Overview of Video Coding Systems

[0066] FIG. 1 illustrates a video coding system according to the present disclosure.

[0067] A video coding system according to one embodiment may include a source device (10) and a receiving device (20). The source device (10) may transmit encoded video and / or image information or data to the receiving device (20) via a digital storage medium or a network in the form of a file or streaming.

[0068] A source device (10) according to an embodiment may include a video source generation unit (11), an encoding device (12), and a transmission unit (13). A receiving device (20) according to an embodiment may include a reception unit (21), a decoding device (22), and a rendering unit (23). The encoding device (12) may be referred to as a video / image encoding device, and the decoding device (22) may be referred to as a video / image decoding device. The transmission unit (13) may be included in the encoding device (12). The reception unit (21) may be included in the decoding device (22). The rendering unit (23) may include a display unit, and the display unit may be configured as a separate device or an external component.

[0069] The video source generation unit (11) can obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source generation unit (11) can include a video / image capture device and / or a video / image generation device. The video / image capture device can include, for example, one or more cameras, a video / image archive including previously captured video / images, etc. The video / image generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate video / images. For example, a virtual video / image can be generated through a computer, etc., in which case the video / image capture process can be replaced with a process of generating related data.

[0070] The encoding device (12) can encode input video / images. The encoding device (12) can perform a series of procedures such as prediction, transformation, and quantization to improve compression and encoding efficiency. The encoding device (12) can output encoded data (encoded video / image information) in the form of a bitstream.

[0071] The transmission unit (13) can transmit encoded video / image information or data output in the form of a bitstream to the reception unit (21) of the reception device (20) via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) can include an element for generating a media file through a predetermined file format and can include an element for transmission via a broadcasting / communication network. The reception unit (21) can extract / receive the bitstream from the storage medium or network and transmit it to the decoding device (22).

[0072] The decoding device (22) can decode video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device (12).

[0073] The rendering unit (23) can render the decrypted video / image. The rendered video / image can be displayed through the display unit.

[0074] Overview of the video encoding device

[0075] FIG. 2 is a schematic diagram illustrating an image encoding device to which an embodiment according to the present disclosure can be applied.

[0076] As illustrated in FIG. 2, the image encoding device (100) may include an image segmentation unit (110), a subtraction unit (115), a transformation unit (120), a quantization unit (130), an inverse quantization unit (140), an inverse transformation unit (150), an addition unit (155), a filtering unit (160), a memory (170), an inter prediction unit (180), an intra prediction unit (185), and an entropy encoding unit (190). The inter prediction unit (180) and the intra prediction unit (185) may be collectively referred to as a “prediction unit.” The transformation unit (120), the quantization unit (130), the inverse quantization unit (140), and the inverse transformation unit (150) may be included in a residual processing unit. The residual processing unit may further include a subtraction unit (115).

[0077] All or at least some of the plurality of components constituting the video encoding device (100) may be implemented as a single hardware component (e.g., an encoder or a processor) according to an embodiment. In addition, the memory (170) may include a decoded picture buffer (DPB) and may be implemented by a digital storage medium.

[0078] The image segmentation unit (110) can segment an input image (or picture, frame) input to the image encoding device (100) into one or more processing units. For example, the processing unit may be called a coding unit (CU). The coding unit may be obtained by recursively segmenting a coding tree unit (CTU) or a largest coding unit (LCU) according to a QT / BT / TT (Quad-tree / binary-tree / ternary-tree) structure. For example, one coding unit may be segmented into a plurality of coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary-tree structure. For segmenting the coding unit, the quad-tree structure may be applied first, and the binary-tree structure and / or the ternary-tree structure may be applied later. The coding procedure according to the present disclosure may be performed based on the final coding unit that is no longer segmented. The maximum coding unit can be used directly as the final coding unit, and the coding unit of the lower depth obtained by dividing the maximum coding unit can be used as the final concatenated unit. Here, the coding procedure may include procedures such as prediction, transformation, and / or restoration described below. As another example, the processing unit of the coding procedure may be a prediction unit (PU) or a transformation unit (TU). The prediction unit and the transformation unit may each be divided or partitioned from the final coding unit. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit that derives a transform coefficient and / or a unit that derives a residual signal from a transform coefficient.

[0079] The prediction unit (inter-prediction unit (180) or intra-prediction unit (185)) can perform prediction on a block to be processed (current block) and generate a predicted block including prediction samples for the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. The prediction unit can generate various information regarding the prediction of the current block and transmit the information to the entropy encoding unit (190). The information regarding the prediction can be encoded by the entropy encoding unit (190) and output in the form of a bitstream.

[0080] The intra prediction unit (185) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the intra prediction mode and / or intra prediction technique. The intra prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of detail in the prediction direction. However, this is merely an example, and a greater or lesser number of directional prediction modes may be used depending on the settings. The intra prediction unit (185) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0081] The inter prediction unit (180) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different from each other. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), etc. A reference picture including the above temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (180) may construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (180) may use the motion information of neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the current block can be signaled by using the motion vector of the surrounding blocks as the motion vector predictor and encoding the motion vector difference and an indicator for the motion vector predictor. The motion vector difference can mean the difference between the motion vector of the current block and the motion vector predictor.

[0082] The prediction unit can generate a prediction signal based on various prediction methods and / or prediction techniques described below. For example, the prediction unit can apply intra prediction or inter prediction to predict the current block, and can also apply intra prediction and inter prediction simultaneously. A prediction method that simultaneously applies intra prediction and inter prediction to predict the current block may be called combined inter and intra prediction (CIIP). In addition, the prediction unit may perform intra block copy (IBC) to predict the current block. Intra block copy can be used for video / image coding of content such as games, such as screen content coding (SCC). IBC is a method of predicting the current block using a previously restored reference block within the current picture located at a predetermined distance from the current block. When IBC is applied, the location of the reference block within the current picture can be encoded as a vector (block vector) corresponding to the predetermined distance. IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives reference blocks within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in the present disclosure.

[0083] The prediction signal generated through the prediction unit can be used to generate a restoration signal or a residual signal. The subtraction unit (115) can generate a residual signal (residual block, residual sample array) by subtracting the prediction signal (predicted block, predicted sample array) output from the prediction unit from the input image signal (original block, original sample array). The generated residual signal can be transmitted to the conversion unit (120).

[0084] The transform unit (120) can apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. The transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0085] The quantization unit (130) can quantize the transform coefficients and transmit them to the entropy encoding unit (190). The entropy encoding unit (190) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information. The quantization unit (130) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0086] The entropy encoding unit (190) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoding unit (190) can also encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling information, transmitted information and / or syntax elements mentioned in the present disclosure may be encoded through the encoding procedure described above and included in the bitstream.

[0087] The above bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) for transmitting the signal output from the entropy encoding unit (190) and / or a storage unit (not shown) for storing the signal may be provided as an internal / external element of the video encoding device (100), or the transmission unit may be provided as a component of the entropy encoding unit (190).

[0088] The quantized transform coefficients output from the quantization unit (130) can be used to generate a residual signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (140) and inverse transformation unit (150), a residual signal (residual block or residual samples) can be restored.

[0089] The addition unit (155) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (180) or the intra prediction unit (185). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The addition unit (155) can be called a reconstructor or a reconstructed block generation unit. The generated reconstructed signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0090] The filtering unit (160) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (160) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (170), specifically, in the DPB of the memory (170). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (160) can generate various information regarding filtering and transmit the information to the entropy encoding unit (190), as described later in the description of each filtering method. The information regarding filtering may be encoded by the entropy encoding unit (190) and output in the form of a bitstream.

[0091] The modified restored picture transmitted to the memory (170) can be used as a reference picture in the inter prediction unit (180). Through this, when inter prediction is applied, the image encoding device (100) can avoid prediction mismatch between the image encoding device (100) and the image decoding device, and can also improve encoding efficiency.

[0092] The DPB in the memory (170) can store a modified reconstructed picture to be used as a reference picture in the inter prediction unit (180). The memory (170) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been reconstructed. The stored motion information can be transferred to the inter prediction unit (180) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (170) can store reconstructed samples of reconstructed blocks in the current picture and transfer them to the intra prediction unit (185).

[0093] Video Decoding Device Overview

[0094] FIG. 3 is a schematic diagram illustrating an image decoding device to which an embodiment according to the present disclosure can be applied.

[0095] As illustrated in FIG. 3, the image decoding device (200) may be configured to include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transformation unit (230), an addition unit (235), a filtering unit (240), a memory (250), an inter prediction unit (260), and an intra prediction unit (265). The inter prediction unit (260) and the intra prediction unit (265) may be collectively referred to as a “prediction unit.” The inverse quantization unit (220) and the inverse transformation unit (230) may be included in a residual processing unit.

[0096] All or at least some of the plurality of components constituting the video decoding device (200) may be implemented as a single hardware component (e.g., a decoder or processor) depending on the embodiment. In addition, the memory (170) may include a DPB and may be implemented by a digital storage medium.

[0097] The video decoding device (200) that receives a bitstream including video / image information can restore the image by performing a process corresponding to the process performed in the video encoding device (100) of FIG. 2. For example, the video decoding device (200) can perform decoding using a processing unit applied in the video encoding device. Therefore, the processing unit for decoding may be, for example, a coding unit. The coding unit may be a coding tree unit or may be obtained by dividing a maximum coding unit. In addition, the restored image signal decoded and output by the video decoding device (200) can be reproduced through a reproduction device (not shown).

[0098] The video decoding device (200) can receive a signal output from the video encoding device of FIG. 2 in the form of a bitstream. The received signal can be decoded through the entropy decoding unit (210). For example, the entropy decoding unit (210) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The video decoding device may additionally use information on the parameter set and / or the general constraint information to decode the image. The signaling information, received information, and / or syntax elements mentioned in the present disclosure can be obtained from the bitstream by being decoded through the decoding procedure. For example, the entropy decoding unit (210) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements required for image restoration and the quantized values ​​of transform coefficients for residuals. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using the syntax element information to be decoded and the decoding information of the surrounding block and the decoding target block or the information of the symbol / bin decoded in the previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (210) is provided to the prediction unit (inter prediction unit (260) and intra prediction unit (265)), and the residual value on which entropy decoding is performed by the entropy decoding unit (210), i.e., quantized transform coefficients and related parameter information, can be input to the inverse quantization unit (220). In addition, information regarding filtering among the information decoded by the entropy decoding unit (210) can be provided to the filtering unit (240). Meanwhile, a receiving unit (not shown) that receives a signal output from an image encoding device may be additionally provided as an internal / external element of the image decoding device (200), or the receiving unit may be provided as a component of an entropy decoding unit (210).

[0099] Meanwhile, the video decoding device according to the present disclosure may be referred to as a video / video / picture decoding device. The video decoding device may include an information decoder (video / video / picture information decoder) and / or a sample decoder (video / video / picture sample decoder). The information decoder may include an entropy decoding unit (210), and the sample decoder may include at least one of an inverse quantization unit (220), an inverse transformation unit (230), an addition unit (235), a filtering unit (240), a memory (250), an inter prediction unit (260), and an intra prediction unit (265).

[0100] The inverse quantization unit (220) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (220) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the image encoding device. The inverse quantization unit (220) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0101] In the inverse transform unit (230), the transform coefficients can be inversely transformed to obtain a residual signal (residual block, residual sample array).

[0102] The prediction unit can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block based on the prediction information output from the entropy decoding unit (210), and can determine a specific intra / inter-prediction mode (prediction technique).

[0103] The fact that the prediction unit can generate a prediction signal based on various prediction methods (techniques) described later is the same as that mentioned in the description of the prediction unit of the image encoding device (100).

[0104] The intra prediction unit (265) can predict the current block by referring to samples within the current picture. The description of the intra prediction unit (185) can be equally applied to the intra prediction unit (265).

[0105] The inter prediction unit (260) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (260) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes (techniques), and the information about the prediction can include information indicating the mode (technique) of inter prediction for the current block.

[0106] The addition unit (235) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (including the inter prediction unit (260) and / or the intra prediction unit (265)). When there is no residual for the block to be processed, such as when the skip mode is applied, the predicted block can be used as the restoration block. The description of the addition unit (155) can be equally applied to the addition unit (235). The addition unit (235) can be called a restoration unit or a restoration block generation unit. The generated restoration signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after going through filtering as described below.

[0107] The filtering unit (240) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (240) can apply various filtering methods to the restored picture to generate a modified restored picture, and store the modified restored picture in the memory (250), specifically, in the DPB of the memory (250). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0108] The (modified) reconstructed picture stored in the DPB of the memory (250) can be used as a reference picture in the inter prediction unit (260). The memory (250) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transferred to the inter prediction unit (260) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (250) can store reconstructed samples of reconstructed blocks within the current picture and transfer them to the intra prediction unit (265).

[0109] In this specification, the embodiments described in the filtering unit (160), the inter prediction unit (180), and the intra prediction unit (185) of the image encoding device (100) can be applied to the filtering unit (240), the inter prediction unit (260), and the intra prediction unit (265) of the image decoding device (200) in the same or corresponding manner, respectively.

[0110] General Video Coding Procedures

[0111] In image / video coding, pictures that constitute an image / video can be encoded / decoded according to a series of decoding orders. The picture order corresponding to the output order of the decoded pictures can be set differently from the decoding order, and based on this, not only forward prediction but also backward prediction can be performed during inter prediction.

[0112] FIG. 4 illustrates an example of a schematic picture decoding procedure to which the embodiments of the present disclosure are applicable. In FIG. 4, S410 may be performed in the entropy decoding unit (210) of the decoding device described above in FIG. 3, S420 may be performed in the prediction unit including the intra prediction unit (265) and the inter prediction unit (260), S430 may be performed in the residual processing unit including the inverse quantization unit (220) and the inverse transformation unit (230), S440 may be performed in the addition unit (235), and S450 may be performed in the filtering unit (240). S410 may include an information decoding procedure described in the present disclosure, S420 may include an inter / intra prediction procedure described in the present disclosure, S430 may include a residual processing procedure described in the present disclosure, S440 may include a block / picture restoration procedure described in the present disclosure, and S450 may include an in-loop filtering procedure described in the present disclosure.

[0113] Referring to FIG. 4, the picture decoding procedure may schematically include a procedure for obtaining image / video information (through decoding) from a bitstream (S410), a procedure for restoring a picture (S420 to S440), and an in-loop filtering procedure for a restored picture (S450), as shown in the description for FIG. 3. The picture restoration procedure may be performed based on prediction samples and residual samples obtained through the inter / intra prediction (S420) and residual processing (S430, inverse quantization and inverse transformation for quantized transform coefficients) processes described in the present disclosure. A modified restored picture can be generated through an in-loop filtering procedure for a restored picture generated through the above picture restoration procedure, and the modified restored picture can be output as a decoded picture and can be stored in a decoded picture buffer or memory (250) of a decoding device to be used as a reference picture in an inter prediction procedure when decoding a subsequent picture. In some cases, the in-loop filtering procedure may be omitted, in which case the restored picture can be output as a decoded picture and can be stored in a decoded picture buffer or memory (250) of a decoding device to be used as a reference picture in an inter prediction procedure when decoding a subsequent picture. The in-loop filtering procedure (S450) may include a deblocking filtering procedure, a sample adaptive offset (SAO) procedure, an adaptive loop filter (ALF) procedure, and / or a bi-lateral filter procedure, as described above, and some or all of them may be omitted. Additionally, one or more of the deblocking filtering procedure, the sample adaptive offset (SAO) procedure, the adaptive loop filter (ALF) procedure, and the bi-lateral filter procedure may be applied sequentially, or all of them may be applied sequentially.For example, the SAO procedure may be performed after a deblocking filtering procedure is applied to a restored picture. Or, for example, the ALF procedure may be performed after a deblocking filtering procedure is applied to a restored picture. This may also be performed in an encoding device.

[0114] FIG. 5 illustrates an example of a schematic picture encoding procedure to which the embodiment(s) of the present disclosure are applicable. In FIG. 5, S510 may be performed in a prediction unit including an intra prediction unit (185) or an inter prediction unit (180) of the encoding device described above in FIG. 2, S520 may be performed in a residual processing unit including a transform unit (120) and / or a quantization unit (130), and S530 may be performed in an entropy encoding unit (190). S510 may include an inter / intra prediction procedure described in the present disclosure, S520 may include a residual processing procedure described in the present disclosure, and S530 may include an information encoding procedure described in the present disclosure.

[0115] Referring to FIG. 5, the picture encoding procedure may include, as schematically shown in the description of FIG. 2, a procedure for encoding information for picture restoration (e.g., prediction information, residual information, partitioning information, etc.) and outputting it in the form of a bitstream, as well as a procedure for generating a restored picture for the current picture and a procedure (optional) for applying in-loop filtering to the restored picture. The encoding device can derive (corrected) residual samples from quantized transform coefficients through the inverse quantization unit (140) and the inverse transformation unit (150), and can generate a restored picture based on the prediction samples, which are the output of S510, and the (corrected) residual samples. The restored picture generated in this way may be the same as the restored picture generated by the decoding device described above. Through the in-loop filtering procedure for the above-mentioned restored picture, a modified restored picture can be generated, which can be stored in the decoded picture buffer or memory (170), and, as in the case of the decoding device, can be used as a reference picture in the inter prediction procedure when encoding a subsequent picture. As described above, some or all of the above-mentioned in-loop filtering procedure can be omitted depending on the case. When the above-mentioned in-loop filtering procedure is performed, (in-loop) filtering-related information (parameters) can be encoded in the entropy encoding unit (190) and output in the form of a bitstream, and the decoding device can perform the in-loop filtering procedure in the same manner as the encoding device based on the above-mentioned filtering-related information.

[0116] This in-loop filtering procedure can reduce noise that occurs during image / video coding, such as blocking artifacts and ringing artifacts, and improve subjective / objective visual quality. Furthermore, by performing the in-loop filtering procedure in both the encoding device and the decoding device, the encoding device and the decoding device can produce identical prediction results, thereby increasing the reliability of picture coding and reducing the amount of data that must be transmitted for picture coding.

[0117] As described above, a picture restoration procedure can be performed not only in a decoding device but also in an encoding device. A restoration block can be generated based on intra-prediction / inter-prediction for each block, and a restoration picture including the restoration blocks can be generated. If the current picture / slice / tile group is an I picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based only on intra-prediction. On the other hand, if the current picture / slice / tile group is a P or B picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based on intra-prediction or inter-prediction. In this case, inter-prediction may be applied to some blocks in the current picture / slice / tile group, and intra-prediction may be applied to some remaining blocks. The color component of a picture may include a luma component and a chroma component, and unless explicitly limited in the present disclosure, the methods and embodiments proposed in the present disclosure may be applied to the luma component and the chroma component.

[0118] Examples of coding hierarchies and structures

[0119] The coded video / image according to the present disclosure can be processed according to, for example, the coding layers and structures described below.

[0120] Figure 6 is a diagram illustrating the hierarchical structure of a coded video. The coded video can be divided into a video coding layer (VCL), which handles the decoding processing of the video and the video itself, a subsystem that transmits and stores the encoded information, and a network abstraction layer (NAL), which exists between the VCL and the subsystem and is responsible for network adaptation functions.

[0121] In VCL, VCL data containing compressed image data (slice data) can be generated, or a parameter set containing information such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), etc., or an SEI (Supplemental Enhancement Information) message additionally required for the image decoding process can be generated.

[0122] In NAL, a NAL unit can be created by adding header information (NAL unit header) to an RBSP (Raw Byte Sequence Payload) generated from a VCL. At this time, RBSP refers to slice data, parameter sets, SEI messages, etc. generated from a VCL. The NAL unit header can include NAL unit type information that is specific to the RBSP data included in the NAL unit.

[0123] As illustrated in the above drawing, NAL units can be divided into VCL NAL units and Non-VCL NAL units according to the RBSP generated from VCL. A VCL NAL unit may refer to a NAL unit that contains information about an image (slice data), and a Non-VCL NAL unit may refer to a NAL unit that contains information necessary for decoding an image (parameter set or SEI message).

[0124] The above-described VCL NAL units and non-VCL NAL units can be transmitted over a network by attaching header information according to the data specifications of the lower system. For example, NAL units can be transformed into data formats of a certain standard, such as the H.266 / VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted over various networks.

[0125] As described above, a NAL unit can have a NAL unit type specified according to the RBSP data structure included in the NAL unit, and information about the NAL unit type can be stored and signaled in the NAL unit header.

[0126] For example, depending on whether a NAL unit contains information about a picture (slice data), it can be broadly classified into VCL NAL unit types and Non-VCL NAL unit types. The VCL NAL unit type can be classified according to the nature and type of the picture contained in the VCL NAL unit, and the Non-VCL NAL unit type can be classified according to the type of parameter set, etc.

[0127] Below are examples of specific NAL unit types, based on the type of parameter set / information contained in the Non-VCL NAL unit type.

[0128] - DCI (Decoding capability information) NAL unit: Type for NAL units containing DCI

[0129] - VPS (Video Parameter Set) NAL unit: Type for NAL unit containing VPS

[0130] - SPS (Sequence Parameter Set) NAL unit: Type for NAL units containing SPS

[0131] - PPS (Picture Parameter Set) NAL unit: Type for NAL units containing PPS

[0132] - APS (Adaptation Parameter Set) NAL unit: Type for NAL units containing APS

[0133] - PH (Picture header) NAL unit: Type for NAL unit including PH

[0134] The above-described NAL unit types have syntax information for the NAL unit type, and the syntax information can be stored and signaled in the NAL unit header. For example, the syntax information can be nal_unit_type, and NAL unit types can be specified by the nal_unit_type value.

[0135] Meanwhile, as described above, a single picture may include multiple slices, and a single slice may include a slice header and slice data. In this case, a single picture header may be additionally added for multiple slices (a set of slice headers and slice data) within a single picture. The picture header (picture header syntax) may include information / parameters that can be commonly applied to the picture.

[0136] The above slice header (slice header syntax) may include information / parameters that can be commonly applied to the slices. The APS (APS syntax) or PPS (PPS syntax) may include information / parameters that can be commonly applied to one or more slices or pictures. The SPS (SPS syntax) may include information / parameters that can be commonly applied to one or more sequences. The VPS (VPS syntax) may include information / parameters that can be commonly applied to multiple layers. The DCI (DCI syntax) may include information / parameters that can be commonly applied to the entire video. The DCI may include information / parameters related to decoding capability. In the present disclosure, the high level syntax (HLS) may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DCI syntax, picture header syntax, and slice header syntax. Meanwhile, the low level syntax (LLS) in the present disclosure may include, for example, slice data syntax, CTU syntax, coding unit syntax, transformation unit syntax, and the like.

[0137] In the present disclosure, image / video information encoded from an encoding device to a decoding device and signaled in the form of a bitstream may include information related to partitioning within a picture, intra / inter prediction information, residual information, in-loop filtering information, etc., and may also include information of the slice header, information of the picture header, information of the APS, information of the PPS, information of the SPS, information of the VPS, and / or information of the DCI. In addition, the image / video information may further include general constraint information and / or information of a NAL unit header.

[0138] Picture information signaling using NAL units

[0139] Picture information can be signaled on a per-NAL unit basis. For example, picture information can be signaled as described below. A sublayer is a temporally scalable layer of a temporally scalable bitstream, which consists of VCL NAL units with a given value of the variable TemporalId and associated non-VCL NAL units. Here, the variable TemporalId can be derived as follows.

[0140] [Mathematical Formula 1]

[0141] TemporalId = nuh_temporal_id_plus1 - 1

[0142] The syntax element nuh_temproal_id_plus1 for signaling the value of the variable TemporalId can be signaled through the NAL unit header of the NAL unit. If the value of nal_unit_type in the NAL unit header is in the range of values ​​from IDR_W_RADL to RSV_IRAP_12, the value of TemporalId can be forced to 0. If the value of nal_unit_type is equal to STSA_NUT and the value of vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is 1, the value of TemporalId can be restricted not to be 0. The values ​​of TemporalId can be the same for all VCL NAL units of an AU. The value of TemporalId of a coded picture, PU or AU can be the value of TemporalId of a VCL NAL unit of the corresponding coded picture, PU or AU. The value of TemporalId of a sublayer representation may be the largest value among the TemporalIds of all VCL NAL units in a single sublayer representation.

[0143] The value of TemporalId for non-VCL NAL units that are not VCL NAL units may be restricted as follows:

[0144] - When nal_unit_type is equal to DCI_NUT, VPS_NUT or SPS_NUT, TemporalId may be restricted to have a value of 0, and the value of TemporalId of the AU containing the corresponding NAL unit may be restricted to be 0.

[0145] - Otherwise, if nal_unit_type is equal to PH_NUT, the value of TemporalId may be constrained to be equal to the TemporalId of the PU containing the NAL unit.

[0146] - Otherwise, if nal_unit_type is equal to EOS_NUT or EOB_NUT, TemporalID may be constrained to be equal to 0.

[0147] - Otherwise, if nal_unit_type is equal to AUD_NUT, FD_NUT, PREFIX_SEI_NUT, or SUFFIX_SEI_NUT, TemporalId may be constrained to have the same value as the TemporalId of the AU containing that NAL unit.

[0148] - Otherwise, if nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT or SUFFIX_APS_NUT, the value of TemporalId may be restricted to have a value greater than or equal to the TemporalId of the PU containing the corresponding NAL unit.

[0149] For example, if the NAL unit is a non-VCL NAL, the value of TemporalId can be equal to the smallest value among the TemporalId values ​​of all AUs to which the non-VCL NAL unit applies. If the value of nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT or SUFFIX_APS_NUT, the value of TemporalId can have a value greater than or equal to the TemporalId of the AU that contains it. This is because all PPSs and APSs can be included at the beginning of the bitstream (e.g., this information is transmitted out-of-band and the receiver places it at the beginning of the bitstream). Here, the first picture to be encoded can have a TemporalId with a value of 0.

[0150] Multi-layer based encoding

[0151] The image / video coding according to the present disclosure may include multi-layer-based image / video coding. The multi-layer-based image / video coding may include scalable coding. In multi-layer-based coding or scalable coding, input signals may be processed layer-by-layer. Depending on the layer, input signals (input images / pictures) may have different values ​​for at least one of resolution, frame rate, bit depth, color format, aspect ratio, and view. In this case, by utilizing the differences between layers (e.g., based on scalability), prediction between layers can be performed, thereby reducing redundant transmission / processing of information and increasing compression efficiency.

[0152] FIG. 7 illustrates a schematic block diagram of a multi-layer encoding device (700) to which an embodiment(s) of the present disclosure can be applied and in which encoding of a multi-layer-based video / image signal is performed.

[0153] The multi-layer encoding device (700) of FIG. 7 may include the encoding device of FIG. 2. Compared to FIG. 2, the multi-layer encoding device (700) of FIG. 7 does not illustrate the image segmentation unit (110) and the addition unit (155), but the multi-layer encoding device (700) may include the image segmentation unit (110) and the addition unit (155). In one embodiment, the image segmentation unit (110) and the addition unit (155) may be included on a layer-by-layer basis. Hereinafter, the description of FIG. 7 will focus on multi-layer-based prediction. For example, in addition to the contents described below, the multi-layer encoding device (700) of FIG. 7 may include technical ideas for the encoding device described above with reference to FIG. 2.

[0154] For convenience of explanation, a multi-layer structure consisting of two layers is illustrated in FIG. 7. However, embodiments of the present disclosure are not limited to two layers, and a multi-layer structure to which embodiments of the present disclosure are applied may include two or more layers.

[0155] Referring to FIG. 7, the encoding device (700) includes an encoding unit (700-1) for layer 1 and an encoding unit (700-0) for layer 0. Layer 0 may be a base layer, a reference layer, or a lower layer, and layer 1 may be an enhancement layer, a current layer, or an upper layer.

[0156] The encoding unit (700-1) of layer 1 may include a prediction unit (720-1), a residual processing unit (730-1), a filtering unit (760-1), a memory (710-1), an entropy encoding unit (740-1), and a MUX (Multiplexer, 770). In one embodiment, the MUX may be included as an external component.

[0157] The encoding unit (700-0) of layer 0 may include a prediction unit (720-0), a residual processing unit (730-0), a filtering unit (760-0), a memory (710-0), and an entropy encoding unit (740-0).

[0158] The prediction unit (720-0, 720-1) can perform prediction based on various prediction techniques as described above for the input image. For example, the prediction unit (720-0, 720-1) can perform inter prediction and intra prediction. The prediction unit (720-0, 720-1) can perform prediction in a predetermined processing unit. The unit for performing prediction may be a coding unit (CU) or a transform unit (TU). Based on the prediction result, a predicted block (including prediction samples) can be generated, and based on this, the residual processing unit can derive a residual block (including residual samples).

[0159] Inter prediction allows prediction to be performed based on information from at least one of the previous and / or subsequent pictures of the current picture, thereby generating a prediction block. Intra prediction allows prediction to be performed based on surrounding samples within the current picture, thereby generating a prediction block.

[0160] As a mode or method of inter prediction, various prediction mode methods described above can be used. In inter prediction, a reference picture can be selected for the current block, which is the prediction target, and a reference block corresponding to the current block can be selected within the reference picture. The prediction unit (720-0, 720-1) can generate a predicted block based on the reference block.

[0161] Additionally, the prediction unit (720-1) can perform prediction for layer 1 using information of layer 0. In the present disclosure, a method of predicting information of a current layer using information of another layer is referred to as inter-layer prediction for convenience of explanation.

[0162] Information of the current layer predicted using information of another layer (e.g. predicted by inter-layer prediction) may be at least one of texture, motion information, unit information, and certain parameters (e.g. filtering parameters, etc.).

[0163] Additionally, information of other layers used for prediction for the current layer (e.g. used for inter-layer prediction) may be at least one of texture, motion information, unit information, and predetermined parameters (e.g. filtering parameters, etc.).

[0164] In inter-layer prediction, the current block is a block within the current picture within the current layer (e.g., layer 1), and may be a block to be encoded. The reference block is a block within a picture (reference picture) belonging to the same access unit (AU) as the picture to which the current block belongs (current picture) in the layer (reference layer, e.g., layer 0) referenced for prediction of the current block, and may be a block corresponding to the current block.

[0165] An example of inter-layer prediction is inter-layer motion prediction, which uses motion information from a reference layer to predict motion information for the current layer. Inter-layer motion prediction allows motion information for the current block to be predicted using motion information from a reference block. In other words, when deriving motion information according to the inter-prediction mode described below, motion information candidates can be derived based on motion information from an inter-layer reference block, rather than from temporally neighboring blocks.

[0166] When applying inter-layer motion prediction, the prediction unit (720-1) may also scale and utilize motion information of a reference block of a reference layer (i.e., an inter-layer reference block).

[0167] As another example of interlayer prediction, interlayer texture prediction can use the texture of a restored reference block as a prediction value for the current block. In this case, the prediction unit (720-1) can scale the texture of the reference block by upsampling. Interlayer texture prediction can also be called interlayer (restored) sample prediction or simply interlayer prediction.

[0168] Another example of inter-layer prediction, inter-layer parameter prediction, allows the derived parameters of a reference layer to be reused in the current layer, or parameters for the current layer can be derived based on the parameters used in the reference layer.

[0169] Another example of inter-layer prediction is inter-layer residual prediction, which uses residual information from other layers to predict the residual of the current layer and performs prediction for the current block based on this.

[0170] In inter-layer differential prediction, which is another example of inter-layer prediction, prediction for the current block can be performed by using the difference between the reconstructed picture of the current layer and the upsampled or downsampled reconstructed picture of the reference layer.

[0171] Another example of inter-layer prediction, inter-layer syntax prediction, uses syntax information from a reference layer to predict or generate the texture of the current block. The syntax information from the referenced reference layer may include information about the intra prediction mode and motion information.

[0172] The various prediction methods using the interlayer described above may be used multiple times when making predictions for a specific block.

[0173] Here, as examples of inter-layer prediction, inter-layer texture prediction, inter-layer motion prediction, inter-layer unit information prediction, inter-layer parameter prediction, inter-layer residual prediction, inter-layer differential prediction, inter-layer syntax prediction, etc. are described, but the inter-layer prediction applicable in the present invention is not limited thereto.

[0174] For example, inter-layer prediction can be applied as an extension of inter-prediction for the current layer. That is, inter-prediction for the current block can be performed by including a reference picture derived from the reference layer among the reference pictures that can be referenced for inter-prediction of the current block.

[0175] In this case, the inter-layer reference picture may be included in the reference picture list for the current block. The prediction unit (720-1) may perform inter prediction for the current block using the inter-layer reference picture.

[0176] Here, the inter-layer reference picture may be a reference picture configured by sampling the restored picture of the reference layer so that it corresponds to the current layer. Therefore, if the restored picture of the reference layer corresponds to the picture of the current layer, the restored picture of the reference layer can be used as the inter-layer reference picture without sampling. For example, if the width and height of the samples in the restored picture of the reference layer and the restored picture of the current layer are the same, and the offset between the top left, top right, bottom left, and bottom right of the picture of the reference layer and the top left, top right, bottom left, and bottom right of the picture of the current layer is 0, the restored picture of the reference layer can be used as the inter-layer reference picture of the current layer without resampling.

[0177] Additionally, the restored picture of the reference layer from which the inter-layer reference picture is derived may be a picture belonging to the same AU as the current picture to be encoded.

[0178] When performing inter prediction for the current block by including an inter-layer reference picture in the reference picture list, the position of the inter-layer reference picture in the reference picture list may be different in the reference picture lists L0 and L1. For example, in the reference picture list L0, the inter-layer reference picture may be located after the short-term reference pictures preceding the current picture, and in the reference picture list L1, the inter-layer reference picture may be located at the end of the reference picture list.

[0179] Here, the reference picture list L0 is the reference picture list used for inter prediction of a P slice or the first reference picture list used for inter prediction of a B slice. The reference picture list L1 is the second reference picture list used for inter prediction of a B slice.

[0180] Therefore, the reference picture list L0 may be composed in the following order: short-term reference picture(s) before the current picture, inter-layer reference pictures, short-term reference pictures(s) after the current picture, and long-term reference pictures. The reference picture list L1 may be composed in the following order: short-term reference picture(s) after the current picture, short-term reference pictures(s) before the current picture, long-term reference pictures, and inter-layer reference pictures.

[0181] Here, a P slice (predictive slice) is a slice on which intra prediction is performed or inter prediction is performed using at most one motion vector and reference picture index per prediction block. A B slice (bi-predictive slice) is a slice on which intra prediction is performed or prediction is performed using at most two motion vectors and reference picture index per prediction block. In this regard, an I slice (intra slice) is a slice on which only intra prediction is applied.

[0182] Additionally, when performing inter prediction for the current block based on a reference picture list including inter-layer reference pictures, the reference picture list may include multiple inter-layer reference pictures derived from multiple layers.

[0183] When multiple inter-layer reference pictures are included, the inter-layer reference pictures may be interleaved in the reference picture lists L0 and L1. For example, assume that two inter-layer reference pictures, inter-layer reference pictures ILRPi and inter-layer reference pictures ILRPj, are included in the reference picture list used for inter prediction of the current block. In this case, in the reference picture list L0, ILRPi may be positioned after the short-term reference pictures preceding the current picture, and ILRPj may be positioned at the end of the list. Additionally, in the reference picture list L1, ILRPi may be positioned at the end of the list, and ILRPj may be positioned after the short-term reference pictures following the current picture.

[0184] In this case, the reference picture list L0 may be composed of the following order: short-term reference picture(s) before the current picture, inter-layer reference picture ILRPi, short-term reference picture(s) after the current picture, long-term reference picture, inter-layer reference picture ILRPj. The reference picture list L1 may be composed of the following order: short-term reference picture(s) after the current picture, inter-layer reference picture ILRPj, short-term reference picture(s) before the current picture, long-term reference picture, inter-layer reference picture ILRPi.

[0185] In addition, one of the two inter-layer reference pictures may be an inter-layer reference picture derived from a scalable layer in terms of resolution, and the other may be an inter-layer reference picture derived from a layer providing a different view. In this case, for example, if ILRPi is an inter-layer reference picture derived from a layer providing a different resolution, and ILRPj is an inter-layer reference picture derived from a layer providing a different view, then in the case of scalable video coding that supports only scalability excluding views, the reference picture list L0 may be composed of the short-term reference picture(s) before the current picture, the inter-layer reference picture ILRPi, the short-term reference picture(s) after the current picture, and the long-term reference picture, in that order, and the reference picture list L1 may be composed of the short-term reference picture(s) after the current picture, the short-term reference picture(s) before the current picture, the long-term reference picture, and the inter-layer reference picture ILRPi, in that order.

[0186] Meanwhile, in inter-layer prediction, information of an inter-layer reference picture may use only sample values, only motion information (motion vector), or both sample values ​​and motion information. When a reference picture index indicates an inter-layer reference picture, the prediction unit (720-1) may use only the sample values ​​of the inter-layer reference picture, only the motion information (motion vector) of the inter-layer reference picture, or both the sample values ​​and motion information of the inter-layer reference picture, depending on the information received from the encoding device.

[0187] In the case of using only the sample values ​​of the inter-layer reference picture, the prediction unit (720-1) can derive samples of a block specified by a motion vector in the inter-layer reference picture as prediction samples of the current block. In the case of scalable video coding that does not consider a view, the motion vector in inter prediction (inter-layer prediction) using the inter-layer reference picture can be set to a fixed value (e.g., 0).

[0188] In the case where only the motion information of the interlayer reference picture is used, the prediction unit (720-1) can use the motion vector specified in the interlayer reference picture as a motion vector predictor for deriving the motion vector of the current block. In addition, the prediction unit (720-1) can also use the motion vector specified in the interlayer reference picture as the motion vector of the current block.

[0189] In the case where both samples and motion information of an interlayer reference picture are used, the prediction unit (720-1) can use samples of an area corresponding to the current block in the interlayer reference picture and motion information (motion vector) specified in the interlayer reference picture to predict the current block.

[0190] When inter-layer prediction is applied, the encoding device can transmit to the decoding device a reference index indicating an inter-layer reference picture in the reference picture list, and can also transmit to the decoding device information specifying which information (sample information, motion information, or sample information and motion information) to use from the inter-layer reference picture, that is, information specifying the type of dependency (dependency) regarding inter-layer prediction between two layers.

[0191] FIG. 8 is a schematic block diagram of a decoding device to which embodiments of the present disclosure may be applied, and in which decoding of a multi-layer-based video / image signal is performed. The decoding device of FIG. 8 may include the decoding device of FIG. 3 . The rearrangement unit illustrated in FIG. 8 may be omitted or included in the dequantization unit. The description of this drawing focuses on multi-layer-based prediction. Other descriptions may include the description of the decoding device described in FIG. 3 .

[0192] In the example of FIG. 8, a multi-layer structure consisting of two layers is described as an example for convenience of explanation. However, it should be noted that the embodiments of the present disclosure are not limited thereto, and a multi-layer structure to which the embodiments of the present disclosure are applied may include two or more layers.

[0193] Referring to FIG. 8, a decoding device (800) may include a decoding unit (800-1) for layer 1 and a decoding unit (800-0) for layer 0. The decoding unit (800-1) of layer 1 may include an entropy decoding unit (810-1), a residual processing unit (820-1), a prediction unit (830-1), an adder (840-1), a filtering unit (850-1), and a memory (860-1). The decoding unit (800-0) of layer 0 may include an entropy decoding unit (810-0), a residual processing unit (820-0), a prediction unit (830-0), an adder (840-0), a filtering unit (850-0), and a memory (860-0).

[0194] When a bitstream containing image information is transmitted from an encoding device, DEMUX (805) can demultiplex the information by layer and transmit it to a decoding device for each layer.

[0195] The entropy decoding unit (810-1, 810-0) can perform decoding corresponding to the coding method used in the encoding device. For example, if CABAC is used in the encoding device, the entropy decoding unit (810-1, 810-0) can also perform entropy decoding using CABAC.

[0196] When the prediction mode for the current block is the intra prediction mode, the prediction unit (830-1, 830-0) can perform intra prediction for the current block based on surrounding restored samples within the current picture.

[0197] When the prediction mode for the current block is the inter prediction mode, the prediction unit (830-1, 830-0) can perform inter prediction for the current block based on information included in at least one picture among the previous picture or the subsequent picture of the current picture. Some or all of the motion information required for inter prediction can be derived by checking the information received from the encoding device and correspondingly thereto.

[0198] When skip mode is applied as a mode of inter prediction, the residual is not transmitted from the encoding device and the prediction block can be used as a restoration block.

[0199] Meanwhile, the prediction unit (830-1) of layer 1 may perform inter prediction or intra prediction using only information within layer 1, or may perform inter-layer prediction using information from another layer (layer 0).

[0200] Information of the current layer predicted using information of another layer (e.g. predicted by inter-layer prediction) may include at least one of texture, motion information, unit information, and predetermined parameters (e.g. filtering parameters, etc.).

[0201] Additionally, information of other layers used for prediction of the current layer (e.g. used for inter-layer prediction) may include at least one of texture, motion information, unit information, and predetermined parameters (e.g. filtering parameters, etc.).

[0202] In inter-layer prediction, the current block is a block within the current picture in the current layer (e.g., layer 1), and may be a block to be decoded. The reference block is a block within a picture (reference picture) belonging to the same access unit (AU) as the picture to which the current block belongs (current picture) in the layer (reference layer, e.g., layer 0) referenced for prediction of the current block, and may be a block corresponding to the current block.

[0203] The multi-layer decoding device (800) can perform inter-layer prediction as described above in the multi-layer encoding device (700). For example, the multi-layer decoding device (800) can perform inter-layer texture prediction, inter-layer motion prediction, inter-layer unit information prediction, inter-layer parameter prediction, inter-layer residual prediction, inter-layer differential prediction, inter-layer syntax prediction, etc. as described above in the multi-layer encoding device (700), and the inter-layer prediction applicable in the present disclosure is not limited thereto.

[0204] The prediction unit (830-1) can perform inter-layer prediction using an inter-layer reference picture when a reference picture index received from an encoding device or a reference picture index derived from a neighboring block indicates an inter-layer reference picture within a reference picture list. For example, when the reference picture index indicates an inter-layer reference picture, the prediction unit (830-1) can derive a sample value of an area specified by a motion vector in the inter-layer reference picture as a prediction block for the current block.

[0205] In this case, the inter-layer reference picture may be included in the reference picture list for the current block. The prediction unit (830-1) may perform inter prediction for the current block using the inter-layer reference picture.

[0206] Here, as described above in the multi-layer encoding device (700), in the operation of the multi-layer decoding device (800), the inter-layer reference picture may be a reference picture configured by sampling the restored picture of the reference layer to correspond to the current layer. The processing for the case where the restored picture of the reference layer corresponds to the picture of the current layer may also be processed in the same manner as in the encoding process.

[0207] In addition, as described above in the multi-layer encoding device (700), in the operation of the multi-layer decoding device (800), the restored picture of the reference layer from which the inter-layer reference picture is derived may be a picture belonging to the same AU as the current picture that is the encoding target.

[0208] In addition, as described above in the multi-layer encoding device (700), in the operation of the multi-layer decoding device (800), when performing inter prediction for the current block by including an inter-layer reference picture in the reference picture list, the position of the inter-layer reference picture in the reference picture list may be different in the reference picture lists L0 and L1.

[0209] In addition, as described above in the multi-layer encoding device (700), in the operation of the multi-layer decoding device (800), when inter prediction for the current block is performed based on a reference picture list including inter-layer reference pictures, the reference picture list may include a plurality of inter-layer reference pictures derived from a plurality of layers, and the arrangement of the inter-layer reference pictures may also be performed so as to correspond to what was described above in the encoding process.

[0210] In addition, as described above in the multi-layer encoding device (700), in the operation of the multi-layer decoding device (800), information of the inter-layer reference picture may be used only as sample values, only as motion information (motion vector), or both as sample values ​​and motion information.

[0211] The multi-layer decoding device (800) can receive a reference index indicating an inter-layer reference picture in a reference picture list from the multi-layer encoding device (700) and perform inter-layer prediction based on the reference index. In addition, the multi-layer decoding device (800) can also receive, from the multi-layer encoding device (700), information indicating which information (sample information, motion information, sample information and motion information) to use from the inter-layer reference picture, i.e., information specifying the type of dependency (dependency) regarding inter-layer prediction between two layers.

[0212] HLS (High Level Syntax) Signaling and Semantics

[0213] As previously explained, HLS can be encoded and / or signaled for video and / or image encoding. As previously explained, the video / image information in the present disclosure can be included in HLS. Furthermore, the image / video encoding method can be performed based on this image / video information.

[0214] Video Parameter Set signaling

[0215] For a multi-layer bitstream with inter-layer dependencies, an available set of layers can be decoded. A video parameter set (VPS) is a parameter set used to transmit layer information. The layer information may include, for example, information about an output layer set (OLS), information about a profile tier level, information about the relationship between an OLS and a hypothetical reference decoder, information about the relationship between an OLS and a DPB, and the like.

[0216] Before being referenced, a VPS raw byte sequence payload (RBSP) must be available to the decoding process, either by being contained in at least one Access Unit (AU) with TemporalID 0 or by being provided by external means. All VPS NAL units with a vps_video_parameter_set_id of a given value within the CVS must have identical contents.

[0217] The syntax sequentially illustrated in FIGS. 9 and 10 exemplarily illustrates the syntax structure of a VPS according to one embodiment of the present disclosure. The syntax elements of FIGS. 9 and 10 are described below.

[0218] vps_video_parameter_set_id provides an identifier for a VPS that can be referenced by other syntax elements. Other syntax elements can use vps_video_parameter_set_id to reference the VPS. The value of vps_video_parameter_set_id must be greater than 0.

[0219] vps_max_layers_minus1 can represent the maximum number of layers allowed in an individual CVS referencing a VPS. For example, vps_max_layers_minus1 plus 1 can represent the maximum number of layers allowed in an individual CVS referencing a VPS.

[0220] The value of vps_max_sublayers_minus1 plus 1 may represent the maximum number of temporal sublayers that may exist in a layer in an individual CVS referencing the above VPS.

[0221] A value of 1 for vps_all_layers_same_num_sublayers_flag may indicate that the number of temporal sublayers is the same for all layers in individual CVSs referencing the VPS. A value of 0 for vps_all_layers_same_num_sublayers_flag may indicate that the number of temporal sublayers may or may not be the same for layers in individual CVSs referencing the VPS. If the value of vps_all_layers_same_num_sublayers_flag is not provided in the bitstream, the value of vps_all_layers_same_num_sublayers_flag may be derived as 1.

[0222] A value of 1 for vps_all_independent_layers_flag may indicate that all layers in the CVS are independently encoded without inter-layer prediction. A value of 0 for vps_all_independent_layers_flag may indicate that at least one layer in the CVS may have been encoded using inter-layer prediction.

[0223] vps_layer_id[ i ] can represent the nuh_layer_id value of the ith layer. For any two non-negative integer values ​​m and n, if m is less than n, vps_layer_id[m] can be restricted to have a value less than vps_layer_id[n]. Here, nuh_layer_id is a syntax element signaled in the NAL unit header and can represent the identifier of the NAL unit.

[0224] A value of 1 of vps_independent_layer_flag[ i ] may indicate that inter-layer prediction is not applied to the layer corresponding to index i. A value of 0 of vps_independent_layer_flag[ i ] may indicate that inter-layer prediction may be applied to the layer corresponding to index i, and the syntax element vps_direct_ref_layer_flag[ i ][ j ] is obtained from the VPS. Here, j may have a value from 0 to i-1. Meanwhile, if the value of vps_independent_layer_flag[ i ] does not exist in the bitstream, its value may be derived as 1.

[0225] A value of 1 for vps_max_tid_ref_present_flag[ i ] may indicate that the syntax element vps_max_tid_il_ref_pics_plus1[ i ][ j ] is provided from the bitstream. A value of 0 for vps_max_tid_ref_present_flag[ i ] may indicate that the syntax element vps_max_tid_il_ref_pics_plus1[ i ][ j ] is not provided from the bitstream.

[0226] A value of vps_direct_ref_layer_flag[ i ][ j ] of 0 may indicate that the layer with index j is not a direct reference layer of the layer with index i. A value of vps_direct_ref_layer_flag[ i ][ j ] of 1 may indicate that the layer with index j is a direct reference layer of the layer with index i. For i and j having a range of values ​​from 0 to vps_max_layers_minus1, if the value of vps_direct_ref_layer_flag[ i ][ j ] is not obtained from the bitstream, its value may be derived as 0. If the value of vps_independent_layer_flag[ i ] is 0, there may exist at least one j such that the value of vps_direct_ref_layer_flag[ i ][ j ] is 1, and in this case, the range of the value of j may be a range of values ​​from 0 to i-1.

[0227] The variables NumDirectRefLayers[ i ], DirectRefLayerIdx[ i ][ d ], NumRefLayers[ i ], RefLayerIdx[ i ][ r ], and LayerUsedAsRefLayerFlag[ j ] can be derived as shown in Fig. 11.

[0228] The variable GeneralLayerIdx[ i ] represents the layer index of the layer whose value of nuh_layer_id is the same as vps_layer_id[ i ] and can be derived as shown in Fig. 12.

[0229] A value of 0 in vps_max_tid_il_ref_pics_plus1[ i ][ j ] may indicate that a picture in the j-th layer that is neither a GDR picture nor an IRAP picture and has a value of 0 in the syntax element ph_recovery_poc_cnt indicating the recovery point of the decoded picture is not used as an inter-layer reference picture for decoding pictures in the i-th layer. A value greater than 0 in vps_max_tid_il_ref_pics_plus1[ i ][ j ] may indicate that a picture in the j-th layer that has a TemporalId greater than vps_max_tid_il_ref_pics_plus1[ i ][ j ] - 1 is not used as an inter-layer reference picture for decoding pictures in the i-th layer. Meanwhile, if the value of vps_max_tid_il_ref_pics_plus1[ i ][ j ] is not obtained from the bitstream, its value can be derived as vps_max_sublayers_minus1 + 1.

[0230] A value of 1 for vps_each_layer_is_an_ols_flag may indicate that each OLS has only one layer, and that each layer belonging to the CVS referencing the VPS is an OLS with a single contained layer as its own output layer. A value of 0 for vps_each_layer_is_an_ols_flag may indicate that the OLS may contain more than one layer. In one embodiment, if the value of vps_max_layers_minus1 is 0, the value of vps_each_layer_is_an_ols_flag may be derived as 1. Otherwise, if the value of vps_all_independent_layers_flag is 0, the value of vps_each_layer_is_an_ols_flag may be derived as 0.

[0231] A value of 0 for vps_ols_mode_idc can indicate that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1 + 1. The ith OLS can contain layers with layer indices from 0 to i. And for each OLS, the highest layer among the OLSs can be output.

[0232] A value of 1 in vps_ols_mode_idc can indicate that the total number of OLSs specified by VPS is equal to vps_max_layers_minus1 + 1. The ith OLS can contain layers with layer indices from 0 to i. And for each OLS, all layers of the OLS can be output.

[0233] A value of 2 for vps_ols_mode_idc explicitly signals the total number of OLSs specified by the VPS, and for each OLS the output layer is explicitly signaled, and other layers may be direct or reference layers of the output layer of the OLS.

[0234] The value of vps_ols_mode_idc can be from 0 to 2. The value 3 of vps_ols_mode_idc can be reserved for future use. If the value of vps_all_independent_layers_flag is 1 and the value of vps_each_layer_is_an_ols_flag is 0, the value of vps_ols_mode_idc can be derived as 2.

[0235] The value obtained by adding 1 to vps_num_output_layer_sets_minus1 can represent the total number of OLSs specified by VPS when the value of vps_ols_mode_idc is a certain value (e.g., when the value is 2). The variable TotalNumOlss representing the total number of OLSs specified by VPS can be derived as shown in Fig. 13.

[0236] The value 1 of vps_ols_output_layer_flag[ i ][ j ] can indicate that the layer whose nuh_layer_id value is equal to vps_layer_id[ j ] when the value of vps_ols_mode_idc is 2 is the output layer of the ith OLS. The value 0 of vps_ols_output_layer_flag[ i ][ j ] can indicate that the layer whose nuh_layer_id value is equal to vps_layer_id[ j ] when the value of vps_ols_mode_idc is 2 is not the output layer of the ith OLS.

[0237] The variable NumOutputLayersInOls[ i ] representing the number of output layers in the ith OLS, the variable NumSubLayersInLayerInOLS[ i ][ j ] representing the number of sublayers existing in the jth layer in the ith OLS, the variable OutputLayerIdInOls[ i ][ j ] representing the nuh_layer_id value of the jth output layer in the ith OLS, and the variable LayerUsedAsOutputLayerFlag[ k ] representing whether the kth layer is used as an output layer in at least one OLS can be derived as shown in the pseudocodes shown in succession in FIGS. 14 and 15.

[0238] For each value of i from 0 to vps_max_layers_minus1, the values ​​of LayerUsedAsRefLayerFlag[ i ] and LayerUsedAsOutputLayerFlag[ i ] can be enforced not to be both 0. For example, no layer can be enforced to be neither the output layer of at least one OLS nor the direct reference layer of another layer.

[0239] For each OLS, at least one layer can be enforced to be the output layer. For example, for each value of i from 0 to TotalNumOlss - 1, the value of NumOutputLayersInOls[ i ] can be enforced to be greater than or equal to 1.

[0240] The variable NumLayersInOls[ i ] representing the number of layers in the ith OLS, the variable LayerIdInOls[ i ][ j ] representing the value of nuh_layer_id of the jth layer in the ith OLS, the variable NumMultiLayerOlss representing the number of multi-layer OLSs (e.g., OLSs including one or more layers), and the variable MultiLayerOlsIdx[ i ] representing the index into the list of multi-layer OLSs for the ith OLS when the value of NumLayersInOls[ i ] is greater than 0 can be derived as in the pseudocode of Fig. 16.

[0241] The variable OlsLayerIdx[ i ][ j ], which represents the OLS layer index of the layer with the same nuh_layer_id as LayerIdInOls[ i ][ j ], can be derived as in the pseudocode of Fig. 17.

[0242] The lowest layer present in each OLS may be constrained to be an independent layer. For example, for each i ranging from 0 to TotalNumOlss - 1, the value of vps_independent_layer_flag[ GeneralLayerIdx[ LayerIdInOls[ i ][ 0 ] ] ] may be constrained to be 1. Each layer may be constrained to be included in at least one OLS specified by the VPS.

[0243] Improvements to APS signaling

[0244] If it is not necessary to decode the output layer, during the extraction process, VCL NAL units having a TemporalId that is equal to or less than the target highest TemporalId in the non-output layer can be removed. However, a problem occurs in that APS NAL units existing in these layers and temporal sub-layers are not properly removed. Typically, in this case, the highest output layer tends not to use not only the VCL from the temporal sub-layer, but also the parameter set such as APS and / or PPS therefrom. Here, the APS may be configured to include ALF (adaptive loop filter) parameters, LMCS (luma mapping with chroma scaling) parameters, or scaling list parameters, depending on the APS type.

[0245] To solve the above problem, PPS and / or APS can be selected as parameter sets that are not used as references for decoding pictures in output layers during the extraction process. In addition to parameter sets, non-VCL NAL units that are not used as references for decoding pictures in output layers during the extraction process can also be selected as not used as references for decoding pictures in output layers during the extraction process. In addition, APS can be prevented from being extracted from the output bitstream.

[0246] For such processing, the syntax element vps_max_tid_il_ref_pics_plus1[ i ][ j ] described above can be redefined. For example, the first value (e.g., 0) of vps_max_tid_il_ref_pics_plus1[ i ][ j ] can indicate that a picture of the j-th layer that is neither a GDR picture nor an IRAP picture and has ph_recovery_poc_cnt value of 0 is not used as an inter-layer reference picture for decoding pictures of the i-th layer. A value greater than 0 of vps_max_tid_il_ref_pics_plus1[ i ][ j ] may indicate that pictures and parameter sets having TemporalId greater than vps_max_tid_il_ref_pics_plus1[ i ][ j ] - 1 in the j-th layer are not used as references for decoding pictures in the i-th layer. Meanwhile, if the value of vps_max_tid_il_ref_pics_plus1[ i ][ j ] is not obtained from the bitstream, its value may be derived as vps_max_sublayers_minus1 + 1.

[0247] Meanwhile, a sub-bitstream can be derived according to the following sub-bitstream extraction process. The sub-bitstream extraction process can take as input the variable inBitstream representing the input bitstream, the target OLS index targetOlsIdx, and the target TemporalId tIdTarget. The variable OutBitstream representing the output sub-bitstream can be derived as follows.

[0248] First, a sub-bitstream can be determined based on the value of the input bitstream (S1810). For example, the variable outBitstream representing the output sub-bitstream can be set to be identical to the variable inBitstream representing the input bitstream.

[0249] All PPS, APS, and VCL NAL units that satisfy all of the following conditions 1 to 3 from outBitstream, and their associated non-VCL NAL units, may be removed (S1820). Here, the associated non-VCL NAL units may be non-VCL NAL units that have a PayloadType other than 0, 1, or 130, and a nal_unit_type of any one of PH_NUT, FD_NUT, SUFFIX_SEI_NUT, and PREFIX_SEI_NUT. Fig. 19 shows NAL unit types.

[0250] (Condition 1) The nal_unit_type of the VCL NAL unit is equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or RASL_NUT, or the nal_unit_type is equal to GDR_NUT and the related ph_recovery_poc_cnt is not 0.

[0251] (Condition 2) nuh_layer_id is equal to LayerIdInOls[ targetOlsIdx ][ j ], where j has a value from 0 to NumLayersInOls[ targetOlsIdx ] - 1.

[0252] (Condition 3) TemporalId has a value greater than or equal to NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ].

[0253] Encoding and decoding methods

[0254] Below, an image encoding method and a decoding method performed by an image encoding device and an image decoding device according to an embodiment are described. The operation of the decoding device is first described.

[0255] As described above, when vps_max_tid_il_ref_pics_plus1[ i ][ j ] has a value greater than 0, for inter-layer prediction of the i-th layer (current layer), pictures and parameter sets having TemporalId greater than vps_max_tid_il_ref_pics_plus1[ i ][ j ] - 1 in the j-th layer (reference layer) may not be used as references. In addition, as described with reference to FIGS. 14 and 15, NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ] may be derived based on vps_max_tid_il_ref_pics_plus1[ i ][ j ]. Therefore, pictures and parameter sets with TemporalId greater than or equal to NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ] may not be used as references for inter-layer prediction of the current layer.

[0256] FIG. 20 is a diagram illustrating a method by which an image decoding device according to one embodiment decodes an image. The image decoding device according to one embodiment includes a memory and a processor, and the decoding device can perform decoding according to the embodiments described below through the operation of the processor.

[0257] According to an embodiment, a decoding device may obtain maximum temporal identifier information for inter-layer prediction (S2010). Specifically, the decoding device may obtain maximum temporal identifier information indicating a maximum temporal identifier of a reference layer referenced for inter-layer prediction of a current layer from a bitstream. In the present disclosure, the meaning of “referenced for inter-layer prediction of the current layer” may include “referenceable for inter-layer prediction of the current layer.” Here, the maximum temporal identifier information may be the syntax element vps_max_tid_il_ref_pics_plus1[i][j] described above.

[0258] Next, the decoding device may perform a sub-bitstream extraction process from the bitstream based on the maximum time identifier information (S2020). In the sub-bitstream extraction process of step S2020, the decoding device may remove a parameter set that is not referenced for inter-layer prediction of the current layer from among the parameter sets of the reference layer. Here, the parameter set may be an Adaptive Parameter Set (APS).

[0259] Specifically, the decoding device can derive NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ] based on the maximum time identifier information obtained from the bitstream. At this time, for example, the methods of FIGS. 14 and 15 can be used. The decoding device can extract a sub-bitstream from which a parameter set that is not referenced for inter-layer prediction of the current layer is removed from among the parameter sets of the reference layer by performing the sub-bitstream extraction process described with reference to FIG. 18 based on the derived NumSubLayersInLayerInOLS.

[0260] The above maximum time identifier information indicates the maximum time identifier of the reference layer referenced for decoding the picture of the current layer. Among the parameter sets of the reference layer, a parameter set having a time identifier greater than the maximum time identifier is not referenced for decoding the picture of the current layer.

[0261] In addition, the maximum time identifier information indicates the maximum time identifier of the reference layer referenced for decoding the picture of the current layer. Among the pictures of the reference layer, pictures having a time identifier greater than the maximum time identifier are not referenced for decoding the picture of the current layer.

[0262] Meanwhile, the decoding device may further include a step of obtaining (extracting) a sub-bitstream from the bitstream based on the number of sub-layers for the current layer. The sub-bitstream is obtained by removing a predetermined parameter set from the bitstream, and the predetermined parameter set may be determined based on a size comparison of a time identifier of the predetermined parameter set and the number of sub-layers determined for a layer corresponding to the predetermined parameter set. Here, the predetermined parameter set may be an Adaptive Parameter Set (APS).

[0263] The above-mentioned predetermined parameter set can be determined based on whether the value of the time identifier of the above-mentioned predetermined parameter set is greater than or equal to the number of sub-layers determined for the layer corresponding to the above-mentioned predetermined parameter set among the layers of the predetermined OLS (Output Layer Set).

[0264] For example, whether to remove a given parameter set can be determined based on whether the TemporalId of the nuh_layer_id and TemporalId of the given parameter set has a value greater than or equal to NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ]. At this time, NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ] can be derived by the method described in FIGS. 14 and 15. For example, NumSubLayersInLayerInOLS[ ][ ] can be derived based on vps_max_tid_il_ref_pics_plus1.

[0265] Meanwhile, the above-described OLS may be any one of the OLSs identified by information signaled by a VPS (video parameter set).

[0266] In the example described with reference to FIG. 20, the current layer may be an output layer, and the reference layer may be a non-output layer.

[0267] FIG. 21 is a diagram illustrating a method for encoding an image by an encoding device according to an embodiment. The image encoding device according to an embodiment includes a memory and a processor, and the encoding device can perform encoding in a manner corresponding to the decoding of a decoding device according to an embodiment described below by the operation of the processor.

[0268] An encoding device according to an embodiment may determine maximum time identifier information based on a maximum time identifier for inter-layer prediction (S2110). Specifically, the encoding device may determine a maximum time identifier of a reference layer referenced for inter-layer prediction of a current layer, and determine maximum time identifier information indicating the determined maximum time identifier. In the present disclosure, the meaning of “something referenced for inter-layer prediction of the current layer” may include “something that can be referenced for inter-layer prediction of the current layer.”

[0269] Next, the encoding device can generate a bitstream by encoding maximum time identifier information indicating a maximum time identifier (S2120). Here, the maximum time identifier information may be the syntax element vps_max_tid_il_ref_pics_plus1[i][j] described above.

[0270] The above maximum time identifier information can be used in the sub-bitstream extraction process. Specifically, the above maximum time identifier information can be used to remove parameter sets that are not referenced for inter-layer prediction of the current layer among the parameter sets of the reference layer. Here, the parameter set can be an Adaptive Parameter Set (APS).

[0271] Specifically, based on the above maximum time identifier information, NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ] can be derived. At this time, for example, the methods of FIGS. 14 and 15 can be used. In addition, based on the derived NumSubLayersInLayerInOLS, by performing the sub-bitstream extraction process described with reference to FIG. 18, a sub-bitstream from which a parameter set that is not referenced for inter-layer prediction of the current layer is removed from among the parameter sets of the reference layer can be extracted.

[0272] The above maximum time identifier represents the maximum time identifier value of the reference layer referenced for encoding the picture of the current layer. Among the parameter sets of the reference layer, a parameter set having a time identifier greater than the maximum time identifier is not referenced for encoding the picture of the current layer.

[0273] In addition, the maximum time identifier indicates the maximum time identifier of the reference layer referenced for encoding the picture of the current layer. Among the pictures of the reference layer, pictures having a time identifier greater than the maximum time identifier are not referenced for encoding the picture of the current layer.

[0274] Meanwhile, a sub-bitstream can be obtained (extracted) based on the number of sub-layers for the current layer from a bitstream generated according to the above encoding method. For example, the sub-bitstream can be obtained by removing a predetermined parameter set from the bitstream. In addition, the predetermined parameter set can be determined based on a size comparison of a time identifier of the predetermined parameter set and the number of sub-layers determined for a layer corresponding to the predetermined parameter set. Here, the predetermined parameter set can be an APS (Adaptive Parameter Set).

[0275] For example, the predetermined parameter set may be determined based on whether the value of the time identifier of the predetermined parameter set is greater than or equal to the number of sub-layers determined for the layer corresponding to the predetermined parameter set among the layers of the predetermined OLS (Output Layer Set).

[0276] For example, whether to remove a given parameter set can be determined based on whether the TemporalId has a value greater than or equal to NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ] for the nuh_layer_id and TemporalId of the given parameter set.

[0277] Meanwhile, the OLS may be any one of the OLSs identified by information signaled by a video parameter set (VPS).

[0278] In the example described with reference to FIG. 21, the current layer may be an output layer, and the reference layer may be a non-output layer.

[0279] Application Examples

[0280] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps may be included, or some steps may be excluded and additional steps may be included.

[0281] In the present disclosure, a video encoding device or video decoding device performing a predetermined operation (step) may perform an operation (step) of checking the conditions or circumstances under which the operation (step) is performed. For example, if it is described that a predetermined operation is performed when a predetermined condition is satisfied, the video encoding device or video decoding device may perform an operation of checking whether the predetermined condition is satisfied and then perform the predetermined operation.

[0282] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0283] Additionally, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general processors, controllers, microcontrollers, microprocessors, etc.

[0284] In addition, the video decoding device and the video encoding device to which the embodiments of the present disclosure are applied may be included in a multimedia broadcasting transmitting and receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as a video communication, a mobile streaming device, a storage medium, a camcorder, a video-on-demand (VoD) service providing device, an OTT (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, a video phone video device, and a medical video device, and may be used to process a video signal or a data signal. For example, the OTT (Over the top video) device may include a game console, a Blu-ray player, an Internet-connected TV, a home theater system, a smartphone, a tablet PC, a DVR (Digital Video Recorder), and the like.

[0285] FIG. 22 is a diagram illustrating a content streaming system to which an embodiment of the present disclosure can be applied.

[0286] As illustrated in FIG. 22, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0287] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, generates a bitstream, and transmits it to the streaming server. Alternatively, if multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server may be omitted.

[0288] The above bitstream can be generated by an image encoding method and / or an image encoding device to which an embodiment of the present disclosure is applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0289] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server may control commands / responses between each device within the content streaming system.

[0290] The streaming server can receive content from a media repository and / or encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0291] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0292] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0293] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0294] Embodiments according to the present disclosure can be used to encode / decode images.

Claims

1. In a video decoding method performed by a video decoding device, A step of obtaining maximum time identifier information from a bitstream, which indicates the maximum time identifier of a reference layer referenced for inter-layer prediction of the current layer; and A step of performing a sub-bitstream extraction process from the bitstream based on the maximum time identifier information, An image decoding method in which, in the above sub-bitstream extraction process, a parameter set that is not referenced for inter-layer prediction of the current layer among the parameter sets of the reference layer is removed from the bitstream.

2. In paragraph 1, The above parameter set is an image decoding method that is an adaptive parameter set (APS).

3. In paragraph 1, A method for decoding an image in which a parameter set having a time identifier greater than the maximum time identifier among the parameter sets of the above reference layer is not referenced for inter-layer prediction of the current layer.

4. In paragraph 3, A method for decoding an image in which a picture having a time identifier greater than the maximum time identifier among the pictures of the above reference layer is not referenced for inter-layer prediction of the current layer.

5. In paragraph 3, A method for decoding an image in which the above reference layer is a non-output layer.

6. An image decoding device including a memory and at least one processor, At least one processor, Obtain maximum time identifier information from the bitstream, which indicates the maximum time identifier of the reference layer referenced for inter-layer prediction of the current layer, A sub-bitstream extraction process is performed from the bitstream based on the above maximum time identifier information, An image decoding device in which, in the above sub-bitstream extraction process, a parameter set that is not referenced for inter-layer prediction of the current layer among the parameter sets of the reference layer is removed from the bitstream.

7. In a video encoding method performed by a video encoding device, A step of determining maximum time identifier information indicating the maximum time identifier of a reference layer referenced for inter-layer prediction of the current layer; and comprising a step of encoding the above maximum time identifier information into a bitstream, The above maximum time identifier information is used to perform a sub-bitstream extraction process from the bitstream, An image encoding method in which, in the above sub-bitstream extraction process, a parameter set that is not referenced for inter-layer prediction of the current layer among the parameter sets of the reference layer is removed from the bitstream.

8. In paragraph 7, The above parameter set is an image encoding method called APS (Adaptive Parameter Set).

9. In paragraph 7, A method for encoding an image, wherein a parameter set having a time identifier greater than the maximum time identifier among the parameter sets of the above reference layer is not referenced for inter-layer prediction of the current layer.

10. In paragraph 9, A method for encoding an image, wherein among the pictures of the reference layer, a picture having a time identifier greater than the maximum time identifier is not referenced for inter-layer prediction of the current layer.

11. In paragraph 9, A method of encoding an image in which the above reference layer is a non-output layer.

12. A computer-readable recording medium storing a bitstream generated by the image encoding method of Article 7.

Images