Video or video coding based on NAL unit related information

Abstract

To provide a video based on NAL unit related information.SOLUTION: According to the disclosure of the present document, based on NAL unit type related information on whether a picture has mixed NAL unit types, NAL unit types for slices within the picture can be determined, and for the picture having the mixed NAL unit types, not only IRAP but also a form mixed with NAL units of other types can be provided.SELECTED DRAWING: Figure 19

Description

Technical Field

[0001] The present technology relates to video or video coding, for example, coding technology based on NAL (network abstraction layer) unit related information.

Background Art

[0002] In recent years, the demand for high-resolution and high-quality videos / videos such as 4K or 8K or higher UHD (Ultra High Definition) videos / videos has been increasing in various fields. As the video / video data becomes higher in resolution and quality, the amount of information or bits transmitted relatively increases compared to the existing video / video data. Therefore, when transmitting video data using a medium such as an existing wired or wireless broadband line or storing video / video data using an existing storage medium, the transmission cost and storage cost increase.

[0003] In addition, in recent years, the interest and demand for immersive media such as VR (Virtual Reality), AR (Artificial Realtiy) contents, and holograms have been increasing, and the broadcasting of videos / videos having video characteristics different from those of real videos, such as game videos, has been increasing.

[0004] Therefore, there is a need for a highly efficient video / video compression technology to effectively compress, transmit, store, and reproduce the information of high-resolution and high-quality videos / videos having various characteristics as described above.

[0005] In addition, a solution for improving the efficiency of video / video coding is required. Therefore, a solution for effectively signaling and coding information related to the NAL (network abstraction layer) unit is required.

Summary of the Invention

Problems to be Solved by the Invention

[0006] The technical problem of this document is to provide a method and apparatus for enhancing the efficiency of video / video coding.

[0007] Another technical problem of this document is to provide a method and apparatus for enhancing the efficiency of video / video coding based on NAL unit related information.

[0008] Yet another technical problem of this document is to provide a method and apparatus for enhancing the efficiency of video / video coding for a picture having a mixed NAL unit type.

[0009] Yet another technical problem of this document is to provide a method and apparatus for making reference picture list related information exist for a slice within a picture having a specific NAL unit type or be signaled for a picture having a mixed NAL unit type.

Means for Solving the Problem

[0010] According to an embodiment of this document, based on NAL unit type related information regarding whether a picture has a mixed NAL unit type, the NAL unit type for a slice within the picture can be determined. For example, based on NAL unit type related information regarding that a picture has a mixed NAL unit type, the first NAL unit for the first slice of the picture and the second NAL unit for the second slice of the picture may have different NAL unit types. Alternatively, based on NAL unit type related information regarding that a picture does not have a mixed NAL unit type, the first NAL unit for the first slice of the picture and the second NAL unit for the second slice of the picture may have the same NAL unit type.

[0011] According to one embodiment of this document, based on the case of allowing a NAL unit type with a mixed picture, for a slice having a specific NAL unit type within a picture, signaling-related information of a reference picture list can be made to exist.

[0012] According to one embodiment of this document, a method for decoding video / video executed by a decoding device is provided. The method for decoding the video / video can include the method disclosed in the embodiment of this document.

[0013] According to one embodiment of this document, a decoding device for executing decoding of video / video is provided. The decoding device can perform the method disclosed in the embodiment of this document.

[0014] According to one embodiment of this document, a method for encoding video / video executed by an encoding device is provided. The method for encoding the video / video can include the method disclosed in the embodiment of this document.

[0015] According to one embodiment of this document, an encoding device for executing encoding of video / video is provided. The encoding device can perform the method disclosed in the embodiment of this document.

[0016] According to one embodiment of this document, a computer-readable digital storage medium storing encoded video / video information generated by the method for encoding video / video disclosed in at least one of the embodiments of this document is provided.

[0017] According to one embodiment of this document, a computer-readable digital storage medium storing encoded information or encoded video / video information that causes a decoding device to perform the method for decoding video / video disclosed in at least one of the embodiments of this document is provided.

Advantages of the Invention

[0018] This document can have various effects. For example, according to one embodiment of this document, the compression efficiency of general video can be increased. Also, according to one embodiment of this document, based on NAL unit related information, the efficiency of video coding can be increased. Further, according to one embodiment of this document, the efficiency of video coding for a picture having a mixed NAL unit type can be increased. Also, according to one embodiment of this document, for a picture having a mixed NAL unit type, reference picture list related information can be effectively signaled and coded. Further, according to one embodiment of this document, by allowing a picture that includes a leading picture NAL unit type (e.g., RASL_NUT, RADL_NUT) and other non-IRAP NAL unit types (e.g., TRAIL_NUT, STSA, NUT) in a mixed form, not only IRAP but also other types of NAL units are provided in a mixed form for a picture having a mixed NAL unit type, and through this, it can have more flexible characteristics.

[0019] The effects obtained through the specific embodiments of this document are not limited to the effects listed above. For example, there may be various technical effects that a person having ordinary skill in the related art can understand or derive from this document. Thus, the specific effects of this document are not limited to what is explicitly described in this document, and may include various effects that can be understood or derived from the technical features of this document.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Best Mode for Carrying Out the Invention

[0021] This document can be modified in various ways and can have various embodiments. Specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit this document to specific embodiments. The terms commonly used in this specification are merely used to describe specific embodiments and are not used with the intention of limiting the technical idea of this document. Singular expressions include plural expressions unless the context clearly indicates otherwise. Terms such as "including" or "having" in this specification are intended to specify the existence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and it should be understood that the existence or addition possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof, etc. is not precluded in advance.

[0022] On the other hand, each configuration in the drawings described in this document is independently illustrated for the convenience of explaining different characteristic functions. It does not mean that each configuration is implemented by separate hardware or separate software. For example, among each configuration, two or more configurations can be combined to form one configuration, and one configuration can also be divided into multiple configurations. Embodiments in which each configuration is integrated and / or separated are included in the scope of rights of this document as long as they do not deviate from the essence of this document.

[0023] In this document, "A or B" can mean "only A", "only B", or "both A and B". In other words, in this document, "A or B" can be interpreted as "A and / or B". For example, in this document, "A, B or C" can mean "only A", "only B", "only C", or "any combination of A, B and C".

[0024] The slashes ( / ) and commas used in this document can mean "and / or". For example, "A / B" can mean "A and / or B". Thus, "A / B" can mean "only A", "only B", or "both A and B". For example, "A, B, C" can mean "A, B or C".

[0025] In this document, "at least one of A and B" can mean "only A", "only B", or "both A and B". Also, in this document, expressions such as "at least one of A or B" and "at least one of A and / or B" can be interpreted in the same way as "at least one of A and B".

[0026] Also, in this document, "at least one of A, B and C" may mean "only A", "only B", "only C", or "any combination of A, B and C". Also, "at least one of A, B or C" or "at least one of A, B and / or C" may mean "at least one of A, B and C".

[0027] Also, the parentheses used in this document may mean "for example". Specifically, when "prediction (intra prediction)" is displayed, "intra prediction" may be proposed as an example of "prediction". In other words, "prediction" in this document is not limited to "intra prediction", and "intra prediction" may be proposed as an example of "prediction". Also, when "prediction (i.e., intra prediction)" is displayed, "intra prediction" may be proposed as an example of "prediction".

[0028] This document relates to video / video coding. For example, the methods / examples disclosed in this document can be applied to the methods disclosed in the VVC (versatile video coding) standard. Also, the methods / examples disclosed in this document can be applied to the methods disclosed in the EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard) or next-generation video / video coding standards (e.g., 267 or H.268, etc.).

[0029] This document presents various examples related to video / video coding, and unless otherwise mentioned, the above examples may be combined with each other.

[0030] In this document, "video" may mean a collection of a series of "images" over time. "Picture" generally means a unit indicating one video in a specific time period, and "slice" / "tile" is a unit constituting a part of a picture in coding. A slice / tile may include one or more CTUs (Coding Tree Units). One picture may be composed of one or more slices / tiles. A tile is a rectangular area of CTUs within a specific column of tiles in a picture and within a specific column of tiles. The column of the tiles is a rectangular area of CTUs, the rectangular area has the same height as the height of the picture, and the width may be specified by a syntax element in the picture parameter set. The row of the tiles is a rectangular area of CTUs, the rectangular area has a width specified by a syntax element in the picture parameter set, and the height may be the same as the height of the picture. Tile scan may indicate a specific sequential ordering of CTUs that partition a picture, the CTUs may be continuously arranged in a raster scan of the CTUs within a tile, and the tiles in a picture may be continuously arranged in a raster scan of the tiles of the picture. A slice may exclusively include an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile of a picture in a single NAL unit.

[0031] On the other hand, one picture can be divided into two or more sub-pictures. A sub-picture may be a rectangular area of one or more slices in a picture.

[0032] A pixel or pel can mean the smallest unit that makes up one picture (or video). Also, the term "sample" may be used as a term corresponding to a pixel. A sample generally indicates a pixel or a pixel value, and may indicate only the pixel / pixel value of the luma component, or may indicate only the pixel / pixel value of the chroma component. Alternatively, a sample may mean a pixel value in the spatial domain, and when such a pixel value is converted to the frequency domain, it may mean a conversion coefficient in the frequency domain.

[0033] A unit can indicate the basic unit of video processing. A unit can include at least one of a specific region of a picture and information regarding the region. One unit can include one luma block and two chroma (e.g., cb, cr) blocks. A unit may, in some cases, be used interchangeably with terms such as "block" or "area". In a general case, an M×N block can include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0034] Also, in this document, at least one of quantization / inverse quantization and / or transformation / inverse transformation may be omitted. When quantization / inverse quantization is omitted, the quantized transform coefficients may be called transform coefficients. When transformation / inverse transformation is omitted, the transform coefficients may be called coefficients or residual coefficients, or, for the sake of consistency of expression, may still be called transform coefficients.

[0035] In this document, the quantized transform coefficients and the transform coefficients may each be referred to as transform coefficients and scaled transform coefficients, respectively. In this case, the residual information may include information regarding the transform coefficients, and the information regarding the transform coefficients may be signaled via a residual coding syntax. The transform coefficients may be derived based on the residual information (or the information regarding the transform coefficients), and the scaled transform coefficients may be derived via an inverse transform (scaling) with respect to the transform coefficients. The residual samples may be derived based on an inverse transform (transformation) with respect to the scaled transform coefficients. This may be applied / expressed similarly in other parts of this document.

[0036] In this document, the technical features separately described within one drawing may be realized separately or simultaneously.

[0037] Hereinafter, with reference to the accompanying drawings, the preferred embodiments of this document will be described in more detail. Hereinafter, the same reference numerals will be used for the same components in the drawings, and the redundant description for the same components may be omitted.

[0038] FIG. 1 schematically shows an example of a video / image coding system that can be applied to an embodiment of this document.

[0039] As shown in FIG. 1, the video / image coding system can include a first device (source device) and a second device (receiving device). The source device can transmit the encoded video / image information or data to the receiving device in a file or streaming form via a digital storage medium or a network.

[0040] The source device can include a video source, an encoding device, and a transmission unit. The receiving device can include a receiving unit, a decoding device, and a renderer. The encoding device can be referred to as a video / video encoding device, and the decoding device can be referred to as a video / video decoding device. A transmitter can be provided in the encoding device. A receiver can be provided in the decoding device. The renderer can include a display unit, and the display unit can also be composed of a separate device or an external component.

[0041] The video source can obtain video / video through processes such as video / video capture, synthesis, or generation. The video source can include a video / video capture device and / or a video / video generation device. The video / video capture device can include, for example, one or more cameras, a video / video archive including previously captured video / video, etc. The video / video generation device can include, for example, a computer, a tablet, and a smartphone, etc., and can (electronically) generate video / video. For example, virtual video / video can be generated through a computer or the like, and in this case, the video / video capture process can be replaced by the process of generating related data.

[0042] The encoding device can encode the input video / video. The encoding device can perform a series of procedures such as prediction, transformation, quantization, etc. for compression and coding efficiency. The encoded data (encoded video / video information) can be output in the form of a bitstream.

[0043] The transmitting unit can transmit the encoded video / video information or data output in bitstream form to the receiving unit of the receiving device via a digital storage medium or network in file or streaming form. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating a media file via a predetermined file format and can include elements for transmission via a broadcast / communication network. The receiving unit can receive / extract the bitstream and transmit it to the decoding device.

[0044] The decoding device can decode the video / video by performing a series of procedures such as inverse quantization, inverse transformation, prediction, etc., corresponding to the operation of the encoding device.

[0045] The renderer can render the decoded video / video. The rendered video / video can be displayed via the display unit.

[0046] Figure 2 is a diagram schematically explaining the configuration of a video / video encoding device to which the embodiments of this document can be applied. Hereinafter, the encoding device can include a video encoding device and / or a video encoding device.

[0047] As shown in FIG. 2, the encoding device 200 can be configured to include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 can include an inter-prediction unit 221 and an intra-prediction unit 222. The residual processor 230 can include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 can further include a subtractor 231. The adder 250 can be called a reconstructor or a reconstructed block generator. The above-described image partitioner 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 can be configured by one or more hardware components (e.g., an encoder chipset or a processor) according to an embodiment. Also, the memory 270 can include a DPB (decoded picture buffer) and can also be configured by a digital storage medium. The hardware component can further include the memory 270 as an internal / external component.

[0048] The video segmentation unit 210 can divide the input video (or picture, frame) input to the encoding device 200 into one or more processing units. As an example, the processing unit can be called a coding unit (CU). In this case, the coding unit can be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) by a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, one coding unit can be divided into multiple coding units with a deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure can be applied first, and then the binary-tree structure and / or the ternary structure can be applied. Or, the binary-tree structure can also be applied first. The coding procedure according to the present disclosure can be performed based on the final coding unit that is no longer divided. In this case, based on the coding efficiency according to the video characteristics, etc., the largest coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units with a deeper depth so that the coding unit with the optimal size can be used as the final coding unit. Here, the coding procedure can include procedures such as prediction, conversion, and restoration described later. As another example, the processing unit can further include a prediction unit (PU: Prediction Unit) or a transform unit (TU: Transform Unit). In this case, the prediction unit and the transform unit can each be divided or partitioned from the aforementioned final coding unit.The prediction unit can be a unit of sample prediction, and the conversion unit can be a unit for deriving a conversion coefficient and / or a unit for deriving a residual signal from the conversion coefficient.

[0049] The unit can, in some cases, be used interchangeably with terms such as a block or an area. In general, an M×N block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or a pel in one picture (or video).

[0050] The encoding device 200 can subtract a prediction signal (predicted block, predicted sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from an input video signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as shown in the figure, the unit that subtracts the prediction signal (predicted block, predicted sample array) from the input video signal (original block, original sample array) within the encoder 200 can be called the subtraction unit 231. The prediction unit can perform a prediction on a processing target block (hereinafter referred to as the current block) and generate a predicted block including predicted samples for the current block. The prediction unit can determine whether intra prediction or inter prediction is applied in units of the current block or CU. The prediction unit can generate various pieces of information related to prediction, such as prediction mode information, and transmit them to the entropy encoding unit 240 as described later in the description of each prediction mode. The information related to prediction can be encoded by the entropy encoding unit 240 and output in the form of a bit stream.

[0051] The intra prediction unit 222 can predict the current block by referring to samples within the current picture. The samples to be referred to can be located adjacent to the current block or can be located remotely, depending on the prediction mode. In intra prediction, the prediction mode can include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes can include, for example, the DC mode and the Planar mode. The directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is an example, and a greater or lesser number of directional prediction modes can be used depending on the setting. The intra prediction unit 222 can also determine the prediction mode to be applied to the current block by using the prediction mode applied to the adjacent block.

[0052] The inter prediction unit 221 can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between adjacent blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the adjacent blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block can be the same or different. The temporal neighboring blocks can be called by names such as collocated reference blocks and collocated CUs (col CUs), and the reference picture including the temporal neighboring blocks can also be called a collocated picture (colPic). For example, the inter prediction unit 221 can construct a motion information candidate list based on adjacent blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction can be performed based on various prediction modes. For example, in the case of the skip mode and the merge mode, the inter prediction unit 221 can use the motion information of adjacent blocks as the motion information of the current block. In the case of the skip mode, unlike the merge mode, a residual signal may not be transmitted.In the case of the motion information prediction (motion vector prediction, MVP) mode, the motion vector of an adjacent block is used as a motion vector predictor, and the motion vector difference is signaled to indicate the motion vector of the current block.

[0053] The prediction unit 220 can generate a prediction signal based on various prediction methods described later. For example, the prediction unit can apply intra prediction or inter prediction for predicting a block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). Also, the prediction unit can be based on the intra block copy (IBC) prediction mode or the palette mode for predicting a block. The IBC prediction mode or the palette mode can be used for content video / motion video coding such as games, for example, like SCC (screen content coding). IBC basically performs prediction within the current picture, but can be performed in the same way as inter prediction in terms of deriving a reference block within the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document. The palette mode can be regarded as an example of intra coding or intra prediction. When the palette mode is applied, the sample values in the picture can be signaled based on the information regarding the palette table and the palette index.

[0054] The prediction signal generated via the prediction unit (including the inter prediction unit 221 and / or the intra prediction unit 222) may be used to generate a restored signal or may be used to generate a residual signal. The conversion unit 232 can apply a conversion technique to the residual signal to generate transform coefficients. For example, the conversion technique may include at least one of DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT means the conversion obtained from this graph when representing the relationship information between pixels by a graph. CNT means the conversion obtained based on generating a prediction signal using all previously reconstructed pixels. Also, the conversion process may be applied to a pixel block having the same size of a square or may be applied to a block of a variable size that is not square.

[0055] The quantization unit 233 quantizes the transform coefficients and transmits them to the entropy encoding unit 240. The entropy encoding unit 240 can encode the quantized signal (information regarding the quantized transform coefficients) and output it as a bitstream. The information regarding the quantized transform coefficients can be called residual information. The quantization unit 233 can reorder the quantized transform coefficients in block form into a one-dimensional vector form based on the coefficient scan order, and can also generate the information regarding the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit 240 can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit 240 can also encode, together or separately, information necessary for video / image restoration (e.g., values of syntax elements, etc.) in addition to the quantized transform coefficients. The encoded information (e.g., encoded video / video information) can be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / video information can further include information regarding various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS), etc. Also, the video / video information can further include general constraint information. In this document, the information and / or syntax elements transmitted / signaled from the encoding device to the decoding device can be included in the video / video information.The video / video information can be encoded via the encoding procedure described above and included in the bitstream. The bitstream can be transmitted via a network or stored in a digital storage medium. Here, the network can include a broadcast network and / or a communication network, etc., and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit 240 can be configured as an internal / external element of the encoding device 200 by a transmission unit (not shown) for transmission and / or a storage unit (not shown) for storage, or the transmission unit can also be included in the entropy encoding unit 240.

[0056] The quantized transform coefficients output from the quantization unit 233 can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients via the inverse quantization unit 234 and the inverse transformation unit 235, a residual signal (residual block or residual sample) can be restored. The addition unit 155 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the restored residual signal to the prediction signal output from the inter prediction unit 221 or the intra prediction unit 222. When there is no residual for the block to be processed, as in the case where the skip mode is applied, the predicted block can be used as the reconstructed block. The addition unit 250 can be called a restoration unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra prediction of the next block to be processed within the current picture and, as will be described later, can also be used for inter prediction of the next picture after passing through filtering.

[0057] On the other hand, LMCS (luma mapping with chroma scaling) can also be applied during the picture encoding and / or restoration process.

[0058] The filtering unit 260 can apply filtering to the restored signal to improve the subjective / objective image quality. For example, the filtering unit 260 can apply various filtering methods to the restored picture to generate a modified restored picture, and store the modified restored picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit 260 can generate various information related to filtering and transmit it to the entropy encoding unit 240 as will be described later in the description of each filtering method. The information related to filtering can be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0059] The modified restored picture transmitted to the memory 270 can be used as a reference picture by the inter prediction unit 221. Through this, when inter prediction is applied, the encoding device can avoid prediction mismatches between the encoding device 100 and the decoding device, and can also improve the encoding efficiency.

[0060] The DPB in memory 270 can store the corrected restored picture for use as a reference picture in the inter prediction unit 221. Memory 270 can store the motion information of the blocks for which the motion information in the current picture has been derived (or encoded) and / or the motion information of the blocks in the already restored picture. The stored motion information can be transmitted to the inter prediction unit 221 for utilization as the motion information of spatially adjacent blocks or temporally adjacent blocks. Memory 270 can store the restored samples of the restored blocks in the current picture and can transmit them to the intra prediction unit 222.

[0061] FIG. 3 is a drawing schematically explaining the configuration of a video / video decoding apparatus to which this document can be applied.

[0062] As shown in FIG. 3, the decoding apparatus 300 can be configured to include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filtering unit 350, and a memory 360. The predictor 330 can include an inter prediction unit 331 and an intra prediction unit 332. The residual processor 320 can include a dequantizer 321 and an inverse transformer 321. The above-described entropy decoder 310, residual processor 320, predictor 330, adder 340, and filtering unit 350 can be configured by one hardware component (for example, a decoder chipset or a processor) according to an embodiment. Further, the memory 360 can include a DPB (decoded picture buffer) and can also be configured by a digital storage medium. The hardware component can further include the memory 360 as an internal / external component.

[0063] If a bitstream including video / video information is input, the decoding device 300 can restore the video corresponding to the process in which the video / video information was processed by the encoding device in FIG. 3. For example, the decoding device 300 can derive units / blocks based on the block division related information obtained from the bitstream. The decoding device 300 can perform decoding using the processing units applied in the encoding device. Therefore, the processing unit for decoding can be, for example, a coding unit, and the coding unit can be divided according to a quad tree structure, a binary tree structure, and / or a ternary tree structure from a coding tree unit or a maximum coding unit. One or more transform units can be derived from the coding unit. Then, the restored video signal decoded and output via the decoding device 300 can be played back via a playback device.

[0064] The decoding device 300 can receive the signal output from the encoding device in FIG. 3 in the form of a bitstream, and the received signal can be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 can parse the bitstream to derive information (e.g., video / video information) necessary for video restoration (or picture restoration). The video / video information can further include information regarding various parameter sets such as an Adaptation Parameter Set (APS), a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), or a Video Parameter Set (VPS). Also, the video / video information can further include general constraint information. The decoding device can further decode a picture based on the information regarding the parameter set and / or the general constraint information. The signaling / received information and / or syntax elements described later in this document can be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 can decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the value of the syntax element necessary for video restoration and the quantized value of the transform coefficient regarding the residual. More specifically, the CABAC entropy decoding method receives the bin corresponding to each syntax element in the bitstream, determines a context model using the syntax element information to be decoded, the information adjacent to the decoding target block, and the decoding information of the decoding target block or the information of the symbol / bin decoded in the previous step, predicts the occurrence probability of the bin according to the determined context model, performs arithmetic decoding of the bin, and can generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model by using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit 310, the information related to prediction is provided to the prediction unit (inter prediction unit 332 and intra prediction unit 331), and the residual value obtained by performing entropy decoding in the entropy decoding unit 310, that is, the quantized transform coefficient and related parameter information, can be input to the residual processing unit 320. The residual processing unit 320 can derive a residual signal (residual block, residual sample, residual sample array). Also, among the information decoded by the entropy decoding unit 310, the information related to filtering can be provided to the filtering unit 350. On the other hand, a receiving unit (not shown) that receives the signal output from the encoding device can be further configured as an internal / external element of the decoding device 300, or the receiving unit can be a component of the entropy decoding unit 310. On the other hand, the decoding device according to this document can be called a video / video / picture decoding device, and the decoding device can be classified into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The information decoder can include the entropy decoding unit 310, and the sample decoder can include at least one of the inverse quantization unit 321, inverse transform unit 322, addition unit 340, filtering unit 350, memory 360, inter prediction unit 332, and intra prediction unit 331.

[0065] In the inverse quantization unit 321, the quantized transform coefficients can be inverse quantized to output the transform coefficients. The inverse quantization unit 321 can reorder the quantized transform coefficients in a two-dimensional block form. In this case, the reordering can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit 321 can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) to obtain the transform coefficients.

[0066] In the inverse transform unit 322, the transform coefficients are inverse transformed to obtain a residual signal (residual block, residual sample array).

[0067] The prediction unit can perform prediction on the current block and generate a predicted block including the predicted samples for the current block. The prediction unit can determine whether intra prediction or inter prediction is applied to the current block based on the information regarding the prediction output from the entropy decoding unit 310, and can determine a specific intra / inter prediction mode.

[0068] The prediction unit 320 can generate a prediction signal based on various prediction methods described later. For example, for prediction of one block, the prediction unit can apply not only intra prediction or inter prediction, but also can apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). Also, the prediction unit can be based on the intra block copy (IBC) prediction mode or the palette mode for prediction of the block. The IBC prediction mode or the palette mode can be used for content video / moving image coding such as games, for example, like SCC (screen content coding). IBC basically performs prediction within the current picture, but can be performed in the same way as inter prediction in terms of deriving a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. The palette mode can be regarded as an example of intra coding or intra prediction. When the palette mode is applied, information regarding the palette table and the palette index can be included in and signaled in the video / video information.

[0069] The intra prediction unit 331 can predict the current block by referring to samples within the current picture. The samples to be referred to can be located adjacent to or away from the current block depending on the prediction mode. In intra prediction, the prediction mode can include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 331 can also determine the prediction mode to be applied to the current block using the prediction mode applied to the adjacent block.

[0070] The inter prediction unit 332 can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between adjacent blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the adjacent blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit 332 can construct a motion information candidate list based on adjacent blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and the information regarding the prediction can include information indicating the mode of inter prediction for the current block.

[0071] The addition unit 340 can generate a restored signal (restored picture, restored block, restored sample array) by adding the obtained residual signal to a prediction signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 332 and / or the intra prediction unit 331). When there is no residual for the processing target block, as in the case where the skip mode is applied, the predicted block can be used as the restored block.

[0072] The addition unit 340 can be referred to as a restoration unit or a restoration block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and as will be described later, can be output after filtering, or can also be used for inter prediction of the next picture.

[0073] On the other hand, LMCS (luma mapping with chroma scaling) can also be applied during the picture decoding process.

[0074] The filtering unit 350 can apply filtering to the restoration signal to improve the subjective / objective image quality. For example, the filtering unit 350 can apply various filtering methods to the restored picture to generate a modified restored picture, and can transmit the modified restored picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0075] The (modified) restored picture stored in the DPB of the memory 360 can be used as a reference picture in the inter prediction unit 332. The memory 360 can store the motion information of the blocks for which the motion information in the current picture has been derived (or decoded) and / or the motion information of the blocks in the already restored picture. The stored motion information can be transmitted to the inter prediction unit 260 for utilization as the motion information of spatially adjacent blocks or temporally adjacent blocks. The memory 360 can store the restored samples of the restored blocks in the current picture and can transmit them to the intra prediction unit 331.

[0076] In this specification, the embodiments described in the filtering unit 260, the inter prediction unit 221, and the intra prediction unit 222 of the encoding device 200 can be applied to the filtering unit 350, the inter prediction unit 332, and the intra prediction unit 331 of the decoding device 300 in the same or corresponding manner, respectively.

[0077] As described above, in performing video coding, prediction is executed to increase the compression efficiency. Through this, a predicted block including prediction samples for the current block, which is a block to be coded, can be generated. Here, the predicted block includes prediction samples in the spatial domain (or pixel domain). The predicted block is also derived in the encoding device and the decoding device, and the encoding device can increase the video coding efficiency by signaling information (residual information) regarding the residual between the original block and the predicted block, which is not the original sample value of the original block, to the decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and combine the residual block and the predicted block to generate a restored block including restored samples, and can generate a restored picture including the restored block.

[0078] The residual information can be generated through the conversion and quantization procedures. For example, an encoding device can derive a residual block between the original block and the predicted block, execute a conversion procedure on the residual samples (residual sample array) included in the residual block to derive conversion coefficients, execute a quantization procedure on the conversion coefficients to derive quantized conversion coefficients, and thus signal the relevant residual information (via a bitstream) to a decoding device. Here, the residual information can include information such as the value information, position information, conversion technique, conversion kernel, quantization parameter, etc. of the quantized conversion coefficients. A decoding device can execute an inverse quantization / inverse conversion procedure based on the residual information to derive residual samples (or a residual block). A decoding device can generate a restored picture based on the predicted block and the residual block. Also, an encoding device can inverse quantize / inverse transform the quantized conversion coefficients for reference in inter-prediction of subsequent pictures to derive a residual block, and generate a restored picture based on this.

[0079] On the one hand, as described above, when making a prediction for the current block, intra prediction or inter prediction can be applied. As one embodiment, when applying inter prediction to the current block, the prediction unit (more specifically, the inter prediction unit) of the encoding / decoding device can perform inter prediction on a block-by-block basis to derive prediction samples. Inter prediction can indicate a prediction derived in a manner dependent on data elements (e.g., sample values, motion information, etc.) of pictures other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) for the current block can be induced based on a reference block (reference sample array) specified by a motion vector on a reference picture pointed to by a reference picture index. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.) information. When inter prediction is applied, the neighboring blocks can include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block can be called by names such as a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block can also be called a collocated picture (colPic). For example, a motion information candidate list can be constructed based on the neighboring blocks of the current block, and flag or index information indicating which candidate is selected (used) can be signaled to derive the motion vector and / or reference picture index of the current block.Inter prediction is performed based on various prediction modes. For example, in the case of skip mode and merge mode, the motion information of the current block may be the same as the motion information of the selected neighboring block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the selected neighboring block can be used as a motion vector predictor, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0080] The motion information can include L0 motion information and / or L1 motion information according to an inter prediction type (such as L0 prediction, L1 prediction, Bi prediction, etc.). The motion vector in the L0 direction may be referred to as an L0 motion vector or MVL0, and the motion vector in the L1 direction may be referred to as an L1 motion vector or MVL1. The prediction based on the L0 motion vector may be called L0 prediction, the prediction based on the L1 motion vector may be called L1 prediction, and the prediction based on both the L0 motion vector and the L1 motion vector may be called bi (Bi) prediction. Here, the L0 motion vector can indicate a motion vector related to the reference picture list L0 (L0), and the L1 motion vector can indicate a motion vector related to the reference picture list L1 (L1). The reference picture list L0 can include a reference picture that is earlier than the current picture in output order, and the reference picture list L1 can include a picture that is later than the current picture in output order. The earlier picture may be referred to as a forward (reference) picture, and the later picture may be referred to as a backward (reference) picture. The reference picture list L0 can further include a picture that is later than the current picture in output order as a reference picture. In this case, the earlier picture is indexed first within the reference picture list L0, and the later picture may be indexed thereafter. The reference picture list L1 can further include a picture that is earlier than the current picture in output order as a reference picture. In this case, the later picture is indexed first within the reference picture list 1, and the earlier picture may be indexed thereafter. Here, the output order may correspond to the POC (picture order count) order (order).

[0081] FIG. 4 shows an example of a schematic video / video encoding procedure to which the embodiments of this document are applicable. In FIG. 4, S400 can be executed by the prediction unit 220 of the encoding device detailed in FIG. 2, S410 can be executed by the residual processing unit 230, and S420 can be executed by the entropy encoding unit 240. S400 can include the inter / intra prediction procedures described in this document, S410 can include the residual processing procedures described in this document, and S420 can include the encoding procedures of the information described in this document.

[0082] Referring to FIG. 4, the video / video encoding procedure is not only a procedure of encoding information (e.g., prediction information, residual information, partitioning information, etc.) for restoring a picture schematically as shown in the description regarding FIG. 2 and outputting it in the form of a bitstream, but also can include a procedure of generating a restored picture for the current picture, and a procedure (optional) of applying in-loop filtering to the restored picture. The encoding device can derive (modified) residual samples from the quantized transform coefficients via the inverse quantization unit 234 and the inverse transform unit 235, and can generate a restored picture based on the prediction samples that are the output of S400 and the (modified) residual samples. The restored picture generated in this way can be the same as the restored picture generated by the decoding device described above. A restored picture modified via the in-loop filtering procedure for the restored picture is generated, which is stored in the decoded picture buffer or memory 270, and can be used as a reference picture in the inter prediction procedure when encoding subsequent pictures, similar to the case of the decoding device. As described above, in some cases, part or all of the in-loop filtering procedure can be omitted. When the in-loop filtering procedure is performed, the (in-loop) filtering-related information (parameters) can be encoded by the entropy encoding unit 240 and output in the form of a bitstream, and the decoding device can perform the in-loop filtering procedure in the same way as the encoding device based on the filtering-related information.

[0083] Through such in-loop filtering procedures, noises generated during image / video coding, such as blocking artifacts and ringing artifacts, can be reduced, and subjective / objective visual quality can be improved. Also, by both the encoding device and the decoding device performing the in-loop filtering procedure, the encoding device and the decoding device can derive the same prediction result, enhancing the reliability of picture coding and reducing the amount of data that must be transmitted for picture coding.

[0084] As described above, picture restoration procedures are performed not only in the decoding device but also in the encoding device. Restoration blocks can be generated based on intra prediction / inter prediction for each block unit, and a restored picture including the restoration blocks can be generated. When the current picture / slice / tile group is an I picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based on intra prediction only. On the other hand, when the current picture / slice / tile group is a P or B picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based on intra prediction or inter prediction. In this case, inter prediction may be applied to some blocks within the current picture / slice / tile group, and intra prediction may be applied to the remaining some blocks. The color components of a picture may include a luma component and a chroma component, and unless explicitly restricted in this document, the methods and embodiments proposed in this document can be applied to the luma component and the chroma component.

[0085] FIG. 5 shows an example of a schematic video / video decoding procedure to which the embodiments of this document are applicable. In FIG. 5, S500 can be executed by the entropy decoding unit 310 of the decoding apparatus detailed in FIG. 3, S510 can be executed by the prediction unit 330, S520 can be executed by the residual processing unit 320, S530 can be executed by the addition unit 340, and S540 can be executed by the filtering unit 350. S500 can include the decoding procedure of the information described in this document, S510 can include the inter / intra prediction procedure described in this document, S520 can include the residual processing procedure described in this document, S530 can include the block / picture restoration procedure described in this document, and S540 can include the in-loop filtering procedure described in this document.

[0086] Referring to FIG. 5, the picture decoding procedure can generally include a video / video information acquisition procedure (S500) from the bitstream (through decoding) as shown in the description regarding FIG. 3, a picture restoration procedure (S510 to S530), and an in-loop filtering procedure (S540) for the restored picture. The picture restoration procedure can be executed based on the predicted samples and the residual samples obtained through the inter / intra prediction (S510) and the residual processing (S520, inverse quantization and inverse transformation for the quantized transform coefficients) processes described in this document. Through the in-loop filtering procedure for the restored picture generated through the picture restoration procedure, a modified restored picture is generated, and the modified restored picture is output as the decoded picture, and is also stored in the decoded picture buffer or memory 360 of the decoding apparatus, and can be used as a reference picture in the inter prediction procedure when decoding subsequent pictures.

[0087] Depending on the case, the in-loop filtering procedure may be omitted. In this case, the restored picture is output as the decoded picture, stored in the decoded picture buffer or memory 360 of the decoding apparatus, and can be used as a reference picture in the inter prediction procedure when decoding subsequent pictures. The in-loop filtering procedure (S540) may include, as described above, a deblocking filtering procedure, a SAO (sample adaptive offset) procedure, an ALF (adaptive loop filter) procedure, and / or a bilateral filter procedure, etc., and some or all of them may be omitted. Also, one or a part of the deblocking filtering procedure, SAO (sample adaptive offset) procedure, ALF (adaptive loop filter) procedure, and bilateral filter procedure may be sequentially applied, or all of them may be sequentially applied. For example, after the deblocking filtering procedure is applied to the restored picture, the SAO procedure may be executed. Or, for example, after the deblocking filtering procedure is applied to the restored picture, the ALF procedure may be executed. This can be similarly executed in the encoding apparatus.

[0088] On the other hand, as described above, the encoding apparatus can perform entropy encoding based on various encoding methods such as exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. Also, the decoding apparatus can perform entropy decoding based on coding methods such as exponential Golomb coding, CAVLC or CABAC. Hereinafter, the entropy encoding / decoding procedure will be described.

[0089] FIG. 6 schematically shows an example of an entropy encoding method to which the embodiments of this document are applicable, and FIG. 7 schematically shows an entropy encoding unit in an encoding device. The entropy encoding unit in the encoding device of FIG. 7 can be applied to be identical or corresponding to the entropy encoding unit 240 of the encoding device 200 in FIG. 2 described above.

[0090] Referring to FIGS. 6 and 7, an encoding device (entropy encoding unit) can execute an entropy coding procedure for video / video information. The video / video information may include partitioning-related information, prediction-related information (e.g., inter / intra prediction partition information, intra prediction mode information, inter prediction mode information, etc.), residual information, in-loop filtering-related information, etc., or may include various syntax elements related thereto. Entropy coding can be executed in units of syntax elements. S600 to S610 can be executed by the entropy encoding unit 240 of the encoding device 200 in FIG. 2 described above.

[0091] The encoding device can perform binarization on a target syntax element (S600). Here, the binarization can be based on various binarization methods such as Truncated Rice binarization process, Fixed-length binarization process, etc., and the binarization method for the target syntax element can be predefined. The binarization procedure can be executed by the binarization unit 242 in the entropy encoding unit 240.

[0092] The encoding device can perform entropy encoding on a target syntax element (S610). The encoding device can encode the bin string of the target syntax element based on entropy coding techniques such as CABAC (context-adaptive arithmetic coding) or CAVLC (context-adaptive variable length coding), based on regular coding (context-based), or based on bypass coding, and its output can be included in the bitstream. The entropy encoding procedure can be executed by the entropy encoding processing unit 243 in the entropy encoding unit 240. As described above, the bitstream can be transmitted to the decoding device via a (digital) storage medium or a network.

[0093] FIG. 8 schematically shows an example of an entropy decoding method to which the embodiments of this document are applicable, and FIG. 9 schematically shows the entropy decoding unit in the decoding device. The entropy decoding unit in the decoding device of FIG. 9 can be applied to be identical or corresponding to the entropy decoding unit 310 of the decoding device 300 in FIG. 3 described above.

[0094] Referring to FIGS. 8 and 9, the decoding device (entropy decoding unit) can decode the encoded video / video information. The video / video information can include partitioning-related information, prediction-related information (e.g., inter / intra prediction division information, intra prediction mode information, inter prediction mode information, etc.), residual information, in-loop filtering-related information, etc., or can include various syntax elements related thereto. Entropy coding can be performed in units of syntax elements. S800 to S810 can be executed by the entropy decoding unit 310 of the decoding device 300 in FIG. 3 described above.

[0095] The decoding device can perform binarization on a target syntax element (S800). Here, the binarization can be based on various binarization methods such as Truncated Rice binarization process, Fixed-length binarization process, etc., and the binarization method for the target syntax element can be predefined. The decoding device can derive an available bit string (bit string candidate) for the available values of the target syntax element through the binarization procedure. The binarization procedure can be executed by the binarization unit 312 in the entropy decoding unit 310.

[0096] The decoding device can perform entropy decoding on a target syntax element (S810). The decoding device sequentially decodes and parses each bin for the target syntax element from the input bits in the bitstream, and compares the derived bit string with the available bit strings for the syntax element. If the derived bit string is the same as one of the available bit strings, the value corresponding to the bit string can be derived as the value of the syntax element. If not, after further parsing the next bit in the bitstream, the above-described procedure can be re-executed. Through such a process, without using start bits or end bits for specific information (specific syntax elements) in the bitstream, the information can be signaled using variable-length bits. Through this, relatively fewer bits can be allocated for lower values, and the overall coding efficiency can be improved.

[0097] The decoding device can decode each bin in the bit string based on context or bypass based on entropy coding techniques such as CABAC or CAVLC. Here, the bitstream can contain various information for video / video decoding as described above. As previously mentioned, the bitstream can be transmitted to the decoding device via a (digital) storage medium or a network.

[0098] Figure 10 exemplarily shows a hierarchical structure for coded video.

[0099] Referring to Figure 10, the coded video is divided into a video decoding process of the video, a VCL (Video Coding Layer), a lower system that transmits and stores the encoded information, and a NAL (Network Abstraction Layer) that exists between the VCL and the lower system and is responsible for network adaptation functions.

[0100] In the VCL, it is possible to generate VCL data including compressed video data (slice data), or generate a parameter set including information such as a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), a Video Parameter Set (VPS), or a SEI (Supplemental Enhancement Information) message that is additionally required in the video decoding process.

[0101] In the NAL, a NAL unit can be generated by adding header information (NAL unit header) to the RBSP (Raw Byte Sequence Payload) generated in the VCL. At this time, the RBSP refers to slice data, parameter sets, SEI messages, etc. generated in the VCL. The NAL unit header may include NAL unit type information specified by the RBSP data included in the NAL unit.

[0102] In addition, NAL units can be classified into VCL NAL units and Non-VCL NAL units according to the RBSP generated by VCL. A VCL NAL unit may mean a NAL unit containing information (slice data) for video, and a Non-VCL NAL unit may mean a NAL unit containing information (parameter set or SEI message) necessary for decoding video.

[0103] VCL NAL units and Non-VCL NAL units can be transmitted via a network with header information attached according to the data standard of the lower system. For example, NAL units can be transformed into data forms of predetermined standards such as the H.266 / VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted via various networks.

[0104] As described above, the NAL unit type can be specified according to the RBSP data structure (structure) included in the corresponding NAL unit, and information about such NAL unit types can be stored in the NAL unit header and signaled.

[0105] For example, depending on whether the NAL unit contains information (slice data) for video, it can be roughly classified into a VCL NAL unit type and a Non-VCL NAL unit type. The VCL NAL unit type can be classified according to the nature and type of the picture contained in the VCL NAL unit, and the Non-VCL NAL unit type can be classified according to the type of parameter set, etc.

[0106] The following is an example of the NAL unit type specified according to the type of parameter set included in the Non-VCL NAL unit type, etc.

[0107] - APS (Adaptation Parameter Set) NAL unit: Type for NAL units containing APS

[0108] - DPS (Decoding Parameter Set) NAL unit: Type for NAL units containing DPS

[0109] - VPS (Video Parameter Set) NAL unit: Type for NAL units containing VPS

[0110] - SPS (Sequence Parameter Set) NAL unit: Type for NAL units containing SPS

[0111] - PPS (Picture Parameter Set) NAL unit: Type for NAL units containing PPS

[0112] - PH (Picture header) NAL unit: Type for NAL units containing PH

[0113] The above-mentioned NAL unit types have syntax information for the NAL unit type, and the syntax information can be stored in the NAL unit header and signaled. For example, the syntax information is nal_unit_type, and the NAL unit type can be specified by the nal_unit_type value.

[0114] On the one hand, as described above, one picture may include a plurality of slices, and one slice may include a slice header and slice data. In this case, one picture header may be further added for a plurality of slices (a set of slice headers and slice data) within one picture. The picture header (picture header syntax) may include information / parameters that can be commonly applied to the picture. In this document, tile groups may be used interchangeably with or in place of slices or pictures. Also, in this document, tile group headers may be used interchangeably with or in place of slice headers or picture headers.

[0115] The slice header (slice header syntax) may include information / parameters that can be commonly applied to the slice. APS (APS syntax) or PPS (PPS syntax) may include information / parameters that can be commonly applied to one or more slices or pictures. SPS (SPS syntax) may include information / parameters that can be commonly applied to one or more sequences. VPS (VPS syntax) may include information / parameters that can be commonly applied to multiple layers. DPS (DPS syntax) may include information / parameters that can be commonly applied to the entire video. DPS may include information / parameters related to the concatenation of CVS (coded video sequence). In this document, the high level syntax (HLS) may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, picture header syntax, and slice header syntax.

[0116] In this document, the video / video information encoded by the encoding device and signaled in the form of a bitstream to the decoding device may include not only information related to partitioning within a picture, intra / inter prediction information, residual information, in-loop filtering information, etc., but also information included in the slice header, information included in the picture header, information included in the APS, information included in the PPS, information included in the SPS, information included in the VPS, and / or information included in the DPS. Further, the video / video information may further include information of the NAL unit header.

[0117] As described above, HLS (High level syntax) can be coded / signaled for video / video coding. In this document, the video / video information may include HLS. For example, a coded picture can be composed of one or more slices. The parameters describing the coded picture are signaled within the picture header (PH), and the parameters describing the slice can be signaled within the slice header (SH). The PH can be sent to its NAL unit type. The SH can be present at the start of the NAL unit containing the payload of the slice (i.e., slice data). The detailed content regarding the syntax and semantics of the PH and SH is as disclosed in the VVC standard. Each picture can be associated with a PH. The picture can be composed of different types of slices which are intra-coded slices (i.e., I slices) and inter-coded slices (i.e., P-slices and B-slices). As a result, the PH may include the syntax elements necessary for the intra-slices of the picture and the inter-slices of the picture.

[0118] On the one hand, generally, one NAL unit type can be set for one picture. The NAL unit type can be signaled via nal_unit_type in the NAL unit header of the NAL unit including the slice. nal_unit_type is syntax information for specifying the NAL unit type, that is, as shown in Table 1 or Table 2 below, it can specify the type of the RBSP data structure included in the NAL unit.

[0119] The following Table 1 shows an example of the codes of the NAL unit types and the classes of the NAL unit types.

[0120]

Table 1

[0121] Alternatively, as an example, the codes of the NAL unit types and the classes of the NAL unit types can also be defined as shown in Table 2 below.

[0122]

Table 2-1

[0123]

Table 2-2

[0124] As shown in Table 1 or Table 2 above, the name and value of the NAL unit type can be identified by the RBSP data structure included in the NAL unit, and the NAL unit can be classified into a VCL NAL unit type and a Non-VCL NAL unit type according to whether the NAL unit contains information (slice data) for video. The VCL NAL unit type can be classified according to the nature and type of the picture, etc., and the Non-VCL NAL unit type can be classified according to the type of parameter set, etc. For example, the NAL unit type can be identified according to the nature and type of the picture included in the VCL NAL unit as follows.

[0125] TRAIL: Indicates the type for a NAL unit containing slice data coded for a trailing picture / sub-picture. For example, nal_unit_type can be defined as TRAIL_NUT, and the value of nal_unit_type can be identified as 0.

[0126] Here, a trailing picture refers to a picture that follows a picture that is randomly accessible in both the output order and the decoding order. A trailing picture can be a non-IRAP picture that follows a related IRAP picture in the output order and is not an STSA picture. For example, a trailing picture related to an IRAP picture follows the IRAP picture in the decoding order. A picture that follows a related IRAP picture in the output order and precedes the related IRAP picture in the decoding order is not allowed.

[0127] STSA (Step-wise Temporal Sub-layer Access): Indicates the type for a NAL unit containing slice data coded for an STSA picture / sub-picture. For example, nal_unit_type can be defined as STSA_NUT, and the value of nal_unit_type can be identified as 1.

[0128] Here, an STSA picture is a picture in a bitstream that supports temporal scalability and allows switching between temporal sublayers, and in a lower sublayer, it is a picture that indicates a position where upswitching to a higher sublayer one level above the lower sublayer is possible. An STSA picture does not use a picture that has the same TemporalId and is in the same layer as the STSA picture for inter-prediction reference. In the decoding order, a picture that follows an STSA picture with the same TemporalId and in the same layer as the STSA picture does not use a picture that is before the STSA picture with the same TemporalId and in the same layer as the STSA picture in the decoding order for inter-prediction reference. An STSA picture is an STSA picture that enables upswitching to the sublayer containing the STSA picture immediately in the lower sublayer. In this case, the picture to be coded must not belong to the lowest sublayer. That is, an STSA picture must always have a TemporalId greater than 0.

[0129] RADL (random access decodable leading (picture)): It indicates the type for an NAL unit containing coded slice data of an RADL picture / sublpicture. For example, nal_unit_type can be defined as RADL_NUT, and the value of nal_unit_type can be specified as 2.

[0130] Here, all RADL pictures are leading pictures. RADL pictures are not used as reference pictures for the decoding process of the trailing pictures of the same related IRAP picture. Specifically, a RADL picture having a nuh_layer_id such as layerId is not used as a reference picture for the decoding process of a picture having a nuh_layer_id such as layerId that follows the IRAP picture related to the RADL picture in output order. When the field_seq_flag (i.e., sps_field_seq_flag) is 0, all RADL pictures (i.e., if there are RADL pictures) precede all non-leading pictures of the same related IRAP picture in decoding order. On the other hand, a leading picture refers to a picture that precedes the related IRAP picture in output order.

[0131] RASL (random access skipped leading (picture)): Indicates the type for a NAL unit containing coded slice data of a RASL picture / sub-picture. For example, nal_unit_type can be defined as RASL_NUT, and the value of nal_unit_type can be specified as 3.

[0132] Here, all RASL pictures are the leading pictures of the associated CRA pictures. If the associated CRA picture has a NoOutputBeforeRecoveryFlag whose value is 1, since the RASL picture may contain a reference to a picture that does not exist in the bitstream, the RASL picture may not be output and may not be decoded accurately. The RASL picture is not used as a reference picture for the decoding process of non-RASL pictures of the same layer. However, the RADL sub-pictures in the RASL pictures of the same layer can be used for inter prediction for the collocated RADL sub-pictures in the RADL pictures associated with the same CRA picture as the RASL picture. When field_seq_flag (i.e., sps_field_seq_flag) is 0, all RASL pictures (i.e., if RASL pictures exist) precede all non-leading pictures of the same associated CRA picture in decoding order.

[0133] There may be reserved nal_unit_types for non-IRAP VCL NAL unit types. For example, nal_unit_type can be defined as RSV_VCL_4 to RSV_VCL_6, and the values of nal_unit_type can be specified as 4 to 6 respectively.

[0134] Here, IRAP (intra random access point) is information indicating an NAL unit for a picture where random access is possible. The IRAP picture can be a CRA picture or an IDR picture. For example, the IRAP picture refers to a picture having an NAL unit type defined with nal_unit_type being IDR_W_RADL, IDR_N_LP, CRA_NU as in Table 1 or Table 2 above, and the values of nal_unit_type can be specified as 7 to 9 respectively.

[0135] The IRAP picture does not use the reference picture in the same layer for inter prediction during the decoding process. In other words, the IRAP picture does not refer to other pictures other than itself for inter prediction during the decoding process. In the decoding order, the first picture in the bitstream becomes an IRAP or GDR picture. For a single-layer bitstream, if the parameter set required when it is necessary to be referred to is available, all subsequent non-RASL pictures and IRAP pictures of the CLVS (coded layer video sequence) in the decoding order can be accurately decoded without performing the decoding process of the pictures preceding the IRAP picture in the decoding order.

[0136] The value of the mixed_nalu_types_in_pic_flag for the IRAP picture is 0. When the value of the mixed_nalu_types_in_pic_flag for the picture is 0, one slice in the picture may have a NAL unit type (nal_unit_type) within the range from IDR_W_RADL to CRA_NUT (for example, the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), and all other slices in the picture may have the same NAL unit type (nal_unit_type). In this case, the picture can be regarded as an IRAP picture.

[0137] IDR (instantaneous decoding refresh): Indicates the type for the NAL unit containing the coded slice data of the IDR picture / subpicture. For example, the nal_unit_type for the IDR picture / subpicture can be defined as IDR_W_RADL or IDR_N_LP, and the values of nal_unit_type can be specified as 7 or 8 respectively.

[0138] Here, an IDR picture does not use inter prediction in the decoding process (i.e., does not refer to other pictures other than itself for inter prediction), and can be the first picture in the decoding order within the bitstream, or can be shown later in the bitstream (i.e., later than the first). Each IDR picture is the first picture of a CVS (coded video sequence) in the decoding order. For example, if an IDR picture has a correlation with a decodable leading picture, the NAL unit type of the IDR picture may be indicated as IDR_W_RADL, and if the IDR picture has no correlation with a leading picture, the NAL unit type of the IDR picture may be indicated as IDR_N_LP. That is, an IDR picture with an NAL unit type of IDR_W_RADL may not have a related RASL picture existing in the bitstream, but may have a related RADL picture in the bitstream. An IDR picture with an NAL unit type of IDR_N_LP does not have a related leading picture existing in the bitstream.

[0139] CRA (clean random access): Indicates the type for a NAL unit containing coded slice data of a CRA picture / subpicture. For example, nal_unit_type can be defined as CRA_NUT, and the value of nal_unit_type can be specified as 9.

[0140] Here, a CRA picture does not use inter prediction in the decoding process (i.e., without referring to other pictures other than itself for inter prediction), and it can be the first picture in the decoding order within the bitstream or be shown later in the bitstream (i.e., later than the first). A CRA picture may have related RADL or RASL pictures existing within the bitstream. For a CRA picture with the value of NoOutputBeforeRecoveryFlag being 1, the related RASL picture may not be output by the decoder. This is because it may contain references to pictures that do not exist within the bitstream, and in this case, it cannot be decoded.

[0141] GDR (gradual decoding refresh): It indicates the type for an NAL unit containing coded slice data of a GDR picture / subpicture. For example, nal_unit_type can be defined as GDR_NUT, and the value of nal_unit_type can be specified as 10.

[0142] Here, the value of pps_mixed_nalu_types_in_pic_flag for a GDR picture can be 0. When the value of pps_mixed_nalu_types_in_pic_flag for a picture is 0 and one slice within the picture has the NAL unit type of GDR_NUT, all other slices within the picture shall have the value of the same NAL unit type (nal_unit_type). In this case, the picture can become a GDR picture after receiving the first slice.

[0143] Also, for example, according to the type of parameters included in a Non-VCL NAL unit, the NAL unit type can be identified, and as shown in Table 1 or Table 2 above, a VPS_NUT indicating the type for an NAL unit including a video parameter set, an SPS_NUT indicating the type for an NAL unit including a sequence parameter set, a PPS_NUT indicating the type for an NAL unit including a picture parameter set, an NAL unit type (nal_unit_type) such as a PH_NUT indicating the type for an NAL unit including a picture header may be included.

[0144] On the other hand, a bitstream that supports temporal scalability (or a temporally scalable bitstream) includes information for a temporally scaled temporal layer. The information for the temporal layer can be the identification information of the temporal layer specified by the temporal scalability of the NAL unit. For example, the identification information of the temporal layer can use the temporal_id syntax information, and the temporal_id syntax information can be stored in the NAL unit header in an encoding device and signaled to a decoding device. Hereinafter, in this specification, the temporal layer may also be referred to as a sub-layer or a temporal sub-layer or a temporal scalable layer.

[0145] FIG. 11 is a diagram showing the temporal layer structure for an NAL unit in a bitstream that supports temporal scalability.

[0146] When the bitstream supports temporal scalability, the NAL units included in the bitstream have identification information for the temporal layer (e.g., temporal_id). As an example, a temporal layer composed of NAL units with a temporal_id value of 0 can provide the lowest temporal scalability, and a temporal layer composed of NAL units with a temporal_id value of 2 can provide the highest temporal scalability.

[0147] In FIG. 11, the box marked with I refers to an I picture, and the box marked with B refers to a B picture. Also, the arrows indicate the reference relationship regarding whether a picture references other pictures.

[0148] As shown in FIG. 11, the NAL units of the temporal layer with a temporal_id value of 0 are reference pictures that can be referenced by the NAL units of the temporal layer with a temporal_id value of 0, 1, or 2. The NAL units of the temporal layer with a temporal_id value of 1 are reference pictures that can be referenced by the NAL units of the temporal layer with a temporal_id value of 1 or 2. The NAL units of the temporal layer with a temporal_id value of 2 may be reference pictures that can be referenced by the NAL units of the same temporal layer, i.e., the temporal layer with a temporal_id value of 2, or may be non-reference pictures that are not referenced by other pictures.

[0149] If, as shown in FIG. 11, the NAL units of the temporal layer with a temporal_id value of 2, i.e., the highest temporal layer, are non-reference pictures, such NAL units can be extracted (or removed) from the bitstream without affecting other pictures during the decoding process.

[0150] On one hand, among the above-mentioned NAL unit types, the IDR and CRA types indicate information of NAL units including pictures that allow random access (or splicing), that is, RAP (Random Access Point) or IRAP (Intra Random Access Point) pictures that become random access points. In other words, an IRAP picture can be an IDR or CRA picture and can only contain I slices. In the bitstream, the first picture in the decoding order is an IRAP picture.

[0151] If an IRAP picture (IDR or CRA picture) is included in the bitstream, there may be pictures whose output order precedes that of the IRAP picture but whose decoding order follows it. Such pictures are called Leading Pictures (LP).

[0152] Figure 12 is a diagram for explaining pictures that allow random access.

[0153] Pictures that allow random access, that is, RAP or IRAP pictures that become random access points, are the first pictures in the decoding order in the bitstream during random access and only contain I slices.

[0154] Figure 12 shows the output order (output order or display order) and decoding order of pictures. As shown, the output order and decoding order of pictures may be different from each other. For convenience, pictures are described by being divided into predetermined groups.

[0155] Pictures belonging to the first group (I) indicate pictures that precede in all of the output order and decoding order compared to IRAP pictures, and pictures belonging to the second group (II) indicate pictures that precede in the output order compared to IRAP pictures but follow in the decoding order. Pictures of the third group (III) follow in both the output order and decoding order compared to IRAP pictures.

[0156] Pictures of the first group (I) can be decoded and output regardless of the IRAP pictures.

[0157] Pictures belonging to the second group (II) that are output prior to the IRAP pictures are called leading pictures, and leading pictures may cause problems in the decoding process when the IRAP picture is used as a random access point.

[0158] Pictures belonging to the third group (III) whose output and decoding order follow the IRAP pictures are called normal pictures. Normal pictures are not used as reference pictures for leading pictures.

[0159] The random access point where random access occurs in the bitstream is the IRAP picture, and random access starts while the first picture of the second group (II) is being output.

[0160] Figure 13 is a diagram for explaining the IDR picture.

[0161] An IDR picture is a picture that becomes a random access point when a Group of Picture has a closed structure. Since an IDR picture is an IRAP picture as described above, it only contains I slices and may be the first picture in the decoding order in the bitstream or may come in the middle of the bitstream. When an IDR picture is decoded, all the reference pictures stored in the DPB (decoded picture buffer) are marked as "unused for reference".

[0162] The bars shown in FIG. 13 represent pictures, and the arrows indicate the reference relationships regarding whether a picture can use other pictures as reference pictures. The x marks shown on the arrows indicate that the picture cannot reference the picture pointed to by the arrow.

[0163] As shown, the picture with a POC of 32 is an IDR picture. The pictures with POCs from 25 to 31 and output before the IDR picture are leading pictures 1310. The pictures with POCs of 33 or more correspond to normal pictures 1320.

[0164] In the output order, the leading pictures 1310 preceding the IDR picture can use the IDR picture and other leading pictures as reference pictures, but in the output order and decoding order, they cannot use the past pictures 1330 preceding the leading pictures 1310 as reference pictures.

[0165] In the output order and decoding order, the normal pictures 1320 following the IDR picture can be decoded by referring to the IDR picture, leading pictures, and other normal pictures.

[0166] FIG. 14 is a diagram for explaining a CRA picture.

[0167] A CRA picture is a picture that becomes a random access point when the Group of Picture has an open structure. As described above, since a CRA picture is also an IRAP picture, it only contains I slices and may be the first picture in the bitstream in decoding order or may be in the middle of the bitstream for normal play.

[0168] The bars shown in FIG. 14 represent pictures, and the arrows indicate the reference relationship regarding whether a picture can use other pictures as reference pictures. The x mark displayed on the arrow indicates that the picture or pictures cannot reference the picture pointed to by the arrow.

[0169] In output order, the leading picture 1410 preceding the CRA picture can use all of the CRA picture, other leading pictures, and the past pictures 1430 preceding the leading picture 1410 in output order and decoding order as reference pictures.

[0170] On the contrary, in output order and decoding order, the normal picture 1420 following the CRA picture can be decoded by referring to the CRA picture and other normal pictures. The normal picture 1420 may not use the leading picture 1410 as a reference picture.

[0171] On the one hand, in the VVC standard, the picture to be coded (i.e., the current picture) can include slices of other NAL unit types. Whether the current picture includes slices of other NAL unit types can be indicated based on the syntax element mixed_nalu_types_in_pic_flag. For example, when the current picture includes slices of other NAL unit types, the value of the syntax element mixed_nalu_types_in_pic_flag can be indicated as 1. At this time, the current picture must refer to a PPS that includes mixed_nalu_types_in_pic_flag with a value of 1. The semantics of the flag (mixed_nalu_types_in_pic_flag) are as follows.

[0172] When the value of the syntax element mixed_nalu_types_in_pic_flag is 1, it can indicate that each picture referring to the PPS has one or more VCL NAL units, the VCL NAL units do not have the same value of NAL unit type (nal_unit_type), and the picture is not an IRAP picture.

[0173] When the value of the syntax element mixed_nalu_types_in_pic_flag is 0, it can indicate that each picture referring to the PPS has one or more VCL NAL units, and the VCL NAL units of each picture referring to the PPS have the same value of NAL unit type (nal_unit_type).

[0174] When the value of no_mixed_nalu_types_in_pic_constraint_flag is 1, the value of mixed_nalu_types_in_pic_flag must be 0. The no_mixed_nalu_types_in_pic_constraint_flag syntax element indicates the constraint regarding whether the value of mixed_nalu_types_in_pic_flag for a picture must be 0. For example, based on the no_mixed_nalu_types_in_pic_constraint_flag information signaled from a higher-level syntax (e.g., PPS) or a syntax containing constraint-related information (e.g., GCI; general constraints information), it can be determined whether the value of mixed_nalu_types_in_pic_flag must be 0.

[0175] For each slice having a NAL unit type value nalUnitTypeA within the range from IDR_W_RADL to CRA_NUT (e.g., the NAL unit type values are from 7 to 9 in Table 1 or Table 2) in a picture picA that also includes one or more slices having other NAL unit types (i.e., when the value of mixed_nalu_types_in_pic_flag for picture picA is 1), the following can be applied.

[0176] - The slice must belong to a subpicture subpicA for which the value of the corresponding subpic_treated_as_pic_flag[i] is 1. Here, subpic_treated_as_pic_flag[i] is information regarding whether the i-th subpicture of each coded picture in CLVS is treated as a picture in the decoding process excluding the in-loop filtering operation. For example, when the value of subpic_treated_as_pic_flag[i] is 1, it may indicate that the i-th subpicture is treated as a picture in the decoding process excluding the in-loop filtering operation. Alternatively, when the value of subpic_treated_as_pic_flag[i] is 0, it may indicate that the i-th subpicture is not treated as a picture in the decoding process excluding the in-loop filtering operation.

[0177] - The slice must not belong to a subpicture of picA that contains a VCL NAL unit having a NAL unit type (nal_unit_type) different from nalUnitTypeA.

[0178] - For all subsequent PUs in CLVS in decoding order, RefPicList[0] or RefPicList[1] of the slice within subpicA should not contain pictures that precede picA in decoding order with active entries.

[0179] To operate the concepts as described above, it can be specified as follows. For example, for the VCL NAL units of a specific picture, it can be applied as follows.

[0180] - When the value of mixed_nalu_types_in_pic_flag is 0, the value of the NAL unit type (nal_unit_type) must be the same for all coded slice NAL units within the picture. A picture or a PU may be considered to have the same NAL unit type as the coded slice NAL units within the picture or PU.

[0181] - Otherwise (when the value of mixed_nalu_types_in_pic_flag is 1), all one or more VCL NAL units must have a specific value of NAL unit type within the range from IDR_W_RADL to CRA_NUT (e.g., the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), and all other VCL NAL units must have a specific value of NAL unit type within the range from TRAIL_NUT to RSV_VCL_6 (e.g., the value of the NAL unit type is from 0 to 6 in Table 1 or Table 2 above), or must have the same NAL unit type as GRA_NUT.

[0182] In the current VVC standard, for pictures with mixed NAL unit types, there may be at least the following problems.

[0183] 1. When a picture contains IDR and non-IRAP NAL units, if the signaling for the RPL (reference picture list) is present in the slice header, such signaling must also be present in the header of the IDR slice. The RPL signaling is present in the slice header of the IDR slice when the value of sps_idr_rpl_present_flag is 1. Currently, even when there is one or more pictures with a mixed NAL unit type, the value of such a flag (sps_idr_rpl_present_flag) can be 0. Here, the sps_idr_rpl_present_flag syntax element can indicate whether the RPL syntax element can be present in the slice header of a slice with a NAL unit type such as IDR_N_LP or IDR_W_RADL. For example, when the value of sps_idr_rpl_present_flag is 1, it can indicate that the RPL syntax element can be present in the slice header of a slice with a NAL unit type such as IDR_N_LP or IDR_W_RADL. Alternatively, when the value of sps_idr_rpl_present_flag is 0, it can indicate that the RPL syntax element is not present in the slice header of a slice with a NAL unit type such as IDR_N_LP or IDR_W_RADL.

[0184] 2. When the current picture refers to a PPS where the value of mixed_nalu_types_in_pic_flag is 1, one or more of the VCL NAL units of the current picture must all have a NAL unit type of a specific value within the range from IDR_W_RADL to CRA_NUT (e.g., the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), and all other VCL NAL units must either have a NAL unit type of a specific value within the range from TRAIL_NUT to RSV_VCL_6 (e.g., the value of the NAL unit type is from 0 to 6 in Table 1 or Table 2 above), or have the same NAL unit type as GRA_NUT. Such a constraint applies only to the current picture including the case of a mixed IRAP and non-IRAP NAL unit type. However, it has not yet been properly applied to pictures including the case of a mixed RASL / RADL and non-IRAP NAL unit type.

[0185] In this document, a solution to the above-mentioned problems is provided. That is, as described above, a picture (i.e., the current picture) including two or more sub-pictures may have a mixed NAL unit type. In the case of the current VVC standard, a picture having a mixed NAL unit type may have a form mixed with an IRAP NAL unit type and a non-IRAP NAL unit type. However, although a leading picture related to the CRA NAL unit type may sometimes have a form mixed with a non-IRAP NAL unit type, such a picture having a mixed NAL unit type is not currently supported by the standard. Therefore, a solution method for a picture having a form mixed with a CRA NAL unit type and a non-IRAP NAL unit type is needed.

[0186] Therefore, this document provides a method for allowing a picture that includes a reading picture NAL unit type (e.g., RASL_NUT, RADL_NUT) and other non-IRAP NAL unit types (e.g., TRAIL_NUT, STSA, NUT) in a mixed form. Also, this document defines the constraints for making the reference picture list exist or be signaled when an IDR sub-picture and other non-IRAP sub-pictures are mixed. Therefore, for a picture having a mixed NAL unit type, by providing a form that is mixed not only with IRAP but also with CRA NAL units, it is made to have more flexible characteristics.

[0187] For example, it can be applied as in the following embodiments, whereby the above problems can be solved. The following embodiments may be applied individually or in combination.

[0188] As one embodiment, when allowing a picture to have a mixed NAL unit type (when the value of mixed_nal_types_in_pic_flag is 1), the signaling for the reference picture list can also exist for a slice having an IDR type of NAL unit type (e.g., IDR_W_RADL or IDR_N_LP). This constraint can be shown as follows.

[0189] - If there is at least one PPS that references an SPS where the value of mixed_nal_types_in_pic_flag equals 1, the value of sps_idr_rpl_present_flag must be 1. Such a constraint can be a requirement for the conformance of the bitstream.

[0190] Alternatively, as one embodiment, for a picture having a mixed NAL unit type, allow the picture to include slices having specific NAL unit types of a leading picture (e.g., RADL or RASL) and non-leading pictures, non-IRAP. This can be shown as follows.

[0191] For the VCL NAL units of a specific picture, the following can be applied.

[0192] - If the value of mixed_nalu_types_in_pic_flag is 0, the value of the NAL unit type (nal_unit_type) must be the same for all coded slice NAL units in the picture. The picture or PU can be regarded as having the same NAL unit type as the coded slice NAL unit within the picture or the PU.

[0193] - Otherwise (if the value of mixed_nalu_types_in_pic_flag is 1), one of the following must be satisfied (i.e., one of the following can have a true value).

[0194] 1) All one or more VCL NAL units must have a specific value of NAL unit type (nal_unit_type) within the range from IDR_W_RADL to CRA_NUT (e.g., the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), and all other VCL NAL units must have a specific value of NAL unit type within the range from TRAIL_NUT to RSV_VCL_6 (e.g., the value of the NAL unit type is from 0 to 6 in Table 1 or Table 2 above), or must have the same NAL unit type as GRA_NUT.

[0195] 2) All of the one or more VCL NAL units must have a specific value of the same NAL unit type as RADL_NUT (for example, the value of the NAL unit type in Table 1 or Table 2 is 2) or RASL_NUT (for example, the value of the NAL unit type in Table 1 or Table 2 is 3), and all of the other VCL NAL units must have a specific value of the same NAL unit type as TRAIL_NUT (for example, the value of the NAL unit type in Table 1 or Table 2 is 0), STSA_NUT (for example, the value of the NAL unit type in Table 1 or Table 2 is 1), RSV_VCL_4 (for example, the value of the NAL unit type in Table 1 or Table 2 is 4), RSV_VCL_5 (for example, the value of the NAL unit type in Table 1 or Table 2 is 5), RSV_VCL_6 (for example, the value of the NAL unit type in Table 1 or Table 2 is 6), or GRA_NUT.

[0196] On the other hand, this document proposes a method for providing a picture having a mixed NAL unit type as described above for a single-layer bitstream. As one embodiment, in the case of a single-layer bitstream, the following constraints may apply.

[0197] - Each picture except the first picture in decode order within the bitstream is considered to be related to a previous IRAP picture in decode order.

[0198] - If a picture is a leading picture of an IRAP picture, it must be a RADL or RASL picture.

[0199] - If a picture is a trailing picture of an IRAP picture, it must not be a RADL or RASL picture.

[0200] - There must be no RASL pictures in the bitstream related to an IDR picture.

[0201] - In the bitstream associated with an IDR picture where the NAL unit type (nal_unit_type) is IDR_N_LP, there shall be no RADL pictures.

[0202] When referenced, if each parameter set is available, random access can be performed at the position of the IRAP PU by discarding all PUs before the IRAP PU (also, the IRAP picture and all subsequent non-RASL pictures can be accurately decoded in decode order).

[0203] - Pictures preceding the IRAP picture in decode order shall precede the IRAP picture in output order and shall precede the RADL pictures associated with the IRAP picture in output order.

[0204] - RASL pictures associated with a CRA picture shall precede the RADL pictures associated with the CRA picture in output order.

[0205] - RASL pictures associated with a CRA picture shall follow in output order the IRAP pictures that precede the CRA picture in decode order.

[0206] - If the value of field_seq_flag is 0 and the current picture is a leading picture associated with an IRAP picture, it shall precede in decode order all non-leading pictures associated with the same IRAP picture. Otherwise, if pictures picA and picB are the first and last leading pictures associated with an IRAP picture in decode order respectively, there shall be at most one non-leading picture that precedes picA in decode order, and there shall be no non-leading pictures between picA and picB in decode order.

[0207] The following drawings are created to illustrate a specific example of this document. Since the names of specific devices, specific terms, and names (such as the names of syntax / syntax elements, etc.) described in the drawings are presented exemplarily, the technical features of this document are not limited to the specific names used in the following drawings.

[0208] Figure 15 schematically shows an example of a video / video encoding method to which the embodiment of this document is applicable. The method disclosed in Figure 15 can be executed by the encoding device 200 disclosed in Figure 2.

[0209] Referring to Figure 15, the encoding device can determine the NAL unit type for a slice within a picture (S1500).

[0210] For example, as described in Table 1 and Table 2 above, the encoding device can determine the NAL unit type according to the nature and type of a picture or sub-picture, etc., and can determine the NAL unit type for each slice based on the NAL unit type of the picture or sub-picture.

[0211] For example, for the case where the value of mixed_nalu_types_in_pic_flag is 0, the slices within the picture related to the PPS can be determined to have the same NAL unit type. That is, when the value of mixed_nalu_types_in_pic_flag is 0, the NAL unit type defined in the first NAL unit header of the first NAL unit including information for the first slice of the picture is the same as the NAL unit type defined in the second NAL unit header of the second NAL unit including information for the second slice of the same picture. Or, for the case where the value of mixed_nalu_types_in_pic_flag is 1, the slices within the picture related to the PPS can be determined to have other NAL unit types. Here, the NAL unit type for a slice within a picture can be determined based on the method proposed in the foregoing embodiments.

[0212] The encoding device can generate NAL unit type related information (S1510). The NAL unit type related information may include information / syntax elements related to the NAL unit type described in the above-described embodiments and / or the above Tables 1 and 2. For example, the information related to the NAL unit type may include the mixed_nalu_types_in_pic_flag syntax element included in the PPS. Also, the information related to the NAL unit type may include the nal_unit_type syntax element in the NAL unit header of the NAL unit including information for the coded slice.

[0213] The encoding device can generate a bitstream (S1520). The bitstream may include at least one NAL unit including video information for the coded slice. Also, the bitstream may include the PPS.

[0214] FIG. 16 schematically shows an example of a video / video decoding method to which the embodiments of this document are applicable. The method disclosed in FIG. 16 can be executed by the decoding device 300 disclosed in FIG. 3.

[0215] Referring to FIG. 16, the decoding device can receive a bitstream (S1600). Here, the bitstream may include at least one NAL unit including video information for the coded slice. Also, the bitstream may include the PPS.

[0216] The decoding device can acquire NAL unit type related information (S1610). The NAL unit type related information may include information / syntax elements related to the NAL unit type described in the above-described embodiments and / or the above Tables 1 and 2. For example, the information related to the NAL unit type may include the mixed_nalu_types_in_pic_flag syntax element included in the PPS. Further, the information related to the NAL unit type may include the nal_unit_type syntax element in the NAL unit header of the NAL unit including information for the coded slice.

[0217] The decoding device can determine the NAL unit type for the slice within the picture (S1620).

[0218] For example, when the value of mixed_nalu_types_in_pic_flag is 0, the slices within the picture related to the PPS use the same NAL unit type. That is, when the value of mixed_nalu_types_in_pic_flag is 0, the NAL unit type defined in the first NAL unit header of the first NAL unit including information for the first slice of the picture is the same as the NAL unit type defined in the second NAL unit header of the second NAL unit including information for the second slice of the same picture. Alternatively, when the value of mixed_nalu_types_in_pic_flag is 1, the slices within the picture related to the PPS use other NAL unit types. Here, the NAL unit type for the slice within the picture can be determined based on the method proposed in the above-described embodiments.

[0219] The decoding device can decode / restore samples / blocks / slices based on the NAL unit type of the slice. The samples / blocks within the slice can be decoded / restored based on the NAL unit type of the slice.

[0220] For example, when a first NAL unit type is set for the first slice of the current picture and a second NAL unit type (different from the first NAL unit type) is set for the second slice of the current picture, the samples / blocks in the first slice or the first slice itself can be decoded / restored based on the first NAL unit type, and the samples / blocks in the second slice or the second slice itself can be decoded / restored based on the second NAL unit type.

[0221] FIGS. 17 and 18 schematically show an example of a video / visual encoding method and related components according to an embodiment of this document.

[0222] The method disclosed in FIG. 17 can be executed by the encoding device 200 disclosed in FIG. 2 or FIG. 18. Here, the encoding device 200 disclosed in FIG. 18 is a simplified representation of the encoding device 200 disclosed in FIG. 2. Specifically, steps S1700 to S1720 in FIG. 17 can be executed by the entropy encoding unit 240 disclosed in FIG. 2, and according to the embodiment, each step can also be executed by the video segmentation unit 210, prediction unit 220, residual processing unit 230, addition unit 340, etc. disclosed in FIG. 2. Furthermore, the method disclosed in FIG. 17 can be executed including the embodiments described above in this document. Therefore, in FIG. 17, regarding the content overlapping with the above-described embodiments, specific descriptions will be omitted or simplified.

[0223] Referring to FIG. 17, the encoding device can determine the NAL unit type for the NAL unit in the current picture (S1700).

[0224] The current picture may include a plurality of slices, and one slice may include a slice header and slice data. Also, a NAL unit can be generated by adding a NAL unit header to the slice (slice header and slice data). The NAL unit header may include NAL unit type information specified by the slice data included in the NAL unit.

[0225] As an embodiment, the encoding device can generate a first NAL unit for a first slice in the current picture and a second NAL unit for a second slice in the current picture. Further, the encoding device can determine a first NAL unit type for the first slice and a second NAL unit type for the second slice according to the types of the first and second slices.

[0226] For example, as shown in Table 1 or Table 2, the NAL unit type may include TRAIL_NUT, STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, CRA_NUT, etc. based on the type of slice data included in the NAL unit. Also, the NAL unit type may be signaled based on the nal_unit_type syntax element in the NAL unit header. The nal_unit_type syntax element is syntax information for specifying the NAL unit type, and as shown in Table 1 or Table 2, it may be indicated by a specific value corresponding to a specific NAL unit type.

[0227] The encoding device can generate NAL unit type-related information for the NAL unit type (S1710).

[0228] The NAL unit type related information may include information / syntax elements related to the NAL unit type described in the above-described embodiments and / or the above Table 1 and Table 2. For example, the NAL unit type related information may be information regarding whether the current picture has a mixed NAL unit type, and may be indicated by the mixed_nalu_types_in_pic_flag syntax element included in the PPS. For example, when the value of the mixed_nalu_types_in_pic_flag syntax element is 0, it may indicate that the NAL units in the current picture have the same NAL unit type. Alternatively, when the value of the mixed_nalu_types_in_pic_flag syntax element is 1, it may indicate that the NAL units in the current picture have other NAL unit types.

[0229] As one embodiment, when all the NAL unit types for the NAL units in the current picture are the same, the encoding device can generate NAL unit type related information (e.g., mixed_nalu_types_in_pic_flag) having a value of 0. Alternatively, when the NAL unit types for the NAL units in the current picture are not the same, the encoding device can generate NAL unit type related information (e.g., mixed_nalu_types_in_pic_flag) having a value of 1.

[0230] That is, based on NAL unit type-related information regarding that the current picture has a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice of the current picture and the second NAL unit for the second slice of the current picture may have different NAL unit types. Alternatively, based on NAL unit type-related information regarding that the current picture does not have a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 0), the first NAL unit for the first slice of the current picture and the second NAL unit for the second slice of the current picture may have the same NAL unit type.

[0231] As an example, based on NAL unit type-related information regarding that the current picture has a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice may have a leading picture NAL unit type, and the second NAL unit for the second slice may have a non-IRAP NAL unit type or a non-leading picture NAL unit type. Here, the leading picture NAL unit type may include a RADL NAL unit type or a RASL NAL unit type, and the non-IRAP NAL unit type or the non-leading picture NAL unit type may include a trail NAL unit type or a STSA NAL unit type.

[0232] Alternatively, as an example, based on NAL unit type related information regarding having a NAL unit type with a currently mixed picture (e.g., the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice may have an IRAP NAL unit type, and the second NAL unit for the second slice may have a non-IRAP NAL unit type or a non-reference picture NAL unit type. Here, the IRAP NAL unit type may include an IDR NAL unit type (i.e., IDR_N_LP NAL or IDR_W_RADL NAL unit type) or a CRA NAL unit type, and the non-IRAP NAL unit type or non-reference picture NAL unit type may include a trail NAL unit type or a STSA NAL unit type. Also, depending on the embodiment, the non-IRAP NAL unit type or non-reference picture NAL unit type may sometimes only refer to the trail NAL unit type.

[0233] According to an embodiment, based on the case of allowing a currently mixed picture to have a NAL unit type, for a slice having an IDR NAL unit type (e.g., IDR_W_RADL or IDR_N_LP) within the currently mixed picture, there must be signaling related information for the reference picture list. The signaling related information for the reference picture list may indicate information regarding whether the syntax element for the signaling of the reference picture list exists in the slice header of the slice. That is, based on the value of the signaling related information for the reference picture list being 1, the syntax element for the signaling of the reference picture list may exist in the slice header of the slice having an IDR NAL unit type. Alternatively, based on the value of the signaling related information for the reference picture list being 0, the syntax element for the signaling of the reference picture list may not exist in the slice header of the slice having an IDR NAL unit type.

[0234] For example, the signaling related information of the reference picture list may be the sps_idr_rpl_present_flag syntax element described above. When the value of sps_idr_rpl_present_flag is 1, it may indicate that the syntax element for the signaling of the reference picture list may exist in the slice header of a slice having a NAL unit type such as IDR_N_LP or IDR_W_RADL. Alternatively, when the value of sps_idr_rpl_present_flag is 0, it may indicate that the syntax element for the signaling of the reference picture list does not exist in the slice header of a slice having a NAL unit type such as IDR_N_LP or IDR_W_RADL.

[0235] The encoding device can encode video / video information including NAL unit type related information (S1720).

[0236] For example, when the first NAL unit for the first slice in the current picture and the second NAL unit for the second slice in the current picture have different NAL unit types, the encoding device can encode video / video information including NAL unit type related information (for example, mixed_nalu_types_in_pic_flag) having a value of 1. Alternatively, when the first NAL unit for the first slice in the current picture and the second NAL unit for the second slice in the current picture have the same NAL unit type, the encoding device can encode video / video information including NAL unit type related information (for example, mixed_nalu_types_in_pic_flag) having a value of 0.

[0237] Also, for example, the encoding device can encode video / video information including nal_unit_type information indicating each NAL unit type for the slices in the current picture.

[0238] Furthermore, for example, the encoding device can encode video / video information including signaling related information (e.g., sps_idr_rpl_present_flag) of the reference picture list.

[0239] Also, for example, the encoding device can encode video / video information including NAL units for slices within the current picture.

[0240] The video / video information including various information as described above can be encoded and output in the form of a bitstream. The bitstream can be transmitted to the decoding device via a network or a (digital) storage medium. Here, the network can include a broadcast network and / or a communication network, etc., and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

[0241] FIG. 19 and FIG. 20 schematically show an example of a video / video decoding method and related components according to an embodiment of this document.

[0242] The method disclosed in FIG. 19 can be executed by the decoding device 300 disclosed in FIG. 3 or FIG. 20. Here, the decoding device 300 disclosed in FIG. 20 is a simplified representation of the decoding device 300 disclosed in FIG. 3. Specifically, steps S1900 to S1920 in FIG. 19 can be executed by the entropy decoding unit 310 disclosed in FIG. 3, and depending on the embodiment, each step can also be executed by the residual processing unit 320, prediction unit 330, addition unit 340, etc. disclosed in FIG. 3. Furthermore, the method disclosed in FIG. 19 can be executed in this document including the aforementioned embodiments. Therefore, in FIG. 19, regarding the content overlapping with the aforementioned embodiments, specific explanations will be omitted or simplified.

[0243] Referring to FIG. 19, the decoding device can obtain video / video information including NAL unit type related information from the bitstream (S1900).

[0244] As an embodiment, the decoding device can parse the bitstream to derive information (e.g., video / video information) necessary for video restoration (or picture restoration). At this time, the video information may include the NAL unit type-related information (e.g., mixed_nalu_types_in_pic_flag) described above, the nal_unit_type information indicating each NAL unit type for the slice in the current picture, the signaling-related information of the reference picture list (e.g., sps_idr_rpl_present_flag), the NAL unit for the slice in the current picture, and the like. That is, the video information may include various information necessary in the decoding process and can be decoded based on coding methods such as exponential Golomb coding, CAVLC, or CABAC.

[0245] As described above, the NAL unit type-related information may include information / syntax elements related to the NAL unit type described in the above-described embodiments and / or Tables 1 and 2. For example, the NAL unit type-related information may be information regarding whether the current picture has a mixed NAL unit type and may be indicated by the mixed_nalu_types_in_pic_flag syntax element included in the PPS. For example, when the value of the mixed_nalu_types_in_pic_flag syntax element is 0, it may indicate that the NAL units in the current picture have the same NAL unit type. Alternatively, when the value of the mixed_nalu_types_in_pic_flag syntax element is 1, it may indicate that the NAL units in the current picture have other NAL unit types.

[0246] The decoding device can determine the NAL unit type for the NAL units in the current picture based on the NAL unit type-related information (S1910).

[0247] The current picture may include a plurality of slices, and one slice may include a slice header and slice data. Also, by adding an NAL unit header to a slice (slice header and slice data), an NAL unit can be generated. The NAL unit header may include NAL unit type information specified by the slice data included in the NAL unit.

[0248] For example, as shown in Table 1 or Table 2 above, the NAL unit type may include TRAIL_NUT, STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, CRA_NUT, etc., based on the type of slice data included in the NAL unit. Also, the NAL unit type may be signaled based on the nal_unit_type syntax element in the NAL unit header. The nal_unit_type syntax element is syntax information for specifying the NAL unit type, and as shown in Table 1 or Table 2 above, it may be indicated by a specific value corresponding to a specific NAL unit type.

[0249] As one embodiment, based on NAL unit type related information regarding that the current picture has a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 1), the decoding device can determine that the first NAL unit for the first slice of the current picture and the second NAL unit for the second slice of the current picture have different NAL unit types. Alternatively, based on NAL unit type related information regarding that the current picture does not have a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 0), the decoding device can determine that the first NAL unit for the first slice of the current picture and the second NAL unit for the second slice of the current picture have the same NAL unit type.

[0250] As an example, based on NAL unit type related information regarding that the current picture has a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice may have a leading picture NAL unit type, and the second NAL unit for the second slice may have a non-IRAP NAL unit type or a non-leading picture NAL unit type. Here, the leading picture NAL unit type may include a RADL NAL unit type or a RASL NAL unit type, and the non-IRAP NAL unit type or the non-leading picture NAL unit type may include a trail NAL unit type or a STSA NAL unit type.

[0251] Alternatively, as an example, based on NAL unit type related information regarding that the current picture has a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice may have an IRAP NAL unit type, and the second NAL unit for the second slice may have a non-IRAP NAL unit type or a non-leading picture NAL unit type. Here, the IRAP NAL unit type may include an IDR NAL unit type (i.e., IDR_N_LP NAL or IDR_W_RADL NAL unit type) or a CRA NAL unit type, and the non-IRAP NAL unit type or the non-leading picture NAL unit type may include a trail NAL unit type or a STSA NAL unit type. Also, according to an embodiment, the non-IRAP NAL unit type or the non-leading picture NAL unit type may sometimes only refer to a trail NAL unit type.

[0252] According to an embodiment, based on the case where it is allowed to have a NAL unit type in which the current picture is mixed, for a slice having an IDR NAL unit type (e.g., IDR_W_RADL or IDR_N_LP) in the current picture, there must be signaling related information of the reference picture list. The signaling related information of the reference picture list may indicate information regarding whether a syntax element for the signaling of the reference picture list exists in the slice header of the slice. That is, based on the value of the signaling related information of the reference picture list being 1, a syntax element for the signaling of the reference picture list may exist in the slice header of a slice having an IDR NAL unit type. Alternatively, based on the value of the signaling related information of the reference picture list being 0, a syntax element for the signaling of the reference picture list may not exist in the slice header of a slice having an IDR NAL unit type.

[0253] For example, the signaling related information of the reference picture list may be the above-described sps_idr_rpl_present_flag syntax element. When the value of sps_idr_rpl_present_flag is 1, it may indicate that a syntax element for the signaling of the reference picture list may exist in the slice header of a slice having a NAL unit type such as IDR_N_LP or IDR_W_RADL. Alternatively, when the value of sps_idr_rpl_present_flag is 0, it may indicate that a syntax element for the signaling of the reference picture list does not exist in the slice header of a slice having a NAL unit type such as IDR_N_LP or IDR_W_RADL.

[0254] The decoding device can decode / restore the current picture based on the NAL unit type (S1920).

[0255] For example, for the first slice in the current picture determined to be of the first NAL unit type and the second slice in the current picture determined to be of the second NAL unit type, the decoding device can decode / restore the first slice based on the first NAL unit type and decode / restore the second slice based on the second NAL unit type. Also, the decoding device can decode / restore samples / blocks in the first slice based on the first NAL unit type and decode / restore samples / blocks in the second slice based on the second NAL unit type.

[0256] In the above-described embodiments, the method is described based on a flowchart as a series of steps or blocks, but the embodiments of this document are not limited to the order of the steps, and a certain step may occur in a different order from the steps described above or simultaneously. Also, those skilled in the art can understand that the steps shown in the flowchart are not exclusive, and other steps may be included, or one or more steps of the flowchart may be deleted without affecting the scope of this document.

[0257] The method according to the above-described document can be implemented in the form of software, and the encoding device and / or decoding device according to this document can be included in a device that performs video processing, such as a TV, a computer, a smartphone, a set-top box, a display device, etc.

[0258] In this document, when an embodiment is implemented in software, the foregoing method may be implemented by modules (processes, functions, etc.) that perform the foregoing functions. The modules may be stored in a memory and executed by a processor. The memory may be internal or external to the processor and may be connected to the processor by various well-known means. The processor may include an ASIC (application-specific integrated circuit), other chip sets, logic circuits, and / or data processing devices. The memory may include a ROM (read-only memory), a RAM (random access memory), a flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described in this document may be implemented and executed on a processor, a microprocessor, a controller, or a chip. For example, the functional units shown in each drawing may be implemented and executed on a computer, a processor, a microprocessor, a controller, or a chip. In this case, information for implementation (for example, information on instructions) or an algorithm may be stored in a digital storage medium.

[0259] In addition, the decoding device and encoding device to which this document is applicable may be included in a multimedia broadcast transceiver, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conferencing device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camcorder, an on-demand video (VoD) service providing device, an OTT video (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, a VR (virtual reality) device, an AR (augmented reality) device, a videophone video device, a transportation means terminal (e.g., a vehicle terminal including an autonomous driving vehicle, an airplane terminal, a ship terminal, etc.) and a medical video device, etc., and may be used to process video signals or data signals. For example, the OTT video (Over the top video) device may include a game console, a Blu-ray player, an Internet access TV, a home theater system, a smartphone, a tablet PC, a DVR (Digital Video Recoder), etc.

[0260] In addition, the processing method to which the embodiments of this document are applied can be produced in the form of a program executed by a computer and can be stored in a recording medium readable by the computer. Multimedia data having a data structure according to the embodiments of this document can also be stored in a recording medium readable by the computer. The recording medium readable by the computer includes all types of storage devices and distributed storage devices in which data readable by the computer is stored. The recording medium readable by the computer can include, for example, Blu-ray Disc (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Further, the recording medium readable by the computer includes a medium embodied in the form of a carrier wave (for example, transmission via the Internet). Also, a bitstream generated by an encoding method can be stored in a recording medium readable by the computer or transmitted via a wired or wireless communication network.

[0261] In addition, the embodiments of this document can be embodied in a computer program product by program code, and the program code can be executed by a computer according to the embodiments of this document. The program code can be stored on a carrier readable by a computer.

[0262] FIG. 21 shows an example of a content streaming system to which the embodiments disclosed in this document can be applied.

[0263] Referring to FIG. 21, a content streaming system to which the embodiments of this document are applied can include a large encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0264] The encoding server compresses the content input from multimedia input devices such as smartphones, cameras, camcorders, etc. into digital data to generate a bitstream, and serves to transmit this to the streaming server. As another example, when multimedia input devices such as smartphones, cameras, camcorders, etc. directly generate a bitstream, the encoding server may be omitted.

[0265] The bitstream can be generated by an encoding method or a bitstream generation method to which the embodiments of this document are applied. The streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0266] The streaming server transmits multimedia data to a user device based on a user request via a web server. The web server serves as a medium to inform the user of what services are available. If the user requests a desired service from the web server, the web server transmits this to the streaming server, and the streaming server transmits multimedia data to the user. At this time, the content streaming system can include another control server. In this case, the control server serves to control commands / responses between each device within the content streaming system.

[0267] The streaming server can receive content from a media repository and / or an encoding server. For example, when it comes to receiving content from the encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0268] In the example of the user device, there may be a mobile phone, a smart phone, a laptop computer, a digital broadcast terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (for example, a smartwatch, smart glass, HMD (head mounted display)), a digital TV, a desktop computer, a digital signage, and the like.

[0269] Each server in the content streaming system can be operated as a distributed server, and in this case, the data received by each server can be distributedly processed.

[0270] The claims described in this specification can be combined in various ways. For example, the technical features of the method claims in this specification can be combined and implemented as a device, and the technical features of the device claims in this specification can be combined and implemented as a method. Also, the technical features of the method claims in this specification and the technical features of the device claims can be combined and implemented as a device, and the technical features of the method claims in this specification and the technical features of the device claims can be combined and implemented as a method.

Claims

1. A video decoding method executed by a decoding device, comprising: obtaining video information including network abstraction layer (NAL) unit type related information from a bitstream; determining a NAL unit type for a NAL unit in a current picture based on the NAL unit type related information; and decoding the current picture based on the NAL unit type. The NAL unit type related information is information regarding whether the current picture has a mixed NAL unit type; The video information further includes NAL unit type restriction information for restricting the NAL unit type related information from having the mixed NAL unit type, The NAL unit type related information is determined based on the NAL unit type restriction information; Based on the NAL unit type related information regarding the current picture having mixed NAL unit types, a first NAL unit type for a first slice of the current picture is different from a second NAL unit type for a second slice of the current picture; Based on the first NAL unit type being an instantaneous decoding refresh (IDR) NAL unit type, a value of information related to reference picture list signaling is equal to 1; A video decoding method, wherein based on the value of information related to the reference picture list signaling being equal to 1, a syntax element for the reference picture list signaling is present in a slice header of the first slice having the IDR NAL unit type.

2. A video encoding method performed by an encoding device, comprising: determining a network abstraction layer (NAL) unit type for a NAL unit in a current picture; generating NAL unit type related information based on the NAL unit type; encoding video information including the NAL unit type associated information; The NAL unit type related information is information regarding whether the current picture has a mixed NAL unit type; The video information further includes NAL unit type restriction information for restricting the NAL unit type related information from having the mixed NAL unit type, The NAL unit type related information is determined based on the NAL unit type restriction information; Based on the NAL unit type related information regarding the current picture having mixed NAL unit types, a first NAL unit type for a first slice of the current picture is different from a second NAL unit type for a second slice of the current picture having a different NAL unit type; 11. A video encoding method, comprising: based on the first NAL unit type being an instantaneous decoding refresh (IDR) NAL unit type, a syntax element for signaling a reference picture list is present in a slice header of the first slice having the IDR NAL unit type.

3. A method for transmitting data for video, comprising: obtaining the video bitstream, The bitstream is generated based on determining a network abstraction layer (NAL) unit type for a NAL unit in a current picture, generating NAL unit type related information based on the NAL unit type, and encoding video information including the NAL unit type related information. transmitting the data including the bitstream; The NAL unit type related information is information regarding whether the current picture has a mixed NAL unit type, The video information further includes NAL unit type restriction information for restricting the NAL unit type related information from having the mixed NAL unit type, The NAL unit type related information is generated based on the NAL unit type restriction information; Based on the NAL unit type related information regarding the current picture having mixed NAL unit types, a first NAL unit type for a first slice of the current picture is different from a second NAL unit type for a second slice of the current picture having a different NAL unit type; A method for transmitting data, based on the first NAL unit type being an instantaneous decoding refresh (IDR) NAL unit type, a syntax element for signaling a reference picture list is present in a slice header of the first slice having the IDR NAL unit type.

Images