Image or video coding based on NAL unit related information

Abstract

To provide a video based on NAL unit related information.SOLUTION: According to the disclosure of the present document, based on NAL unit type related information on whether a picture has mixed NAL unit types, NAL unit types for slices within the picture can be determined, and for the picture having the mixed NAL unit types, not only IRAP but also a form mixed with NAL units of other types can be provided.SELECTED DRAWING: Figure 19

Description

Technical Field

[0001] The present technology relates to video or video coding, for example, coding technology based on NAL (network abstraction layer) unit related information.

Background Art

[0002] In recent years, the demand for high-resolution and high-quality video / video such as 4K or UHD (Ultra High Definition) video / video of 8K or higher has been increasing in various fields. As the video / video data becomes higher in resolution and quality, the amount of information or bits transmitted relatively increases compared to the existing video / video data. Therefore, when transmitting video data using a medium such as an existing wired or wireless broadband line, or storing video / video data using an existing storage medium, the transmission cost and storage cost increase.

[0003] In addition, in recent years, the interest and demand for immersive media such as VR (Virtual Reality), AR (Artificial Reality) content, and holograms have been increasing, and the broadcast of video / video having video characteristics different from those of real-world video, such as game video, has been increasing.

[0004] Therefore, there is a need for a highly efficient video / video compression technology to effectively compress, transmit, store, and reproduce the information of high-resolution and high-quality video / video having various characteristics as described above.

[0005] In addition, a solution for improving the efficiency of video / video coding is required. Therefore, a solution for effectively signaling and coding information related to the NAL (network abstraction layer) unit is required.

Summary of the Invention

Problems to be Solved by the Invention

[0006] The technical problem of this document is to provide a method and an apparatus for enhancing the efficiency of video / videotape coding.

[0007] Another technical problem of this document is to provide a method and an apparatus for enhancing the efficiency of video / videotape coding based on NAL unit related information.

[0008] Yet another technical problem of this document is to provide a method and an apparatus for enhancing the efficiency of video / videotape coding for a picture having a mixed NAL unit type.

[0009] Yet another technical problem of this document is to provide a method and an apparatus for making the reference picture list related information exist in a slice within a picture having a specific NAL unit type or be signaled for a picture having a mixed NAL unit type.

Means for Solving the Problem

[0010] According to an embodiment of this document, based on NAL unit type related information regarding whether a picture has a mixed NAL unit type, the NAL unit type for a slice within the picture can be determined. For example, based on the NAL unit type related information regarding that a picture has a mixed NAL unit type, the first NAL unit for the first slice of the picture and the second NAL unit for the second slice of the picture may have different NAL unit types. Alternatively, based on the NAL unit type related information regarding that a picture does not have a mixed NAL unit type, the first NAL unit for the first slice of the picture and the second NAL unit for the second slice of the picture may have the same NAL unit type.

[0011] According to one embodiment of this document, based on the case of allowing a NAL unit type with a mixed picture, for a slice having a specific NAL unit type within a picture, signaling related information of a reference picture list can be made to exist.

[0012] According to one embodiment of this document, a method for decoding video / images executed by a decoding device is provided. The method for decoding the video / images can include the method disclosed in the embodiment of this document.

[0013] According to one embodiment of this document, a decoding device for executing decoding of video / images is provided. The decoding device can perform the method disclosed in the embodiment of this document.

[0014] According to one embodiment of this document, a method for encoding video / images executed by an encoding device is provided. The method for encoding the video / images can include the method disclosed in the embodiment of this document.

[0015] According to one embodiment of this document, an encoding device for executing encoding of video / images is provided. The encoding device can perform the method disclosed in the embodiment of this document.

[0016] According to one embodiment of this document, a computer-readable digital storage medium storing encoded video / image information generated by the method for encoding video / images disclosed in at least one of the embodiments of this document is provided.

[0017] According to one embodiment of this document, a computer-readable digital storage medium storing encoded information or encoded video / image information that causes a decoding device to perform the method for decoding video / images disclosed in at least one of the embodiments of this document is provided.

Advantages of the Invention

[0018] This document can have various effects. For example, according to one embodiment of this document, the compression efficiency of general video can be increased. Also, according to one embodiment of this document, based on NAL unit related information, the efficiency of video coding can be increased. Furthermore, according to one embodiment of this document, the efficiency of video coding for a picture having a mixed NAL unit type can be increased. Also, according to one embodiment of this document, for a picture having a mixed NAL unit type, reference picture list related information can be effectively signaled and coded. Furthermore, according to one embodiment of this document, by allowing a picture that includes a leading picture NAL unit type (e.g., RASL_NUT, RADL_NUT) and other non-IRAP NAL unit types (e.g., TRAIL_NUT, STSA, NUT) in a mixed form, not only IRAP but also other types of NAL units can be provided in a mixed form for a picture having a mixed NAL unit type, and through this, more flexible characteristics can be achieved.

[0019] The effects obtained through the specific embodiments of this document are not limited to the effects listed above. For example, there may be various technical effects that a person having ordinary skill in the related art can understand or derive from this document. Thus, the specific effects of this document are not limited to those explicitly described in this document and may include various effects that can be understood or derived from the technical features of this document.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Best Mode for Carrying Out the Invention

[0021] This document can be modified in various ways and can have various embodiments. Specific embodiments are illustrated in the drawings and will be described in detail. However, this is not intended to limit this document to specific embodiments. The terms commonly used in this specification are merely used to describe specific embodiments and are not used with the intention of limiting the technical idea of this document. Singular expressions include plural expressions unless the context clearly indicates otherwise. Terms such as "including" or "having" in this specification are intended to specify the existence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and it should be understood that the existence or possibility of addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof, etc. is not precluded in advance.

[0022] On the other hand, each configuration in the drawings described in this document is independently illustrated for the convenience of explaining different characteristic functions. It does not mean that each configuration is implemented by separate hardware or separate software. For example, among the configurations, two or more configurations can be combined to form one configuration, and one configuration can also be divided into multiple configurations. Embodiments in which each configuration is integrated and / or separated are included in the scope of rights of this document as long as they do not deviate from the essence of this document.

[0023] In this document, "A or B" may mean "only A", "only B", or "both A and B". In other words, in this document, "A or B" may be interpreted as "A and / or B". For example, in this document, "A, B or C" may mean "only A", "only B", "only C", or "any combination of A, B and C".

[0024] The slashes ( / ) and commas used in this document may mean "and / or". For example, "A / B" may mean "A and / or B". Thus, "A / B" may mean "only A", "only B", or "both A and B". For example, "A, B, C" may mean "A, B or C".

[0025] In this document, "at least one of A and B" may mean "only A", "only B", or "both A and B". Also, in this document, expressions such as "at least one of A or B" and "at least one of A and / or B" may be interpreted in the same way as "at least one of A and B".

[0026] Also, in this document, "at least one of A, B and C" may mean "only A", "only B", "only C", or "any combination of A, B and C". Also, "at least one of A, B or C" and "at least one of A, B and / or C" may mean "at least one of A, B and C".

[0027] Also, the parentheses used in this document may mean "for example". Specifically, when it is displayed as "prediction (intra prediction)", "intra prediction" may be proposed as an example of "prediction". In other words, the "prediction" in this document is not limited to "intra prediction", and "intra prediction" may be proposed as an example of "prediction". Also, when it is displayed as "prediction (i.e., intra prediction)", "intra prediction" may be proposed as an example of "prediction".

[0028] This document relates to video / video coding. For example, the methods / examples disclosed in this document can be applied to the methods disclosed in the VVC (versatile video coding) standard. Also, the methods / examples disclosed in this document can be applied to the methods disclosed in the EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard) or next-generation video / video coding standards (e.g., 267 or H.268, etc.).

[0029] This document presents various examples related to video / video coding, and unless otherwise mentioned, the above examples may be combined with each other.

[0030] In this document, video may mean a set of a series of images according to the flow of time. Picture generally means a unit indicating one video in a specific time period, and slice / tile is a unit constituting a part of a picture in coding. A slice / tile may include one or more CTUs (Coding Tree Units). One picture may be composed of one or more slices / tiles. A tile is a rectangular area of CTUs within a specific column of tiles in a picture and within a specific column of tiles. The column of the tiles is a rectangular area of CTUs, the rectangular area has the same height as the height of the picture, and the width may be specified by a syntax element in the picture parameter set. The row of the tiles is a rectangular area of CTUs, the rectangular area has a width specified by a syntax element in the picture parameter set, and the height may be the same as the height of the picture. Tile scan may indicate a specific sequential ordering of CTUs that partition a picture, the CTUs may be continuously arranged in a raster scan of the CTUs within a tile, and the tiles in a picture may be continuously arranged in a raster scan of the tiles of the picture. A slice may exclusively include an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile of an integer number of pictures, and may be included in a single NAL unit.

[0031] On the other hand, one picture can be divided into two or more sub-pictures. A sub-picture can be a rectangular area of one or more slices in a picture.

[0032] A pixel or pel may mean the smallest unit that constitutes one picture (or video). Also, the term "sample" may be used as a term corresponding to a pixel. A sample generally indicates a pixel or a pixel value, and may indicate only the pixel / pixel value of the luma component, or may indicate only the pixel / pixel value of the chroma component. Alternatively, a sample may mean a pixel value in the spatial domain, and when such a pixel value is converted to the frequency domain, it may mean a conversion coefficient in the frequency domain.

[0033] A unit may indicate the basic unit of video processing. A unit may include at least one of a specific region of a picture and information regarding the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. A unit may, in some cases, be used interchangeably with terms such as "block" or "area". In a general case, an M×N block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0034] Also, in this document, at least one of quantization / inverse quantization and / or transformation / inverse transformation may be omitted. When quantization / inverse quantization is omitted, the quantized transform coefficients may be called transform coefficients. When transformation / inverse transformation is omitted, the transform coefficients may be called coefficients or residual coefficients, or may still be called transform coefficients for the sake of consistency of expression.

[0035] In this document, the quantized transform coefficients and the transform coefficients may each be referred to as transform coefficients and scaled transform coefficients, respectively. In this case, the residual information may include information regarding the transform coefficients, and the information regarding the transform coefficients may be signaled via a residual coding syntax. The transform coefficients may be derived based on the residual information (or the information regarding the transform coefficients), and the scaled transform coefficients may be derived via an inverse transform (scaling) with respect to the transform coefficients. The residual samples may be derived based on an inverse transform (transformation) with respect to the scaled transform coefficients. This may be applied / expressed in a similar manner in other parts of this document.

[0036] In this document, the technical features separately described within one drawing may be realized separately or simultaneously.

[0037] Hereinafter, with reference to the attached drawings, the preferred embodiments of this document will be described in more detail. Hereinafter, the same reference numerals will be used for the same components in the drawings, and duplicate descriptions of the same components may be omitted.

[0038] FIG. 1 schematically shows an example of a video / image coding system that can be applied to an embodiment of this document.

[0039] As shown in FIG. 1, the video / image coding system can include a first device (source device) and a second device (receiving device). The source device can transmit the encoded video / image information or data in a file or streaming form to the receiving device via a digital storage medium or a network.

[0040] The source device can include a video source, an encoding device, and a transmission unit. The receiving device can include a receiving unit, a decoding device, and a renderer. The encoding device can be referred to as a video / video encoding device, and the decoding device can be referred to as a video / video decoding device. A transmitter can be provided in the encoding device. A receiver can be provided in the decoding device. The renderer can include a display unit, and the display unit can also be composed of a separate device or an external component.

[0041] The video source can obtain video / video through processes such as video / video capture, synthesis, or generation. The video source can include a video / video capture device and / or a video / video generation device. The video / video capture device can include, for example, one or more cameras, a video / video archive containing previously captured video / video, etc. The video / video generation device can include, for example, a computer, a tablet, and a smartphone, etc., and can (electronically) generate video / video. For example, virtual video / video can be generated through a computer or the like, and in this case, the video / video capture process can be replaced by the process of generating related data.

[0042] The encoding device can encode the input video / video. The encoding device can perform a series of procedures such as prediction, transformation, quantization, etc. for compression and coding efficiency. The encoded data (encoded video / video information) can be output in the form of a bitstream.

[0043] The transmitting unit can transmit the encoded video / video information or data output in bitstream form to the receiving unit of the receiving device via a digital storage medium or a network in file or streaming form. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating a media file via a predetermined file format and can include elements for transmission via a broadcast / communication network. The receiving unit can receive / extract the bitstream and transmit it to the decoding device.

[0044] The decoding device can decode the video / video by performing a series of procedures such as inverse quantization, inverse transformation, prediction, etc. corresponding to the operation of the encoding device.

[0045] The renderer can render the decoded video / video. The rendered video / video can be displayed via the display unit.

[0046] Figure 2 is a diagram schematically explaining the configuration of a video / video encoding device to which the embodiments of this document can be applied. Hereinafter, the encoding device can include a video encoding device and / or a video encoding device.

[0047] As shown in FIG. 2, the encoding apparatus 200 can be configured to include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 can include an inter-prediction unit 221 and an intra-prediction unit 222. The residual processor 230 can include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 can further include a subtractor (231). The adder 250 can be called a reconstructor or a reconstructed block generator. The above-described image partitioner 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 can be configured by one or more hardware components (e.g., an encoder chipset or a processor) according to an embodiment. Also, the memory 270 can include a DPB (decoded picture buffer) and can also be configured by a digital storage medium. The hardware component can further include the memory 270 as an internal / external component.

[0048] The video segmentation unit 210 can divide the input video (or picture, frame) input to the encoding device 200 into one or more processing units. As an example, the processing unit can be called a coding unit (CU). In this case, the coding unit can be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) by a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, one coding unit can be divided into multiple coding units with a deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure can be applied first, and then the binary-tree structure and / or the ternary structure can be applied. Or, the binary-tree structure can also be applied first. The coding procedure according to the present disclosure can be performed based on the final coding unit that is no longer divided. In this case, based on the coding efficiency according to the video characteristics, etc., the largest coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units with a deeper depth so that a coding unit with an optimal size can be used as the final coding unit. Here, the coding procedure can include procedures such as prediction, transformation, and restoration described later. As another example, the processing unit can further include a prediction unit (PU: Prediction Unit) or a transform unit (TU: Transform Unit). In this case, the prediction unit and the transform unit can each be divided or partitioned from the aforementioned final coding unit.The prediction unit can be a unit of sample prediction, and the conversion unit can be a unit for deriving a conversion coefficient and / or a unit for deriving a residual signal from the conversion coefficient.

[0049] The unit can, in some cases, be used interchangeably with terms such as block or area. In general, an M×N block can represent a set of samples or transform coefficients, etc., consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or a pel in one picture (or video).

[0050] The encoding device 200 can subtract the prediction signal (predicted block, predicted sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from the input video signal (original block, original sample array) to generate a residual signal (residual signal, residual block, residual sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as shown in the figure, the unit that subtracts the prediction signal (prediction block, prediction sample array) from the input video signal (original block, original sample array) within the encoder 200 can be called the subtraction unit 231. The prediction unit can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a predicted block including prediction samples for the current block. The prediction unit can determine whether intra prediction or inter prediction is applied in units of the current block or CU. The prediction unit can generate various pieces of information related to prediction, such as prediction mode information, and transmit them to the entropy encoding unit 240 as described later in the description of each prediction mode. The information related to prediction can be encoded by the entropy encoding unit 240 and output in the form of a bit stream.

[0051] The intra prediction unit 222 can predict the current block by referring to samples within the current picture. The samples to be referred to can be located adjacent to the current block or can be located remotely depending on the prediction mode. In intra prediction, the prediction mode can include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes can include, for example, the DC mode and the Planar mode. The directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of fineness of the prediction direction. However, this is an example, and more or fewer directional prediction modes can be used depending on the setting. The intra prediction unit 222 can also determine the prediction mode to be applied to the current block using the prediction mode applied to the adjacent blocks.

[0052] The inter prediction unit 221 can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between adjacent blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the adjacent blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block can be the same or different. The temporal neighboring blocks can be called by names such as collocated reference blocks and collocated CUs (col CUs), and the reference picture including the temporal neighboring blocks can also be called a collocated picture (colPic). For example, the inter prediction unit 221 can construct a motion information candidate list based on adjacent blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter prediction unit 221 can use the motion information of adjacent blocks as the motion information of the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the case of the motion vector prediction (MVP) mode, the motion vector of an adjacent block is used as a motion vector predictor, and the motion vector difference is signaled to indicate the motion vector of the current block.

[0053] The prediction unit 220 can generate a prediction signal based on various prediction methods described below. For example, for the prediction of one block, the prediction unit can apply not only intra prediction or inter prediction but also simultaneously apply intra prediction and inter prediction. This can be called combined inter and intra prediction (CIIP). Also, the prediction unit can be based on the intra block copy (IBC) prediction mode or the palette mode for the prediction of a block. The IBC prediction mode or the palette mode can be used for content video / motion video coding such as games, for example, like SCC (screen content coding). IBC basically performs prediction within the current picture, but can be performed in the same way as inter prediction in terms of deriving a reference block within the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document. The palette mode can be regarded as an example of intra coding or intra prediction. When the palette mode is applied, the sample values in the picture can be signaled based on the information regarding the palette table and the palette index.

[0054] The prediction signal generated via the prediction unit (including the inter-prediction unit 221 and / or the intra-prediction unit 222) may be used to generate a restored signal or may be used to generate a residual signal. The conversion unit 232 can apply a conversion technique to the residual signal to generate transform coefficients. For example, the conversion technique may include at least one of DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT means the conversion obtained from this graph when representing the relationship information between pixels as a graph. CNT means the conversion obtained based on generating a prediction signal using all previously reconstructed pixels. Also, the conversion process may be applied to a pixel block having the same size of a square, or may be applied to a block of a variable size that is not square.

[0055] The quantization unit 233 quantizes the transform coefficients and transmits them to the entropy encoding unit 240. The entropy encoding unit 240 can encode the quantized signal (information regarding the quantized transform coefficients) and output it as a bitstream. The information regarding the quantized transform coefficients can be referred to as residual information. The quantization unit 233 can reorder the block-form quantized transform coefficients in a one-dimensional vector form based on the coefficient scan order, and can also generate the information regarding the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit 240 can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit 240 can encode, together or separately, information necessary for video / image restoration (e.g., values of syntax elements, etc.) in addition to the quantized transform coefficients. The encoded information (e.g., encoded video / video information) can be transmitted or stored in units of NAL (network abstraction layer) units in the form of a bitstream. The video / video information can further include information regarding various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS), etc. Also, the video / video information can further include general constraint information. In this document, the information and / or syntax elements transmitted / signaled from the encoding device to the decoding device can be included in the video / video information.The video / video information can be encoded through the encoding procedure described above and included in the bitstream. The bitstream can be transmitted via a network or stored in a digital storage medium. Here, the network can include a broadcast network and / or a communication network, etc., and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit 240 can be configured as an internal / external element of the encoding device 200 by a transmission unit (not shown) for transmission and / or a storage unit (not shown) for storage, or the transmission unit can also be included in the entropy encoding unit 240.

[0056] The quantized transform coefficients output from the quantization unit 233 can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients via the inverse quantization unit 234 and the inverse transformation unit 235, a residual signal (residual block or residual sample) can be restored. The addition unit 155 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the restored residual signal to the prediction signal output from the inter prediction unit 221 or the intra prediction unit 222. When there is no residual for the block to be processed, as in the case where the skip mode is applied, the predicted block can be used as the reconstructed block. The addition unit 250 can be called a restoration unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra prediction of the next block to be processed within the current picture and, as will be described later, can also be used for inter prediction of the next picture after passing through filtering.

[0057] On the other hand, LMCS (luma mapping with chroma scaling) can also be applied during the picture encoding and / or restoration process.

[0058] The filtering unit 260 can apply filtering to the restored signal to improve the subjective / objective image quality. For example, the filtering unit 260 can apply various filtering methods to the restored picture to generate a modified restored picture, and store the modified restored picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, and the like. The filtering unit 260 can generate various information related to filtering and transmit it to the entropy encoding unit 240 as will be described later in the description of each filtering method. The information related to filtering can be encoded by the entropy encoding unit 240 and output in the form of a bit stream.

[0059] The modified restored picture transmitted to the memory 270 can be used as a reference picture by the inter prediction unit 221. When inter prediction is applied through this, the encoding device can avoid prediction mismatches between the encoding device 100 and the decoding device, and can also improve the encoding efficiency.

[0060] The DPB of memory 270 can store the corrected restored picture for use as a reference picture in the inter prediction unit 221. Memory 270 can store the motion information of the blocks for which the motion information in the current picture has been derived (or encoded) and / or the motion information of the blocks in the already restored picture. The stored motion information can be transmitted to the inter prediction unit 221 for utilization as the motion information of spatially adjacent blocks or temporally adjacent blocks. Memory 270 can store the restored samples of the restored blocks in the current picture and can transmit them to the intra prediction unit 222.

[0061] FIG. 3 is a drawing schematically explaining the configuration of a video / video decoding apparatus to which this document can be applied.

[0062] As shown in FIG. 3, the decoding apparatus 300 can be configured to include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filtering unit 350, and a memory 360. The predictor 330 can include an inter prediction unit 331 and an intra prediction unit 332. The residual processor 320 can include a dequantizer 321 and an inverse transformer 321. The above-described entropy decoder 310, residual processor 320, predictor 330, adder 340, and filtering unit 350 can be configured by one hardware component (for example, a decoder chipset or a processor) according to an embodiment. Also, the memory 360 can include a DPB (decoded picture buffer) and can also be configured by a digital storage medium. The hardware component can further include the memory 360 as an internal / external component.

[0063] If a bitstream including video / video information is input, the decoding device 300 can restore the video corresponding to the process in which the video / video information was processed by the encoding device in FIG. 3. For example, the decoding device 300 can derive units / blocks based on the block division related information obtained from the bitstream. The decoding device 300 can perform decoding using the processing units applied in the encoding device. Therefore, the processing unit for decoding can be, for example, a coding unit, and the coding unit can be divided according to a quad-tree structure, a binary tree structure, and / or a ternary tree structure from a coding tree unit or a maximum coding unit. One or more transform units can be derived from the coding unit. Then, the restored video signal decoded and output via the decoding device 300 can be played back via a playback device.

[0064] The decoding device 300 can receive the signal output from the encoding device in FIG. 3 in the form of a bitstream, and the received signal can be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 can parse the bitstream to derive information (e.g., video / video information) necessary for video restoration (or picture restoration). The video / video information can further include information regarding various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Also, the video / video information can further include general constraint information. The decoding device can further decode a picture based on the information regarding the parameter set and / or the general constraint information. The signaling / received information and / or syntax elements described later in this document can be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 can decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the value of the syntax element necessary for video restoration and the quantized value of the transform coefficient regarding the residual. More specifically, the CABAC entropy decoding method receives the bin corresponding to each syntax element in the bitstream, determines a context model using the syntax element information to be decoded, the information adjacent to the decoding target block, and the information of the symbol / bin decoded in the previous step, predicts the occurrence probability of the bin based on the determined context model, performs arithmetic decoding of the bin, and can generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model by using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit 310, the information related to prediction is provided to the prediction unit (inter prediction unit 332 and intra prediction unit 331), and the residual value obtained by performing entropy decoding in the entropy decoding unit 310, that is, the quantized transform coefficient and related parameter information, can be input to the residual processing unit 320. The residual processing unit 320 can derive a residual signal (residual block, residual sample, residual sample array). Also, among the information decoded by the entropy decoding unit 310, the information related to filtering can be provided to the filtering unit 350. On the other hand, a receiving unit (not shown) that receives the signal output from the encoding device can be further configured as an internal / external element of the decoding device 300, or the receiving unit can be a component of the entropy decoding unit 310. On the other hand, the decoding device according to this document can be called a video / video / picture decoding device, and the decoding device can be classified into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The information decoder can include the entropy decoding unit 310, and the sample decoder can include at least one of the inverse quantization unit 321, inverse transform unit 322, addition unit 340, filtering unit 350, memory 360, inter prediction unit 332, and intra prediction unit 331.

[0065] In the inverse quantization unit 321, the quantized transform coefficients can be inverse quantized to output the transform coefficients. The inverse quantization unit 321 can reorder the quantized transform coefficients in a two-dimensional block form. In this case, the reordering can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit 321 can perform inverse quantization on the quantized transform coefficients using quantization parameters (for example, quantization step size information) to obtain the transform coefficients.

[0066] In the inverse transform unit 322, the transform coefficients are inverse transformed to obtain a residual signal (residual block, residual sample array).

[0067] The prediction unit can perform prediction on the current block and generate a predicted block including the predicted samples for the current block. The prediction unit can determine whether intra prediction or inter prediction is applied to the current block based on the information regarding the prediction output from the entropy decoding unit 310, and can determine a specific intra / inter prediction mode.

[0068] The prediction unit 320 can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction for the prediction of one block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). Also, the prediction unit can be based on the intra block copy (IBC) prediction mode or the palette mode for the prediction of a block. The IBC prediction mode or the palette mode can be used for content video / moving image coding such as games, for example, like SCC (screen content coding). IBC basically performs prediction within the current picture, but can be performed in the same way as inter prediction in terms of deriving a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. The palette mode can be regarded as an example of intra coding or intra prediction. When the palette mode is applied, information regarding the palette table and the palette index can be included in and signaled in the video / moving image information.

[0069] The intra prediction unit 331 can predict the current block by referring to samples within the current picture. The samples to be referred to can be located adjacent to or away from the current block depending on the prediction mode. In intra prediction, the prediction mode can include a plurality of non - directional modes and a plurality of directional modes. The intra prediction unit 331 can also determine the prediction mode to be applied to the current block using the prediction mode applied to an adjacent block.

[0070] The inter prediction unit 332 can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between adjacent blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the adjacent blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit 332 can configure a motion information candidate list based on adjacent blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and the information regarding the prediction can include information indicating the mode of inter prediction for the current block.

[0071] The adder 340 can generate a restored signal (restored picture, restored block, restored sample array) by adding the obtained residual signal to a prediction signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 332 and / or the intra prediction unit 331). When there is no residual for the block to be processed, as in the case where the skip mode is applied, the predicted block can be used as the restored block.

[0072] The addition unit 340 can be called a restoration unit or a restoration block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, can be output after filtering as described later, or can also be used for inter prediction of the next picture.

[0073] On the other hand, LMCS (luma mapping with chroma scaling) can also be applied during the picture decoding process.

[0074] The filtering unit 350 can apply filtering to the restoration signal to improve the subjective / objective image quality. For example, the filtering unit 350 can apply various filtering methods to the restored picture to generate a modified restored picture, and can transmit the modified restored picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0075] The (modified) restored picture stored in the DPB of the memory 360 can be used as a reference picture by the inter prediction unit 332. The memory 360 can store the motion information of the blocks for which the motion information in the current picture has been derived (or decoded) and / or the motion information of the blocks in the already restored picture. The stored motion information can be transmitted to the inter prediction unit 260 for utilization as the motion information of spatially adjacent blocks or temporally adjacent blocks. The memory 360 can store the restored samples of the restored blocks in the current picture and can transmit them to the intra prediction unit 331.

[0076] In this specification, the embodiments described in the filtering unit 260, the inter prediction unit 221, and the intra prediction unit 222 of the encoding apparatus 200 can be applied in the same or corresponding manner to the filtering unit 350, the inter prediction unit 332, and the intra prediction unit 331 of the decoding apparatus 300, respectively.

[0077] As described above, when performing video coding, prediction is performed to increase the compression efficiency. Through this, a predicted block including prediction samples for the current block, which is the block to be coded, can be generated. Here, the predicted block includes prediction samples in the spatial domain (or pixel domain). The predicted block is also derived in the encoding apparatus and the decoding apparatus, and the encoding apparatus can increase the video coding efficiency by signaling information (residual information) regarding the residual between the original block and the predicted block, which is not the original sample value of the original block, to the decoding apparatus. The decoding apparatus can derive a residual block including residual samples based on the residual information, and combine the residual block and the predicted block to generate a restored block including restored samples, and can generate a restored picture including the restored block.

[0078] The residual information can be generated through the conversion and quantization procedures. For example, an encoding device can derive a residual block between the original block and the predicted block, execute a conversion procedure on the residual samples (residual sample array) included in the residual block to derive conversion coefficients, execute a quantization procedure on the conversion coefficients to derive quantized conversion coefficients, and thus signal the relevant residual information (via a bitstream) to a decoding device. Here, the residual information can include information such as the value information, position information, conversion technique, conversion kernel, quantization parameter, etc. of the quantized conversion coefficients. The decoding device can execute an inverse quantization / inverse conversion procedure based on the residual information to derive residual samples (or a residual block). The decoding device can generate a restored picture based on the predicted block and the residual block. Also, the encoding device can inverse quantize / inverse convert the quantized conversion coefficients for reference in the inter prediction of subsequent pictures to derive a residual block and generate a restored picture based on this.

[0079] On the one hand, as described above, when predicting the current block, intra prediction or inter prediction can be applied. As one embodiment, when applying inter prediction to the current block, the prediction unit (more specifically, the inter prediction unit) of the encoding / decoding apparatus can perform inter prediction in block units to derive prediction samples. Inter prediction can indicate a prediction derived in a method that depends on data elements (such as sample values or motion information) of pictures other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) for the current block can be induced based on a reference block (reference sample array) specified by a motion vector on the reference picture pointed to by the reference picture index. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block can be predicted in block, sub-block, or sample units based on the correlation of motion information between the surrounding blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.) information. When inter prediction is applied, the surrounding blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called by names such as a collocated reference block or a collocated CU (colCU), and the reference picture including the temporal neighboring block may also be called a collocated picture (colPic). For example, a motion information candidate list can be configured based on the surrounding blocks of the current block, and flag or index information indicating which candidate is selected (used) can be signaled to derive the motion vector and / or reference picture index of the current block.Inter prediction is performed based on various prediction modes. For example, in the case of the skip mode and the merge mode, the motion information of the current block may be the same as that of the selected neighboring block. In the case of the skip mode, different from the merge mode, a residual signal may not be transmitted. In the case of the motion vector prediction (MVP) mode, the motion vector of the selected neighboring block can be used as a motion vector predictor, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0080] The motion information can include L0 motion information and / or L1 motion information according to the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). The motion vector in the L0 direction may be referred to as an L0 motion vector or MVL0, and the motion vector in the L1 direction may be referred to as an L1 motion vector or MVL1. The prediction based on the L0 motion vector may be referred to as L0 prediction, the prediction based on the L1 motion vector may be referred to as L1 prediction, and the prediction based on both the L0 motion vector and the L1 motion vector may be referred to as bi (Bi) prediction. Here, the L0 motion vector can indicate a motion vector related to the reference picture list L0 (L0), and the L1 motion vector can indicate a motion vector related to the reference picture list L1 (L1). The reference picture list L0 can include, as reference pictures, pictures that are earlier in output order than the current picture, and the reference picture list L1 can include pictures that are later in output order than the current picture. The earlier picture may be referred to as a forward (reference) picture, and the later picture may be referred to as a backward (reference) picture. The reference picture list L0 can further include, as reference pictures, pictures that are later in output order than the current picture. In this case, the earlier picture may be indexed first within the reference picture list L0, and the later picture may be indexed thereafter. The reference picture list L1 can further include, as reference pictures, pictures that are earlier in output order than the current picture. In this case, the later picture may be indexed first within the reference picture list 1, and the earlier picture may be indexed thereafter. Here, the output order may correspond to the POC (picture order count) order (order).

[0081] FIG. 4 shows an example of a schematic video / video encoding procedure to which the embodiments of this document are applicable. In FIG. 4, S400 can be executed by the prediction unit 220 of the encoding device detailed in FIG. 2, S410 can be executed by the residual processing unit 230, and S420 can be executed by the entropy encoding unit 240. S400 can include the inter / intra prediction procedures described in this document, S410 can include the residual processing procedures described in this document, and S420 can include the encoding procedures of the information described in this document.

[0082] Referring to FIG. 4, the video / video encoding procedure is not only a procedure of encoding information (e.g., prediction information, residual information, partitioning information, etc.) for restoring a picture schematically as shown in the description regarding FIG. 2 and outputting it in the form of a bitstream, but also a procedure of generating a restored picture for the current picture and a procedure (optional) of applying in-loop filtering to the restored picture. The encoding device can derive (modified) residual samples from the quantized transform coefficients via the inverse quantization unit 234 and the inverse transform unit 235, and can generate a restored picture based on the prediction samples that are the output of S400 and the (modified) residual samples. The restored picture generated in this way can be the same as the restored picture generated by the decoding device described above. A restored picture modified via the in-loop filtering procedure for the restored picture is generated, which is stored in the decoded picture buffer or memory 270 and can be used as a reference picture in the inter prediction procedure when encoding subsequent pictures, similar to the case of the decoding device. As described above, in some cases, part or all of the in-loop filtering procedure can be omitted. When the in-loop filtering procedure is performed, (in-loop) filtering-related information (parameters) can be encoded by the entropy encoding unit 240 and output in the form of a bitstream, and the decoding device can perform the in-loop filtering procedure in the same way as the encoding device based on the filtering-related information.

[0083] Through such in-loop filtering procedures, it is possible to reduce noises generated during the coding of images / videos, such as blocking artifacts and ringing artifacts, and improve the subjective / objective visual quality. Also, by both the encoding device and the decoding device performing the in-loop filtering procedures, the encoding device and the decoding device can derive the same prediction results, enhance the reliability of picture coding, and reduce the amount of data that must be transmitted for picture coding.

[0084] As described above, picture restoration procedures are performed not only in the decoding device but also in the encoding device. Restoration blocks can be generated based on intra prediction / inter prediction for each block unit, and a restored picture including the restoration blocks can be generated. When the current picture / slice / tile group is an I picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based only on intra prediction. On the other hand, when the current picture / slice / tile group is a P or B picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based on intra prediction or inter prediction. In this case, inter prediction may be applied to some blocks within the current picture / slice / tile group, and intra prediction may be applied to the remaining some blocks. The color components of a picture may include a luma component and a chroma component, and unless explicitly restricted in this document, the methods and embodiments proposed in this document can be applied to the luma component and the chroma component.

[0085] FIG. 5 shows an example of a schematic video / video decoding procedure to which the embodiments of this document are applicable. In FIG. 5, S500 can be executed by the entropy decoding unit 310 of the decoding apparatus detailed in FIG. 3, S510 can be executed by the prediction unit 330, S520 can be executed by the residual processing unit 320, S530 can be executed by the addition unit 340, and S540 can be executed by the filtering unit 350. S500 can include the decoding procedure of the information described in this document, S510 can include the inter / intra prediction procedure described in this document, S520 can include the residual processing procedure described in this document, S530 can include the block / picture restoration procedure described in this document, and S540 can include the in-loop filtering procedure described in this document.

[0086] Referring to FIG. 5, the picture decoding procedure can generally include a video / video information acquisition procedure (S500) from the bitstream (through decoding) as shown in the description regarding FIG. 3, a picture restoration procedure (S510 - S530), and an in-loop filtering procedure (S540) for the restored picture. The picture restoration procedure can be executed based on the predicted samples and the residual samples obtained through the inter / intra prediction (S510) and the residual processing (S520, inverse quantization and inverse transformation for the quantized transform coefficients) processes described in this document. Through the in-loop filtering procedure for the restored picture generated through the picture restoration procedure, a modified restored picture is generated, and the modified restored picture is output as the decoded picture, and is also stored in the decoded picture buffer or memory 360 of the decoding apparatus, and can be used as a reference picture in the inter prediction procedure when decoding subsequent pictures.

[0087] In some cases, the in-loop filtering procedure may be omitted. In this case, the restored picture is output as a decoded picture and stored in the decoded picture buffer or memory 360 of the decoding device, and can be used as a reference picture in the inter prediction procedure when decoding subsequent pictures. The in-loop filtering procedure (S540) may include, as described above, a deblocking filtering procedure, a SAO (sample adaptive offset) procedure, an ALF (adaptive loop filter) procedure, and / or a bilateral filter procedure, etc., and a part or all of them may be omitted. Also, one or a part of the deblocking filtering procedure, SAO (sample adaptive offset) procedure, ALF (adaptive loop filter) procedure, and bilateral filter procedure may be sequentially applied, or all of them may be sequentially applied. For example, after the deblocking filtering procedure is applied to the restored picture, the SAO procedure may be executed. Or, for example, after the deblocking filtering procedure is applied to the restored picture, the ALF procedure may be executed. This can be similarly executed in the encoding device.

[0088] On the other hand, as described above, the encoding device can perform entropy encoding based on various encoding methods such as exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. Also, the decoding device can perform entropy decoding based on coding methods such as exponential Golomb coding, CAVLC, or CABAC. Hereinafter, the entropy encoding / decoding procedure will be described.

[0089] FIG. 6 schematically shows an example of an entropy encoding method to which the embodiment of this document is applicable, and FIG. 7 schematically shows an entropy encoding unit in an encoding apparatus. The entropy encoding unit in the encoding apparatus of FIG. 7 can be applied to be identical or corresponding to the entropy encoding unit 240 of the encoding apparatus 200 in FIG. 2 described above.

[0090] Referring to FIGS. 6 and 7, an encoding apparatus (entropy encoding unit) can execute an entropy coding procedure for video / video information. The video / video information may include partitioning related information, prediction related information (e.g., inter / intra prediction partition information, intra prediction mode information, inter prediction mode information, etc.), residual information, in-loop filtering related information, etc., or may include various syntax elements related thereto. Entropy coding can be executed in units of syntax elements. S600 to S610 can be executed by the entropy encoding unit 240 of the encoding apparatus 200 in FIG. 2 described above.

[0091] The encoding apparatus can perform binarization on a target syntax element (S600). Here, the binarization can be based on various binarization methods such as Truncated Rice binarization process, Fixed-length binarization process, etc., and the binarization method for the target syntax element can be predefined. The binarization procedure can be executed by a binarization unit 242 in the entropy encoding unit 240.

[0092] The encoding device can perform entropy encoding on a target syntax element (S610). The encoding device can encode the bin string of the target syntax element based on entropy coding techniques such as CABAC (context - adaptive arithmetic coding) or CAVLC (context - adaptive variable length coding), based on regular coding (context - based) or bypass coding, and its output can be included in the bitstream. The entropy encoding procedure can be executed by the entropy encoding processing unit 243 in the entropy encoding unit 240. As described above, the bitstream can be transmitted to the decoding device via a (digital) storage medium or a network.

[0093] FIG. 8 schematically shows an example of an entropy decoding method to which the embodiments of this document are applicable, and FIG. 9 schematically shows the entropy decoding unit in the decoding device. The entropy decoding unit in the decoding device of FIG. 9 can be applied to be identical or corresponding to the entropy decoding unit 310 of the decoding device 300 in FIG. 3 described above.

[0094] Referring to FIGS. 8 and 9, the decoding device (entropy decoding unit) can decode the encoded video / video information. The video / video information can include partitioning - related information, prediction - related information (e.g., inter / intra prediction partition information, intra prediction mode information, inter prediction mode information, etc.), residual information, in - loop filtering - related information, etc., or can include various syntax elements related thereto. Entropy coding can be performed in units of syntax elements. S800 to S810 can be executed by the entropy decoding unit 310 of the decoding device 300 in FIG. 3 described above.

[0095] The decoding device can perform binarization on a target syntax element (S800). Here, the binarization can be based on various binarization methods such as Truncated Rice binarization process, Fixed-length binarization process, etc., and the binarization method for the target syntax element can be predefined. The decoding device can derive an available bit string (bit string candidate) for the available values of the target syntax element through the binarization procedure. The binarization procedure can be executed by the binarization unit 312 in the entropy decoding unit 310.

[0096] The decoding device can perform entropy decoding on a target syntax element (S810). The decoding device sequentially decodes and parses each bin for the target syntax element from the input bits in the bit stream, and compares the derived bit string with the available bit strings for the syntax element. If the derived bit string is the same as one of the available bit strings, the value corresponding to the bit string can be derived as the value of the syntax element. If not, after further parsing the next bit in the bit stream, the above-described procedure can be re-executed. Through such a process, without using start bits or end bits for specific information (specific syntax elements) in the bit stream, the information can be signaled using variable-length bits. Through this, relatively fewer bits can be allocated for lower values, and the overall coding efficiency can be improved.

[0097] The decoding device can decode each bin in the bit string based on context or bypass based on entropy coding techniques such as CABAC or CAVLC. Here, the bit stream can contain various information for video / video decoding as described above. As previously mentioned, the bit stream can be transmitted to the decoding device via a (digital) storage medium or a network.

[0098] Figure 10 exemplarily shows a hierarchical structure for coded video.

[0099] Referring to Figure 10, the coded video is divided into a video decoding process of the video, a VCL (Video Coding Layer) that handles itself, a lower-level system that transmits and stores the encoded information, and a NAL (Network Abstraction Layer) that exists between the VCL and the lower-level system and is responsible for network adaptation functions.

[0100] In the VCL, it is possible to generate VCL data including compressed video data (slice data), or generate a parameter set including information such as a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), a Video Parameter Set (VPS), or a SEI (Supplemental Enhancement Information) message that is additionally required in the video decoding process.

[0101] In the NAL, it is possible to generate a NAL unit by adding header information (NAL unit header) to the RBSP (Raw Byte Sequence Payload) generated in the VCL. At this time, the RBSP refers to slice data, parameter sets, SEI messages, etc. generated in the VCL. The NAL unit header may include NAL unit type information specified by the RBSP data included in the NAL unit.

[0102] In addition, NAL units can be classified into VCL NAL units and Non-VCL NAL units according to the RBSP generated by VCL. A VCL NAL unit can mean a NAL unit containing information (slice data) for video, and a Non-VCL NAL unit can mean a NAL unit containing information (parameter set or SEI message) necessary for decoding video.

[0103] VCL NAL units and Non-VCL NAL units can be transmitted via a network with header information attached according to the data standards of the lower-level system. For example, NAL units can be transformed into data forms of predetermined standards such as the H.266 / VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted via various networks.

[0104] As described above, the NAL unit type can be specified by the RBSP data structure (structure) included in the corresponding NAL unit, and information about such NAL unit types can be stored in the NAL unit header and signaled.

[0105] For example, depending on whether the NAL unit contains information (slice data) for video, it can be roughly classified into a VCL NAL unit type and a Non-VCL NAL unit type. The VCL NAL unit type can be classified according to the nature and type of the picture contained in the VCL NAL unit, and the Non-VCL NAL unit type can be classified according to the type of the parameter set, etc.

[0106] The following is an example of the NAL unit type specified according to the type of parameter set included in the Non-VCL NAL unit type, etc.

[0107] - APS (Adaptation Parameter Set) NAL unit: Type for NAL units containing APS

[0108] - DPS (Decoding Parameter Set) NAL unit: Type for NAL units containing DPS

[0109] - VPS (Video Parameter Set) NAL unit: Type for NAL units containing VPS

[0110] - SPS (Sequence Parameter Set) NAL unit: Type for NAL units containing SPS

[0111] - PPS (Picture Parameter Set) NAL unit: Type for NAL units containing PPS

[0112] - PH (Picture header) NAL unit: Type for NAL units containing PH

[0113] The above-mentioned NAL unit types have syntax information for the NAL unit types, and the syntax information can be stored in the NAL unit header and signaled. For example, the syntax information is nal_unit_type, and the NAL unit type can be specified by the nal_unit_type value.

[0114] On the one hand, as described above, one picture may include a plurality of slices, and one slice may include a slice header and slice data. In this case, one picture header may be further added for a plurality of slices (a set of slice headers and slice data) within one picture. The picture header (picture header syntax) may include information / parameters applicable in common to the picture. In this document, a tile group may be used interchangeably with or in place of a slice or a picture. Also, in this document, a tile group header may be used interchangeably with or in place of a slice header or a picture header.

[0115] The slice header (slice header syntax) may include information / parameters applicable in common to the slice. APS (APS syntax) or PPS (PPS syntax) may include information / parameters applicable in common to one or more slices or pictures. SPS (SPS syntax) may include information / parameters applicable in common to one or more sequences. VPS (VPS syntax) may include information / parameters applicable in common to multiple layers. DPS (DPS syntax) may include information / parameters applicable in common to the entire video. DPS may include information / parameters related to the concatenation of CVS (coded video sequence). In this document, the high level syntax (HLS) may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, picture header syntax, and slice header syntax.

[0116] In this document, the video / video information encoded by an encoding device and signaled in the form of a bitstream to a decoding device may include not only information related to partitioning within a picture, intra / inter prediction information, residual information, in-loop filtering information, etc., but also information included in a slice header, information included in a picture header, information included in an APS, information included in a PPS, information included in an SPS, information included in a VPS, and / or information included in a DPS. Further, the video / video information may further include information of an NAL unit header.

[0117] As described above, HLS (High level syntax) can be coded / signaled for video / video coding. In this document, the video / video information may include HLS. For example, a coded picture can be composed of one or more slices. Parameters describing the coded picture can be signaled within a picture header (PH), and parameters describing a slice can be signaled within a slice header (SH). The PH can be sent to its NAL unit type. The SH can be present at the start part of the NAL unit including the payload of the slice (i.e., slice data). Details regarding the syntax and semantics of the PH and SH are as disclosed in the VVC standard. Each picture can be associated with a PH. The picture can be composed of different types of slices which are intra-coded slices (i.e., I slices) and inter-coded slices (i.e., P-slices and B-slices). As a result, the PH can include syntax elements necessary for the intra-slices of the picture and the inter-slices of the picture.

[0118] On the one hand, generally, one NAL unit type can be set for one picture. The NAL unit type can be signaled via nal_unit_type in the NAL unit header of the NAL unit including the slice. nal_unit_type is syntax information for specifying the NAL unit type, that is, as shown in Table 1 or Table 2 below, it can specify the type of the RBSP data structure included in the NAL unit.

[0119] The following Table 1 shows an example of the codes of the NAL unit type and the classes of the NAL unit type.

[0120]

Table 1

[0121] Alternatively, as an example, the codes of the NAL unit type and the classes of the NAL unit type can also be defined as shown in Table 2 below.

[0122]

Table 2-1

[0123]

Table 2-2

[0124] As shown in Table 1 or Table 2 above, the name and value of the NAL unit type can be identified by the RBSP data structure included in the NAL unit, and the NAL unit can be classified into a VCL NAL unit type and a Non-VCL NAL unit type according to whether the NAL unit contains information (slice data) for video. The VCL NAL unit type can be classified according to the nature and type of the picture, etc., and the Non-VCL NAL unit type can be classified according to the type of parameter set, etc. For example, the NAL unit type can be identified according to the nature and type of the picture included in the VCL NAL unit as follows.

[0125] TRAIL: Indicates the type for a NAL unit containing slice data coded for a trailing picture / sub-picture. For example, nal_unit_type can be defined as TRAIL_NUT, and the value of nal_unit_type can be identified as 0.

[0126] Here, a trailing picture refers to a picture that follows a picture that is randomly accessible in both the output order and the decoding order. A trailing picture can be a non-IRAP picture that follows the related IRAP picture in the output order and is not an STSA picture. For example, a trailing picture associated with an IRAP picture follows the IRAP picture in the decoding order. A picture that follows the related IRAP picture in the output order and precedes the related IRAP picture in the decoding order is not allowed.

[0127] STSA (Step-wise Temporal Sub-layer Access): Indicates the type for a NAL unit containing slice data coded for an STSA picture / sub-picture. For example, nal_unit_type can be defined as STSA_NUT, and the value of nal_unit_type can be identified as 1.

[0128] Here, an STSA picture is a picture in a bitstream that supports temporal scalability and allows switching between temporal sublayers, and in a lower sublayer, it is a picture that indicates a position where up-switching to a higher sublayer one level higher than the lower sublayer is possible. An STSA picture does not use a picture in the same TemporalId and the same layer as the STSA picture for reference in inter prediction. In the decoding order, a picture that follows an STSA picture in the same TemporalId and the same layer as the STSA picture does not use a picture before the STSA picture in the same TemporalId and the same layer as the STSA picture in the decoding order for reference in inter prediction. An STSA picture is an STSA picture that enables up-switching to the sublayer containing the STSA picture in the immediately lower sublayer. In this case, the picture to be coded must not belong to the lowest sublayer. That is, an STSA picture must always have a TemporalId greater than 0.

[0129] RADL (random access decodable leading (picture)): Indicates the type for an NAL unit containing coded slice data of an RADL picture / sublpicture. For example, nal_unit_type can be defined as RADL_NUT, and the value of nal_unit_type can be specified as 2.

[0130] Here, all RADL pictures are leading pictures. RADL pictures are not used as reference pictures for the decoding process of trailing pictures of the same associated IRAP picture. Specifically, a RADL picture with a nuh_layer_id such as layerId is not used as a reference picture for the decoding process of a picture with a nuh_layer_id such as layerId that follows the IRAP picture associated with the RADL picture in output order. When the field_seq_flag (i.e., sps_field_seq_flag) is 0, all RADL pictures (i.e., if RADL pictures exist) precede all non-leading pictures of the same associated IRAP picture in decoding order. On the other hand, a leading picture refers to a picture that precedes the associated IRAP picture in output order.

[0131] RASL (random access skipped leading (picture)): Indicates the type for NAL units containing coded slice data of RASL pictures / subpictures. For example, nal_unit_type can be defined as RASL_NUT, and the value of nal_unit_type can be specified as 3.

[0132] Here, all RASL pictures are the leading pictures of the associated CRA pictures. If the associated CRA picture has a NoOutputBeforeRecoveryFlag whose value is 1, since the RASL picture may include a reference to a picture that does not exist in the bitstream, the RASL picture may not be output and may not be decoded accurately. The RASL picture is not used as a reference picture for the decoding process of non-RASL pictures of the same layer. However, the RADL sub-pictures in the RASL pictures of the same layer can be used for inter prediction for the collocated RADL sub-pictures in the RADL pictures associated with the same CRA picture as the RASL picture. When field_seq_flag (i.e., sps_field_seq_flag) is 0, all RASL pictures (i.e., if RASL pictures exist) precede all non-leading pictures of the same associated CRA picture in decoding order.

[0133] There may be reserved nal_unit_types for non-IRAP VCL NAL unit types. For example, nal_unit_type can be defined as RSV_VCL_4 to RSV_VCL_6, and the values of nal_unit_type can be specified as 4 to 6 respectively.

[0134] Here, IRAP (intra random access point) is information indicating a NAL unit for a picture where random access is possible. The IRAP picture can be a CRA picture or an IDR picture. For example, the IRAP picture refers to a picture having a NAL unit type defined with nal_unit_type being IDR_W_RADL, IDR_N_LP, CRA_NU as in Table 1 or Table 2 above, and the values of nal_unit_type can be specified as 7 to 9 respectively.

[0135] The IRAP picture does not use a reference picture in the same layer for inter prediction during the decoding process. In other words, the IRAP picture does not refer to other pictures other than itself for inter prediction during the decoding process. In the decoding order, the first picture in the bitstream becomes an IRAP or GDR picture. For a single-layer bitstream, if the parameter set required when reference is necessary is available, all subsequent non-RASL pictures and IRAP pictures of the CLVS (coded layer video sequence) in the decoding order can be accurately decoded without performing the decoding process of the pictures preceding the IRAP picture in the decoding order.

[0136] The value of the mixed_nalu_types_in_pic_flag for the IRAP picture is 0. When the value of the mixed_nalu_types_in_pic_flag for the picture is 0, one slice in the picture may have a NAL unit type (nal_unit_type) within the range from IDR_W_RADL to CRA_NUT (for example, the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), and all other slices in the picture may have the same NAL unit type (nal_unit_type). In this case, the picture can be regarded as an IRAP picture.

[0137] IDR (instantaneous decoding refresh): Indicates the type for the NAL unit containing the coded slice data of the IDR picture / subpicture. For example, the nal_unit_type for the IDR picture / subpicture can be defined as IDR_W_RADL or IDR_N_LP, and the values of nal_unit_type can be specified as 7 or 8 respectively.

[0138] Here, an IDR picture does not use inter prediction in the decoding process (i.e., it does not refer to other pictures other than itself for inter prediction), and it can be the first picture in the decoding order within the bitstream or be shown later in the bitstream (i.e., not the first but later). Each IDR picture is the first picture of a CVS (coded video sequence) in the decoding order. For example, if an IDR picture has a relationship with a decodable leading picture, the NAL unit type of the IDR picture may be indicated as IDR_W_RADL, and if the IDR picture has no relationship with a leading picture, the NAL unit type of the IDR picture may be indicated as IDR_N_LP. That is, an IDR picture with an NAL unit type of IDR_W_RADL may not have a related RASL picture existing in the bitstream but may have a related RADL picture in the bitstream. An IDR picture with an NAL unit type of IDR_N_LP does not have a related leading picture existing in the bitstream.

[0139] CRA (clean random access): It indicates the type for a NAL unit containing coded slice data of a CRA picture / subpicture. For example, nal_unit_type can be defined as CRA_NUT, and the value of nal_unit_type can be specified as 9.

[0140] Here, the CRA picture does not use inter prediction in the decoding process (i.e., without referring to other pictures other than itself for inter prediction), and it can be the first picture in the decoding order in the bitstream or be shown later in the bitstream (i.e., later than the first). The CRA picture may have related RADL or RASL pictures existing in the bitstream. For a CRA picture with the value of NoOutputBeforeRecoveryFlag being 1, the related RASL picture may not be output by the decoder. This is because it may contain references to pictures that do not exist in the bitstream, and in this case, it cannot be decoded.

[0141] GDR (gradual decoding refresh): Indicates the type for the NAL unit containing the coded slice data of the GDR picture / subpicture. For example, nal_unit_type can be defined as GDR_NUT, and the value of nal_unit_type can be specified as 10.

[0142] Here, the value of pps_mixed_nalu_types_in_pic_flag for the GDR picture can be 0. When the value of pps_mixed_nalu_types_in_pic_flag for a picture is 0 and one slice in the picture has the NAL unit type of GDR_NUT, all other slices in the picture shall have the value of the same NAL unit type (nal_unit_type). In this case, the picture can become a GDR picture after receiving the first slice.

[0143] Also, for example, according to the types of parameters included in the Non-VCL NAL unit, the NAL unit type can be determined. As shown in Table 1 or Table 2 above, it may include NAL unit types (nal_unit_type) such as VPS_NUT indicating the type for the NAL unit containing the video parameter set, SPS_NUT indicating the type for the NAL unit containing the sequence parameter set, PPS_NUT indicating the type for the NAL unit containing the picture parameter set, and PH_NUT indicating the type for the NAL unit containing the picture header.

[0144] On the other hand, a bitstream that supports temporal scalability (or a temporally scalable bitstream) includes information for a temporally scaled temporal layer. The information for the temporal layer can be the identification information of the temporal layer specified by the temporal scalability of the NAL unit. For example, the identification information of the temporal layer can use the temporal_id syntax information, and the temporal_id syntax information can be stored in the NAL unit header by the encoding device and signaled to the decoding device. Hereinafter, in this specification, the temporal layer may also be referred to as a sub-layer or a temporal sub-layer or a temporal scalable layer.

[0145] FIG. 11 is a diagram showing the temporal layer structure for the NAL unit in a bitstream that supports temporal scalability.

[0146] When a bitstream supports temporal scalability, the NAL units included in the bitstream have identification information for the temporal layer (e.g., temporal_id). As an example, the temporal layer composed of NAL units with a temporal_id value of 0 can provide the lowest temporal scalability, and the temporal layer composed of NAL units with a temporal_id value of 2 can provide the highest temporal scalability.

[0147] In FIG. 11, the box marked with I refers to an I picture, and the box marked with B refers to a B picture. Also, the arrows indicate the reference relationship regarding whether a picture references other pictures.

[0148] As shown in FIG. 11, the NAL units of the temporal layer with a temporal_id value of 0 are reference pictures that can be referenced by the NAL units of the temporal layers with temporal_id values of 0, 1, or 2. The NAL units of the temporal layer with a temporal_id value of 1 are reference pictures that can be referenced by the NAL units of the temporal layers with temporal_id values of 1 or 2. The NAL units of the temporal layer with a temporal_id value of 2 may be reference pictures that can be referenced by the NAL units of the same temporal layer, i.e., the temporal layer with a temporal_id value of 2, or may be non-reference pictures that are not referenced by other pictures.

[0149] If, as shown in FIG. 11, the NAL units of the temporal layer with a temporal_id value of 2, i.e., the highest temporal layer, are non-reference pictures, such NAL units can be extracted (or removed) from the bitstream without affecting other pictures during the decoding process.

[0150] On the one hand, among the above-mentioned NAL unit types, the IDR and CRA types indicate NAL units containing pictures that allow random access (or splicing), that is, RAP (Random Access Point) or IRAP (Intra Random Access Point) pictures that become random access points. In other words, an IRAP picture can be an IDR or CRA picture and can only contain I slices. In the bitstream, the first picture in the decoding order is an IRAP picture.

[0151] If an IRAP picture (IDR or CRA picture) is included in the bitstream, there may be pictures whose output order precedes that of the IRAP picture but whose decoding order follows it. Such pictures are called Leading Pictures (LP).

[0152] Figure 12 is a diagram for explaining pictures that allow random access.

[0153] Pictures that allow random access, that is, RAP or IRAP pictures that become random access points, are the first pictures in the decoding order in the bitstream during random access and only contain I slices.

[0154] Figure 12 shows the output order (output order or display order) and decoding order of pictures. As shown, the output order and decoding order of pictures may be different from each other. For convenience, pictures are described by being divided into predetermined groups.

[0155] Pictures belonging to the first group (I) indicate pictures that precede the IRAP picture in all of the output order and the decoding order. Pictures belonging to the second group (II) indicate pictures that precede the IRAP picture in the output order but follow it in the decoding order. Pictures of the third group (III) follow the IRAP picture in both the output order and the decoding order.

[0156] Pictures of the first group (I) can be decoded and output regardless of the IRAP picture.

[0157] Pictures belonging to the second group (II) that are output prior to the IRAP picture are called leading pictures. Leading pictures may cause problems during the decoding process when the IRAP picture is used as a random access point.

[0158] Pictures belonging to the third group (III) whose output and decoding order follow the IRAP picture are called normal pictures. Normal pictures are not used as reference pictures for leading pictures.

[0159] The random access point where random access occurs in the bitstream is the IRAP picture, and random access starts while the first picture of the second group (II) is being output.

[0160] Figure 13 is a diagram for explaining the IDR picture.

[0161] An IDR picture is a picture that becomes a random access point when a Group of Picture has a closed structure. Since an IDR picture is an IRAP picture as described above, it contains only I slices and may be the first picture in the decoding order in the bitstream or may come in the middle of the bitstream. When an IDR picture is decoded, all reference pictures stored in the DPB (decoded picture buffer) are marked as "unused for reference".

[0162] The bars shown in FIG. 13 represent pictures, and the arrows indicate the reference relationships regarding whether a picture can use other pictures as reference pictures. The x marks shown on the arrows indicate that the picture cannot reference the picture pointed to by the arrow.

[0163] As shown, the picture with a POC of 32 is an IDR picture. The pictures with POCs from 25 to 31 and output before the IDR picture are leading pictures 1310. The pictures with POCs of 33 or more correspond to normal pictures 1320.

[0164] In the output order, the leading pictures 1310 preceding the IDR picture can use the IDR picture and other leading pictures as reference pictures, but in the output order and decoding order, the past pictures 1330 preceding the leading pictures 1310 cannot be used as reference pictures.

[0165] In the output order and decoding order, the normal pictures 1320 following the IDR picture can be decoded by referring to the IDR picture, leading pictures, and other normal pictures.

[0166] FIG. 14 is a diagram for explaining a CRA picture.

[0167] A CRA picture is a picture that becomes a random access point when the Group of Picture has an open structure. As described above, since a CRA picture is also an IRAP picture, it only contains I slices and may be the first picture in the bitstream in decoding order, or may be in the middle of the bitstream for normal playback.

[0168] The bars shown in FIG. 14 represent pictures, and the arrows indicate the reference relationship regarding whether a picture can use other pictures as reference pictures. The x marks shown on the arrows indicate that the picture or pictures cannot reference the picture pointed to by the arrow.

[0169] In output order, the leading picture 1410 preceding the CRA picture can use all of the CRA picture, other leading pictures, and the past pictures 1430 preceding the leading picture 1410 in output order and decoding order as reference pictures.

[0170] On the contrary, in output order and decoding order, the normal picture 1420 following the CRA picture can be decoded by referring to the CRA picture and other normal pictures. The normal picture 1420 may not use the leading picture 1410 as a reference picture.

[0171] On the one hand, in the VVC standard, the picture to be coded (i.e., the current picture) can include slices of other NAL unit types. Whether the current picture includes slices of other NAL unit types can be indicated based on the syntax element mixed_nalu_types_in_pic_flag. For example, when the current picture includes slices of other NAL unit types, the value of the syntax element mixed_nalu_types_in_pic_flag can be indicated as 1. At this time, the current picture must refer to a PPS that includes mixed_nalu_types_in_pic_flag with a value of 1. The semantics of the flag (mixed_nalu_types_in_pic_flag) are as follows.

[0172] When the value of the syntax element mixed_nalu_types_in_pic_flag is 1, it can indicate that each picture referring to the PPS has one or more VCL NAL units, the VCL NAL units do not have the same value of NAL unit type (nal_unit_type), and the picture is not an IRAP picture.

[0173] When the value of the syntax element mixed_nalu_types_in_pic_flag is 0, it can indicate that each picture referring to the PPS has one or more VCL NAL units, and the VCL NAL units of each picture referring to the PPS have the same value of NAL unit type (nal_unit_type).

[0174] When the value of no_mixed_nalu_types_in_pic_constraint_flag is 1, the value of mixed_nalu_types_in_pic_flag must be 0. The no_mixed_nalu_types_in_pic_constraint_flag syntax element indicates the constraint regarding whether the value of mixed_nalu_types_in_pic_flag for a picture must be 0. For example, based on the no_mixed_nalu_types_in_pic_constraint_flag information signaled from a higher-level syntax (e.g., PPS) or a syntax containing information regarding constraints (e.g., GCI; general constraints information), it can be determined whether the value of mixed_nalu_types_in_pic_flag must be 0.

[0175] For each slice in picture picA that includes one or more slices with other values of NAL unit type (i.e., when the value of mixed_nalu_types_in_pic_flag for picture picA is 1), and having an NAL unit type value nalUnitTypeA within the range from IDR_W_RADL to CRA_NUT (e.g., the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), the following can be applied.

[0176] - The slice must belong to a subpicture subpicA for which the value of the corresponding subpic_treated_as_pic_flag[i] is 1. Here, subpic_treated_as_pic_flag[i] is information regarding whether the i-th subpicture of each coded picture in CLVS is treated as a picture in the decoding process excluding the in-loop filtering operation. For example, when the value of subpic_treated_as_pic_flag[i] is 1, it may indicate that the i-th subpicture is treated as a picture in the decoding process excluding the in-loop filtering operation. Alternatively, when the value of subpic_treated_as_pic_flag[i] is 0, it may indicate that the i-th subpicture is not treated as a picture in the decoding process excluding the in-loop filtering operation.

[0177] - The slice must not belong to a subpicture of picA that contains a VCL NAL unit having a NAL unit type (nal_unit_type) different from nalUnitTypeA.

[0178] - For all subsequent PUs in CLVS in decoding order, RefPicList[0] or RefPicList[1] of the slice within subpicA should not contain pictures that precede picA in decoding order with active entries.

[0179] To operate the concepts as described above, it can be specified as follows. For example, for the VCL NAL units of a specific picture, it can be applied as follows.

[0180] - When the value of mixed_nalu_types_in_pic_flag is 0, the value of the NAL unit type (nal_unit_type) must be the same for all coded slice NAL units within the picture. A picture or a PU may be considered to have the same NAL unit type as the coded slice NAL units within the picture or the PU.

[0181] - Otherwise (when the value of mixed_nalu_types_in_pic_flag is 1), all one or more VCL NAL units must have a NAL unit type of a specific value within the range from IDR_W_RADL to CRA_NUT (for example, the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), and all other VCL NAL units must have a NAL unit type of a specific value within the range from TRAIL_NUT to RSV_VCL_6 (for example, the value of the NAL unit type is from 0 to 6 in Table 1 or Table 2 above), or must have the same NAL unit type as GRA_NUT.

[0182] Currently, in the VVC standard, for pictures with mixed NAL unit types, there may be at least the following problems.

[0183] 1. When a picture contains IDR and non-IRAP NAL units, and the signaling for the RPL (reference picture list) is present in the slice header, such signaling must also be present in the header of the IDR slice. The RPL signaling is present in the slice header of the IDR slice when the value of sps_idr_rpl_present_flag is 1. Currently, even when there is one or more pictures with a mixed NAL unit type, the value of such a flag (sps_idr_rpl_present_flag) can be 0. Here, the sps_idr_rpl_present_flag syntax element can indicate whether the RPL syntax element can be present in the slice header of a slice with a NAL unit type such as IDR_N_LP or IDR_W_RADL. For example, when the value of sps_idr_rpl_present_flag is 1, it can indicate that the RPL syntax element can be present in the slice header of a slice with a NAL unit type such as IDR_N_LP or IDR_W_RADL. Alternatively, when the value of sps_idr_rpl_present_flag is 0, it can indicate that the RPL syntax element is not present in the slice header of a slice with a NAL unit type such as IDR_N_LP or IDR_W_RADL.

[0184] 2. When the current picture refers to a PPS with the value of mixed_nalu_types_in_pic_flag being 1, one or more of the VCL NAL units of the current picture must all have a NAL unit type of a specific value within the range from IDR_W_RADL to CRA_NUT (for example, the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), and all other VCL NAL units must either have a NAL unit type of a specific value within the range from TRAIL_NUT to RSV_VCL_6 (for example, the value of the NAL unit type is from 0 to 6 in Table 1 or Table 2 above), or have the same NAL unit type as GRA_NUT. Such a constraint is only applicable to the current picture including the case of a mixed IRAP and non-IRAP NAL unit type. However, it has not yet been properly applied to pictures including the case of a mixed RASL / RADL and non-IRAP NAL unit type.

[0185] This document provides solutions to the above-mentioned problems. That is, as described above, a picture (i.e., the current picture) including two or more sub-pictures may have a mixed NAL unit type. In the case of the current VVC standard, a picture with a mixed NAL unit type may have a form mixed with an IRAP NAL unit type and a non-IRAP NAL unit type. However, although a leading picture related to the CRA NAL unit type may sometimes have a form mixed with a non-IRAP NAL unit type, such a picture with a mixed NAL unit type is not currently supported by the standard. Therefore, a solution for a picture having a form mixed with a CRA NAL unit type and a non-IRAP NAL unit type is needed.

[0186] Therefore, this document provides a method for allowing a picture that includes a reading picture NAL unit type (e.g., RASL_NUT, RADL_NUT) and other non-IRAP NAL unit types (e.g., TRAIL_NUT, STSA, NUT) in a mixed form. Further, this document defines the constraints for making the reference picture list exist or be signaled when an IDR sub-picture and other non-IRAP sub-pictures are mixed. Therefore, for a picture having a mixed NAL unit type, not only IRAP but also a CRA NAL unit is provided in a mixed form to have more flexible characteristics.

[0187] For example, it can be applied as in the following embodiments, thereby solving the above problems. The following embodiments may be applied individually or in combination.

[0188] As one embodiment, when allowing a picture to have a mixed NAL unit type (when the value of mixed_nal_types_in_pic_flag is 1), the signaling for the reference picture list can also exist for a slice having an IDR type of NAL unit type (e.g., IDR_W_RADL or IDR_N_LP). This constraint can be shown as follows.

[0189] - When there is at least one PPS that refers to an SPS where the value of mixed_nal_types_in_pic_flag equals 1, the value of sps_idr_rpl_present_flag must be 1. Such a constraint can be a requirement for the conformance of the bitstream.

[0190] Alternatively, as one embodiment, for a picture having a mixed NAL unit type, it is allowed for the picture to include slices having a specific NAL unit type of a leading picture (e.g., RADL or RASL) and a specific NAL unit type of a non-leading picture, non-IRAP. This can be shown as follows.

[0191] For the VCL NAL units of a specific picture, the following can be applied.

[0192] - When the value of mixed_nalu_types_in_pic_flag is 0, the value of the NAL unit type (nal_unit_type) must be the same for all coded slice NAL units within the picture. The picture or PU can be regarded as having the same NAL unit type as the coded slice NAL unit within the picture or the PU.

[0193] - Otherwise (when the value of mixed_nalu_types_in_pic_flag is 1), one of the following must be satisfied (i.e., one of the following can have a true value).

[0194] 1) All one or more VCL NAL units must have a NAL unit type (nal_unit_type) of a specific value within the range from IDR_W_RADL to CRA_NUT (e.g., the value of the NAL unit type is from 7 to 9 in Table 1 or Table 2 above), and all other VCL NAL units must have a NAL unit type of a specific value within the range from TRAIL_NUT to RSV_VCL_6 (e.g., the value of the NAL unit type is from 0 to 6 in Table 1 or Table 2 above), or must have the same NAL unit type as GRA_NUT.

[0195] 2) Any one or more VCL NAL units must have a specific value of the same NAL unit type as RADL_NUT (for example, the value of the NAL unit type in Table 1 or Table 2 is 2) or RASL_NUT (for example, the value of the NAL unit type in Table 1 or Table 2 is 3), and any other VCL NAL units must have a specific value of the same NAL unit type as TRAIL_NUT (for example, the value of the NAL unit type in Table 1 or Table 2 is 0), STSA_NUT (for example, the value of the NAL unit type in Table 1 or Table 2 is 1), RSV_VCL_4 (for example, the value of the NAL unit type in Table 1 or Table 2 is 4), RSV_VCL_5 (for example, the value of the NAL unit type in Table 1 or Table 2 is 5), RSV_VCL_6 (for example, the value of the NAL unit type in Table 1 or Table 2 is 6), or GRA_NUT.

[0196] On the other hand, this document also proposes a method for providing a picture with a mixed NAL unit type as described above for a single-layer bitstream. As an embodiment, in the case of a single-layer bitstream, the following constraints may apply.

[0197] - Except for the first picture in decoding order within the bitstream, each picture is considered to be related to a previous IRAP picture in decoding order.

[0198] - If a picture is a leading picture of an IRAP picture, it must be a RADL or RASL picture.

[0199] - If a picture is a trailing picture of an IRAP picture, it must not be a RADL or RASL picture.

[0200] - There must be no RASL pictures in the bitstream related to an IDR picture.

[0201] - In the bitstream associated with an IDR picture where the NAL unit type (nal_unit_type) is IDR_N_LP, there shall be no RADL pictures.

[0202] When referenced, if each parameter set is available, random access can be performed at the position of the IRAP PU by discarding all PUs before the IRAP PU (also, the IRAP picture and all subsequent non-RASL pictures can be accurately decoded in decode order).

[0203] - Pictures preceding an IRAP picture in decode order shall precede the IRAP picture in output order and shall precede the RADL pictures associated with the IRAP picture in output order.

[0204] - RASL pictures associated with a CRA picture shall precede the RADL pictures associated with the CRA picture in output order.

[0205] - RASL pictures associated with a CRA picture shall follow in output order the IRAP pictures that precede the CRA picture in decode order.

[0206] - If the value of field_seq_flag is 0 and the current picture is a leading picture associated with an IRAP picture, it shall precede in decode order all non-leading pictures associated with the same IRAP picture. Otherwise, if pictures picA and picB are the first and last leading pictures associated with an IRAP picture in decode order respectively, there shall be at most one non-leading picture that precedes picA in decode order, and there shall be no non-leading pictures between picA and picB in decode order.

[0207] The following drawings are created to illustrate a specific example of this document. Since the names of specific devices, specific terms and names described in the drawings (for example, the names of syntax / syntax elements, etc.) are presented exemplarily, the technical features of this document are not limited to the specific names used in the following drawings.

[0208] FIG. 15 schematically shows an example of a video / video encoding method to which the embodiments of this document are applicable. The method disclosed in FIG. 15 can be executed by the encoding device 200 disclosed in FIG. 2.

[0209] Referring to FIG. 15, the encoding device can determine the NAL unit type for a slice within a picture (S1500).

[0210] For example, as described in Table 1 and Table 2 above, the encoding device can determine the NAL unit type according to the nature and type of a picture or sub-picture, etc., and can determine the NAL unit type for each of the slices based on the NAL unit type of the picture or sub-picture.

[0211] For example, for the case where the value of mixed_nalu_types_in_pic_flag is 0, the slices within the picture related to the PPS can be determined to have the same NAL unit type. That is, when the value of mixed_nalu_types_in_pic_flag is 0, the NAL unit type defined in the first NAL unit header of the first NAL unit including information for the first slice of the picture is the same as the NAL unit type defined in the second NAL unit header of the second NAL unit including information for the second slice of the same picture. Alternatively, for the case where the value of mixed_nalu_types_in_pic_flag is 1, the slices within the picture related to the PPS can be determined to have other NAL unit types. Here, the NAL unit type for a slice within a picture can be determined based on the method proposed in the foregoing embodiments.

[0212] The encoding device can generate NAL unit type related information (S1510). The NAL unit type related information may include information / syntax elements related to the NAL unit type described in the foregoing embodiments and / or the aforementioned Tables 1 and 2. For example, the information related to the NAL unit type may include the mixed_nalu_types_in_pic_flag syntax element included in the PPS. Further, the information related to the NAL unit type may include the nal_unit_type syntax element in the NAL unit header of the NAL unit including information for the coded slice.

[0213] The encoding device can generate a bitstream (S1520). The bitstream may include at least one NAL unit including video information for the coded slice. Further, the bitstream may include the PPS.

[0214] FIG. 16 schematically shows an example of a video / video decoding method to which the embodiments of this document are applicable. The method disclosed in FIG. 16 can be executed by the decoding device 300 disclosed in FIG. 3.

[0215] Referring to FIG. 16, the decoding device can receive a bitstream (S1600). Here, the bitstream may include at least one NAL unit including video information for the coded slice. Further, the bitstream may include the PPS.

[0216] The decoding device can obtain NAL unit type related information (S1610). The NAL unit type related information may include information / syntax elements related to the NAL unit type described in the foregoing embodiments and / or the foregoing Tables 1 and 2. For example, the information related to the NAL unit type may include the mixed_nalu_types_in_pic_flag syntax element included in the PPS. Further, the information related to the NAL unit type may include the nal_unit_type syntax element in the NAL unit header of the NAL unit including information for the coded slice.

[0217] The decoding device can determine the NAL unit type for the slice within the picture (S1620).

[0218] For example, when the value of mixed_nalu_types_in_pic_flag is 0, the slices within the picture related to the PPS use the same NAL unit type. That is, when the value of mixed_nalu_types_in_pic_flag is 0, the NAL unit type defined in the first NAL unit header of the first NAL unit including information for the first slice of the picture is the same as the NAL unit type defined in the second NAL unit header of the second NAL unit including information for the second slice of the same picture. Alternatively, when the value of mixed_nalu_types_in_pic_flag is 1, the slices within the picture related to the PPS use other NAL unit types. Here, the NAL unit type for the slice within the picture can be determined based on the method proposed in the foregoing embodiments.

[0219] The decoding device can decode / restore samples / blocks / slices based on the NAL unit type of the slice. The samples / blocks within the slice can be decoded / restored based on the NAL unit type of the slice.

[0220] For example, when a first NAL unit type is set for the first slice of the current picture and a second NAL unit type (different from the first NAL unit type) is set for the second slice of the current picture, the samples / blocks within the first slice or the first slice itself can be decoded / restored based on the first NAL unit type, and the samples / blocks within the second slice or the second slice itself can be decoded / restored based on the second NAL unit type.

[0221] FIGS. 17 and 18 schematically show an example of a video / visual encoding method and related components according to an embodiment of the present document.

[0222] The method disclosed in FIG. 17 can be executed by the encoding device 200 disclosed in FIG. 2 or FIG. 18. Here, the encoding device 200 disclosed in FIG. 18 is a simplified representation of the encoding device 200 disclosed in FIG. 2. Specifically, steps S1700 to S1720 in FIG. 17 can be executed by the entropy encoding unit 240 disclosed in FIG. 2, and depending on the embodiment, each step can also be executed by the video segmentation unit 210, prediction unit 220, residual processing unit 230, addition unit 340, etc. disclosed in FIG. 2. Furthermore, the method disclosed in FIG. 17 can be executed including the embodiments described above in this document. Therefore, in FIG. 17, regarding the content overlapping with the above-described embodiments, specific explanations will be omitted or simplified.

[0223] Referring to FIG. 17, the encoding device can determine the NAL unit type for the NAL units within the current picture (S1700).

[0224] The current picture may include a plurality of slices, and one slice may include a slice header and slice data. Also, an NAL unit header can be added to the slice (slice header and slice data) to generate an NAL unit. The NAL unit header may include NAL unit type information specified by the slice data included in the NAL unit.

[0225] As one embodiment, the encoding device can generate a first NAL unit for a first slice in the current picture and a second NAL unit for a second slice in the current picture. Further, the encoding device can determine a first NAL unit type for the first slice and a second NAL unit type for the second slice according to the types of the first and second slices.

[0226] For example, as shown in Table 1 or Table 2 above, the NAL unit type may include TRAIL_NUT, STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, CRA_NUT, etc. based on the type of slice data included in the NAL unit. Also, the NAL unit type may be signaled based on the nal_unit_type syntax element in the NAL unit header. The nal_unit_type syntax element is syntax information for identifying the NAL unit type, and as shown in Table 1 or Table 2 above, it may be indicated by a specific value corresponding to a specific NAL unit type.

[0227] The encoding device can generate NAL unit type-related information for the NAL unit type (S1710).

[0228] The NAL unit type related information may include information / syntax elements related to the NAL unit type described in the foregoing embodiments and / or the aforementioned Tables 1 and 2. For example, the NAL unit type related information may be information regarding whether the current picture has a mixed NAL unit type, and may be indicated by the mixed_nalu_types_in_pic_flag syntax element included in the PPS. For example, when the value of the mixed_nalu_types_in_pic_flag syntax element is 0, it may indicate that the NAL units within the current picture have the same NAL unit type. Alternatively, when the value of the mixed_nalu_types_in_pic_flag syntax element is 1, it may indicate that the NAL units within the current picture have other NAL unit types.

[0229] As one embodiment, when all of the NAL unit types for the NAL units within the current picture are the same, the encoding device can generate NAL unit type related information (e.g., mixed_nalu_types_in_pic_flag) having a value of 0. Alternatively, when the NAL unit types for the NAL units within the current picture are not the same, the encoding device can generate NAL unit type related information (e.g., mixed_nalu_types_in_pic_flag) having a value of 1.

[0230] That is, based on the NAL unit type related information regarding that the current picture has a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice of the current picture and the second NAL unit for the second slice of the current picture may have different NAL unit types. Alternatively, based on the NAL unit type related information regarding that the current picture does not have a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 0), the first NAL unit for the first slice of the current picture and the second NAL unit for the second slice of the current picture may have the same NAL unit type.

[0231] As an example, based on the NAL unit type related information regarding that the current picture has a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice may have a leading picture NAL unit type, and the second NAL unit for the second slice may have a non-IRAP NAL unit type or a non-leading picture NAL unit type. Here, the leading picture NAL unit type may include a RADL NAL unit type or a RASL NAL unit type, and the non-IRAP NAL unit type or the non-leading picture NAL unit type may include a trail NAL unit type or a STSA NAL unit type.

[0232] Alternatively, as an example, based on NAL unit type related information regarding having a NAL unit type with a currently mixed picture (for example, the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice may have an IRAP NAL unit type, and the second NAL unit for the second slice may have a non-IRAP NAL unit type or a non-reference picture NAL unit type. Here, the IRAP NAL unit type may include an IDR NAL unit type (i.e., IDR_N_LP NAL or IDR_W_RADL NAL unit type) or a CRA NAL unit type, and the non-IRAP NAL unit type or non-reference picture NAL unit type may include a trail NAL unit type or a STSA NAL unit type. Also, depending on the embodiment, the non-IRAP NAL unit type or non-reference picture NAL unit type may sometimes only refer to the trail NAL unit type.

[0233] According to an embodiment, based on the case of allowing a currently mixed picture to have a NAL unit type, for a slice having an IDR NAL unit type (e.g., IDR_W_RADL or IDR_N_LP) within the currently mixed picture, signaling related information of the reference picture list must exist. The signaling related information of the reference picture list may indicate information regarding whether a syntax element for the signaling of the reference picture list exists in the slice header of the slice. That is, based on the value of the signaling related information of the reference picture list being 1, a syntax element for the signaling of the reference picture list may exist in the slice header of the slice having an IDR NAL unit type. Alternatively, based on the value of the signaling related information of the reference picture list being 0, a syntax element for the signaling of the reference picture list may not exist in the slice header of the slice having an IDR NAL unit type.

[0234] For example, the signaling related information of the reference picture list may be the sps_idr_rpl_present_flag syntax element described above. When the value of sps_idr_rpl_present_flag is 1, it may indicate that the syntax element for the signaling of the reference picture list may be present in the slice header of a slice having a NAL unit type such as IDR_N_LP or IDR_W_RADL. Alternatively, when the value of sps_idr_rpl_present_flag is 0, it may indicate that the syntax element for the signaling of the reference picture list is not present in the slice header of a slice having a NAL unit type such as IDR_N_LP or IDR_W_RADL.

[0235] The encoding device can encode video / video information including NAL unit type related information (S1720).

[0236] For example, when the first NAL unit for the first slice in the current picture and the second NAL unit for the second slice in the current picture have different NAL unit types, the encoding device can encode video / video information including NAL unit type related information (for example, mixed_nalu_types_in_pic_flag) having a value of 1. Alternatively, when the first NAL unit for the first slice in the current picture and the second NAL unit for the second slice in the current picture have the same NAL unit type, the encoding device can encode video / video information including NAL unit type related information (for example, mixed_nalu_types_in_pic_flag) having a value of 0.

[0237] Also, for example, the encoding device can encode video / video information including nal_unit_type information indicating each NAL unit type for the slices in the current picture.

[0238] Furthermore, for example, an encoding device can encode video / video information including signaling related information of a reference picture list (e.g., sps_idr_rpl_present_flag).

[0239] Also, for example, an encoding device can encode video / video information including NAL units for slices within the current picture.

[0240] Video / video information including various information as described above can be encoded and output in the form of a bitstream. The bitstream can be transmitted to a decoding device via a network or a (digital) storage medium. Here, the network can include a broadcast network and / or a communication network, etc., and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

[0241] FIG. 19 and FIG. 20 schematically show an example of a video / video decoding method and related components according to an embodiment of this document.

[0242] The method disclosed in FIG. 19 can be executed by the decoding device 300 disclosed in FIG. 3 or FIG. 20. Here, the decoding device 300 disclosed in FIG. 20 is a simplified representation of the decoding device 300 disclosed in FIG. 3. Specifically, steps S1900 to S1920 in FIG. 19 can be executed by the entropy decoding unit 310 disclosed in FIG. 3, and according to the embodiment, each step can also be executed by the residual processing unit 320, the prediction unit 330, the addition unit 340, etc. disclosed in FIG. 3. Furthermore, the method disclosed in FIG. 19 can be executed in this document including the aforementioned embodiments. Therefore, in FIG. 19, regarding the content overlapping with the aforementioned embodiments, specific explanations will be omitted or simplified.

[0243] Referring to FIG. 19, the decoding device can obtain video / video information including NAL unit type related information from the bitstream (S1900).

[0244] As an embodiment, the decoding device can parse a bitstream to derive information (e.g., video / video information) necessary for video restoration (or picture restoration). At this time, the video information may include the NAL unit type-related information (e.g., mixed_nalu_types_in_pic_flag) described above, nal_unit_type information indicating each NAL unit type for a slice in the current picture, signaling-related information of the reference picture list (e.g., sps_idr_rpl_present_flag), NAL units for a slice in the current picture, and the like. That is, the video information may include various information necessary in the decoding process and can be decoded based on a coding method such as exponential Golomb coding, CAVLC, or CABAC.

[0245] As described above, the NAL unit type-related information may include information / syntax elements related to the NAL unit type described in the above-described embodiments and / or Table 1 and Table 2. For example, the NAL unit type-related information may be information regarding whether the current picture has a mixed NAL unit type and may be indicated by the mixed_nalu_types_in_pic_flag syntax element included in the PPS. For example, when the value of the mixed_nalu_types_in_pic_flag syntax element is 0, it may indicate that the NAL units in the current picture have the same NAL unit type. Alternatively, when the value of the mixed_nalu_types_in_pic_flag syntax element is 1, it may indicate that the NAL units in the current picture have other NAL unit types.

[0246] The decoding device can determine the NAL unit type for the NAL units in the current picture based on the NAL unit type-related information (S1910).

[0247] The current picture may include a plurality of slices, and one slice may include a slice header and slice data. Also, a NAL unit can be generated by adding a NAL unit header to the slice (slice header and slice data). The NAL unit header may include NAL unit type information specified by the slice data included in the NAL unit.

[0248] For example, as shown in Table 1 or Table 2 above, the NAL unit type may include TRAIL_NUT, STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, CRA_NUT, etc. based on the type of slice data included in the NAL unit. Also, the NAL unit type may be signaled based on the nal_unit_type syntax element in the NAL unit header. The nal_unit_type syntax element is syntax information for specifying the NAL unit type, and as shown in Table 1 or Table 2 above, it may be indicated by a specific value corresponding to a specific NAL unit type.

[0249] As one embodiment, based on NAL unit type related information regarding that the current picture has a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 1), the decoding device can determine that the first NAL unit for the first slice of the current picture and the second NAL unit for the second slice of the current picture have different NAL unit types. Alternatively, based on NAL unit type related information regarding that the current picture does not have a mixed NAL unit type (for example, the value of mixed_nalu_types_in_pic_flag is 0), the decoding device can determine that the first NAL unit for the first slice of the current picture and the second NAL unit for the second slice of the current picture have the same NAL unit type.

[0250] As an example, based on NAL unit type related information regarding that the current picture has a mixed NAL unit type (e.g., the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice may have a leading picture NAL unit type, and the second NAL unit for the second slice may have a non-IRAP NAL unit type or a non-leading picture NAL unit type. Here, the leading picture NAL unit type may include a RADL NAL unit type or a RASL NAL unit type, and the non-IRAP NAL unit type or the non-leading picture NAL unit type may include a trail NAL unit type or a STSA NAL unit type.

[0251] Alternatively, as an example, based on NAL unit type related information regarding that the current picture has a mixed NAL unit type (e.g., the value of mixed_nalu_types_in_pic_flag is 1), the first NAL unit for the first slice may have an IRAP NAL unit type, and the second NAL unit for the second slice may have a non-IRAP NAL unit type or a non-leading picture NAL unit type. Here, the IRAP NAL unit type may include an IDR NAL unit type (i.e., an IDR_N_LP NAL or an IDR_W_RADL NAL unit type) or a CRA NAL unit type, and the non-IRAP NAL unit type or the non-leading picture NAL unit type may include a trail NAL unit type or a STSA NAL unit type. Also, according to an embodiment, the non-IRAP NAL unit type or the non-leading picture NAL unit type may sometimes be referred to only as a trail NAL unit type.

[0252] According to an embodiment, based on the case where it is allowed to have a NAL unit type in which the current picture is mixed, for a slice having an IDR NAL unit type (e.g., IDR_W_RADL or IDR_N_LP) in the current picture, there must be signaling related information of a reference picture list. The signaling related information of the reference picture list may indicate information regarding whether a syntax element for the signaling of the reference picture list exists in the slice header of the slice. That is, based on the value of the signaling related information of the reference picture list being 1, a syntax element for the signaling of the reference picture list may exist in the slice header of a slice having an IDR NAL unit type. Alternatively, based on the value of the signaling related information of the reference picture list being 0, a syntax element for the signaling of the reference picture list may not exist in the slice header of a slice having an IDR NAL unit type.

[0253] For example, the signaling related information of the reference picture list may be the aforementioned sps_idr_rpl_present_flag syntax element. When the value of sps_idr_rpl_present_flag is 1, it may indicate that a syntax element for the signaling of the reference picture list may exist in the slice header of a slice having a NAL unit type such as IDR_N_LP or IDR_W_RADL. Alternatively, when the value of sps_idr_rpl_present_flag is 0, it may indicate that a syntax element for the signaling of the reference picture list does not exist in the slice header of a slice having a NAL unit type such as IDR_N_LP or IDR_W_RADL.

[0254] The decoding device can decode / restore the current picture based on the NAL unit type (S1920).

[0255] For example, for a first slice in a current picture determined to be of a first NAL unit type and a second slice in the current picture determined to be of a second NAL unit type, the decoding device can decode / restore the first slice based on the first NAL unit type and decode / restore the second slice based on the second NAL unit type. Also, the decoding device can decode / restore samples / blocks in the first slice based on the first NAL unit type and decode / restore samples / blocks in the second slice based on the second NAL unit type.

[0256] In the above-described embodiments, the method is described based on a flowchart as a series of steps or blocks, but the embodiments of this document are not limited to the order of the steps, and a certain step may occur in a different order from the steps described above or simultaneously. Also, those skilled in the art can understand that the steps shown in the flowchart are not exclusive, and other steps may be included, or one or more steps of the flowchart may be deleted without affecting the scope of this document.

[0257] The method according to the above-described document can be implemented in the form of software, and the encoding device and / or decoding device according to this document can be included in a device that performs video processing, such as a TV, a computer, a smartphone, a set-top box, a display device, etc.

[0258] In this document, when an embodiment is implemented in software, the aforementioned method can be implemented by modules (processes, functions, etc.) that perform the aforementioned functions. The modules can be stored in a memory and executed by a processor. The memory may be internal or external to the processor and may be connected to the processor by various well-known means. The processor may include an ASIC (application-specific integrated circuit), other chip sets, logic circuits, and / or data processing devices. The memory may include a ROM (read-only memory), a RAM (random access memory), a flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described in this document can be implemented and executed on a processor, a microprocessor, a controller, or a chip. For example, the functional units shown in each drawing can be implemented and executed on a computer, a processor, a microprocessor, a controller, or a chip. In this case, information for implementation (e.g., information on instructions) or algorithms can be stored in a digital storage medium.

[0259] In addition, the decoding device and encoding device to which this document is applicable may be included in a multimedia broadcast transceiver, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conferencing device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camcorder, an on-demand video (VoD) service providing device, an OTT video (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, a VR (virtual reality) device, an AR (argumente reality) device, a video phone video device, a transportation means terminal (e.g., a vehicle terminal including an autonomous driving vehicle, an airplane terminal, a ship terminal, etc.), and a medical video device, etc., and may be used to process a video signal or a data signal. For example, the OTT video (Over the top video) device may include a game console, a Blu-ray player, an Internet access TV, a home theater system, a smartphone, a tablet PC, a DVR (Digital Video Recoder), etc.

[0260] In addition, the processing method to which the embodiments of this document are applied can be produced in the form of a program executed by a computer and can be stored in a recording medium readable by the computer. Multimedia data having a data structure according to the embodiments of this document can also be stored in a recording medium readable by the computer. The recording medium readable by the computer includes all types of storage devices and distributed storage devices in which data readable by the computer is stored. The recording medium readable by the computer can include, for example, Blu-ray Disc (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Also, the recording medium readable by the computer includes a medium embodied in the form of a carrier wave (for example, transmission via the Internet). Further, a bitstream generated by an encoding method can be stored in a recording medium readable by the computer or can be transmitted via a wired or wireless communication network.

[0261] In addition, the embodiments of this document can be embodied in a computer program product by program code, and the program code can be executed by a computer according to the embodiments of this document. The program code can be stored on a carrier readable by a computer.

[0262] FIG. 21 shows an example of a content streaming system to which the embodiments disclosed in this document can be applied.

[0263] Referring to FIG. 21, the content streaming system to which the embodiments of this document are applied can include a large encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0264] The encoding server compresses the content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream, and plays the role of transmitting this to the streaming server. As another example, when a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server can be omitted.

[0265] The bitstream can be generated by an encoding method or a method for generating a bitstream to which the embodiments of this document are applied. The streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0266] The streaming server transmits multimedia data to a user device based on a user request via a web server. The web server plays the role of a medium for informing the user of what services are available. If the user requests a desired service from the web server, the web server transmits this to the streaming server, and the streaming server transmits multimedia data to the user. At this time, the content streaming system can include another control server. In this case, the control server plays the role of controlling commands / responses between each device in the content streaming system.

[0267] The streaming server can receive content from a media repository and / or an encoding server. For example, when it comes to receiving content from the encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0268] In the example of the user device, there may be a mobile phone, a smart phone, a laptop computer, a digital broadcast terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (for example, a smartwatch, smart glass, an HMD (head mounted display)), a digital TV, a desktop computer, a digital signage, etc.

[0269] Each server in the content streaming system can be operated as a distributed server. In this case, the data received by each server can be distributedly processed.

[0270] The claims described in this specification can be combined in various ways. For example, the technical features of the method claims in this specification can be combined and implemented as a device, and the technical features of the device claims in this specification can be combined and implemented as a method. Also, the technical features of the method claims in this specification and the technical features of the device claims can be combined and implemented as a device, and the technical features of the method claims in this specification and the technical features of the device claims can be combined and implemented as a method.

Claims

1. In a video decoding method executed by a decoding apparatus, a step of obtaining video information including network abstraction layer (NAL) unit type related information from a bitstream; a step of determining an NAL unit type for an NAL unit in a current picture based on the NAL unit type related information; a step of decoding the current picture based on the NAL unit type, and including: the NAL unit type related information is information regarding whether the current picture has a mixed NAL unit type; the video information further includes NAL unit type restriction information to limit that the NAL unit type related information has the mixed NAL unit type; the NAL unit type related information is determined based on the NAL unit type restriction information; based on the NAL unit type related information regarding that the current picture has a mixed NAL unit type, a first NAL unit type for a first slice of the current picture is different from a second NAL unit type for a second slice of the current picture; based on that the first NAL unit type is an instantaneous decoding refresh (IDR) NAL unit type, a value of information related to signaling of a reference picture list is equal to 1; A video decoding method, wherein based on that the value of the information related to signaling of the reference picture list is equal to 1, a syntax element for signaling of the reference picture list exists in a slice header of the first slice having the IDR NAL unit type.

2. In a video encoding method executed by an encoding apparatus, a step of determining an NAL unit type for an NAL unit in a current picture; a step of generating NAL unit type related information based on the NAL unit type; a step of encoding video information including the NAL unit type related information, and including: the NAL unit type related information is information regarding whether the current picture has a mixed NAL unit type; The video information further includes NAL unit type restriction information for restricting that the NAL unit type related information has the mixed NAL unit type. The NAL unit type related information is determined based on the NAL unit type restriction information. Based on the NAL unit type related information regarding that the current picture has the mixed NAL unit type, a first NAL unit type for a first slice of the current picture is different from a second NAL unit type for a second slice of the current picture having a different NAL unit type. Based on the first NAL unit type being an IDR (instantaneous decoding refresh) NAL unit type, a syntax element for signaling of a reference picture list exists in a slice header of the first slice having the IDR NAL unit type, a video encoding method.

3. A method for transmitting data for a video, a step of obtaining a bitstream of the video, where the bitstream is generated based on determining an NAL (network abstraction layer) unit type for an NAL unit in a current picture, generating NAL unit type related information based on the NAL unit type, and encoding video information including the NAL unit type related information, a step. a step of transmitting the data including the bitstream, and the NAL unit type related information is information regarding whether the current picture has the mixed NAL unit type. The video information further includes NAL unit type restriction information for restricting that the NAL unit type related information has the mixed NAL unit type. The NAL unit type related information is generated based on the NAL unit type restriction information. Based on the NAL unit type related information regarding that the current picture has the mixed NAL unit type, a first NAL unit type for a first slice of the current picture is different from a second NAL unit type for a second slice of the current picture having a different NAL unit type. Based on the fact that the first NAL unit type is an IDR (instantaneous decoding refresh) NAL unit type, a syntax element for signaling a reference picture list exists in a slice header of the first slice having the IDR NAL unit type, a method for transmitting data.

Images