Media file processing method and device

Abstract

According to an embodiment of the present document, there is provided a media file processing method executed by a media file processing device, the method comprising the steps of recovering a sequence of access units for a first OLS, determining whether a first condition and a second condition are true, and inserting an EOS NAL unit as many times as the number of layers applied to the first OLS based on the first condition and the second condition being true, wherein the first condition is whether an OLS different from the first OLS is selected next to the sequence of access units, and the second condition is whether the sequence of access units does not end with an EOS NAL unit or does not end with an EOB NAL unit in each layer of the first OLS.

Description

[Technical field]

[0001] This document relates to video / image coding technology, and more particularly to a method and apparatus for processing a media file for image information coded in a video / image coding system. [Background technology]

[0002] Recently, the demand for high-resolution, high-quality video / images such as 4K or 8K or higher UHD (Ultra High Definition) video / images is increasing in various fields. As the video / image data has higher resolution and quality, the amount of information or bits transmitted relatively increases compared to existing video / image data, so that when video / image data is transmitted using existing media such as wired or wireless broadband lines or when video / image data is stored using existing storage media, the transmission and storage costs increase.

[0003] In addition, interest and demand for immersive media such as virtual reality (VR), artificial reality (AR) content and holograms has been increasing recently, and the broadcast of videos / images with different image characteristics from real images, such as game images, is increasing.

[0004] Therefore, there is a demand for a highly efficient image compression technology in order to effectively compress and transmit, store, and play back high-resolution, high-quality video / image information having the above-mentioned various characteristics. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem of this document is to provide a method and apparatus for improving video / image coding efficiency.

[0006] Another technical problem of this document is to provide a method and apparatus for generating a media file for coded video information.

[0007] Another technical problem of this document is to provide a method and apparatus for processing a media file for coded video information. [Means for solving the problem]

[0008] According to an embodiment of the present document, there is provided a media file generating method executed by a media file generating device, the method including the steps of generating a sequence of access units for a first Output Layer Set (OLS), determining whether a first condition and a second condition are true, inserting an End Of Sequence (EOS) Network Abstraction Layer (NAL) unit as many times as the number of layers applied to the first OLS based on whether the first condition and the second condition are true, and generating the media file including the first OLS, wherein the first condition is whether an OLS different from the first OLS is selected next to the sequence of access units, and the second condition is whether the sequence of access units does not end with an EOS NAL unit or an End Of Bitstream (EOB) NAL unit in each layer of the first OLS.

[0009] According to another embodiment of the present document, there is provided a media file generating device, the media file generating device including a media file generating unit that generates a sequence of access units for a first Output Layer Set (OLS), determines whether a first condition and a second condition are true, and inserts an End Of Sequence (EOS) Network Abstraction Layer (NAL) unit as many times as the number of layers applied to the first OLS based on whether the first condition and the second condition are true, and generates the media file including the first OLS, the first condition being whether an OLS different from the first OLS is selected next to the sequence of access units, and the second condition being whether the sequence of access units does not end with an EOS NAL unit or an End Of Bitstream (EOB) NAL unit in each layer of the first OLS.

[0010] According to another embodiment of the present document, there is provided a media file processing method executed by a media file processing device, the method including the steps of recovering a sequence of access units for a first Output Layer Set (OLS), determining whether a first condition and a second condition are true, and inserting an End Of Sequence (EOS) Network Abstraction Layer (NAL) unit as many times as the number of layers applied to the first OLS based on whether the first condition and the second condition are true, wherein the first condition is whether an OLS different from the first OLS is selected next to the sequence of access units, and the second condition is whether the sequence of access units does not end with an EOS NAL unit or an End Of Bitstream (EOB) NAL unit in each layer of the first OLS.

[0011] According to another embodiment of the present document, there is provided a media file processing device, comprising: a receiving unit for acquiring a media file; and a media file processing unit for recovering a sequence of access units for a first Output Layer Set (OLS) of the media file, determining whether a first condition and a second condition are true, and inserting an End Of Sequence (EOS) Network Abstraction Layer (NAL) unit as many times as the number of layers applied to the first OLS if the first condition and the second condition are true, wherein the first condition is whether an OLS different from the first OLS is selected next to the sequence of access units, and the second condition is whether the sequence of access units does not end with an EOS NAL unit or an End Of Bitstream (EOB) NAL unit in each layer of the first OLS.

[0012] According to another embodiment of the present document, there is provided a computer readable digital storage medium having stored thereon a media file generated by a media file generating method, the computer readable digital storage medium comprising the steps of: constructing sub-picture tracks each including sub-picture image data; constructing a base track referencing the sub-picture tracks; and generating the media file including the sub-picture tracks and the base track, wherein samples of the base track form rectangular regions with holes and no overlap, the holes indicating areas not covered by at least one of the samples of the referenced sub-picture tracks, and the overlaps indicating areas covered two or more times by samples of the referenced sub-picture tracks. Effect of the Invention

[0013] According to this document, even if only the maximum temporal ID is changed, unnecessary operations of inserting EOS NAL units can be prevented, thereby improving overall coding efficiency.

[0014] According to this document, it is possible to prevent the burden caused by inserting an EOS NAL unit at an appropriate position within an access unit, thereby improving overall coding efficiency. [Brief description of the drawings]

[0015] [Figure 1] 1 illustrates generally an example of a video / image coding system in which embodiments of the present document may be applied; [Diagram 2] FIG. 1 is a diagram illustrating the configuration of a video / image encoding device to which the embodiments of this document can be applied. [Diagram 3] FIG. 1 is a diagram illustrating the configuration of a video / image decoding device to which an embodiment of the present document can be applied. [Figure 4] 1 shows an exemplary hierarchical structure for coded video / images. [Diagram 5] 1 illustrates a schematic example of a media file structure. [Figure 6] 1 illustrates a schematic example of a media file structure. [Figure 7] 1 illustrates an example of the general operation of a DASH-based adaptive streaming model. [Figure 8] 1 illustrates an exemplary method for generating a media file to which the embodiments proposed in this document are applied. [Figure 9] 10 illustrates an exemplary method for decoding a media file generated by applying the embodiments proposed in this document. [Figure 10] 1 illustrates a schematic diagram of a media file generation method by a media file generation device according to the present document. [Figure 11] 1 illustrates a schematic diagram of a media file generation device for performing a media file generation method according to the present document; [Figure 12]1 illustrates a schematic diagram of a media file processing method according to the present document. [Figure 13] 1 illustrates a schematic representation of a media file processing device for performing a media file processing method according to the present document; [Figure 14] 1 illustrates an exemplary structural diagram of a content streaming system to which the embodiments of this document are applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] This document may be modified in various ways and may have various embodiments, and a specific embodiment will be illustrated in the drawings and described in detail. However, this is not intended to limit the embodiments of this document to the specific embodiment. The terms used in this document are merely used to describe a specific embodiment, and are not intended to limit the technical ideas of this document. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this document, the terms "include" or "have" specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the document, and should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0017] Meanwhile, each configuration in the drawings described in this document is illustrated independently for the convenience of explaining the different characteristic functions, and does not mean that each configuration is realized by separate hardware or software. For example, two or more of each configuration may be combined to form one configuration, and one configuration may be divided into multiple configurations. An embodiment in which each configuration is integrated and / or separated is also included in the scope of this document as long as it does not deviate from the essence of this document.

[0018] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Hereinafter, the same reference numerals will be used for the same components in the drawings, and duplicated descriptions of the same components may be omitted.

[0019] FIG. 1 illustrates a schematic diagram of an example of a video / image coding system in which embodiments of the present document may be applied.

[0020] As shown in Figure 1, the video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device in a file or streaming form via a digital recording medium or a network.

[0021] The source device may comprise a video source, an encoding device, and a sending unit. The receiving device may comprise a receiving unit, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may comprise a display unit, which may be a separate device or an external component.

[0022] A video source may acquire video / images through a video / image capture, synthesis, or generation process, etc. A video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process in which the associated data is generated.

[0023] An encoding device can encode an input video / image. The encoding device can perform a series of steps such as prediction, transformation, quantization, etc. for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0024] The transmitting unit may transmit the encoded video / image information or data output in the form of a bitstream to a receiving unit of a receiving device via a digital recording medium or a network in the form of a file or streaming. The digital recording medium may include various recording media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit may include an element for generating a media file through a predetermined file format and may include an element for transmission via a broadcasting / communication network. The receiving unit may receive / extract the bitstream and transmit it to a decoding device.

[0025] The decoding device can decode the video / image by performing a series of steps such as inverse quantization, inverse transformation, prediction, etc., which correspond to the operations of the encoding device.

[0026] The renderer can render the decoded video / images, and the rendered video / images can be displayed via a display unit.

[0027] This document relates to video / image coding. For example, the methods / embodiments disclosed in this document can be applied to methods disclosed in the versatile video coding (VVC) standard, the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd generation of audio video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).

[0028] In this document, various embodiments relating to video / image coding are presented, which, unless otherwise stated, may also be implemented in combination with each other.

[0029] In this document, a video may refer to a collection of a series of images over time. A picture generally refers to a unit showing one image at a specific time, and a subpicture / slice / tile is a unit that constitutes a part of a picture in coding. A subpicture / slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more subpictures / slices / tiles. A picture may be composed of one or more tile groups. A tile group may include one or more tiles. A brick may refer to a rectangular area of ​​a row of CTUs in a tile in a picture. A tile may be partitioned into multiple bricks, and each brick may be composed of one or more rows of CTUs in the tile. A tile that is not partitioned into multiple bricks may also be called a brick. A brick scan may indicate a particular sequential ordering of CTUs partitioning a picture, where the CTUs may be aligned in a brick-CTU raster scan, the bricks in a tile may be aligned contiguously in a raster scan of the bricks of the tile, and the tiles in a picture may be aligned contiguously in a raster scan of the tiles of the picture. A sub-picture may also indicate a rectangular region of one or more slices in a picture. That is, a sub-picture may include one or more slices that collectively cover a rectangular region of a picture. A tile is a particular tile column and a rectangular region of a CTU within the particular tile column. The tile column is a rectangular region of a CTU, where the rectangular region has a height equal to the height of the picture, and a width may be specified by a syntax element in a picture parameter set. The tile row is a rectangular region of a CTU, where the rectangular region has a width equal to the height of the picture, and a height is equal to the height of the picture.A tile scan may indicate a particular sequential ordering of CTUs that partition a picture, and the CTUs may be consecutively aligned in a raster scan of CTUs within a tile, and tiles within a picture may be consecutively aligned in a raster scan of the tiles of the picture. A slice may contain an integer number of bricks of a picture, and the integer number of bricks may be contained in one NAL unit. A slice may consist of a number of complete tiles or may be a continuous sequence of complete bricks of one tile. In this document, tile group and slice may be used interchangeably. For example, in this document, tile group / tile group header may be referred to as slice / slice header.

[0030] A pixel or a pel may refer to the smallest unit that constitutes one picture (or image). A term corresponding to a pixel may be "sample." A sample may generally indicate a pixel or a pixel value, may indicate only a pixel / pixel value of a luma component, or may indicate only a pixel / pixel value of a chroma component.

[0031] A unit may indicate a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. A unit may include one luma block and two chroma (e.g., cb, cr) blocks. A unit may be used in combination with terms such as block or area, depending on the case. In a general case, an M×N block may include a sample (or sample array) consisting of M columns and N rows, or a set (or array) of transform coefficients.

[0032] In this document, "A or B" can mean "A only," "B only," or "both A and B." In other words, in this document, "A or B" can be interpreted as "A and / or B." For example, in this document, "A, B or C" can mean "A only," "B only," "C only," or "any combination of A, B and C."

[0033] The slash ( / ) and comma used in this document can mean "and / or." For example, "A / B" can mean "A and / or B." This allows "A / B" to mean "A only," "B only," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

[0034] In this document, "at least one of A and B" can mean "A only," "B only," or "both A and B." Also, in this document, the expressions "at least one of A or B" and "at least one of A and / or B" can be interpreted as "at least one of A and B."

[0035] In this document, "at least one of A, B and C" can mean "A only," "B only," "C only," or "any combination of A, B and C." Additionally, "at least one of A, B or C" and "at least one of A, B and / or C" can mean "at least one of A, B and C."

[0036] Furthermore, parentheses used in this document may mean "for example." Specifically, when "prediction (intra prediction)" is displayed, "intra prediction" is proposed as an example of "prediction." In other words, "prediction" in this document is not limited to "intra prediction," and "intra prediction" is proposed as an example of "prediction." Furthermore, when "prediction (i.e., intra prediction)" is displayed, "intra prediction" is proposed as an example of "prediction."

[0037] In this document, technical features described individually in one drawing may be embodied individually or simultaneously.

[0038] The following drawings are created to explain a specific example of the present document. The names of specific devices and names of specific signals / messages / fields described in the drawings are presented for illustrative purposes only, and the technical features of the present document are not limited to the specific names used in the following drawings.

[0039] 2 is a diagram for explaining the configuration of a video / image encoding device to which the embodiments of this document can be applied. Hereinafter, the video encoding device may include an image encoding device.

[0040] As shown in FIG. 2, the encoding device 200 may be configured to include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter predictor 221 and an intra predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. The image division unit 210, the prediction unit 220, the residual processing unit 230, the entropy encoding unit 240, the addition unit 250, and the filtering unit 260 may be configured by one or more hardware components (e.g., an encoder chip set or a processor) according to an embodiment. In addition, the memory 270 may include a decoded picture buffer (DPB) and may be configured by a digital recording medium. The hardware components may further include the memory 270 as an internal / external component.

[0041] The image division unit 210 may divide an input image (or picture, frame) input to the encoding device 200 into one or more processing units. As an example, the processing units may be called coding units (CUs). In this case, the coding units may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be divided into a plurality of coding units of a deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quad-tree structure may be applied first, and a binary tree structure and / or a ternary structure may be applied later. Alternatively, a binary tree structure may be applied first. The coding procedure according to this document may be performed based on a final coding unit that is not further divided. In this case, the largest coding unit may be used as the final coding unit immediately based on coding efficiency according to image characteristics, or the coding unit may be recursively divided into coding units of lower depths as necessary, and a coding unit of an optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit described above.The prediction unit is a unit of sample prediction, and the transform unit is a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0042] The unit may be used interchangeably with terms such as block or area. In the general case, an M×N block may refer to a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally refer to a pixel or pixel value, may refer to only a pixel / pixel value of a luma component, or may refer to only a pixel / pixel value of a chroma component. A sample may be used as a term corresponding to a pixel or pel of one picture (or image).

[0043] The encoding device 200 may generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from an input image signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as illustrated, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input image signal (original block, original sample array) in the encoder 200 may be called a subtraction unit 231. The prediction unit may perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including a prediction sample for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied in units of the current block or CU. The prediction unit may generate various information related to prediction, such as prediction mode information, and transmit the information to the entropy encoding unit 240, as will be described later in the description of each prediction mode. The prediction information can be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0044] The intra prediction unit 222 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from the current block depending on the prediction mode. In the intra prediction, the prediction mode may include a plurality of non-directional modes and a plurality of directional modes. The non-directional mode may include, for example, a DC mode and a planar mode. The directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the fineness of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the setting. The intra prediction unit 222 may also determine the prediction mode to be applied to the current block using the prediction mode applied to the neighboring block.

[0045] The inter prediction unit 221 may derive a predicted block for a current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring block may include a spatial neighboring block present in the current picture and a temporal neighboring block present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be called a collocated picture (colPic). For example, the inter prediction unit 221 may generate information indicating which candidate is used to derive a motion vector and / or a reference picture index of the current block by forming a motion information candidate list based on neighboring blocks. Inter prediction may be performed based on various prediction modes, and for example, in the case of a skip mode and a merge mode, the inter prediction unit 221 may use motion information of a neighboring block as motion information of the current block. In the case of the skip mode, unlike the merge mode, a residual signal may not be transmitted.In the case of a motion vector prediction (MVP) mode, the motion vector of a neighboring block can be used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0046] The prediction unit 220 may generate a prediction signal based on various prediction methods described later. For example, the prediction unit may apply intra prediction or inter prediction for prediction of one block, or may simultaneously apply intra prediction and inter prediction. This may be called combined inter and intra prediction (CIIP). The prediction unit may also be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image / video coding such as games, for example, screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in deriving a reference block within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described in this document. The palette mode may be seen as an example of intra coding or intra prediction. When the palette mode is applied, a sample value within a picture may be signaled based on information regarding a palette table and a palette index.

[0047] The prediction signal generated through the prediction unit (including the inter prediction unit 221 and / or the intra prediction unit 222) may be used to generate a restored signal or may be used to generate a residual signal. The transform unit 232 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, the GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. The CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process may be applied to pixel blocks having the same square size, or may be applied to non-square, variable-sized blocks.

[0048] The quantizer 233 quantizes the transform coefficients and transmits the quantized transform coefficients to the entropy encoder 240, which may encode the quantized signal (information on the quantized transform coefficients) and output the quantized signal as a bitstream. The information on the quantized transform coefficients may be referred to as residual information. The quantizer 233 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and may generate information on the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder 240 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 240 may encode information required for video / image restoration (e.g., values ​​of syntax elements, etc.) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of network abstraction layer (NAL) units. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / image information may further include general constraint information. Information and / or syntax elements transmitted / signaled from an encoding device to a decoding device in this document may be included in the video / image information. The video / image information may be encoded through the above-mentioned encoding procedure and included in the bitstream.The bitstream may be transmitted via a network or may be stored in a digital recording medium. Here, the network may include a broadcasting network and / or a communication network, and the digital recording medium may include various recording media such as a USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitting unit (not shown) for transmitting the signal output from the entropy encoding unit 240 and / or a storing unit (not shown) for storing the signal may be configured as an internal / external element of the encoding device 200, or the transmitting unit may be included in the entropy encoding unit 240.

[0049] The quantized transform coefficients output from the quantizer 233 may be used to generate a prediction signal. For example, a residual signal (residual block or residual sample) may be restored by applying inverse quantization and inverse transform to the quantized transform coefficients via the inverse quantizer 234 and the inverse transformer 235. The adder 250 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit 221 or the intra prediction unit 222. When there is no residual for the current block, such as when a skip mode is applied, a predicted block may be used as a reconstructed block. The adder 250 may be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next current block in the current picture, and may also be used for inter prediction of the next picture after filtering, as described below.

[0050] Meanwhile, luma mapping with chroma scaling (LMCS) may be applied during picture encoding and / or reconstruction.

[0051] The filtering unit 260 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 260 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc. The filtering unit 260 may generate various information related to filtering and transmit it to the entropy encoding unit 240, as will be described later in relation to each filtering method. The information related to filtering may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0052] The modified reconstructed picture transmitted to the memory 270 may be used as a reference picture in the inter prediction unit 221. When inter prediction is applied through this, the encoding apparatus may avoid prediction mismatch between the encoding apparatus 200 and the decoding apparatus 300, and may also improve encoding efficiency.

[0053] The memory 270DPB may store the modified reconstructed picture to be used as a reference picture in the inter prediction unit 221. The memory 270 may store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter prediction unit 221 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 270 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 222.

[0054] FIG. 3 is a diagram illustrating a schematic configuration of a video / image decoding device to which the embodiments of this document can be applied.

[0055] Referring to FIG. 3, the decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an intra predictor 331 and an inter predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. The entropy decoding unit 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 may be configured as one hardware component (e.g., a decoder chipset or processor) according to an embodiment. Also, the memory 360 may include a decoded picture buffer (DPB) and may be configured as a digital storage medium. The hardware components may further include a memory 360 as an internal / external component.

[0056] When a bitstream including video / image information is input, the decoding apparatus 300 can restore an image corresponding to a process in which the video / image information is processed by the encoding apparatus of FIG. 2. For example, the decoding apparatus 300 can derive a unit / block based on block division related information obtained from the bitstream. The decoding apparatus 300 can perform decoding using a processing unit applied in the encoding apparatus. Thus, the processing unit of the decoding is, for example, a coding unit, and the coding unit can be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units can be derived from the coding unit. Then, the restored image signal decoded and output by the decoding apparatus 300 can be reproduced by a reproduction device.

[0057] The decoding device 300 may receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal may be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 may derive information (e.g., video / image information) required for image restoration (or picture restoration) by parsing the bitstream. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The decoding device may further decode pictures based on information on the parameter set and / or the general constraint information. Signaling / received information and / or syntax elements described later in this document may be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 may decode information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration, quantized values ​​of transform coefficients related to residuals, etc. More specifically, the CABAC entropy decoding method may receive bins corresponding to each syntax element in the bitstream, determine a context model using information on the syntax element to be decoded and decode information on the neighboring and blocks to be decoded, or information on symbols / bins decoded in a previous step, predict the occurrence probability of bins according to the determined context model, and perform arithmetic decoding of the bins to generate symbols corresponding to the values ​​of each syntax element. In this case, the CABAC entropy decoding method may update the context model using information on the decoded symbol / bin for the context model of the next symbol / bin after determining the context model.Among the information decoded by the entropy decoding unit 310, information related to prediction is provided to a prediction unit (inter prediction unit 332 and intra prediction unit 331), and residual values ​​entropy decoded by the entropy decoding unit 310, i.e., quantized transform coefficients and related parameter information, may be input to the residual processing unit 320. The residual processing unit 320 may derive a residual signal (residual block, residual sample, residual sample array). Also, among the information decoded by the entropy decoding unit 310, information related to filtering may be provided to the filtering unit 350. Meanwhile, a receiving unit (not shown) for receiving a signal output from the encoding device may be further configured as an internal / external element of the decoding device 300, or the receiving unit may be a component of the entropy decoding unit 310. Meanwhile, the decoding device according to this document may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit 310, and the sample decoder may include at least one of the inverse quantization unit 321, the inverse transform unit 322, the addition unit 340, the filtering unit 350, the memory 360, the inter prediction unit 332, and the intra prediction unit 331.

[0058] The inverse quantization unit 321 may inverse quantize the quantized transform coefficients to output transform coefficients. The inverse quantization unit 321 may rearrange the quantized transform coefficients in a two-dimensional block form. In this case, the rearrangement may be performed based on a coefficient scan order performed in the encoding device. The inverse quantization unit 321 may perform inverse quantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0059] The inverse transform unit 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0060] The prediction unit may perform prediction on a current block and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on information on the prediction output from the entropy decoding unit 310, and may determine a specific intra / inter prediction mode.

[0061] The prediction unit 320 may generate a prediction signal based on various prediction methods described later. For example, the prediction unit may apply intra prediction or inter prediction for prediction of one block, and may simultaneously apply intra prediction and inter prediction. This may be called combined inter and intra prediction (CIIP). In addition, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image / video coding such as games, for example, screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in deriving a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode may be seen as an example of intra coding or intra prediction. When the palette mode is applied, information regarding a palette table and a palette index may be included in the video / image information and signaled.

[0062] The intra prediction unit 331 may predict a current block by referring to samples in a current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from the current block depending on a prediction mode. In intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 331 may also determine a prediction mode to be applied to the current block using a prediction mode applied to a neighboring block.

[0063] The inter prediction unit 332 may derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted from an inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter prediction unit 332 may configure a motion information candidate list based on the neighboring blocks and derive a motion vector and / or a reference picture index for the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating a mode of inter prediction for the current block.

[0064] The adder 340 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to a prediction signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 332 and / or the intra prediction unit 331). When there is no residual for a current block, such as when a skip mode is applied, the predicted block may be used as a reconstructed block.

[0065] The adder 340 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of a next block to be processed in a current picture, and may be output after filtering as described below, or may be used for inter prediction of a next picture.

[0066] Meanwhile, LMCS (luma mapping with chroma scaling) can be applied during the picture decoding process.

[0067] The filtering unit 350 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 350 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may transmit the modified reconstructed picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0068] The (modified) reconstructed picture stored in the DPB of the memory 360 may be used as a reference picture in the inter prediction unit 332. The memory 360 may store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter prediction unit 260 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 360 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 331.

[0069] In this document, the embodiments described for the filtering unit 260, the inter prediction unit 221, and the intra prediction unit 222 of the encoding device 200 can also be applied identically or correspondingly to the filtering unit 350, the inter prediction unit 332, and the intra prediction unit 331 of the decoding device 300, respectively.

[0070] FIG. 4 shows an exemplary hierarchical structure for coded video / pictures.

[0071] Referring to Figure 4, the coded video / image can be divided into a VCL (video coding layer) that handles the video / image decoding process and itself, a subsystem that transmits and stores the coded information, and a NAL (network abstraction layer) that exists between the VCL and the subsystem and is responsible for network adaptation functions.

[0072] For example, in the VCL, VCL data including compressed image data (slice data) can be generated, or a parameter set including a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), a Video Parameter Set (VPS), or a Supplemental Enhancement Information (SEI) message additionally required for the video decoding process can be generated.

[0073] For example, in NAL, a NAL unit can be generated by adding header information (NAL unit header) to a raw byte sequence payload (RBSP) generated in a VCL. In this case, the RBSP can refer to slice data, a parameter set, an SEI message, etc. generated in the VCL. The NAL unit header can include NAL unit type information specified by the RBSP data included in the corresponding NAL unit.

[0074] For example, as shown in Figure 4, NAL units can be classified into VCL NAL units and non-VCL NAL units according to the RBSP generated in the VCL. The VCL NAL unit can refer to a NAL unit that includes information (slice data) on an image, and the non-VCL NAL unit can refer to a NAL unit that includes information (parameter set or SEI message) required for image decoding.

[0075] The above-mentioned VCL NAL unit and Non-VCL NAL unit can be transmitted over a network by attaching header information according to a data standard of a subsystem. For example, the NAL unit can be converted into a predetermined standard data format such as H.266 / VVC file format, real-time transport protocol (RTP), Transport Stream (TS), etc., and can be transmitted over various networks.

[0076] Also, as described above, the NAL unit type of a NAL unit can be specified by the RBSP data structure included in the corresponding NAL unit, and information on the NAL unit type can be stored and signaled in the NAL unit header.

[0077] For example, NAL units can be classified into VCL NAL unit types and Non-VCL NAL unit types depending on whether they contain information about a video (slice data), and VCL NAL unit types can be classified according to the characteristics and type of a picture included in the VCL NAL unit, and Non-VCL NAL unit types can be classified according to the type of a parameter set.

[0078] The following are examples of NAL unit types specified by the types of parameter sets contained in the Non-VCL NAL unit types:

[0079] -APS (Adaptation Parameter Set) NAL unit: Type for NAL units containing APS

[0080] -DPS (Decoding Parameter Set) NAL unit: Type for NAL units containing DPS

[0081] -VPS (Video Parameter Set) NAL unit: Type for NAL units that contain VPS

[0082] -SPS (Sequence Parameter Set) NAL unit: Type for NAL units containing SPS

[0083] -PPS (Picture Parameter Set) NAL unit: Type for NAL units containing PPS

[0084] -PH (Picture header) NAL unit: Type for NAL units containing PH

[0085] The above-mentioned NAL unit types may have syntax information for the NAL unit types, and the syntax information may be stored and signaled in a NAL unit header. For example, the syntax information may be nal_unit_type, and the NAL unit type may be specified in the nal_unit_type value.

[0086] Meanwhile, one picture may include a plurality of slices, and a slice may include a slice header and slice data. In this case, one picture header may be added for a plurality of slices (a set of slice headers and slice data). A picture header (picture header syntax) may include information / parameters that may be commonly applied to pictures. A slice header (slice header syntax) may include information / parameters that may be commonly applied to slices. An APS (APS syntax) or a PPS (PPS syntax) may include information / parameters that may be commonly applied to one or more slices or pictures. An SPS (SPS syntax) may include information / parameters that may be commonly applied to one or more sequences. A VPS (VPS syntax) may include information / parameters that may be commonly applied to a plurality of layers. A DPS (DPS syntax) may include information / parameters that may be commonly applied to an entire image. A DPS may include information / parameters related to concatenation of a coded video sequence (CVS).

[0087] In this document, the image / video information encoded from an encoding device to a decoding device and signaled in the form of a bitstream may include not only partitioning-related information within a picture, intra / inter prediction information, inter-layer prediction-related information, residual information, in-loop filtering information, etc., but also information included in the slice header, information included in the picture header, information included in the APS, information included in the PPS, information included in the SPS, information included in the VPS, and / or information included in the DPS. In addition, the image / video information may further include information of a NAL unit header.

[0088] Meanwhile, the encoded image / video information may be configured based on a media file format to generate a media file. For example, the encoded image / video information may form a media file (segment) based on one or more NAL units / sample entries for the encoded image / video information. The media file may include sample entries and tracks. For example, a media file (segment) may include various records, and each record may include information related to an image / video or information related to a media file format. Also, for example, one or more NAL units may be stored in a configuration record (or decoder configuration record, or VVC decoder configuration record) field of a media file. Here, a field may also be called a syntax element.

[0089] For example, the ISO Base Media File Format (ISOBMFF) can be used as a media file format to which the methods / embodiments disclosed in this document can be applied. ISOBMFF can be used as the basis for many codec encapsulation formats, such as the AVC file format, the HEVC file format, and / or the VVC file format, and many multimedia container formats, such as the MPEG-4 file format, the 3GPP file format (3GP), and / or the DVB file format, etc. In addition to continuous media such as audio and video, static media such as images and metadata can also be stored in a file according to ISOBMFF. Files structured by ISOBMFF can be used for a variety of purposes, such as local media file playback, progressive downloading of a remote file, segments for Dynamic Adaptive Streaming over HTTP (DASH), containers and packetization instructions for streaming content, and recording of received real-time media streams.

[0090] A "box", which will be described later, is a basic syntax element of ISOBMFF. An ISOBMFF file may consist of a sequence of boxes, and a box may contain other boxes. For example, a movie box (a box with a grouping type of "moov") may contain metadata for continuous media streams in a media file, and each stream may be represented in a track in the file. Metadata for a track may be contained in a track box (a box with a grouping type of "trak"), and the media content of a track may be contained in a media data box (a box with a grouping type of "mdat") or directly in a separate file. The media content of a track may consist of a sequence of samples, such as audio or video access units. For example, an ISOBMFF can specify types of tracks such as a media track containing elementary media streams, a hint track containing media transmission instructions or indicating received packet streams, and a timed metadata track containing time synchronized metadata.

[0091] Also, although ISOBMFF was designed for storage, it is also very useful for streaming, e.g. progressive download or DASH. For streaming purposes, movie fragments defined in ISOBMFF can be used. A fragmented ISOBMFF file can be represented with two tracks, e.g. associated with video and audio. For example, after receiving a 'moov' box, if random access is included, all movie fragments 'moof' can be decoded together with the associated media data.

[0092] Additionally, the metadata for each track may include a list of sample description entries that provide the coding or encapsulation format used for the track and initialization data required for processing the format, and each sample may be associated with one of the sample description entries for the track.

[0093] ISOBMFF allows for the specification of sample-specific metadata through a variety of mechanisms. Certain boxes within the Sample Table box (boxes with grouping type "stbl") can be standardized to accommodate common requirements. For example, the Sync Sample box (boxes with grouping type "stss") can be used to list the random access samples of a track. The sample grouping mechanism allows for the mapping of samples by a four-character grouping type with sample groups that share the same property as specified in the sample group description entry in the file. Various grouping types are specified in ISOBMFF.

[0094] 5 and 6 are diagrams showing an example of a media file structure.

[0095] A media file may include at least one box. Here, a box is a data block or object that includes media data or metadata related to the media data. The boxes may form a hierarchical structure, which allows data to be classified and the media file to be in a form suitable for storing and / or transmitting large amounts of media data. In addition, the media file may have a structure that allows a user to easily access media information, such as moving to a specific point in the media content.

[0096] A media file may contain an ftyp box, a moov box, and / or an mdat box.

[0097] The ftyp box (file type box) may provide file type or compatibility related information for the corresponding media file. The ftyp box may include configuration version information for the media data of the corresponding media file. A decoding device may classify the corresponding media file by referring to the ftyp box.

[0098] The moov box (movie box) is a box that contains metadata for the media data of the corresponding media file. The moov box can act as a container for all metadata. The moov box is the highest hierarchical box among the metadata-related boxes. Depending on the embodiment, only one moov box can exist in a media file.

[0099] The mdat box (media data box) is a box that contains the actual media data of the corresponding media file. The media data may include audio samples and / or video samples, and the mdat box may act as a container for such media samples.

[0100] Depending on the embodiment, the moov box may further include an mvhd box, a trak box, and / or an mvex box as sub-boxes.

[0101] The mvhd box (Movie Header Box) can contain media presentation related information of the media data contained in the corresponding media file, i.e., the mvhd box can contain information such as the media creation time, change time, time standard, duration, etc. of the corresponding media presentation.

[0102] A trak box (track box) can provide information related to the track of the corresponding media data. A trak box can include information such as stream-related information, presentation-related information, and access-related information for an audio track or a video track. There can be multiple trak boxes depending on the number of tracks.

[0103] Depending on the embodiment, the trak box may further include a tkhd box (track header box) as a subordinate box. The tkhd box may include information on the corresponding track indicated by the trak box. The tkhd box may include information such as the creation time, modification time, and track identifier of the corresponding track.

[0104] An mvex box (movie extend box) can indicate that the media file contains a moof box, described below. To know all media samples of a particular track, the moof box must be scanned.

[0105] Depending on the embodiment, a media file may be divided into a plurality of fragments (500). This allows the media file to be divided and stored or transmitted. The media data (mdat box) of the media file is divided into a plurality of fragments, and each fragment may include a moof box and a divided mdat box. Depending on the embodiment, information of the ftyp box and / or the moov box is required to utilize the fragments.

[0106] The moof box (movie fragment box) can provide metadata for the media data of the corresponding fragment. The moof box is the highest-level box among the metadata-related boxes of the corresponding fragment.

[0107] The mdat box (media data box) can contain actual media data as described above. This mdat box can contain media samples of the media data corresponding to each corresponding fragment.

[0108] Depending on the embodiment, the moof box may further include an mfhd box and / or a traf box as sub-boxes.

[0109] The mfhd box (movie fragment header box) can contain information related to the relationship between multiple divided fragments. The mfhd box can contain a sequence number and indicate which data the media data of the corresponding fragment is divided into. In addition, the mfhd box can be used to check whether any of the divided data is missing.

[0110] The traf box (track fragment box) can contain information about the corresponding track fragment. The traf box can provide metadata about the divided track fragments included in the corresponding fragment. The traf box can provide metadata so that media samples in the corresponding track fragment can be decoded / played. There can be multiple traf boxes depending on the number of track fragments.

[0111] According to an embodiment, the traf box described above may further include a tfhd box and / or a trun box as sub-boxes.

[0112] The tfhd box (track fragment header box) can contain header information for the corresponding track fragment. The tfhd box can provide basic sample size, duration, offset, identifier, and other information for the media sample of the track fragment indicated by the traf box.

[0113] A trun box (track fragment run box) can contain information related to the corresponding track fragment. The trun box can contain information such as the duration, size, and playback time of each media sample.

[0114] The above mentioned media files or fragments of media files can be treated and transmitted as segments, which can be initialization segments and / or media segments.

[0115] The file in the illustrated embodiment (510) is a file including information related to the initialization of the media decoder, excluding media data. This file may, for example, correspond to the initialization segment described above. The initialization segment may include the ftyp box and / or the moov box described above.

[0116] The file in the illustrated embodiment (520) is a file including the fragment described above. This file may, for example, correspond to the media segment described above. The media segment may include the moof box and / or the mdat box described above. The media segment may further include a styp box and / or a sidx box.

[0117] The styp box (segment type box) may provide information for identifying media data of a split fragment. The styp box may perform the same role as the ftyp box described above for a split fragment. According to an embodiment, the styp box may have the same format as the ftyp box.

[0118] The sidx box (segment index box) can provide information indicating an index for a split fragment, thereby indicating which fragment the corresponding split fragment is.

[0119] In accordance with the embodiment (530), an ssix box may be further included, and the ssix box (subsegment index box) may provide information indicating the index of a subsegment when the segment is further divided into subsegments.

[0120] A box in a media file can include more expanded information based on the box or full box format as in the illustrated embodiment 550. In this embodiment, the size field and the largesize field can indicate the length of the corresponding box in bytes, etc. The version field can indicate the version of the corresponding box format. The type field can indicate the type or identifier of the corresponding box. The flags field can indicate flags associated with the corresponding box, etc.

[0121] Meanwhile, the video / image-related fields (attributes) in this document can be included and transmitted in a DASH-based adaptive streaming model.

[0122] 7 shows an example of the overall operation of the DASH-based adaptive streaming model. The DASH-based adaptive streaming model according to the illustrated embodiment (700) describes the operation between an HTTP server and a DASH client. Here, DASH (Dynamic Adaptive Streaming over HTTP) is a protocol for supporting HTTP-based adaptive streaming, and can support dynamic streaming according to network conditions. This allows AV content playback to be provided seamlessly.

[0123] First, the DASH client can obtain the MPD. The MPD can be delivered from a service provider such as an HTTP server. The DASH client can request a corresponding segment from a server using access information to the segment described in the MPD. Here, this request can be performed reflecting the network state.

[0124] After acquiring the relevant segments, the DASH client processes them in the media engine and displays them on the screen. The DASH client can request and acquire the necessary segments by reflecting the playback time and / or network conditions in real time (Adaptive Streaming). This allows the content to be played seamlessly.

[0125] A Media Presentation Description (MPD) is a file that includes detailed information for enabling a DASH client to dynamically obtain a segment, and can be expressed in XML format.

[0126] The DASH client controller can generate commands to request MPD and / or segments reflecting the network conditions, and can control the acquired information so that it can be used by internal blocks such as the media engine.

[0127] The MPD Parser can parse the retrieved MPD in real time, allowing the DASH client controller to generate commands that can retrieve the required segments.

[0128] The segment parser can parse the acquired segments in real time. Depending on the information contained in the segments, internal blocks such as the media engine can perform specific operations.

[0129] The HTTP client can request the required MPD and / or segments from the HTTP server, and can transfer the MPD and / or segments obtained from the server to the MPD parser or segment parser.

[0130] The Media Engine can display content on the screen using the media data included in the segment, utilizing the information in the MPD.

[0131] The DASH data model may have a hierarchical structure (710). A media presentation may be described by an MPD. The MPD may describe a temporal sequence of multiple periods that make up the media presentation. A period may represent a section of media content.

[0132] In one section, data can be included in an adaptation set. An adaptation set is a collection of media content components that can be interchanged with each other. An adaptation can include a collection of representations. A representation can correspond to a media content component. Within a representation, content can be divided in time into multiple segments for proper accessibility and delivery. A URL for each segment can be provided to access the respective segment.

[0133] MPD can provide information related to a media presentation, and a period element, an adaptation set element, and a representation element can be described for a corresponding period, an adaptation set, and a representation, respectively. A representation can be divided into sub-representations, and a sub-representation element can be described for a corresponding sub-representation.

[0134] Here, common attributes / elements can be defined, which can be applied to (included in) adaptation sets, representations, subrepresentations, etc. Among the common attributes / elements are essential properties and / or supplemental properties.

[0135] The essential properties are information including elements that are considered essential for processing the corresponding media presentation related data. The supplemental properties are information including elements that can be used for processing the corresponding media presentation related data. When a descriptor described below according to an embodiment is transmitted via an MPD, it can be defined and transmitted within the essential properties and / or the supplemental properties.

[0136] Meanwhile, a "sample" as described below refers to all data associated with a single time or a single element of one of the three sample arrays (Y, Cb, Cr) that represent a picture. For example, when the term "sample" is used in the context of a track (of a media file format), it can refer to all data associated with a single time of the track. Here, the time is a decoding time or a composition time. Also, for example, when the term "sample" is used in the context of a picture, i.e., when used in picture terms such as "luma sample," it can refer to a single element in one of the three sample arrays that represent a picture.

[0137] Meanwhile, to store VVC content, three types of elementary streams can be defined as follows:

[0138] - a video elementary stream that contains a VCL NAL unit and does not contain any parameter set, DCI or OPI NAL units, where all parameter set, DCI and OPI NAL units can be stored in a sample entry or in a sample entry.

[0139] - A video and parameter set elementary stream containing a VCL NAL unit and which may contain a parameter set, DCI or OPI NAL unit and having a sample entry or sample entries in which the parameter set, DCI or OPI NAL unit is stored.

[0140] - a non-VCL elementary stream that contains non-VCL NAL units synchronized with the elementary streams contained in the video track, where a VVC non-VCL track does not contain parameter set, DCI or OPI NAL units within the sample entries.

[0141] Meanwhile, a VVC video stream can be defined as follows:

[0142] For example, a VVC sample entry can be defined as a sample entry whose sample entry type is "vvc1" or "vvi1". Each sample entry of a VVC track is a VVC sample entry. A VVC sample entry can include a VVC Configuration Box, defined below. The VVC Configuration Box can include a VVC decoder configuration record that contains decoder configuration information.

[0143] Also, an optional BitRateBox can be present in the VVC Sample Entry to signal bit rate information of the VVC video stream, and an Extension Descriptor can also be present that should be inserted into the Elementary Stream Descriptor when used in MPEG-4.

[0144] Additionally, video sections using other configurations or parameter sets may be represented using multiple sample entries, as permitted by the ISO Base Media File Format specification.

[0145] If a VVC subpicture track can be consumed without other VVC subpicture tracks and contains a conforming VVC bitstream, then the regular VVC sample entry ("vvc1" or "vvi1") can be used for the VVC subpicture track. Otherwise, the "vvs1" sample entry can be used for the VVC subpicture track and the following constraints can be applied to the track:

[0146] The -track_in_movie flag is equivalent to 0.

[0147] -A track contains only one sample entry.

[0148] - Tracks are referenced by one or more VVC base tracks via 'subp' track references.

[0149] - DCI, OPI, VPS, SPS, PPS, AUD, PH, EOS, EOB and other AU or picture level non-VCL NAL units SHOULD be absent both in sample entries and in samples of the "vvs1" track.

[0150] -Unless otherwise specified, child boxes of a video sample entry (e.g. CleanApertureBox and PixelAspectRatioBox) should not be present in the sample entry and will be ignored if present.

[0151] A sample is not marked as a sync sample unless all VCL NAL units contained in the sample comply with the sync sample requirements.

[0152] - There is no composition time offset information for samples in the 'vvs1' track.

[0153] -Subsample information for samples in the 'vvs1' track MAY be present. If present, the subsample information MUST conform to the subsample definition for VVC.

[0154] A VVC track may also contain a 'subp' track reference with an entry containing one of the track_ID values ​​of a VVC subpicture track or the track_group_id value of an 'alte' track group of a VVC subpicture track. A 'spor' type sample group is present in each VVC base track. A dedicated sample entry type for a VVC base track may indicate the VVC base track via the track type's codecs MIME parameter.

[0155] Also, a sample entry of type "vvs1" may contain a VvcNALUConfigBox. If a VVC subpicture track is referenced by a VVC base track that contains a "spor" sample group description entry with subpic_id_info_flag set to 1, then the VVC subpicture track may potentially contain a subpicture ID sample group description that uses the default sample grouping mechanism.

[0156] Also, if the sample entry name is "vvc1" or "vvi1", the stream to which this sample item applies is a compliant VVC stream when viewed by a VVC decoder operating with the configuration (including profile, tier, and level) provided in the VVCConfigurationBox.

[0157] Also, if the sample entry name is "vvc1", the value of array_completeness is equal to 1 for the DCI, VPS, SPS, and PPS NAL unit arrays, and equal to 0 for all other arrays. Also, if the sample entry name is "vvi1", the value of array_completeness is equal to 0 for all arrays.

[0158] Also, if a track does not essentially contain a VVC bitstream and does not indicate a VVC bitstream after checking the 'subp' and 'vvcN' track references (if present), the track may contain an 'oref' track reference to a track carrying a 'vopi' sample group or an Operating Points Entity Group. For example, if a single-layer VVC bitstream contains two temporal sublayers stored in different tracks, a track containing a sublayer with TemporalId=1 may contain an 'oref' track reference to a track containing a sublayer with TemporalId=0.

[0159] Meanwhile, operating point information of the ISO based media file format (ISOBMF) for VVC can be signaled to samples in a group box with a grouping type of 'vopi' or an entity group with a grouping type of 'opeg'. The operating point information is necessary to identify samples and sample entries for each operating point.

[0160] Meanwhile, an operating point is a temporal subset of an OSL that can be identified by an output layer set (OLS) index and the highest value of TemporalId. Each operating point can be associated with a profile, tier, and level (i.e., PTL) that defines the conformance point of the operating point.

[0161] Also, for example, an operating point entity group can be defined to provide track mapping for operating points and profile level information for operating points. If the 'opeg' entity group is present, the file can include a file-level Extended Type Box with a brand combination that includes the 'unif' brand as specified in ISO / IEC 14496-12.

[0162] When aggregating samples of tracks mapped to the operation points described in the operation point entity group, the implicit reconstruction process does not need to remove any more VCL NAL units to obtain a conforming VVC bitstream. Tracks that belong to the operation point entity group have track references of type 'oref' for the group_id indicated in the operation point entity group and do not contain a 'vopi' sample group.

[0163] In addition, all entity_id values ​​included in the operation point entity group must indicate track IDs of tracks belonging to the same VVC bitstream. If present, the OperatingPointGroupBox is included in the GroupsListBox of the file-level MetaBox and is not included in MetaBoxes at other levels than the file level. Here, the OperatingPointGroupBox can indicate the operation point entity group.

[0164] The syntax of the above-mentioned action point entity group is as shown in the following table.

[0165] [Table 1-1]

[0166] [Table 1-2]

[0167] In addition, the semantics of the syntax of the action point entity group are as shown in the following table.

[0168] [Table 2-1]

[0169] [Table 2-2]

[0170] [Table 2-3]

[0171] [Table 2-4]

[0172] [Table 2-5]

[0173] Also, for example, a media file may include a VVC bitstream entity group, which may be defined to provide a VVC track of a single VVC elementary stream.

[0174] For example, if there is an EntityToGroupBox with grouping_type "vvcb", the following constraints may apply:

[0175] 1) An EntityToGroupBox must be contained in a GroupsListBox of a file-level MetaBox, and not in a MetaBox of any other level.

[0176] 2) The entity_id value included in the EntityToGroupBox indicates the track IDs of all VVC tracks that carry one VVC elementary stream.

[0177] 3) The first entity_id value included in the EntityToGroupBox indicates the track ID of the VVC track that carries the "vopi" sample group.

[0178] 4) A VVC track having a track ID value represented by the second or subsequent entity_id value included in the EntityToGroupBox does not convey the "vopi" sample group.

[0179] Also, for example, the media file may include decoder configuration information for the image / video content, i.e., the media file may include a VVC decoder configuration record that includes the decoder configuration information.

[0180] When the VVC decoder configuration record is stored in a sample entry, the VVC decoder configuration record may include the parameter set, DCI, OPI, and SEI NAL units as well as the size of the length field used for each sample to indicate the length of the NAL units included in the VVC decoder configuration record. The VVC decoder configuration record may be externally framed (the size of the VVC decoder configuration record is provided in the structure that contains the VVC decoder configuration record).

[0181] The VVC decoder configuration record may also include a version field. The version in this document may define version 1 of the VVC decoder configuration record. Incompatible changes to the VVC decoder configuration record may be indicated by a change in the version number. If the version number is not recognized, readers should not decode the VVC decoder configuration record or the stream to which the record applies.

[0182] Compatible extensions to the VVC Decoder Configuration Record do not change the Configuration Version Code. Readers must be prepared to ignore data that they do not recognize beyond the definition of the data they understand.

[0183] If a track essentially contains a VVC bitstream or is resolving via a 'subp' track reference, then a VvcPtlRecord must be present in the decoder configuration record, in which case the specific output layer set for the VVC bitstream can be indicated in the output_layer_set_idx field. Also, if ptl_present_flag in a track's decoder configuration record is 0, then the track must have an 'oref' track reference to an ID that can reference a VVC track or an 'opeg' entity group.

[0184] When a stream described in the VVC Decoder Configuration Record is decoded, the values ​​of the syntax elements VvcPTLRecord, chroma_format_idc, and bit_depth_minus8 may be valid for all referenced parameter sets. In particular, the following restrictions may apply:

[0185] - The profile indication general_profile_idc indicates the profile adhered to by the OLS identified by output_layer_set_idx in the configuration record.

[0186] - The tier indication general_tier_flag may indicate a tier greater than or equal to the highest tier indicated in the profile_tier_level() syntax structure (all parameter sets) adhered to by the OLS identified by output_layer_set_idx in the configuration record.

[0187] -Each bit of general_constraint_info can be set only if the bit is set in all general_constraints_info() syntax structures of the profile_tier_level() syntax structure (all parameter sets) that the OLS identified by output_layer_set_idx in the configuration record complies with.

[0188] -The level indication general_level_idc may indicate a level of capability that is greater than or equal to the highest level of the profile_tier_level() syntax structure (all parameter sets) adhered to by the OLS identified by output_layer_set_idx in the configuration record.

[0189] In addition, the following constraints may be applied to chroma_format_idc:

[0190] - If the VVC stream to which the configuration record applies is a single-layer bitstream, the value of sps_chroma_format_idc defined in ISO / IEC 23090-3 MUST be the same for all SPS referenced by the VCL NAL units of the sample to which the current sample entry description applies, and chroma_format_idc MUST be the same as sps_chroma_format_idc.

[0191] - Otherwise (if the VVC stream to which the configuration record applies is a multi-layer bitstream), the value of vps_ols_dpb_chroma_format[MultiLayerOlsIdx[output_layer_set_idx]] defined in ISO / IEC 23090-3 MUST be the same for all CVS to which the current sample entry description applies, and chroma_format_idc MUST be the same as the value of vps_ols_dpb_chroma_format[MultiLayerOlsIdx[output_layer_set_idx]].

[0192] In addition, the following constraints may be applied to bit_depth_minus8.

[0193] -If the VVC stream to which the configuration record applies is a single-layer bitstream, the value of sps_bitdepth_minus8 defined in ISO / IEC 23090-3 must be the same for all SPS referenced by the VCL NAL units of the sample to which the current sample entry description applies, and the value of bit_depth_minus8 must be the same as the value of sps_bitdepth_minus8.

[0194] - Otherwise (if the VVC stream to which the configuration record applies is a multi-layer bitstream), the value of vps_ols_dpb_bitdepth_minus8[MultiLayerOlsIdx[output_layer_set_idx]] defined in ISO / IEC 23090-3 shall be the same for all VVCs to which the current sample entry description applies, and the value of bit_depth_minus8 shall be the same as the value of vps_ols_dpb_bitdepth_minus8[MultiLayerOlsIdx[output_layer_set_idx]].

[0195] In addition, the following constraints may be applied to picture_width:

[0196] - If the VVC stream to which the configuration record applies is a single-layer bitstream, the value of sps_pic_width_max_in_luma_samples defined in ISO / IEC 23090-3 must be the same for all SPS referenced by the VCL NAL units of the sample to which the current sample entry description applies, and the value of picture_width must be the same as the value of sps_pic_width_max_in_luma_samples.

[0197] - Otherwise (if the VVC stream to which the configuration record applies is a multi-layer bitstream), the value of vps_ols_dpb_pic_width[MultiLayerOlsIdx[output_layer_set_idx]] defined in ISO / IEC 23090-3 shall be the same for all VVCs to which the current sample entry description applies, and the value of picture_width shall be the same as the value of vps_ols_dpb_pic_width[MultiLayerOlsIdx[output_layer_set_idx]].

[0198] In addition, the following constraints may be applied to picture_height:

[0199] - If the VVC stream to which the configuration record applies is a single-layer bitstream, the value of sps_pic_height_max_in_luma_samples defined in ISO / IEC 23090-3 shall be the same for all SPS referenced by the VCL NAL units of the sample to which the current sample entry description applies, and the value of picture_height shall be the same as the value of sps_pic_height_max_in_luma_samples.

[0200] - Otherwise (if the VVC stream to which the configuration record applies is a multi-layer bitstream), the value of vps_ols_dpb_pic_height[MultiLayerOlsIdx[output_layer_set_idx]] defined in ISO / IEC 23090-3 shall be the same for all VVCs to which the current sample entry description applies, and the value of picture_height shall be the same as the value of vps_ols_dpb_pic_height[MultiLayerOlsIdx[output_layer_set_idx]].

[0201] An explicit indication for the chroma format and bit depth, as well as other important format information used in the VVC video elementary stream, can be provided in the VVC Decoder Configuration Record. If the color space or bit depth indications differ in the VUI information of the two sequences, two different VVC sample entries are required.

[0202] The VVC decoder configuration record also has a set of arrays that carry initialization NAL units. The NAL unit types can be restricted to indicate only DCI, OPI, VPS, SPS, PPS, prefix APS, and prefix SEI NAL units. NAL unit types reserved in ISO / IEC 23090-3 and this document may be defined in the future, and readers must ignore arrays with reserved or unpermitted values ​​of NAL unit types.

[0203] Meanwhile, the array is in the following order: DCI, OPI, VPS, SPS, PPS, prefix APS, prefix SEI.

[0204] The syntax of the above-mentioned VVC decoder configuration record is as shown in the table below.

[0205] [Table 3-1]

[0206] [Table 3-2]

[0207] Also, the semantics of the syntax of the VVC decoder configuration record are as shown in the following table.

[0208] [Table 4-1]

[0209] [Table 4-2]

[0210] [Table 4-3]

[0211] On the other hand, the VVC file format defined in this document defines the following types of tracks:

[0212] a) VVC track: A VVC track contains NAL units in samples and sample entries, possibly references other VVC tracks that contain other sublayers of the VVC bitstream, and possibly references VVC subpicture tracks to indicate the VVC bitstream. If a VVC track references a VVC subpicture track, the VVC track can be called a VVC base track.

[0213] b) VVC non-VCL track: APSs carrying ALF, LMCS or scaling list parameters, and other non-VCL NAL units can be stored in a VVC non-VCL track and can be transmitted via this track separately from tracks containing VCL NAL units.

[0214] c) VVC subpicture track: A VVC subpicture track may contain one of the following:

[0215] For example, a VVC sub-picture track may contain a sequence of one or more VVC sub-pictures.

[0216] Also for example, a VVC sub-picture track may contain one or more complete slices forming a rectangular area.

[0217] On the other hand, a sample in a VVC Sub-picture track can contain one of the following:

[0218] For example, a sample of a VVC subpicture track may contain one or more complete subpictures as specified in ISO / IEC 23090-3 that are contiguous in decoding order.

[0219] Also, for example, samples of a VVC sub-picture track may include one or more complete slices as specified in ISO / IEC 23090-3 that form a rectangular region and are contiguous in decoding order.

[0220] On the other hand, the VVC sub-pictures or slices contained in any sample of a VVC sub-picture track are consecutive in decoding order.

[0221] The VVC non-VCL track and the VVC sub-picture track enable optimal delivery of VVC video in streaming applications as follows:

[0222] For example, the tracks can each be carried in its own DASH representations, and for decoding and rendering of a subset of the tracks, a DASH representation containing a subset of the VVC subpicture tracks and a DASH representation containing non-VCL tracks can be required. Such an approach can eliminate redundant transmission of APSs and other non-VCL NAL units.

[0223] Meanwhile, the data sharing and VVC bitstream reconstruction process can be performed as follows: where the output of the process is a VVC bitstream, which can be referred to as an output bitstream.

[0224] For example, a file reader must invoke the process if one or more of the following conditions are true:

[0225] i) A VVC bitstream ("vvcb") entity group is present in the file, and a file reader processes the VVC bitstream represented by said entity group to generate an output bitstream.

[0226] ii) Operation point ("opeg") entity groups are present in the file, and the file reader generates an output bitstream using any operation points described by the entity groups.

[0227] iii) The file reader extracts a subset of layers or sublayers of VVC tracks with ptl_present_flag equal to 1 to generate an output bitstream.

[0228] iv) The file reader processes the VVC basic tracks.

[0229] v) The file reader processes a VVC track that has an associated VVC non-VCL track.

[0230] For example, the process may consist of the following sequence of steps:

[0231] 1) If condition i, ii or iii above is true, the operation point is determined at the beginning of the VVC bitstream and can be determined again for any IRAP or GDR access unit.

[0232] When a file reader selects an action point for the first time or selects an action point that is different from any previously selected action point, the file reader MUST indicate to the VVC decoder the output layer set index and the highest TemporalId value of the selected action point by including it in the OPI NAL unit inserted into the output bitstream (the first NAL unit following the AU delimiter NAL unit in the first access unit in which the action point is used, if any), or via other means.

[0233] The next ordering step can be applied to the sequence of access units in decoding order starting from the access unit for which the action point is determined to the end of the bitstream or access unit, where the action point can be determined exclusively next and earlier in decoding order.

[0234] 2) If a VVC bitstream is represented with multiple VVC tracks, the file parser can identify the track required for the selected operation point as follows:

[0235] i) if the operation points in the 'opeg' entity group contain the selected operation point, the file parser may select tracks that belong to the operation point as displayed in the 'opeg' entity group.

[0236] ii) If the 'opeg' entity group does not exist (i.e., if the 'vopi' sample group exists), the file parser may determine whether the VVC track in the 'vvcb' entity group indicates a VVC bitstream. The first entity_id of the 'vvcb' entity group may identify the track that contains the 'vopi' sample group. The operation point mapping to layers and sublayers may be derived from the 'vopi' sample group. The set of tracks that contain the layers and sublayers of the selected operation point, and are therefore required for decoding the selected operation point, may be derived from the 'linf' sample group present in the VVC track of the VVC bitstream. Because a particular layer or sublayer may be represented by one or more tracks, when determining the track required for the operation point, it may be selected from among the set of tracks that contain all of the particular layers or sublayers.

[0237] 3) The access units for the output bitstream can be restored to the decoding time order of samples of the VVC track required for the selected operation point (if condition i, ii or iii above is true) or the VVC basic track (if condition iv above is true) or the VVC track (if condition v above is true).

[0238] If multiple tracks contain data for an access unit, alignment of each sample in the tracks may be performed based on sample decoding times.

[0239] The sequence of access units can be reconstructed at each sample of the required tracks by repeatedly invoking the implicit reconstruction process described below.

[0240] The reconstructed access units may be arranged in the output bitstream in order of increasing decoding time.

[0241] 4) Additionally, a file reader MUST include an EOS NAL unit in each layer of an applied operation point in the output bitstream if all of the following two conditions are true:

[0242] - Selecting an operating point where the sequence of access units is different from the one previously selected.

[0243] - The sequence of access units does not end with an EOS or EOB NAL unit in every layer of the applied operation point.

[0244] When a VVC bitstream is represented with multiple VVC tracks, the sample decoding times must be such that when the tracks are combined into a single bitstream that is aligned with increasing decoding time, the access unit order is modified as specified in ISO / IEC 23090-3.

[0245] Additionally, an implicit reconstruction process of a VVC bitstream can be performed as follows, which can represent the reconstruction of an access unit from time-aligned samples having a current decoding time among:

[0246] - Required VVC track(s),

[0247] - Associated VVC non-VCL tracks (if any) and

[0248] -referenced VVC subpicture tracks (if any).

[0249] When reconstructing a bitstream that includes sublayers whose VCL NAL units have a TemporalId greater than 0, all lower sublayers in the same layer (i.e., sublayers whose VCL NAL units have a smaller TemporalId) can also be included in the resulting bitstream.

[0250] Also, if the sample(s) with the current decoding time contains a VCL NAL unit with a TemporalId greater than the largest TemporalId included in the selected operation point, the access unit is not restored from the current decoding time.

[0251] Also, when reconstructing an access unit, sample picture units (as specified in ISO / IEC 23090-3) having the same decoding time may be arranged in the access unit in ascending order of nuh_layer_id value. For example, if the condition applies, the following steps may be performed:

[0252] - If a sample of a track including the first picture unit of a sample is indicated as belonging to a sample group 'aud' including ols_idx and lower_ols_idx corresponding to a target operation point in an AUD NAL unit, the AUD NAL unit in the 'aud' sample group is the first NAL unit and can be placed in an AU.

[0253] If a track sample is indicated as belonging to sample group 'eos' including ols_idx, max_tid and lower_ols_idx, then the EOS NAL unit in the 'eos' sample group corresponding to the target operation point may be placed in the AU at the indicated position, i.e., after the eos_position-th NAL unit of the reconstructed AU excluding the AUD NAL unit, if inserted by sample group 'aud'.

[0254] -If a track sample is indicated as belonging to a sample group 'eos' including ols_idx, max_tid and lower_ols_idx, the EOB NAL unit in the 'eob' sample group corresponding to the target operation point can be placed in the AU next to all NAL units of the AU (including the EOS NAL unit).

[0255] Only picture units of layers and sublayers that are in the target operation point can be included in the output bitstream.

[0256] Also, when reconstructing an access unit based on the action point associated with output layer set index i (in a for loop on num_olss of the 'vopi' sample group), the following may apply.

[0257] When recovering an access unit, for each layer of the output layers set to index j ranging from 0 to layer_count[i]-1, if num_ref_sublayers_in_layer_in_ols[i][j] is greater than 0, the VCL NAL unit may belong to a sublayer of a layer with TemporalId less than or equal to Min(num_ref_sublayers_in_layer_in_ols[i][j]-1, max_temporal_id), where max_temporal_id is the value of the corresponding syntax element for the operation point that is included in the resulting bitstream, so that the required track can be selected.

[0258] -When recovering an access unit, for each layer of the output layer set to index j ranging from 0 to layer_count[i]-1, if num_ref_sublayers_in_layer_in_ols[i][j] is equal to 0, only IRAP and GDR picture units with ph_recovery_poc_cnt equal to 0 from all picture units of the reference layer are included in the resulting bitstream, and the required track can be selected thereby.

[0259] In addition, if an access unit of a VVC track contains an unspecified NAL unit type (NAL units having a nal_unit_type in the range UNSPEC_28..UNSPEC_31 (inclusive), i.e., units having a nal_unit_type value in the range 28 to 31 defined in ISO / IEC 23090-3), the unspecified NAL unit type can be discarded from the final reconstructed bitstream.

[0260] Also, if the VVC track contains a "subp" track reference, each picture unit can be restored as specified in clause 11.6.3 of ISO / IEC 23090-3. Also, if the VVC track contains a "recr" track reference, each picture unit can be restored as specified in clause 11.6.6 of ISO / IEC 23090-3. The process of clause 11.6.3 or 11.6.6 of ISO / IEC 23090-3 can be repeated for each layer of the target operation point in ascending order of nuh_layer_id ordering.

[0261] The reconstructed access units may be arranged in the VVC bitstream in order of increasing decoding time.

[0262] Since a particular layer or sublayer may be represented by one or more tracks, when determining the track required for an action point, it may be selected from a track set that includes all of the particular layer or sublayer.

[0263] Furthermore, the picture unit recovery process for samples of a VVC track that references a VVC sub-picture track can be performed as follows: Samples of a VVC track can be resolved with picture units in the following order:

[0264] - If it is in a time-aligned sample of a sample or an associated VVC non-VCL track, the AUD NAL unit may be included in a picture unit, e.g., the first NAL unit of a sample if the AUD NAL unit is present in the sample.

[0265] - If the sample is the first sample of a sample sequence associated with the same sample entry, the DCI, OPI, parameter set, and SEI NAL units (if any) contained in the sample entry can be included in the picture unit.

[0266] - If there is an associated VVC non-VCL track and the picture unit is the first picture unit of an access unit reconstructed from samples, the following NAL units are included in the picture unit: If there is at least one NAL unit in the time-aligned samples of the associated VVC non-VCL track with nal_unit_type equal to EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT or RSV_NVCL_27 (NAL units with the above NAL unit types cannot precede the first VCL NAL unit in a picture unit), the above NAL units except the first unit in the time-aligned samples of the associated VVC non-VCL track (excluding the AUD NAL unit) may be included. Otherwise, all NAL units of the time-aligned samples of the associated VVC non-VCL track may be included.

[0267] -If a sample has one or more NAL units with nal_unit_type equal to EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT or RSV_NVCL_27 (NAL units with the above NAL unit types cannot precede the first VCL NAL unit in a picture unit), then all NAL units of the sample except the first one can be included in the picture unit; otherwise, all NAL units of the sample can be included in the picture unit.

[0268] - If the reader selects an action point, it must exclude all OPI NAL units stored in the sample entries and samples in the access units restored in all the steps above.

[0269] If a VVC track does not reference a VVC sub-picture track, the NAL units of a sample in the VVC track can be contained in a picture unit.

[0270] If not, the following may be applied:

[0271] - Track references may be resolved as specified in clause 11.6.4 of ISO / IEC 23090-3.

[0272] The parameter set can be updated if necessary as specified in clause 11.6.5 of ISO / IEC 23090-3.

[0273] -Picture units may be added in the order of the VVC subpicture track referenced by the 'subp' track reference (if num_subpic_ref_idx of the same group entry of the 'spor' sample group entry mapped to the sample is 0) or in the order specified in the 'spor' sample group entry mapped to the sample (if num_subpic_ref_idx of the same group entry of the 'spor' sample group entry mapped to the sample is greater than 0) with the contents of resolved samples that are time-aligned (at decoding time) in each referenced VVC subpicture track except for all DCI, OPI, VPS, SPS, PPS, AUD, PH, EOS, and EOB NAL units as well as scalable nesting SEI NAL messages with sn_subpic_flag set to 1.

[0274] Also, if the referenced VVC sub-picture track is associated with a VVC non-VCL track, the resolved samples of the VVC sub-picture track may include the following NAL units:

[0275] - If there is at least one NAL unit in the time-aligned sample of the associated VVC non-VCL track with nal_unit_type equal to EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT or RSV_NVCL_27 (NAL units with the above NAL unit types cannot precede the first VCL NAL unit in a picture unit), the above NAL units except the first unit in the time-aligned sample of the associated VVC non-VCL track (excluding the AUD NAL unit).

[0276] - otherwise, all NAL units of time-aligned samples of the associated VVC non-VCL track.

[0277] - A NAL unit from a sample of the referenced VVC subpicture track.

[0278] - The remaining NAL units (if any) with time-aligned samples of the associated VVC non-VCL track.

[0279] - all NAL units of the sample whose nal_unit_type is EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT, or RSV_NVCL_27.

[0280] - if there is an associated VVC non-VCL track and the picture unit is the last picture unit of an access unit reconstructed from samples, all NAL units of the time-aligned samples of the associated VVC non-VCL track whose nal_unit_type is EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT, or RSV_NVCL_27.

[0281] -All NAL units or NAL unit-like structures of a sample whose nal_unit_type is in the UNSPEC_28..UNSPEC_31 (inclusive) range may be deleted.

[0282] Meanwhile, the sample groups disclosed in this document may include the following sample groups.

[0283] For example, a stream access point sample group can be defined. The stream access point (SAP) sample group "sap" specified in ISO / IEC 14496-12 can be used to provide information for all SAPs.

[0284] For example, if layer_id_method_idc is 0, the SAP can be parsed as follows:

[0285] - If the sample entry type is "vvc1" or "vvi1" and the track does not contain a sublayer with TemporalId equal to 0, the SAP can specify access to all sublayers in the track.

[0286] - Otherwise, the SAP can specify access to all layers in the track.

[0287] Also, for example, a GDR picture in a VVC bitstream can generally be represented as SAP type 4 in the "sap" sample group.

[0288] Also, for example, if a sample of a VVC track references a PPS where pps_mixed_nalu_types_in_pic_flag is 1, and for each subpicture index i in the range from 0 to sps_num_subpics_minus1, sps_subpic_treated_as_pic_flag[i] is 1, and there is at least one IRAP subpicture with the same subpicture index i in the same CLVS or following the current sample, the following may apply:

[0289] -Samples can be shown in SAP Samples of Type 4.

[0290] - The sample can be mapped to a "roll" sample group description entry with a roll_distance value modified for the decoding process to omit decoding of a subpicture with a particular subpicture index before the IRAP subpicture is present.

[0291] If a SAP sample group is used, it must be used in all tracks carrying the same VVC bitstream.

[0292] Also, for example, a random access recovery point sample group can be defined. The random access recovery point sample group "roll" specified in ISO / IEC 14496-12 can be used when providing information on recovery points for gradual decoding refresh. When the "roll" sample group is used with a VVC track, the syntax and semantics of the grouping_type_parameter can be specified similarly to that for the "sap" sample group in ISO / IEC 14496-12. Also, layer_id_method_idc equal to 0 and 1 can be used when the picture of the target layer of the sample mapped to the "roll" sample group is a GDR picture. For example, when layer_id_method_idc is equal to 0, the "roll" sample group can specify behavior for all layers in the track. Also, for example, the semantics of layer_id_method_idc the same as 1 are disclosed in Section 9.5.7 of ISO / IEC 14496-12. For example, layer_id_method_idc the same as 2 and 3 can be used when all pictures of the target layer of samples mapped to the "roll" sample group are not GDR pictures, and the following can be applied to pictures of the target layer that are not GDR pictures.

[0293] - The referenced PPS has pps_mixed_nalu_types_in_pic_flag equal to 1.

[0294] For each subpicture index i in the range -0 to sps_num_subpics_minus1, the following two are all true: sps_subpic_treated_as_pic_flag[i] is 1 and there is at least one IRAP subpicture that follows the current sample in the same CLVS or has the same subpicture index i.

[0295] Also, for example, if layer_id_method_idc is equal to 2, the "roll" sample group can specify behavior for all layers in the track. Also, for example, the semantics of layer_id_method_idc equal to 3 are disclosed in clause 9.5.7 of ISO / IEC 14496-12.

[0296] If a reader uses a sample indicated with layer_id_method_idc equal to 2 or 3 to begin decoding, the reader must additionally modify the SPS, PPS and PH NAL units of the restored bitstream as follows so that the bitstream starting with a sample indicated as belonging to a sample group with layer_id_method_idc of 2 and 3 becomes a matching bitstream.

[0297] -All SPS referenced in the sample may have sps_gdr_enabled_flag equal to 1.

[0298] -All PPSs referenced in the sample may have pps_mixed_nalu_types_in_pic_flag equal to 0.

[0299] All VCL NAL units of an AU recovered from a sample may have the same nal_unit_type as GDR_NUT.

[0300] - All picture headers of an AU recovered from a sample may have a ph_gdr_pic_flag equal to 1 and a ph_recovery_poc_cnt value corresponding to the roll_distance of the 'roll' sample group description entry to which the sample is mapped.

[0301] Also, for example, if a "roll" sample group is associated with a dependent layer, but not a reference layer, the sample group may indicate properties that apply when all reference layers of the dependent layer are available and decoded, and may be used when starting decoding of a predicted layer.

[0302] Also, for example, a layer information sample group can be defined. A list of layers and sublayers that the track carries can be signaled in the layer information ("linf") sample group. If the current track contains more than one layer, more than one sublayer, or one sublayer, but the sublayer is not a sublayer with TemporalId equal to 0, the layer and sublayer information is required when parsing information signaled in other sample groups such as the "sap", "rap", "sync", "tele" and "vopi" sample groups. Also, if the "linf" sample group is present in the VVC track, sub_layer_presence_flags is equal to 0.

[0303] Also, for example, an operating point information sample group can be defined. Applications can be informed of various operating points provided for VVC elementary streams and configurations using the operating point information sample group ("vopi"). The operating points are related to output layer set, max TemporalId value, profile, tier, and level signaling. All the above information can be captured by the "vopi" sample group. In addition to the above information, the operating point information sample group can also be provided with dependency information between layers.

[0304] Also, for example, if there are two or more VVC tracks for a VVC elementary stream and no Operating Point entity group exists for the VVC elementary stream, the following two may all be applied.

[0305] Among the VVC tracks for a VVC elementary stream there is only one track carrying the "vopi" sample group.

[0306] - All other VVC tracks in the VVC elementary stream must have a track reference of type 'vref' to the track carrying the 'vopi' sample group.

[0307] For a particular sample of a given track, the temporally collocated samples of other tracks can be defined to have the same decoding time as the particular sample. k Track T with a "vref" track reference for N Each sample S N The following can be applied to

[0308] -Track T k A temporal collocate sample of S k If there is a sample S N is sample S k can be associated with the same "vopi" sample group entry.

[0309] - If not, sample S N is the number of samples S at the decoding time. N The track preceding T k The last sample in may be associated with the same "vopi" sample group entry.

[0310] Also, if a VVC elementary stream references multiple VPSs, it must contain multiple entries in the sample group description box with grouping_type "vopi". Also, if there is only one VPS, it is better to include the operating points information sample group in the sample table box by including it in each track fragment using a default sample group description index. Also, the grouping_type_parameter is not defined for SampleToGroupBox with grouping type "vopi".

[0311] The "vopi" sample group entry may include a VVC operation point record, the syntax of which is shown in the following table:

[0312] [Table 5-1]

[0313] [Table 5-2]

[0314] [Table 5-3]

[0315] In addition, the semantics of the syntax of the VVC action point record are as shown in the table below.

[0316] [Table 6-1]

[0317] [Table 6-2]

[0318] [Table 6-3]

[0319] [Table 6-4]

[0320] [Table 6-5]

[0321] [Table 6-6]

[0322] Also, for example, an access unit delimiter sample group can be defined, whose sample group description entry can include an AUD NAL unit.

[0323] If a sample is mapped to an access unit delimiter sample group ("aud"), then if the target operation point corresponds to one of the output layer sets represented in the "aud" sample group, it can indicate that the AUD NAL unit contained within the sample group should be inserted into the recovered AU.

[0324] The syntax of the AUD sample entry of the access unit delimiter sample group is as shown in the table below.

[0325] [Table 7]

[0326] In addition, the semantics of the syntax of the AUD sample entry are as shown in the table below.

[0327] [Table 8]

[0328] Also, for example, an end of sequence sample group can be defined, whose sample group description entry can include an EOS NAL unit.

[0329] When a sample is mapped to an end of sequence sample group ("eos"), if the target operation point corresponds to one of the output layer sets and maximum temporal IDs indicated in the "eos" sample group, it can indicate that the EOS NAL unit contained in the sample group should be inserted at the indicated position of the recovered AU.

[0330] The syntax of the sequence end sample entry of the sequence end sample group is as shown in the table below.

[0331] [Table 9]

[0332] In addition, the semantics of the syntax of the sequence end sample entry are as shown in the table below.

[0333] [Table 10]

[0334] Also, for example, an End of bitstream sample group can be defined, whose sample group description entry can include an EOB NAL unit.

[0335] If a sample is mapped to an end of bitstream sample group ("eob"), then if the target operation point corresponds to one of the output layer sets and maximum temporal IDs indicated in the "eob" sample group, it can indicate that the EOB NAL unit contained in the sample group should be inserted into the recovered AU.

[0336] The syntax of the bitstream end sample entry of the bitstream end sample group is as shown in the table below.

[0337] [Table 11]

[0338] Also, the semantics of the syntax of the bitstream end sample entry are as shown in the following table.

[0339] [Table 12]

[0340] Meanwhile, when the target operating point is changed as described above, the current VVC file format specifies the following:

[0341] [Table 13]

[0342] As shown above in Table 13, a file reader MUST include an EOS NAL unit in each layer of an applied operation point in the output bitstream if the following two conditions are all true:

[0343] - Next in the sequence of access units an operation point is selected which is different from the previous operation point.

[0344] The sequence of access units does not end with an EOS or EOB NAL unit in each layer of the applied operation point.

[0345] However, the above process may cause the following problems.

[0346] First, the target operating point has two components including a target OLS and a maximum temporal ID, and even if only the maximum temporal ID is changed, the target operating point is considered to have been changed, but there is no need to call the above-mentioned process. The change in the maximum temporal ID can occur smoothly in response to a change in the network state. Therefore, a method can be proposed to call the above-mentioned process only when a change is related to the target OLS.

[0347] Second, the process requires that the file reader include an EOS NAL unit in each layer of the applied operation point in the output bitstream, which can mean that the file reader should be able to place the EOS NAL unit at the appropriate position within an access unit. This is burdensome. Because VVC allows an EOS NAL unit to be present at any position within an access unit as long as it is not located prior to the associated picture unit, there is no need for the file parser to put the EOS in the correct layer.

[0348] Accordingly, this document proposes solutions to the problems mentioned above. The proposed embodiments may be applied individually or in combination.

[0349] As a first example, this document proposes a method for invoking the process of inserting the EOS NAL unit only when the target OLS is changed, instead of invoking the process of inserting the EOS NAL unit when the target operation point is changed, and thus, when only the maximum temporal ID is changed, the process can be skipped.

[0350] As a second example, this document proposes a method for inserting an EOS NAL unit in the above process, such that a file parser can include the EOS NAL unit without having to place it in as many corresponding layers as there are layers of the target OLS.

[0351] As a third example, this document proposes a scheme that allows for the inclusion of all required EOS NAL units at the end of the access unit.

[0352] As a fourth example, this document proposes ways to make the above process optional.

[0353] As an example, an embodiment reflecting the first to third examples proposed in this document is as follows. The embodiment can be expressed in the VVC file format specification as shown in the following table.

[0354] [Table 14]

[0355] For example, according to Table 14, it can be determined whether the following two conditions are true: For example, the first condition is whether an output layer set (OLS) different from the previous OLS is selected next to the sequence of access units, and for example, the second condition is whether the sequence of access units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied OLS.

[0356] If the above two conditions are true, i.e., if an OLS different from the previous OLS is selected next to the sequence of access units and the sequence of access units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied OLS, the file reader may include as many EOS NAL units in the last access unit as the number of layers applied to the target output layer applied in the output bitstream. The included EOS NAL units may be placed at the end of the access unit in ascending order of the layer IDs of the NAL units.

[0357] As another example, an embodiment reflecting the first to fourth examples proposed in this document is as follows. The embodiment can be expressed in the VVC file format specification as shown in the following table.

[0358] [Table 15]

[0359] For example, according to Table 15, it can be determined whether the following two conditions are true: For example, the first condition is whether an output layer set (OLS) different from the previous OLS is selected next to the sequence of access units, and for example, the second condition is whether the sequence of access units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied OLS.

[0360] If the above two conditions are true, i.e., if an OLS different from the previous OLS is selected next to the sequence of access units and the sequence of access units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied OLS, the file reader may include an EOS NAL unit in the last access unit as many layers as the number of layers applied to the target output layer applied in the output bitstream. The process of including the EOS NAL unit is optional. The included EOS NAL units may be placed at the end of the access unit in ascending order of the layer IDs of the NAL units.

[0361] FIG. 8 illustrates an exemplary media file generation method to which the embodiments proposed in this document are applied.

[0362] Referring to FIG. 8, a first device may configure a track (S800). For example, the first device may configure a track according to the above-described embodiments. For example, the first device may represent a transmission end, an encoding end, or a media file generating end. Also, for example, the first device may configure a sub-picture track and a base track. Also, the first device may include an encoder.

[0363] The first device may generate a media file based on the track (S810). For example, the first device may generate a media file based on the track according to the embodiments described above.

[0364] FIG. 9 illustrates an exemplary method for decoding a media file generated by applying the embodiments proposed in this document.

[0365] 9, a second device may obtain / receive a media file including a track (S900). For example, the second device may obtain / receive a media file including a track according to the above-described embodiment. Also, for example, the second device may represent a receiving end, a decoding end, or a rendering end.

[0366] For example, the media file may include the information set forth in Table 1, Table 3, Table 5, Table 7, Table 9, and / or Table 11.

[0367] The second device may parse / obtain the track (S910). The second device may parse / obtain the track included in the media file. For example, the track may include a sub-picture track or a base track, etc. For example, the second device may parse a sub-picture track and a base track.

[0368] The second device can generate one or more sub-picture slices based on the track, generate reconstructed samples based on the track, and obtain information required for image / video decoding based on the track.

[0369] FIG. 10 illustrates a schematic diagram of a method for generating a media file by a media file generating apparatus according to the present document. The method disclosed in FIG. 10 may be performed by the media file generating apparatus disclosed in FIG. 11. The media file generating apparatus may represent the first apparatus described above. Specifically, for example, S1000 to S1030 in FIG. 10 may be performed by a media file generating unit of the media file generating apparatus. Also, although not shown, the process of encoding a bitstream including video information may be performed by an encoder. The encoder may be included in the media file generating apparatus or may be configured as an external component.

[0370] A media file generating apparatus generates a sequence of access units for a first Output Layer Set (OLS) (S1000). For example, the media file generating apparatus may derive a bitstream including encoded video information and generate a media file for the bitstream. For example, the media file generating apparatus may obtain the encoded video information via a network or a (digital) storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as a USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. Alternatively, for example, the media file generating apparatus may include an encoder and derive the encoded video information. For example, the media file generating apparatus may generate the first OLS for a bitstream including the encoded video information. For example, the media file generating apparatus may generate a sequence of access units for the first OLS.

[0371] The media file generation device determines whether a first condition and a second condition are true (S1010). The media file generation device can determine whether the first condition and the second condition are true.

[0372] For example, the first condition is whether an OLS different from the first OLS is selected next to the sequence of the access units. That is, for example, if an OLS different from the first OLS is selected next to the sequence of the access units, the first condition can be determined to be true. Meanwhile, for example, the OLS different from the first OLS can be represented as a second OLS.

[0373] Also, for example, the second condition is whether the sequence of the access units does not end with an EOS NAL unit or an EOB (End Of Bitstream) NAL unit in each layer of the first OLS. That is, for example, if the sequence of the access units does not end with an EOS NAL unit or an EOB (End Of Bitstream) NAL unit in each layer of the first OLS, the second condition can be determined to be true.

[0374] The media file generating device inserts an EOS (End Of Sequence, EOS) NAL (Network Abstraction Layer, NAL) unit into the first OLS as many times as the number of layers applied to the first OLS based on the first condition and the second condition being true (S1020). For example, when the first condition and the second condition are true, the media file generating device can insert an EOS (End Of Sequence, EOS) NAL (Network Abstraction Layer, NAL) unit into the first OLS as many times as the number of layers applied to the first OLS.

[0375] Also, for example, the EOS NAL unit is not placed in the corresponding layer.

[0376] Also, for example, the media file generation device may insert the EOS NAL unit to be inserted at an end of the last access unit of the sequence in ascending order of the layer IDs of the EOS NAL units. Also, for example, the EOS NAL units to be inserted may be arranged at an end of the access units of the sequence in ascending order of the layer IDs of the EOS NAL units.

[0377] Also, for example, if no OLS different from the first OLS is selected next to the sequence of the generated access units and only the target operation point is changed, the EOS NAL unit is not inserted. For example, if no OLS different from the first OLS is selected next to the sequence of the generated access units and only the max temporal ID is changed, the EOS NAL unit is not inserted.

[0378] The media file generation device generates a media file including the first OLS (S1030). The media file generation device can generate a media file including the first OLS and / or the second OLS.

[0379] Meanwhile, although not shown, the media file generation device may generate the second OLS. For example, the media file generation device may generate a sequence of access units for the second OLS. Also, although not shown, the media file generation device may store the generated media file in a (digital) storage medium or transmit it to the media file processing device via a network or a (digital) storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

[0380] Figure 11 illustrates a schematic diagram of a media file generating apparatus for performing a media file generating method according to the present document. The method disclosed in Figure 10 may be performed by the media file generating apparatus disclosed in Figure 11. Specifically, for example, a media file generating unit of the media file generating apparatus of Figure 11 may perform S1000 to S1030. Also, although not shown, a process of encoding a bitstream including video information may be performed by an encoder of the media file generating apparatus.

[0381] FIG. 12 illustrates a media file processing method by a media file processing device according to the present document. The method disclosed in FIG. 12 may be executed by the media file processing device disclosed in FIG. 13. The media file processing device may represent the second device described above. Specifically, for example, S1200 to S1220 in FIG. 12 may be executed by a media file processing unit of the media file processing device. The media file processing unit may include a file parser and / or a file reader. Also, although not illustrated, the process of obtaining a media file may be executed by a receiving unit, the process of deriving a bitstream including a sequence of access units for a first OLS and a sequence of access units for an OLS different from the first OLS may be executed by a media file processing unit, and the process of decoding the bitstream may be executed by a decoder. The decoder may be included in the media file processing device or may be configured as an external component.

[0382] A media file processing device reconstructs a sequence of access units for a first Output Layer Set (OLS) (S1200). For example, the media file processing device may obtain a media file via a network or a (digital) storage medium. The media file may include the sequence of access units for the first OLS. The media file processing device may reconstruct the sequence of access units for the first OLS. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as a USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

[0383] The media file processing device determines whether a first condition and a second condition are true (S1210). The media file processing device can determine whether the first condition and the second condition are true.

[0384] For example, the first condition is whether an OLS different from the first OLS is selected next to the sequence of the access units. That is, for example, if an OLS different from the first OLS is selected next to the sequence of the access units, the first condition can be determined to be true. Meanwhile, for example, the OLS different from the first OLS can be represented as a second OLS.

[0385] Also, for example, the second condition is whether the sequence of the access units does not end with an EOS NAL unit or an EOB (End Of Bitstream) NAL unit in each layer of the first OLS. That is, for example, if the sequence of the access units does not end with an EOS NAL unit or an EOB (End Of Bitstream) NAL unit in each layer of the first OLS, the second condition can be determined to be true.

[0386] The media file processing device inserts an EOS (End Of Sequence, EOS) NAL (Network Abstraction Layer, NAL) unit into the first OLS based on the first condition and the second condition being true (S1220). For example, when the first condition and the second condition are true, the media file processing device can insert an EOS (End Of Sequence, EOS) NAL (Network Abstraction Layer, NAL) unit into the first OLS based on the first condition and the second condition being true.

[0387] Also, for example, the EOS NAL unit is not placed in the corresponding layer.

[0388] Also, for example, the media file processing device may insert the EOS NAL unit to be inserted at an end of the last access unit of the sequence in ascending order of the layer IDs of the EOS NAL units. Also, for example, the EOS NAL units to be inserted may be arranged at an end of the access units of the sequence in ascending order of the layer IDs of the EOS NAL units.

[0389] Also, for example, if no OLS different from the first OLS is selected next to the sequence of the restored access units and only the target operation point is changed, the EOS NAL unit is not inserted. For example, if no OLS different from the first OLS is selected next to the sequence of the restored access units and only the max temporal ID is changed, the EOS NAL unit is not inserted.

[0390] Meanwhile, although not shown, the media file processing device may recover a sequence of access units for the second OLS. Also, the media file processing device may derive a bitstream including a sequence of access units for the first OLS and a sequence of access units for the second OLS, and may decode the bitstream. The bitstream may be referred to as a VVC bitstream or an output bitstream. For example, the media file processing device may decode video information in the bitstream and generate a recovered picture based on the video information.

[0391] Fig. 13 is a schematic diagram of a media file processing device that executes a media file processing method according to the present document. The method disclosed in Fig. 12 can be executed by the media file processing device disclosed in Fig. 13. Specifically, for example, a media file processing unit of the media file processing device of Fig. 12 can execute S1200 to S1220 of Fig. 12. Meanwhile, although not shown, the media file processing device may include a decoder, the process of obtaining a media file may be executed by a receiving unit, the process of deriving a bitstream including a sequence of access units for a first OLS and a sequence of access units for an OLS different from the first OLS may be executed by a media file processing unit, and the process of decoding the bitstream may be executed by a decoder.

[0392] According to the above-mentioned document, even if only the maximum temporal ID is changed, unnecessary operations of inserting EOS NAL units can be prevented, thereby improving overall coding efficiency.

[0393] In addition, according to this document, it is possible to prevent burden caused by inserting an EOS NAL unit at an appropriate position within an access unit, thereby improving overall coding efficiency.

[0394] In the above-described embodiments, the method is described with reference to a flow chart with a series of steps or blocks, but this document is not limited to the order of steps, and certain steps may occur in a different order or simultaneously with other steps than those described above. Also, those skilled in the art will understand that the steps shown in the flow chart are not exclusive, and other steps may be included, or one or more steps of the flow chart may be deleted without affecting the scope of this document.

[0395] The embodiments described in this document may be implemented and executed on a processor, a microprocessor, a controller, or a chip. For example, the functional units illustrated in the drawings may be implemented and executed on a computer, a processor, a microprocessor, a controller, or a chip. In this case, information (e.g., information on instructions) or algorithms for implementation may be stored in a digital recording medium.

[0396] In addition, the devices to which the embodiments of this document are applied may include multimedia broadcast transmitting / receiving devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video interactive devices, real-time communication devices such as video communication, mobile streaming devices, storage media, camcorders, custom video (video on demand, VOD) service providing devices, OTT video (over the top video) devices, Internet streaming service providing devices, three-dimensional (3D) video devices, virtual reality (VR) devices, augmented reality (AR) devices, image telephone video devices, transportation terminals (e.g., vehicle terminals, airplane terminals, ship terminals, etc.), and medical video devices, and may be used to process video signals or data signals. For example, OTT video (over the top video) devices may include game consoles, Blu-ray players, Internet-connected TVs, home theater systems, smartphones, tablet PCs, DVRs (Digital Video Recorders), etc.

[0397] In addition, the processing method to which the embodiments of this document are applied may be produced in the form of a program executed by a computer and may be stored in a computer-readable recording medium. Multimedia data having a data structure according to this document may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium may include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable recording medium may also include a medium realized in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bit stream generated by the encoding method may be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0398] Furthermore, the embodiments of the present document may be embodied in a computer program product having program code, the program code being executable by a computer in accordance with the embodiments of the present document. The program code may be stored on a computer readable carrier.

[0399] FIG. 14 exemplarily illustrates a structural diagram of a content streaming system to which the embodiments of this document are applied.

[0400] A content streaming system to which the embodiments of this document are applied can broadly include an encoding server, a streaming server, a web server, a media repository, a user device, and a multimedia input device.

[0401] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server may be omitted.

[0402] The bitstream may be generated by an encoding method or a bitstream generation method to which an embodiment of this document is applied, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0403] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server transmits multimedia data to the user. In this case, the content streaming system may include a separate control server, and in this case, the control server controls commands / responses between devices in the content streaming system.

[0404] The streaming server may receive content from a media repository and / or an encoding server. For example, when the content is received from the encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a certain period of time in order to provide a smooth streaming service.

[0405] Examples of the user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glass, head mounted displays (HMDs)), digital TVs, desktop computers, digital signage, etc. Each server in the content streaming system can be operated as a distributed server, and in this case, data received by each server can be processed in a distributed manner.

[0406] The claims herein may be combined in various ways. For example, the technical features of the method claims herein may be combined and embodied in an apparatus, the technical features of the device claims herein may be combined and embodied in a method, the technical features of the method claims herein and the technical features of the device claims herein may be combined and embodied in an apparatus, and the technical features of the method claims herein and the technical features of the device claims herein may be combined and embodied in a method.

Claims

1. 1. A method for processing a media file, comprising: recovering a sequence of access units for a first Output Layer Set (OLS); determining whether a first condition and a second condition are true; and inserting an End Of Sequence (EOS) Network Abstraction Layer (NAL) unit as many times as the number of layers applied to the first OLS based on the first condition and the second condition being true; the first condition is whether an OLS different from the first OLS is selected next to the sequence of access units; The second condition is whether the sequence of access units does not end with an EOS NAL unit or an EOB (End Of Bitstream) NAL unit in each layer of the first OLS.

2. The method of claim 1 , wherein the inserted EOS NAL units are arranged at an end of an access unit of the sequence in ascending order of layer IDs of the EOS NAL units.

3. The method of claim 1 , wherein the inserted EOS NAL unit is not placed in a corresponding layer.

4. 2. The method of claim 1, wherein the EOS NAL unit is not inserted if no OLS different from the first OLS is selected next to the sequence of the restored access units and only a target operating point is changed.

5. 2. The method of claim 1, wherein if no OLS different from the first OLS is selected next to the sequence of the recovered access units and only a max temporal ID is changed, the EOS NAL unit is not inserted.

6. 2. The method of claim 1, wherein the step of inserting an EOS NAL unit is optional.

7. A media file processing device for executing a media file processing method, A receiver for acquiring a media file; a media file processing unit that restores a sequence of access units for a first Output Layer Set (OLS) of the media file, determines whether a first condition and a second condition are true, and inserts End Of Sequence (EOS) Network Abstraction Layer (NAL) units as many as the number of layers applied to the first OLS based on whether the first condition and the second condition are true; the first condition is whether an OLS different from the first OLS is selected next to the sequence of access units; The second condition is whether the sequence of access units does not end with an EOS NAL unit or an EOB (End Of Bitstream) NAL unit in each layer of the first OLS.

8. The media file processing device of claim 7, wherein the inserted EOS NAL units are arranged at an end of an access unit of the sequence in ascending order of layer IDs of the EOS NAL units.

9. The media file processing device of claim 7, wherein the inserted EOS NAL unit is not placed in a corresponding layer.

10. 8. The media file processing device of claim 7, wherein the EOS NAL unit is not inserted if no OLS different from the first OLS is selected next to the sequence of the restored access units and only a target operating point is changed.

11. 8. The media file processing device of claim 7, wherein if no OLS different from the first OLS is selected next to the sequence of the recovered access units and only a max temporal ID is changed, the EOS NAL unit is not inserted.

12. The media file processing device of claim 7, wherein the step of inserting the EOS NAL unit is optional.

13. 1. A method for generating a media file, comprising: generating a sequence of access units for a first Output Layer Set (OLS); determining whether a first condition and a second condition are true; inserting an End Of Sequence (EOS) Network Abstraction Layer (NAL) unit into the first OLS as many times as the number of layers applied to the first OLS based on the first condition and the second condition being true; generating the media file including the first OLS; the first condition is whether an OLS different from the first OLS is selected next to the sequence of access units; The second condition is whether the sequence of access units does not end with an EOS NAL unit or an EOB (End Of Bitstream) NAL unit in each layer of the first OLS.

14. The method of claim 13, wherein the inserted EOS NAL units are arranged at an end of an access unit of the sequence in ascending order of layer IDs of the EOS NAL units.

15. The method of claim 13, wherein the inserted EOS NAL unit is not placed in a corresponding layer.

Images