Media file processing method and apparatus therefor

Abstract

According to one embodiment of the present application, there is provided a method of processing a media file by a media file processing apparatus. The method includes the steps of: recovering a sequence of access units for a first OLS; determining whether a first condition and a second condition are true; and based on the first condition and the second condition being true, inserting as many EOS NAL units as a number of layers applied to the first OLS, wherein the first condition is about whether an OLS different from the first OLS is selected after the sequence of access units, and the second condition is about whether the sequence of access units does not end with an EOS NAL unit or does not end with an EOB NAL unit in each layer of the first OLS.

Description

Technical Field

[0001] This disclosure relates to video / image coding technology, and more specifically, to methods and apparatus for processing media files for encoded image information in a video / image coding system. Background Technology

[0002] Recently, there has been a growing demand for high-resolution, high-quality images / videos (e.g., 4K, 8K, or more Ultra High Definition (UHD) video / images) across various fields. As video / image resolution or quality increases, a relatively large amount of information or bits is transmitted compared to conventional video / image data. Therefore, if video / image data is transmitted via media such as existing wired / wireless broadband lines or stored in traditional storage media, it can easily increase transmission and storage costs.

[0003] Furthermore, there is growing interest and demand for virtual reality (VR) and artificial reality (AR) content, as well as immersive media such as holograms; and the broadcasting of images / videos that present image / video characteristics different from those of actual videos / images, such as game images / videos, is also growing.

[0004] Therefore, efficient image compression technology is needed to effectively compress and send, store, or play high-resolution, high-quality video / images that exhibit the various characteristics described above. Summary of the Invention

[0005] Technical challenges

[0006] This disclosure provides a method and apparatus for improving video / image coding efficiency.

[0007] This disclosure also provides a method and apparatus for generating media files for encoded image information.

[0008] This disclosure also provides a method and apparatus for processing media files containing encoded image information.

[0009] Technical solution

[0010] According to embodiments of this document, a media file creation method performed by a media file creation device is provided. The method includes: generating a sequence of access units for a first output layer set (OLS); determining whether a first condition and a second condition are true; inserting as many sequence end (EOS) network abstraction layer (NAL) units as the number of layers applied to the first OLS, based on the first and second conditions being true; and generating a media file including the first OLS, wherein the first condition is whether an OLS different from the first OLS is selected after the sequence of access units, and wherein the second condition is whether the sequence of access units in each layer of the first OLS does not end with an EOS NAL unit or a bitstream end (EOB) NAL unit.

[0011] According to another embodiment of this document, a media file creation device is provided. This media file generation device includes a media file generator that generates a sequence of access units for a first output layer set (OLS), determines whether a first condition and a second condition are true, inserts as many sequence end (EOS) network abstraction layer (NAL) units as the number of layers applied to the first OLS based on the first and second conditions being true, and generates a media file including the first OLS, wherein the first condition is whether an OLS different from the first OLS is selected after the sequence of access units, and wherein the second condition is whether the sequence of access units in each layer of the first OLS does not end with an EOS NAL unit or a bitstream end (EOB) NAL unit.

[0012] According to another embodiment of this document, a media file processing method performed by a media file processing device is provided. The method includes: reconstructing a sequence of access units for a first output layer set (OLS); determining whether a first condition and a second condition are true; and, based on the first and second conditions being true, inserting an equal number of sequence end (EOS) network abstraction layer (NAL) units as applied to the first OLS, wherein the first condition is whether an OLS different from the first OLS is selected after the sequence of access units, and wherein the second condition is whether the sequence of access units in each layer of the first OLS does not end with an EOS NAL unit or a bitstream end (EOB) NAL unit.

[0013] According to another embodiment of this document, a media file processing apparatus is provided. A media file processing apparatus includes a receiver that acquires a media file; and a media file processor that reconstructs a sequence of access units for a first output layer set (OLS) of the media file, determines whether a first condition and a second condition are true, and, based on the first and second conditions being true, inserts as many sequence end (EOS) network abstraction layer (NAL) units as the number of layers applied to the first OLS, wherein the first condition is whether an OLS different from the first OLS is selected after the sequence of access units, and wherein the second condition is whether the sequence of access units in each layer of the first OLS does not end with an EOS NAL unit or a bitstream end (EOB) NAL unit.

[0014] According to another embodiment of this disclosure, a computer-readable digital storage medium is provided for storing a media file generated by a media file generation method. In the computer-readable digital storage medium, the method includes the following steps: configuring sub-image tracks, each including sub-image image data; configuring a base track referencing the sub-image tracks; and generating a media file including the sub-image tracks and the base track, wherein samples of the base track form rectangular regions without holes or overlaps, the holes indicating regions not covered by at least one sample of the referenced sub-image track, and the overlaps indicating regions covered twice or more by the referenced sub-image track.

[0015] Beneficial effects

[0016] According to this document, even changing only the maximum time ID can prevent unnecessary operations of inserting EOS NAL cells, thereby improving overall coding efficiency.

[0017] According to this document, the overhead that occurs when inserting EOS NAL units into the appropriate positions within access units can be prevented, thereby improving overall coding efficiency. Attached Figure Description

[0018] Figure 1 Examples of video / image coding systems applicable to embodiments of this disclosure are illustrated schematically.

[0019] Figure 2 This is a schematic diagram illustrating the configuration of a video / image encoding device to which embodiments of the present disclosure may be applied.

[0020] Figure 3 This is a schematic diagram illustrating the configuration of a video / image decoding device to which embodiments of the present disclosure may be applied.

[0021] Figure 4 An example illustrates a layer structure for encoded video / images.

[0022] Figure 5 and Figure 6 An example of a media file structure is illustrated.

[0023] Figure 7 An example of the overall operation of the DASH-based adaptive flow model is shown.

[0024] Figure 8 An exemplary embodiment is provided for a method of generating media files using the embodiments proposed in this disclosure.

[0025] Figure 9 An exemplary method is illustrated for decoding a media file generated by applying the embodiments proposed in this disclosure.

[0026] Figure 10 The method for generating media files using a device for generating media files according to the present disclosure is illustrated schematically.

[0027] Figure 11 An apparatus for generating media files according to the present disclosure is illustrated, the apparatus performing a method for generating media files.

[0028] Figure 12 The illustration illustrates a method for processing media files using a device for processing media files according to the present disclosure.

[0029] Figure 13 An apparatus for processing media files according to the present disclosure is illustrated, the apparatus performing a method for processing media files.

[0030] Figure 14 A structural diagram of a content streaming system applying this disclosure is illustrated. Detailed Implementation

[0031] This disclosure may be modified in various forms, and particular embodiments thereof will be described and illustrated in the accompanying drawings. However, the embodiments are not intended to limit this disclosure. The terminology used in the following description is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. Singular expressions include plural expressions, provided that they are not clearly interpreted differently. Terms such as “comprising” and “having” are intended to indicate the presence of features, numbers, steps, operations, elements, components or combinations thereof used in the following description, and therefore it should be understood that the possibility of having or adding one or more different features, numbers, steps, operations, elements, components or combinations thereof is not excluded.

[0032] Furthermore, for ease of illustration of different specific functions, the elements in the accompanying drawings described in this disclosure are drawn independently and do not imply that these elements are implemented by independent hardware or independent software. For example, two or more elements may be combined to form a single element, or a single element may be divided into multiple elements. Embodiments in which elements are combined and / or divided are part of this disclosure without departing from its concept.

[0033] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, throughout the drawings, the same reference numerals may be used to indicate the same elements, and similar descriptions of similar elements will be omitted.

[0034] Figure 1 Examples of video / image coding systems to which the implementation methods of this document can be applied are illustrated.

[0035] Reference Figure 1 A video / image encoding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device in the form of a file or stream via a digital storage medium or network.

[0036] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0037] Video sources can acquire video / images through processes that capture, synthesize, or generate video / images. Video sources may include video / image capture devices and / or video / image generation devices. For example, a video / image capture device may include one or more cameras, a video / image archive containing previously captured video / images, etc. For example, a video / image generation device may include a computer, tablet computer, and smartphone, and can generate video / images (electronically). For example, virtual video / images can be generated via a computer, etc. In this case, the video / image capture process can be replaced by a process that generates related data.

[0038] Encoding devices can encode input video / images. For compression and encoding efficiency, encoding devices can perform a series of processes such as prediction, transformation, and quantization. The encoded data (encoded video / image information) can be output as a bitstream.

[0039] The transmitter can transmit encoded video / image information or data, output in bitstream form, to the receiver of the receiving device in the form of a file or stream via digital storage media or a network. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter may include elements for generating media files according to a predetermined file format and may include elements for transmission over a broadcast / communication network. The receiver can receive / extract the bitstream and transmit the received bitstream to a decoding device.

[0040] Decoding devices can decode video / images by performing a series of processes such as inverse quantization, inverse transform, and prediction, which correspond to the operations of encoding devices.

[0041] The renderer can render the decoded video / images. The rendered video / images can be displayed on a monitor.

[0042] This disclosure relates to video / image coding. For example, the methods / implementations disclosed in this disclosure can be applied to methods disclosed in the Universal Video Coding (VVC) standard, the Basic Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second-generation Audio Video Coding (AVS2) standard, or next-generation video / image coding standards (e.g., H.267 or H.268).

[0043] This disclosure presents various implementations of video / image coding, and unless otherwise stated, these implementations may be combined with each other.

[0044] In this disclosure, video can refer to a series of images over time. An image generally refers to a unit representing an image within a specific time period, and sub-images / slices / tiles are units that constitute part of an image during encoding. Sub-images / slices / tiles may include one or more coding tree units (CTUs). An image may consist of one or more sub-images / slices / tiles. An image may consist of one or more tile groups. A tile group may include one or more tiles. A tile can represent a rectangular region of a CTU row within a tile in an image. A tile can be divided into multiple tiles, each tile consisting of one or more CTU rows within the tile. A tile that is not divided into multiple tiles can also be called a tile. A tile scan is a specific ordering of the CTUs used to segment the image, wherein the CTUs are sequentially ordered in the CTU raster scan within the tile, tiles within a tile are sequentially ordered in the raster scan of the tiles of the tile, and tiles in an image are sequentially ordered in the raster scan of the tiles of the image. Furthermore, a sub-image can represent a rectangular region of one or more slices within an image. In other words, a sub-image can contain one or more slices that collectively cover a rectangular area of ​​the image. A tile is a rectangular area of ​​a CTU within a specific tile column and a specific tile row in the image. A tile column is a rectangular area of ​​a CTU having a height equal to the height of the image and a width specified by a syntax element in the image parameter set. A tile row is a rectangular area of ​​a CTU having a height specified by a syntax element in the image parameter set and a width equal to the width of the image. A tile scan is a specific ordering of the CTUs that segment the image, where the CTUs are ordered consecutively in a CTU raster scan within a tile, and the tiles in the image are ordered consecutively in a tile raster scan within the image. A slice comprises an integer number of tiles of an image that can be specifically contained within a single NAL unit. A slice can consist of multiple complete tiles or a consecutive sequence of complete tiles of a single tile. Tile groups and slices can be used interchangeably in this disclosure. For example, in this disclosure, a tile group / tile group header can be referred to as a slice / slice header.

[0045] A pixel or cell (pel) can refer to the smallest unit that makes up a picture (or image). Additionally, "sample" can be used as the term corresponding to a pixel. A sample can typically represent a pixel or pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0046] A unit can represent a basic unit of image processing. A unit may include a specific region of an image and at least one of the information associated with that region. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "region". In general, an M×N block may include a set (or array) of samples (or sample arrays) or transform coefficients in M ​​columns and N rows.

[0047] In this disclosure, "A or B" may mean "A only", "B only", or "both A and B". In other words, in this disclosure, "A or B" may be interpreted as indicating "A and / or B". For example, in this disclosure, "A, B or C" may mean "A only", "B only", "C only", or "any combination of A, B, and C".

[0048] The forward slash " / " or comma used in this disclosure can mean "and / or". For example, "A / B" can mean "A and / or B". Therefore, "A / B" can mean "A only", "B only", or "both A and B". For example, "A, B, C" can mean "A, B, or C".

[0049] In this disclosure, "at least one of A and B" may mean "only A", "only B" or "both A and B". Furthermore, in this disclosure, the expression "at least one of A or B" or "at least one of A and / or B" may be interpreted as the same as "at least one of A and B".

[0050] Additionally, in this disclosure, "at least one of A, B, and C" may mean "A only", "B only", "C only" or "any combination of A, B, and C". Furthermore, "at least one of A, B, or C" or "at least one of A, B, and / or C" may mean "at least one of A, B, and C".

[0051] Furthermore, the parentheses used in this disclosure may mean "for example". Specifically, when indicating "prediction (intra-frame prediction)", "intra-frame prediction" may be proposed as an example of "prediction". In other words, "prediction" in this disclosure may not be limited to "intra-frame prediction", and "intra-frame prediction" may be proposed as an example of "prediction". Moreover, even when indicating "prediction (i.e., intra-frame prediction)", "intra-frame prediction" may be proposed as an example of "prediction".

[0052] The technical features described individually in one of the accompanying drawings of this disclosure can be implemented individually or simultaneously.

[0053] The following figures were created to illustrate specific examples of this disclosure. Since the names of specific devices or signals / messages / fields described in the figures are presented as examples, the technical features of this disclosure are not limited to the specific names used in the following figures.

[0054] Figure 2 This is a schematic diagram illustrating the configuration of a video / image encoding apparatus to which embodiments of the present disclosure may be applied. In the following, the video encoding apparatus may include an image encoding apparatus.

[0055] Reference Figure 2The encoding device 200 includes an image segmenter 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transform 232, a quantizer 233, an inverse quantizer 234, and an inverse transform 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstruction block generator. According to embodiments, the image segmenter 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). Additionally, the memory 270 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware component may also include the memory 270 as an internal / external component.

[0056] Image segmenter 210 can segment an input image (or picture or frame) input to encoding device 200 into one or more processing units. For example, a processing unit may be referred to as a coding unit (CU). In this case, the coding unit may be recursively segmented from a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree-binary-trinary tree (QTBTTT) structure. For example, a coding unit may be segmented into multiple deeper coding units based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure may be applied first, followed by a binary tree structure and / or a ternary structure. Alternatively, a binary tree structure may be applied first. The encoding process according to this disclosure may be performed based on the final coding unit that is no longer segmented. In this case, the maximum coding unit may be used as the final coding unit based on image characteristics, coding efficiency, etc., or, if necessary, the coding unit may be recursively segmented into deeper coding units, and the coding unit with the optimal size may be used as the final coding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes (described later). As another example, the processor may also include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be split or divided from the aforementioned final encoding unit. The prediction unit may be a unit for predicting samples, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving residual signals from transform coefficients.

[0057] In some cases, a unit can be used interchangeably with terms such as a block or region. Generally, an M×N block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can typically represent a pixel or pixel value, either representing only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component. A sample can be used as a term corresponding to a picture (or image) of pixels or cells.

[0058] In the encoding device 200, a residual signal (residual block, residual sample array) is generated by subtracting the prediction signal (prediction block, prediction sample array) output from the inter-frame predictor 221 or the intra-frame predictor 222 from the input image signal (original block, original sample array), and the generated residual signal is sent to the converter 232. In this case, as shown, the portion in the encoder 200 that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) can be referred to as the subtractor 231. The predictor can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a prediction block that includes the prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction based on the current block or CU. As described later in the description of the various prediction modes, the predictor can generate various types of information related to the prediction (e.g., prediction mode information) and send the generated information to the entropy encoder 240. The information about the prediction can be encoded in the entropy encoder 240 and output as a bitstream.

[0059] Intra-predictor 222 can predict the current block by referencing samples in the current image. Depending on the prediction mode, the referenced samples may be located near or separated from the current block. In intra-prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. For example, non-directional modes may include DC mode and planar mode. For example, depending on the level of detail in the prediction direction, the directional modes may include 33 or 65 directional prediction modes. However, this is just an example, and more or fewer directional prediction modes may be used depending on the settings. Intra-predictor 222 can determine the prediction mode to be applied to the current block by using the prediction modes applied to neighboring blocks.

[0060] Inter-frame predictor 221 can deduce the predicted block of the current block based on a reference block (reference sample array) specified by a motion vector on a reference image. Here, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a juxtaposed reference block, juxtaposed CU (colCU), etc., and the reference image including the temporally neighboring block may be referred to as a juxtaposed image (colPic). For example, inter-frame predictor 221 can configure a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to deduce the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 221 can use motion information of neighboring blocks as motion information for the current block. In skip mode, unlike merge mode, residual signals may not be sent. In motion vector prediction (MVP) mode, motion vectors of neighboring blocks can be used as motion vector predictors, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0061] Predictor 220 can generate a prediction signal based on various prediction methods described below. For example, the predictor can apply not only intra-frame prediction or inter-frame prediction to predict a block, but also both intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined intra-frame and inter-frame prediction (CIIP). Alternatively, the predictor can predict blocks based on an intra-block copy (IBC) prediction mode or a palette mode. The IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction in the current frame, but can be performed similarly to inter-frame prediction, such that a reference block is derived in the current frame. That is, IBC can use at least one inter-frame prediction technique described in this disclosure. The palette mode can be considered as an example of intra-frame coding or intra-frame prediction. When a palette mode is applied, sample values ​​within the frame can be signaled based on information about the palette table and palette index.

[0062] The predicted signal generated by the predictor (including inter-frame predictor 221 and / or intra-frame predictor 222) can be used to generate a reconstructed signal or a residual signal. Transformer 232 can generate transform coefficients by applying transform techniques to the residual signal. For example, the transform technique may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen–Loève Transform (KLT), Graphical Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to a transform obtained from a graphic when the relationship information between pixels is represented graphically. CNT refers to a transform generated based on the predicted signal generated using all previously reconstructed pixels. Furthermore, the transform processing can be applied to square pixel blocks of the same size or to blocks of variable size other than square.

[0063] Quantizer 233 quantizes the transform coefficients and sends them to entropy encoder 240, which encodes the quantized signal (information about the quantized transform coefficients) and outputs a bitstream. This information about the quantized transform coefficients can be referred to as residual information. Quantizer 233 can rearrange the block-type quantized transform coefficients into a one-dimensional vector based on the coefficient scan order, and generate information about the quantized transform coefficients based on this one-dimensional vector. Entropy encoder 240 can perform various encoding methods such as exponential Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). Entropy encoder 240 can encode information required for video / image reconstruction other than the quantized transform coefficients (e.g., values ​​of syntax elements, etc.) together or separately. The encoded information (e.g., encoded video / image information) can be sent or stored in NAL (Network Abstraction Layer) units as a bitstream. The video / image information may also include information about various parameter sets, such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). Additionally, the video / image information may include general constraint information. In this disclosure, information and / or syntax elements transmitted from the encoding device / notified by signal to the decoding device may be included in the video / image information. The video / image information may be encoded by the above-described encoding process and included in a bitstream. The bitstream may be transmitted via a network or stored in a digital storage medium. The network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that transmits the signal output from the entropy encoder 240 and / or a storage unit (not shown) that stores the signal may be included as an internal / external element of the encoding device 200, and alternatively, the transmitter may be included in the entropy encoder 240.

[0064] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying inverse quantization and inverse transform to the quantized transform coefficients via inverse quantizer 234 and inverse transformer 235. Adder 250 adds the reconstructed residual signal to the prediction signal output from inter-frame predictor 221 or intra-frame predictor 222 to generate a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array). If the block to be processed has no residual (such as in the case of applying a skip mode), the prediction block can be used as a reconstructed block. Adder 250 may be referred to as a reconstructor or reconstructed block generator. As described below, the generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image and can be filtered for inter-frame prediction of the next image.

[0065] In addition, luminance mapping with chroma scaling (LMCS) can be applied during image encoding and / or reconstruction.

[0066] Filter 260 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 260 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 270 (specifically, the DPB of memory 270). Various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. Filter 260 can generate various types of filtering-related information and send the generated information to entropy encoder 240, as described later in the description of the various filtering methods. The filtering-related information can be encoded by entropy encoder 240 and output as a bitstream.

[0067] The modified reconstructed image sent to memory 270 can be used as a reference image in inter-frame predictor 221. When inter-frame prediction is applied by the encoding device, prediction mismatch between the encoding device 200 and the decoding device can be avoided and encoding efficiency can be improved.

[0068] The DPB of memory 270 can store a modified reconstructed image used as a reference image in inter-frame predictor 221. Memory 270 can store motion information of blocks in the current image that derive (or encode) motion information and / or motion information of already reconstructed blocks in the image. The stored motion information can be sent to inter-frame predictor 221 and used as motion information for spatially or temporally neighboring blocks. Memory 270 can store reconstructed samples of reconstructed blocks in the current image and can transmit the reconstructed samples to intra-frame predictor 222.

[0069] Figure 3 This is a schematic diagram illustrating the configuration of a video / image decoding device to which embodiments of the present disclosure can be applied.

[0070] Reference Figure 3 The decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an intra-frame predictor 331 and an inter-frame predictor 332. The residual processor 320 may include an inverse quantizer 321 and an inverse transformer 322. According to embodiments, the entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 may be configured by hardware components (e.g., a decoder chipset or processor). Additionally, the memory 360 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware components may also include the memory 360 as an internal / external component.

[0071] When the input includes a bitstream containing video / image information, the decoding device 300 can reconstruct and... Figure 2 The encoding device processes video / image information corresponding to the image. For example, the decoding device 300 can deduce units / blocks based on block segmentation information obtained from the bitstream. The decoding device 300 can use a processor applied in the encoding device to perform decoding. Therefore, for example, the processor for decoding can be an encoding unit, and the encoding unit can be segmented from encoding tree units or maximum encoding units according to a quadtree structure, binary tree structure, and / or ternary tree structure. One or more transform units can be derived from the encoding units. The reconstructed image signal decoded and output by the decoding device 300 can be reproduced by a reproduction device.

[0072] Decoding device 300 can receive from Figure 2The encoding device outputs a signal in the form of a bitstream, and the received signal can be decoded by the entropy decoder 310. For example, the entropy decoder 310 can parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information may also include information about various parameter sets, such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). In addition, the video / image information may also include general constraint information. The decoding device can also decode the picture based on the information about the parameter sets and / or general constraint information. The information and / or syntax elements notified / received by signals, as described later in this disclosure, can be decoded and obtained from the bitstream through the decoding process. For example, the entropy decoder 310 decodes the information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, or CABAC, and outputs the quantized values ​​of the syntax elements and transform coefficients of the residuals required for image reconstruction. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using information about the target syntax element, decoding information about the target block, or information about symbols / bins decoded in a previous stage, and performs arithmetic decoding on the bins by predicting the probability of bin occurrence based on the determined context model, generating symbols corresponding to the values ​​of each syntax element. In this case, the CABAC entropy decoding method can update the context model after determining the context model by using the information of the decoded symbols / bins for the context model of the next symbol / bin. Information related to prediction from the information decoded by the entropy decoder 310 can be provided to the predictors (inter-frame predictor 332 and intra-frame predictor 331), and the residual values ​​of the entropy decoding performed in the entropy decoder 310 (i.e., quantized transform coefficients and related parameter information) can be input to the residual processor 320. The residual processor 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). Additionally, information about filtering from the information decoded by the entropy decoder 310 can be provided to the filter 350. Furthermore, a receiver (not shown) for receiving signals output from the encoding device may be configured as an internal / external component of the decoding device 300, or the receiver may be a component of the entropy decoder 310. Additionally, the decoding device according to this disclosure may be referred to as a video / image / picture decoding device, and the decoding device may be classified as an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoder 310, and the sample decoder may include at least one of an inverse quantizer 321, an inverse transformer 322, an adder 340, a filter 350, a memory 360, an inter-frame predictor 332, and an intra-frame predictor 331.

[0073] Dequantizer 321 can dequantize the quantized transform coefficients and output the transform coefficients. Dequantizer 321 can rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device. Dequantizer 321 can obtain the transform coefficients by performing dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information).

[0074] The inverse transformer 322 performs inverse transformation on the transformation coefficients to obtain the residual signal (residual block, residual sample array).

[0075] The predictor can perform prediction on the current block and generate a prediction block that includes prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on information about the prediction output from the entropy decoder 310, and can determine a specific intra-frame / inter-frame prediction mode.

[0076] Predictor 320 can generate a prediction signal based on various prediction methods. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but also apply intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined intra-frame and inter-frame prediction (CIIP). Alternatively, the predictor can predict blocks based on an intra-block copy (IBC) prediction mode or a palette mode. IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction in the current frame, but can be performed similarly to inter-frame prediction, such that a reference block is derived in the current frame. That is, IBC can use at least one inter-frame prediction technique described in this disclosure. Palette mode can be considered as an example of intra-frame coding or intra-frame prediction. When applying palette mode, sample values ​​within the frame can be signaled based on information about the palette table and palette index.

[0077] Intra-predictor 331 can predict the current block by referencing samples in the current image. Depending on the prediction mode, the referenced samples may be located near or separated from the current block. In intra-prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. Intra-predictor 331 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0078] Inter-frame predictor 332 can deduce the predicted block of the current block based on a reference block (reference sample array) specified by a motion vector on a reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. For example, inter-frame predictor 332 can configure a motion information candidate list based on neighboring blocks and deduce the motion vector and / or reference image index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and information about the prediction may include information indicating the inter-frame prediction mode of the current block.

[0079] Adder 340 generates a reconstruction signal (reconstructed image, reconstruction block, reconstruction sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (including inter-frame predictor 332 and / or intra-frame predictor 331). If the block to be processed has no residual (e.g., when a skip mode is applied), the prediction block can be used as the reconstruction block.

[0080] Adder 340 can be referred to as a reconstructor or reconstruction block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image, and can be output through filtering as described below, or it can be used for inter-frame prediction of the next image.

[0081] In addition, Luminance Mapping with Chroma Scaling (LMCS) can be applied in image decoding processing.

[0082] Filter 350 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 360 (specifically, the DPB of memory 360). For example, various filtering methods may include deblocking filtering, sample adaptive shifting, adaptive loop filtering, bilateral filtering, etc.

[0083] The (modified) reconstructed image stored in the DPB of memory 360 can be used as a reference image in inter-frame predictor 332. Memory 360 can store motion information of blocks in the current image from which motion information is derived (or decoded) and / or motion information of already reconstructed blocks in the image. The stored motion information can be sent to inter-frame predictor 332 as motion information of spatially or temporally neighboring blocks. Memory 360 can store reconstructed samples of reconstructed blocks in the current image and transmit the reconstructed samples to intra-frame predictor 331.

[0084] In this disclosure, the embodiments described in the filter 260, inter-frame predictor 221, and intra-frame predictor 222 of the encoding device 200 can be applied in the same way as or respectively corresponding to the filter 350, inter-frame predictor 332, and intra-frame predictor 331 of the decoding device 300. This can also be applied to unit 332 and intra-frame predictor 331.

[0085] Figure 4 An example illustrates the layer structure of an encoded video / image.

[0086] Reference Figure 4 The encoded images / videos are divided into a Video Coding Layer (VCL), a lower-level system, and a Network Abstraction Layer (NAL). The VCL performs the decoding process of the images / videos and handles the decoding process. The lower-level system sends and stores the encoded information. The Network Abstraction Layer (NAL) exists between the VCL and the lower-level system and is used to perform network adaptation functions.

[0087] For example, VCL data can be generated in VCL, which includes compressed image data (slice data), or picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS), or parameter sets that include supplementary enhancement information (SEI) messages required for the image decoding process.

[0088] For example, in NAL, header information (NAL cell data) is added to the raw byte sequence payload (RSRP) generated in VCL to create the NAL cell. In this case, the slice data, parameter set, SEI message, etc., generated in VCL can be referred to as RBSP. The NAL cell header may include NAL cell type information specified according to the RSRP data included in the corresponding NAL cell.

[0089] For example, such as Figure 4 As shown, NAL units can be classified into VCL NAL units and non-VCL NAL units based on the RSRP generated in the VCL. A VCL NAL unit can refer to a NAL unit that includes information about the information (slice data), while a non-VCL NAL unit can refer to a NAL unit that includes information required for decoding the image (parameter set or SEI message).

[0090] VCL NAL units and non-VCL NAL units can be sent over a network, with header information added according to the subsystem's data standard. For example, NAL units can be converted into predetermined standard data formats such as H.266 / VVC file format, Real-time Transport Protocol (RTP), Transport Streaming (TS), etc., and transmitted over various networks.

[0091] Furthermore, as described above, regarding NAL cells, the NAL cell type can be specified according to the RBSP data structure included in the corresponding NAL cell, and information about the NAL cell type can be stored in the NAL cell header and notified by a signal.

[0092] For example, NAL units can be classified into VCL NAL unit types and non-VCL NAL unit types based on whether the NAL unit includes information about the image (slice data). Furthermore, VCL NAL unit types can be classified based on the nature and type of the image included in the VCL NAL unit, and non-VCL NAL units can be classified based on the type of parameter set.

[0093] The following is an example of a NAL unit type specified based on the type of the parameter set included in a non-VCL NAL unit type.

[0094] - Adaptive Parameter Set (APS) NAL Units: Types of NAL units that include the APS.

[0095] - Decoding Parameter Set (DPS) NAL Unit: The type of NAL unit that includes the DPS.

[0096] - Video Parameter Set (VPS) NAL Unit: Includes the type of NAL unit for the VPS.

[0097] - Sequence Parameter Set (SPS) NAL Units: Types of NAL units that include SPS.

[0098] - Picture Parameter Set (PPS) NAL Unit: The type of NAL unit that includes the PPS.

[0099] - Picture Header (PH) NAL Unit: Includes the type of NAL unit for the PH.

[0100] The aforementioned NAL unit type can have syntax information for the NAL unit type, and this syntax information can be stored in the NAL unit header and signaled. For example, the syntax information can be nal_unit_type, and the NAL unit type can be specified as the value of nal_unit_type.

[0101] Furthermore, an image can include multiple slices, and each slice can include a slice header and slice data. In this case, an image header can be added for multiple slices (a set of slice headers and slice data). The image header (image header syntax) can include information / parameters that are typically applied to an image. The slice header (slice header syntax) can include information / parameters that are typically applied to a slice. APS (APS syntax) or PPS (PPS syntax) can include information / parameters that are typically applied to one or more slices or images. SPS (SPS syntax) can include information / parameters that are typically applied to one or more sequences. VPS (VPS syntax) can include information / parameters that are typically applied to multiple layers. DPS (DPS syntax) can include information / parameters that are typically applied to the entire image. DPS can include information / parameters related to the concatenation of coded video sequences (CVS).

[0102] In this disclosure, image / video information encoded from an encoding device to a decoding device and signaled in the form of a bitstream may include intra-picture segmentation related information, intra-frame / inter-frame prediction information, inter-layer prediction related information, residual information, and intra-loop filtering information, and may include information included in the APS, information included in the PPS, information included in the SPS, information included in the VPS, and / or information included in the DPS. Furthermore, the image / video information may also include information from the NAL unit header.

[0103] Furthermore, the encoded image / video information described above can be configured based on the media file format to facilitate the generation of media files. For example, the encoded image / video information can be used to form a media file (segment) based on one or more NAL units / sample entries of the encoded image / video information. The media file may include sample entries and tracks. For example, the media file (segment) may include various records, and each record includes information related to the image / video or information related to the media file format. Additionally, for example, one or more NAL units may be stored in the configuration record (or decoder configuration record or VVC decoder configuration record) field of the media file. Here, fields can also be referred to as syntax elements.

[0104] For example, the ISO Basic Media File Format (ISOBMFF) can be used as a media file format to which the methods / implementations disclosed in this disclosure can be applied. ISOBMFF can be used based on many codec encapsulation formats (e.g., VC file format, HEVC file format, and / or VVC file format) and many multimedia container formats (e.g., MPEG-4 file format, 3GPP file format (3GP), and / or DVB file format). Furthermore, in addition to continuous media such as audio and video, static media such as images and metadata can be stored in files according to ISOBMFF. File structuring according to ISOBMFF can be used for various purposes, including local media file playback, progressive download of remote files, segmentation (DASH) for dynamic adaptive streaming over HTTP, containerization and packetization instructions for content to be streamed, recording of received real-time media streams, etc.

[0105] The 'box' described below can be a basic syntax element of ISOBMFF. An ISOBMFF file can consist of a series of boxes, and another box can be included within a box. For example, a movie box (a box with grouping type 'moov') can include metadata for a continuous media stream that includes the media file, and each stream can be displayed as a track in the file. Metadata can be included in track boxes (boxes with grouping type 'trak'), and the media content of a track can be included in media data boxes (boxes with grouping type 'mdat') or directly in a separate file. The media content of a track can consist of a sequence of samples such as audio or video access units. For example, ISOBMFF can specify tracks of types such as media tracks that include basic media streams, cue tracks that include media transmission instructions or indicate received packet streams, and timing metadata that includes time synchronization metadata.

[0106] Furthermore, while ISOBMFF is designed for storage purposes, it is also very useful for streaming (e.g., progressive downloads or DASH). Movie clips defined in ISOBMFF can be used for streaming purposes. Fragmented ISOBMFF files can be represented by, for example, two tracks associated with video and audio. For example, when random access is included after receiving a 'moov' box, all movie clips 'moof' can be decoded along with their associated media data.

[0107] In addition, the metadata for each track may include a list of sample description entries that specify the encoding or encapsulation format used for the track and provide the initialization data required to process the corresponding format. Furthermore, each sample may be cascaded to one of the sample description entries for the track.

[0108] When using ISOBMFF, sample-specific metadata can be specified through various mechanisms. Specific boxes within a sample table frame (a box with grouping type 'stb1') can be standardized to meet general requirements. For example, a synchronization sample frame (a box with grouping type 'stss') can be used to list randomly accessed samples. When using sample grouping mechanisms, samples can be mapped based on a four-character grouping type by sharing sample groups that describe entries for the same sample group. Various grouping types can be specified in ISOBMFF.

[0109] Figure 5 and Figure 6 An example of a media file structure is shown.

[0110] A media file may include at least one frame. Here, a frame may be a data block or object that includes media data or metadata associated with the media data. The frames may be in a hierarchical structure, thus allowing for data categorization, and the media file may have a format suitable for storing and / or transmitting large volumes of media data. Furthermore, the media file may have a structure that allows users to easily access media information such as navigating to specific media content points.

[0111] Media files may include ftyp boxes, moov boxes, and / or mdat boxes.

[0112] The ftyp box (file type box) provides information about the file type or compatibility of the corresponding media file. The ftyp box can include configuration version information about the media data of the corresponding media file. Decoders can refer to the ftyp box to identify the corresponding media file.

[0113] A moov frame (movie frame) can be a frame that includes metadata about the media data of the corresponding media file. A moov frame can serve as a container for all metadata. The moov frame can be the highest-level frame associated with metadata. Depending on the implementation, only one moov frame may exist in a media file.

[0114] A mdat frame (media data frame) can be a frame containing the actual media data of a corresponding media file. Media data can include audio samples and / or video samples. The mdat frame can be used as a container to hold such media samples.

[0115] According to the implementation method, the above-mentioned moov box may also include the mvhd box, trak box and / or mvex box as lower-level boxes.

[0116] The MVHD header frame can include information related to the media presentation of the media data included in the corresponding media file. In other words, the MVHD header frame can include information such as the media generation time, change time, time standard, and time period of the corresponding media presentation.

[0117] A trak box provides information about the corresponding media track. A trak box can include information such as streaming-related information, rendering-related information, and access-related information about audio or video tracks. Multiple trak boxes can exist depending on the number of tracks.

[0118] The `trak` box can also include a `tkhd` box (track header box) as a lower-level box. The `tkhd` box can include information about the track indicated by the `trak` box. The `tkhd` box can include information such as the creation time, change time, and track identifier of the corresponding track.

[0119] The mvex box (movie extension box) can indicate that the corresponding media file may have a moof box, which will be described later. Scanning the moof boxes may be necessary to identify all media samples for a specific track.

[0120] According to one implementation, a media file can be divided into multiple segments (500). Therefore, the media file can be fragmented and stored or transmitted. The media data (mdat frames) of the media file can be divided into multiple segments, each segment including a moof frame and the divided mdat frames. According to one implementation, information from ftyp frames and / or moov frames may be required to use the segment.

[0121] A moof (movie clip frame) can provide metadata about the media data of the corresponding clip. The moof can be the highest-level frame among the frames associated with the metadata of the corresponding clip.

[0122] The mdat frame (media data frame) can include the actual media data as described above. The mdat frame can include media samples corresponding to the media data of each segment it corresponds to.

[0123] According to the implementation method, the above-mentioned moof box may also include an mfhd box and / or a traf box as a lower-level box.

[0124] The MFHD (Movie Clip Header) box can include information about the relationships between the segmented clips. The MFHD box can indicate the order of the media data after segmentation by including sequence numbers. Furthermore, the MFHD box can be used to check for missing data in the segmented data.

[0125] A TRAF frame (track segment frame) can contain information about the corresponding track segment. A TRAF frame can provide metadata about the segmented track segments included in the corresponding segment. The TRAF frame provides metadata that allows media samples within the corresponding track segment to be decoded / reproduced. Depending on the number of track segments, multiple TRAF frames can exist.

[0126] According to the implementation method, the above-mentioned traf box may also include a tfhd box and / or a trun box as a lower-level box.

[0127] The TFHD frame (track segment header frame) can include header information for the corresponding track segment. The TFHD frame can provide information such as the basic sample size, time period, offset, and identifier of the media sample for the track segment indicated by the TRAFF frame mentioned above.

[0128] The trun box can include information related to the corresponding track segment. The trun box can include information such as the time period, size, and playback time of each media sample.

[0129] The aforementioned media files and their segments can be processed into segments and sent. Segmentation may include initialization segments and / or media segments.

[0130] In addition to media data, the file in the illustrated embodiment (510) may also include information related to media decoder initialization. For example, the file may correspond to the aforementioned initialization segment. The initialization segment may include the aforementioned ftyp box and / or moov box.

[0131] The file in the illustrated embodiment (520) may include the aforementioned segments. For example, the file may correspond to the aforementioned media segments. The media segments may also include styp boxes and / or sidx boxes.

[0132] A styp box (segment type box) can provide information for identifying media data after segmentation. A styp box can be used as the aforementioned ftyp box for segmented data. In some implementations, the styp box can have the same format as the ftyp box.

[0133] The sidx box (segment index box) can provide information indicating the index of the segment after division. Therefore, the order of the segmented parts can be indicated.

[0134] According to implementation (530), an ssix box may also be included. The ssix box (subsegment index box) can provide information indicating the index of the subsegment when a segment is divided into subsegments.

[0135] The boxes in the media file may include further extended information based on boxes or FullBoxes, as shown in the illustrated embodiment (550). In this embodiment, the size and largesize fields may represent the length of the corresponding box in bytes. The version field may indicate the version of the corresponding box format. The type field may indicate the type or identifier of the corresponding box. The flag field may indicate a flag associated with the corresponding box.

[0136] Furthermore, according to this disclosure, the fields (properties) of the video / image can be forwarded while being included in a DASH-based adaptive streaming model.

[0137] Figure 7 A diagram illustrating the overall operation of the DASH-based adaptive streaming model is provided. The DASH-based adaptive streaming model according to the illustrated embodiment (700) illustrates the operation between an HTTP server and a DASH client. Here, Dynamic Adaptive Streaming over HTTP (DASH) is a protocol used to support HTTP-based adaptive streaming and can dynamically support streaming based on network conditions. As a result, AV content can be reproduced without interruption.

[0138] First, the DASH client can obtain the MPD. The MPD can be delivered from a service provider such as an HTTP server. The DASH client can use the segment access information described in the MPD to request the segment from the server. Here, this request can be performed taking network conditions into account.

[0139] After acquiring the segments, the DASH client can process them in the media engine and display them on the screen. The DASH client can request and acquire necessary segments in real time, taking into account playback time and / or network conditions (adaptive streaming). As a result, content can be played back without interruption.

[0140] A Media Presentation Description (MPD) is a file that includes detailed information about segments that enables DASH clients to dynamically retrieve them, and can be expressed in XML format.

[0141] The DASH client controller can generate commands for requesting MPD and / or segments, taking network conditions into account. Additionally, the controller can control how the acquired information is used in internal blocks, such as those of the media engine.

[0142] The MPD parser can parse the acquired MPD in real time. Therefore, the DASH client controller can generate commands to obtain the required segments.

[0143] The segment parser can parse the acquired segments in real time. Based on the information included in the segments, including the media engine's internal blocks, specific operations can be performed.

[0144] An HTTP client can request the required MPD and / or segments from an HTTP server. Alternatively, the HTTP client can pass the MPD and / or segments obtained from the server to an MPD parser or a segment parser.

[0145] The media engine can use media data included in segments to display content on the screen. Here, multiple pieces of information from the MPD can be used.

[0146] The DASH data model can have a hierarchical structure (710). Media presentation can be described by MPD. MPD can describe the time series of multiple time periods that form the media presentation. A time period can represent a part of the media content.

[0147] Within a single section, multiple data items can be included in an adapter set. An adapter set can be a collection of multiple media content components that can be exchanged with each other. An adapter set can include a collection of representations. Representations can correspond to media content components. Within a representation, content can be divided into multiple segments over time, which can be used for appropriate access and delivery. A URL for each segment can be provided to facilitate access to each segment.

[0148] MPD can provide information related to media presentation, and the time period element, adaptation set element, and representation element can describe the time period, adaptation set, and representation, respectively. Representations can be divided into sub-representations, and sub-representation elements can describe the corresponding sub-representations.

[0149] Public properties / elements can be defined, which can be applied to (including) adaptation sets, representations, sub-representations, etc. Basic properties and / or supplementary properties can exist within public properties / elements.

[0150] The basic properties may include information that is considered necessary for processing media presentation-related data. The supplementary properties may include information that can be used to process media presentation-related data. When transmitted via MPD, the descriptors described in the following embodiments may be defined in the basic properties and / or supplementary properties and transmitted.

[0151] Furthermore, the 'sample' described below can refer to a single element of one of the three sample arrays (Y, Cb, and Cr) of an image, or all the data associated with a single time. For example, when the term 'sample' is used in the context of a track (in a media file format), 'sample' can mean all the data associated with a single time of the corresponding track. Here, time can be decoding time or synthesis time. Additionally, for example, when the term 'sample' is used in the context of an image—that is, when 'sample' is used as a term for an image such as a 'luminance sample'—'sample' can refer to a single element of one of the three sample arrays of an image.

[0152] To store VVC content, three types of basic streams can be defined as follows: - A video base stream containing VCL NAL units but not parameter sets, DCI, or OPINAL units. Here, all parameter sets, DCI, and OPINAL units can be stored in one or more sample entries.

[0153] - A video and parameter set basic stream containing VCL NAL units, which may contain parameter sets, DCI or OPINAL units, and has one or more sample entries storing parameter sets, DCI or OPINAL units.

[0154] - A non-VCL elementary stream containing non-VCL NAL units synchronized with the elementary stream contained in the video track. Here, the VVC non-VCL track does not contain parameter sets, DCI, or OPI NAL units from sample entries.

[0155] Meanwhile, VVC video streams can be defined as follows.

[0156] For example, a VVC sample entry can be defined as a sample entry whose sample entry type is "vvc1" or "vvi1". Each sample entry in a VVC track can be a VVC sample entry. A VVC sample entry can include a VVC configuration box as defined below. A VVC configuration box can include a VVC decoder configuration record containing decoder configuration information.

[0157] Additionally, an optional BitRateBox can exist in the VVC sample entry to signal the bitrate information of the VVC video stream. Furthermore, when used in MPEG-4, an extended descriptor may also exist that must be inserted into the basic stream descriptor.

[0158] In addition, as permitted by the ISO Basic Media File Format specification, multiple sample entries can be used to display sections of video using different configurations or parameter sets.

[0159] If a VVC subpicture track can be consumed without another VVC subpicture track and contains a consistent VVC bitstream, then a regular VVC sample entry (“vvc1” or “vvi1”) can be used for the VVC subpicture track. Otherwise, a “vvs1” sample entry is used for the VVC subpicture track and the following constraints can be applied to that track.

[0160] The track_in_movie flag is equal to 0.

[0161] - Each track contains only one sample entry.

[0162] - A track is referenced by one or more VVC base tracks via a "subp" track reference.

[0163] - DCI, OPI, VPS, SPS, PPS, AUD, PH, EOS, EOB and other AU or picture-level non-VCL NAL units must not appear in either the sample entry or the sample of the "vvs1" track.

[0164] - Unless otherwise specified, sub-frames of video sample entries (e.g., CleanApertureBox and PixelAspectRatioBox) must not appear in the sample entry and are ignored if they do.

[0165] - A sample will not be marked as a synchronous sample unless all VCL NAL units included in the sample meet the synchronous sample requirements.

[0166] - Synthesis time offset information for samples in track "vvs1" is not available.

[0167] – Subsample information may exist for samples in the “vvs1” orbital. If subsample information exists, it must conform to the definition of a subsample for VVC.

[0168] Additionally, a VVC track can include a "subp" track reference and an entry containing either the track_ID value of the VVC subpicture track or the track_group_id value of the "alte" track group of the VVC subpicture track. Sample groups of type "spor" can reside within each VVC base track. The dedicated sample entry type for a VVC base track can be indicated by the codec MIME parameter of the track type.

[0169] Additionally, sample entries of type "vvs1" can include VvcNALUConfigBox. If a VVC subpicture track is referenced by a VVC base track containing a "spor" sample group description entry with subpic_id_info_flag equal to 1, the VVC subpicture track may potentially include a subpicture ID sample group description using the default sample grouping mechanism.

[0170] Additionally, if the sample entry name is "vvc1" or "vvi1", the stream to which this sample entry is applied can be a compatible VVC stream, as seen by the VVC decoder operating on the configuration (including profile, hierarchy, and level) provided in the VVCConfigurationBox.

[0171] Additionally, if the sample entry name is "vvc1", the value of array_completeness can be equal to 1 for arrays of DCI, VPS, SPS, and PPS NAL cells, and equal to 0 for all other arrays. Conversely, if the sample entry name is "vvi1", the value of array_completeness may be equal to 0 for all arrays.

[0172] Additionally, if a track does not contain a VVC bitstream by default and does not indicate a VVC bitstream after checking the "subp" and "vvcN" track references (if present), then the track may contain a track carrying a "vopi" sample group or an "oref" track reference for an operation point entity group. For example, if a single-layer VVC bitstream contains two time sub-layers stored in different tracks, the track containing the sub-layer with a TemporalId of 1 may contain an "oref" track reference to the track containing the sub-layer with a TemporalId of 0.

[0173] Simultaneously, the operation point information for the ISO Basic Media File Format (ISOBMF) used in VVC can be transmitted as samples within a group frame with grouping type "vopi" or an entity group with grouping type "opeg". Operation point information may be required to identify the sample and sample entry for each operation point.

[0174] Additionally, an operation point can be a temporal subset of the Output Layer Set (OLS), which can be identified using the OLS index and the highest value of TemporalId. Each operation point can be associated with a profile, hierarchy, and level (i.e., PTL) that defines the consistency point of the operation point.

[0175] Additionally, for example, an operation point entity group can be defined to provide track mapping and profile-level information for operation points. If an "opeg" entity group exists, the file may contain a file-level ExtendedTypeBox with a brand combination including the "unif" brand specified in ISO / IEC 14496-12.

[0176] When aggregating track samples mapped to the operands described in the operand entity group, the implicit reconstruction process does not require removing any additional VCL NAL units to obtain a consistent VVC bitstream. Tracks belonging to the operand entity group can have track references of type "oref" for the group_id indicated in the operand entity group, and may not include the "vopi" sample group.

[0177] Additionally, all entity_id values ​​included in the operation point entity group must indicate the track ID of a track belonging to the same VVC bitstream. If present, the OperatingPointGroupBox is included in the GroupsListBox of the file-level MetaBox and may not be included in other file-level MetaBoxes. Here, the OperatingPointGroupBox can represent an operation point entity group.

[0178] The syntax for the above operation point entity group can be shown in the table below.

[0179] Table 1

[0180] In addition, the semantics of the syntax for operation point entity groups can be seen in the following table.

[0181] Table 2

[0182] Additionally, for example, media files may include VVC bitstream entity groups. A VVC bitstream entity group can be defined as a VVC track that provides a single VVC elementary stream.

[0183] For example, if there exists an EntityToGroupBox with grouping_type "vvcb", the following constraints can be applied.

[0184] 1) EntityToGroupBox must be included in the GroupsListBox of the file-level MetaBox and not in any other level MetaBox.

[0185] 2) The entity_id value included in EntityToGroupBox represents the track ID of all VVC tracks that carry a VVC basic flow.

[0186] 3) The first entity_id value included in EntityToGroupBox represents the track ID of the VVC track that carries the “vopi” sample group.

[0187] 4) VVC tracks with a track ID value indicated by a second or later entity_id value contained in EntityToGroupBox do not carry “vopi” sample groups.

[0188] Furthermore, for example, media files may include decoder configuration information about the image / video content. That is, media files may include VVC decoder configuration records that include decoder configuration information.

[0189] If the VVC decoder configuration record is stored in a sample entry, the VVC decoder configuration record may include the size of a length field used in each sample to indicate the length of the NAL unit included in the VVC decoder configuration record, as well as the parameter set, DCI, OPI, and SEI NAL units. The VVC decoder configuration record may be externally framed (the size of the VVC decoder configuration record is provided in the structure containing the VVC decoder configuration record).

[0190] Furthermore, the VVC decoder configuration record may also include a version field. The version in this disclosure can define version 1 of the VVC decoder configuration record. Changes incompatible with the VVC decoder configuration record can be indicated as changes to the version number. When the version number is not recognized, the reader does not need to decode the VVC decoder configuration record or the stream to which the record is applied.

[0191] Compatible extensions to the VVC decoder configuration record can be made without changing the configuration version code. The reader needs to be prepared to ignore unrecognized data that exceeds the data definition understood by the reader.

[0192] If a track primarily contains a VVC bitstream or is resolved via a "subp" track reference, a VvcPtlRecord must be present in the decoder configuration record. In this case, the specific set of output layers for the VVC bitstream can be indicated by the output_layer_set_idx field. Additionally, if the ptl_present_flag in the decoder configuration record for a track is 0, then the track must have an "oref" track reference that can reference the ID of a VVC track or an "opeg" entity group.

[0193] When decoding the stream described in the VVC decoder configuration record, the values ​​of the syntax elements VvcPTLRecord, chroma_format_idc, and bit_depth_minus8 are valid for all referenced parameter sets. In particular, the following restrictions may apply.

[0194] - The profile indicator general_profile_idc indicates the profile that OLS conforms to, identified by output_layer_set_idx in the configuration record.

[0195] - The tier indicator general_tier_flag can indicate the tier of the highest tier indicated in the profile_tier_level() syntax structure (all parameter sets) that conforms to OLS, which is greater than or equal to the tier of the highest tier indicated by the tier, identified by the output_layer_set_idx in the configuration record.

[0196] - Each bit of General_constraint_info can only be set if the bit is set in all General_constraints_info() syntax structures in the profile_tier_level() syntax structure (all parameter sets) that OLS conforms to, and is identified by output_layer_set_idx in the configuration record.

[0197] - The level indicator general_level_idc can indicate the level of capability that is equal to or greater than the highest level of the profile_tier_level() syntax structure (all parameter sets) followed by OLS, as identified by output_layer_set_idx in the configuration record.

[0198] Additionally, the following constraints can be applied to chroma_format_idc.

[0199] - If the VVC stream of the application configuration record is a single-layer bit stream, the value of sps_chroma_format_idc as defined in ISO / IEC 23090-3 must be the same in all SPSs referenced by the VCL NAL unit of the sample describing the application by the current sample entry, and chroma_format_idc must be the same as sps_chroma_format_idc.

[0200] - Otherwise (if the VVC stream used for configuration recording is a multi-layer bitstream), the value of vps_ols_dpb_chroma_format[MultiLayerOlsIdx[output_layer_set_idx]] as defined in ISO / IEC 23090-3 must be the same for all CVSs applied for the current sample entry description, and chroma_format_idc must be the same as the value of vps_ols_dpb_chroma_format[MultiLayerOlsIdx[output_layer_set_idx]].

[0201] Additionally, the following constraints can be applied to bit_depth_minus8.

[0202] - If the VVC stream of the application configuration record is a single-layer bit stream, the value of sps_bitdepth_minus8 as defined in ISO / IEC 23090-3 must be the same in all SPSs referenced by the VCL NAL cell of the sample describing the application by the current sample entry, and the value of bit_depth_minus8 must be the same as the value of sps_bitdepth_minus8.

[0203] - Otherwise (if the VVC stream to which the configuration record is applied is a multi-layer bitstream), the value of vps_ols_dpb_bitdepth_minus8[MultiLayerOlsIdx[output_layer_set_idx]] as defined in ISO / IEC 23090-3 must be the same for all CVSs to which the current sample entry description is applied, and the value of bit_depth_minus8 must be the same as the value of vps_ols_dpb_bitdepth_minus8[MultiLayerOlsIdx[output_layer_set_idx]].

[0204] Additionally, the following constraints can be applied to picture_width.

[0205] - If the VVC stream of the application configuration record is a single-layer bitstream, then as defined in ISO / IEC 23090-3, the value of sps_pic_width_max_in_luma_samples must be the same in all SPS referenced by the VCL NAL cell of the sample described by the current sample entry of the application, and the value of picture_width must be the same as the value of sps_pic_width_max_in_luma_samples.

[0206] - Otherwise (if the VVC stream used for configuration recording is a multi-layer bitstream), the value of vps_ols_dpb_pic_width[MultiLayerOlsIdx[output_layer_set_idx]] as defined in ISO / IEC 23090-3 must be the same for all CVSs applied to the current sample entry description, and the value of picture_width must be the same as the value of vps_ols_dpb_pic_width[MultiLayerOlsIdx[output_layer_set_idx]].

[0207] Additionally, the following constraints can be applied to picture_height.

[0208] - If the VVC stream of the application configuration record is a single-layer bit stream, then as defined in ISO / IEC 23090-3, the value of sps_pic_height_max_in_luma_samples must be the same in all SPS referenced by the VCL NAL cell of the sample described by the current sample entry of the application, and the value of picture_height must be the same as the value of sps_pic_height_max_in_luma_samples.

[0209] - Otherwise (if the VVC stream used for configuration recording is a multi-layer bitstream), the value of vps_ols_dpb_pic_height[MultiLayerOlsIdx[output_layer_set_idx]] as defined in ISO / IEC 23090-3 must be the same for all CVSs used for the current sample entry description, and the value of picture_height must be the same as the value of vps_ols_dpb_pic_height[MultiLayerOlsIdx[output_layer_set_idx]].

[0210] It can provide explicit indications of chroma format and bit depth, as well as other important format information used in the VVC video base stream, in the VVC decoder configuration record. If the color space or bit depth indications in the VUI information of two sequences are different, two different VVC sample entries may be required.

[0211] Additionally, the VVC decoder configuration record can have a set of arrays carrying initial NAL units. NAL unit types can be limited to those representing only DCI, OPI, VPS, SPS, PPS, prefix APS, and prefix SEI NAL units. For NAL unit types reserved in ISO / IEC 23090-3, and NAL unit types that may be defined in this document in the future, the reader may need to ignore arrays with reserved or disallowed NAL unit type values.

[0212] Meanwhile, the array can be configured in the following order: DCI, OPI, VPS, SPS, PPS, prefix APS, and prefix SEI.

[0213] The syntax for the VVC decoder configuration record described above can be shown in the table below.

[0214] Table 3

[0215] Additionally, the semantics of the syntax used for VVC decoder configuration records can be seen in the following table.

[0216] Table 4

[0217] Additionally, the following types of tracks are defined according to the VVC file format of this disclosure.

[0218] a) VVC Track: A VVC track represents a VVC bitstream by including NAL units in the samples and sample entries, and possibly by referencing other VVC tracks that include other sub-layers of the VVC bitstream, and possibly by referencing VVC subpicture tracks. When a VVC track is called a VVC subpicture track, it is referred to as a VVC base track.

[0219] b) VVC non-VCL track: An APS carrying ALF, LMCS, or scaling list parameters, and other non-VCL NAL units can be stored in the VVC non-VCL track, which can be transmitted via a track separate from the track containing the VCL NAL units.

[0220] c) VVC sub-picture track: A VVC sub-picture track may include any of the following.

[0221] For example, a VVC subpicture track can include a sequence of one or more VVC subpictures.

[0222] For example, a VVC sub-picture track can include one or more complete slices that form a rectangular region.

[0223] A sample of a VVC sub-image track may include any of the following.

[0224] For example, a sample of a VVC subpicture track may include one or more complete subpictures that are consecutive in decoding order as specified in ISO / IEC 23090-3.

[0225] For example, samples of VVC sub-picture tracks may include one or more complete slices that form rectangular regions and are consecutive in the decoding order, as specified in ISO / IEC 23090-3.

[0226] VVC subpicks, or slices of random samples included in VVC subpick tracks, can be consecutive in the decoding order.

[0227] The VVC non-VCL track and VVC sub-picture track enable VVC video to be delivered within streaming applications in the following preferred manner.

[0228] For example, tracks can be carried separately via their independent DASH representations, and a subset of DASH representations including VVC sub-picture tracks and a subset of DASH representations including non-VCL tracks may be required for decoding and rendering. This method can eliminate redundant transmissions of APS and other non-VCL NAL units.

[0229] Simultaneously, it can perform data sharing and VVC bitstream reconstruction processes as follows. Here, the output of this process can be a VVC bitstream, which can be referred to as the output bitstream.

[0230] For example, a file reader may need to invoke the process if one or more of the following conditions are met: i) There are VVC bitstream (“vvcb”) entity groups in the file, and the file reader processes the VVC bitstream represented by the entity groups to generate the output bitstream.

[0231] ii) There is an oppoint (“opeg”) entity group in the file, and the file reader uses any oppoint described by the entity group to generate the output bit stream.

[0232] iii) The file reader extracts a subset of layers or sublayers of VVC tracks where ptl_present_flag is 1 to generate an output bitstream.

[0233] iv) The file reader handles the VVC default track.

[0234] v) The file reader processes VVC tracks with associated VVC non-VCL tracks.

[0235] For example, the process can consist of a sequence of the following steps.

[0236] 1) If conditions i, ii, or iii mentioned above are true, the operation point is determined at the beginning of the VVC bitstream, and the operation point can be determined again for any IRAP or GDR access unit.

[0237] If the file reader selects an operation point for the first time or selects an operation point different from the previously selected operation point, the file reader may need to indicate the output layer set index and the highest TemporalId value of the selected operation point to the VVC decoder by including it in the OPI NAL unit inserted into the output bitstream (from the first access unit using the operation point (if any) to the first NAL unit after the AU delimiter NAL unit) or otherwise.

[0238] The following sorting steps can be applied to a sequence of access units in decoding order, starting from the access unit for which an operation point is determined, and ending at the end of the bitstream or access unit. Here, the next operation point can be determined exclusively, even if it is earlier in the decoding order.

[0239] 2) If the VVC bitstream is represented by multiple VVC tracks, the file parser can identify the track required for the selected operation point as follows.

[0240] i) If the operation point of the “opeg” entity group includes the selected operation point, the file parser can select a track that belongs to the operation point indicated in the “opeg” entity group.

[0241] ii) If the “opeg” entity group does not exist (i.e., the “vopi” sample group exists), the file parser can determine whether the VVC track in the “vvcb” entity group represents a VVC bitstream. The first entity_id of the “vvcb” entity group can identify the track containing the “vopi” sample group. The mapping from operation point to layer and sublayer can be derived from the “vopi” sample group. The set of tracks containing the selected operation point and its sublayer, and therefore requiring decoding of the selected operation point, can be derived from the “linf” sample group present in the VVC tracks of the VVC bitstream. Because a particular layer or sublayer can be represented by one or more tracks, when identifying the track required for the operation point, it can be selected from the set of tracks that include all specific layers or sublayers.

[0242] 3) The access unit to the output bitstream can be recovered according to the decoding time order of the selected operation point (if conditions i, ii or iii above are true) or the sample of the VVC basic track (if condition iv mentioned above is true) or the VVC track required by the VVC track (if condition v above is true).

[0243] When multiple tracks contain data for accessing cells, alignment of each sample in a track can be performed based on the sample decoding time.

[0244] The sequence of access units can be reconstructed from each sample of the desired track by repeatedly invoking the implicit reconstruction process, which will be described later.

[0245] The recovered access units can be placed in the output bitstream in ascending order of decoding time.

[0246] 4) Additionally, if both of the following conditions are true, the file reader may need to include EOS NAL units in each layer at the operation point applied to the output bitstream.

[0247] - The sequence of accessed units is selected from operation points that are different from those previously selected.

[0248] - The sequence of accessed units does not end with the EOS NAL unit or EOB NAL unit in each layer of the applied operation point.

[0249] When a VVC bitstream is displayed as multiple VVC tracks, once the tracks are combined into a single aligned bitstream, the decoding time of the sample may need to be modified as specified in ISO / IEC 23090-3, thereby increasing the decoding time.

[0250] Additionally, the implicit reconstruction of the VVC bitstream process can be performed as follows. This process can represent the recovery of access units from samples that are time-aligned with the current decoding time.

[0251] - Required VVC track, - Relevant VVC non-VCL tracks (if any), and - Referenced VVC sub-image tracks (if any).

[0252] When a VCL NAL cell reconstructs a bitstream containing a sub-layer with a TemporalId greater than 0, all lower sub-layers within the same layer (i.e., sub-layers where the VCL NAL cell has a smaller TemporalId) can also be included in the resulting bitstream.

[0253] Additionally, if a sample with the current decoding time includes a VCL NAL unit with a TemporalId greater than the maximum TemporalId contained in the selected operation point, the access unit cannot be recovered from the current decoding time.

[0254] Additionally, when restoring access units, image units of samples with the same decoding time (as specified in ISO / IEC 23090-3) can be placed in the access units in ascending order of their nuh_layer_id values. For example, if conditions apply, the following steps can be performed: - If the samples of the orbit containing the first image unit are labeled as belonging to the sample group "aud" with ols_idx and lower_ols_idx corresponding to the target operation points within the AUD NAL unit. Here, the AUD NAL unit in the "aud" sample group can be placed in the AU as the first NAL unit.

[0255] - If a sample in the orbit is marked as belonging to the sample group "eos" containing ols_idx, max_tid, and lower_ols_idx, then the EOS NAL cell in the "eos" sample group corresponding to the target operation point can be placed in the AU at the indicated position. That is, when inserted by the sample group "aud", it can be placed after the eos_position-th NAL cell of the reconstructed AU, excluding the AUD NAL cell.

[0256] - If a sample in the orbit is labeled as belonging to the sample group "eos" containing ols_idx, max_tid, and lower_ols_idx, then the EOB NAL cell in the sample group "eob" corresponding to the target operation point can be placed in the AU following all NAL cells in the AU (including the EOS NAL cell).

[0257] Only the image units of the layer and sublayer at the target operation point can be included in the output bitstream.

[0258] Additionally, when resuming access units based on the operation point associated with the output layer set index i (in the for loop of num_olss for the “vopi” sample group), the following can be applied.

[0259] - When resuming access to a unit, for each output layer with index j set to the range of 0 to layer_count[i]-1, if num_ref_sublayers_in_layer_in_ols[i][j] is greater than 0, then the VCL NAL unit can belong to a sublayer of the layer where the VCL NAL unit has a TemporalId less than or equal to Min(num_ref_sublayers_in_layer_in_ols[i][j]-1, max_temporal_id). Here, max_temporal_id is the value of the syntax element corresponding to the operation point and is included in the resulting bitstream, and the necessary tracks can be selected accordingly.

[0260] - When recovering access units, for each layer of the output layer with index j set to the range of 0 to layer_count[i]-1, if num_ref_sublayers_in_layer_in_ols[i][j] equals 0, then based on all reference images of the referenced layer, only IRAP image units and GDR image units with ph_recovery_poc_cnt equal to 0 are included in the obtained bitstream, and the necessary tracks can be selected accordingly.

[0261] Additionally, if the accessed units of the VVC track contain unspecified NAL unit types (NAL units with nal_unit_type in the range UNSPEC_28..UNSPEC_31 (inclusive), i.e., units with nal_unit_type values ​​in the range 28 to 31 as defined in ISO / IEC 23090-3), the unspecified NAL unit types may be discarded from the eventually recovered bitstream.

[0262] Additionally, if the VVC track contains a "subp" track reference, each picture unit can be recovered according to section 11.6.3 of ISO / IEC 23090-3. Similarly, if the VVC track contains a "recr" track reference, each picture unit can be recovered according to section 11.6.6 of ISO / IEC 23090-3. The procedures in section 11.6.3 or 11.6.6 of ISO / IEC 23090-3 can be repeated for each layer of the target operation point in ascending order of nuh_layer_id.

[0263] The recovered access units can be placed in the VVC bitstream in ascending order of decoding time.

[0264] Because a specific layer or sublayer can be represented by one or more tracks, when identifying the track required for an operation point, a selection can be made from a set of tracks that include all specific layers or sublayers.

[0265] Additionally, the image cell recovery process from samples of a VVC track referencing a VVC sub-image track can be performed as follows. Samples of a VVC track can be parsed into image cells in the following order.

[0266] - If an AUD NAL cell is in a sample or a time-aligned sample with an associated VVC non-VCL track, it can be included in the picture cell. For example, if an AUD NAL cell exists in a sample, it may be the first NAL cell in the sample.

[0267] - If the sample is the first sample in a sequence of samples associated with the same sample entry, the DCI, OPI, parameter set, and SEI NAL units (if any) included in the sample entry can be included in the image unit.

[0268] - If an associated VVC non-VCL track exists and the picture unit is the first picture unit of the access unit recovered from the sample, then the next NAL unit is included in the picture unit. If at least one NAL unit exists in the time-aligned sample of an associated VVC non-VCL track (EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT, or with nal_unit_type equal to RSV_NVCL_27, and (NAL units with the above NAL unit types cannot precede the first VCL NAL unit in the picture unit), then the above NAL units (excluding AUD NAL units) other than the first unit can be included in the time-aligned sample of the associated VVC non-VCL track. In other cases, all NAL units of the time-aligned sample of the associated VVC non-VCL track can be included.

[0269] - If a sample has one or more NAL units with nal_unit_type equal to EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT, or RSV_NVCL_27 (NAL units with the above NAL unit types cannot precede the first VCL NAL unit in the unit picture), then the NAL units of the sample, except for the first unit, can be included in the picture unit; otherwise, all NAL units of the sample can be included in the picture unit.

[0270] - If the reader selects an operation point, the reader may need to exclude sample entries and all OPINAL cells stored in the sample from the access cells recovered in all the above steps.

[0271] - If the VVC track does not refer to the VVC sub-picture track, then the NAL cell of the sample in the VVC track can be included in the picture cell.

[0272] In addition, if this is not the case, the following can be applied.

[0273] - Track references can be resolved in accordance with Clause 11.6.4 of ISO / IEC 23090-3.

[0274] - If necessary, the parameter set may be updated in accordance with Clause 11.6.5 of ISO / IEC 23090-3.

[0275] - Following the order of the VVC subpicture tracks referenced in the "subp" track reference (if the num_subpic_ref_idx of the same group entry mapped to the sample's "spor" sample group entry is 0) or the order specified in the "spor" sample group entry mapped to the sample (if the num_subpic_ref_idx of the same group entry mapped to the sample's "spor" sample group entry is greater than 0), and all DCI, OPI, VPS, SPS, PPS, AUD, PH, EOS, EOB NAL units, the picture unit can be added to the content of the parsed sample with time alignment (at decoding time) in each referenced VVC subpicture track, except for the expandable nested SEI NAL message where sn_subpic_flag is 1.

[0276] Additionally, if the referenced VVC sub-picture track is associated with a VVC non-VCL track, the parsed sample of the VVC sub-picture track may include the following NAL units.

[0277] - If at least one NAL unit exists in a time-aligned sample of an EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT, or associated VVC non-VCL track with nal_unit_type equal to RSV_NVCL_27 (NAL units with the above-mentioned NAL unit types cannot precede the first VCL NAL unit in the picture unit), the above-mentioned NAL unit (excluding AUD NAL units) excludes the first unit in the time-aligned sample of the associated VVC non-VCL track.

[0278] - Otherwise, all NAL cells of the time-aligned sample of the associated VVC non-VCL track.

[0279] - NAL cells from the referenced VVC sub-image track samples.

[0280] - Remaining NAL cells (if any) in time-aligned samples of associated VVC non-VCL tracks.

[0281] - All NAL units in the sample that have EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT or RSV_NVCL_27 nal_unit_type.

[0282] - If an associated VVC non-VCL track exists and the picture unit is the last picture unit of the access unit recovered from the sample, then all NAL units of the sample with associated VVC non-VCL tracks of time alignment with nal_unit_type of EOS_NUT, EOB_NUT, SUFFIX_APS_NUT, SUFFIX_SEI_NUT, FD_NUT or RSV_NVCL_27.

[0283] - All NAL units or similar structures with nal_unit_type in samples within the range of UNSPEC_28 to UNSPEC_31 (inclusive) can be deleted.

[0284] In addition, the sample groups disclosed in this document may include the following sample groups.

[0285] For example, a sample group of flow access points can be defined. The sample group "sap" of flow access points (SAPs) specified in ISO / IEC 14496-12 can be used to provide information about all SAPs.

[0286] For example, if layer_id_method_idc is 0, then SAP can be interpreted as follows.

[0287] - If the sample entry type is "vvc1" or "vvi1" and the track does not contain a sub-layer with a TemporalId of 0, SAP can specify access to all sub-layers in the track.

[0288] - Otherwise, SAP can specify access to any layer in the track.

[0289] Additionally, for example, the GDR image of a VVC bitstream can often be displayed as SAP type 4 in the "sap" sample group.

[0290] Furthermore, for example, if a sample in a VVC track references a PPS with pps_mixed_nalu_types_in_pic_flag equal to 1, and sps_subpic_treatment_as_pic_flag[i] is 1 for each subpic index i ranging from 0 to sps_num_subpics_minus1, and there is at least one IRAP subpic with the same subpic index i in the same CLVS or after the current sample, then the following can be applied.

[0291] - Samples can be labeled as type 4 SAP samples.

[0292] - Samples can be mapped to "roll" sample group description entries with a roll_distance value modified for the decoding process, so that decoding of sub-images with specific sub-image indices can be omitted before IRAP sub-images exist.

[0293] If using SAP sample groups, it may be necessary to use them on all tracks carrying the same VVC bitstream.

[0294] Additionally, for example, random access recovery point sample groups can be defined. The random access recovery point sample group "roll" specified in ISO / IEC 14496-12 can be used to provide information about recovery points for progressive decoding refresh. When the "roll" sample group is used with a VVC track, the syntax and semantics of the grouping_type_parameter can be specified as the same as those used for the "sap" sample group in ISO / IEC 14496-12. Furthermore, when the image of the target layer mapped to the sample of the "roll" sample group is a GDR image, layer_id_method_idc can be used with values ​​equal to 0 and 1. For example, if layer_id_method_idc equals 0, the "roll" sample group can specify the behavior of all layers in the track. Additionally, for example, the semantics of layer_id_method_idc equal to 1 can be disclosed in section 9.5.7 of ISO / IEC 14496-12. For example, when all images in the target layer of the samples mapped to the "roll" sample group are not GDR images, layer_id_method_idc of values ​​equal to 2 and 3 can be used, and the following can be applied to images in the target layer that are not GDR images.

[0295] - The referenced PPS has pps_mixed_nalu_types_in_pic_flag equal to 1.

[0296] - For each subpick index i in the range from 0 to sps_num_subpics_minus1, both of the following can be true: sps_subpic_treatment_as_pic_flag[i] is 1, there exists at least one IRAP subpick that follows or has the same subpick index i as the current sample in the same CLVS.

[0297] Furthermore, for example, if layer_id_method_idc equals 2, the “roll” sample group can specify the behavior of all layers in the track. Additionally, for example, the semantics of layer_id_method_idc equaling 3 can be disclosed in section 9.5.7 of ISO / IEC 14496-12.

[0298] If the reader begins decoding using a sample indicated by layer_id_method_idc equal to 2 or 3, then in order to ensure that the bitstream starting with a sample marked as belonging to the sample group with layer_id_method_idc of 2 and 3 is a matching bitstream, the reader may need to further modify the SPS, PPS, and PH NAL units of the recovered bitstream as follows.

[0299] - All SPS referenced in the sample can have a sps_gdr_enabled_flag equal to 1.

[0300] - All PPS referenced in the sample can have pps_mixed_nalu_types_in_pic_flag equal to 0.

[0301] - All VCL NAL units of the AU recovered from the sample can have the same nal_unit_type as GDR_NUT.

[0302] - All image headers of the AU recovered from the sample can have a ph_gdr_pic_flag equal to 1 and a ph_recovery_poc_cnt value corresponding to the roll_distance of the “roll” sample group description entry to which the sample is mapped.

[0303] Furthermore, for example, if the "roll" sample group is related to the dependency layer but not to the referencing layer, then the sample group can represent the characteristics applied when all referencing layers of the dependency layer are available and decoded. The sample group can then be used to begin decoding the prediction layer.

[0304] Additionally, for example, layer information sample groups can be defined. A list of layers and sublayers carried by a track can be signaled as layer information (“linf”) sample groups. If the current track contains two or more layers, two or more sublayers, or one sublayer that is not a sublayer with a TemporalId of 0, then layer and sublayer information may be needed to interpret the information signaled in different sample groups, such as “sap,” “rap,” “sync,” “tele,” and “vopi” sample groups. Furthermore, if a “linf” sample group exists in a VVC track, then sub_layer_presence_flags can be equal to 0.

[0305] Additionally, for example, operation point information sample groups can be defined. Applications can use operation point information sample groups (“vopi”) to provide various operation point notifications for the VVC basic flow and configuration. Operation points can be associated with output layer sets, maximum TemporalId values, profiles, tiers, and level signaling. All of the above information can be captured through “vopi” sample groups. In addition to the above information, operation point information sample groups can also provide dependency information between layers.

[0306] Additionally, for example, if there is more than one VVC track for a VVC elementary flow and there is no operation point entity group for a VVC elementary flow, then both of the following can be applied.

[0307] - In the VVC tracks used for the VVC basic stream, only one track can exist that carries the "vopi" sample group.

[0308] All other VVC tracks in the VVC basic stream can have track references of type "vref" to the track that carries the "vopi" sample group.

[0309] For a specific sample on a given track, samples juxtaposed temporally on other tracks can be defined as having the same decoding time as that specific sample. The following can be applied for each sample SN on track TN that has a track reference to track Tk carrying the "vref" sample group.

[0310] -If orbit T exists k Time-juxtaposed samples S k Then sample S N It can be associated with sample S k The same "vopi" sample group entries.

[0311] -Otherwise, sample S N It can be associated with the "vopi" sample group entry, which is equal to the entry that precedes sample S during decoding. N orbit T k The final sample.

[0312] Additionally, if the VVC base stream references multiple VPSs, it may be necessary to include multiple entries in a sample group description box with grouping_type "vopi". Alternatively, if there is only one VPS, it is better to include the operation point information sample group in the sample table box rather than using the default sample group description index and including it in each track segment. Furthermore, the grouping_type_parameter cannot be defined for a SampleToGroupBox with grouping type "vopi".

[0313] The “vopi” sample group entries can include VVC operation point records. The syntax for VVC operation point records is shown in the table below.

[0314] Table 5

[0315] In addition, the semantics of the syntax used for VVC operation point recording can be shown in the following table.

[0316] Table 6

[0317] Additionally, for example, an access cell delimiter sample group can be defined. The sample group description entry of the sample group can include AUD NAL cells.

[0318] If a sample is mapped to an access unit limiter sample group (“aud”), and the target operation point corresponds to one of the output layer sets indicated within the “aud” sample group, then the AUD NAL units contained within the sample group can be indicated to be inserted into the recovered AU.

[0319] The syntax for AUD sample entries in the access unit delimiter sample group can be shown in the table below.

[0320] Table 7

[0321] In addition, the semantics of the syntax used for AUD sample entries can be seen in the table below.

[0322] Table 8

[0323] Additionally, for example, the end of a sequence sample group can be defined. The sample group description entry for a sample group can include an ESNAL unit.

[0324] If a sample is mapped to the end of the sequence sample group (“eos”), and if the target operation point corresponds to one of the output layer set and the maximum time ID indicated within the “eos” sample group, then it can indicate the position at which the EOS NAL cells included in the sample group should be inserted into the indicated position of the recovered AU.

[0325] The syntax for the sequence end sample entry of the sequence end sample group can be shown in the table below.

[0326] Table 9

[0327] In addition, the semantics of the syntax used for the end of a sequence sample entry can be shown in the table below.

[0328] Table 10

[0329] Additionally, for example, the end of a bitstream sample group can be defined. The sample group description entry for a sample group can include an EOB NAL unit.

[0330] If a sample is mapped to the end-of-bitstream sample group (“eob”), and the target operation point corresponds to one of the output layer set and maximum time ID indicated within the “eob” sample group, then the EOB NAL unit included in the sample group can be indicated to be inserted into the recovered AU.

[0331] The syntax for the bitstream end sample entry in the bitstream end sample group can be shown in the table below.

[0332] Table 11

[0333] In addition, the semantics of the syntax used for the end-of-bitstream sample entries can be shown in the table below.

[0334] Table 12

[0335] Meanwhile, when the target operation point changes as described above, the current VVC file format is specified as follows.

[0336] Table 13

[0337] As shown in Table 13 above, if both of the following conditions are true, the file reader may need to include EOS NAL units in each layer that serves as the operation point for the output bitstream application.

[0338] - After the sequence of accessing units, select an operation point that is different from the previous operation point.

[0339] - The sequence of accessed units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied operation point.

[0340] However, the above process may cause at least the following problems.

[0341] First, the target operation point has two components: the target OLS and the maximum time ID. Even if only the maximum time ID changes, the target operation point is considered to have changed, but the aforementioned procedure may not need to be invoked. Changes in the maximum time ID can occur smoothly in response to changes in network state. Therefore, a method can be proposed such that the procedure is invoked only when the change is related to the target OLS.

[0342] Secondly, this process requires the file reader to include EOS NAL units as the output bitstream in each layer of the applied operation point. This may mean that the file reader must be able to place the EOS NAL units in the appropriate positions within the access unit. This can be a burden. Because VVC allows EOS NAL units to exist anywhere within the access unit, as long as they do not precede the associated image unit, the file parser does not need to place the EOS in the correct layer.

[0343] Therefore, this document proposes solutions to the problems mentioned above. The proposed embodiments can be applied individually or in combination.

[0344] As a first example, instead of invoking the procedure for inserting the EOS NAL cell when the target operation point changes, this paper proposes a method that invokes the procedure for inserting the EOS NAL cell only when the target OLS changes. Therefore, if only the maximum time ID changes, this procedure can be skipped.

[0345] As a second example, when inserting EOS NAL cells in the above process, this document proposes a method that allows the file parser to include as many EOS NAL cells as there are layers in the target OLS, without having to place them in the corresponding layers.

[0346] As a third example, this document proposes a method that allows all necessary EOS NAL cells to be placed at the end of the access cell.

[0347] As a fourth example, this document suggests making the above process optional.

[0348] As examples, embodiments reflecting the first to third examples presented in this document can be as follows. The above embodiments can be expressed as the VVC file format specification shown in the table below.

[0349] Table 14

[0350] For example, based on Table 14 above, it can be determined whether the following two conditions are true. For example, the first condition could be whether an OLS different from the previous output layer set (OLS) is selected after the sequence of accessed units. Additionally, for example, the second condition could be whether the sequence of accessed units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied OLS.

[0351] If both of the above conditions are true—that is, after the sequence of access units, an OLS different from the previous OLS is selected, and if the sequence of access units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied OLS—then the file reader can include in the last access unit the same number of EOS NAL units as the target output layer applied as the output bitstream. The included EOS NAL units can be placed at the end of the access unit in ascending order of the NAL unit's layer ID.

[0352] As another example, embodiments reflecting the first to fourth examples presented in this document can be as follows. The above embodiments can be expressed as the VVC file format specification shown in the table below.

[0353] Table 15

[0354] For example, based on Table 15 above, it can be determined whether the following two conditions are true. For example, the first condition could be whether an OLS different from the previous output layer set (OLS) is selected after the sequence of accessed units. Additionally, for example, the second condition could be whether the sequence of accessed units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied OLS.

[0355] If both of the above conditions are true—that is, after the sequence of access units, an OLS different from the previous OLS is selected, and if the sequence of access units does not end with an EOS NAL unit or an EOB NAL unit in each layer of the applied OLS—then the file reader can include in the last access unit the same number of EOS NAL units as the target output layer applied as the output bitstream. The process of including EOS NAL units can be optional. The included EOS NAL units can be placed at the end of the access unit in ascending order of the NAL unit's layer ID.

[0356] Figure 8 An example of a media file generation method applying one embodiment of the present disclosure is illustrated.

[0357] Reference Figure 8The first device can configure the track (S800). For example, the first device can configure the track according to the foregoing embodiments. For example, the first device can refer to a sending end, an encoding end, or a media file generating end. For example, the first device can configure sub-image tracks and basic tracks. The first device may include an encoder.

[0358] The first device can generate a media file based on the track (S810). For example, the first device can generate a media file based on the track according to the aforementioned embodiments.

[0359] Figure 9 A method for decoding a media file generated by applying the embodiments proposed in this disclosure is illustrated.

[0360] Reference Figure 9 The second device can acquire / receive a media file including tracks (S900). For example, the second device can acquire / receive a media file including tracks according to the foregoing embodiments. For example, the second device can refer to a receiving end, a decoding end, or a rendering end.

[0361] For example, media files may include information described in Tables 1, 3, 5, 7, 9 and / or 11.

[0362] The second device can parse / obtain tracks (S910). The second device can parse / obtain tracks included in the media file. For example, tracks may include sub-image tracks or base tracks. For example, the second device can parse sub-image tracks and base tracks.

[0363] The second device can generate slices of one or more sub-images based on the track. The second device can generate reconstructed samples based on the track. The second device can obtain the information needed for image / video decoding based on the track.

[0364] Figure 10 This illustration schematically demonstrates a method for creating media files based on a media file generation device derived from this document. Figure 10 The method disclosed in the article can be used by Figure 11 The media file creation device disclosed herein shall be used to perform the operation. The media file creation device may represent the first device mentioned above. Specifically, for example, Figure 10 Steps S1000 to S1030 can be performed by the media file creation unit of the media file creation device. Additionally, although not shown, the process of encoding the bitstream including video information can be performed by an encoder. The encoder can be included in the media file creation device or can be configured as an external component.

[0365] The media file generation device generates a sequence of access units for the first output layer set (OLS) (S1000). For example, the media file generation device can export a bitstream including encoded video information and generate a media file for that bitstream. For example, the media file generation device can obtain encoded video information via a network or a (digital) storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. Alternatively, for example, the media file generation device may include an encoder and export encoded video information. For example, the media file generation device can generate a first OLS for the bitstream including encoded video information. For example, the media file generation device can generate a sequence of access units for the first OLS.

[0366] The media file creation device determines whether the first condition and the second condition are true (S1010). The media file generation device can determine whether the first condition and the second condition are true.

[0367] For example, the first condition could be whether an OLS different from the first OLS is selected after the sequence of access units. That is, for example, if an OLS different from the first OLS is selected after the sequence of access units, the first condition can be determined to be true. Meanwhile, for example, the OLS different from the first OLS can be referred to as the second OLS.

[0368] Furthermore, for example, the second condition could be whether the sequence of accessed units in each layer of the first OLS does not end with an EOS NAL unit or an end-of-bit stream (EOB) NAL unit. That is, for example, if the sequence of accessed units in each layer of the first OLS does not end with an EOS NAL unit or an end-of-bit stream (EOB) NAL unit, then the second condition can be determined to be true.

[0369] Based on the fact that the first and second conditions are true, the media file generation device inserts a Sequence End (EOS) Network Abstraction Layer (NAL) unit equal to the number of layers applied to the first OLS (S1020). For example, if the first and second conditions are true, the media file generation device can insert a Sequence End (EOS) Network Abstraction Layer (NAL) unit equal to the number of layers applied to the first OLS into the first OLS.

[0370] Moreover, for example, EOS NAL cells may not be placed in the corresponding layer.

[0371] Alternatively, for example, the media file generation device can insert the EOS NAL units at the end of the last access unit of the sequence in ascending order of their layer IDs. Alternatively, for example, the inserted EOS NAL units can be placed at the end of the access unit of the sequence in ascending order of their layer IDs.

[0372] Additionally, for example, if no OLS different from the first OLS is selected after the generated sequence of access units and only the target operation point is changed, then an EOS NAL unit may not need to be inserted. For example, if no OLS different from the first OLS is selected after the generated sequence of access units and only the maximum time ID is changed, then an EOS NAL unit may not need to be inserted.

[0373] The media file generation device generates a media file including a first OLS (S1030). The media file generation device can generate a media file including a first OLS and / or a second OLS.

[0374] Simultaneously, although not shown, the media file generation device can generate a second OLS. For example, the media file creation device can generate a sequence of access units for the second OLS. Additionally, although not shown, the media file generation device can store the generated media file in a (digital) storage medium, or send it to a media file processing device via a network or (digital) storage medium. Here, the network can include broadcast networks and / or communication networks, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0375] Figure 11 This schematically illustrates a media file generation device that performs the media file generation method according to this document. Figure 10 The method disclosed in the article can be used by Figure 11 The publicly available media file creation device is used to perform this. Specifically, for example, Figure 11 The media file generation unit of the media file generation device can execute steps S1000 to S1030. Additionally, although not shown, the process of encoding the bitstream including video information can be performed by the encoder of the media file creation device.

[0376] Figure 12 This document schematically illustrates a media file processing method of a media file processing device according to this document. Figure 12 The method disclosed in the article can be derived from Figure 13 The media file processing device disclosed herein shall be used to perform the operation. The media file processing device may represent the second device described above. For a specific example, Figure 12S1200 to S1220. These can be performed by the media file processing unit of the media file processing device. The media file processing unit may include a file parser and / or a file reader. Additionally, although not shown, the process of acquiring the media file can be performed by the receiving unit, the process of deriving a bitstream including a sequence of access units for a first OLS and a sequence of access units for an OLS different from the first OLS can be performed by the media file processing unit, and the process of decoding the bitstream can be performed by the decoder. The decoder may be included in the media file processing device or may be configured as an external component.

[0377] The media file processing device recovers the sequence of access units used for the first output layer set (OLS) (S1200). For example, the media file processing device can obtain a media file via a network or a (digital) storage medium. The media file may include the sequence of access units used for the first OLS. The media file processing device can reconstruct the sequence of access units used for the first OLS. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0378] The media file processing device determines whether the first condition and the second condition are true (S1210). The media file processing device can determine whether the first condition and the second condition are true.

[0379] For example, the first condition could be whether an OLS different from the first OLS is selected after the sequence of access units. That is, for example, if an OLS different from the first OLS is selected after the sequence of access units, the first condition can be determined to be true. Meanwhile, for example, the OLS different from the first OLS can be referred to as the second OLS.

[0380] Furthermore, for example, the second condition could be whether the sequence of accessed units in each layer of the first OLS does not end with an EOS NAL unit or an end-of-bit stream (EOB) NAL unit. That is, for example, if the sequence of accessed units in each layer of the first OLS does not end with an EOS NAL unit or an end-of-bit stream (EOB) NAL unit, then the second condition can be determined to be true.

[0381] Based on the fact that the first and second conditions are true, the media file processing device inserts a Sequence End (EOS) Network Abstraction Layer (NAL) unit equal to the number of layers applied to the first OLS (S1220). For example, if the first and second conditions are true, the media file processing device may insert a Sequence End (EOS) Network Abstraction Layer (NAL) unit equal to the number of layers applied to the first OLS into the first OLS.

[0382] Moreover, for example, EOS NAL cells may not be placed in the corresponding layer.

[0383] Alternatively, for example, a media file processing device can insert the EOS NAL units at the end of the last access unit in the sequence, according to the ascending order of their layer IDs. Alternatively, for example, the inserted EOS NAL units can be placed at the end of the access unit in the sequence, according to the ascending order of their layer IDs.

[0384] Additionally, for example, if no OLS different from the first OLS is selected after the sequence of recovered access cells and only the target operation point is changed, then an EOS NAL cell may not need to be inserted. For example, if no OLS different from the first OLS is selected after the sequence of recovered access cells and only the maximum time ID is changed, then an EOS NAL cell may not need to be inserted.

[0385] Simultaneously, although not shown, the media file processing device can recover the order of access units used for the second OLS. Furthermore, the media file processing device can export a bitstream including the access unit sequence for the first OLS and the access unit sequence for the second OLS, and can decode the bitstream. This bitstream can be referred to as a VVC bitstream or an output bitstream. For example, the media file processing device can decode the video information in the bitstream and generate a recovered image based on the video information.

[0386] Figure 13 This schematically illustrates a media file processing device that performs the media file processing method according to this document. Figure 12 The method disclosed in the article can be derived from Figure 13 The media file processing device disclosed in the document is used to perform this operation. Specifically, for example, Figure 12 The media file processing unit of the media file processing device can perform Figure 12 Steps S1200 to S1220. Meanwhile, although not shown, the media file processing device may include a decoder. The process of acquiring the media file may be performed by the receiving unit, the process of deriving a bitstream including a sequence of access units for a first OLS and a sequence of access units for an OLS different from the first OLS may be performed by the media file processing unit, and the process of decoding the bitstream may be performed by the decoder.

[0387] According to the aforementioned document, even changing only the maximum time ID can prevent unnecessary operations of inserting EOS NAL cells, thereby improving overall coding efficiency.

[0388] In addition, according to this document, the overhead that occurs when inserting EOS NAL units into the appropriate positions within the access units can be prevented, thereby improving overall coding efficiency.

[0389] In the above embodiments, the method is described based on a flowchart having a series of steps or blocks. This disclosure is not limited to the order of the steps or blocks described above. Some steps or blocks may occur simultaneously or in a different order than the other steps or blocks described above. Furthermore, those skilled in the art will understand that the steps shown in the above flowchart are not exclusive and may include other steps, or one or more steps in the flowchart may be deleted without affecting the scope of this disclosure.

[0390] The embodiments described in this specification can be implemented on a processor, microprocessor, controller, or chip. For example, the functional unit shown in each figure can be implemented on a computer, processor, microprocessor, controller, or chip. In this case, the information for implementation (e.g., information about instructions) or the algorithm can be stored in a digital storage medium.

[0391] Furthermore, the devices to which this disclosure applies may include multimedia broadcasting transmitting / receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video chat devices, real-time communication devices such as video communication, mobile streaming devices, storage media, portable cameras, VoD service providing devices, over-the-top (OTT) video devices, internet streaming service providing devices, three-dimensional (3D) video devices, virtual reality (VR) devices, augmented reality (AR) devices, teleconferencing video devices, transportation user equipment (e.g., vehicle user equipment, aircraft user equipment, ship user equipment, etc.), and medical video devices, and may be used to process video signals or data signals. For example, over-the-top (OTT) video devices may include game consoles, Blu-ray players, connected TVs, home theater systems, smartphones, tablet PCs, and digital video recorders (DVRs), etc.

[0392] Furthermore, the processing methods to which this disclosure applies can be generated in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data having the data structure according to this disclosure can also be stored in a computer-readable recording medium. A computer-readable recording medium includes all types of storage devices storing data readable by a computer system. For example, a computer-readable recording medium can include BD, Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, a computer-readable recording medium includes media implemented in the form of a carrier wave (e.g., transmitted via the Internet). Furthermore, bitstreams generated by encoding methods can be stored in a computer-readable recording medium or transmitted via wired / wireless communication networks.

[0393] Furthermore, embodiments of this disclosure can be implemented as a computer program product based on program code, and the program code can be executed on a computer through embodiments of this disclosure. The program code can be stored on a computer-readable medium.

[0394] Figure 14 A structural diagram of a content streaming system applying this disclosure is illustrated.

[0395] A content streaming system using embodiments of this disclosure may substantially include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.

[0396] An encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and then sends the bitstream to a streaming server. As another example, when a multimedia input device such as a smartphone, camera, or camcorder generates a bitstream directly, the encoding server can be omitted.

[0397] Bitstreams can be generated using the encoding methods or bitstream generation methods applied to the embodiments of this disclosure. Furthermore, the streaming server can temporarily store the bitstream during transmission or reception.

[0398] The streaming server sends multimedia data to the user's device via a web server based on the user's request, and the web server acts as a medium for notifying the user of services. When a user requests a desired service from the web server, the web server delivers the request to the streaming server, and the streaming server sends the multimedia data to the user. In this scenario, the content streaming system may include a separate control server. In this case, the control server is used to control the commands / responses between the various devices in the content streaming system.

[0399] A streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a predetermined period of time.

[0400] Examples of user devices may include mobile phones, smartphones, laptops, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, board PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays), digital TVs, desktop computers, digital signage, etc. Each server in the content streaming system can operate as a distributed server, and in this case, data received from each server can be distributed.

[0401] The claims described in this disclosure can be combined in various ways. For example, the technical features in the method claims of this disclosure can be combined to implement a device, and the technical features in the device claims of this disclosure can be combined to implement a method. Furthermore, the technical features in the method claims and the device claims of this disclosure can be combined to implement a device, and the technical features in the method claims and the device claims of this disclosure can be combined to implement a method.

Claims

1. A method for processing media files, the method comprising: Reconstruct the sequence of access units used for the first output layer set (OLS); Determine whether the first and second conditions are true; as well as Based on the fact that the first and second conditions are true, insert the same number of Sequence End (EOS) Network Abstraction Layer (NAL) units as the number of layers applied to the first OLS. The first condition is whether an OLS different from the first OLS is selected after the sequence of access units. The second condition is whether, in each layer of the first OLS, the sequence of access units does not end with an EOSNAL unit or a bitstream-end (EOB)NAL unit, and The inserted EOS NAL units are arranged in ascending order of their layer IDs at the end of the access units in the sequence.

2. The method according to claim 1, wherein, If no OLS other than the first OLS is selected after the sequence of the reconstructed access units and only the target operation point is changed, then the EOS NAL unit is not inserted.

3. The method according to claim 1, wherein, If no OLS other than the first OLS is selected after the sequence of the reconstructed access units and only the maximum time ID is changed, then the EOS NAL unit is not inserted.

4. A media file processing apparatus for performing a media file processing method, comprising: A receiver configured to acquire media files; as well as A media file processor configured to reconstruct the sequence of access units for a first output layer set (OLS) of the media file, determine whether a first condition and a second condition are true, and, based on the first condition and the second condition being true, insert an equal number of sequence end (EOS) network abstraction layer (NAL) units as applied to the first OLS. The first condition is whether an OLS different from the first OLS is selected after the sequence of accessed units. The second condition is whether, in each layer of the first OLS, the sequence of access units does not end with an EOSNAL unit or a bitstream-end (EOB)NAL unit, and The inserted EOS NAL units are arranged in ascending order of their layer IDs at the end of the access units in the sequence.

5. The media file processing apparatus according to claim 4, wherein, If no OLS other than the first OLS is selected after the sequence of the reconstructed access units and only the target operation point is changed, then the EOS NAL unit is not inserted.

6. The media file processing apparatus according to claim 4, wherein, If no OLS other than the first OLS is selected after the sequence of the reconstructed access units and only the maximum time ID is changed, then the EOS NAL unit is not inserted.

7. A method for generating a media file, the method comprising: Generate a sequence of access units for the first output layer set (OLS); Determine whether the first and second conditions are true; If the first condition and the second condition are true, insert the same number of Sequence End (EOS) Network Abstraction Layer (NAL) units as the number of layers applied to the first OLS; as well as Generate the media file that includes the first OLS. The first condition is whether an OLS different from the first OLS is selected after the sequence of access units. The second condition is whether, in each layer of the first OLS, the sequence of access units does not end with an EOSNAL unit or a bitstream-end (EOB)NAL unit, and The inserted EOS NAL units are arranged in ascending order of their layer IDs at the end of the access units in the sequence.

Images