Method, apparatus and computer program product for video encoding and video decoding

By managing and parsing sub-picture tracks in container files, using sample group description entries to indicate sub-picture layout and selecting appropriate sub-picture tracks, the problem of video encoding and decoding in the prior art is solved, and efficient video is achieved Bitstream reconstruction and flexible sub-picture merging.

CN114930868BActive Publication Date: 2024-11-26NOKIA TECHNOLOGIES OY
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202080091336.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-12-31
Filing Date
2020-12-17
Publication Date
2024-11-26
Estimated Expiration
2040-12-17

AI Technical Summary

Technical Problem

The prior art is difficult to effectively manage and merge sub-picture tracks in video encoding and decoding, resulting in low storage and transmission efficiency and the inability to flexibly select sub-picture subsets of multiple source bitstreams.

Method used

Reconstructing video bitstreams by writing and parsing sub-picture tracks in a container file, using sample group description entries to indicate sub-picture layouts and selecting appropriate sub-picture tracks.

Benefits of technology

Improves the storage and transmission efficiency of video encoding and decoding, allows flexible selection and merging of sub-pictures, reduces byte counting overhead, and supports time-varying sub-picture layout and parameter assignment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114930868B_ABST
    Figure CN114930868B_ABST
Patent Text Reader

Abstract

The embodiment relates to a method for writing two or more sub-picture tracks in a container file; writing a basic track in the container file, the basic track being intended to be parsed into a video bitstream; indicating a sub-picture layout in the basic track; writing a sample group description entry in the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; and indicating samples of the basic track in the container file, the sample group description entry being intended to be used to reconstruct a video bitstream for the samples of the basic track. The embodiment also relates to a parsing method, and a technical device for implementing the writing method and the parsing method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present solution generally relates to video encoding and video decoding. Background Art

[0002] This section is intended to provide a background or context to the invention described in the claims. The description herein may include concepts that could be pursued but not necessarily have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

[0003] A video coding system may include an encoder that transforms an input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form, for example, so that the video information can be stored / transmitted at a lower bit rate than might otherwise be required. Summary of the invention

[0004] The scope of protection sought by various embodiments of the present invention is defined by the independent claims. Embodiments and features described in this specification that do not belong to the scope of the independent claims, if any, are to be interpreted as examples useful for understanding various embodiments of the present invention.

[0005] An improved method and a technical device for implementing the method have now been invented. Various aspects include a method, an apparatus and a computer-readable medium including a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims.

[0006] According to a first aspect, a method is provided, the method comprising writing two or more sub-picture tracks in a container file; writing a basic track in the container file, the basic track being intended to be decomposed into a video bitstream; indicating a sub-picture layout in the basic track; writing in the container file an indication of a sample group description entry, the sample group description entry being for a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; and indicating samples of the basic track in the container file, the sample group description entry being intended to be used to reconstruct a video bitstream for the samples of the basic track.

[0007] According to a second aspect, a method is provided, the method comprising parsing a sub-picture layout from a basic track of a container file; parsing a sample group description entry from the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; when the sample group description entry indicates a group of sub-picture tracks, selecting a second sub-picture track from the group of sub-picture tracks; parsing which sample set of the basic track from the container file the sample group description entry is intended to be used to reconstruct a video bitstream; and reconstructing a coded picture of a video bitstream corresponding to the sample set from the container file by time-aligning samples of the first sub-picture track or the second sub-picture track included for each sub-picture position of the sub-picture layout.

[0008] According to a third aspect, a device is provided, the device comprising a component for writing two or more sub-picture tracks in a container file; a component for writing a basic track in the container file, the basic track being intended to be decomposed into a video bitstream; a component for indicating a sub-picture layout in the basic track; a component for writing a sample group description entry in the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; and a component for indicating samples of the basic track in the container file, the sample group description entry being intended to be used to reconstruct a video bitstream for the samples of the basic track.

[0009] According to one embodiment, the apparatus further comprises means for writing track references from the base track to a list of entries in the container file, each entry identifying a sub-picture track or a track group of sub-picture tracks; and wherein for each sub-picture position in the sub-picture layout, the sample group description entry comprises, for each sub-picture position, an index to the list of entries for each sub-picture position, the index indicating the first sub-picture track or the group of sub-picture tracks.

[0010] According to one embodiment, the sample group description entry includes an indication as to whether the sub-picture identification is carried in a picture header or a parameter set included in the base track.

[0011] According to one embodiment, the sample group description entry includes one or more of the following:

[0012] - the length of the picture identifier syntax element;

[0013] - the bit position of the first sub-picture identifier syntax element;

[0014] - A flag indicating whether a start code emulation prevention byte is present before or within the sub-picture identifier syntax element.

[0015] According to one embodiment, a sample entry of a sub-picture track includes one or more of the following:

[0016] - sub-picture identifier;

[0017] - Sub-picture location identifier.

[0018] According to one embodiment, the apparatus further comprises means for writing sample groups for picture header NAL units in the container file.

[0019] According to a fourth aspect, a device is provided, the device comprising a component for parsing a sub-picture layout from a basic track of a container file; a component for parsing a sample group description entry from the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; a component for selecting a second sub-picture track from the group of sub-picture tracks when the sample group description entry indicates a group of sub-picture tracks; a component for parsing which sample set of the basic track from the container file the sample group description entry is intended to be used to reconstruct a video bitstream; and a component for reconstructing a coded picture of a video bitstream corresponding to the sample set from the container file by including time-aligned samples of the first sub-picture track or the second sub-picture track for each sub-picture position of the sub-picture layout.

[0020] According to one embodiment, the apparatus further comprises reading, from the container file, track references from the base track to a list of entries, each entry identifying a sub-picture track or a track group of sub-picture tracks; and wherein for each sub-picture position in the sub-picture layout, the sample group description entry comprises, for each sub-picture position, an index into the list of entries for each sub-picture position, the index indicating a first sub-picture track or the group of sub-picture tracks.

[0021] According to one embodiment, the sample group description entry includes an indication of whether the sub-picture identification is carried in a picture header or a parameter set included in the base track.

[0022] According to one embodiment, the sample group description entry includes one or more of the following:

[0023] - the length of the picture identifier syntax element;

[0024] - the bit position of the first sub-picture identifier syntax element;

[0025] - A flag indicating whether a start code emulation prevention byte is present before or within the sub-picture identifier syntax element.

[0026] According to one embodiment, a sample entry of a sub-picture track includes one or more of the following:

[0027] - sub-picture identifier;

[0028] - Sub-picture location identifier.

[0029] According to one embodiment, the apparatus further comprises means for reading a sample group for a picture header NAL unit from the container file.

[0030] According to one embodiment, the apparatus further comprises means for indicating a mapping of sub-picture identifiers to sub-picture layouts.

[0031] According to an embodiment, the means for indicating is configured to override the sub-picture identifier in the parameter set or the picture header by using one or more of the following:

[0032] a) draw a conclusion as to whether the sub-picture identifier is carried in a parameter set and / or a picture header;

[0033] b) if more than one parameter set or picture header contains the sub-picture identifier, a conclusion is drawn about the precedence between parameter sets and picture headers and the parameter set or picture header with the highest priority is selected;

[0034] c) if a picture header is selected for overlay, the picture header for overlay is selected as a picture header present in the sample or a picture header of a sample grouping mapped to the sample in the base track;

[0035] d) Modify the selected parameter set or picture header to contain the sub-picture identifier of the selected sub-picture track.

[0036] According to one embodiment, for option d), the apparatus comprises means for performing the following modification: starting from the bit position of the first sub-picture identifier element and in the order specified in the sample group description entry, overwriting each sub-picture identifier element value with a sub-picture identifier from each selected sub-picture track.

[0037] According to one embodiment, the apparatus further comprises at least one processor and a memory including computer program code.

[0038] According to a fourth aspect, there is provided a computer program product comprising computer program code, the computer program code being configured to, when executed on at least one processor, cause an apparatus or system to implement a method according to any one of the embodiments.

[0039] According to one embodiment, a computer program product according to an embodiment is embodied on a non-transitory computer readable medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Hereinafter, various embodiments will be described in more detail with reference to the accompanying drawings, in which

[0041] Figure 1 A first example using a VVC sub-picture track is shown;

[0042] Figure 2 A second example using a VVC sub-picture track is shown;

[0043] Figure 3 is a flow chart illustrating a method according to one embodiment;

[0044] Figure 4 is a flow chart illustrating a method according to another embodiment;

[0045] Figure 5 An apparatus according to an embodiment is shown;

[0046] Figure 6 shows an encoding process according to one embodiment; and

[0047] Figure 7 A decoding process according to one embodiment is shown. DETAILED DESCRIPTION

[0048] In the following, several embodiments will be described in the context of a video coding arrangement. However, it should be noted that the present embodiments are not necessarily limited to this particular arrangement.

[0049] The Advanced Video Coding standard (which may be abbreviated as AVC or H.264 / AVC) was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by these two parent standardization organizations and is referred to as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There are multiple versions of the H.264 / AVC standard, each of which integrates new extensions or features into the specification. These extensions include scalable video coding (SVC) and multi-view video coding (MVC).

[0050] The High Efficiency Video Coding standard (which may be abbreviated as HEVC or H.265 / HEVC) was developed by the Joint Collaboration Team on Video Coding (JCT-VC) of VCEG and MPEG. The standard is published by these two parent standardization organizations and is called ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Extensions to H.265 / HEVC include scalable, multi-view, three-dimensional, and fidelity range extensions, which may be referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. Unless otherwise stated, references to H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC, and REXT in this description for the purpose of understanding the definitions, structures, or concepts of these standard specifications should be understood as references to the latest versions of these standards available prior to the date of this application.

[0051] A generic video coding standard (VVC, H.266 or H.266 / VVC) is currently being developed by the Joint Video Experts Team (JVET) which is a collaboration between ISO / IEC MPEG and ITU-T VCEG.

[0052] This section will describe some key definitions, bitstream and coding structures and concepts of H.264 / AVC and HEVC and some of their extensions as examples of video encoders, decoders, encoding methods, decoding methods and bitstream structures in which embodiments may be implemented. Some key definitions, bitstream and coding structures and concepts of H.264 / AVC are the same as those in the HEVC standard, and therefore, they will be described jointly below. Aspects of the various embodiments are not limited to H.264 / AVC or HEVC or their extensions, but rather are described for one possible basis on which the embodiments may be partially or fully implemented.

[0053] A video codec may include an encoder that transforms an input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a visual form. The compressed representation may be referred to as a bitstream or video bitstream. The video encoder and / or video decoder may also be separate from each other, i.e., need not form a codec. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0054] A hybrid video codec (e.g., ITU-T H.264) can encode video information in two stages. First, the pixel values ​​in a particular picture area (or "block") are predicted, for example, by a motion compensation component (looking up and indicating an area in one of the previously encoded video frames that closely corresponds to the block being encoded) or by a spatial component (using the pixel values ​​around the block to be encoded in a specified manner). Then, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This can be accomplished by transforming the difference in pixel values ​​using a specified transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate).

[0055] In temporal prediction, the prediction source is a previously decoded picture (also called a reference picture). In intra-block copy (IBC; also called intra-block copy prediction or current picture reference), prediction is applied similarly to temporal prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced during the prediction process. Inter-layer or inter-view prediction can be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter-frame prediction may refer only to temporal prediction, while in other cases, inter-frame prediction may be collectively referred to as temporal prediction, as well as any of intra-block copy, inter-layer prediction, and inter-view prediction, provided that they are performed in the same or similar process as temporal prediction. Inter-frame prediction or temporal prediction may sometimes be referred to as motion compensation or motion compensated prediction.

[0056] Intra prediction exploits the fact that neighboring pixels within the same picture may be correlated. Intra prediction can be performed in the spatial domain or in the transform domain, i.e., sample values ​​or transform coefficients can be predicted. Intra prediction can be utilized in intra coding where inter prediction is not applied.

[0057] One result of the encoding process is a set of encoding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy encoded more efficiently if they are first predicted from spatially or temporally neighboring parameters. For example, a motion vector can be predicted from spatially neighboring motion vectors, and only the difference relative to the motion vector predictor can be encoded. Prediction of encoding parameters and intra-frame prediction can be collectively referred to as intra-picture prediction.

[0058] Entropy encoding / decoding can be performed in a variety of ways. For example, context-based encoding / decoding can be applied, in which both the encoder and the decoder modify the context state of the encoding parameters based on the encoding parameters of the previous encoding / decoding. For example, the context-based encoding can be context-adaptive binary arithmetic coding (CABAC) or context-based variable length coding (CAVLC) or any similar entropy encoding. Entropy encoding / decoding can alternatively or additionally be performed using a variable length coding scheme, such as Huffman encoding / decoding or Exp-Golomb encoding / decoding. The decoding of the encoding parameters from the entropy coded bitstream or codeword can be called parsing.

[0059] Video coding standards may specify bitstream syntax and semantics and the decoding process for error-free bitstreams, while the encoding process may not be specified, but the encoder may only need to produce a conforming bitstream. The conformance of the bitstream and the decoder may be verified using a hypothetical reference decoder (HRD). These standards may contain coding tools that help deal with transmission errors and losses, but the use of these tools in encoding may be optional, and the decoding process for erroneous bitstreams may not be specified.

[0060] A syntax element may be defined as a data element represented in a bitstream. A syntax structure may be defined as zero or more syntax elements present together in a specified order in a bitstream.

[0061] In most cases, the basic unit for encoder input and decoder output, respectively, is a picture. A picture given as input to an encoder may also be referred to as a source picture, and a picture decoded by a decoder may be referred to as a decoded picture or a reconstructed picture.

[0062] The source picture and the decoded picture each include one or more sample arrays, such as one of the following sample array sets:

[0063] - Luminance (Y) only (monochrome).

[0064] - Luma and two chrominances (YCbCr or YCgCo).

[0065] - Green, Blue and Red (GBR, also known as RGB).

[0066] - An array representing otherwise unspecified monochromatic or tristimulus color sampling (e.g., YZX, also called XYZ).

[0067] In the following, these arrays may be referred to as luma (or L or Y) and chroma, where the two chroma arrays may be referred to as Cb and Cr; regardless of the actual color representation method used. The actual color representation method used may be indicated, for example, in the encoded bitstream, such as using the Video Usability Information (VUI) syntax of HEVC, etc. A component may be defined as an array or a single sample from one of the three sample arrays (luminance and two chroma), or an array or a single sample of an array constituting a picture in a monochrome format.

[0068] A picture can be defined as a frame or a field. A frame consists of a matrix of luma samples and possibly corresponding chroma samples. A field is a set of alternating sample rows of a frame, and can be used as encoder input when the source signal is interlaced. The chroma sample array may not be present (and therefore monochrome sampling may be used), or the chroma sample array may be subsampled compared to the luma sample array.

[0069] Some chroma formats can be summarized as follows:

[0070] - In monochrome sampling, there is only one sample array, which nominally can be considered as a brightness array.

[0071] - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array.

[0072] - In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luma array.

[0073] - In 4:4:4 sampling, when separate color planes are not used, each of the two chroma arrays has the same height and width as the luma array.

[0074] The coding format or standard may allow encoding the sample arrays as separate color planes into the bitstream, and decoding the separate encoded color planes separately from the bitstream. When separate color planes are used, each of them is processed separately (by the encoder and / or decoder) as a picture with monochrome sampling.

[0075] When chroma subsampling is used (e.g., 4:2:0 or 4:2:2 chroma sampling), the position of chroma samples relative to luma samples may be determined on the encoder side (e.g., as a preprocessing step or as part of encoding). The chroma sample positions relative to luma sample positions may be predefined, for example, in a coding standard such as H.264 / AVC or HEVC, or may be indicated in the bitstream, for example as part of a VUI of H.264 / AVC or HEVC.

[0076] Typically, the (multiple) source video sequences provided as encoding input may represent interlaced source content or progressive source content. For interlaced source content, fields of opposite parity have been captured at different times. Progressive source content contains captured frames. The encoder can encode the fields of interlaced source content in two ways: a pair of interlaced fields can be encoded as a coded frame, or a field can be encoded as a coded field. Similarly, the encoder can encode the frames of progressive source content in two ways: the frames of progressive source content can be encoded as a coded frame or a pair of coded fields. A field pair or complementary field pair can be defined as two fields adjacent to each other in decoding and / or output order, which have opposite parity (i.e., one is the top field and the other is the bottom field) and do not belong to any other complementary field pair. Some video coding standards or schemes allow mixing coded frames and coded fields in the same coded video sequence. In addition, it is possible to implement the prediction of coded fields and / or the prediction of coded frames of complementary field pairs (coded as fields) from fields in coded frames in encoding and / or decoding.

[0077] Partitioning can be defined as dividing a set into subsets such that every element of the set is in exactly one of the subsets.

[0078] In the draft version of H.266 / VVC, the following partitioning applies. Note that what is described here may still develop in subsequent draft versions of H.266 / VVC before the standard is finalized. The picture is partitioned into coding tree units (CTUs) with a maximum size of 128×128, although the encoder can choose to use a smaller size, such as 64×64. The coding tree unit (CTU) is first partitioned by a quaternary tree (also called a quadtree) structure. The leaf nodes of the quadtree can then be further partitioned by a multi-type tree structure. There are four types of splitting in the multi-type tree structure, namely vertical binary splitting, horizontal binary splitting, vertical ternary splitting, and horizontal ternary splitting. The leaf nodes of the multi-type tree are called coding units (CUs). CU, PU, ​​and TU have the same block size unless the CU is too large for the maximum transform length. The segmentation structure of the CTU is a quadtree with nested multi-type trees, using binary and ternary splitting, that is, the concepts of separate CU, PU, ​​and TU are not used unless a CU that is too large for the maximum transform length is required. A CU may have a square or rectangular shape.

[0079] The decoder reconstructs the output video by applying a prediction component similar to the encoder to form a predicted representation of the pixel block (using the motion or spatial information created by the encoder and stored in a compressed representation) and applying prediction error decoding (the inverse operation of prediction error encoding to recover the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing it on for display and / or storing it as a prediction reference for upcoming frames in the video sequence.

[0080] The filtering may, for example, include one or more of: deblocking, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF).

[0081] The deblocking loop filter may include multiple filtering modes or strengths, which may be adaptively selected based on characteristics of blocks adjacent to the boundary (such as quantization parameter values ​​and / or signaling included in the bitstream by the encoder). For example, the deblocking loop filter may include a normal filtering mode and a strong filtering mode, which may differ in the number of filter taps (i.e., the number of samples filtered on both sides of the boundary) and / or the filter tap values. For example, filtering of two samples along both sides of the boundary may be performed using a filter with an impulse response of (379-3) / 16, while omitting the potential impact of a clipping operation.

[0082] Motion information can be indicated by a motion vector associated with each motion compensated image block in a video codec. Each of these motion vectors represents the displacement of a prediction source block in an image block in a picture to be encoded (on the encoder side) or to be decoded (on the decoder side) and a picture in a previously encoded or decoded picture. In order to efficiently represent motion vectors, these motion vectors can be differentially encoded relative to block-specific prediction motion vectors. Predicted motion vectors can be created in a predefined manner, such as calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another way to create a motion vector prediction is to generate a candidate prediction list from adjacent blocks and / or collocated blocks in a temporal reference picture, and signal the selected candidate as a motion vector predictor. In addition to predicting motion vector values, reference indexes of previously encoded / decoded pictures can also be predicted. Reference indexes can be predicted from adjacent blocks and / or collocated blocks in a temporal reference picture. In addition, efficient video codecs can employ additional motion information encoding / decoding mechanisms, often referred to as merge modes, in which all motion field information including motion vectors and corresponding reference picture indexes for each available reference picture list is predicted and used without any modification / correction. Similarly, predicting motion field information is performed using motion field information of neighboring blocks and / or collocated blocks in temporal reference pictures, and the used motion field information is signaled in a list of motion field candidate lists populated with motion field information of available neighboring / collocated blocks.

[0083] Video codecs can support motion compensated prediction from one source image (uni-prediction) and from two sources (bi-prediction). In the case of uni-prediction, a single motion vector is applied, while in the case of bi-prediction, two motion vectors are signaled and the motion compensated predictions from the two sources are averaged to create the final sample prediction. In the case of weighted prediction, the relative weights of the two predictions can be adjusted, or a signal offset can be added to the prediction signal.

[0084] In addition to applying motion compensation to inter-picture prediction, a similar approach can be applied to intra-picture prediction. In this case, the displacement vector indicates where a block of samples can be copied from the same picture to form a prediction for the block to be encoded or decoded. This intra-frame block copying approach can significantly improve coding efficiency in the presence of repetitive structures within a frame, such as text or other graphics.

[0085] The prediction residual after motion compensation or intra prediction can be transformed using a transform kernel (such as DCT) before encoding. The reason for this is that there is usually still some correlation between the residuals, and in many cases, the transform can help reduce this correlation and provide more efficient coding.

[0086] Video encoders can use Lagrangian cost functions to find the best coding mode, e.g., the desired macroblock mode and associated motion vectors. This cost function uses a weight factor λ to relate the (exact or estimated) image distortion caused by the lossy coding method to the (exact or estimated) amount of information required to represent the pixel values ​​in the image region:

[0087] C=D+λR (Equation 1) where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., mean square error) taking into account the mode and motion vectors, and R is the number of bits required to represent the data required to reconstruct the image block in the decoder (including the amount of data representing the candidate motion vectors).

[0088] Some codecs use the concept of picture order count (POC). The value of POC is obtained for each picture and does not decrease as the picture position in the output order increases. Therefore, POC indicates the output order of the pictures. POC can be used in the decoding process, for example, for implicit scaling of motion vectors and for reference picture list initialization. In addition, POC can be used in the verification of the consistency of the output order.

[0089] In video coding standards, a compliant bitstream must be able to be decoded by an assumed reference decoder, which can be conceptually connected to the output of the encoder and includes at least a pre-decoder buffer, a decoder, and an output / display unit. This virtual decoder can be called an assumed reference decoder (HRD) or a video buffer verifier (VBV). A stream is compliant if it can be decoded by the HRD without buffer overflow or, in some cases, without underflow. A buffer overflow occurs if more bits are put into the buffer when the buffer is full. A buffer underflow occurs if some bits are not in the buffer when some bits are to be taken from the buffer for decoding / playback. One of the motivations of HRD is to avoid so-called malicious bitstreams, which will consume a lot of resources and actual decoder implementations will not be able to handle.

[0090] The HRD model may include instantaneous decoding, and the input bitrate of the coded picture buffer (CPB) of the HRD may be viewed as a constraint on the decoding rate of the encoded data by the encoder and the bitstream, and a requirement on the processing rate by the decoder. The encoder may include the CPB specified in the HRD, which is used to verify and control whether the buffer constraints are adhered to in the encoding. The decoder implementation may also have a CPB that may, but need not, operate similarly or identically to the CPB specified for the HRD.

[0091] A decoded picture buffer (DPB) may be used in an encoder and / or decoder. There may be two reasons for buffering decoded pictures, namely, for reference in inter-frame prediction and for reordering decoded pictures to output order. Certain coding formats (such as HEVC) provide great flexibility for both reference picture marking and output reordering, and separate buffers for reference picture buffering and output picture buffering may waste memory resources. Therefore, the DPB may include a unified decoded picture buffering process for reference pictures and output reordering. When a decoded picture is no longer used as a reference and does not need to be output, the decoded picture can be removed from the DPB. The HRD may also include the DPB. The DPB implemented by the HRD and the decoder may, but need not, operate identically.

[0092] The output order may be defined as the order in which decoded pictures are output from the decoded picture buffer (for decoded pictures to be output from the decoded picture buffer).

[0093] The decoder and / or HRD may include a picture output process. The output process may be viewed as a process in which the decoder provides a decoded and cropped picture as an output of the decoding process. The output process may be part of a video coding standard, for example as part of a hypothetical reference decoder specification. In output cropping, rows and / or columns of samples may be removed from a decoded picture according to a cropping rectangle to form an output picture. A cropped decoded picture may be defined as the result of cropping a decoded picture based on a consistent cropping window specified, for example, in a sequence parameter set referenced by a corresponding codec picture.

[0094] One or more syntax structures for (decoded) reference picture marking may be present in a video coding system. An encoder generates an instance of the syntax structure, e.g., in each coded picture, and a decoder decodes an instance of the syntax structure, e.g., from each coded picture. For example, decoding of the syntax structure may enable a picture to be adaptively marked as "used for reference" or "not used for reference".

[0095] The reference picture set (RPS) syntax structure of HEVC is an example of a syntax structure for reference picture marking. The reference picture set that is valid or active for a picture includes all reference pictures that can be used as reference for the picture and all reference pictures that remain marked as "used for reference" for any subsequent pictures in decoding order. Reference pictures that remain marked as "used for reference" for any subsequent pictures in decoding order but are not used as reference pictures for the current picture or image slice may be considered inactive. For example, they may not be included in the initial reference picture list(s).

[0096] In some coding formats and codecs, a distinction is made between so-called short-term and long-term reference pictures. This distinction may affect some decoding processes, such as motion vector scaling. The syntax structure(s) used to mark reference pictures may indicate whether a picture is marked as "used for long-term reference" or "used for short-term reference".

[0097] In some coding formats, the reference picture used for inter prediction can be indicated by an index to a reference picture list. In some codecs, two reference picture lists (reference picture list 0 and reference picture list 1) are generated for each bidirectionally predicted (B) slice, and one reference picture list (reference picture list 0) is formed for each inter-coded (P) slice.

[0098] In VVC, the reference picture list is indicated directly in the reference picture list syntax structure. When a picture appears in any reference picture list of the current picture (in an active or inactive entry in any reference picture list), it is marked as "used for long-term reference" or "used for short-term reference". When a picture does not exist in the reference picture list of the current picture, it is marked as "unused for reference". The abbreviation RPL can be used to refer to the reference picture list syntax structure and / or one or more reference picture lists. The number of active entries in the reference picture list can be indicated by the encoder and / or decoded by the decoder, and can indicate the number of pictures that can be used as a reference for prediction of the current picture starting from the first list entry. Entries in the reference picture list that are not present in the active entry can be defined as inactive entries, are not used as references for prediction of the current picture, and can be used as references for prediction of subsequent pictures in decoding order.

[0099] A decoded picture buffer (DPB) may be used in an encoder and / or decoder. There may be two reasons for buffering decoded pictures, namely, for reference in inter-frame prediction and for reordering decoded pictures to output order. Since some codecs (such as VVC) provide flexibility for both reference picture marking and output reordering, separate buffers for reference picture buffering and output picture buffering may waste memory resources. Therefore, the DPB may include a unified decoded picture buffering process for reference pictures and output reordering. When a decoded picture is no longer used as a reference and does not need to be output, the decoded picture can be removed from the DPB.

[0100] Scalable video coding refers to a coding structure in which one bitstream can contain multiple representations of content at different bitrates, resolutions, or frame rates. In these cases, the receiver can extract the desired representation based on its characteristics (e.g., the resolution that best matches the display device). Alternatively, a server or network element can extract the portion of the bitstream to be sent to the receiver based on, for example, the network characteristics or processing capabilities of the receiver. A scalable bitstream can include a "base layer" that provides the lowest quality video available and one or more enhancement layers that enhance the video quality when received and decoded with lower layers. In order to improve the coding efficiency of the enhancement layer, the coded representation of the layer can depend on the lower layer. For example. The motion and pattern information of the enhancement layer can be predicted from the lower layer. Similarly, the pixel data of the lower layer can be used to create predictions for the enhancement layer.

[0101] A scalable video codec for quality scalability (also known as signal-to-noise ratio or SNR) and / or spatial scalability can be implemented as follows. For the base layer, a conventional non-scalable video encoder and decoder is used. The reconstructed / decoded pictures of the base layer are included in the reference picture buffer of the enhancement layer. In H.264 / AVC, HEVC and similar codecs that use (multiple) reference picture lists for inter-frame prediction, the base layer decoded pictures can be inserted into (multiple) reference picture lists for encoding / decoding of enhancement layer pictures, similar to the decoded reference pictures of the enhancement layer. Therefore, the encoder can select the base layer reference picture as an inter-frame prediction reference and indicate its use, for example using a reference picture index in the encoded bitstream. The decoder decodes the base layer picture from the bitstream (e.g., from the reference picture index) to be used as an inter-frame prediction reference for the enhancement layer. When the decoded base layer picture is used as a prediction reference for the enhancement layer, it is called an inter-layer reference picture.

[0102] Scalability patterns or scalability dimensions can include, but are not limited to, the following:

[0103] • Quality scalability: Base layer pictures are encoded with lower quality than enhancement layer pictures, which can be achieved, for example, by using quantization parameter values ​​in the base layer (ie, larger quantization step sizes for transform coefficient quantization) than in the enhancement layer.

[0104] • Spatial scalability: Base layer pictures are coded at a lower resolution (ie, fewer samples) than enhancement layer pictures. Spatial scalability and quality scalability can sometimes be considered the same type of scalability.

[0105] • Bit-depth scalability: Base layer pictures are encoded at a lower bit depth (eg, 8 bits) than enhancement layer pictures (eg, 10 or 12 bits).

[0106] • Dynamic range scalability: Scalable layers represent different dynamic ranges and / or images acquired using different tone mapping functions and / or different optical transfer functions.

[0107] • Chroma format scalability: Base layer pictures provide lower spatial resolution in chroma sample arrays (eg, encoded in 4:2:0 chroma format) than enhancement layer pictures (eg, 4:4:4 format).

[0108] Color gamut scalability: Enhancement layer pictures have a richer / wider range of color representation than base layer pictures, e.g., the enhancement layer may have the UHDTV (ITU-R BT.2020) color gamut while the base layer may have the ITU-R BT.709 color gamut.

[0109] Region of Interest (ROI) Scalability: The enhancement layer represents a spatial subset of the base layer. ROI scalability can be used together with other types of scalability, such as quality or spatial scalability, so that the enhancement layer provides higher subjective quality for the spatial subset.

[0110] • View scalability, also known as multi-view coding. The base layer represents the first view, while the enhancement layer represents the second view.

[0111] Depth scalability, also known as depth enhancement coding: One or some layers of the bitstream may represent (multiple) texture views, while other layers or layers may represent (multiple) depth views.

[0112] In all the above scalability cases, the base layer information can be used to encode the enhancement layers to minimize the additional bitrate overhead.

[0113] Scalability can be enabled in two basic ways: by introducing new coding modes to perform prediction of pixel values ​​or syntax from lower layers of the scalable representation, or by placing lower layer pictures into a reference picture buffer (Decoded Picture Buffer DPB) of a higher layer.

[0114] The first approach is more flexible and thus can provide better coding efficiency in most cases. However, the second approach of reference frame based scalability can be implemented very efficiently with minimal changes to the single layer codec while still achieving most of the available coding efficiency gains. Essentially, the reference frame based scalability codec can be implemented by using the same hardware or software implementation for all layers, with only attention paid to DPB management.

[0115] The basic unit for the output of encoders of some coding formats (such as HEVC and VVC) and the input of decoders of some coding formats (such as HEVC and VVC) is the Network Abstraction Layer (NAL) unit. For transmission over a packet-oriented network or storage in a structured file, the NAL unit can be encapsulated into packets or similar structures.

[0116] A byte stream format may be specified for NAL unit streams in a transport or storage environment that does not provide a frame structure. The byte stream format separates NAL units from each other by prepending a start code in front of each NAL unit. To avoid false detection of NAL unit boundaries, an encoder may run a byte-oriented start code emulation prevention algorithm that adds emulation prevention bytes to the NAL unit payload if a start code would otherwise be present. To enable straightforward gateway operation between packet-oriented and stream-oriented systems, start code emulation prevention may always be performed, regardless of whether the byte stream format is used.

[0117] A NAL unit may be defined as a syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of a Raw Byte Sequence Payload (RBSP), interspersed with emulation prevention bytes as necessary. An RBSP may be defined as a syntax structure containing an integer number of bytes encapsulated in a NAL unit. An RBSP is either empty or in the form of a string of data bits containing the syntax elements, followed by an RBSP stop bit, and then zero or more subsequent bits equal to 0.

[0118] The NAL unit includes a header and a payload. In VVC, a two-byte NAL unit header is used for all specified NAL unit types, while in other codecs, the NAL unit header can be similar to that in VVC.

[0119] In VVC, the NAL unit header includes a five-bit NAL unit type indication (nal_unit_type), a three-bit nuh_temporal_id_plus1 indication for the temporal level or sublayer (which may need to be greater than or equal to 1), and a six-bit nuh_layer_id syntax element. The nuh_temporal_id_plus1 syntax element can be regarded as a temporal identifier for the NAL unit, and the zero-based TemporalId variable can be derived as follows: TemporalId = nuh_temporal_id_plus1-1. The abbreviation TID can be used interchangeably with the TemporalId variable. TemporalId equal to 0 corresponds to the lowest temporal level. The value of nuh_temporal_id_plus1 must be non-zero to avoid start code emulation involving two NAL unit header bytes. The bitstream created by excluding all VCL NAL units with a TemporalId greater than or equal to the selected value and including all other VCL NAL units is consistent. Therefore, a picture with TemporalId equal to tid_value does not use any picture with TemporalId greater than tid_value as an inter-frame prediction reference. A sublayer or temporal sublayer can be defined as a temporal scalable layer (or temporal layer TL) of a temporal scalable bitstream. Such a temporal scalable layer may include VCL NAL units and associated non-VCL NAL units with a specific value of the TemporalId variable. nuh_layer_id can be understood as a scalable layer identifier.

[0120] NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units can be coded slice NAL units. In HEVC and VVC, VCL NAL units contain syntax elements representing one or more CUs. In HEVC and VVC, NAL unit type values ​​within a certain range indicate VCL NAL units, and VCL NAL unit types can indicate picture types.

[0121] An image can be split into independently encodable and decodable image segments (e.g., slices or tiles or tile groups). Such image segments can be processed in parallel. A "slice" in this specification may refer to an image segment consisting of a certain number of basic coding units processed in a default encoding or decoding order, and a "tile" may refer to an image segment that has been defined as a rectangular image area. A tile group may be defined as a group of one or more tiles. Image segments can be encoded as separate units in a bitstream, such as VCL NAL units in HEVC and VVC. The encoded image segments may include a header and a payload, wherein the header contains parameter values ​​required to decode the payload.

[0122] In the HEVC standard, a picture can be partitioned into tiles, which are rectangular and contain an integer number of CTUs. In the HEVC standard, the partitioning into tiles forms a grid, which is characterized by a list of tile column widths (in CTUs) and a list of tile row heights (in CTUs). Slices are ordered consecutively in the bitstream in the raster scan order of the tile grid. A tile can contain an integer number of slices.

[0123] In HEVC, a slice includes an integer number of CTUs. If tiles are not used, the CTUs are scanned in the raster scan order of CTUs within a tile or within a picture. A slice may contain an integer number of tiles, or a slice may be contained in a tile. Within a CTU, a CU has a specific scan order.

[0124] In HEVC, a slice is defined as an integer number of coding tree units contained in an independent slice segment and all subsequent dependent slice segments (if any) before the next independent slice segment (if any) within the same access unit. In HEVC, a slice segment is defined as an integer number of coding tree units that are ordered consecutively in a tile scan and contained in a single NAL (Network Abstraction Layer) unit. The division of each picture into slice segments is a partitioning. In HEVC, an independent slice segment is defined as a slice segment whose values ​​of the syntax elements of the slice segment header are not inferred from the values ​​of the previous slice segment, and a dependent slice segment is defined as a slice segment whose values ​​of some syntax elements of the slice segment header are inferred from the values ​​of the previous slice segment in decoding order. In HEVC, a slice header is defined as a slice segment header of an independent slice segment, which is the current slice segment or the independent slice segment before the current dependent slice segment, and a slice segment header is defined as a portion of a coded slice segment containing data elements related to the first or all coding tree units represented in the slice segment. If tiles are not used, the CU is scanned in the raster scan order of the LCU within the tile or within the picture. Within an LCU, CUs have a specific scanning order.

[0125] In the draft version of VVC, the partitioning of a picture into tiles is defined as follows. A picture is divided into one or more tile rows and one or more tile columns. The partitioning of a picture into tiles forms a tile grid characterized by a list of tile column widths (in CTUs) and a list of tile row heights (in CTUs). A tile is a sequence of coding tree units (CTUs) that covers a "cell" in the tile grid, i.e., a rectangular area of ​​a picture.

[0126] In the draft version of VVC, the partitioning of a picture into slices is defined as follows. Two slice modes are supported, namely, raster scan slice mode and rectangular slice mode. In raster scan slice mode, a slice contains a sequence of tiles in a raster scan of tiles of a picture. In rectangular slice mode, a slice contains either an integer number of complete tiles that together form a rectangular region of a picture, or an integer number of complete CTU rows of tiles. A slice is a VCL NAL unit.

[0127] In the draft version of VVC, a slice (also called a coded slice) may include a slice header and slice data. A slice header may be defined as a portion of a coded slice that contains data elements related to all tiles or CTU rows within a tile represented in the slice.

[0128] The motion constrained tile set (MCTS) constrains the inter prediction process in encoding so that there are no sample values ​​outside the motion constrained tile set, and no sample values ​​are used for inter prediction of any sample in the motion constrained tile set at the fractional sample position obtained using one or more sample values ​​outside the motion constrained tile set. In addition, the encoding of the MCTS is constrained so that the candidate motion vector is not obtained from a block outside the MCTS. This can be enforced by turning off the temporal motion vector prediction of HEVC, or prohibiting the encoder from using the TMVP candidate or any motion vector prediction candidate after the TMVP candidate in the merge or AMVP candidate list for the PU located to the left of the right tile boundary of the MCTS, except for the last PU in the lower right corner of the MCTS. In general, an MCTS can be defined as a tile set that is independent of any sample values ​​and coded data (such as motion vectors) outside the MCTS. An MCTS sequence can be defined as a sequence of corresponding MCTSs in one or more coded video sequences, etc. In some cases, MCTS may be required to form a rectangular area. It should be understood that, depending on the context, MCTS can refer to a tile set within a picture or a corresponding tile set in a picture sequence. The corresponding tile sets may, but typically do not have to, be collocated in a picture sequence. A motion limited tile set may be considered an independently coded tile set, as it may be decoded without other tile sets.

[0129] Note that sample positions used in inter prediction may be saturated by the encoding and / or decoding process such that positions that were originally outside of the picture are saturated to point to corresponding boundary samples of the picture. Therefore, if a tile boundary is also a picture boundary, then in some use cases the encoder may allow motion vectors to effectively cross that boundary, or allow motion vectors to effectively result in fractional sample interpolation that will reference positions outside that boundary because the sample positions are saturated onto the boundary. In other use cases, specifically if a coded tile can be extracted from a bitstream where it is located adjacent to a picture boundary to another bitstream where the tile is located not adjacent to a picture boundary, then the encoder may constrain motion vectors to be on a picture boundary, similar to any MCTS boundary.

[0130] The VVC draft standard supports sub-pictures (also called sub-pictures). A sub-picture can be defined as a rectangular area of ​​one or more slices within a picture, where one or more slices are complete. Therefore, a sub-picture includes one or more slices that collectively cover a rectangular area of ​​a picture. The slices of a sub-picture may need to be rectangular slices. The partitioning of a picture into sub-pictures (also called sub-picture layout or layout of sub-pictures) can be indicated in the SPS and / or decoded from the SPS. One or more of the following properties can be indicated (e.g., by an encoder) or decoded (e.g., by a decoder) or inferred (e.g., by an encoder and / or decoder) for sub-pictures collectively or individually for each sub-picture: i) whether the sub-picture is treated as a picture during decoding; in some cases, the property does not include loop filtering operations that can be separately indicated / decoded / inferred; ii) whether loop filtering operations are performed across sub-picture boundaries. Treating a sub-picture as a picture during decoding can include saturating sample positions in inter-frame prediction that would otherwise be outside the sub-picture to the sub-picture boundary.

[0131] It is to be understood that the embodiments described with reference to MCTS may also be implemented using sub-pictures (as specified in the VVC draft standard), and the embodiments described with reference to sub-pictures may also be implemented using MCTS (as described above).

[0132] A non-VCL NAL unit may be, for example, one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of bitstream NAL unit, or a filler data NAL unit. A parameter set may be required to reconstruct a decoded picture, whereas many other non-VCL NAL units are not required to reconstruct decoded sample values.

[0133] Some coding formats specify parameter sets that can carry parameter values ​​required for decoding or reconstructing decoded pictures. Parameters that remain unchanged through the encoded video sequence can be included in a sequence parameter set (SPS). If the CVS contains multiple layers, the SPS may be valid only for a subset of these layers. In addition to the parameters that may be required for the decoding process, the sequence parameter set may optionally include video availability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation. The picture parameter set (PPS) contains parameters that may not change in several coded pictures. The picture parameter set may include parameters that can be referenced by coded image fragments of one or more coded pictures.

[0134] A decoding parameter set (DPS) may carry parameters applied to a bitstream. A DPS may be specified to contain attributes and / or constraints that are not required for decoding a VCL NAL unit. A video parameter set (VPS) may carry parameters that are commonly applied to multiple layers. An adaptive parameter set (APS) may be defined as a syntax structure applicable to zero or more slices. There may be different types of adaptation parameter sets. An adaptation parameter set may, for example, contain filtering parameters for a particular type of filter. In the VVC draft standard, three types of APS are specified as parameters that carry one of the following: an adaptive loop filter (ALF), a luminance map with chroma scaling (LMCS), and a scaling list. A scaling list may be defined as a list that associates each frequency index with a scaling factor for a scaling process that multiplies a transform coefficient level by a scaling factor to obtain a transform coefficient.

[0135] A parameter set may be activated when referenced, for example by its identifier. For example, a header of an image segment (such as a slice header) may contain an identifier of a PPS activated for decoding a codec picture containing the image segment. When the PPS is activated, the PPS may contain an identifier of an activated SPS. Activation of a parameter set of a particular type may result in deactivation of a previously activated parameter set of the same type. In addition to or in lieu of activating a parameter set, a parameter set may be referenced or referred to by a syntactic structure containing an identifier of the parameter set. For example, a slice header may reference a PPS whose identifier is included in a slice header and an SPS whose identifier is included in the referenced PPS.

[0136] In the VVC draft standard, the PPS syntax includes a 1-bit syntax element (i.e., flag) named mixed_nalu_types_in_pic_flag. When equal to 1, mixed_nalu_types_in_pic_flag specifies that each picture that references the PPS has more than one VCL NAL unit, and the VCL NAL units do not have the same nal_unit_type value, and the picture is not an IRAP picture. mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture that references the PPS has one or more VCL NAL units, and the VCL NAL units of each picture that references the PPS have the same nal_unit_type value.

[0137] Instead of or in addition to parameter sets at different hierarchical levels (e.g., sequence and picture), the video coding format may include a header syntax structure, such as a sequence header or a picture header. A sequence header may precede any other data of the coded video sequence in bitstream order. A picture header may precede any coded video data of a picture in bitstream order.

[0138] In VVC, a picture header (PH) can be defined as a syntax structure containing syntax elements that apply to all slices of a coded picture. In other words, it contains information common to all slices of a coded picture associated with the PH. The picture header syntax structure is specified as an RBSP and is included in a NAL unit.

[0139] The phrases along a bitstream (e.g., indicated along a bitstream) or along a coding unit of a bitstream (e.g., indicated along a coding tile) may be used in the claims and described embodiments to refer to transmission, signaling, or storage in such a way that "out-of-band" data is associated with the bitstream or coding unit, respectively, but is not included therein. The phrases decoding along a bitstream or along a coding unit of a bitstream, etc. may refer to decoding the referenced out-of-band data associated with the bitstream or coding unit, respectively (which may be obtained from an out-of-band transmission, signaling, or storage). For example, the phrase along a bitstream may be used when the bitstream is contained in a container file, such as a file conforming to the ISO base media file format, and certain file metadata is stored in the file in such a way that the metadata is associated with the bitstream, such as a box in a sample entry of a track containing the bitstream, a sample group of a track containing the bitstream, or a timed metadata track associated with a track containing the bitstream.

[0140] A codec picture is an encoded representation of a picture.

[0141] An access unit may include coded video data and associated other data for a single time instance. In VVC, an access unit (AU) may be defined as a group of picture units (PUs) that belong to different layers and contain coded pictures associated with the same time for output from a decoded picture buffer (DPB). In VVC, a picture unit (PU) may be defined as a group of NAL units that are associated with each other according to specified classification rules, are continuous in decoding order, and contain only one coded picture. In addition to VCL NAL units containing coded pictures, picture units may also contain non-VCL NAL units.

[0142] Codec pictures may be required to appear in a specific order within an access unit. For example, a codec picture with nuh_layer_id equal to nuhLayerIdA may need to precede all codec pictures with nuh_layer_id greater than nuhLayerIdA in the same access unit in decoding order.

[0143] A bitstream may be defined as a sequence of bits, which in some coding formats or standards may be in the form of a NAL unit stream or a byte stream that forms a representation of a coded picture and associated data that forms one or more coded video sequences. The first bitstream may be followed by a second bitstream in the same logical channel, such as in the same file or in the same connection of a communication protocol. An elementary stream (in the context of video coding) may be defined as a sequence of one or more bitstreams. In some coding formats or standards, the end of the first bitstream may be indicated by a specific NAL unit, which may be referred to as an end-of-bitstream (EOB) NAL unit and is the last NAL unit of the bitstream.

[0144] A coded video sequence (CVS) may be defined as a sequence of coded pictures in decoding order that is independently decodable and followed by another coded video sequence or the end of a bitstream.

[0145] A coded layer video sequence (CLVS) may be defined as a sequence of picture units (PUs) with the same scalable layer (e.g., with the same nuh_layer_id value in VVC), which in decoding order includes a CLVS start picture unit (CLVSSPU), followed by zero or more PUs that are not CLVSS PUs, including all subsequent PUs, but excluding any subsequent PU that is a CLVSS PU. A picture unit (PU) may be defined as a codec picture and all non-VCL NAL units associated with a codec picture. A CLVSS PU may be defined as a PU that is allowed to start a CLVS, i.e., a PU that can start the decoding process of a layer. A CLVSS PU may, for example, include an IRAP picture or a progressive decoding refresh (GDR) picture.

[0146] A group of pictures (GOP) and its characteristics can be defined as follows. A GOP can be decoded regardless of whether any previous pictures have been decoded. An open GOP is a group of pictures where, when decoding starts from the initial intra picture of an open GOP, pictures preceding the initial intra picture in output order may not be decoded correctly. In other words, pictures of an open GOP can refer (in inter prediction) to pictures belonging to a previous GOP. A HEVC decoder or a VVC decoder can identify an intra picture that starts an open GOP because a specific NAL unit type, a CRA NAL unit type, can be used for its coded slices. A closed GOP is a group of pictures where all pictures can be decoded correctly when decoding starts from the initial intra picture of a closed GOP. In other words, no picture in a closed GOP references any picture in a previous GOP. An open GOP coding structure may be more efficient in compression than a closed GOP coding structure because it has greater flexibility in selecting reference pictures.

[0147] When describing the video codec and example embodiments, the following description may be used to specify the writing and / or parsing process of each syntax element:

[0148] -u(n): Use an unsigned integer of n bits. When n is "v" in the syntax table, the number of bits varies in a way that depends on the values ​​of other syntax elements. The pairing process for this descriptor is specified by the n next bits in the bitstream, which are interpreted as the binary representation of the unsigned integer, with the most significant bit written first.

[0149] -ue(v): Unsigned integer exponential-Golomb coding (also called exp-Golom coding) syntax element, written with the left bit first.

[0150] For example, the following table can be used to convert an Exponential Golomb bit string to a code number (codeNum):

[0151] Bit string codeNum 1 0 0 1 0 1 0 1 1 2 0 0 1 0 0 3 0 0 1 0 1 4 0 0 1 1 0 5 0 0 1 1 1 6 0 0 0 1 0 0 0 7 0 0 0 1 0 0 1 8 0 0 0 1 0 1 0 9 … …

[0152] Available media file format standards include ISO Base Media File Format (ISO / IEC 14496-12, which can be abbreviated as ISOBMFF), MPEG-4 File Format (ISO / IEC 14496-14, also known as MP4 format), NAL Unit Structured Video File Format (ISO / IEC 14496-15) and 3GPP File Format (3GPP TS 26.244, also known as 3GP format). ISO file format is the basis for obtaining all the above file formats (excluding ISO file format itself). These file formats (including ISO file format itself) are generally referred to as ISO file format family.

[0153] Some concepts, structures and specifications of ISOBMFF are described below as an example of a container file format, based on which embodiments can be implemented. Aspects of the present invention are not limited to ISOBMFF, but are described for one possible basis on which the present invention can be partially or completely implemented.

[0154] The basic building block in the ISO base media file format is called a box. Each box has a header and a payload. The box header indicates the type of box and the size of the box in bytes. The type of box can be represented by a 32-bit unsigned integer or equivalently by a four-character code (4CC) that can be written in quotes, such as "mdat". Boxes can contain other boxes, and the ISO file format specifies which box types are allowed in a particular type of box. In addition, the presence of certain boxes in each file may be mandatory, while the presence of other boxes may be optional. In addition, for certain box types, multiple boxes may be allowed to exist in a file. Therefore, the ISO base media file format can be considered to specify a hierarchy of boxes. According to ISOBMFF, a file includes metadata encapsulated in a box and may also include media data encapsulated in a box. The media data may alternatively exist in (multiple) other files referenced by a file that conforms to ISOBMFF. A FullBox is a box that additionally contains an 8-bit version field and a 24-bit flag field in its box header. The syntax of a box may be specified using the Syntax Description Language (SDL) defined in ISO / IEC 14496-1.

[0155] In files conforming to the ISO Base Media File Format, media data may be provided in a media data "mdat" box (also called MediaDataBox). A file conforming to ISOBMFF may contain zero or more "mdat" boxes.

[0156] An Identified Media Data Box (also referred to as IdentifiedMediaDataBox, "imda") may have the same semantics as a MediaDataBox, but it may also contain an identifier for setting a data reference to the contained media data. The identifier may, for example, be the first element contained by the Identified Media Data Box. The syntax of an Identified Media Data Box may be specified as follows, where imda_identifier is an identifier of the box. Note that while a 32-bit unsigned integer type of imda_identifier is used in the syntax, other field lengths and other primitive data types (e.g., strings) are possible in similar Identified Media Data Box structures. The syntax of an Identified Metadata Box is provided below:

[0157]

[0158] This description also applies to IdentifiedMediaDataBox wherever a MediaDataBox, an "mdat" box, or a MediaDataBox is referenced.

[0159] In files conforming to the ISO base media file format, a movie "moov" box (also called MovieBox) may be used to encapsulate metadata. In some cases, both media data and movie boxes may need to exist in order for the file to be operable. A movie "moov" box may include one or more tracks, and each track may reside in a corresponding TrackBox ("trak"). A track may be one of multiple types, including a media track, which refers to samples formatted according to a media compression format (and its encapsulation of the ISO base media file format). A track may be viewed as a logical channel. TrackBox contains a TrackHeaderBox, which contains a track identifier, i.e., a track_ID syntax element, which is an integer that uniquely identifies the track throughout the life cycle of the presentation.

[0160] Movie fragments may be used, for example, when recording content to an ISO file, for example, to avoid data loss when the recording application crashes, memory space is insufficient, or other events occur. Without movie fragments, data loss may occur because the file format may require that all metadata (e.g., movie boxes) be written to one contiguous area of ​​the file. In addition, when recording a file, there may not be enough memory space (e.g., random access memory RAM) to buffer the available storage size of the movie box, and to recalculate the contents of the movie box when the movie may be closed too slowly. In addition, movie fragments may allow the file to be recorded and played back simultaneously using a conventional ISO file parser. In addition, progressive download may require a shorter initial buffering duration, for example, when receiving and playing a file simultaneously when movie fragments are used, and the initial movie box is smaller than a file with the same media content but structured without movie fragments.

[0161] The movie fragment feature may allow metadata that may otherwise reside in the movie box to be split into multiple fragments. Each fragment may correspond to a certain time segment of a track. In other words, the movie fragment feature may enable the interleaving of file metadata and media data. As a result, the size of the movie box may be limited and the above-mentioned use cases may be enabled.

[0162] In some examples, the media samples for a movie fragment may reside in an mdat box when located in the same file as the moov box. However, for metadata for the movie fragment, a "moof" box may be provided. The "moof" box may include information about a specific playback time duration that was previously in the "moov" box. The moov box itself may still represent a valid movie, but in addition, it may include an "mvex" box to indicate that the movie fragment will follow in the same file. The movie fragment may extend the presentation associated with the "moov" box in time.

[0163] There may be a set of track fragments within a movie fragment, including from zero to multiple track fragments per track. A track fragment may include from zero to multiple track runs (also called track fragment runs), where each document is a continuous sample run of the track. In these structures, many fields are optional and can be set by default. The metadata that can be included in the "moof" box may be limited to a subset of the metadata that can be included in the moov box, and in some cases may be encoded in different ways. Detailed information about the boxes that can be included in the "moof box" can be found in the ISO basic media file format specification. A self-contained movie fragment can be defined as including a "moof" box and an mdat box that are continuous in file order, and wherein the mdat box contains samples of the movie fragment (the "moof" box provides its metadata) and does not contain samples of any other movie fragments (i.e., any other "moof" box).

[0164] TrackBox and track fragments (in TrackFragmentBox) include decoding and synthesis time information of samples within the scope of TrackBox and track fragment, respectively. The decoding time indicates the time when the sample is intended to be decoded, while the synthesis time indicates the time when the sample is intended to be played. Two samples in different tracks can be considered to be time-aligned when their decoding time or synthesis time is the same. The term time alignment can refer to one or both of the alignment of decoding time and / or synthesis time. Sometimes, the term time alignment can refer to the alignment of decoding time only.

[0165] A track reference mechanism can be used to associate tracks with each other. TrackReferenceBox consists of box(es), each of which provides a reference from the contained track to a group of other tracks identified by their track_ID value(s) or, as described below, by their track_group_id value(s). These references are marked by the box type of the contained box(es) (i.e., the four-character code of the box).

[0166] A TrackGroupBox contained in a TrackBox can indicate groups of tracks where each group shares specific characteristics or the tracks within the group have specific relationships. A box contains zero or more boxes, and the specific characteristics or relationships are indicated by the box types of the contained boxes. The contained boxes are of type TrackGroupTypeBox or are derived from TrackGroupTypeBox. TrackGroupTypeBox is similar to FullBox, but also includes an identifier that can be used to summarize tracks that belong to the same track group. Tracks that contain contained boxes of the same type in a TrackGroupBox and have the same identifier values ​​within those contained boxes belong to the same track group.

[0167] The ISO base media file format contains three timed metadata mechanisms that can be associated with a particular sample: sample groups, timed metadata tracks, and sample auxiliary information. Resulting specifications may provide similar functionality through one or more of these three mechanisms.

[0168] Sample grouping in the ISO base media file format and the resulting files thereof may be defined as assigning each sample in a track as a member of a sample group based on a grouping criterion. Sample groups in a sample grouping are not limited to consecutive samples, but may also contain non-adjacent samples. Since there may be more than one sample grouping for samples in a track, each sample grouping may have a type field to indicate the type of grouping. Sample grouping may be represented by two linked data structures: (1) SampleToGroupBox ("sbgp" box) represents the assignment of samples to sample groups; and (2) SampleGroupDescriptionBox ("sgpd" box) contains a sample group entry for each sample group describing the properties of the group. SampleToGroupBox and SampleGroupDescriptionBox may have multiple instances based on different grouping criteria. These may be distinguished by a type field used to indicate the type of grouping. SampleToGroupBox may include a grouping_type_parameter field, which may be used, for example, to indicate a subtype of the grouping.

[0169] Mapping a sample to a sample group description index equal to 0 indicates that the sample is not a member of any group of this type. If the sum of the sample counts in the SampleToGroupBox is less than the total sample count, or there is no SampleToGroupBox applied to some samples (e.g., it is not present in the track fragment), then these samples are associated with the group identified by the default_group_description_index in the SampleGroupDescriptionBox (if any), otherwise not associated with any group. Therefore, the default sample group description index (i.e., the default_group_description_index in the SampleGroupDescriptionBox) specifies the index of the sample group description entry that applies to all samples in the track for which no sample-to-group mapping is provided via the SampleToGroupBox. When the default_group_description_index is not present in the SampleGroupDescriptionBox, it is inferred to be equal to 0 (indicating that the sample is not mapped to a group description of this type).

[0170] Several types of stream access points (SAPs) have been specified in ISOBMFF, including the following. SAP type 1 corresponds to what is known in some coding schemes as a "closed GOP random access point" (where all pictures are correctly decodable in decoding order, resulting in a continuous temporal sequence of correctly decoded pictures without gaps), and in addition, the first picture in the decoding order is also the first picture in the presentation order. SAP type 2 corresponds to what is known in some coding schemes as a "closed GOP random access point" (where all pictures are correctly decodable in decoding order, resulting in a continuous temporal sequence of correctly decoded pictures without gaps), where the first picture in the decoding order may not be the first picture in the presentation order. SAP type 3 corresponds to what is known in some coding schemes as an "open GOP random access point", where there may be some pictures in the decoding order that cannot be correctly decoded and have a presentation time less than the intra-coded pictures associated with the SAP.

[0171] Stream access points (also or alternatively referred to as layer access points) for layered coding may similarly be defined in a layer-by-layer manner. The SAP of a layer may be defined as a position in a layer (or the like) that enables playback of the layer to start using only information starting at that position, assuming that the reference layers of the layer have been decoded earlier.

[0172] The stream access point (SAP) sample group specified in ISOBMFF identifies the sample as a specified SAP type. The grouping_type_parameter of the SAP sample group includes the fields target_layers and layer_id_method_idc. target_layers specifies the target layer of the indicated SAP. The semantics of target_layers may depend on the value of layer_id_method_idc. layer_id_method_idc specifies the semantics of target_layers. layer_id_method_idc equal to 0 specifies that the target layer includes all layers represented by the track. The sample group description entry of the SAP sample group includes the fields dependent_flag and SAP_type. For non-layered media, dependent_flag may need to be 0. dependent_flag equal to 1 specifies that the reference layer (if any) used to predict the target layer may have to be decoded to access the samples of the sample group. dependent_flag equal to 0 specifies that the reference layer (if any) used to predict the target layer does not need to be decoded to access any SAP of the sample group. The sap_type value in the range of 1 to 6 (inclusive) specifies the SAP type of the associated sample.

[0173] A sync sample may be defined as a sample corresponding to either SAP type 1 or SAP type 2. A sync sample may be considered a media sample that starts a new independent sequence of samples; if decoding starts from a sync sample, then it and subsequent samples in decoding order may all be decoded correctly, and the resulting set of decoded samples forms a correct representation of the media starting with the decoded sample with the earliest composition time. Sync samples may be indicated using a SyncSampleBox (for samples whose metadata is present in a TrackBox) or in a sample flag indicated or inferred for a track fragment run (more specifically the sample_is_non_sync_sample flag).

[0174] The draft revision of ISO / IEC 14496-15 includes the specification of alternative extraction ("alte") track groups. Members of an "alte" track group are alternatives for use as extraction sources. For HEVC, track group members with track_group_type equal to "alte" may be defined as alternatives for use as a source for a "scal" or "sabt" track reference. File writers may indicate in a file that an "alte" track group contains tracks that may be used as alternatives for extraction sources.

[0175] The identifier of the 'alte' track group may be taken from the same numbering space as the track identifiers. In other words, the identifier of the 'alte' track group may need to be different from all track identifier values. Thus, the 'alte' track group identifier may be used where track identifiers are normally used. In particular, the 'alte' track group identifier may be used as a track reference indicating a source for extraction. The value of (flag&1) may be set equal to 1 in a TrackGroupTypeBox of type 'alte' to indicate the uniqueness of the track_group_id as specified in ISO / IEC 14496-12.

[0176] In addition to or instead of the track ID values, a TrackReferenceTypeBox with reference_type equal to track_ref_4cc may list the track_group_id values ​​of the "alte" track group(s) containing the same alter_track_ref_4cc value. For example, an extractor track may point to the "alte" track group via a "scal" track reference, in addition to or instead of the individual tracks. Any individual track of the "alte" track group is a suitable source for extraction. A player or file reader etc. may change the source track for extraction at the point where the track being switched to has a sync sample or a SAP sample of type 1 or 2.

[0177] Files conforming to ISOBMFF may contain any non-timed object, called an item, meta-item, or metadata item, in a meta-box (four-character code: "meta"). Although the name of the meta-box refers to metadata, an item may contain metadata or media data in general. Meta-boxes may be located at the top level of a file, within a movie box (four-character code: "moov"), and within a track box (four-character code: "trak"), but at most one meta-box may occur at each file level, movie level, or track level. A meta-box may need to contain a HandlerBox ("hdlr") box to indicate the structure or format of the contents of the "meta" box. A meta-box may list and characterize any number of items that may be referenced, and each item may be associated with a file name and uniquely identified with the file by an item identifier (item_id), which is an integer value. Metadata items may be stored, for example, in an item data box ("idat") box of a metadata box or in an "mdat" box or reside in a separate file. If the metadata is located outside the file, its location may be declared by a DataInformationBox (four-character code: "dinf"). In the specific case where metadata is formatted using the Scalable Markup Language (XML) syntax and needs to be stored directly in the MetaBox, the metadata can be encapsulated into an XMLBox (four-character code: "xml") or a BinaryXMLBox (four-character code: "bxml"). Items can be stored as continuous byte ranges, or can be stored in several ranges, each of which is a continuous byte range. In other words, items can be stored in segments as ranges, for example to enable interleaving. A range is a continuous subset of resource bytes. Resources can be formed by cascading ranges. The ItemPropertiesBox can associate any item with a set of ordered item properties. Item properties can be considered as small data records. The ItemPropertiesBox consists of two parts: an ItemPropertyContainerBox that contains an implicitly indexed list of item properties, and one or more ItemPropertyAssociationBoxes that associate items with item properties.

[0178] Entity grouping is similar to track grouping, but both tracks and items (e.g., image items) can be grouped in the same group. The syntax for entity grouping can be specified as follows:

[0179]

[0180] group_id is a non-negative integer assigned to a particular grouping, and may not be equal to any group_id value of any other EntityToGroupBox, any item_ID value of the hierarchy level (file, movie, or track) containing the GroupsListBox (containing the EntityToGroupBox), or any track_ID value (when the GroupsListBox is contained in the file level). num_entities_in_group specifies the number of entity_id values ​​mapped to this entity group. entity_id is resolved to an item when an item with item_ID equal to entity_id exists in the hierarchy level (file, movie, or track) containing the GroupsListBox (containing the EntityToGroupBox), or to a track when a track with track_ID equal to entity_id exists and the GroupsListBox is contained in the file level.

[0181] High Efficiency Image File Format (HEIF) is a standard developed by the Moving Picture Experts Group (MPEG) for storing images and image sequences. HEIF includes features built on top of the ISO Base Media File Format (ISOBMFF) used. The ISOBMFF structure and features are largely used in the design of HEIF. The basic design of HEIF includes still images stored as items and image sequences stored as tracks.

[0182] In the context of HEIF, the following boxes may be included within the root-level "meta" box and may be used as described below. In HEIF, the handler value for the Handler box of the "meta" box is "pict". Resources containing coded media data (whether within the same file, or in an external file identified by a uniform resource identifier) ​​are broken down via data information ("dinf") boxes, while item location ("iloc") boxes store the location and size of each item in the referenced file. The item reference ("iref") box records the relationship between items using typed references. If there is an item in a collection of items that is considered to be most important in some way than the others, that item is signaled by the primary item ("pitm") box. In addition to the boxes mentioned here, the "meta" box has the flexibility to include other boxes as needed to describe the item.

[0183] HEIF supports derived image items. An item is a derived image item when it includes a 'dimg' item reference to another item. A derived image is obtained by performing a specified operation (also called an image operation), such as a rotation, on a specified input image. The operation performed to obtain the derived image is identified by the item_type of the item. An image item used as input to the derived image may be an encoded image, or may be another derived image item.

[0184] Any number of image items may be included in the same file. Given a collection of images stored using the "meta" box method, it is sometimes necessary to define certain relationships between the images. Examples of such relationships include indicating a cover image for the collection, providing thumbnails for some or all images in a collection, and associating some or all images in a collection with auxiliary images such as alpha planes. A cover image in a collection of images is indicated using the "pitm" box. Thumbnails or auxiliary images are linked to the main image item using an item reference of type "thmb" or "auxl", respectively.

[0185] In VVC, the partitioning of a picture into sub-pictures (also called sub-picture layout or sub-picture layout) can be indicated in a sequence parameter set (SPS) and / or decoded from a sequence parameter set (SPS). In VVC draft 7, the SPS syntax indicates the partitioning of a picture into sub-pictures by providing syntax elements indicating the following items for each sub-picture in a coding tree unit (CTU): the x and y coordinates of the upper left corner of the sub-picture, the width of the sub-picture, and the height of the sub-picture. Thus, the sub-picture layout indicates the position, width, and height of the sub-picture within the picture, but does not assign any particular identifier of the sub-picture or sub-picture sequence to the sub-picture layout.

[0186] In addition to the sub-picture layout, one or more of the following properties may be indicated (e.g., by an encoder) or decoded (e.g., by a decoder) or inferred (e.g., by an encoder and / or decoder) for the sub-pictures collectively or individually for each sub-picture:

[0187] i) whether the sub-picture is treated as a picture during decoding; in some cases, this attribute does not include loop filtering operations that can be separately indicated / decoded / inferred;

[0188] ii) Whether to perform loop filtering operations across sub-picture boundaries.

[0189] In VVC draft 7, the sub-picture identifier (i.e., the syntax element slice_subpic_id) is indicated in the slice header (e.g., by the encoder) and / or decoded from the slice header (e.g., by the decoder). slice_subpic_id specifies the sub-picture identifier of the sub-picture containing the slice. slice_subpic_id is encoded as a fixed-length unsigned integer u(v), whose length is indicated in the referenced SPS, PPS (picture parameter set) or PH (picture header); when the length is not indicated, the length is obtained based on the number of sub-pictures indicated in the referenced SPS.

[0190] When rectangular slices are used in VVC draft 7, the slice header includes a slice_address syntax element, which is the slice index of the slice within the sub-picture identified by slice_subpic_id.

[0191] In VVC Draft 7, the SPS, PPS or PH contains a list of sub-picture identifier values, i.e., sps_subpic_id[i], pps_subpic_id[i] or ph_subpic_id[i], respectively, where i is in the range of 0 to the number of sub-pictures in the sub-picture layout minus 1 (inclusive). The sub-picture identifier value list can be indicated in the SPS to be unchanged for the entire coded video sequence. If the SPS allows changing the sub-picture identifier value list within the coded video sequence, pps_subpic_id[i] or ph_subpic_id[i] (whichever exists and applies to the picture) specifies the sub-picture ID of the i-th sub-picture. When both the PPS applied to a picture and the picture header of the picture contain a list of sub-picture identifier values, the list in the picture header takes precedence. The decoder can use the sub-picture identifier value list to locate the decoded sub-picture in the correct position according to the sub-picture layout.

[0192] In VVC draft 7, the PH syntax associated with the sub-picture identifier value can be as follows:

[0193]

[0194]

[0195] In VVC draft 7, the semantics of the PH syntax element associated with the sub-picture identifier value can be specified as follows:

[0196] -ph_subpic_id_signalling_present_flag equal to 1 specifies that sub-picture ID mapping is signaled in the PH. -ph_subpic_id_signalling_present_flag equal to 0 specifies that sub-picture ID mapping is not signaled in the PH.

[0197] -ph_subpic_id_len_minus1 plus 1 specifies the number of bits used to represent the syntax element ph_subpic_id[i].

[0198] For bitstream conformance, it may be required that the value of ph_subpic_id_len_minus1 should be the same for all PHs referenced by a codec picture in a CLVS.

[0199] - ph_subpic_id[i] specifies the sub-picture ID of the i-th sub-picture. The length of the ph_subpic_id[i] syntax element is ph_subpic_id_len_minus1+1 bits.

[0200] It is envisioned that extracting sub-picture sequences from one or more “source” VVC bitstreams and merging the extracted sub-picture sequences into a “destination” VVC bitstream may be performed as follows:

[0201] - When encoding a source VVC bitstream, the slice_subpic_id value of each sub-picture sequence may be chosen to be different from other slice_subpic_id values ​​in all source VVC bitstreams, and the lengths of the slice_subpic_id syntax elements in the source VVC bitstreams may be chosen to be the same.

[0202] - The SPS(s) of the destination VVC bitstream are created or rewritten based on the SPS(s) of the source VVC bitstream. SPS creation may include one or more of the following:

[0203] o The sub-picture layout indicated in each SPS is created based on the sub-pictures merged into the destination VVC bitstream o The picture width and height are indicated in each SPS based on the sub-pictures merged into the destination VVC bitstream

[0204] -The PPS(s) of the destination VVC bitstream are created or rewritten based on the PPS(s) of the source VVC bitstream. PPS creation may include one or more of the following:

[0205] o Indicate the picture width and height in each PPS based on the sub-picture merged into the destination VVC bitstream

[0206] o Create a list of sub-picture identifier values, pps_subpic_id[i], from the sub-pictures merged into the destination VVC bitstream, where i is in the range of 0 to the number of sub-pictures in the sub-picture layout minus 1.

[0207] - The picture header of the destination VVC bitstream is created or rewritten based on the corresponding picture header of the source VVC bitstream. Picture header creation may include one or more of the following:

[0208] o Create a list of sub-picture identifier values, ph_subpic_id[i], based on the sub-pictures that are merged into the destination VVC bitstream, where i is in the range of 0 to the number of sub-pictures in the sub-picture layout minus 1.

[0209] - For each codec picture in the destination VVC bitstream, include the sub-pictures from the corresponding codec picture in the source VVC bitstream(s), e.g., in the order indicated by the list of sub-picture identifier values.

[0210] As described above, the VVC sub-picture feature enables the extraction and merging of sub-pictures without modifying the VCL NAL unit (ie, slice). Therefore, the fundamental difference between VVC sub-picture extraction and merging and HEVC motion restricted tile set extraction and merging is that there is no need to rewrite the slice header.

[0211] AVC and HEVC Extractors

[0212] The extractors specified in ISO / IEC 14496-15 for H.264 / AVC and HEVC enable compact formation of tracks that extract NAL unit data by reference. An extractor is a NAL unit-like structure. A NAL unit-like structure may be specified to include a NAL unit header and a NAL unit payload, like any NAL unit, but start code emulation prevention (which is required for NAL units) may not be followed in a NAL unit-like structure. For HEVC, an extractor contains one or more constructors. A sample constructor extracts NAL unit data from a sample of another track by reference. An inline constructor includes NAL unit data. The term inline may be defined, for example, in relation to a data unit, to indicate that the containing syntax structure contains or carries a data unit (as opposed to including a data unit by reference or through a data pointer). When an extractor is processed by a file reader that requires it, the extractor is logically replaced by the bytes produced when the contained constructor is decomposed in order of appearance. Nested extraction may not be allowed, for example, bytes referenced by a sample constructor should not contain an extractor; an extractor shall not directly or indirectly reference another extractor. An extractor may contain one or more constructors for extracting data from the current track or from another track that is linked to the track in which the extractor is located via a track reference of type "scal". The bytes of an extracted extractor may represent one or more complete NAL units. An extracted extractor starts with a valid length field and a NAL unit header. The bytes of the sample constructor are copied only from a single identified sample in the track referenced by the indicated "scal" track reference. Alignment is done in decoding time, i.e. using only the sample schedule and then a count offset in the number of samples. An extractor is a media level concept and therefore applies to the target track before any edit lists are taken into account. (However, one would normally expect the edit lists in both tracks to be identical).

[0213] The following syntax can be used:

[0214]

[0215] The semantics can be defined as follows:

[0216] - NALUnitHeader(): The first two bytes of the HEVC NAL unit. A specific nal_unit_type value indicates an extractor, for example nal_unit_type equals 49.

[0217] -constructor_type specifies the constructor being used.

[0218] - EndOfNALUnit() is a function that returns 0 (false) if there is more data in this extractor; otherwise it returns 1 (true).

[0219] A sample constructor can have the following syntax:

[0220]

[0221] track_ref_index identifies the source track from which the data is extracted. track_ref_index is the index of a track reference of type "scal". The index value of the first track reference is 1; the value 0 is reserved.

[0222] The samples in the track from which data is extracted are aligned in time or immediately preceding in the media decoding timeline, i.e. using only the time-to-sample table, adjusted by the sample containing the extractor at an offset specified by sample_offset. sample_offset gives the relative index of the sample in the linked track, which should be used as a source of information. Sample 0 (zero) is the sample with the same or closest previous decoding time compared to the decoding time of the sample containing the extractor; sample 1 (one) is the next sample, sample -1 (minus 1) is the previous sample, and so on.

[0223] data_offset: The offset of the first byte in the reference sample to be copied. If the extraction starts from the first data byte in this sample, the offset value is 0.

[0224] data_length: The number of bytes to copy.

[0225] The syntax for an inline constructor can be specified as follows:

[0226]

[0227] length: The number of bytes belonging to the InlineConstructor following this field.

[0228] inline_data: The data bytes to be returned when resolving an inline constructor.

[0229] The HEVC slice segment header can be overridden by the extractor track through an inline constructor (which can directly include the slice segment header).

[0230] HEVC tile base track and tile track

[0231] HEVC tile base track represents the bitstream by implicitly combining sample data from the referenced HEVC tile track. In HEVC, a "sabt" track reference is used to reference a tile track from a tile base track, and the tile ordering is indicated by the order of the tile tracks contained by the "sabt" track reference. In addition, in HEVC, a tile track is a "tbas" track reference to a tile base track.

[0232] The samples of the HEVC tile track contain slice segments. The samples of the HEVC tile track may include a sequence of motion constrained tile sets.

[0233] VVC sub-picture track group (in the working draft of VVC bearer in ISOBMFF of MPEG N18856)

[0234] It is planned to specify VVC storage in ISOBMFF-compliant files as (multiple) new clauses in ISO / IEC 14496-15. At the time of writing this document, a working draft (WD) for VVC carriage in ISOBMFF in MPEG N18856 is available. The WD includes a sub-picture track group sub-clause, which is included in the WD for exploration and discussion. The following paragraphs summarize the feature and contain an excerpt from the WD.

[0235] Sub-picture tracks use regular VVC sample entries. Track grouping is defined, which provides information indicating the level of consistency of the merged bitstream from several sub-picture tracks. Track grouping provides guidance to simplify the generation of parameter sets for the reconstructed "destination" bitstream.

[0236] When the coded sub-pictures to be jointly decoded within the group are interchangeable, i.e., the player selects multiple active tracks from a set of sampled sub-pictures with the same horizontal contribution, the sample group type "acgl" (active common group level) indicates the combination rule and the level_idc of the resulting combination when jointly decoding.

[0237] When there are coded sub-pictures with different properties (eg, different resolutions) selected for joint decoding, the sample group type "amgl" (active multi-group level) indicates a combination rule and the level_idc of the resulting combination at the time of joint decoding.

[0238] The syntax can be specified as follows:

[0239]

[0240] The semantics can be specified as follows:

[0241] track_group_type indicates the grouping type.

[0242] When track_group_type is equal to "acgl", this indicates that the track belongs to a group of tracks with the same track_group_id value. When track_group_type is equal to "amgl", this indicates that the track belongs to a group of tracks with the same track_group_id value and a subgroup with track_subgroup_id value.

[0243] num_active_tracks

[0244] When a subset of num_active_tracks tracks is selected from a group with the same track_group_id value, the group with track_group_id is played corresponding to the level of level_idc.

[0245] num_subgroup_ids

[0246] The number of separate subgroups each identified by the same track_subgroup_id value, with different subgroups identified by different track_subgroup_id values, for which the level_idc value is indicated when a group of sub-picture tracks are jointly decoded.

[0247] included_track_subgroup_id[i]

[0248] When a subset of tracks with the same track_group_id value is played, the subset includes num_active_tracks[i] tracks selected from each corresponding track subgroup with track_subgroup_id equal to included_subgroup_id[i], where i ranges from 0 to num_subgroup_ids, and the group played corresponds to the level of level_idc.

[0249] In order to realize the storage and merging of VVC sub-picture tracks, methods such as extracting tracks have the following disadvantages or problems:

[0250] - Extractor-like approaches are expensive in terms of byte count overhead. Since extractors are included in every sample, they incur a relatively high cost in terms of byte count overhead;

[0251] - The extractor-like approach is not necessarily flexible, as it can make any modifications to the NAL units, such as rewriting the slice header, and no changes are required to the VCL NAL units in VVC sub-picture merging.

[0252] When reconstructing a bitstream from an HEVC tile base track and a reference HEVC tile track, the slice segment header is not rewritten when decomposing the HEVC tile base track. Therefore, the HEVC tile base track can only represent the source bitstream, not a subset of it. Therefore, in order to implement the storage of VVC sub-picture tracks and their merging, the tiled base track and tiled track methods of the HEVC file format are used:

[0253] - No flexibility for the player to select more than one sub-picture subset of the source VVC bitstream;

[0254] - Cannot flexibly change the sub-picture layout over time.

[0255] A VVC sub-picture track group method in WD (MPEG N18856):

[0256] - The client is required to parse the (multiple) source VVC bitstreams and decode parts of them (parameter sets PH). Parsing and decoding may be a relatively complex operation involving, for example:

[0257] o Remove the start code emulation prevention bytes from the NAL unit to obtain the RBSP (Raw Byte Sequence Payload);

[0258] o Track implicit references from each slice header to picture header, and explicit references from each slice header to PPS, SPS and VPS (Video Parameter Set) and between different types of parameter sets. Parse and decode syntax element values ​​based on referenced syntax structures.

[0259] - The client is required to "assemble" the destination VVC bitstream and encode its parts (parameter set PH). "Assembling" and encoding may be a relatively complex operation involving, for example:

[0260] o Rewrite parameter sets and picture headers based on the source sub-picture track selected for merging

[0261] o Track implicit references from each slice header to the picture header, and explicit references from each slice header to PPS, SPS and VPS and between different types of parameter sets. Encode syntax element values ​​based on the referenced syntax structures.

[0262] o Insert start code emulation prevention bytes to obtain NAL units from the created RBSP.

[0263] - When a sub-picture track is not intended to be played without other sub-picture tracks, each sub-picture track is required to redundantly contain parameter sets and picture headers.

[0264] This embodiment provides an improved solution for storing a VVC sub-picture in a container file. The container file may be a file compliant with ISOBMFF.

[0265] In encoding of a bitstream, two or more sub-picture tracks are written to a container file. According to one embodiment, the writing is performed by a file writer or similar entity, which generates an instance of a sample group in a container file containing two or more sub-picture tracks (or the like). In addition, a basic track is written to the container file, the basic track being intended to be decomposed into a video bitstream. In the basic track, a sub-picture layout is indicated. In addition, a sample group description entry is written to the container file. The sample group description entry indicates a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout. The samples of the basic track for which the sample group description entry is intended to be used to reconstruct the video bitstream are also indicated in the container file.

[0266] In decoding of a bitstream, a sub-picture layout is parsed from a basic track of a container file. In addition, a sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout is parsed from the container file. When the sample group description entry indicates a group of sub-picture tracks, a second sub-picture track is selected from the group of sub-picture tracks. In addition, which sample set of the basic track is parsed from the container file, the sample group description entry is intended to be used to reconstruct the video bitstream. According to one embodiment, a file reader or similar entity parses an instance of a sample group from a container file containing two or more sub-picture tracks (or the like). Finally, by including a time-aligned sample of the first sub-picture track or the second sub-picture track for each sub-picture position of the sub-picture layout, a coded picture of the video bitstream corresponding to the sample set is reconstructed from the container file.

[0267] The first sub-picture track includes a sub-picture sequence for a corresponding sub-picture position, and any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position. A VVC sub-picture track can be defined to contain a sub-picture sequence with the same sub-picture identifier (ID). In a first alternative, a VVC sub-picture track can be defined to contain one or more temporally cascaded sub-picture sequences, which sub-pictures can have different sub-picture identifiers. When the sub-picture identifier changes, the first sub-picture in decoding order can be a sub-picture of a CLVSS picture. In a second alternative, a VVC sub-picture track can be defined to contain one or more sub-picture sequences. When there is more than one sub-picture sequence, the sub-pictures of each sequence are spatially adjacent and can be regarded as a single sub-picture. The first alternative and the second alternative can also be combined into a third alternative.

[0268] When describing the present embodiment, the terms "base track" and "VVC base track" may be used interchangeably. Similarly, the terms "sub-picture track" and "VVC sub-picture track" may be used interchangeably. Even if the embodiment is described with reference to a VVC base track and / or a VVC sub-picture track, the embodiment is not limited to VVC, but is applicable to any video coding scheme having a concept equivalent to or similar to a VVC sub-picture.

[0269] The present scheme also provides a new type of sample group, which is further described in the following paragraphs. Hereinafter, the new type of sample group has a four-character code "spor" (for "sub-picture order"), but it should be understood that any other four-character code can be used as well.

[0270] The "spor" sample group is intended for a VVC base track, which a file reader or similar entity decomposes into a VVC destination bitstream. The VVC base track references a VVC sub-picture track, from which the VVC destination bitstream may be derived, as will be described.

[0271] Each sample group description entry of a "spor" sample group indicates a sub-picture in decoding order, where the sub-picture is indicated by an index of a specific type of track reference. Hereinafter, a track reference of type "subp" is used, but it is understood that any other four-character code may be used as well.

[0272] The syntax of a "spor" sample group description entry can be specified as follows:

[0273]

[0274] The loop over the subp_track_ref_idx values ​​specifies the indices of track references of type "subp" in decoding order. When the track references are otherwise decomposed in the order given in the sample group description entries, a valid VVC destination bitstream is produced.

[0275] In another embodiment, each sample group description entry of a "spor" sample group indicates a sub-picture in decoding order, wherein the sub-picture is indicated using an identifier of a track (track_ID) or a track group (track_group_id).

[0276] In another embodiment, each sample group description entry of the "spor" sample group indicates a sub-picture in decoding order, where the sub-picture is indicated by an index of an entity group of a specific or indicated type and an indicated or inferred identifier (group_id). For example, the entity group identifier can be inferred to be the group_id of such an entity group, where the track_ID value of the VVC basic track appears as the first listed entity_id. The entity group may contain (as entity_id values) the track_ID value of the VVC sub-picture track and / or the item_ID value of the VVC sub-picture image item. A player or the like can resolve a reference to the index of the entity group by selecting one of the entities of the group (excluding the entity indicating the VVC basic track, if any).

[0277] In another embodiment, each sample group description entry of a "spor" sample group indicates a sub-picture in decoding order, where the sub-picture is indicated using an index of a specific type of track reference. The track reference can be an identifier of a track (track_ID), an identifier of a specific type of track group (track_group_id), or an identifier of a specific type of entity group (group_id). When the track reference is an identifier of a track group, a player or the like resolves the reference by selecting one of the tracks in the track group. When the track reference is an identifier of an entity group, a player or the like resolves the reference by selecting one of the entities in the entity group, where the entity can be, for example, an image item.

[0278] Embodiments that are able to reference both samples and image items from a "spor" sample group description entry can be used, for example, to combine a sub-picture of a static background image item and a dynamically changing foreground sub-picture. There may be many applications or use cases that can benefit from such a combination. For example, a movie screen or a television with 2D video content may be embedded in a 360° static image background, and omnidirectional viewing may occur, such as with a head-mounted display. In another example, a movie image, etc. is synthesized where only (multiple) parts of the scene are dynamic and the remaining area is static.

[0279] In one embodiment, the sample group description entry further includes one or more of the following:

[0280] - a first indication of whether a sub-picture ID is carried in an SPS NAL unit included in a VVC basic track;

[0281] - a second indication as to whether the sub-picture ID is carried in a PPS NAL unit included in the VVC basic track;

[0282] - A third indication as to whether a sub-picture ID is carried in a picture header (PH) NAL unit included in a VVC basic track (referred to as ph_subpic_id_flag in the following syntax).

[0283] In one embodiment, if the first indication indicates that the sub-picture ID is carried in a SPSNAL unit included in the VVC basic track, the sample group description entry includes one or more of the following:

[0284] – The SPS NAL unit that applies to the samples mapped to this sample group description entry

[0285] - The SPS RBSP that applies to the samples mapped to this sample group description entry

[0286] - the SPS identifier value that references the corresponding identifier value in the SPS provided in the VVC basic track (i.e., sps_seq_parameter_set_id in the VVC draft standard), applied to the SPS of the sample mapped to this sample group description entry

[0287] - an index between the SPS NAL units provided in the sample entries of the VVC base track, pointing to the SPS NAL unit that applies to the sample mapped to this sample group description entry - an index between the NAL units (of any type) provided in the sample entries of the VVC base track, pointing to the SPS NAL unit that applies to the sample mapped to this sample group description entry

[0288] - the length (in bits) of the sub-picture identifier (i.e., sps_subpic_id[i]) syntax element in the SPS RBSP,

[0289] - the bit position of the first sub-picture identifier syntax element in the referenced or included SPS NAL unit or SPS RBSP (i.e., sps_subpic_id[0]), and

[0290] - A flag indicating whether a start code emulation prevention byte (i.e., sps_subpic_id[i]) is present before or within the sub-picture identifier syntax element. This flag may alternatively be expressed to indicate whether the bit position of sps_subpic_id[i] with any valid i value in the SPS NAL unit is different from the bit position in the SPS RBSP.

[0291] In one embodiment, if the second indication indicates that the sub-picture ID is carried in a PPSNAL unit included in the VVC basic track, the sample group description entry includes one or more of the following:

[0292] – the PPS NAL unit that applies to the samples mapped to this sample group description entry;

[0293] - the PPS RBSP that applies to samples mapped to this sample group description entry;

[0294] - the PPS identifier value referencing the corresponding identifier value (i.e. pps_pic_parameter_set_id in the VVC draft standard) in the PPS provided in the VVC basic track, applied to the PPS of the samples mapped to this sample group description entry;

[0295] - an index among the PPS NAL units provided in the sample entries of the VVC base track, which points to the PPS NAL unit that applies to the sample mapped to this sample group description entry;

[0296] - an index among the NAL units (of any type) provided in the sample entries of the VVC elementary track, pointing to the PPS NAL unit that applies to the sample mapped to this sample group description entry;

[0297] - the length (in bits) of the sub-picture identifier (i.e., pps_subpic_id[i]) syntax element in the PPS RBSP;

[0298] - the bit position of the first sub-picture identifier syntax element in the referenced or included PPS NAL unit or PPS RBSP (ie, pps_subpic_id[0]); and

[0299] - A flag indicating whether a start code emulation prevention byte (i.e., pps_subpic_id[i]) is present before or within the sub-picture identifier syntax element. This flag may alternatively be expressed to indicate whether the bit position of pps_subpic_id[i] with any valid i value in the PPS NAL unit is different from the bit position in the PPS RBSP.

[0300] In one embodiment, if the third indication indicates that the sub-picture ID is carried in a picture header (PH) NAL unit included in the VVC basic track, the sample group description entry includes one or more of the following:

[0301] - the length (in bits) of the sub-picture identifier (i.e., ph_subpic_id[i]) syntax element in the PH RBSP (referred to as ph_subpic_id_len_minus1 in the syntax below),

[0302] - the bit position of the first sub-picture identifier syntax element (i.e., ph_subpic_id[0] in the PH RBSP (referred to as ph_subpic_id_bit_pos in the syntax below), and

[0303] - A flag (referred to as ph_start_code_emul_flag in the syntax below) indicating whether a start code emulation prevention byte (i.e., ph_subpic_id[i]) is present before or within the sub-picture identifier syntax element. This flag may alternatively be expressed to indicate whether the bit position of ph_subpic_id[i] with any valid i value in the PH NAL unit is different from the bit position in the PH RBSP.

[0304] In an example embodiment related to the third indication, the syntax of the "spor" sample group description entry may be specified as follows:

[0305]

[0306] The syntax related to the first indication and / or the second indication may be implemented similarly to the above syntax.The syntax may cover aspects related to one or more of the first indication, the second indication and the third indication similar to the above syntax.

[0307] In the following, the VVC sub-picture track and the VVC base track are discussed in a more detailed manner.

[0308] Track group for VVC sub-picture tracks

[0309] According to one embodiment, the group of sub-picture tracks is indicated by a file writer or similar entity as a track group in a container file, and / or is parsed from a track group in a container by a file reader or similar entity. For example, the group of sub-picture tracks can be an "alte" track group that collects VVC sub-picture tracks. From these VVC sub-picture tracks, any track can be selected interchangeably to decompose the VVC basic track. Tracks in the same "alte" track group have the same width, height, and sub-picture border attributes.

[0310] Although the present embodiment refers to a track group having an "alte" four-character code, the embodiment is generally applicable to any four-character code for a track group.

[0311] Sample entry for a VVC sub-picture track

[0312] If a VVC sub-picture track is suitable for use without other VVC sub-picture tracks, regular VVC sample entries (ie, "vvc1" or "vvi1" according to MPEG N18856) may be used.

[0313] According to one embodiment, a special sample entry type, "vvs1" in this article (but the embodiment generally applies to any four-character code), is used for VVC sub-picture tracks. It can be specified that when this special sample entry type is used, VPS, DPS, SPS, PPS, AUD (Access Unit Delimiter), PH, EOS (End of Sequence) and EOB (End of Bitstream) NAL units are not present in the sample entry and samples.

[0314] According to one embodiment, a sample entry of a VVC sub-picture track includes one or more of the following:

[0315] - Sub-picture ID, which may need to be equal to the value of slice_subpic_id in all slices present in the samples of the track.

[0316] -Sub-picture position ID. When VVC sub-picture tracks have the same sub-picture position ID value, they represent the same original content and have the same width and height.

[0317] According to one embodiment, a sample entry for a VVC sub-picture track having two or more sub-pictures includes one or more of the following:

[0318] - Sub-picture ID, which may need to be equal to the value of slice_subpic_id in the slice of each sub-picture present in the samples of the track.

[0319] -Sub-picture position ID. When VVC sub-picture tracks have the same sub-picture position ID value, they represent the same original content and have the same width and height.

[0320] For example, a sample entry of a VVC sub-picture track may contain a specific box, referred to herein as a sub-picture properties box (SubPicPropertiesBox), which carries the above-mentioned syntax elements.

[0321] VVC basic track

[0322] According to one embodiment, a VVC base track has a conventional VVC sample entry type (i.e., "vvc1" or "vvi1" according to MPEG N18856). By including a "subp" track reference in a track, a file writer or similar entity can indicate that the track is a VVC base track, rather than a conventional self-contained VVC track. In another embodiment, a specific sample entry type(s) (e.g., "vvcb") may be specified to indicate that the track is a VVC base track.

[0323] Although the embodiments refer to a sample entry type having a "vvcb" four-character code, the embodiments apply generally to any four-character code of the sample entry type to indicate a VVC base track.

[0324] According to one embodiment, samples of the VVC base track do not contain VCL NAL units. According to another embodiment, samples of the VVC base track are allowed to contain VCL NAL units. A file writer or similar entity may, for example, include in the VVC base track VCL NAL units of such sub-picture sequences that are intended to always be present in the destination VVC bitstream.

[0325] According to one embodiment, instances of parameter sets (such as VPS, DPS, SPS and / or PPS NAL units) applied to the decomposed VVC basic track (i.e. the destination VVC bitstream) are included in sample entries or samples of the VVC basic track, e.g. by a file writer.

[0326] Some of the above embodiments allow reference to samples and image items from a "spor" sample group description entry. According to one embodiment, when the sample group description entry is decomposed so that one or more samples of the VVC sub-picture track and one or more VVC sub-picture items are used to decompose samples of the VVC basic track, a picture-level syntax (such as a PPS with mixed_nalu_types_in_pic_flag equal to 1) is used to indicate that the codec picture decomposed from the VVC basic track sample contains mixed NAL unit types. In one embodiment, a file writer or the like includes and / or references a picture-level syntax, such as a PPS, which indicates mixed NAL unit types in all samples of the VVC basic track, which types are mapped to sample group description entries for both reference samples and image items. In one embodiment, the syntax element indicating the picture order count of the sub-picture image item (e.g., slice_pic_order_cnt_lsb in the VVC draft standard) is rewritten (as part of parsing samples of the VVC basic track) to be the same as the value(s) of the corresponding syntax element(s) in the selected VVC sub-picture track.

[0327] According to one embodiment, instances of EOS and EOB NAL units (if any) applied to the decomposed VVC basic track (i.e., the destination VVC bitstream) are included in the samples of the VVC basic track. It may be required that they cannot be inherited by any other NAL unit in the same sample.

[0328] This embodiment is further discussed with the help of the following examples.

[0329] Mixed quality sub-image merging

[0330] Figure 1is an example of how VVC sub-picture tracks can be used for viewport-dependent omnidirectional video streams. Here, the term "omnidirectional" can refer to media content that can have a larger spatial extent than the field of view of the device rendering the content. Omnidirectional content can, for example, cover substantially 360 degrees in the horizontal dimension and substantially 180 degrees in the vertical dimension, but omnidirectional can also refer to content that covers less than a 360-degree view in the horizontal direction and / or a 180-degree view in the vertical direction.

[0331] An omnidirectional image can be represented by a sphere that has been mapped to a two-dimensional image plane using an equirectangular projection (ERP). In this case, the horizontal coordinate can be considered equivalent to the longitude, while the vertical coordinate can be considered equivalent to the latitude, without applying any transformation or scaling. The ERP image can be formed by a set of input images, such as fisheye images from a camera array or camera device with multiple lenses and sensors, which are stitched onto a spherical image. The spherical image is further projected onto a cylinder (without top and bottom surfaces). The cylinder is unfolded to form a two-dimensional projection framework. In practice, one or more of the proposed steps can be combined; for example, the input image can be projected directly onto the cylinder without an intermediate projection onto the sphere. The projection structure for the equirectangular projection can be viewed as a cylinder comprising a single surface.

[0332] In general, omnidirectional content can be mapped onto different types of solid geometry, such as polyhedra (i.e., three-dimensional solid objects containing planar polygonal faces, straight edges, and sharp corners or vertices, such as a cube or pyramid), cylinders (by projecting a spherical image onto a cylinder, as described above using equirectangular projection), cylinders (without first projecting directly onto a sphere), cones, etc., and then unfolded to a two-dimensional image plane.

[0333] On the other hand, a viewport can be defined as the area of ​​an omnidirectional image or video that is suitable for display and viewing by a user. The current viewport (which may be simply referred to as the viewport) can be defined as the portion of the spherical video that is currently displayed and therefore viewable by the user(s). At any point in time, the video rendered by an application on a head mounted display (HMD) renders a portion of the 360-degree video, which is called the viewport.

[0334] Viewport-dependent video refers to content where the region(s) within the viewport are different from the rest of the omnidirectional content (e.g., by encoding the viewport at a higher quality). Such content can be provided by a sender to a recipient device based on the viewport direction.

[0335] The per-region packing information may be encoded as metadata in or along the bitstream. For example, the packing information may include a per-region mapping between a predefined or indicated source format (e.g., according to an indicated projection format, such as ERP) and a packed frame format (e.g., a decoded picture or sub-picture represented by a track). The per-region packing information may be included in a VVC sub-picture track (e.g., by a file writer) to indicate which spherical region is covered by the track, and may be parsed from the VVC sub-picture track (e.g., by a player) to derive which spherical region is covered by the track.

[0336] The per-region packing information may be used in various embodiments to select (multiple) VVC sub-picture tracks from a track group or entity group. For example, the spherical area covered by the VVC sub-picture track may be inferred from the per-region packing information, and when the spherical area intersects the viewport, the VVC sub-picture track may be selected from the track group or entity group.

[0337] Next, the rectangle-by-region packing metadata is described: For each region, the metadata defines a rectangle in the projected picture, a corresponding rectangle in the decoded picture, and optional transformations of rotation by 90, 180, or 270 degrees and / or horizontal and / or vertical mirroring. For example, a rectangle can be indicated by the position of the upper left corner and the lower right corner. The mapping can include resampling. Since the size of the corresponding rectangles in the projected and decoded pictures may be different, the mechanism infers resampling by region.

[0338] When the VVC sub-picture track is used for a viewport-dependent omnidirectional video stream, a 4×2 tile grid can be used to form sub-pictures of equirectangular projection (ERP) pictures during encoding. Two VVC bitstreams derived from the same source content are encoded with different picture qualities and bitrates.

[0339] To create a VVC sub-picture track, each sub-picture sequence can be included in one VVC sub-picture track. Each pair of VVC sub-picture tracks that represent the same content (i.e., have the same position in the ERP picture) is indicated as a member of the same "alte" track group. The track_group_id values ​​g1, ..., g8 are uniquely selected and are not equal to any track_ID value.

[0340] A VVC base track is created to contain a track reference of type "subp" with track_group_id values ​​g1, ..., g8 listed. All track_group_id values ​​g1, ..., g8 may be listed. A VVC base track may also contain a "spor" sample group with a single sample group description entry containing an index list of "subp" track references, i.e., values ​​1, ..., 8, and may be indicated as a default value applied to all samples of the track.

[0341] In player operation, the player can select, for example, quality and / or bitrate to receive each VVC sub-picture track in the same "alte" track group based on certain conditions (e.g., viewing orientation, network bandwidth). In this example, the player receives VVC sub-picture tracks 1, 2, 5, and 6 at a certain quality, and receives VVC sub-picture tracks 3, 4, 7, and 8 at another quality. The VVC base track is used to reconstruct a VVC bitstream that can be decoded using a single VVC decoder.

[0342] Mixed-resolution sub-image merging

[0343] Another example is discussed below with reference to a particular resolution or pixel count of width and / or height. It should be understood that the resolution and pixel count are given as examples and the example embodiments may be similarly applied to any selection of resolution and pixel count.

[0344] In terms of sampling density, a cubemap resolution with cube faces of 1536×1536 samples can be considered roughly equivalent to 6144×3072 ERP. In the presented arrangement, the high-resolution tiles are extracted from a cubemap with a face size of 1536×1536 and can be considered to cover a hemisphere. The remaining tiles can be extracted from a cubemap with a quarter resolution compared to the high-resolution bitstream. It is claimed that this resolution meets the capabilities of a head-mounted display with a Quad HD (2560×1440) display panel. It is also claimed that the proposed scheme provides a reasonable margin for changes in viewing direction caused, for example, by head movements on HMD-based viewing. This arrangement is Figure 2 and explained below.

[0345] In encoding, the content can be encoded at two spatial resolutions with cube face sizes of 1536 × 1536 and 768 × 768. In both bitstreams, a 6 × 4 sub-picture grid can be used.

[0346] To create a VVC sub-picture track, the encoded sub-picture sequence is stored as a corresponding VVC sub-picture track. Sub-picture tracks with the same width and height are indicated as members of the same "alte" track group. The track_group_id values ​​g1 and g2 are uniquely selected and are not equal to any track_ID value,

[0347] To create a VVC base track, the SPS of the VVC base track specifies Figure 2The sub-picture layout shown. The VVC base track contains a track reference of type "subp" listing track_group_id values ​​g1 and g2. The VVC base track also contains a "spor" sample group with a single sample group description entry that contains an index list of the "subp" track reference and is indicated as the default sample group description entry that applies to all samples of the track.

[0348] In player operation, the player can select sub-pictures covering the viewport from the high-resolution encoding and the remaining sub-pictures from the low-resolution encoding. 12 VVC sub-picture tracks derived from the high-resolution bitstream are selected, while the complementary 12 VVC sub-picture tracks are derived from the low-resolution bitstream.

[0349] Need to understand, Figure 2 The sample group description entries presented in are only one example of what may result in a valid bitstream reconstructed from a VVC base track. The sample group description entries may equally well contain track reference indices in any order of corresponding samples that are permitted by the VVC standard and described in the SPS and that apply to the VVC base track. In the VVC draft standard, it is required that the decoding and bitstream order of sub-pictures of a picture must be such that sub-pictures adjacent to the top and left boundaries of any particular sub-picture must precede that particular sub-picture in decoding and bitstream order. Storage of picture headers for VVC base tracks

[0350] According to one embodiment, a picture header (PH) NAL unit may be included in samples of a VVC base track (eg, by a file writer) and / or parsed from samples of a VVC base track (eg, by a file reader).

[0351] According to one embodiment, a new type of sample group is specified for carrying PH NAL units, as described in the subsequent paragraphs. In one embodiment, a file writer or similar entity generates an instance of a sample group in a container file. In one embodiment, a file reader or similar entity parses an instance of a sample group from a container file.

[0352] Hereinafter, sample groups of the new type have the four-character code "phdr" (standing for "picture header"), but it will be appreciated that any other four-character code may be used as well.

[0353] In various embodiments, a "phdr" sample group description entry includes one of the following:

[0354] - a PH NAL unit and zero or more other NAL units, such as a SEI NAL unit;

[0355] -PH NAL unit;

[0356] - a PH NAL unit without a NAL unit header;

[0357] -PH RBSP.

[0358] If a sample is not mapped to any sample group description entry of the "phdr" sample group, the sample may be required to contain a PH NAL unit.

[0359] It has been proposed that the picture header syntax contains syntax elements for deriving a picture order count (POC). Since access units of a coded video sequence have different POCs, using a "phdr" sample group to carry a picture header with correct values ​​for POC-related syntax elements would result in an impractical number of sample group description entries. In one embodiment, the "phdr" sample group description entry contains one or more of the following:

[0360] - Indication(s) as to whether the POC-related syntax element(s) in the picture header apply to the samples mapped to this sample group description entry or will be overwritten (when reconstructing the bitstream).

[0361] - indication(s) of an indication of how timing information (such as composition time) of the track relates to the POC value. For example, a composition time scaling factor may be indicated to convert the composition time to a POC value. The composition time difference may be derived from the composition time of the current sample minus the composition time of the previous sample to which the POC-related syntax element is so applied. The POC difference may be derived by dividing the composition time difference by the composition time scaling factor. The POC value may be derived by adding the POC difference to the derived POC value of the previous sample to which the POC-related syntax element is so applied. The POC-related syntax element value may be derived from the POC value.

[0362] - The starting bit position of the POC related syntax element(s) within the picture header, such as the POC LSB value.

[0363] - The length (in bits) of the POC related syntax element(s).

[0364] - A flag indicating whether there are(s) start code emulation prevention bytes before or within the POC related syntax element(s).

[0365] The use of the "phdr" sample group may not be limited to the VVC base track, but the "phdr" sample group may also or alternatively be used for regular VVC tracks and VVC sub-picture tracks. The embodiments described below may be used separately and are not limited to use with the "spor" sample group.

[0366] If the PH NAL units are identical, in several codec pictures the storage of PH NAL units can be improved in terms of number of bytes by using the "phdr" sample group.

[0367] Reconstruct the destination VVC bitstream

[0368] According to one embodiment, a file reader or similar entity reconstructs a VVC bitstream by decomposing a VVC basic track. The VVC basic track can be parsed in the decoding order of samples of the VVC basic track. The samples of the VVC basic track are parsed into access units in the destination VVC bitstream. The access unit can be parsed into a VCL NAL unit containing NAL units carried in the samples of the VVC basic track and at least a VVC sub-picture track selected in a referenced VVC sub-picture track, where the reference can be indicated by a "spor" sample group description entry applied to the samples of the VVC basic track, and in some embodiments, by a "subp" track reference, as described in other embodiments.

[0369] According to one embodiment, a file reader or similar entity indicates the mapping of sub-picture IDs to sub-picture layouts by overriding a sub-picture ID syntax element in the SPS, PPS(s) or picture header.

[0370] In one embodiment, a file reader or similar entity infers which syntax structure is modified by using one or more of the Override Sub-Picture ID syntax elements in the following steps:

[0371] - A file reader or similar entity infers which syntax structures contain the sub-picture ID syntax element, for example based on an indication in the "spor" sample group description entry of samples mapped to the VVC base track. In another example, the sub-picture ID is inferred to be carried in the SPS, PPS or PH NAL unit by parsing the SPS, PPS or PH NAL unit, respectively.

[0372] - If more than one syntax structure applicable to a sample contains a sub-picture ID syntax element, a file reader or similar entity may infer the priority order of the containing syntax structures including the sub-picture ID syntax element applicable to a particular sample(s). For example, a sub-picture ID in a PH may override a sub-picture ID in a PPS and an SPS. The syntax structure with the highest priority may be selected for modification.

[0373] - The syntax structure to be modified may result in the entire syntax structure being repeated in the reconstructed bitstream. For example, if the PPS is modified by overwriting the sub-picture ID syntax element with a different set of sub-picture IDs that were previously used in a PPS with the same PPS identifier value, then the file reader or similar entity includes a copy of the entire PPS NAL unit in the codec pictures of the reconstructed bitstream to which the set of sub-picture IDs applies.

[0374] If the sub-picture ID syntax element is overridden in the PH NAL unit, the PH NAL unit that performs the overriding is selected as the PH NAL unit present in the sample (if any), or the PH NAL unit that maps to the "phdr" sample grouping of the sample in the VVC basic track (if any).

[0375] In one embodiment, a file reader or similar entity overrides the syntax structure that selects the sub-picture ID to be modified to include the selected sub-picture track. In one embodiment, the modification of the "spor" sample group description entry mapped to the sample is performed as follows:

[0376] o Starting from the bit position of the first sub-picture identifier syntax element in the syntax structure indicated in the sample group description entry (e.g., the bit position of ph_subpic_id[0] in the picture header to be modified), overwrite each sub-picture identifier syntax element (e.g., the ph_subpic_id[i] value in the picture header) with the sub-picture ID from each selected VVC sub-picture track in the order specified in the sample group description entry.

[0377] The NAL units included in an access unit are arranged in an order that complies with the constraints in the VVC specification. It is worth noting that these NAL units may have undergone rewriting of sub-picture identifier values ​​as described above. In one embodiment, a file reader or similar entity arranges the NAL units in the order of the following bullet points (where "sample" refers to a sample of the VVC basic track being parsed as an access unit):

[0378] - when present in the sample (and the first NAL unit), the AUD NAL unit (if any);

[0379] - Parameter set NAL units and SEI NAL units (if any) contained in the sample entry when the sample is the first sample in a sequence of samples associated with the same sample entry. Note that it may be required that only prefix APS and prefix SEI NAL units can be present in a sample entry.

[0380] - the NAL units present in the sample and preceding the first of the following in the sample, if any:

[0381] ο the PH NAL unit in the sample (if any),

[0382] o the first VCL NAL unit in the sample (if any),

[0383] o the first NAL unit that is allowed to be the last NAL unit of an AU according to the VVC specification (if any),

[0384] οEnd of sample.

[0385] - The PH NAL unit present in the sample or mapped to the PH NAL unit of the "phdr" sample grouping of the sample, if any.

[0386] - Decompose the contents of the sample from each referenced VVC sub-picture track in the order specified in the "spor" sample group description entry that maps to this sample, excluding all VPS, DPS, SPS, PPS, AUD, PH, EOS, and EOB NAL units (if any). Track references are decomposed as specified below.

[0387] o Note that if the reference VVC sub-picture track is associated with an APS track, the parsed samples contain the APS NAL unit(s) that are temporally aligned to the samples in the APS track, if any.

[0388] - The remaining NAL units in the sample (not included in the above access unit).

[0389] The tracking reference index for the "spor" sample group description entry can be broken down as follows:

[0390] If the track reference points to the track ID of a VVC sub-picture track, the track reference is resolved to a VVC sub-picture track.

[0391] - Otherwise (the track reference points to the "alte" track group), the track reference is resolved to any track of the "alte" track group. It may be required that if a particular track reference index value was resolved to a particular track in the previous sample, it should be resolved in the current sample to any of the following cases:

[0392] ο the same specific track, or

[0393] o Any other track in the same "alte" track group containing synchronized samples, or in one embodiment, SAP sample type 3 that is time aligned with the current sample.

[0394] According to another embodiment, a VVC encoded image may be stored in a file conforming to an image file format (e.g., by a file writer) and / or parsed from a file conforming to an image file format (e.g., by a file reader), similar to but not necessarily identical to the High Efficiency Image File Format (HEIF, ISO / IEC 23008-12), as described below. The file writer may form separate items from each sub-picture of the VVC encoded image. The file writer may form and / or the file reader may parse the resulting image item, which may be referred to as a "VVC obtained image item" or a "VVC base image item" that merges the VVC encoded sub-pictures into a codec picture. The file writer includes and / or the file reader may parse an item reference from the VVC base image item to the sub-picture item in an ordered manner to indicate (by the file writer) or decompose (by the file reader) the decoding or bitstream order of the sub-pictures in the codec picture. Such a reference may be accomplished using a new type of item reference ("iref") of type "spir". Although the embodiment references an item reference of type "spir", the embodiment is generally also applicable to any other four-character code.

[0395] According to another embodiment, the file writer can create and / or the file reader can parse an entity group containing sub-picture image items, wherein any single image item can be used interchangeably to resolve a "spir" item reference pointing to an entity group ID. Thus, in order to parse the image items obtained by the VVC, the player selects the sub-picture image items from the entity group. The player can, for example, select the sub-picture image items that cover the viewport and omit the sub-picture image items that do not cover the viewport.

[0396] According to another embodiment, instead of using a "spir" item reference, the VVC sub-picture item and the VVC basic item are included in an entity group, which lists the resulting VVC image item (or VVC basic item) and other VVC sub-picture items, including additional attributes in the entity group (if necessary).

[0397] Figure 3is a flow chart illustrating a method according to an embodiment. A method comprises writing 310 two or more sub-picture tracks in a container file; writing 320 a base track in the container file, the base track being intended to be decomposed into a video bitstream; indicating 330 a sub-picture layout in the base track; writing 340 a sample group description entry in the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track comprises a sub-picture sequence for the respective sub-picture position, and wherein any track in the group of sub-picture tracks comprises a valid sub-picture sequence for the respective sub-picture position; and indicating 350 samples of the base track in the container file, the sample group description entry being intended to be used for reconstructing the video bitstream for the sample group description entry.

[0398] An apparatus according to one embodiment includes means for writing two or more sub-picture tracks in a container file; means for writing a base track in the container file, the base track being intended to be decomposed into a video bitstream; means for indicating a sub-picture layout in the base track; means for writing a sample group description entry in the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; and means for indicating samples of the base track in the container file, the sample group description entry being intended for use in reconstructing a video bitstream for the sample group description entry. The means includes at least one processor and a memory including computer program code, wherein the processor may also include processor circuitry. The memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to perform methods according to various embodiments.

[0399] Figure 44 is a flow chart illustrating a method according to another embodiment. A method includes parsing 410 a sub-picture layout from a basic track of a container file; parsing 420 a sample group description entry from the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; when the sample group description entry indicates a group of sub-picture tracks, selecting 430 a second sub-picture track from the group of sub-picture tracks; parsing 440 from the container file which set of samples of the basic track the sample group description entry is intended to be used for reconstructing a video bitstream; reconstructing 450 a coded picture of a video bitstream corresponding to the group of samples from the container file by including, for each sub-picture position of the sub-picture layout, time-aligned samples of the first sub-picture track or the second sub-picture track.

[0400] An apparatus according to an embodiment includes a component for parsing a sub-picture layout from a basic track of a container file; a component for parsing a sample group description entry from the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; a component for selecting a second sub-picture track from the group of sub-picture tracks when the sample group description entry indicates a group of sub-picture tracks; a component for parsing a sample set of a basic track from the container file, the sample group description entry being intended to be used to reconstruct a video bitstream for the sample set of the basic track; a component for reconstructing a coded picture of a video bitstream corresponding to the sample set from the container file by including time-aligned samples of the first sub-picture track or the second sub-picture track for each sub-picture position of the sub-picture layout. The apparatus includes at least one processor and a memory including computer program code, wherein the processor may also include a processor circuit system. The memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to perform a method according to various embodiments.

[0401] Figure 5 An example of a data processing system for an apparatus is illustrated in FIG. Several functions may be performed using a single physical device, for example, all computing processes may be performed in a single processor if desired. The data processing system includes a main processing unit 100, a memory 102, a storage device 104, an input device 106, an output device 108, and a graphics subsystem 110, all of which are connected to each other via a data bus 112.

[0402] Main processing unit 100 is a conventional processing unit arranged to process data within a data processing system. Main processing unit 100 may include or be implemented as one or more processors or processor circuitry. Memory 102, storage device 104, input device 106, and output device 108 may include conventional components recognized by those skilled in the art. Memory 102 and storage device 104 store data in data processing system 100.

[0403] For example, according to various embodiments Figure 3 or Figure 4 The computer program code of the method shown in the flowchart of resides in the memory 102. The input device 106 inputs data into the system, and the output device 108 receives data from the data processing system and forwards the data to, for example, a display. The data bus 112 is a conventional data bus, and although shown as a single line, it can be any combination of the following: processor bus, PCI bus, graphics bus, ISA bus. Therefore, it is easy for those skilled in the art to recognize that the device can be any data processing device, such as a computer device, a personal computer, a server computer, a mobile phone, a smart phone or an Internet access device, such as an Internet tablet.

[0404] Figure 6 An example of a video encoder is illustrated, where In: image to be encoded; P'n: predicted representation of an image block; Dn: prediction error signal; D'n: reconstructed prediction error signal; I'n: preliminary reconstructed image; R'n: final reconstructed image; T, T-1: transform and inverse transform; Q, Q-1: quantization and inverse quantization; E: entropy coding; RFM: reference frame memory; Pinter: inter-frame prediction; Pintra: intra-frame prediction; MS: mode selection; F: filtering. Figure 7 A block diagram of a video decoder is illustrated, wherein P'n: predicted representation of an image block; D'n: reconstructed prediction error signal; I'n: preliminary reconstructed image; R'n: final reconstructed image; T-1: inverse transform; Q-1: inverse quantization; E-1: entropy decoding; RFM: reference frame memory; P: prediction (internal or intra); F: filtering. The apparatus according to the embodiment may include only an encoder or a decoder or both.

[0405] Various embodiments may provide advantages. For example, compared to using track references without sample grouping (as in HEVC tiled tracks and HEVC tiled tracks):

[0406] οThe use of sample groups provides the possibility for time-varying sub-picture layouts.

[0407] o The use of sample groups provides the possibility to assign additional parameters (eg, the length of the sub-picture identifier and / or the bit position of the first sub-picture identifier in the PH RBSP). The additional parameters may be time-varying.

[0408] Furthermore, the byte count overhead of sample groups is not significant. For example, it is assumed that the bit position of ph_subpic_id[0] may remain constant in all PH NAL units. Therefore, all pictures of a coded video sequence may be mapped to the same sample group description entry. If the same SPS is used throughout the bitstream, the use of a default sample group description entry may be indicated in the SampleGroupDescriptionBox, and SampleToGroupBoxes may not be present. Furthermore, when a track group of a sub-picture track is referenced by a "subp" track reference, the reader may be free to select a subset of sub-pictures of the source bitstream, or select sub-pictures from more than one source bitstream, or a combination of both, depending on how the track group is formed. Furthermore, reconstructing the VVC bitstream from the selected sub-picture track may only require overwriting the sub-picture identifier (e.g., ph_subpic_id[i]) syntax element in the picture header as instructed in the "spor" sample group.

[0409] Various embodiments may be implemented with the aid of a computer program code that resides in a memory and causes an associated device to perform the method. For example, a device may include circuitry and electronics for processing, receiving, and transmitting data, computer program code in a memory, and a processor that causes the device to perform features of an embodiment when the computer program code is run. In addition, a network device such as a server may include circuitry and electronics for processing, receiving, and transmitting data, computer program code in a memory, and a processor that causes the network device to perform features of an embodiment when the computer program code is run. The computer program code includes one or more operating characteristics. The above-mentioned operating characteristics are defined by configuration of the above-mentioned computer based on the type of the above-mentioned processor, wherein the system can be connected to the above-mentioned processor via a bus, wherein the programmable operating characteristics of the system include writing two or more sub-picture tracks in a container file; writing a basic track in the container file, which is intended to be decomposed into a video bitstream; indicating a sub-picture layout in the basic track; writing a sample group description entry in the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; and indicating samples of the basic track in the container file, the sample group description entry being intended to be used to reconstruct the video bitstream for the samples of the basic track. According to another embodiment, a programmable operational feature of the system includes parsing a sub-picture layout from a basic track of a container file; parsing a sample group description entry from the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and wherein any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position; when the sample group description entry indicates a group of sub-picture tracks, selecting a second sub-picture track from the group of sub-picture tracks; parsing from the container file which sample set of the basic track the sample group description entry is intended to be used to reconstruct a video bitstream; and reconstructing from the container file a coded picture of the video bitstream corresponding to the sample set by including time-aligned samples of the first sub-picture track or the second sub-picture track for each sub-picture position of the sub-picture layout.

[0410] A computer program product according to an embodiment may be embodied on a non-transitory computer readable medium. According to another embodiment, the computer program product may be downloaded in the form of data packets via a network.

[0411] Some embodiments have been described with respect to a specific grammar of a grammatical structure. The embodiments apply to generating entities of the described grammatical structure as well as reading, parsing and / or decoding entities of the described grammatical structure.

[0412] If desired, the different functions discussed herein may be performed in a different order and / or simultaneously with other functions. In addition, if desired, one or more of the above functions and embodiments may be optional or may be combined.

[0413] Although various aspects of the embodiment are set out in the independent claims, further aspects comprise other combinations of features from the described embodiments and / or the dependent claims with features of the independent claims and not just the combinations explicitly set out in the claims.

[0414] It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, several variations and modifications may be made without departing from the scope of the present disclosure as defined in the appended claims.

Claims

1. A method for processing a video, comprising: Write two or more sub-picture tracks in a container file; writing in the container file an elementary track, the elementary track being intended to be decomposed into a video bitstream; indicating a sub-picture layout in the base track; writing a sample group description entry in the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position, and wherein the sample group description entry includes an indication of whether a sub-picture identification is carried in a picture header or a parameter set included in the basic track; and Samples of the base track are indicated in the container file, the sample group description entries being intended to be used to reconstruct the video bitstream for the samples of the base track.

2. A method for processing a video, comprising: Parse the sub-image layout from the base track of the container file; parsing a sample group description entry from the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track includes a sub-picture sequence for the corresponding sub-picture position, and any track in the group of sub-picture tracks includes a valid sub-picture sequence for the corresponding sub-picture position, and wherein the sample group description entry includes an indication of whether a sub-picture identification is carried in a picture header or a parameter set included in the basic track; When the sample group description entry indicates a group of sub-picture tracks, selecting a second sub-picture track from the group of sub-picture tracks; parsing from the container file which sample set of the elementary track the sample group description entry is intended to be used for reconstructing a video bitstream; A codec picture of a video bitstream corresponding to the set of samples is reconstructed from the container file by including, for each sub-picture position of the sub-picture layout, time-aligned samples of the first sub-picture track or the second sub-picture track.

3. A device for processing a video, comprising: A component for writing two or more sub-picture tracks in a container file; means for writing in said container file an elementary track, said elementary track being intended to be decomposed into a video bitstream; means for indicating a sub-picture layout in said basic track; means for writing a sample group description entry in the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track comprises a sub-picture sequence for the corresponding sub-picture position, and any track in the group of sub-picture tracks comprises a valid sub-picture sequence for the corresponding sub-picture position, and wherein the sample group description entry comprises an indication of whether a sub-picture identification is carried in a picture header or a parameter set included in the basic track; and means for indicating, in the container file, samples of the elementary track, the sample group description entries being intended to be used for reconstructing the video bitstream for the samples of the elementary track.

4. The apparatus according to claim 3, further comprising: Means for writing in the container file a track reference from the base track to a list of entries, each entry identifying a sub-picture track or a track group of sub-picture tracks; and wherein for each sub-picture position in the sub-picture layout, the sample group description entry comprises an index into the list of entries for each sub-picture position, wherein the index indicates the sub-picture track or the group of sub-picture tracks.

5. The apparatus according to claim 3 or 4, wherein the sample group description entry comprises one or more of the following: The length of the picture identifier syntax element; a bit position of the first sub-picture identifier syntax element; A flag indication of whether a start code emulation prevention byte is present before or within the sub-picture identifier syntax element.

6. The apparatus of claim 3, wherein the sample entry of the sub-picture track comprises one or more of the following: sub-picture identifier; Sub-image location identifier.

7. The apparatus according to claim 3 or 4, further comprising means for writing a sample group for a picture header NAL unit in the container file.

8. An apparatus for processing a video, comprising: A component used to parse the sub-image layout from the basic track of the container file; means for parsing a sample group description entry from the container file, the sample group description entry indicating a first sub-picture track or a group of sub-picture tracks for each sub-picture position in the sub-picture layout, wherein the first sub-picture track comprises a sub-picture sequence for the corresponding sub-picture position, and any track in the group of sub-picture tracks comprises a valid sub-picture sequence for the corresponding sub-picture position, and wherein the sample group description entry comprises an indication of whether a sub-picture identification is carried in a picture header or a parameter set included in the basic track; means for selecting a second sub-picture track from a group of sub-picture tracks when the sample group description entry indicates the group of sub-picture tracks; for parsing from said container file which sample set of said elementary track said sample set description entry is intended to be used for reconstructing a component of a video bitstream; Means for reconstructing a codec picture of a video bitstream corresponding to the set of samples from the container file by including, for each sub-picture position of the sub-picture layout, time-aligned samples of the first sub-picture track or the second sub-picture track.

9. The apparatus according to claim 8, further comprising: means for reading from said container file track references from said base track to a list of entries, each entry identifying a sub-picture track or a track group of sub-picture tracks; And wherein for each sub-picture position in the sub-picture layout, the sample group description entry comprises an index to the list of entries for each sub-picture position, wherein the index indicates a first sub-picture track or the group of sub-picture tracks.

10. The apparatus according to claim 8 or 9, wherein the sample group description entry comprises one or more of the following: The length of the picture identifier syntax element; a bit position of the first sub-picture identifier syntax element; A flag indication of whether a start code emulation prevention byte is present before or within the sub-picture identifier syntax element.

11. The apparatus according to claim 8 or 9, wherein the sample entry of the sub-picture track comprises one or more of the following: sub-picture identifier; a sub-picture location identifier; and The apparatus further comprises means for reading a sample group for a picture header NAL unit from the container file; and Means for indicating a mapping of sub-picture identifiers to said sub-picture layout.

12. The apparatus of claim 11, wherein the means for indicating is configured to override the sub-picture identifier in a parameter set or a picture header by using one or more of the following options: a) inferring whether the sub-picture identifier is carried in a parameter set and / or a picture header; b) if more than one parameter set or picture header contains a sub-picture identifier, inferring a precedence order between parameter sets and picture headers and selecting said parameter set or said picture header having the highest priority; c) if a picture header is selected for overlay, selecting the picture header for overlay as the picture header present in a sample or a picture header of a sample group mapped to the sample in the base track; d) modifying the selected parameter set or picture header to contain the sub-picture identifier of the selected sub-picture track.

13. The apparatus of claim 12, wherein for option d), the apparatus comprises means for performing the following modification: Starting at the bit position of the first sub-picture identifier element and in the order specified in the sample group description entry, overwrite each sub-picture identifier element value with a sub-picture identifier from each selected sub-picture track.

Citation Information

Patent Citations

  • A method, an apparatus and a computer program product for processing image sequence tracks

    CN107534801A

  • Generation device, identification information generation method, reproduction device, and image reproduction method

    CN110100435A

  • Method, device, and computer program for encapsulating partitioned timed media data using sub-track feature

    US20160165321A1

  • Signalling of video content including sub-picture bitstreams for video coding

    US20180277164A1

  • Method, device, and computer program for encapsulating and parsing timed media data

    WO2017202799A1