Video data stream, video encoder, apparatus and method for hrd timing correction, and further addition for scalable and mergeable bitstream

Abstract

To provide a video data stream for achieving a video encoder, a video decoder, an encoding and decoding method, and a high-level video encoding concept.SOLUTION: A video data stream having a video encoded in a video encoder comprises an indication that indicates whether one or more scalable nested supplemental enhancement information messages comprising timing information for each of one or more output layer sets are present within the video data stream.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to video encoding and decoding, and in particular to a video encoder, a video decoder, an encoding and decoding method, and a video data stream for implementing advanced video coding concepts.

[0002] With H.265 / HEVC (HEVC=High Efficiency Video Coding), video codecs are already provided as tools to improve or further enable parallel processing in encoders and / or decoders. For example, HEVC supports subdivision of a picture into an array of tiles that are coded independently of each other. Another concept supported by HEVC is related to WPP, according to which CTU rows or lines of a picture can be processed in parallel from left to right, e.g. in stripes (CTU=coding tree unit), if some minimum CTU offset is respected during processing of consecutive CTU lines. However, it would be advantageous to have a video codec available at any time that supports parallel processing capabilities of video encoders and / or video decoders more efficiently.

[0003] Usually, in video coding, the coding process of picture samples requires smaller partitions, where the samples are divided into several rectangular regions for joint processing such as predictive coding or transform coding. Therefore, pictures are partitioned into blocks of a certain size that is constant during the coding of a video sequence. In the H.264 / AVC standard, fixed-size blocks of 16x16 samples, so-called macroblocks, are used (AVC=Advanced Video Coding).

[0004] In the latest HEVC standard (see [1]), there are coding tree blocks (CTBs) or coding tree units (CTUs) of maximal size 64x64 samples. In further descriptions of HEVC, the more general term CTU is used for such kind of blocks.

[0005] The CTUs are processed in raster scan order, starting with the top-left CTU and processing the CTUs in the picture line-wise to the bottom-right CTU.

[0006] The coded CTU data is organized into a kind of container called a slice. Originally, in previous video coding standards, a slice means a segment containing one or more consecutive CTUs of a picture. Slices are used to segment the coded data. Historically, the term slice continues to be applied, since from another point of view, a complete picture can also be defined as one large segment. In addition to the coded picture samples, slices also contain additional information related to the coding process of the slice itself, which is placed in the so-called slice header.

[0007] According to the state of the art, the VCL (Video Coding Layer) also includes techniques for fragmentation and spatial partitioning. Such partitioning can be applied to video coding for various reasons, such as load balancing in parallel processing, matching CTU size in network transmission, error mitigation, etc.

[0008] Bitstreams specified in video coding standards have HRD conformance related information. This conformance consists of a hypothetical reference decoder (HRD) that includes a buffer model that assumes that NAL units enter a coded picture buffer (CPB) before the decoder and are removed from there at specific times that ensure that the CPB size is not exceeded (buffer overrun) or that NAL units do not arrive later than they should be removed (buffer underrun). Furthermore, this model consists of a decoded picture buffer (DPB) from which decoded pictures are output when they are no longer needed for prediction, and whose size is similarly constrained in many implementations. The timing information of the HRD is conveyed in the bitstream by so-called SEI messages, in particular the buffering period SEI message that defines specific timing information of a buffering period (BP) (multiple access units or AUs), the picture timing (PT) SEI message that conveys timing information of a single associated AU, and the decoding unit information (DUI) SEI message that conveys timing information of an associated AU subset, i.e., a decoding unit or DU.

[0009] Bitstreams specified in video coding standards contain information related to Hypothetical Reference Decoder (HRD) conformance, which consists of a hypothetical buffer model that assumes that NAL units enter the Coded Picture Buffer (CPB) and are removed from it at specific times to ensure that the CPB size is not exceeded (buffer overrun) or that NAL units do not arrive later than they need to be removed (buffer underrun).

[0010] If the bitstream is a scalable bitstream, pruning can be performed to obtain sub-bitstreams that are also conforming bitstreams. For example, if there is an output layer set (OLS) that includes three layers with resolution scalability (e.g., a 480p base layer, a first enhancement layer at 720p, and a second enhancement layer at 1080p), hereafter referred to as B3, two sub-bitstreams can be obtained, one with two layers (480p and 720p) B2 and the other with one layer (480p) B1. Similarly, OLS can be used for temporal scalability, where B3, B2, and B1 have the same resolution but different frame rates.

[0011] Obviously, such bitstreams B3, B2 and B1 have different HRD conformance since the required CPB size, bit rate and timing information may differ.

[0012] The different CPB sizes and bit rates are indicated in the VPS as characteristics of the defined output layer set (3 in the described example). The different timing information is provided by so-called nesting SEI messages. The nested SEI messages may contain nested buffering period SEI and picture timing SEI messages that are applied to the sub-bitstreams that can be obtained by bitstream pruning (bitstream extraction). Then, during this operation (extraction or pruning is performed), for example, the buffering period SEI message and the picture timing SEI message of the input bitstream B3 are removed from the bitstream and also the NAL units belonging to the second enhancement layer are removed. Furthermore, the buffering period SEI message and the picture timing SEI message corresponding to the bitstream B2, carried in a nested SEI message, are placed in the bitstream from the nested SEI message bitstream and thus replace the removed ones.

[0013] It is an object of the present invention to provide an improved video encoding and video decoding concept.

[0014] The object of the present invention is solved by the subject matter of the independent claims.

[0015] Preferred embodiments are provided in the dependent claims.

[0016] According to one embodiment, a video data stream having video encoded therein is provided, the video data stream including an indication of whether one or more scalable nested Supplemental Enhancement Information (SEI) messages are present within the video data stream that include timing information for each of one or more output layer sets.

[0017] Further, according to an embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder for generating a video data stream such that the video data stream includes an indication of whether one or more scalable nested supplemental enhancement information messages including respective timing information for one or more output layer sets are present within the video data stream.

[0018] Further, according to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The apparatus is for processing the input bitstream to obtain sub-bitstreams. An indication indicates whether one or more scalable nested supplemental enhancement information messages are present in the video data stream, the supplemental enhancement information messages including respective timing information for one or more sets of output layers.

[0019] Further, according to an embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method including generating a video data stream such that the video data stream includes an indication of whether one or more scalable nested supplemental enhancement information messages including respective timing information for one or more output tier sets are present within the video data stream.

[0020] Further, according to one embodiment, there is provided a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The method includes processing the input bitstream to obtain sub-bitstreams. An indication is provided of whether one or more scalable nested supplemental enhancement information messages are present in the video data stream that include timing information for each of one or more output layer sets.

[0021] Furthermore, a computer program is provided for implementing one of the above methods when executed on a computer or signal processor.

[0022] Further, according to one embodiment, a video data stream having video encoded therein is provided, wherein an indication in the video data stream indicates whether timing information of the sub-bitstream is obtained from one or more non-scalable nested picture timing supplemental enhancement information messages of the video data stream.

[0023] Further according to an embodiment, there is provided a video data stream having video encoded therein, where a first indication in the video data stream indicates whether timing information of a sub-bitstream is obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream and / or a second indication in the video data stream indicates whether said timing information of a sub-bitstream is obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0024] Furthermore, according to an embodiment, the video data stream includes one or more non-scalable nested supplemental enhancement information messages including timing information. If the video data stream includes a scalable nested supplemental enhancement information message including timing information, this indicates that, depending on the scalable nested supplemental enhancement information message, all of the one or more non-scalable nested timing information extension information messages are substituted by a scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information). Or, a subset including at least one of the one or more non-scalable nested timing information extension information messages is substituted by the non-scalable nested supplemental enhancement information message including said timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information).

[0025] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream such that an indication in the video data stream indicates whether timing information of a sub-bitstream is obtained from one or more non-scalable nested picture timing supplemental enhancement information messages of the video data stream.

[0026] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, wherein a first indication in the video data stream indicates whether timing information of a sub-bitstream is obtained from the one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a second indication in the video data stream indicates whether timing information of a sub-bitstream is obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0027] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates a video data stream, such that the video data stream includes one or more non-scalable nested supplemental enhancement information messages including timing information. If the video data stream includes a scalable nested supplemental enhancement information message including timing information, this indicates that, depending on the scalable nested supplemental enhancement information message, all of the one or more non-scalable nested timing information enhancement information messages are substituted by a scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information). Or, a subset including at least one of the one or more non-scalable nested timing information supplemental enhancement information messages is substituted by the non-scalable nested supplemental enhancement information message including the timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information).

[0028] Further, according to an embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the apparatus being for processing the input bitstream to obtain a sub-bitstream, an indication in the video data stream indicating whether timing information of the sub-bitstream is obtained from one or more non-scalable nested picture timing supplemental enhancement information messages of the video data stream.

[0029] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the apparatus being adapted to process the input bitstream to obtain a sub-bitstream, a first indication in the video data stream indicating whether timing information of the sub-bitstream is obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a second indication in the video data stream indicating whether timing information of the sub-bitstream is obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0030] Further, according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having a video encoded therein. The apparatus is for processing the input bitstream to obtain sub-bitstreams. The video data stream includes one or more non-scalable nested supplemental enhancement information messages including timing information. If the video data stream includes a scalable nested supplemental enhancement information message including timing information, the apparatus is responsive to the scalable nested supplemental enhancement information message to substitute all of the one or more non-scalable nested timing information enhancement information messages by a scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information). Or, a subset including at least one of the one or more non-scalable nested timing information supplemental enhancement information messages is substituted by a non-scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information).

[0031] Further, according to an embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method including generating a video data stream such that an indication within the video data stream is that timing information of a sub-bitstream is obtained from one or more non-scalable nested picture timing supplemental enhancement information messages of the video data stream.

[0032] Further, according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the method including processing the input bitstream to obtain a sub-bitstream, an indication in the video data stream indicating whether timing information of the sub-bitstream is obtained from one or more non-scalable nested picture timing supplemental enhancement information messages of the video data stream.

[0033] Further, according to an embodiment, a method is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, wherein a first indication in the video data stream indicates whether timing information of the sub-bitstreams is obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a second indication in the video data stream indicates whether timing information of the sub-bitstreams is obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0034] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the method comprising: processing the input bitstream to obtain a sub-bitstream, where a first indication in the video data stream indicates whether timing information of the sub-bitstream is obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a second indication in the video data stream indicates whether timing information of the sub-bitstream is obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0035] Further, according to an embodiment, a method is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating a video data stream such that the video data stream includes one or more non-scalable nested supplemental enhancement information messages including timing information. If the video data stream includes a scalable nested supplemental enhancement information message including timing information, this indicates that at least one of the one or more non-scalable nested picture timing supplemental enhancement information messages is substituted by a scalable nested supplemental enhancement information message including timing information.

[0036] Further, according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The method includes processing the input bitstream to obtain sub-bitstreams. The video data stream includes one or more non-scalable nested supplemental enhancement information messages that include timing information. If the video data stream includes a scalable nested supplemental enhancement information message that includes timing information, the method includes at least one of the one or more non-scalable nested picture timing supplemental enhancement information messages being substituted by a scalable nested supplemental enhancement information message that includes timing information.

[0037] Furthermore, a computer program is provided for implementing one of the above methods when executed on a computer or signal processor.

[0038] Further, according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, where for each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set are equal for all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0039] Further, according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, and for each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video data stream including an indication of whether the buffer period information and / or picture timing information of the output layer set are equal in all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0040] Further, according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, where for each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer periodicity information and / or picture timing information of an output layer set, the buffer periodicity information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0041] Further according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, where for each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0042] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates a video data stream, such that the video data stream includes a plurality of access units. If, for each access unit of the plurality of access units, the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video encoder generates the video data stream, such that the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set are equal for all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0043] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates a video data stream, such that the video data stream includes a plurality of access units. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0044] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates a video data stream, such that the video data stream includes a plurality of access units. If, for each access unit of the plurality of access units, the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of the output layer set, the video encoder generates the video data stream, such that the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0045] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The apparatus processes the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set are equal for all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0046] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The apparatus processes the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units. If the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set are equal in all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0047] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units, and the apparatus processes the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0048] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The apparatus processes the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of the output layer set, the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0049] Further according to an embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating a video data stream such that the video data stream includes a plurality of access units. For each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the method includes generating the video data stream such that the buffer period information and / or picture timing information of the output layer set are equal for all two or more scalable nested supplemental enhancement information messages of the access unit.

[0050] Further, according to an embodiment, a method is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating a video data stream such that the video data stream includes a plurality of access units. For each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output tier set, the method includes generating the video data stream such that the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output tier set are equal for all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0051] Further according to an embodiment, a method is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating a video data stream, such that the video data stream includes a plurality of access units. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages that include buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0052] Further according to an embodiment, a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method includes generating a video data stream such that the video data stream includes a plurality of access units, and if for each access unit of the plurality of access units, the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of the output layer set, the method includes generating the video data stream such that the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0053] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The method includes processing the access unit of the input bitstream to obtain a sub-bitstream. For each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set are equal for all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0054] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The method includes processing the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set are equal in all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0055] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The method includes processing the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0056] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The method includes processing the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of the output layer set, the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0057] Furthermore, a computer program is provided for implementing one of the above methods when executed on a computer or signal processor.

[0058] Further according to an embodiment there is provided a video data stream having video encoded therein, the video data stream comprising a plurality of access units, the video data stream comprising a spreading factor dependent on a number of sub-bitstreams of the video data stream or the video data stream comprising a clock sub-tick value dependent on a highest sub-bitstream among the sub-bitstreams of the video data stream.

[0059] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream, such that the video data stream comprises a plurality of access units. Further, the video encoder generates the video data stream, such that the video data stream comprises a spreading factor that is dependent on a number of sub-bitstreams of the video data stream, or the video encoder generates the video data stream, such that the video data stream comprises a clock sub-tick value that is dependent on the highest sub-bitstream of the sub-bitstreams of the video data stream.

[0060] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream, such that the video data stream includes sub-layer specific frame rate information for the sub-layer, and / or the video encoder generating the video data stream, such that the video data stream includes sub-layer specific frame display time information for the sub-layer.

[0061] Further according to an embodiment, there is provided a video decoder for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video data stream comprising a plurality of access units, the video decoder is for decoding the video data stream to decode the video, the video data stream comprising a spreading factor dependent on a number of sub-bitstreams of the video data stream, the video decoder is for decoding the video using the spreading factor, or the video data stream comprises a clock sub-tick value dependent on a highest sub-bitstream of the sub-bitstreams of the video data stream, the video decoder is for decoding the video using the clock sub-tick value.

[0062] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. A video decoder is for decoding the video data stream to decode the video. The video data stream includes sub-layer specific frame rate information of a sub-layer and / or the video data stream includes sub-layer specific frame display period information for the sub-layer. The decoder is for determining a spreading factor using the sub-layer specific frame rate information of the sub-layer and / or using the sub-layer specific frame display period information.

[0063] Further according to an embodiment there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method comprising: generating a video data stream, such that the video data stream comprises a plurality of access units, the method comprising: generating a video data stream, such that the video data stream comprises a spreading factor that is dependent on a number of sub-bitstreams of the video data stream, or the method comprises generating a video data stream, such that the video data stream comprises a clock sub-tick value that is dependent on a highest sub-bitstream of the sub-bitstreams of the video data stream.

[0064] Further according to an embodiment, there is provided a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video data stream including a plurality of access units, the method including decoding the video data stream to decode the video, the video data stream including a spreading factor that is dependent on a number of sub-bitstreams of the video data stream, the method including decoding the video using the spreading factor, or the video data stream including a clock sub-tick value that is dependent on a highest sub-bitstream of the sub-bitstreams of the video data stream, the method including decoding the video using the clock sub-tick value.

[0065] Further according to an embodiment is a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The method includes decoding the video data stream to decode the video. The video data stream includes sub-layer specific frame rate information for a sub-layer and / or the video data stream includes sub-layer specific frame display period information for a sub-layer. The method includes determining a spreading coefficient using the sub-layer specific frame rate information of the sub-layer and / or using the sub-layer specific frame display period information.

[0066] Furthermore, a computer program is provided for implementing one of the above methods when executed on a computer or signal processor.

[0067] According to an embodiment, a video decoder is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video decoder is adapted to decode the video data stream to decode the video. To decode the video, the video decoder estimates a coded picture buffer size for a sub-picture as a function of information in the video data stream indicating current coded picture buffer size information.

[0068] Further according to an embodiment, a video decoder is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video decoder being configured to decode the video data stream to decode said video, for decoding the video, the video decoder being configured to estimate a sub-picture bitrate as a function of information in the video data stream indicating bitrate information of a currently coded video sequence.

[0069] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, and a video decoder for decoding the video data stream to decode said video, wherein to decode the video, the video decoder receives a coded picture buffer size of a sub-picture coded in the video data stream and uses the coded picture buffer size of the sub-picture to decode the video and / or the video decoder receives a bitrate of the sub-picture coded in the video data stream and decodes the video using the bitrate of the sub-picture.

[0070] Further, according to an embodiment, a video data stream is provided having video encoded therein, the video data stream for example comprising the syntax element cpb_size_value_minus1[i][j] and the syntax element cpb_size_scale, or the video data stream for example comprising the syntax element bit_rate_value_minus1[i][j] and the syntax element bit_rate_scale.

[0071] Further, according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including an indication indicating whether a coded picture buffer size of a sub-picture should be estimated using current coded picture buffer size information and / or the video data stream including an indication indicating whether a bitrate of a sub-picture is estimated using bitrate information of a currently coded video sequence.

[0072] Further according to an embodiment there is provided a video data stream having video encoded therein, such that the video data stream includes an indication indicating whether a coded picture buffer size for a sub-picture has been encoded in the video data stream or whether the coded picture buffer size for a sub-picture should be estimated and / or the video data stream includes an indication indicating whether a bitrate for a sub-picture has been encoded in the video data stream or whether the bitrate for a sub-picture needs to be estimated.

[0073] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream, such that the video data stream includes for example the syntax element cpb_size_value_minus1[i][j] and the syntax element cpb_size_scale. Or, the video encoder generates the video data stream, such that the video data stream includes the syntax element bit_rate_value_minus1[i][j] and the syntax element bit_rate_scale.

[0074] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream such that the video data stream includes an indication indicating whether a coded picture buffer size of a sub-picture should be estimated using current coded picture buffer size information and / or the video encoder generating the video data stream such that the video data stream includes generating an indication indicating whether a bitrate of the sub-picture is estimated using currently coded video sequence bitrate information.

[0075] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream such that the video data stream includes an indication indicating whether a coded picture buffer size of a sub-picture has been encoded in the video data stream or whether a coded picture buffer size of a sub-picture should be estimated, and / or the video encoder generating the video data stream such that the video data stream includes an indication indicating whether a bitrate of a sub-picture has been encoded in the video data stream or whether a bitrate of a sub-picture needs to be estimated.

[0076] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that, when the video data stream includes common decoding unit removal timing information and multiple extractable sub-bitstreams, each of the multiple sub-bitstreams is specific to an output layer set, and parsing each output layer set specific hypothetical reference decoder parameter in a video parameter set or a sequence parameter set or a supplemental enhancement information message of the video data stream includes either a spreading factor or an absolute value of a tick divisor in scaling the removal timing of the common decoding unit.

[0077] Further according to an embodiment, there is provided a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the method comprising: decoding the video data stream to decode the video, the method comprising: estimating a coded picture buffer size of a sub-picture as a function of information in the video data stream indicating current coded picture buffer size information.

[0078] Further according to an embodiment there is provided a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the method comprising: decoding the video data stream to decode the video, for decoding the video the method comprises estimating a sub-picture bitrate as a function of information in the video data stream indicative of currently coded video sequence bitrate information.

[0079] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the method comprising: decoding the video data stream to decode the video, for decoding the video, the method comprising receiving a coded picture buffer size of a sub-picture coded in the video data stream and using the coded picture buffer size of the sub-picture to decode the video, and / or the method comprising receiving a bitrate of the sub-picture coded in the video data stream and decoding the video using the bitrate of the sub-picture.

[0080] Further according to an embodiment there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method comprising generating the video data stream such that the video data stream includes syntax elements cpb_siges_veret_minus1[i][j] and cpb_zesic_class, or the method comprises generating the video data stream such that the video data stream includes syntax elements bit_rate_value_minus1[i][j] and bit_rate_scale.

[0081] Further according to an embodiment there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method comprising generating the video data stream such that the video data stream includes an indication indicating whether a coded picture buffer size of a sub-picture should be estimated using current coded picture buffer size information and / or the method comprises generating the video data stream such that the video data stream includes an indication indicating whether a bitrate of a sub-picture is estimated using bitrate information of a currently coded video sequence.

[0082] Further according to an embodiment there is provided a method for encoding video into a video data stream such that the video data stream has video encoded therein, the method comprising: generating a video data stream such that the video data stream includes an indication indicating whether a coded picture buffer size for a sub-picture has been encoded within the video data stream or whether a coded picture buffer size for a sub-picture should be estimated, and / or the method comprises generating a video data stream such that the video data stream includes an indication indicating whether a bit rate for a sub-picture has been encoded within the video data stream or whether a bit rate for a sub-picture needs to be estimated.

[0083] Furthermore, a computer program is provided for implementing one of the above methods when executed on a computer or signal processor.

[0084] According to an embodiment, there is provided a video data stream having video encoded therein, the video data stream including a plurality of access units, the video data stream further including delta time information for each of two or more decoding units of the access units of the plurality of access units, a decoding unit removal time for each of the two or more decoding units of the access units being dependent on the access unit removal time of the access unit and dependent on the delta time information of the decoding unit.

[0085] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream, such that the video data stream includes a plurality of access units. Further, the video encoder generates the video data stream, such that the video data stream includes delta time information for each of two or more decoding units of an access unit of the plurality of access units, where a decoding unit removal time of each decoding unit of the two or more decoding units of the access unit depends on the access unit removal time of the access unit and depends on the delta time information of the decoding unit.

[0086] Further, according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes a plurality of access units. A video decoder is for decoding the video data stream to decode video. Furthermore, the video data stream includes delta time information for each of two or more decoding units of an access unit of the plurality of access units, a decoding unit removal time of each decoding unit of the two or more decoding units of the access unit is dependent on the access unit removal time of the access unit and dependent on the delta time information of the decoding unit, and the video decoder is for decoding the video data stream using the delta time information for each of the two or more decoding units of the access unit.

[0087] Further according to an embodiment there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method including generating a video data stream such that the video data stream includes a plurality of access units, the method further including generating the video data stream such that the video data stream includes delta time information for each of two or more decoding units of an access unit of the plurality of access units, and a decoding unit removal time of each decoding unit of the two or more decoding units of the access unit is dependent on the access unit removal time of the access unit and dependent on the delta time information of the decoding unit.

[0088] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes a plurality of access units. The method includes decoding the video data stream to decode the video. Furthermore, the video data stream includes delta time information for each of two or more decoding units of an access unit of the plurality of access units, a decoding unit removal time of each decoding unit of the two or more decoding units of the access unit is dependent on the access unit removal time of the access unit and dependent on the delta time information of the decoding unit, and the method includes decoding the video data stream using the delta time information for each of the two or more decoding units of the access unit.

[0089] Furthermore, a computer program is provided for implementing one of the above methods when executed on a computer or signal processor. [Brief description of the drawings]

[0090] [Figure 1] 1 illustrates a video encoder for encoding video into a video data stream according to an embodiment; [Diagram 2] 1 shows an apparatus for receiving an input video data stream according to an embodiment. [Diagram 3] 1 illustrates a video decoder for receiving a video data stream having video stored therein, according to an embodiment; [Figure 4] The change in removal time when there are three decode units per access unit is shown. [Diagram 5] Two layers are shown, along with the removal times for access units and decode units for both layers. [Figure 6] 1 shows a video encoder. [Figure 7] 1 shows a video decoder. [Figure 8]It shows the relationship between a reconstructed signal, e.g. a reconstructed picture, on the one hand, and a combination of a prediction residual signal and a prediction signal signaled in a data stream, on the other hand. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0091] The following description of the figures begins with the presentation of an illustration of an encoder and decoder of a block-based predictive codec for encoding pictures of a video, to form an example of an encoding framework in which embodiments of the present invention may be incorporated. The respective encoders and decoders are described with respect to Figures 6 to 8. Below, an illustration of embodiments of the inventive concepts is presented together with an explanation of how such concepts may be incorporated into the respective encoders and decoders of Figures 6 and 7, although the embodiments described in Figures 1 to 3 and thereafter may also be used to form encoders and decoders that do not operate according to the encoding framework underlying the encoders and decoders of Figures 6 and 7.

[0092] FIG. 6 shows a video encoder, illustratively an apparatus for predictively coding a picture 12 into a data stream 14 using transform-based residual coding. The apparatus, or encoder, is indicated using the reference symbol 10. FIG. 7 shows a corresponding video decoder 20, e.g., apparatus 20, which is also configured to predictively decode a picture 12' from a data stream 14 using transform-based residual decoding, the apostrophe being used to indicate that the picture 12' reconstructed by the decoder 20 deviates from the picture 12 originally coded by apparatus 10 in terms of coding losses introduced by quantization of the predictive residual signal. Although FIGS. 6 and 7 illustratively use transform-based predictive residual coding, embodiments of the present application are not limited to this type of predictive residual coding. This also applies to other details described with respect to FIGS. 6 and 7, as outlined hereafter.

[0093] The encoder 10 is configured to subject the prediction residual signal to a spatial-to-spectral transform and to encode the prediction residual signal into a data stream 14, in which the prediction residual signal is obtained. Similarly, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and subject the prediction residual signal to a spectral-to-spatial transform, in which the prediction residual signal is obtained.

[0094] Internally, the encoder 10 may include a prediction residual signal former 22, which generates a prediction residual 24 to measure the deviation of a prediction signal 26 from an original signal, e.g., the picture 12. The prediction residual signal former 22 may, for example, be a subtractor, which subtracts the prediction signal from the original signal, e.g., the picture 12. The encoder 10 then further includes a transformer 28, which subjects the prediction residual signal 24 to a spatial-to-spectral transformation to obtain a spectral domain prediction residual signal 24'. The spectral domain prediction residual signal is then subjected to quantization by a quantizer 32 and included by the encoder 10. The quantized prediction residual signal 24" is therefore encoded into the bitstream 14. To this end, the encoder 10 may optionally include an entropy coder 34, which entropy codes the transformed and quantized prediction residual signal into the data stream 14. The prediction signal 26 is generated by a prediction stage 36 of the encoder 10 based on the prediction residual signal 24", which is encoded into the data stream 14 and is decodable therefrom. To this end, the prediction stage 36 may internally include, as shown in FIG. 6, an inverse quantizer 38 for inverse quantizing the prediction residual signal 24″ to obtain a spectral domain prediction residual signal 24′″ corresponding to the signal 24′ except for quantization losses, followed by an inverse transformer 40 for subjecting the latter prediction residual signal 24′″ to an inverse transformation, e.g. a spectral to spatial transformation, to obtain a prediction residual signal 24″″ corresponding to the original prediction residual signal 24 except for quantization losses. A combiner 42 of the prediction stage 36 then recombines the prediction signal 26 and the prediction residual signal 24″″, e.g. by addition, to obtain a reconstructed signal 46, e.g. a reconstruction of the original signal 12. The reconstructed signal 46 may correspond to the signal 12′. A prediction module 44 of the prediction stage 36 then generates the prediction signal 26 based on the signal 46, e.g. by using spatial prediction (e.g. intra-picture prediction) and / or temporal prediction (e.g. inter-picture prediction).

[0095] Similarly, the decoder 20 may be internally constructed from components corresponding to the prediction stage 36 and interconnected in a manner corresponding to the prediction stage, as shown in Figure 7. In particular, an entropy decoder 50 of the decoder 20 may entropy decode a quantized spectral domain prediction residual signal 24'' from the data stream, where an inverse quantizer 52, an inverse transformer 54, a synthesizer 56 and a prediction module 58 are interconnected and cooperate in the manner described above with respect to the modules of the prediction stage 36 to recover a reconstructed signal based on the prediction residual signal 24'', with the output of the synthesizer 56 resulting in a reconstructed signal, i.e., picture 12', as shown in Figure 7.

[0096] Although not specifically mentioned above, it is easily apparent that the encoder 10 may set some coding parameters, including, for example, prediction mode, motion parameters, etc., according to some optimization schemes, such as, for example, a method for optimizing some rate- and distortion-related criteria, such as, for example, coding cost. For example, the encoder 10 and the decoder 20 and corresponding modules 44, 58 may each support different prediction modes, such as intra- and inter-coding modes. The granularity with which the encoder and the decoder switch between their prediction mode types may correspond to the subdivision of the pictures 12 and 12′, respectively, into coding segments or coding blocks. In units of these coding segments, for example, the picture may be subdivided into intra-coded blocks and inter-coded blocks. The intra-coded blocks are predicted based on the already coded / decoded neighborhood of the respective block space, as outlined in more detail below. Several intra-coding modes may be present and selected for each intra-coded segment, including directional or angular intra-coding modes, according to which each segment is filled by extrapolating neighboring sample values ​​along a particular direction specific to each directional intra-coding mode into each intra-coded segment. These intra-coding modes may also include one or more further modes, such as, for example, a DC coding mode, according to which the prediction of each intra-coded block assigns a DC value to all samples in the respective intra-coded segment, and / or a planar intra-coding mode, according to which the prediction of each block is approximated and determined to be a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample positions of each intra-coded block as the slope and offset of the plane defined by the two-dimensional linear function based on neighboring samples. In comparison, inter-coded blocks may be predicted, for example, temporally.In the case of inter-coded blocks, motion vectors may be signaled in the data stream, indicating the spatial displacement of parts of previously coded pictures of the video to which the picture 12 belongs, where the previously coded / decoded pictures are sampled to obtain a prediction signal for the respective inter-coded block. This means that in addition to the coding of the residual signal that the data stream 14 contains, such as the entropy coded transform coefficient levels representing the quantized spectral domain prediction residual signal 24'', the data stream 14 may also contain coded therein coding mode parameters for assigning coding modes to the various blocks, prediction parameters of some of the blocks, such as motion parameters of the inter-coded segments, and optional further parameters, such as parameters for controlling and signaling the subdivision of the pictures 12 and 12' into respective segments. The decoder 20 uses these parameters to subdivide the picture in the same way as the encoder did, to assign the same prediction modes to the segments and to perform the same prediction, resulting in the same prediction signal.

[0097] FIG. 8 shows the relationship between a reconstructed signal, for example a reconstructed picture 12′, on the one hand, and a combination of a prediction residual signal 24″″ signaled in the data stream 14 and a prediction signal 26, on the other hand. As already indicated above, this combination may be additive. The prediction signal 26 is shown in FIG. 8 as a subdivision of the picture area into intra-coded blocks, exemplarily shown using hatching, and inter-coded blocks, exemplarily shown without hatching. This subdivision may be any subdivision, such as a regular subdivision of the picture area into rows and columns of square or non-square blocks, or a multi-tree subdivision of the picture 12 from a tree root block into several leaf blocks of various sizes (such as a quad-tree subdivision, for example), a combination of which is shown in FIG. 8, in which the picture area is first subdivided into rows and columns of a tree root block, which are then further subdivided into one or more leaf blocks according to a recursive multi-tree subdivision.

[0098] Again, data stream 14 may have an intra-coding mode coded therein for intra-coded blocks 80, such that one of several supported intra-coding modes is assigned to each intra-coded block 80. In the case of inter-coded blocks 82, data stream 14 may have one or more motion parameters coded therein. In general, inter-coded blocks 82 are not restricted to being temporally coded. Instead, inter-coded blocks 82 may be any block that is predicted from a previously coded portion beyond the current picture 12 itself, for example, from a previously coded picture of the video to which picture 12 belongs, or from a picture of another view or a hierarchically lower layer if the encoder and decoder are scalable encoder and decoder, respectively.

[0099] In FIG. 8, the prediction residual signal 24'''' is also shown as a subdivision of the picture domain into blocks 84. These blocks may be referred to as transform blocks to distinguish the same from the coding blocks 80 and 82. In fact, FIG. 8 shows that the encoder 10 and the decoder 20 may use two different subdivisions of the picture 12 and the picture 12' into blocks, one into the respective coding blocks 80 and 82, and another into the transform blocks 84. Although both subdivisions may be the same, e.g. the coding blocks 80 and 82, respectively, and may form the transform blocks 84 at the same time, FIG. 8 shows the case, e.g., where the subdivision into the transform blocks 84 forms an extension of the subdivision into the coding blocks 80, 82, so that any boundary between the two blocks 80 and 82 overlaps with the boundary between the two blocks 84, or in other words, where each of the blocks 80, 82 coincides with one of the transform blocks 84 or coincides with a cluster of the transform blocks 84. However, these subdivisions may also be determined or selected independently of one another such that transformation blocks 84 may alternately cross block boundaries between blocks 80 and blocks 82. Thus, as far as the subdivision into transformation blocks 84 is concerned, similar statements are true as those made with regard to the subdivision into blocks 80, 82; for example, block 84 may be the result of a regular subdivision of the picture region into blocks (with or without arrangement into rows and columns), the result of a recursive multi-tree subdivision of the picture region, or a combination thereof or any other type of block result. As an aside, it is noted that blocks 80, 82 and 84 are not limited to being square, rectangular or of any other shape.

[0100] 8 further illustrates that the combination of the prediction signal 26 and the prediction residual signal 24'''' directly results in the reconstructed signal 12'. However, it should be noted that, according to alternative embodiments, more than one prediction signal 26 may be combined with the prediction residual signal 24'''' to result in picture 12'.

[0101] In Fig. 8, the transform blocks 84 have the following significance: The transformer 28 and the inverse transformer 54 perform their transforms in units of their transform blocks 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84. Some codecs allow to skip the transform, so that for some transform blocks 84, the prediction residual signal is directly coded in the spatial domain. However, according to the embodiment below, the encoder 10 and the decoder 20 are configured in such a way that they support several transforms. For example, the transforms supported by the encoder 10 and the decoder 20 are: o DCT-II (or DCT-III) (DCT stands for Discrete Cosine Transform) o DST-IV (DST stands for Discrete Sine Transform) DCT-IV DST-VII o Identity Transformation (IT) may include

[0102] Naturally, the transformer 28 supports all of the forward transform versions of those transforms, but the decoder 20 or inverse transformer 54 supports only their corresponding backward or inverse versions, o Inverse DCT-II (or Inverse DCT-III), o Reverse DST-IV o Inverse DCT-IV o Reverse DST-VII o Identity Transformation (IT) Support.

[0103] In the ensuing description, further details are provided on which transforms may be supported by the encoder 10 and the decoder 20. Note that in any case, the supported transform set may include only one transform, such as one spectral-to-spatial or spatial-to-spectral transform.

[0104] As already outlined above, Figures 6 to 8 are presented as an example in which the inventive concepts further described below may be implemented to form specific examples of encoders and decoders according to the present application. So far, the encoders and decoders of Figures 6 and 7, respectively, may represent possible implementations of the encoders and decoders described below in this specification. However, Figures 6 and 7 are merely examples. However, an encoder according to an embodiment of the present application may use concepts outlined in more detail below to perform block-based encoding of picture 12 that differs from the encoder of Figure 6, for example in that the same is not a video encoder but is still a picture encoder, in that the same does not support inter prediction, or in that the subdivision into blocks 80 is performed in a different way than the way illustrated in Figure 8. Similarly, a decoder according to an embodiment of the present application may perform block-based decoding of picture 12′ from data stream 14 using the encoding concepts further outlined below, but may differ from the decoder 20 of FIG. 7 in, for example, that it is not a video decoder but is still a picture decoder, that it does not support intra prediction, or that it subdivides picture 12′ into blocks in a different way than described with respect to FIG. 8, and / or that it derives prediction residuals from data stream 14 in, for example, the spatial domain rather than the transform domain.

[0105] 1 shows a video encoder 100 for encoding video into a video data stream according to one embodiment. The video encoder 100 is configured to generate a video data stream.

[0106] 2 shows an apparatus 200 for receiving an input video data stream having video encoded therein, according to one embodiment, the apparatus 200 being configured to generate an output video data stream from the input video data stream.

[0107] 3 shows a video decoder 300 for receiving a video data stream having video stored therein, according to one embodiment. The video decoder 300 is configured to decode video from the video data stream.

[0108] Further, a system according to an embodiment is presented, which includes the device of Fig. 2 and the video decoder of Fig. 3. The video decoder (300) of Fig. 3 is configured to receive the output video data stream of the device (200) of Fig. 2. The video decoder 300 of Fig. 3 is configured to decode video from the output video data stream of the device 200 of Fig. 2.

[0109] In one embodiment, the system may further include, for example, the video encoder 100 of Figure 1. The device 200 of Figure 2 may be configured to receive, for example, a video data stream from the video encoder 100 of Figure 1 as an input video data stream.

[0110] The intermediate device 210 (optional) of the apparatus 200 may be configured, for example, to receive a video data stream from the video encoder 100 as an input video data stream and to generate an output video data stream from the input video data stream. For example, the intermediate device may be configured, for example, to modify information (header / metadata information) of the input video data stream and / or may be configured, for example, to remove pictures from the input video data stream and / or may be configured to mix / splice the input video data stream with an additional second bitstream having a second video encoded therein.

[0111] The video decoder 221 (optional) may be configured, for example, to decode video from the output video data stream.

[0112] The hypothetical reference decoder 222 (optional) may be configured, for example, to determine timing information of the video depending on the output video data stream, or may be configured, for example, to determine buffer information of a buffer in which the video or a portion of the video is stored.

[0113] The system includes a video encoder 101 of FIG. 1 and a video decoder 151 of FIG.

[0114] The video encoder 101 is configured to generate an encoded video signal. The video decoder 151 is configured to decode the encoded video signal to reconstruct video pictures.

[0115] Specific embodiments are described below.

[0116] In HEVC, a note in the Extraction Process specification explains the handling of nested SEI messages as follows:

[0117] A "smart" bitstream extractor may include appropriate non-scalable nested buffering picture SEI messages, non-scalable nested picture timing SEI messages, and non-scalable nested decoding unit information SEI messages in the extracted sub-bitstream, provided that the messages applicable to the sub-bitstream were present as scalably nested SEI messages in the original bitstream.

[0118] In VVC, the assumed design is set in place such that the behavior is normatively specified, e.g., for the extraction process defined in JVET-P2001-vC with the embodiment of the present invention added, as follows:

[0119] Sub-bitstream Extraction Process The inputs to this process are the bitstream inBitstream, the target OLS index targetOlsIdx, and the target highest TemporalId value tIdTarget.

[0120] The output of this process is the sub-bitstream outBitstream. It is a requirement of the input bitstream that any output sub-bitstream that is the output of a process specified in this clause shall have targetOlsIdx equal to an index in the list of OLSs specified in the VPS, and tIdTarget equal to any value in the range 0 to 6, inclusive, and that a conforming bitstream shall be one that satisfies the following conditions: · The output sub-bitstream contains at least one VCL NAL unit with nuh_layer_id value equal to each of LayerIdInOls[targetOlsIdx]. · The output sub-bitstream contains at least one VCNLAL unit whose TemporalId is equal to tIdTarget. NOTE - A compliant bitstream contains one or more coded slice NAL units with TemporalId equal to 0, but is not required to contain any coded slice NAL units with nuh_layer_id equal to 0. The output sub-bitstream OutBitstream is derived as follows. The bitstream outBitstream is set to be the same as the bitstream inBitstream. Remove all NAL units with TemporalId greater than tIdTarget from outBitstream. · Remove all NAL units from outBitstream whose nal_unit_type is not equal to any of VPS_NUT, DPS_NUT, and EOB_NUT and whose nuh_layer_id is not included in the list LayerIdInOls[targetOlsIdx]. Remove from outBitstream all SEI NAL units, including any scalable nesting SEI message, where nesting_ols_flag is 1 and there is no value of i in the range from 0 to nesting_num_olss_minus1, inclusive, where NestingOlsIdx[i] is the same as targetOlsIdx. If targetOlsIdx is greater than 0, remove all SEI NAL units from outBitstream that contain non-scalable nested SEI messages with payloadType equal to 0 (buffering period), 1 (picture timing), or 130 (decode unit information).

[0121] According to a particular embodiment: · If outBitstream contains a scalable nesting SEI message with nesting_ols_flag equal to 1 and contains SEI NAL units applicable to outBitstream (NestingOlsIdx[i] equals targetOlsIdx), do the following: From the scalable nested SEI message, extract the appropriate non-scalable nested SEI messages with payloadType equal to 0 (buffering period), 1 (image timing), or 130 (decode information) and put those messages into outBitstream. Remove all SEI NAL units, including scalable nested SEI messages, from outBitstream.

[0122] In the following, we explain the presence of OLS scalable nested SEI messages in the bitstream.

[0123] According to an embodiment, there is provided a video data stream having video encoded therein, the video data stream including an indication of whether one or more scalable nested supplemental enhancement information messages are present within the video data stream that include timing information for each of one or more sets of output layers.

[0124] In an embodiment, the indication is a parameter set flag, e.g., the video data stream includes a parameter set flag indicating whether one or more scalable nested supplemental enhancement information messages containing timing information for each of one or more output layer sets are present in the video data stream.

[0125] According to an embodiment, a video data stream as claimed in claim 2, the sequence parameter sets of the video data stream may for example comprise a parameter set flag.

[0126] In an embodiment, the parameter setting flag is sps_ols_nest_timing_present_flag.

[0127] According to an embodiment, the video data stream may include further supplemental enhancement information messages, which may include, for example, a parameter set flag indicating whether one or more scalable nested supplemental enhancement information messages for each of one or more output layer sets are present in the video data stream.

[0128] In an embodiment, the timing information may include, for example, at least one of picture timing information, buffering period information, and decoding unit information.

[0129] In an embodiment, the timing information is hypothetical reference decoder timing information.

[0130] Further, according to an embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder for generating a video data stream such that the video data stream includes an indication of whether one or more scalable nested supplemental enhancement information messages including respective timing information for one or more output layer sets are present within the video data stream.

[0131] In an embodiment, the indication is a parameter set flag. The video encoder may, for example, be configured to generate the video data stream such that the video data stream may, for example, include a parameter set flag indicating whether one or more scalable nested supplemental enhancement information messages including timing information for each of one or more output layer sets are present in the video data stream.

[0132] In an embodiment, the video encoder may be configured, for example, to generate a video data stream such that a series of parameter sets for the video data stream may include, for example, a decoding parameter set.

[0133] In an embodiment, a video encoder may for example be configured to generate a video data stream such that the parameter set flag is sps_ols_nest_timing_present_flag.

[0134] In an embodiment, the video encoder is, for example, configured to generate the video data stream such that the video data stream may, for example, include further supplemental enhancement information messages, which may, for example, include a parameter set flag. The video encoder may, for example, be configured to generate the video data stream such that the parameter set flag indicates whether one or more scalable nested supplemental enhancement information messages for each of the one or more output layer sets are present in the video data stream.

[0135] According to an embodiment, the timing information may include, for example, at least one of picture timing information, buffering period information, and decoding unit information.

[0136] In an embodiment, the timing information is hypothetical reference decoder timing information.

[0137] Further, according to an embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The apparatus is for processing the input bitstream to obtain sub-bitstreams. An indication indicates whether one or more scalable nested supplemental enhancement information messages are present in the video data stream, the one or more scalable nested supplemental enhancement information messages including respective timing information for one or more output layer sets.

[0138] In an embodiment, the indication is a parameter set flag. The device may for example be configured to process a video data stream that includes a parameter set flag that indicates whether one or more scalable nested supplemental enhancement information messages that include timing information for each of one or more output tier sets are present in the video data stream.

[0139] In an embodiment, the video data stream of a series of parameter sets may, for example, include a parameter set flag.

[0140] According to an embodiment, the parameter setting flag is sps_ols_nest_timing_present_flag.

[0141] In an embodiment, the video data stream may include a further supplemental enhancement information message, which may include, for example, a parameter set flag indicating whether one or more scalable nested supplemental enhancement information messages for each of one or more output layer sets are present in the video data stream. The device may be configured to process the further supplemental enhancement information message, for example.

[0142] According to an embodiment, the timing information may include, for example, at least one of picture timing information, buffering period information, and decoding unit information.

[0143] According to an embodiment, if one or more scalable nested supplemental enhancement information messages include timing information that may include, for example, picture timing information for each of one or more output layer sets present in the video data stream, the device may be configured, for example, to substitute picture timing information of a non-scalable nested picture timing supplemental enhancement information message. If one or more scalable nested supplemental enhancement information messages include timing information including buffering period information for each of one or more output layer sets present in the video data stream, the device may be configured, for example, to substitute buffering period information of a non-scalable nested buffering period supplemental enhancement information message. If one or more scalable nested supplemental enhancement information messages include timing information including decoding unit information for each of one or more output layer sets present in the video data stream, the device may be configured, for example, to substitute decoding unit information of a non-scalable nested decoding unit supplemental enhancement information message.

[0144] In an embodiment, the timing information is hypothetical reference decoder timing information.

[0145] According to an embodiment, the device may be configured to, for example, decode the sub-bitstream to decode the video.

[0146] Further, according to an embodiment, a system for encoding video into a video data stream and for decoding the video is provided. The system includes a video encoder as described above, and an apparatus as described above. The video encoder may for example be configured to encode the video into the video data stream, such that the video data stream has the video encoded therein. The apparatus may for example be configured to receive the video data stream as an input bitstream. Further, the apparatus may for example be configured to process the input bitstream to obtain a sub-bitstream. Further, the apparatus may for example be configured to decode the sub-bitstream to decode the video.

[0147] The VVC draft specification includes a definition of an OLS within a VPS that can also be used for HRD-based conformance testing of OLS sub-bitstreams based on the respective HRD SEI messages (BP, PT, DUI) present in the bitstream in a nested format (scalable nested SEI message nesting). When defining an OLS, it is important to ensure that the HRD SEI messages for each of these OLS are present in the bitstream for conformance testing to be valid.

[0148] Therefore, it is part of the invention that an indication is in the bitstream (e.g., a parameter set flag such as sps_ols_scal_nest_timing_present_flag for SPS or a new SEI message containing such a flag) indicating scalable nested SEI messages to all OLSs must be present in the bitstream.

[0149] The following describes how the HRD SEI is applied to the sub-bitstreams.

[0150] According to an embodiment, a video data stream having video encoded therein is provided, wherein an indication in the video data stream indicates whether timing information of a sub-bitstream is obtained from one or more non-scalable nested picture timing supplemental enhancement information messages of the video data stream.

[0151] In an embodiment, if the indication indicates that sub-bitstream timing information is not obtained from one or more non-scalable nested picture timing supplemental enhancement information messages, this indicates that the one or more non-scalable nested picture timing supplemental enhancement information messages are substituted for one or more scalable nested picture timing supplemental enhancement information messages.

[0152] In an embodiment, the indication is a flag. One of the one or more scalable nested supplemental enhancement information messages may, for example, include a flag indicating whether sub-bitstream timing information may, for example, be configured to be obtained from one or more non-scalable nested picture timing supplemental enhancement information messages.

[0153] In an embodiment, the flag is use_orig_pic_timing_flag.

[0154] In an embodiment, the video data stream may, for example, include a video parameter set, the indication being a flag. The video parameter set may, for example, include a flag indicating whether timing information of the sub-bitstream may, for example, be configured to be obtained from one or more non-scalable nested picture timing supplemental enhancement information messages.

[0155] In an embodiment, the flag is use_orig_pic_timing_flag.

[0156] In an embodiment, the flag is general_same_pic_timing_in_all_ols_flag.

[0157] In an embodiment, the indication indicates that each of the one or more non-scalable nested picture timing supplemental enhancement information messages in each access unit of the one or more access units applies to an access unit of any output layer configured in the video data stream, that there are no scalable nested picture timing supplemental enhancement information messages or that the non-scalable nested picture timing supplemental enhancement information messages in each access unit of the one or more access units may or may not apply to an access unit of any output layer configured in the video data stream, and that a scalable nested picture timing supplemental enhancement information message may be present.

[0158] In an embodiment, if the indication indicates that the timing information of the sub-bitstream may be configured to be obtained from one or more non-scalable nested supplemental picture timing enhancement information messages, at least one of the one or more scalable nested supplemental enhancement information messages occurs before the one or more non-scalable nested supplemental picture timing enhancement information messages in an access unit of the bitstream.

[0159] In an embodiment, the indication is a constraint flag. The one or more non-scalable nested picture timing supplemental enhancement information messages include a constraint flag that indicates whether the one or more non-scalable nested picture timing supplemental enhancement information messages apply to at least one of the one or more sub-bitstreams.

[0160] In an embodiment, the indication is a first indication, a second indication in the video data stream indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a third indication in the video data stream indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0161] According to an embodiment, the indication in the video data stream further indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream and / or the indication in the video data stream further indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0162] Further, according to an embodiment, there is provided a video data stream having video encoded therein, where a first indication in the video data stream indicates whether timing information of the sub-bitstreams is obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream and / or a second indication in the video data stream indicates whether timing information of the sub-bitstreams is obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0163] Furthermore, according to an embodiment, the video data stream includes one or more non-scalable nested supplemental enhancement information messages including timing information. If the video data stream includes a scalable nested supplemental enhancement information message including timing information, this indicates that, depending on the scalable nested supplemental enhancement information message, all of the one or more non-scalable nested timing information extension information messages are substituted by a scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information). Or, a subset including at least one of the one or more non-scalable nested timing information extension information messages is substituted by a scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information).

[0164] According to an embodiment, the one or more non-scalable nested supplemental enhancement information messages comprising timing information are one or more non-scalable nested picture timing supplemental enhancement information messages and the scalable nested supplemental enhancement information messages comprising timing information are scalable nested picture timing supplemental enhancement information messages, or the one or more non-scalable nested supplemental enhancement information messages comprising timing information are one or more non-scalable nested buffering period supplemental enhancement information messages and the scalable nested supplemental enhancement information messages comprising timing information are scalable nested buffering period supplemental enhancement information messages, or the one or more non-scalable nested supplemental enhancement information messages comprising timing information are one or more non-scalable nested decoding unit supplemental enhancement information messages and the scalable nested supplemental enhancement information messages comprising timing information are scalable nested decoding unit supplemental enhancement information messages.

[0165] In an embodiment, where the video data stream may for example include a scalable nested supplemental enhancement information message including the timing information, the scalable nested supplemental enhancement information message including the timing information occurs before one or more non-scalable nested supplemental enhancement information messages including timing information within an access unit of the bitstream.

[0166] According to an embodiment, the timing information is timing information of a hypothetical reference decoder.

[0167] In an embodiment, the sub-bitstream may be output layer set dependent and / or sub-layer dependent and / or sub-picture dependent and / or decoding unit subset dependent.

[0168] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that an indication in the video data stream indicates whether timing information of a sub-bitstream is obtained from one or more non-scalable nested picture timing supplemental enhancement information messages of the video data stream.

[0169] According to an embodiment, if the indication indicates that sub-bitstream timing information is not obtained from one or more non-scalable nested picture timing supplemental enhancement information messages, this indicates that the one or more non-scalable nested picture timing supplemental enhancement information messages are substituted for one or more scalable nested picture timing supplemental enhancement information messages.

[0170] In an embodiment, the indication is a flag. The video encoder may, for example, be configured to generate the video data stream such that one of the one or more scalable nested supplemental enhancement information messages may, for example, include a flag indicating whether sub-bitstream timing information may, for example, be configured to be obtained from one or more non-scalable nested picture timing supplemental enhancement information messages.

[0171] In an embodiment, the flag is use_orig_pic_timing_flag.

[0172] In an embodiment, the video data stream may, for example, include a video parameter set. The indication may, for example, be a flag. The video encoder may, for example, be configured to generate the video data stream such that the video parameter set includes a flag indicating that timing information of the sub-bitstream may, for example, be configured to be obtained from one or more non-scalable nested picture timing supplemental enhancement information messages.

[0173] In an embodiment, the flag is use_orig_pic_timing_flag.

[0174] In an embodiment, the flag is general_same_pic_timing_in_all_ols_flag.

[0175] According to an embodiment, the indication indicates that each of the one or more non-scalable nested picture timing supplemental enhancement information messages in each access unit of the one or more access units applies to an access unit of any output layer configured in the video data stream, that there are no scalable nested picture timing supplemental enhancement information messages or that the non-scalable nested picture timing supplemental enhancement information messages in each access unit of the one or more access units may or may not apply to an access unit of any output layer configured in the video data stream, and that a scalable nested picture timing supplemental enhancement information message may be present.

[0176] In an embodiment, if the indication indicates that the timing information of the sub-bitstream may be configured to be obtained from one or more non-scalable nested supplemental picture timing enhancement information messages, at least one of the one or more scalable nested supplemental enhancement information messages occurs before the one or more non-scalable nested supplemental picture timing enhancement information messages in an access unit of the bitstream.

[0177] According to an embodiment, the indication is a constraint flag. The one or more non-scalable nested picture timing supplemental enhancement information messages may for example be configured such that a video encoder generates the video data stream such that the one or more non-scalable nested picture timing supplemental enhancement information messages include a constraint flag indicating whether the one or more non-scalable nested picture timing supplemental enhancement information messages apply to at least one of the one or more sub-bitstreams.

[0178] In an embodiment, the indication is a first indication, a second indication in the video data stream indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a third indication in the video data stream indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0179] According to an embodiment, the indication in the video data stream further indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream and / or the indication in the video data stream further indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0180] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, wherein a first indication in the video data stream indicates whether timing information of a sub-bitstream is obtained from the one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a second indication in the video data stream indicates whether timing information of a sub-bitstream is obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0181] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates a video data stream, such that the video data stream includes one or more non-scalable nested supplemental enhancement information messages including timing information. If the video data stream includes a scalable nested supplemental enhancement information message including timing information, this indicates that, depending on the scalable nested supplemental enhancement information message, all of the one or more non-scalable nested timing information enhancement information messages are substituted by a scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information). Or, a subset including at least one of the one or more non-scalable nested timing information supplemental enhancement information messages is substituted by the non-scalable nested supplemental enhancement information message including the timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information).

[0182] According to an embodiment, a video encoder may for example be configured to generate the video data stream such that the one or more non-scalable nested supplemental enhancement information messages including timing information are one or more non-scalable nested picture timing supplemental enhancement information messages and the scalable nested supplemental enhancement information messages including timing information are scalable nested picture timing supplemental enhancement information messages, or the video encoder may for example be configured to generate the video data stream such that the one or more non-scalable nested supplemental enhancement information messages including timing information are one or more non-scalable nested buffering period supplemental enhancement information messages and the scalable nested supplemental enhancement information messages including timing information are scalable nested buffering period supplemental enhancement information messages. Or, the video encoder may be configured, for example, to generate the video data stream such that the one or more non-scalable nested supplemental enhancement information messages containing timing information are one or more non-scalable nested decoding unit supplemental enhancement information messages, and the scalable nested supplemental enhancement information messages containing timing information are scalable nested decoding unit supplemental enhancement information messages.

[0183] In an embodiment, where the video data stream may, for example, include a scalable nested supplemental enhancement information message that includes timing information, the video encoder may, for example, be configured to generate the video data stream such that the scalable nested supplemental enhancement information message that includes timing information is generated before one or more non-scalable nested supplemental enhancement information messages that include timing information within an access unit of the bitstream.

[0184] According to an embodiment, the timing information is timing information of a hypothetical reference decoder.

[0185] In an embodiment, the sub-bitstream may be output layer set dependent and / or sub-layer dependent and / or sub-picture dependent and / or decoding unit subset dependent.

[0186] Further, according to an embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the apparatus being for processing the input bitstream to obtain a sub-bitstream, an indication in the video data stream indicating whether timing information of the sub-bitstream is obtained from one or more non-scalable nested picture timing supplemental enhancement information messages of the video data stream.

[0187] According to an embodiment, if the indication indicates that sub-bitstream timing information is not obtained from one or more non-scalable nested picture timing supplemental enhancement information messages, the device indicates that the one or more non-scalable nested picture timing supplemental enhancement information messages may, for example, be configured to be substituted for one or more scalable nested picture timing supplemental enhancement information messages.

[0188] According to an embodiment, the indication may be, for example, a constraint flag. One of the one or more scalable nested supplemental enhancement information messages may, for example, include a flag indicating that timing information of the sub-bitstream may, for example, be configured to be obtained from one or more non-scalable nested picture timing supplemental enhancement information messages.

[0189] In an embodiment, the flag is use_orig_pic_timing_flag.

[0190] In an embodiment, the video data stream may, for example, include a video parameter set. The indication may, for example, be a flag. The video parameter set may, for example, include a flag indicating whether timing information of the sub-bitstream may, for example, be configured to be obtained from one or more non-scalable nested picture timing supplemental enhancement information messages.

[0191] In an embodiment, the flag is same_pic_timing_within_ols_flag.

[0192] According to an embodiment, the flag is general_same_pic_timing_in_all_ols_flag.

[0193] In an embodiment, the indication indicates that each of the one or more non-scalable nested picture timing supplemental enhancement information messages in each access unit of the one or more access units applies to an access unit of any output layer configured in the video data stream, that there are no scalable nested picture timing supplemental enhancement information messages or that the non-scalable nested picture timing supplemental enhancement information messages in each access unit of the one or more access units may or may not apply to an access unit of any output layer configured in the video data stream, and that a scalable nested picture timing supplemental enhancement information message may be present.

[0194] According to an embodiment, if the indication indicates that non-scalable nested picture timing supplemental enhancement information messages in each access unit of one or more access units may or may not be applied to access units of any output layer configured in the video data stream, and that scalable nested picture timing supplemental enhancement information messages may be present, the device is configured to remove from the input bitstream or sub-bitstream all supplemental enhancement information network abstraction layer units with picture timing content, including non-scalable nested supplemental enhancement information messages.

[0195] In an embodiment, if the indication indicates that the timing information of the sub-bitstream may, for example, be configured to be obtained from one or more non-scalable nested picture timing supplemental enhancement information messages, the device may, for example, be configured to process at least one of the one or more scalable nested supplemental enhancement information messages, which occurs before the device processes one or more non-scalable nested picture timing supplemental enhancement information messages in an access unit of the bitstream, before the device may, for example, be configured to process one or more non-scalable nested picture timing supplemental enhancement information messages in the access unit.

[0196] In an embodiment, the indication is a constraint flag. The apparatus may, for example, be configured to process one or more non-scalable nested picture timing supplemental enhancement information messages that include a constraint flag indicating whether the one or more non-scalable nested picture timing supplemental enhancement information messages apply to at least one of the one or more sub-bitstreams.

[0197] In an embodiment, the indication is a first indication, a second indication in the video data stream indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a third indication in the video data stream indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0198] According to an embodiment, the indication in the video data stream further indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream and / or the indication in the video data stream further indicates whether the timing information of the sub-bitstream is further obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0199] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the apparatus being adapted to process the input bitstream to obtain a sub-bitstream, a first indication in the video data stream indicating whether timing information of the sub-bitstream is obtained from one or more non-scalable nested buffering period supplemental enhancement information messages of the video data stream, and / or a second indication in the video data stream indicating whether timing information of the sub-bitstream is obtained from one or more non-scalable nested decoding unit supplemental enhancement information messages of the video data stream.

[0200] Further, according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having a video encoded therein. The apparatus is for processing the input bitstream to obtain sub-bitstreams. The video data stream includes one or more non-scalable nested supplemental enhancement information messages including timing information. If the video data stream includes a scalable nested supplemental enhancement information message including timing information, the apparatus is responsive to the scalable nested supplemental enhancement information message to: extract all of the one or more non-scalable nested timing information enhancement information messages from the scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information); or extract a subset including at least one of the one or more non-scalable nested timing information supplemental enhancement information messages from the scalable nested supplemental enhancement information message including timing information (e.g., the timing information is picture timing information or buffering period information or decoding unit information).

[0201] According to an embodiment, the one or more non-scalable nested supplemental enhancement information messages comprising timing information are one or more non-scalable nested picture timing supplemental enhancement information messages and the scalable nested supplemental enhancement information messages comprising timing information are scalable nested picture timing supplemental enhancement information messages, or the one or more non-scalable nested supplemental enhancement information messages comprising timing information are one or more non-scalable nested buffering period supplemental enhancement information messages and the scalable nested supplemental enhancement information messages comprising timing information are scalable nested buffering period supplemental enhancement information messages, or the one or more non-scalable nested supplemental enhancement information messages comprising timing information are one or more non-scalable nested decoding unit supplemental enhancement information messages and the scalable nested supplemental enhancement information messages comprising timing information are scalable nested decoding unit supplemental enhancement information messages.

[0202] In an embodiment, a video data stream may for example include a scalable nested supplemental enhancement information message including timing information, the scalable nested supplemental enhancement information message including timing information occurring before one or more non-scalable nested supplemental enhancement information messages including timing information within an access unit of the bitstream, and an apparatus may for example be configured to process a scalable nested supplemental enhancement information message including timing information before one or more non-scalable nested supplemental enhancement information messages including timing information.

[0203] According to an embodiment, the timing information is timing information of a hypothetical reference decoder.

[0204] In an embodiment, the sub-bitstream may be output layer set dependent and / or sub-layer dependent and / or sub-picture dependent and / or decoding unit subset dependent.

[0205] According to an embodiment, the device may be configured to, for example, decode the sub-bitstream to decode the video.

[0206] Further, according to an embodiment, a system for encoding video into a video data stream and for decoding the video is provided. The system includes a video encoder as described above, and an apparatus as described above. The video encoder may for example be configured to encode the video into the video data stream, such that the video data stream has the video encoded therein. The apparatus may for example be configured to receive the video data stream as an input bitstream. Further, the apparatus may for example be configured to process the input bitstream to obtain a sub-bitstream. Further, the apparatus may for example be configured to decode the sub-bitstream to decode the video.

[0207] The VVC draft specification includes SEI messages that control the HRD timing operations, namely the buffering period SEI message and the picture timing SEI message when the bitstream is decoded.

[0208] Currently, the VVC draft specification already contains all Picture Timing SEI messages that apply to some target Temporal IDs, i.e., temporal scalability does not necessarily require scalable nesting of additional timing information. Therefore, when extracting sub-bitstreams (e.g., in a layered scenario via OLS or spatially by extraction of sub-pictures), in some cases, it may not be necessary to modify / replace Picture Timing SEI messages in such a way.

[0209] In the following, the output layer set abstraction is described.

[0210] In one embodiment, signaling is added to indicate that the Picture Timing SEI message in the bitstream applies to any sub-bitstream (defined / corresponding to some Output Layer Set (OLS) A) of the bitstream (defined / corresponding to some OLS B), and only the Buffering periodic SEI messages are replaced with scalably nested pairs. Example syntax: [Table 1]

[0211] Currently, scalable nested SEI messages follow the HRD SEI message in an access unit. Therefore, as part of the above embodiment, if use_orig_pic_timing_flag is equal to 1, scalable nested SEI messages, including OLS-specific HRD SEI messages, must occur before their respective PT SEI messages (messages to keep during extraction) in an access unit in bitstream order.

[0212] Or, in an alternative embodiment, the indication is in the VPS as a constraint flag: [Table 2]

[0213] Or, in an alternative embodiment, the indication is in the Picture Timing SEI as a constraint flag.

[0214] The extraction process has been modified.

[0215] In the following, the DPS rewrite during the sub-bitstream extraction process is described.

[0216] The inputs to this process are the bitstream inBitstream, the target OLS index targetOlsIdx, and the target highest TemporalId value tIdTarget.

[0217] The output of this process is the sub-bitstream outBitstream.

[0218] For any output sub-bitstream that is the output of a process specified in this clause with targetOlsIdx equal to an index in the list of OLSs specified in the VPS as the bitstream, and tIdTarget equal to any value in the range 0 to 6 inclusive as the input, and which satisfies the following conditions, it shall be a conforming bitstream: · The output sub-bitstream contains at least one VCL NAL unit with nuh_layer_id equal to each of LayerIdInOls[targetOlsIdx]. · The output sub-bitstream contains at least one VCL NAL unit whose TemporalId is equal to tIdTarget. NOTE - A compliant bitstream contains one or more coded slice NAL units with TemporalId equal to 0, but is not required to contain a coded slice NAL unit with nuh_layer_id equal to 0. The output sub-bitstream OutBitstream is derived as follows. The bitstream outBitstream is set to be the same as the bitstream inBitstream. Remove all NAL units with TemporalId greater than tIdTarget from outBitstream. · Remove all NAL units from outBitstream whose nal_unit_type is not equal to any of VPS_NUT, DPS_NUT, and EOB_NUT and whose nuh_layer_id is not included in the list LayerIdInOls[targetOlsIdx]. Remove from outBitstream all SEI NAL units, including any scalable nesting SEI message, where nesting_ols_flag is 1 and there is no value of i in the range from 0 to nesting_num_olss_minus1, inclusive, where NestingOlsIdx[i] is the same as targetOlsIdx. · If targetOlsIdx is greater than 0, remove all SEI NAL units from outBitstream that contain non-scalable nested SEI messages with payloadType equal to 0 (buffering periodicity), 1 (picture timing), or 130 (decoding unit information). ·If targetOlsIdx is greater than 0 and use_orig_pic_timing_flag / same_pic_timing_within_ols_flag is equal to 0, remove all SEI NAL units, including non-scalable nested SEI messages with payloadType equal to 1 (picture timing), from outBitstream. If outBitstream contains a scalable nesting SEI message and outBitstream contains applicable SEI NAL units (NestingOlsIdx[i] is equal to targetOlsIdx), do the following: From the scalable nested SEI message, extract the appropriate non-scalable nested SEI messages with payloadType equal to 0 (buffering period), 1 (picture timing), or 130 (decoding unit information) and put those messages into outBitstream. Otherwise (when use_orig_pic_timing_flag / same_pic_timing_within_ols_flag is equal to 1), extract from the scalable nested SEI message if there is one applicable for outBitstream (NestingOlsIdx[i] is equal to targetOlsIdx) the appropriate non-scalable nested SEI message with payloadType equal to 0 (buffering period) or 130 (decoding unit information) and put those messages into outBitstream. Remove all SEI NAL units, including scalable nested SEI messages, from outBitstream.

[0219] The indication, e.g., flag, may use, e.g., general_same_pic_timing_in_all_ols_flag, or, e.g., same_pic_timing_within_ols_flag.

[0220] When general_same_pic_timing_in_all_ols_flag (or same_pic_timing_within_ols_flag) is equal to a first value, e.g., 1, it specifies that the non-scalable nested PT SEI messages of each AU apply to the AUs of all OLSs in the bitstream and there are no scalable nested PT SEI messages. When general_same_pic_timing_in_all_ols_flag is equal to a second value, e.g., 0, the non-scalable nested PT SEI messages of each AU may or may not apply to any OLS's AUs in the bitstream and there may be scalable nested PT SEI messages.

[0221] For example, if general_same_pic_timing_in_all_ols_flag is equal to the second value, e.g., 0, the device shall remove from the input bitstream or output bitstream (e.g., outBitstream / sub-bitstream) all SEI NAL units that constitute non-scalable nested SEI messages with payloadType equal to 1 (PT).

[0222] In an embodiment, the buffering period SEI message may also not require replacement by scalable nested transformation in some cases, so that additional indications can be carried in the parameter set or the buffering period SEI message itself, and the indicated timing is also applied to the extracted sub-bitstream, and the extraction process is further modified to preserve the original BP and PT SEI messages according to the respective indications.

[0223] In the following, the extraction of sub-pictures is explained.

[0224] In another embodiment that considers the case of sub-picture sub-bitstream extraction, the respective signals are added to the bitstream (e.g., in the syntax of a sub-picture nested SEI message, a Picture Timing SEI message, or a parameter set), and the Picture Timing SEI message in the bitstream applies to any sub-bitstream defined by a sub-picture or sub-picture set of the bitstream consisting of a combination of all sub-pictures contained in the bitstream, and only the buffering period SEI message is replaced by its sub-picture nested counterpart.

[0225] Alternatively, the buffering period SEI message may also in some cases not require replacement by sub-picture nested transformations, so that additional instructions can be carried in the parameter set or buffering period SEI message itself, with the indicated timing also being applied to the extracted sub-bitstream, and the extraction process being further modified to preserve the original BP and PT SEI messages according to the respective instructions.

[0226] The following describes removal of non-scalable nested HRD SEI messages based on the presence of scalable nested HRD SEI messages.

[0227] In another embodiment of the invention, the scope of non-scalable nested HRD SEI messages (e.g., BP, PT, DUI SEI messages), i.e., the sub-bitstream from which some of them (e.g., only PT SEI messages) or all of them can be extracted (e.g., OLS), is indicated by the presence of a respective alternative SEI message in the form of a scalable nested HRD SEI message, and in the absence of such a message, the respective non-scalable HRD SEI message applies to the OLS. A consequence of this scope indication is that the removal of non-scalable nested SEI messages in the sub-bitstream extraction process depends on the presence of a scalable nested SEI message that can act as a replacement for the removed message, and in the absence of such a scalable nested SEI message, the non-scalable nested SEI message remains in the extracted sub-bitstream.

[0228] In a further embodiment, each scalable nested HRD SEI message is placed before a non-scalable nested HRD SEI message in bitstream order in the access unit to facilitate sequential processing of the bitstream during extraction.

[0229] Below we describe a simplified, scalable nesting.

[0230] According to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, where for each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set are equal for all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0231] Further, according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, and for each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video data stream including an indication of whether the buffer period information and / or picture timing information of the output layer set are equal in all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0232] Further, according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, where for each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer periodicity information and / or picture timing information of an output layer set, the buffer periodicity information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0233] Further according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, where for each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0234] In an embodiment, all scalable nested supplemental enhancement information messages within an access unit whose identifier has the same value and identifies an output layer set may, for example, convey the same buffer period information and / or the same picture timing information.

[0235] According to an embodiment, for a particular access unit of a plurality of access units, any picture timing supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same picture timing information, and / or for a particular access unit of a plurality of access units, any buffer periodicity supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same buffer periodicity information, and / or for a particular access unit of a plurality of access units, any decoding unit supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same decoding unit information.

[0236] In an embodiment, two scalable nested supplemental enhancement information messages of a particular payload type within an access unit, where the identifier has the same value and identifies an output tier set, may for example carry the same content.

[0237] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream, such that the video data stream includes a plurality of access units, for each access unit of the plurality of access units. If an access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video encoder generates the video data stream such that the buffer period information and / or picture timing information of the output layer set are equal in all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0238] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates a video data stream, such that the video data stream includes a plurality of access units. If, for each access unit of the plurality of access units, the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video encoder generates the video data stream, such that the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set are equal for all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0239] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream, such that the video data stream includes a plurality of access units. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video encoder generates the video data stream such that the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0240] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates a video data stream, such that the video data stream includes a plurality of access units. If, for each access unit of the plurality of access units, the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of the output layer set, the video encoder generates the video data stream, such that the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0241] In an embodiment, all scalable nested supplemental enhancement information messages within an access unit whose identifier has the same value and identifies an output layer set may, for example, convey the same buffer period information and / or the same picture timing information.

[0242] According to an embodiment, for a particular access unit of a plurality of access units, any picture timing supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same picture timing information, and / or for a particular access unit of a plurality of access units, any buffer periodicity supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same buffer periodicity information, and / or for a particular access unit of a plurality of access units, any decoding unit supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same decoding unit information.

[0243] In an embodiment, two scalable nested supplemental enhancement information messages of a particular payload type within an access unit, where the identifier has the same value and identifies an output tier set, may for example carry the same content.

[0244] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The apparatus processes the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set are equal for all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0245] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The apparatus processes the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units. If the access unit includes two or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set are equal in all of the two or more scalable nested supplemental enhancement information messages of the access unit.

[0246] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units, and the apparatus processes the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of an output layer set, the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0247] Further according to an embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The input bitstream includes a plurality of access units. The apparatus processes the access units of the input bitstream to obtain sub-bitstreams. For each access unit of the plurality of access units, if the access unit includes three or more scalable nested supplemental enhancement information messages including buffer period information and / or picture timing information of the output layer set, the video data stream includes an indication of whether the buffer period information and / or picture timing information of the output layer set appears only in one of the three or more scalable nested supplemental enhancement information messages and in another one of the three or more scalable nested supplemental enhancement information messages that immediately follows said one of the three or more scalable nested supplemental enhancement information messages.

[0248] In an embodiment, all scalable nested supplemental enhancement information messages within an access unit whose identifier has the same value and identifies an output layer set may, for example, convey the same buffer period information and / or the same picture timing information.

[0249] According to an embodiment, for a particular access unit of a plurality of access units, any picture timing supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same picture timing information, and / or for a particular access unit of a plurality of access units, any buffer periodicity supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same buffer periodicity information, and / or for a particular access unit of a plurality of access units, any decoding unit supplemental enhancement information messages applied to a set of layers and sub-layers of the output layer set may, for example, carry the same decoding unit information.

[0250] In an embodiment, two scalable nested supplemental enhancement information messages of a particular payload type within an access unit, where the identifier has the same value and identifies an output tier set, may for example carry the same content.

[0251] According to an embodiment, if a device finds a buffer period scalable nested supplemental enhancement information message of an output layer set, the device may for example be configured to use the content of the buffer period scalable nested supplemental enhancement information of the output layer set without searching for further buffer period scalable nested supplemental enhancement information messages of the output layer set, and / or if a device finds a picture timing scalable nested supplemental enhancement information message of an output layer set, the device may for example be configured to use the content of the picture timing scalable nested supplemental enhancement information of the output layer set without searching for further picture timing scalable nested supplemental enhancement information messages of the output layer set.

[0252] According to an embodiment, the device may be configured to, for example, decode the sub-bitstream to decode the video.

[0253] Further, according to an embodiment, a system for encoding video into a video data stream is provided. The system includes a video encoder as described above, and an apparatus as described above. The video encoder may for example be configured to encode video into a video data stream, such that the video data stream has the video encoded therein. The apparatus may for example be configured to receive the video data stream as an input bitstream. Further, the apparatus may for example be configured to process the input bitstream to obtain a sub-bitstream. Further, the apparatus may for example be configured to decode the sub-bitstream to decode the video.

[0254] If scalable nested SEI messages can be used to replace the BP and PT SEI messages for all OLS in a bitstream, the current state of the art allows an encoder to distribute the BP and PT across multiple scalable nesting SEI messages. An extractor processing such a bitstream needs to scan all scalable nested SEI messages in an access unit (which may be large given the amount of OLS and repetitions) until it finds the appropriate scalable nested BP and P SEI messages.

[0255] If the encoder places the first scalably nested BP or PTSEI message into an access unit of the bitstream and, in the course of encoding the images of that access unit, finds that an update to the BP or PTSEI message parameters is required, writing another BP or PTSEI message into that access unit of the bitstream will in turn burden the extractor with the need to ensure that it uses the latest or most recent BP or PT SEI message placed in the access unit in the bitstream when abstracting that access unit.

[0256] The present invention simplifies the operation of the extractor by imposing constraints that remove the nonsensical options mentioned above.

[0257] In an embodiment, it may be a requirement for bitstream compatibility, for example, that all scalable SEI messages (i.e., identifiers / indexes identifying OLSs) within an access unit with the same nesting value carry the same content (e.g., the same buffering period information and / or the same picture / timing information). This ensures that the BP and PT SEI messages of an OLS can be found in one scalable nesting SEI message, and that an extractor can obtain all the required information once it finds the scalable nesting SEI message in the target OLS.

[0258] In an embodiment, it may be a bitstream compatibility requirement that, for example, all scalable nested SEI messages within an access unit with the same value of an identifier, e.g., an index, identifying an OLS, carry the same content (e.g., the same buffering period information and / or the same picture / timing information). For example, for a particular access unit, any PT SEI message (e.g., for a particular OLS) that applies to a set of layers and sublayers must have the same payload (e.g., for two PT SEI messages contained in two separate scalable nested SEI messages). The same applies to BPSEI and DUI SEI messages. This ensures that the BP and PT SEI messages of an OLS can be found in one scalable nesting SEI message, and that the extractor can obtain all the required information once it finds the scalable nesting SEI message in the target OLS.

[0259] In an embodiment, it may be a requirement of bitstream compatibility, for example, that all scalable nested SEI messages within an access unit that have the same value of an identifier, e.g., an index, an OLS identification, carry the same content (e.g., the same buffering period information and / or the same picture / timing information). For example, for a particular access unit in a bitstream that carries two scalably nested PT SEI messages that apply to the same OLS, the payloads of these two scalably nested PT SEI messages are equal, and the extractor can be sure that when it finds a scalable nested SEI message in the target OLS, it has all the information it needs.

[0260] In an embodiment, it may be a requirement of bitstream compatibility, for example, that all scalable nested SEI messages within an access unit that have the same value of an identifier, e.g., an index, identifying an OLS, carry the same content (e.g., the same buffering period information and / or the same picture / timing information). For example, the BP and PT SEI messages of an OLS can be found in one scalable nesting SEI message, and an extractor can be sure to obtain all the required information once it finds the scalable nesting SEI message in the target OLS.

[0261] According to an embodiment, it may be a bitstream conformance requirement, for example, that BP and PT SEI messages applicable to OLS are not applicable to OLS without any other scalable nesting SEI message NAL units falling within the two scalable nesting SEI messages.

[0262] In the following, we explain low latency and temporal scalability of DU timing.

[0263] According to an embodiment, there is provided a video data stream having video encoded therein, the video data stream comprising a plurality of access units, the video data stream comprising a spreading factor that depends on a number of sub-bitstreams of the video data stream, or the video data stream comprising a clock sub-tick value that depends on the highest sub-bitstream of the sub-bitstreams of the video data stream.

[0264] According to an embodiment, each of the sub-bitstreams is output layer set dependent and / or sub-layer dependent and / or sub-picture dependent.

[0265] In an embodiment, the decoding unit removal time depends on the removal time of the access unit and the spreading factor.

[0266] According to an embodiment, the video data stream includes a temporal distance, the temporal distance being multiplied by a derived clock sub-tick value, said derived clock sub-tick value being derivable using said spreading factor.

[0267] In an embodiment, each of the plurality of spreading factors is assigned to a sub-layer of a plurality of sub-layers. A video data stream may, for example, include a plurality of access units.

[0268] The clock subtick value depends on the clock ticks, which in turn depends on the spread.

[0269] The clock subtick value is defined as follows: ClockSubTick= = ClockTick÷( tick_divisor_minus2+2 ) * ( tick_divisor_factor_minus1[HTid]+ 1 ) where ClockSubTick is the clock subtick value, Clock Tick is the clock tick, tick_divisor_minus2 is the additional tick divisor, and tick_divisor_factor_minus1[HTidU+00A0] is the time division factor of the video data stream.

[0270] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream, such that the video data stream comprises a plurality of access units. Further, the video encoder generates the video data stream, such that the video data stream comprises a spreading factor that is dependent on a number of sub-bitstreams of the video data stream, or the video encoder generates the video data stream, such that the video data stream comprises a clock sub-tick value that is dependent on the highest sub-bitstream of the sub-bitstreams of the video data stream.

[0271] According to an embodiment, each of the sub-bitstreams is output layer set dependent and / or sub-layer dependent and / or sub-picture dependent.

[0272] In embodiments, a video encoder may for example be configured to generate said video data stream such that a removal time of a decoding unit depends on the removal time of an access unit and said spreading factor.

[0273] According to an embodiment, the video data stream may for example include a temporal distance, which is multiplied by a derived clock sub-tick value, said derived clock sub-tick value being derivable using said spreading factor.

[0274] In an embodiment, each of the multiple spreading factors is assigned to a sub-layer of the multiple sub-layers. A video encoder may be configured to generate a video data stream such that the video data stream may include, for example, multiple spreading factors.

[0275] The clock subtick value depends on the clock ticks, which in turn depends on the spread.

[0276] The clock subtick value is defined as follows: ClockSubTick= = ClockTick÷( tick_divisor_minus2+2 ) * ( tick_divisor_factor_minus1[HTid]+ 1 ) where ClockSubTick is the clock subtick value, Clock Tick is the clock tick, tick_divisor_minus2 is the additional tick divisor, and tick_divisor_factor_minus1[HTidU+00A0] is the video data stream.

[0277] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream, such that the video data stream includes sub-layer specific frame rate information for the sub-layer, and / or the video encoder generating the video data stream, such that the video data stream includes sub-layer specific frame display time information for the sub-layer.

[0278] Further according to an embodiment, there is provided a video decoder for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video data stream comprising a plurality of access units, the video decoder being adapted to decode the video data stream to decode said video, the video data stream comprising a spreading factor dependent on a number of sub-bitstreams of the video data stream, the video decoder being adapted to decode the video using the spreading factor, or the video data stream comprising a clock sub-tick value dependent on a highest sub-bitstream of the sub-bitstreams of the video data stream, the video decoder being adapted to decode the video using the clock sub-tick value.

[0279] According to an embodiment, each of the sub-bitstreams is output layer set dependent and / or sub-layer dependent and / or sub-picture dependent.

[0280] 144. A video decoder according to claim 142 or 143, in an embodiment wherein the decoding unit removal time depends on the access unit removal time and the spreading factor.

[0281] In an embodiment, the video data stream may, for example, include a temporal distance. The video decoder may, for example, be configured to multiply the temporal distance by the derived clock sub-tick value. The video decoder may, for example, be configured to derive the derived clock sub-tick value using a spreading factor.

[0282] In an embodiment, the spreading factor is one of a plurality of spreading factors. Each of the plurality of spreading factors may, for example, be assigned to a sub-layer of a plurality of sub-layers. The video data stream may, for example, include a plurality of access units. The video decoder may, for example, be configured to decode the video using the plurality of spreading factors.

[0283] In an embodiment, the clock subtick value depends on the clock ticks, which in turn depends on the spreading factor.

[0284] In an embodiment, the clock subtick value is defined as follows: ClockSubTick= = ClockTick÷( tick_divisor_minus2+2 ) * ( tick_divisor_factor_minus1[HTid]+ 1 ) where ClockSubTick is the clock subtick value, Clock Tick is the clock tick, tick_divisor_minus2 is the additional tick divisor, and tick_divisor_factor_minus1[HTidU+00A0] is the time division factor of the video data stream.

[0285] Further, according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, a video decoder is for decoding the video data stream to decode said video, the video data stream including sub-layer specific frame rate information of a sub-layer and / or the video data stream including sub-layer specific frame display period information for a sub-layer, the decoder is for determining a spreading factor using the sub-layer specific frame rate information of the sub-layer and / or using the sub-layer specific frame display period information.

[0286] Further, according to an embodiment, there is provided a system for encoding video into a video data stream and for decoding video, the system including a video encoder as described above and an apparatus as described above. The video encoder is for encoding video into a video data stream such that the video data stream has the video encoded therein. The video decoder is for receiving the video data stream and for decoding the video data stream into video.

[0287] Removing a temporal sub-layer modifies the removal times of the remaining access units of the lower temporal sub-layer, especially in the presence of frame reordering. However, in low-latency configurations, especially when DU timing is provided, the removal times are modified in a very structured way, i.e., resulting in decode times that scale or spread over time. Figure 4 illustrates the problem when there are three DUs per access unit.

[0288] Note that the final decoding time of an AU when the entire bitstream is decoded will be different than for only that substream, so the ultra-low latency property will be lost. Typically, this is the result of different decoding capabilities, i.e. a decoder capable of decoding 60 fps can process a frame in 1 / 60th of a second, but a decoder capable of decoding 30 fps can only process a frame in 1 / 30th of a second.

[0289] Note that the removal time of a DU is indicated as a delta of the removal time of an AU. deltaTime is signaled in a picture timing SEI message or a Decoding Unit Information SEI message in number of clock subticks. This deltaTime indicates the removal time of a DU compared to the removal time of an AU. Instead of looping over multiple time sublayers to indicate the DU removal time (as deltaTimes compared to the AU removal time) from the CPB, a spreading factor can be derived. In one embodiment, the spreading factor is derived from sublayer specific frame rate information (e.g., sublayer T0 30fps, sublayer L1 60fps) or sublayer specific frame display period (FrameTimeInterval described in Section 6 below, e.g., sublayer L01 / 30s and sublayer L11 / 60s), e.g. according to the ratio of such information of two such sublayers. Alternatively, such a spreading factor can be indicated. This embodiment adds information of the DU removal time indicating the spreading factor and calculates the corresponding DU time as a delta to the AU removal time.

[0290] Or as shown in one of the following: [Table 3]

[0291] Or, in an alternative embodiment, do the following: [Table 4]

[0292] The derivation of the related variable ClockSubTick in the HRD spec is changed as follows: The variable ClockSubTick is derived as follows and is called Clock Subtick: lockSubTick=ClockTick÷( tick_divisor_minus2+2 ) * ( tick_divisor_factor_minus1[HTid]+ 1 ) (C-2)

[0293] DeltaTime then uses a larger or smaller ClockSubTick depending on the highest sub-layer present in the bitstream.

[0294] Alternatively, the syntax element tick_divisor_minus2 can be duplicated for each sub-layer to indicate the correct tick divisor when the highest temporal sub-layer (HTid) is set equal to the respective sub-layer. In that case, there is no spreading factor, but multiple ClockSubTicks are signaled, each sent for a different value of the highest sub-layer present in the bitstream, so that when a sub-layer is removed from the original bitstream, the ClockSubTick used will be different.

[0295] Below we explain the relevance of the use of common DU timing to OLS extraction.

[0296] If the bitstream contains extractable sub-streams specific to OLS and common DU timing (non-scalable nested PT SEI messages containing common DU timing), in terms of bitrate overhead and processing complexity, it is desirable to reuse these PT SEI messages in the form of comparison against the above indications (use_orig_pic_timing_flag and same_pic_timing_within_ols_flag) instead of providing scalable nested PT SEI messages as an alternative.

[0297] FIG. 5 is a diagram in which two layers are shown on top and removal times for AUs and DUs for both layers are shown below for the full bitstream (e.g., 0th OLS) and for a single layer extracted with layer ID L0 (e.g., 1st OLS).

[0298] In this case, the common timing needs to be scaled in the same way as above by tick_divisor_factor_minus1[] or the absolute representation of the adjusted tick divisor (see DU time spread in the figure). Therefore, in one embodiment, each OLS-specific HRD parameter syntax structure in the VPS carries either the relative factor or the absolute value of the tick divisor to scale the common DU removal timing.

[0299] However, in this usage scenario, the number of DUs per AU changes with the extraction since the AU contains both L0 and L1 images, which means that aspects described below in aspect 6 are also required to derive the correct number of DUs remaining after extraction.

[0300] In the following, the relevance of the use of common DU timing to sub-picture extraction is explained.

[0301] If the bitstream contains extractable sub-by-stream specific to subpictures (subpicture boundary processing for motion compensated prediction is enabled) and common DU timing (non-scalable nested PT SEI messages containing common timing), it is also desirable, from the standpoint of bitrate overhead and processing complexity, to reuse these SEI messages in the manner of comparison against the above indications (use_orig_pic_timing_flag and same_pic_timing_within_ols_flag) instead of providing scalable nested PT SEI messages as an alternative.

[0302] In this case, the common timing needs to be scaled in the same way as above by tick_divisor_factor_minus1[] or the absolute representation of the adjusted tick divisor to derive the correct number of DUs remaining after extraction (see DU time spread in the figure). Therefore, in one embodiment, the sub-picture specific HRD parameters syntax structures in the VPS / SPS carry either the relative factor or the absolute value of the tick divisor to scale the common DU removal timing.

[0303] Alternatively, an SEI message (eg, a sub-picture level information SEI message) may convey scaling information or absolute values ​​to allow proper derivation of the HRD parameters in the extracted sub-bitstream.

[0304] The derivation of the CPB / bitrate size of a sub-picture is described below.

[0305] According to an embodiment, a video decoder is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video decoder is adapted to decode the video data stream to decode the video. To decode the video, the video decoder estimates a coded picture buffer size for a sub-picture as a function of information in the video data stream indicating current coded picture buffer size information.

[0306] According to an embodiment, the video data stream may include a reference level signaled coded picture buffer size.

[0307] In an embodiment, if the current level is equal to a reference level, the video decoder may for example be configured to determine the coded picture buffer size using the coded picture buffer size of the reference level.

[0308] According to an embodiment, the video decoder may for example be configured to estimate the coded picture buffer size depending on the syntax element cpb_size_value_minus1[i][j] of the video data stream.

[0309] In an embodiment, a video decoder may for example be configured to estimate the coded picture buffer size according to the syntax element cpb_size_scale of the video data stream.

[0310] According to an embodiment, the video decoder may be configured to estimate a coded picture buffer size for a sub-picture, e.g., the coded picture buffer size for the video coding layer, e.g., the sub-picture can further estimate a coded picture buffer size with a picture buffer size coded at a network abstraction layer.

[0311] In an embodiment, a video decoder may, for example, be configured to estimate a video coding layer coded picture buffer size and / or a network abstraction layer coded picture buffer size depending on the value of the reference level fragment.

[0312] According to an embodiment, a video decoder may be configured, for example, to estimate a video coding layer coded picture buffer size according to: SubPicCbpSizeVcl[s] = = Floor(( cpb_size_value_minus1[i][j]+ 1 ) * 2 ( 4 + cpb_size_scale ) *RefLevelFraction[i][j]÷256) The apparatus may for example be configured to estimate the network abstraction layer coded picture buffer size according to: SubPicCbpSizeNal[s] = = Floor(( cpb_size_value_minus1[i][j]+ 1 ) * 2 ( 4 + cpb_size_scale )*RefLevelFraction[i][j]÷256), RefLevelFraction is the value of the fraction of the reference level.

[0313] In an embodiment, a video decoder may be configured, for example, to estimate a video coding layer coded picture buffer size according to: SubpicCpbSizeVcl[i][j][k] = Floor(CpbVclFactor*MaxCPB*OlsRefLevelFraction[i][j][k]÷256) SubpicCpbSizeNal[i][j][k]=Floor(CpbNalFactor*MaxCPB*OlsRefLevelFraction[i][j][k]÷256) where i, j, and k are indices and OlsRefLevelFraction[i][j][k] are real numbers.

[0314] Further according to an embodiment, a video decoder is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video decoder being configured to decode the video data stream to decode said video, for decoding the video, the video decoder being configured to estimate a sub-picture bitrate as a function of information in the video data stream indicating bitrate information of a currently coded video sequence.

[0315] According to an embodiment, the video data stream may for example include an indication indicating whether the sub-picture bit rate should be estimated using bit rate information of the currently coded video sequence. If the indication of the video data stream indicates that the sub-picture bit rate is to be estimated using bit rate information of the currently coded video sequence, the video decoder estimates the bit rate using the bit rate information of the currently coded video sequence. If the indication of the video data stream indicates that the sub-picture bit rate is to be estimated without using bit rate information of the currently coded video sequence, the video decoder estimates the bit rate using a predefined value or a worst case value without using bit rate information of the currently coded video sequence.

[0316] In an embodiment, the bitrate information of the currently coded video sequence is a signaled bitrate of a reference level. The video data stream may for example include the signaled bitrate of the reference level. If the current level is equal to the reference level, the video decoder may for example be configured to determine the bitrate of the sub-picture using said signaled bitrate of the reference level.

[0317] According to an embodiment, the video decoder may for example be configured to estimate the bit rate of a sub-picture depending on the syntax element bit_rate_value_minus1[i][j] of the video data stream.

[0318] In an embodiment, the video decoder may for example be configured to estimate the bit rate in response to a syntax element bit_rate_scale of the video data stream.

[0319] According to an embodiment, the video decoder may be configured to estimate a coded picture buffer size for a sub-picture, e.g., the coded picture buffer size for the video coding layer, e.g., the sub-picture can further estimate a coded picture buffer size with a picture buffer size coded at a network abstraction layer.

[0320] In an embodiment, a video decoder may, for example, be configured to estimate a video coding layer coded picture buffer size and / or a network abstraction layer coded picture buffer size depending on the value of the reference level fragment.

[0321] According to an embodiment, a video decoder may be configured, for example, to estimate a video coding layer bit rate for a sub-picture according to: SubPicBitRateVcl[s] = = Floor(( bit_rate_value_minus1[i][j]+1 ) * 2 (6 + bit_rate_scale) *RefLevelFraction[i][j]÷256)

[0322] The device may for example be configured to estimate the network abstraction layer bitrate for a sub-picture by: SubPicBitRateNal[s] = = Floor(( bit_rate_value_minus1[i][j]+1 ) * 2 (6 + bit_rate_scale) *RefLevelFraction[i][j]÷256)

[0323] RefLevelFraction is the value of the reference level fraction.

[0324] In an embodiment, a video decoder may be configured, for example, to estimate a video coding layer bit rate for a sub-picture according to: SubpicBitRateVcl[i][j][k]=Floor(CpbVclFactor*ValBR*OlsRefLevelFraction[0][j][k]÷256)

[0325] SubpicBitRateNal[i][j][k]=Floor(CpbNalFactor*ValBR*OlsRefLevelFraction[0][j][k]÷256) where i, j, and k are indices and OlsRefLevelFraction[i][j][k] are real numbers.

[0326] In an embodiment, i may, for example, indicate an index of a particular indicated reference level, j may, for example, indicate an index of a particular sub-picture of a picture of an access unit of a video data stream, and k may, for example, indicate the index of a maximum temporal sub-layer included in the video data stream and / or at which a video decoder operates.

[0327] According to an embodiment, OlsRefLevelFraction[i][j][k] may for example depend on the variable sli_non_subpic_layers_fraction[i][k], which indicates the i-th fraction of the bitstream level constraint associated with the layers of targetCvss for which sps_num_subpics_minus1 is equal to 0 when Htid is equal to k.

[0328] In an embodiment, If vps_max_layers_minus1 is equal to 0, or there are no layers in the bitstream with sps_num_subpics_minus1 equal to 0, For example, sli_non_subpic_layers_fraction[i][k] is 0,

[0329] If k is less than sli_max_sublayers_minus1 and sli_non_subpic_layers_fraction[i][k] does not exist, For example, sli_non_subpic_layers_fraction[i][k]= sli_non_subpic_layers_fraction[i][k+1].

[0330] According to an embodiment, for example, OlsRefLevelFraction[i][j][k] = =sli_non_subpic_layers_fraction[i][k]+ ( n - sli_non_subpic_layers_fraction[i][k]) ÷ n * (sli_ref_level_fraction_minus1[i][j][k]+ 1). In the formula, n represents a positive integer.

[0331] According to an embodiment, for example, n=256; or n=128; or n=512; or n=1024; or n=2048; or n=4096.

[0332] According to an embodiment, i, j, and k are defined according to sli_ref_level_fraction_minus1, where sli_ref_level_fraction_minus1[i][j][k] plus 1 specifies the i-th part of the level constraint associated with sli_ref_level_idc[i][k] for a subpicture whose subpicture index is equal to j for a layer of targetCvss whose Htid considered as a sublayer index is equal to k and for which sps_num_subpics_minus1 is greater than 0.

[0333] Further according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, and a video decoder for decoding the video data stream to decode said video, wherein to decode the video, the video decoder receives a coded picture buffer size of a sub-picture coded in the video data stream and uses the coded picture buffer size of the sub-picture to decode the video and / or the video decoder receives a bitrate of the sub-picture coded in the video data stream and decodes the video using the bitrate of the sub-picture.

[0334] According to an embodiment, the video data stream may for example include an indication indicating whether a sub-picture coded picture buffer size is encoded in the video data stream or whether the sub-picture coded picture buffer size should be estimated. If the video data stream indication indicates that the sub-picture coded picture buffer size should be estimated, the video decoder estimates the sub-picture coded picture buffer size. If the video data stream indication indicates that the sub-picture coded picture buffer size is encoded in the video data stream, the video decoder uses the sub-picture coded picture buffer size encoded in the video data stream.

[0335] In an embodiment, the video data stream may, for example, include an indication indicating whether the sub-picture bit rate is encoded in the video data stream or whether the sub-picture bit rate should be estimated. If the video data stream indication indicates that the sub-picture bit rate should be estimated, the video decoder estimates the sub-picture bit rate. If the video data stream indication indicates that the sub-picture bit rate is encoded in the video data stream, the video decoder uses the coded picture buffer size of the sub-picture that is encoded in the video data stream.

[0336] According to an embodiment, each of the multiple extractable sub-bitstreams is specific to an output layer set, and the sub-picture is assigned to at least one extractable sub-bitstream of the multiple extractable sub-bitstreams. If the video data stream may for example include common decoding unit removal timing information and multiple extractable sub-bitstreams, the parsing of each output layer set-specific hypothesis reference decoder parameter in a video parameter set or a sequence parameter set or a supplemental enhancement information message of the video data stream may for example include either the absolute value of a spreading factor or a tick divisor in scaling the removal timing of the common decoding unit. The video decoder may for example be configured to process said absolute value of the spreading factor or the tick divisor.

[0337] Further, according to an embodiment, a video data stream having video encoded therein is provided. The video data stream may for example include the syntax element cpb_size_value_minus1[i][j] and the syntax element cpb_size_scale. Or the video data stream may for example include the syntax element bit_rate_value_minus1[i][j] and the syntax element bit_rate_scale.

[0338] Further, according to an embodiment, a video data stream having video encoded therein is provided, the video data stream including an indication indicating whether a coded picture buffer size of a sub-picture should be estimated using current coded picture buffer size information and / or the video data stream generating an indication indicating whether a bitrate of a sub-picture is estimated using bitrate information of a currently coded video sequence.

[0339] According to an embodiment, the current coded picture buffer size information is a reference level signaled coded picture buffer size, the video data stream may for example include a reference level signaled coded picture buffer size, and / or the bitrate information of the currently coded video sequence is a reference level signaled bitrate, the video data stream may for example include a reference level signaled bitrate.

[0340] Further according to an embodiment there is provided a video data stream having video encoded therein, such that the video data stream includes an indication indicating whether a coded picture buffer size for a sub-picture has been encoded in the video data stream or whether the coded picture buffer size for a sub-picture should be estimated and / or the video data stream includes an indication indicating whether a bitrate for a sub-picture has been encoded in the video data stream or whether the bitrate for a sub-picture needs to be estimated.

[0341] According to an embodiment, each of the multiple extractable sub-bitstreams is specific to an output layer set, and a sub-picture is assigned to at least one extractable sub-bitstream of the multiple extractable sub-bitstreams. If the video data stream may for example include common decoding unit removal timing information and multiple extractable sub-bitstreams, the parsing of each output layer set-specific hypothesis reference decoder parameter in a video parameter set or sequence parameter set or supplemental enhancement information message of the video data stream may for example include either a spreading factor or an absolute value of a tick divisor in scaling the removal timing of the common decoding unit.

[0342] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has the video encoded therein. The video data stream may for example include the syntax element cpb_size_value_minus1[i][j] and the syntax element cpb_size_scale. Or the video encoder generates the video data stream, such that the video data stream includes the syntax element bit_rate_value_minus1[i][j] and the syntax element bit_rate_scale.

[0343] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream such that the video data stream includes an indication indicating whether a coded picture buffer size of a sub-picture should be estimated using current coded picture buffer size information and / or the video encoder generating the video data stream such that the video data stream includes generating an indication indicating whether a bitrate of the sub-picture is estimated using currently coded video sequence bitrate information.

[0344] According to an embodiment, the video encoder may, for example, be configured such that the current coded picture buffer size information is a reference level signaled coded picture buffer size and the video data stream may, for example, include the reference level signaled coded picture buffer size, and / or the video encoder may, for example, be configured such that the bitrate information of the currently coded video sequence is a reference level signaled bitrate and the video data stream may, for example, include the reference level signaled bitrate.

[0345] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream such that the video data stream includes an indication indicating whether a coded picture buffer size of a sub-picture has been encoded in the video data stream or whether a coded picture buffer size of a sub-picture should be estimated, and / or the video encoder generating the video data stream such that the video data stream includes an indication indicating whether a bitrate of a sub-picture has been encoded in the video data stream or whether a bitrate of a sub-picture needs to be estimated.

[0346] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that, when the video data stream includes common decoding unit removal timing information and multiple extractable sub-bitstreams, each of the multiple sub-bitstreams is specific to an output layer set, and parsing each output layer set specific hypothetical reference decoder parameter in a video parameter set or a sequence parameter set or a supplemental enhancement information message of the video data stream includes either a spreading factor or an absolute value of a tick divisor in scaling the removal timing of the common decoding unit.

[0347] Further, according to an embodiment, a system for encoding video into a video data stream and for decoding video is provided, the system including a video encoder as described above and a video decoder as described above. The video encoder may for example be configured to encode the video into the video data stream such that the video data stream has the video encoded therein. The video decoder is configured to receive the video data stream and decode the video data stream into video. The current VVC draft specification includes an SEI message that indicates additional information that helps in estimating the level information of subpictures and the level of subpicture sets. The way to achieve this is to signal the portion that a subpicture contributes to a particular signal reference level. Based on the percentage of the level of each subpicture, the variables CPBsize and bitrate and the level of the bitstream of the subsequent subpictures are estimated. The percentage of the cumulative level is also used to derive the CPB size and bitrate of the bitstream consisting of a set of subpictures and also to estimate their levels. However, all these derivations use the MaxCPBSize and / or MaxBitrate of the reference level, which is problematic because most bitstreams may not fully occupy the CPB and bitrate budget up to the MaxCPBSize and / or MaxBitrate of a particular level. Using these values ​​to estimate the level of the merged bitstream of subpicture bitstreams or subpicture set bitstreams will likely result in over-provisioning of the (accumulated) CPB size and bitrate.

[0348] The VVC draft specification includes the derivation of the variables SubPicCpbSizeVcl[i][j] and SubPicCpbSizeNal[i][j], as follows: SubPicCpbSizeVcl[i][j]=Floor(CpbVclFactor*MaxCPB*RefLevelFraction[i][j]÷256) SubPicCpbSizeNal[i][j]=Floor(CpbNalFactor*MaxCPB*RefLevelFraction[i][j]÷256)

[0349] This is because the original bitstream containing all sub-pictures may already have more accurate information about the exact CPB size and bitrate of the bitstream (cpb_size_value_minus1 and cpb_size_scale) and not just the information of the respective maximum values ​​derived by reference from the bitstream level. Therefore, it is an object of the present invention to be able to derive much more accurate values ​​for the respective CBP size and bitrate per sub-picture values ​​and use them for approximation at the sub-picture set level. In a further embodiment, the sub-picture CPB is derived as follows; SubPicCbpSizeVcl[s] = Floor(( cpb_size_value_minus1[i][j]+ 1 ) * 2 ( 4 + cpb_size_scale ) *RefLevelFraction[i][j]÷256) SubPicCbpSizeNal[s] = Floor(( cpb_size_value_minus1[i][j]+ 1 ) * 2 ( 4 + cpb_size_scale ) *RefLevelFraction[i][j]÷256)

[0350] Note that the syntax elements cpb_size_value_minus1[i][j] are sent separately for the Vcl and Nal HRD parameters and are used respectively in the above derivation, therefore the values ​​of SubPicCbpSizeVcl[s] and SubPicCbpSizeVcl[s] are derived from potentially different values.

[0351] Since CPB sizes are signaled for a specific level, the above derivation can only be performed if that level is included as a reference level.

[0352] For bitrates, the respective derivations are modified from using the maximum bitrate as a reference as follows (Be [Vcl / Nal]Factor * May BE), SubPicBitRateVcl[s]= Floor( BrVclFactor * MaxBR *RefLevelFraction[i][j]÷256) SubPicBitRateNal[s]= Floor( BrNalFactor * MaxBR *RefLevelFraction[i][j]÷256) The actual bit rate of the signaled bitstream is used ((bit_rate_value_minus1[i][j]+1) * 2 (6 + bit_rate_scale ), as listed below. SubPicBitRateVcl[s] = Floor(( bit_rate_value_minus1[i][j]+1 ) * 2 (6 + bit_rate_scale) *RefLevelFraction[i][j]÷256) SubPicBitRateNal[s] = Floor(( bit_rate_value_minus1[i][j]+1 ) * 2 (6 + bit_rate_scale) *RefLevelFraction[i][j]÷256) Note that the same applies here as above for CPB size, and that the values ​​of the syntax elements bit_rate_value_minus1[i][j] depend on whether the Nal or Vcl HRD is considered, and therefore the values ​​of SubBitRateVcl[s] and SubPicBitrateVcl[s] are derived to potentially different values.

[0353] In an embodiment, the variables SubpicCpbSizeVcl[i][j][k] and SubpicCpbSizeNal[i][j][k] are derived as follows: SubpicCpbSizeVcl[i][j][k]=Floor(CpbVclFactor*MaxCPB*OlsRefLevelFraction[i][j][k]÷256) SubpicCpbSizeNal[i][j][k]=Floor(CpbNalFactor*MaxCPB*OlsRefLevelFraction[i][j][k]÷256) In an embodiment in which MaxCPB is derived from sli_ref_level_idc[i][k], the variables SubpicCpbSizeVcl[i][j][k] and SubpicCpbSizeNal[i][j][k] are derived as follows: SubpicBitRateVcl[i][j][k]=Floor(CpbVclFactor*ValBR*OlsRefLevelFraction[0][j][k]÷256) SubpicBitRateNal[i][j][k]=Floor(CpbNalFactor*ValBR*OlsRefLevelFraction[0][j][k]÷256)

[0354] For example, the variables OlsRefLevelFraction[i][j][k] are numbers, e.g., real numbers.

[0355] For example, the variable OlsRefLevelFraction[i][j][k] may be derived according to, for example, sli_non_subpic_layers_fraction[i][k]+ ( n - sli_non_subpic_layers_fraction[i][k]) ÷ n * (sli_ref_level_fraction_minus1[i][j][k]+ 1) In the formula, n represents a positive integer, for example, n=256; or, for example, n=128; or, for example, n=512; or, for example, n=1024; or, for example, n=2048; or, for example, n=4096.

[0356] So, for example, OlsRefLevelFraction[i][j][k]= =sli_non_subpic_layers_fraction[i][k]+ ( 256 - sli_non_subpic_layers_fraction[i][k]) ÷ 256 * (sli_ref_level_fraction_minus1[i][j][k]+ 1).

[0357] sli_non_subpic_layers_fraction[i][k] may indicate the i-th portion of the bitstream level constraints associated with layers of targetCvss for which sps_num_subpics_minus1 is equal to 0, e.g., when Htid is equal to k. If vps_max_layers_minus1 is equal to 0 or no layer in the bitstream has sps_num_subpics_minus1 equal to 0, then sli_non_subpic_layers_fraction[i][k] is equal to 0. When k is less than sli_max_sublayers_minus1 and sli_non_subpic_layers_fraction[i][k] is not present, it is inferred to be equal to sli_non_subpic_layers_fraction[i][k+1], and sli_ref_level_fraction_minus1[i][j][k] plus 1 specifies the i-th part of the level constraint, for which sli_ref_level_idc[i][k] is involved, and for which, for subpictures, Htid is equal to k, the subpicture index of the layer in targetCvss equal to j and sps_num_subpics_minus1 is greater than 0. When k is less than sli_max_sublayers_minus1 and sli_ref_level_fraction_minus1[i][j][k] is not present, it is inferred to be equal to sli_ref_level_fraction_minus1[i][j][k+1].

[0358] Alternatively, in another embodiment, the CPB size and / or bit rate for each sub-picture can be directly signaled instead of being derived, or in addition, there can be a gating flag indicating whether the values ​​can be derived or whether they are explicitly signaled.

[0359] The following describes DU timing signaling of Picture Timing SEI.

[0360] According to an embodiment, a video data stream having video encoded therein is provided, the video data stream including a plurality of access units, the video data stream further including delta time information for each of two or more decoding units of the access units of the plurality of access units, a decoding unit removal time for each decoding unit of the two or more decoding units of the access units being dependent on the access unit removal time of the access unit and dependent on the delta time information for the decoding unit.

[0361] According to an embodiment, the video data stream may for example comprise picture timing supplemental extension information, which may for example comprise delta time information of two or more decoding units of said access unit.

[0362] In an embodiment, the delta time information indicates a difference in removal time between two of two or more decoding units of the access unit.

[0363] According to an embodiment, a last decoding unit of the two or more decoding units of said access unit has a removal time equal to the removal time of said access unit.

[0364] In an embodiment, the access unit may, for example, include three or more decoding units, and the difference in removal time is equal for each pair of two consecutive decoding units of the three or more decoding units of the access unit.

[0365] In an embodiment, the picture timing supplemental enhancement information is applied to a sub-bitstream derived from the video data stream, and the number of decoding units remains constant.

[0366] In an embodiment, the frame time interval is signaled in a parameter set of the video data stream, the HRD parameter of the sequence parameter set.

[0367] In an embodiment, the frame time interval can be derived as the difference between the removal times of two consecutive access units at the highest temporal level.

[0368] In an embodiment, each decoding unit of the two or more decoding units of the access unit may for example include one video coding layer network abstraction layer unit.

[0369] In an embodiment, the picture timing supplemental enhancement information is applied to sub-bitstreams derived from a video data stream, where there are different numbers of decoding units.

[0370] In an embodiment, the frame time interval can be derived as follows: Multiply (elemental_duration_in_tc_minus1[maxTiD]+1) by ClockTicks.

[0371] In an embodiment, the video data stream may for example include an indication of whether there is a variation for the video data stream in the number of decoding units.

[0372] In an embodiment, the video data stream does not include an indication of whether there is a variation for the video data stream in the number of decoding units.

[0373] In an embodiment, the number of decoding units in an access unit depends on the frame time interval and the common delay increment.

[0374] In an embodiment, the video data stream includes decoding unit information supplemental enhancement information messages of two or more decoding units of the access unit, the decoding unit information supplemental enhancement information messages of the decoding units may for example include delta time information of the decoding units.

[0375] In an embodiment, the video data stream may, for example, include a minimum picture duration flag in a video parameter set or a sequence parameter set of said video data stream, said minimum picture duration flag indicating whether frame time interval information is present in the absence of a constant frame rate.

[0376] Further, according to an embodiment, a video encoder is provided for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream, such that the video data stream includes a plurality of access units. Furthermore, the video encoder generates the video data stream, such that the video data stream includes delta time information for each of two or more decoding units of an access unit of the plurality of access units, and a decoding unit removal time of each decoding unit of the two or more decoding units of the access unit depends on the access unit removal time of the access unit and depends on the delta time information of the decoding unit.

[0377] According to an embodiment, a video encoder may be configured to generate a video data stream such that the video data stream may include, for example, picture timing supplemental extension information. The video encoder may be configured to generate a video data stream such that the picture timing supplemental extension information may include, for example, delta time information of two or more decoding units of said access unit.

[0378] In an embodiment, the video encoder may for example be configured to generate the video data stream such that the delta time information indicates a removal time difference between two of two or more decoding units of said access unit.

[0379] According to an embodiment, a last decoding unit of the two or more decoding units of said access unit has a removal time equal to the removal time of said access unit.

[0380] In an embodiment, the video encoder may for example be configured to generate the video data stream such that the access unit may for example include three or more decoding units, and the removal time difference is equal for each pair of two consecutive decoding units of the three or more decoding units of the access unit.

[0381] In an embodiment, the picture timing supplemental enhancement information is applied to a sub-bitstream derived from the video data stream, and the number of decoding units remains constant.

[0382] In an embodiment, a video encoder may for example be configured to generate the video data stream such that the frame time interval is signaled in a parameter set of the video data stream, an HRD parameter of a sequence parameter set.

[0383] According to an embodiment, a video encoder may for example be configured to generate a video data stream such that a frame time interval is derivable as the difference between removal times of two successive access units at the highest temporal level.

[0384] In an embodiment, the video encoder may, for example, be configured to generate a video data stream such that each decoding unit of two or more decoding units of the access unit may, for example, include one video coding layer network abstraction layer unit.

[0385] According to an embodiment, a video encoder may for example be configured to generate a video data stream such that picture timing supplemental enhancement information is applied to sub-bitstreams derived from the video data stream and there are different numbers of decoding units.

[0386] In an embodiment, the frame time interval can be derived as follows: Multiply (elemental_duration_in_tc_minus1[maxTiD]+1) by ClockTicks.

[0387] In an embodiment, the video encoder may, for example, be configured to generate the video data stream, such that the video data stream may, for example, include an indication of whether the video data stream is variable in number of decoding units for the video data stream.

[0388] In an embodiment, the video encoder may for example be configured to generate the video data stream such that the video data stream does not include an indication of whether the video data stream is variable in number of decoding units for the video data stream.

[0389] According to an embodiment, the video encoder may for example be configured to generate the video data stream such that the number of decoding units within an access unit depends on the frame time interval and the common delay increment.

[0390] In an embodiment, a video encoder may, for example, be configured to generate a video data stream such that the video data stream may, for example, include decoding unit information supplemental enhancement information messages of two or more decoding units of the access unit. The video encoder may, for example, be configured to generate a video data stream such that the decoding unit information supplemental enhancement information messages of the decoding units may, for example, include the delta time information of the decoding units.

[0391] In an embodiment, the video data stream may, for example, include a minimum picture duration flag in a video parameter set or a sequence parameter set of said video data stream, said minimum picture duration flag indicating whether frame time interval information is present in the absence of a constant frame rate.

[0392] Further, according to an embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video data stream including a plurality of access units, and a video decoder for decoding the video data stream to decode the video, the video data stream including delta time information for each of two or more decoding units of an access unit of the plurality of access units, a decoding unit removal time for each decoding unit of the two or more decoding units of the access unit being dependent on the access unit removal time of the access unit and dependent on the delta time information for the decoding unit, and the video decoder for decoding the video data stream using the delta time information for each of the two or more decoding units of the access unit.

[0393] According to an embodiment, the video data stream may for example comprise picture timing supplemental extension information, which may for example comprise delta time information of two or more decoding units of said access unit.

[0394] In an embodiment, the delta time information indicates a difference in removal time between two of two or more decoding units of the access unit, and the video decoder may be configured, for example, to decode the video data stream using the difference in removal time between the two decoding units.

[0395] According to an embodiment, a last decoding unit of two or more decoding units of the access unit has a removal time equal to the removal time of the access unit, and a video decoder may, for example, be configured to decode the video data stream using the removal time of the access unit.

[0396] In an embodiment, the access unit may, for example, include three or more decoding units, and the difference in removal time is equal for each pair of two consecutive decoding units of the three or more decoding units of the access unit.

[0397] In an embodiment, the picture timing supplemental enhancement information is applied to a sub-bitstream derived from the video data stream, and the number of decoding units remains constant.

[0398] In an embodiment, the frame time interval is signaled in a parameter set of a video data stream, an HRD parameter of a sequence parameter set, and a video decoder may for example be configured to decode said video data stream using said frame time interval.

[0399] According to an embodiment, a frame time interval can be derived as the difference between the removal times of two consecutive access units at the highest temporal level, and a video decoder can, for example, be configured to decode the video data stream using the frame time interval.

[0400] In an embodiment, each decoding unit of the two or more decoding units of the access unit may for example include one video coding layer network abstraction layer unit.

[0401] In an embodiment, the picture timing supplemental enhancement information is applied to sub-bitstreams derived from a video data stream, where there are different numbers of decoding units.

[0402] In an embodiment, the video decoder may be configured to derive the frame time interval, for example, according to the following: (elemental_duration_in_tc_minus1[maxTiD]+1) multiplied by ClockTicks.

[0403] According to an embodiment, the video data stream may, for example, include an indication indicating whether the video data stream is variable in the number of decoding units, and the video decoder may, for example, be configured to decode the video data stream by processing the indication.

[0404] In an embodiment, the video data stream does not include an indication of whether there is a variation for the video data stream in the number of decoding units.

[0405] According to an embodiment, the number of decoding units in an access unit depends on the frame time interval and the common delay increment.

[0406] In an embodiment, the video data stream may include decoding unit information supplemental enhancement information messages of two or more decoding units of the access unit, where the decoding unit information supplemental enhancement information messages of the decoding units may include, for example, delta time information of the decoding units, and a video decoder may, for example, be configured to decode the video data stream using the delta time information of the decoding units.

[0407] In an embodiment, the video data stream may, for example, include a minimum picture duration flag in a video parameter set or a sequence parameter set of said video data stream, said minimum picture duration flag indicating whether frame time interval information is present in the absence of a constant frame rate.

[0408] Further, according to an embodiment, a system for encoding video into a video data stream and for decoding video is provided. The system includes a video encoder as described above, and a video decoder as described above. The video encoder may for example be configured to encode the video into the video data stream, such that the video data stream has the video encoded therein. The video decoder may for example be configured to receive the video data stream and decode the video data stream into video.

[0409] As mentioned above, the DU timing is given as a delta of the AU timing. More specifically, the removal time of a DU is indicated by giving a delta time in either the PictureTimingSEI message or the Decoding Unit Information SEI message, which is related to the removal time of the AU that contains the particular DU.

[0410] If the information is included in the picture timing SEI message, the information signaled is the removal time difference between two DUs. The removal time of the last DU of an AU is the same as the removal time of the AU removal time, any other DU is signaled as the removal time difference to the next DU. There are two options to signal it (highlighted in different colors): [Table 5] The first way to signal it is when the DUs all have the same removal time difference in common. The second case is when the removal time differences between the DUs within an AU are not the same.

[0411] The current syntax of the Picture Timing SEI message in the presence of DU timing precludes the application of aspect 2 described in this invention since the number of DUs may change when sub-bitstreams are extracted from the bitstream.

[0412] In principle, if the common removal time difference between CUs is the same, it may be possible to make the picture timing SEI message still apply even if the number of DUs is changed. In one embodiment, there is a mode in the PT SEI message, where the PT SEI message applies to sub-bitstreams with different numbers of DUs. The number of DUs is derived from other syntax elements. The PT SEI message is modified as follows: [Table 6]

[0413] If du_not_constraint_flag is equal to 0, the value of du_common_cpb_removal_delay_flag is inferred to be 1. The value of num_decoding_units_minus1 is inferred to be equal to FrameTimeInterval divided by (du_common_cpb_removal_delay_increment_minus1 + 1)*ClockSubTicks minus 1.

[0414] The FrameTimeInterval may be signaled in the parameter set, the HRD parameter of the SPS, or may be derived as the difference between the deletion times of two consecutive access units at the highest temporal level.

[0415] In this case, there is an additional constraint that each DU contains one VCL NAL unit.

[0416] To explicitly signal the FrameTimeInterval, one can do the following: [Table 7]

[0417] min_pic_duration_within_cvs_present_flag is an SPS or VPS flag indicating the presence of a FrameTimeInterval in the absence of a constant frame rate, allowing the FrameTimeInterval to be indicated in both the case of a constant frame rate and the case of a non-constant frame rate.

[0418] In the second case, du_not_constraint_flag can be set to 1 only if the frame rate is constant. In that case, the value of FrameTimeInterval is derived as (elemental_duration_in_tc_minus1 [maxTiD]+1) multiplied by ClockTicks.

[0419] Alternatively, the additional signaling flag can be omitted by merging the respective indications with a common DU timing mode signaling as follows: [Table 8]

[0420] Note that the derivation of the number of DUs in an AU above is based on the FrameTimeInterval and a common delay increment (seconds given in number of clock-sub-ticks). Note also that in aspect 4, the clock-sub-ticks vary depending on the temporal sub-layer present in the bitstream. The number of DUs is derived using the clock-sub-ticks of the highest temporal layer, or the spreading factor described in aspect 4 is also taken into account in the FrameTimeInterval, so the same result is obtained.

[0421] Although some aspects are described in terms of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where the blocks or devices correspond to method steps or features of the method steps. Similarly, aspects described in terms of method steps also represent a description of a corresponding block or item or feature of the corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0422] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software or at least partly in hardware or at least partly in software. This implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, having electronically readable control signals, which may be stored thereon and which cooperate (or may cooperate) with a programmable computer system to perform the respective method. Thus, the digital storage medium may be computer readable.

[0423] Some embodiments according to the invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0424] Generally, embodiments of the present invention may be implemented as a computer program product having program code operable to perform one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0425] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0426] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0427] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0428] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals may be adapted to be transferred via a data communication connection, for example via the Internet.

[0429] A further embodiment comprises a processing means, such as for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0430] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0431] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0432] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0433] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0434] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0435] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the configurations and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented as descriptions and explanations of the embodiments herein.

[0436] References [1] ISO / IEC, ITU-T. High efficiency video coding. ITU-T Recommendation H.265 | ISO / IEC 23008 10 (HEVC), edition 1, 2013;edition 2, 2014.

Claims

[Claim 1] 1. A video data stream having video encoded therein, and an indication of whether one or more scalable nested supplemental enhancement information messages containing timing information for each of one or more output layer sets are present within the video data stream.

Images