Video data stream, video encoder, apparatus and method for hypothetical reference decoder and output layer set

Abstract

An apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, and for a playback speed change factor (Nx), the apparatus determines a decoding capability requirement according to a decoding capability requirement limit, where the playback speed change factor (Nx) is a forward playback speed change factor or a reverse playback speed change factor.

Description

[Technical Field]

[0001] The present invention relates to video encoding and video decoding, and in particular to a video encoder, a video decoder, an encoding and decoding method, and a video data stream for implementing advanced video coding concepts.

[0002] H.265 / HEVC (HEVC = High Efficiency Video Coding) is a video codec that already provides tools to improve or further enable parallel processing in the encoder and / or decoder. For example, HEVC supports the subdivision of a picture into an array of tiles that are coded independently of each other. Another concept supported by HEVC relates to WPP, according to which CTU rows or CTU lines of a picture can be processed in parallel from left to right, e.g., in stripes (CTU = coding tree unit), where a certain minimum CTU offset is followed during the processing of consecutive CTU lines. However, it would be desirable to have a video codec that more efficiently supports parallel processing capabilities of a video encoder and / or video decoder.

[0003] Usually, in video coding, the coding process of picture samples requires smaller partitions, where the samples are divided into several rectangular regions for joint processing such as predictive coding or transform coding. Therefore, a picture is partitioned into blocks of a certain size that remains constant during the coding of a video sequence. In the H.264 / AVC standard, fixed-size blocks of 16x16 samples, so-called macroblocks, are used (AVC = Advanced Video Coding).

[0004] In the latest HEVC standard (see [1]), there is a maximum size of a coding tree block (CTB) or coding tree unit (CTU) of 64x64 samples. Further descriptions of HEVC use the more general term CTU for such a type of block.

[0005] The CTUs are processed in raster scan order, starting with the top left CTU and processing the CTUs down to the bottom right CTU, picture line by picture line.

[0006] The coded CTU data is organized into a type of container called a slice. Originally, in previous video coding standards, a slice meant a segment containing one or more consecutive CTUs of a picture. Slices are used to segment coded data. From another perspective, a complete picture can also be defined as one large segment, so historically, the term slice continues to apply. In addition to the coded picture samples, slices also contain additional information related to the coding process of the slice itself, which is located in a so-called slice header.

[0007] According to the state of the art, the VCL (Video Coding Layer) also includes techniques for fragmentation and spatial partitioning. Such partitioning can be applied to video coding for various reasons, such as load balancing in parallel processing, CTU size matching in network transmission, error mitigation, etc.

[0008] Bitstreams specified in video coding standards have HRD conformance-related information. This conformance consists of a Hypothetical Reference Decoder (HRD) that includes a buffer model that assumes that NAL units enter a Coded Picture Buffer (CPB) before the decoder and are removed from there at specific times that ensure that the CPB size is not exceeded (buffer overrun) or that NAL units do not arrive later than they should be removed (buffer underrun). Furthermore, the model consists of a Decoded Picture Buffer (DPB) from which decoded pictures are output once they are no longer needed for prediction, and whose size is similarly constrained in many implementations. The timing information of the HRD is conveyed in the bitstream by so-called SEI messages, in particular a buffering period SEI message that defines the specific timing information of a buffering period (BP) (multiple access units or AUs), a picture timing (PT) SEI message that conveys timing information of a single associated AU, and a decoding unit information (DUI) SEI message that conveys timing information of an associated AU subset, i.e., a decoding unit or DU.

[0009] It is an object of the present invention to provide an improved video encoding and video decoding concept.

[0010] The object of the present invention is solved by the subject matter of the independent claims.

[0011] Preferred embodiments are provided in the dependent claims.

[0012] According to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. For a playback speed change factor (Nx), the apparatus determines a decoding capability requirement according to a decoding capability requirement limit. The playback speed change factor (Nx) is a forward playback speed change factor or a backward playback speed change factor.

[0013] Further, according to an embodiment, there is provided a video data stream having video encoded therein, the video data stream including information regarding a sample rate limitation and / or the video data stream including information regarding a bit rate limitation.

[0014] Further, according to an embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein, wherein the video encoder generates the video data stream, such that the video data stream includes information about a sample rate limitation and / or the video encoder generates the video data stream, such that the video data stream includes information about a bit rate limitation.

[0015] Further, according to an embodiment, there is provided a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. In the case of a playback speed change factor (Nx), the method includes determining a decoding capability requirement according to a decoding capability requirement limit. The playback speed change factor (Nx) is a forward playback speed change factor or a reverse playback speed change factor.

[0016] Further, according to an embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method comprising generating the video data stream such that the video data stream includes information regarding a sample rate limitation, and / or the method comprises generating the video data stream such that the video data stream includes information regarding a bit rate limitation.

[0017] Furthermore, a computer program is provided for implementing one of the above methods when run on a computer or signal processor.

[0018] Further, according to one embodiment, there is provided a video data stream having video encoded therein, the video data stream including a plurality of access units, and a set of access units of the video data stream including a plurality of access units, and the video data stream further includes, for each access unit of the subset of access units, an indication that a seamless splicing point exists after the access unit, the subset of access units being a proper subset of access units of the video data stream.

[0019] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes a plurality of access units, and a group of access units of the video data stream includes a plurality of access units. Further, the video encoder generates the video data stream such that the video data stream includes, for each access unit of the subset of access units, an indication that a seamless splicing point exists after the access unit. Further, the video encoder generates the video data stream such that the access units of the subset are a group of access units of a proper subset of the video data stream.

[0020] Further, according to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The apparatus processes the video data stream. The video data stream includes a plurality of access units, and a set of access units of the video data stream includes a plurality of access units. Further, the video data stream includes, for each access unit of the subset of access units, an indication that a seamless splicing point exists after the access unit. The subset of access units is a proper subset of access units of the video data stream. The apparatus processes the indication.

[0021] Further, according to one embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating a video data stream such that the video data stream includes a plurality of access units, wherein a set of access units of the video data stream includes a plurality of access units. The method further includes generating the video data stream such that the video data stream includes, for each access unit of the subset of access units, an indication that a seamless splicing point exists after the access unit. The method further includes generating the video data stream such that the access units of the subset are a proper subset of access units of the video data stream.

[0022] Further, according to one embodiment, there is provided a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The method includes processing the video data stream. The video data stream includes a plurality of access units, and a set of access units of the video data stream includes a plurality of access units. Further, the video data stream includes, for each access unit of a subset of access units, an indication that a seamless splicing point exists after the access unit. The subset of access units is a proper subset of access units of the video data stream. The method includes processing the indication.

[0023] Furthermore, a computer program is provided for implementing one of the above methods when run on a computer or signal processor.

[0024] Further, according to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The apparatus processes the input bitstream according to an output layer set to obtain sub-bitstreams. If the output layer set does not include a predetermined layer of a plurality of layers present in the video data stream, the apparatus removes at least one non-scalable nested supplemental enhancement information message assigned the predetermined layer. If the output layer set includes the predetermined layer, the apparatus is configured not to remove any non-scalable nested supplemental enhancement information message assigned the predetermined layer.

[0025] Further, according to one embodiment, there is provided a video data stream having video encoded therein, the video data stream including a video parameter set, the video parameter set including a plurality of output layer sets, and if any of the plurality of output layer sets does not include all layers of a plurality of layers present in the video data stream, the video data stream does not include any non-scalable nested supplemental enhancement information messages for a hypothetical reference decoder.

[0026] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes a video parameter set. Further, the video encoder generates the video data stream such that the video parameter set includes multiple output layer sets. If any of the multiple output layer sets does not include all layers of the multiple layers present in the video data stream, the video encoder generates the video data stream such that the video data stream does not include any non-scalable nested supplemental enhancement information messages for a hypothetical reference decoder.

[0027] Further, according to one embodiment, a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, includes processing the input bitstream according to an output layer set to obtain sub-bitstreams. If the output layer set does not include a predetermined layer (vps_layer_id[0]) of a plurality of layers present in the video data stream, the method includes removing at least one non-scalable nested supplemental enhancement information message assigned the predetermined layer (vps_layer_id[0]). If the output layer set includes the predetermined layer (vps_layer_id[0]), the method includes not removing any non-scalable nested supplemental enhancement information message assigned the predetermined layer (vps_layer_id[0]).

[0028] Further, according to one embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating a video data stream such that the video data stream includes a video parameter set. Further, the method includes generating the video data stream such that the video parameter set includes a plurality of output layer sets. If none of the plurality of output layer sets includes all layers of a plurality of layers present in the video data stream, the method includes generating the video data stream such that the video data stream does not include any non-scalable nested supplemental enhancement information messages for a hypothetical reference decoder.

[0029] Further, according to one embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes initial profile tier level information. Furthermore, the video data stream includes a first group of two or more layers. The apparatus processes the input bitstream according to one output layer set of a plurality of output layer sets and generates an output bitstream by removing at least one layer from the first group of two or more layers to obtain a second group of one or more layers, such that the output bitstream includes the second group of one or more layers but does not include the at least one layer removed from the first group. Furthermore, the apparatus removes from the initial profile tier level information those portions of the initial profile tier level information that are not associated with at least one layer of the second group of one or more layers.

[0030] Further, according to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video data stream including scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets, and the apparatus processes the input bitstream according to an output layer set of the plurality of output layer sets to obtain an output bitstream, wherein the output bitstream includes the decoding parameter set information for the output layer set and the output bitstream does not include decoding parameter set information for any other output layer sets of the plurality of output layer sets.

[0031] Further, according to one embodiment, there is provided a video data stream having video encoded therein, the video data stream including scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets.

[0032] Further, according to an embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream such that the video data stream includes scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets.

[0033] Additionally, according to one embodiment, there is provided a video data stream having video encoded therein, the video data stream including an indication of a plurality of operating points, the video data stream including a first mapping for each of the plurality of operating points that assigns one or more profile tier levels of the plurality of profile tier levels to the operating point, and the video data stream including a plurality of second mappings, each of the plurality of mappings assigning one of the plurality of operating points to one output level set of the plurality of output level sets.

[0034] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes an indication of a plurality of operating points. Further, the video encoder generates the video data stream such that the video data stream includes a first mapping for each operating point of the plurality of operating points that assigns one or more profile tier levels of the plurality of profile tier levels to the operating point. Further, the video encoder generates the video data stream such that the video data stream includes a plurality of second mappings, each of the plurality of mappings assigning one of the plurality of operating points to one output level set of the plurality of output level sets.

[0035] Further, according to one embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes an indication of a plurality of operating points. Furthermore, the video data stream includes a first mapping for each operating point of the plurality of operating points, mapping one or more profile tier levels of the plurality of profile tier levels to the operating point. Furthermore, the video data stream includes a plurality of second mappings, each of the plurality of mappings mapping one of the plurality of operating points to one output level set of the plurality of output level sets. The apparatus processes the input bitstream in accordance with an output layer set of the one of the plurality of output layer sets and in accordance with at least one of the second mappings, mapping one of the plurality of operating points to one output level set, and generating an output bitstream.

[0036] Further, according to one embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes initial profile tier level information. Furthermore, the video data stream includes a first group of two or more layers. The method includes generating an output bitstream by processing the input bitstream according to an output layer set of a plurality of output layer sets and removing at least one layer from the first group of two or more layers to obtain a second group of one or more layers, whereby the output bitstream includes the second group of one or more layers but does not include the at least one layer removed from the first group. Furthermore, the method includes removing from the initial profile tier level information those portions of the initial profile tier level information that are not associated with at least one layer of the second group of one or more layers.

[0037] Further, according to one embodiment, there is provided a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video data stream including scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets, the method including processing the input bitstream according to an output layer set of the plurality of output layer sets to obtain an output bitstream, wherein the output bitstream includes the decoding parameter set information for the output layer set and the output bitstream does not include decoding parameter set information for any other output layer sets of the plurality of output layer sets.

[0038] Further, according to an embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein, the method comprising generating a video data stream such that the video data stream includes scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets.

[0039] Additionally, according to one embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating the video data stream such that the video data stream includes an indication of a plurality of operating points. Further, the method includes generating the video data stream such that the video data stream includes a first mapping for each operating point of the plurality of operating points that assigns one or more profile tier levels of a plurality of profile tier levels to the operating point. Further, the method includes generating the video data stream such that the video data stream includes a plurality of second mappings, each of the plurality of mappings assigning one of the plurality of operating points to one output level set of a plurality of output level sets.

[0040] Further, according to one embodiment, a method is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes an indication of a plurality of operating points. Furthermore, the video data stream includes a first mapping for each operating point of the plurality of operating points, mapping one or more profile tier levels of a plurality of profile tier levels to the operating point. Furthermore, the video data stream includes a plurality of second mappings, each of the plurality of mappings mapping one of the plurality of operating points to one output level set of a plurality of output level sets. The method includes processing the input bitstream in accordance with an output layer set of the one of the plurality of output layer sets and in accordance with at least one of the second mappings, mapping one of the plurality of operating points to one output level set, and generating an output bitstream.

[0041] Furthermore, a computer program is provided for implementing one of the above methods when run on a computer or signal processor.

[0042] According to one embodiment, a video data stream having video encoded therein is provided. The video data stream includes a sequence parameter set and a decoding parameter set. The sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. The decoding parameter set includes a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. The first indication value is less than or equal to the second indication value.

[0043] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes a sequence parameter set and a decoding parameter set. The video encoder generates the video data stream such that the sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. Furthermore, the video encoder generates the video data stream such that the decoding parameter set includes a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. Furthermore, the video encoder generates the video data stream such that the first indication value is less than or equal to the second indication value.

[0044] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes a sequence parameter set and a decoding parameter set. Furthermore, the video encoder generates the video data stream such that the sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. Furthermore, the video encoder generates the video data stream such that the decoding parameter set includes a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. The second indication value of the decoding parameter set is an upper bound that takes precedence over the first indication value of the sequence parameter set.

[0045] Further, according to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes a sequence parameter set and a decoding parameter set. The sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. The decoding parameter set includes a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. If the first indication value is greater than the second indication value, the apparatus processes the input bitstream to generate an output bitstream such that the output bitstream includes the second indication value as an indication for the maximum number of sublayers or for the maximum number of sublayers minus the constant value.

[0046] Further, according to one embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating a video data stream such that the video data stream includes a sequence parameter set and a decoding parameter set. Furthermore, the method includes generating the video data stream such that the sequence parameter set includes a first indication value indicating a maximum number of sublayers, or a maximum number of sublayers minus a constant value, according to information stored in the sequence parameter set. Furthermore, the method includes generating the video data stream such that the decoding parameter set includes a second indication value indicating a maximum number of sublayers, or a maximum number of sublayers minus a constant value, according to information stored in the decoding parameter set. The method includes generating the video data stream such that the first indication value is less than or equal to the second indication value.

[0047] Further, according to one embodiment, there is provided a method for encoding video into a video data stream, such that the video data stream has video encoded therein. The method includes generating a video data stream such that the video data stream includes a sequence parameter set and a decoding parameter set. Furthermore, the method includes generating the video data stream such that the sequence parameter set includes a first indication value indicating a maximum number of sublayers, or a maximum number of sublayers minus a constant value, according to information stored in the sequence parameter set. Furthermore, the method includes generating the video data stream such that the decoding parameter set includes a second indication value indicating a maximum number of sublayers, or a maximum number of sublayers minus a constant value, according to information stored in the decoding parameter set. The second indication value of the decoding parameter set is an upper bound that takes precedence over the first indication value of the sequence parameter set.

[0048] Further, according to one embodiment, there is provided a method for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes a sequence parameter set and a decoding parameter set. The sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. The decoding parameter set includes a second indication value indicating the maximum number of sublayers or a maximum number of sublayers minus the constant value according to the information stored in the decoding parameter set. If the first indication value is greater than the second indication value, the method includes processing the input bitstream to generate an output bitstream such that the output bitstream includes the second indication value as an indication for the maximum number of sublayers or for the maximum number of sublayers minus the constant value.

[0049] Furthermore, a computer program is provided for implementing one of the above methods when run on a computer or signal processor. [Brief explanation of the drawings]

[0050] [Figure 1] 1 illustrates a video encoder for encoding video into a video data stream according to one embodiment. [Figure 2] 1 illustrates an apparatus for receiving an input video data stream according to one embodiment. [Figure 3] 1 illustrates a video decoder for receiving a video data stream having video stored therein, according to one embodiment. [Figure 4] An example of using reference picture resampling in addition to temporal scalability is shown. [Figure 5] Two splicing point options are shown: Option I (top) maintains the frame rate throughout the example, and Option II (bottom) has a frame rate drop in the last group of pictures of the bitstream. [Figure 6] 1 shows a video encoder. [Figure 7] 1 shows a video decoder. [Figure 8] It shows the relationship between the reconstructed signal, e.g. the reconstructed picture, on the one hand, and the prediction residual signal and the combination of the prediction signal signaled in the data stream, on the other hand. DETAILED DESCRIPTION OF THE INVENTION

[0051] The following description of the figures begins with presenting a description of an encoder and decoder of a block-based predictive codec for encoding pictures of video, to form an example of an encoding framework in which embodiments of the present invention may be incorporated. Respective encoders and decoders are described with reference to Figures 6 through 8. Below, descriptions of embodiments of the inventive concepts are presented along with an explanation of how such concepts may be incorporated into the respective encoders and decoders of Figures 6 and 7, although the embodiments described in Figures 1 through 3 and thereafter may also be used to form encoders and decoders that do not operate according to the underlying encoding framework of the encoders and decoders of Figures 6 and 7.

[0052] FIG. 6 illustrates a video encoder, illustratively an apparatus for predictively encoding picture 12 into data stream 14 using transform-based residual coding. The apparatus, or encoder, is indicated using the reference symbol 10. FIG. 7 illustrates a corresponding video decoder 20, e.g., apparatus 20, also configured to predictively decode picture 12′ from data stream 14 using transform-based residual decoding; an apostrophe is used to indicate that picture 12′ reconstructed by decoder 20 deviates from picture 12 originally encoded by apparatus 10 in terms of coding loss introduced by quantization of the prediction residual signal. While FIGS. 6 and 7 illustratively use transform-based predictive residual coding, embodiments of the present application are not limited to this type of predictive residual coding. This also applies to other details described with respect to FIGS. 6 and 7, as outlined below.

[0053] The encoder 10 is configured to subject the prediction residual signal to a spatial-to-spectral transform and encode the prediction residual signal into a data stream 14, thereby obtaining the prediction residual signal. Similarly, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and subject the prediction residual signal to a spectral-to-spatial transform, thereby obtaining the prediction residual signal.

[0054] Internally, the encoder 10 may include a prediction residual signal former 22, which generates a prediction residual 24 to measure the deviation of a prediction signal 26 from an original signal, e.g., picture 12. The prediction residual signal former 22 may be, for example, a subtractor that subtracts the prediction signal from the original signal, e.g., picture 12. The encoder 10 then further includes a transformer 28, which subjects the prediction residual signal 24 to a spatial-to-spectral transform to obtain a spectral-domain prediction residual signal 24'. The spectral-domain prediction residual signal is then quantized by a quantizer 32 and included by the encoder 10. The quantized prediction residual signal 24" is therefore coded into the bitstream 14. To this end, the encoder 10 may optionally include an entropy coder 34, which entropy codes the transformed and quantized prediction residual signal into the data stream 14. A prediction signal 26 is generated by a prediction stage 36 of the encoder 10 based on the prediction residual signal 24", which is coded into the data stream 14 and is decodable therefrom. To this end, the prediction stage 36 may internally include, as shown in FIG. 6, an inverse quantizer 38 that inversely quantizes the prediction residual signal 24" to obtain a spectral-domain prediction residual signal 24'" that corresponds to the signal 24' except for quantization losses, and then an inverse transformer 40 that subjects the latter prediction residual signal 24''' to an inverse transform, for example a spectral-to-spatial transform, to obtain a prediction residual signal 24'''' that corresponds to the original prediction residual signal 24 except for quantization losses. A combiner 42 of the prediction stage 36 then recombines the prediction signal 26 and the prediction residual signal 24'''', for example by addition, to obtain a reconstructed signal 46, for example a reconstruction of the original signal 12. The reconstructed signal 46 may correspond to the signal 12'. A prediction module 44 of the prediction stage 36 then generates a prediction signal 26 based on the signal 46, for example by using spatial prediction (such as intra-picture prediction) and / or temporal prediction (such as inter-picture prediction).

[0055] Similarly, decoder 20 may be internally constructed from components corresponding to prediction stage 36 and interconnected in a manner corresponding to the prediction stage, as shown in Figure 7. In particular, entropy decoder 50 of decoder 20 may entropy decode a quantized spectral domain prediction residual signal 24" from the data stream, where inverse quantizer 52, inverse transformer 54, synthesizer 56 and prediction module 58 are interconnected and cooperate in the manner described above with respect to the modules of prediction stage 36 to recover a reconstructed signal based on prediction residual signal 24" such that the output of synthesizer 56 results in a reconstructed signal, i.e., picture 12', as shown in Figure 7.

[0056] Although not specifically described above, it is readily apparent that the encoder 10 may set some coding parameters, including, for example, prediction modes, motion parameters, etc., according to some optimization schemes, such as a method for optimizing some rate- and distortion-related criteria, such as coding cost. For example, the encoder 10 and decoder 20 and corresponding modules 44, 58 may each support different prediction modes, such as intra-coding and inter-coding modes. The granularity at which the encoder and decoder switch between their prediction mode types may correspond to the subdivision of the pictures 12 and 12′, respectively, into coding segments or coding blocks. In units of these coding segments, for example, a picture may be subdivided into intra-coded blocks and inter-coded blocks. The intra-coded blocks are predicted based on their spatial, already coded / decoded neighbors, as outlined in more detail below. Several intra-coding modes may be present and selected for each intra-coded segment, including directional or angular intra-coding modes, according to which each segment is filled by extrapolating neighboring sample values ​​along a specific direction specific to the directional intra-coding mode into the respective intra-coded segment. These intra-coding modes may also include one or more additional modes, such as a DC coding mode, according to which prediction of each intra-coded block assigns a DC value to all samples in the respective intra-coded segment, and / or a planar intra-coding mode, according to which prediction of each block is approximated and determined to be a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample positions of the respective intra-coded block, as determined by the slope and offset of the plane defined by the two-dimensional linear function based on neighboring samples. In contrast, inter-coded blocks may be predicted, for example, temporally.For inter-coded blocks, motion vectors may be signaled within the data stream, indicating the spatial displacement of portions of previously coded pictures of the video to which picture 12 belongs, where the previously coded / decoded pictures are sampled to obtain a prediction signal for each inter-coded block. This means that in addition to the coding of the residual signal that data stream 14 contains, such as entropy-coded transform coefficient levels representing the quantized spectral-domain prediction residual signal 24", data stream 14 may also include coded therein coding mode parameters for assigning coding modes to various blocks, prediction parameters for some of the blocks, such as motion parameters for inter-coded segments, and optional further parameters, such as parameters for controlling and signaling the subdivision of pictures 12 and 12' into their respective segments. Decoder 20 uses these parameters to subdivide the picture in the same way as the encoder did, assign the same prediction modes to the segments, and perform the same prediction, resulting in the same prediction signal.

[0057] FIG. 8 illustrates the relationship between, on the one hand, a reconstructed signal, e.g., a reconstructed picture 12′, and, on the other hand, a combination of a prediction residual signal 24″″ signaled in the data stream 14 and a prediction signal 26. As already indicated above, this combination may be additive. The prediction signal 26 is illustrated in FIG. 8 as a subdivision of the picture region into intra-coded blocks, exemplarily shown using hatching, and inter-coded blocks, exemplarily shown without hatching. This subdivision may be any subdivision, such as a regular subdivision of the picture region into rows and columns of square or non-square blocks, or a multi-tree subdivision of the picture 12 (e.g., quad-tree subdivision) from a tree root block into multiple leaf blocks of various sizes, a combination of which is illustrated in FIG. 8, in which the picture region is first subdivided into rows and columns of a tree root block, and then further subdivided into one or more leaf blocks according to a recursive multi-tree subdivision.

[0058] Again, data stream 14 may have an intra-coding mode coded therein for intra-coded blocks 80, such that one of several supported intra-coding modes is assigned to each intra-coded block 80. In the case of inter-coded blocks 82, data stream 14 may have one or more motion parameters coded therein. In general, inter-coded blocks 82 are not restricted to being temporally coded. Instead, inter-coded blocks 82 may be any blocks that are predicted from previously coded portions beyond current picture 12 itself, for example, from a previously coded picture of the video to which picture 12 belongs, or from a picture of another view or a hierarchically lower layer if the encoder and decoder are scalable encoder and decoder, respectively.

[0059] In FIG. 8 , the prediction residual signal 24″″ is also shown as a subdivision of the picture region into blocks 84. These blocks may be referred to as transform blocks to distinguish them from the coding blocks 80 and 82. In fact, FIG. 8 shows that the encoder 10 and the decoder 20 may use two different subdivisions of the picture 12 and the picture 12′ into blocks, one into the respective coding blocks 80 and 82, and the other into the transform blocks 84. While both subdivisions may be the same, e.g., the coding blocks 80 and 82, respectively, and may simultaneously form the transform blocks 84, FIG. 8 also shows the case where, for example, the subdivision into the transform blocks 84 forms an extension of the subdivision into the coding blocks 80, 82, so that any boundary between the two blocks 80 and 82 overlaps with the boundary between the two blocks 84, or, in other words, each block 80, 82 coincides with one of the transform blocks 84 or with a cluster of transform blocks 84. However, these subdivisions may also be determined or selected independently of one another, such that transform blocks 84 may alternately cross the block boundaries between blocks 80 and 82. Thus, as far as the subdivision into transform blocks 84 is concerned, similar statements are true as made with respect to the subdivision into blocks 80, 82; for example, blocks 84 may be the result of a regular subdivision of a picture region into blocks (with or without arrangement into rows and columns), a recursive multi-tree subdivision of a picture region, or a combination thereof or any other type of block result. As an aside, it is noted that blocks 80, 82, and 84 are not limited to being square, rectangular, or of any other shape.

[0060] 8 further illustrates that the combination of prediction signal 26 and prediction residual signal 24"" directly results in reconstructed signal 12'. Note, however, that in alternative embodiments, more than one prediction signal 26 may be combined with prediction residual signal 24"" to result in picture 12'.

[0061] In Fig. 8, the transform blocks 84 have the following significance: the transformer 28 and the inverse transformer 54 perform their transforms in units of their transform blocks 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84. Some codecs allow to skip transforms, so that for some transform blocks 84, the prediction residual signal is directly coded in the spatial domain. However, according to the embodiment below, the encoder 10 and the decoder 20 are configured in such a way that they support several transforms. For example, the transforms supported by the encoder 10 and the decoder 20 are: DCT-II (or DCT-III) (DCT stands for Discrete Cosine Transform), DST-IV (DST stands for Discrete Sine Transform), DCT-IV, DST-VII, · Identity Transformation (IT), may include:

[0062] Naturally, the transformer 28 supports all of the forward transform versions of those transforms, but the decoder 20 or inverse transformer 54 supports their corresponding backward or inverse versions, Inverse DCT-II (or inverse DCT-III), · Reverse DST-IV, ·Inverse DCT-IV, · Inverted DST-VII, · Identity Transformation (IT), Support.

[0063] Subsequent descriptions provide further details on which transforms may be supported by the encoder 10 and decoder 20. Note that in any case, the supported transform set may include only one transform, such as one spectral-to-spatial or spatial-to-spectral transform.

[0064] As already outlined above, Figures 6 to 8 are presented as an example in which the inventive concepts described further below may be implemented to form specific examples of encoders and decoders according to the present application. Thus far, the encoders and decoders of Figures 6 and 7, respectively, may represent possible implementations of the encoders and decoders described herein below. However, Figures 6 and 7 are merely examples. However, an encoder according to an embodiment of the present application may use concepts outlined in more detail below to perform block-based encoding of picture 12 that differs from the encoder of Figure 6 in that it is not a video encoder but is still a picture encoder, that it does not support inter-prediction, or that the subdivision into blocks 80 is performed in a different way than illustrated in Figure 8. Similarly, a decoder according to an embodiment of the present application may perform block-based decoding of picture 12′ from data stream 14 using the coding concepts further outlined below, but may differ from decoder 20 of FIG. 7 in that, for example, it is not a video decoder but is still a picture decoder, that it does not support intra prediction, or that it subdivides picture 12′ into blocks in a different way than described with respect to FIG. 8, and / or that it derives prediction residuals from data stream 14 in the spatial domain rather than the transform domain, for example.

[0065] 1 shows a video encoder 100 for encoding video into a video data stream according to one embodiment. The video encoder 100 is configured to generate a video data stream.

[0066] 2 shows an apparatus 200 for receiving an input video data stream having video encoded therein, according to one embodiment, and configured to generate an output video data stream from the input video data stream.

[0067] 3 shows a video decoder 300 for receiving a video data stream having video stored therein, according to one embodiment. The video decoder 300 is configured to decode video from the video data stream.

[0068] Further, a system according to an embodiment is provided, which includes the device of Figure 2 and the video decoder of Figure 3. The video decoder (300) of Figure 3 is configured to receive the output video data stream of the device (200) of Figure 2. The video decoder 300 of Figure 3 is configured to decode video from the output video data stream of the device 200 of Figure 2.

[0069] In one embodiment, the system may further include, for example, the video encoder 100 of Figure 1. The apparatus 200 of Figure 2 may be configured to receive, for example, a video data stream from the video encoder 100 of Figure 1 as an input video data stream.

[0070] Intermediate device 210 (optional) of apparatus 200 may be configured, for example, to receive a video data stream from video encoder 100 as an input video data stream and to generate an output video data stream from the input video data stream. For example, the intermediate device may be configured, for example, to modify information (header / metadata information) of the input video data stream and / or may be configured, for example, to remove pictures from the input video data stream and / or may be configured to mix / splice the input video data stream with an additional second bitstream having a second video encoded therein.

[0071] The video decoder 221 (optional) may be configured, for example, to decode video from the output video data stream.

[0072] The hypothetical reference decoder 222 (optional) may be configured, for example, to determine timing information of the video according to the output video data stream, or may be configured, for example, to determine buffer information of a buffer in which the video or a portion of the video is stored.

[0073] The system includes a video encoder 101 of FIG. 1 and a video decoder 151 of FIG.

[0074] The video encoder 101 is configured to generate an encoded video signal. The video decoder 151 is configured to decode the encoded video signal to reconstruct video pictures.

[0075] Below, trick modes and fast playback are explained.

[0076] According to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. For a playback speed change factor (Nx), the apparatus determines a decoding capability requirement according to a decoding capability requirement limit. The playback speed change factor (Nx) is a forward playback speed change factor or a backward playback speed change factor.

[0077] In one embodiment, the decoding capability requirement may be, for example, a modified sample rate. The decoding capability requirement limit may be, for example, a sample rate limit.

[0078] According to one embodiment, the video data stream may include, for example, information regarding sample rate limitations.

[0079] In one embodiment, the device may be configured to determine the modified sample rate depending on, for example, the picture size of the pictures of the video.

[0080] According to one embodiment, the device may be configured to determine the modified sample rate, for example further depending on the frame rate and depending on a playback speed modification factor (Nx). The video data stream may for example contain information about the frame rate.

[0081] In one embodiment, the frame rate depends on one of the one or more sublayers. The video data stream may, for example, include sublayer-specific frame rate information for each of the one or more sublayers. The device may, for example, be configured to obtain the frame rate from the sublayer-specific frame rate information for one of the one or more sublayers.

[0082] According to one embodiment, the device may be configured to determine the modified sample rate depending on a picture size of a picture of the video, for example the largest picture size among a plurality of pictures of the video.

[0083] In one embodiment, the video data stream may include, for example, an application factor, and the device may be configured to calculate the modified sample rate using, for example, the maximum layer picture size and the application factor.

[0084] According to one embodiment, the maximum picture size depends on one sub-layer of the one or more sub-layers. The video data stream may, for example, include sub-layer-specific maximum possible picture size information for each of the one or more sub-layers. The device may, for example, be configured to obtain the maximum picture size from the sub-layer-specific maximum possible picture size information for one of the one or more sub-layers.

[0085] In one embodiment, the video data stream may include, for example, an application factor, and the device may be configured to determine the modified sample rate using, for example, the sample rate of the video and using the application factor.

[0086] According to one embodiment, the decoding capability requirement may be, for example, a modified bit rate. The decoding capability requirement limit may be, for example, a bit rate limit.

[0087] In one embodiment, the video data stream may include, for example, information regarding bit rate limitations.

[0088] According to one embodiment, the device may be configured to determine the modified bit rate, for example, depending on the initial bit rate of the video data stream and depending on a playback speed modification factor (Nx).

[0089] In one embodiment, the initial bit rate depends on one of the one or more sublayers. The video data stream may, for example, include sublayer-specific initial bit rate information for each of the one or more sublayers. The device may, for example, be configured to obtain the initial bit rate from sublayer-specific frame rate information for one of the one or more sublayers.

[0090] According to one embodiment, the video data stream may for example include an IRAP-only flag indicating whether decoding of only IRAP (IRAP = Intra random access point) pictures may take place when the initial playback speed may for example be increased. Depending on the IRAP-only flag the device may for example be configured to read the decoding capability requirements from the video data stream.

[0091] In one embodiment, the video data stream may include a reference picture only flag that indicates whether reference picture only decoding may be performed, for example, when the initial playback speed may be increased, etc. Depending on the reference picture only flag, the device may be configured to, for example, read the decoding capability requirements from the video data stream.

[0092] Further, according to one embodiment, there is provided a video data stream having video encoded therein, the video data stream may for example include information regarding sample rate limitations and / or the video data stream may for example include information regarding bit rate limitations.

[0093] According to one embodiment, the video data stream may include information regarding, for example, the frame rate.

[0094] In one embodiment, the video data stream may include, for example, sub-layer-specific frame rate information for each of one or more sub-layers.

[0095] According to one embodiment, the video data stream may include, for example, application coefficients for determining the modified sample rate.

[0096] In one embodiment, the video data stream may, for example, include sub-layer-specific maximum picture size information for each of one or more sub-layers.

[0097] According to one embodiment, the video data stream may include, for example, sub-layer-specific initial bitrate information for each of one or more sub-layers.

[0098] In one embodiment, the video data stream may include an IRAP-only flag that indicates whether decoding of IRAP pictures only may occur, for example, when the initial playback speed may, for example, be increased.

[0099] According to one embodiment, the video data stream may include a reference picture only flag that indicates whether reference picture only decoding may be performed, for example when the initial playback speed may be increased, for example.

[0100] In one embodiment, depending on the IRAP-only flag, the video data stream may include, for example, level information, and / or depending on the reference picture-only flag, the video data stream may include, for example, level information.

[0101] Further, according to an embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein, wherein the video encoder generates the video data stream, such that the video data stream includes information about a sample rate limitation and / or the video encoder generates the video data stream, such that the video data stream includes information about a bit rate limitation.

[0102] In one embodiment, the video encoder may be configured to generate the video data stream such that the video data stream may include information regarding, for example, frame rate.

[0103] According to one embodiment, the video encoder may be configured, for example, to generate the video data stream such that the video data stream may include sub-layer-specific frame rate information, for example, for each of one or more sub-layers.

[0104] In one embodiment, the video encoder may be configured, for example, to generate the video data stream such that the video data stream may include application coefficients, for example for determining a modified sample rate.

[0105] According to one embodiment, a video encoder may be configured, for example, to generate a video data stream such that the video data stream may include sub-layer-specific maximum picture size information, for example, for each of one or more sub-layers.

[0106] In one embodiment, the video encoder may be configured, for example, to generate the video data stream such that the video data stream may include sub-layer-specific initial bitrate information, for example, for each of one or more sub-layers.

[0107] According to one embodiment, the video encoder may be configured to generate a video data stream such that the video data stream may, for example, include an IRAP-only flag indicating whether decoding of IRAP pictures only may occur when the initial playback speed may, for example, be increased.

[0108] In one embodiment, the video encoder may be configured to generate a video data stream such that the video data stream may, for example, include a reference picture only flag that indicates whether reference picture only decoding may occur when the initial playback speed may, for example, be increased.

[0109] According to one embodiment, the video encoder may be configured to generate a video data stream such that the video data stream may include, for example, level information, e.g., in response to an IRAP-only flag, and / or the video encoder may be configured to generate a video data stream such that the video data stream may include, for example, level information, e.g., in response to a reference picture-only flag.

[0110] In one embodiment, the device may be configured to, for example, decode an input bitstream to decode video.

[0111] Furthermore, according to one embodiment, there is provided a system for encoding video into a video data stream and for decoding the video. The system includes a video encoder as described above and an apparatus as described above. The video encoder encodes the video into the video data stream such that the video data stream has the video encoded therein. The apparatus receives the video data stream as an input bitstream. Furthermore, the apparatus decodes the input bitstream to decode the video.

[0112] Fast forward or reverse (fast reverse play) operations are common operations performed in video applications. Typically, these operations consist of decoding only a subset of the bitstream, which, when decoded, is faster than the rate indicated by the bitstream, e.g., the frame rate. Also, even for non-trick mode operation, the increased speed at which the video is played can be important in some scenarios. While there is no significant difference in fast forward, the difference in fast play considered in this description can be that the secondary target is continuous, smooth playback (possibly with audio playback and synchronization) while the primary target is not; for example, fast forward may be discontinuous playback with many "jumps" in the content. In either case, the result is that the content is played at Nx speed.

[0113] Obviously, the faster the content is played back, the faster the bitstream needs to be decoded, and so higher sample rates and higher bit rates as indicated by the HRD parameters result in higher decoding capability requirements, i.e., higher levels.

[0114] If reference picture resampling (RPR) is not used, i.e., all pictures in the CVS or bitstream have the same size, the sample rate can be easily calculated as the picture size in samples divided by the frame rate and multiplied by a speed-up factor Nx. Similarly, the bit rate is multiplied by the speed-up factor Nx.

[0115] In the first embodiment, the two values ​​derived as described and discussed above are checked against the level limits in VVC, and the level to which the Nx speedup belongs is calculated as the minimum value for which the derived values ​​are less than the level limit.

[0116] Note that frame rate or bit rate being sub-layer specific means that whatever sub-layer is being played (e.g. fast forward is only performed by decoding temporal level 0), the corresponding value is calculated using the sub-layer specific signaled frame rate or bit rate.

[0117] However, when reference picture resampling (RPR) is used, the sample rate cannot be easily derived, i.e., only the worst case for the maximum picture size can be used, and therefore the worst case sample rate is derived. An example of how RPR can be used in addition to temporal scalability is shown in Figure 4.

[0118] FIG. 4 shows an example of using reference picture resampling in addition to temporal scalability.

[0119] In this example, the lowest sub-layer may have a picture size of 1920 x 1080, and the highest sub-layer may have a picture size of 960 x 540. Using the maximum picture sizes, and assuming only a 60 fps bitstream, this results in approximately 124 frames per second. * 10 6The sample rate of the sample is derived, and the actual value is approximately 78 * 10 6 That is, the worst case is 1.6 times higher than the actual value.

[0120] In one embodiment, the bitstream includes an indication for coefficients that should be applied to a worst-case derived sample rate. For signaled coefficients, the worst-case sample rate may be derived, divided by the signaled coefficient, and multiplied by a rate decoding factor to calculate the actual sample rate through a speed-up decoding operation.

[0121] In another embodiment, a maximum picture size value is indicated for each sub-layer, allowing a better approximation or even a correct value to be derived if the picture size does not vary within a sub-layer.

[0122] In addition, a sub-layer may not be used completely. Sometimes fast forward is performed by only decoding and playing either IRAP pictures (e.g., IDR) or pictures that are not non-reference pictures. However, it is not clear what level is required to decode such a sub-bitstream when decoding at Nx rate. In a further embodiment, additional level signaling is IRAP only: If only IRAP is decrypted, Reference pictures only: Non-reference pictures are discarded and the rest are decoded. This is shown in one of two cases:

[0123] The syntax for one such embodiment may be as follows: For profile_tier_level: [Table 1]

[0124] Similar to the irap-only case, the reference picture-only case can be added to the syntax.

[0125] The bit rate and CPB size can similarly be signaled as follows: [Table 2]

[0126] Below, guarding the delay_for_concatenation_ensured_flag is explained.

[0127] According to one embodiment, there is provided a video data stream having video encoded therein, the video data stream including a plurality of access units, and a group of access units of the video data stream including a plurality of access units, and the video data stream further includes, for each access unit of the subset of access units, an indication that a seamless splicing point exists after the access unit, the subset of access units being a proper subset of the access units of the video data stream.

[0128] In one embodiment, a video data stream may include, for example, multiple temporal sublayers, where an indication that a seamless splicing point exists after an access unit is only present in an access unit of a first temporal sublayer group of the multiple temporal sublayers, but an indication that a seamless splicing point exists after the access unit is not present in an access unit of a second temporal sublayer group of the multiple temporal sublayers.

[0129] According to one embodiment, the video data stream may include, for example, supplemental enhancement information for a buffering period, which indicates access units of a subset of the plurality of access units after which a seamless splicing point exists.

[0130] In one embodiment, the video data stream may include, for example, a divisor. A delay value may be assigned, for example, to each access unit of a plurality of access units of the video data stream. An indication that a seamless splicing point exists after an access unit is present only in those access units of the plurality of access units of the video data stream that have an assigned delay value, and the assigned delay value, when divided by the divisor, may be equal to, for example, a remainder value.

[0131] According to one embodiment, the remainder value may be, for example, 0.

[0132] In one embodiment, the video data stream may include, for example, residual values.

[0133] According to one embodiment, the indication for each access unit of the subset of access units indicating that a seamless splicing point exists after the access unit may be, for example, a flag.

[0134] In one embodiment, the flag may be, for example, delay_for_concatenation_ensured_flag.

[0135] According to one embodiment, the flag indicates a first value if a constant frame rate for output pictures resulting from decoding the video data stream cannot e.g. be ensured, and the flag may indicate a second value, different from the first value, if a constant frame rate for output pictures resulting from decoding the video data stream can e.g. be ensured.

[0136] In one embodiment, the flag indicates a second value if the output pictures resulting from decoding the video data stream have equidistant output times.

[0137] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes a plurality of access units, and a group of access units of the video data stream includes a plurality of access units. Further, the video encoder generates the video data stream such that the video data stream includes, for each access unit of the subset of access units, an indication that a seamless splicing point exists after the access unit. Further, the video encoder generates the video data stream such that the access units of the subset are a group of access units of a proper subset of the video data stream.

[0138] In one embodiment, a video encoder may be configured to generate a video data stream such that the video data stream may include, for example, multiple temporal sub-layers. The video encoder may be configured to generate the video data stream such that, for example, an indication that a seamless splicing point exists after an access unit is present only in an access unit of a first temporal sub-layer group of the multiple temporal sub-layers, while the video encoder may be configured to generate the video data stream such that, for example, an indication that a seamless splicing point exists after the access unit is not present in an access unit of a second temporal sub-layer group of the multiple temporal sub-layers.

[0139] According to one embodiment, the video encoder may be configured, for example, to generate the video data stream such that the video data stream may include supplemental enhancement information, for example, for a buffering period. The video encoder may be configured, for example, to generate the video data stream such that the supplemental enhancement information for the buffering period indicates access units of a subset of the plurality of access units after which there is a seamless splicing point.

[0140] In one embodiment, a video encoder may be configured to generate a video data stream such that the video data stream may include, for example, a divisor. The video encoder may be configured to generate a video data stream such that a delay value may be assigned to, for example, each access unit of, for example, a plurality of access units of the video data stream. The video encoder may be configured to generate a video data stream such that, for example, an indication that a seamless splicing point exists after an access unit is present only in those access units of the plurality of access units of the video data stream that have an assigned delay value, and such that, when divided by the divisor, the assigned delay value may be equal to, for example, a remainder value.

[0141] According to one embodiment, the video encoder may be configured to generate the video data stream such that the remainder value may be, for example, zero.

[0142] In one embodiment, the video encoder may be configured to generate the video data stream such that the video data stream may include, for example, residual values.

[0143] According to one embodiment, the video encoder may be configured to generate a video data stream such that an indication for each access unit of a subset of access units may be, for example, a flag, indicating that a seamless splicing point exists after the access unit.

[0144] In one embodiment, the video encoder may be configured to generate the video data stream such that the flag may be, for example, delay_for_concatenation_ensured_flag.

[0145] According to one embodiment, the video encoder may be configured to generate the video data stream such that the flag indicates a first value, e.g., if a constant frame rate for output pictures resulting from decoding the video data stream cannot be ensured. The video encoder may be configured to generate the video data stream such that the flag indicates a second value, different from the first value, e.g., if a constant frame rate for output pictures resulting from decoding the video data stream can be ensured.

[0146] In one embodiment, the video encoder may be configured to generate the video data stream such that the flag indicates a second value if, for example, output pictures resulting from decoding the video data stream have equidistant output times.

[0147] Further, according to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The apparatus processes the video data stream. The video data stream includes a plurality of access units, and a set of access units of the video data stream includes a plurality of access units. Further, the video data stream includes, for each access unit of the subset of access units, an indication that a seamless splicing point exists after the access unit. The subset of access units is a proper subset of access units of the video data stream. The apparatus processes the indication.

[0148] In one embodiment, a video data stream may include, for example, multiple temporal sublayers, where an indication that a seamless splicing point exists after an access unit is only present in an access unit of a first temporal sublayer group of the multiple temporal sublayers, but an indication that a seamless splicing point exists after the access unit is not present in an access unit of a second temporal sublayer group of the multiple temporal sublayers.

[0149] According to one embodiment, the video data stream may include, for example, supplemental enhancement information for a buffering period, the supplemental enhancement information for the buffering period indicating access units of a subset of the plurality of access units after which a seamless splicing point exists. An apparatus may be configured to process, for example, the supplemental enhancement information for the buffering period.

[0150] In one embodiment, the video data stream may include, for example, a divisor. The device may be configured to receive, for example, the divisor. A delay value may be assigned to, for example, each access unit of a plurality of access units of the video data stream. The indication of the presence of a seamless splicing point after an access unit is present only in those access units of the plurality of access units of the video data stream that have an assigned delay value, and the assigned delay value, when divided by the divisor, may be equal to, for example, a remainder value.

[0151] According to one embodiment, the remainder value may be, for example, 0.

[0152] In one embodiment, the video data stream may, for example, include the remainder values, and the device may, for example, be configured to receive the remainder values.

[0153] According to one embodiment, the indication for each access unit of the subset of access units that indicates the presence of a seamless splicing point after the access unit may be, for example, a flag. An apparatus may be configured to process the flag, for example.

[0154] In one embodiment, the flag may be, for example, delay_for_concatenation_ensured_flag.

[0155] According to one embodiment, the flag indicates a first value if a constant frame rate for output pictures resulting from decoding the video data stream cannot e.g. be ensured, and the flag indicates a second value different from the first value if a constant frame rate for output pictures resulting from decoding the video data stream can e.g. be ensured.

[0156] In one embodiment, the flag indicates a second value if the output pictures resulting from decoding the video data stream have equidistant output times.

[0157] According to one embodiment, the device may be configured to, for example, decode an input bitstream to decode video.

[0158] Furthermore, according to one embodiment, there is provided a system for encoding video into a video data stream and for decoding the video. The system includes a video encoder as described above and an apparatus as described above. The video encoder encodes the video into the video data stream such that the video data stream has the video encoded therein. The apparatus receives the video data stream as an input bitstream. Furthermore, the apparatus decodes the input bitstream to decode the video.

[0159] In the current VVC draft specification, there is an indication that seamless splicing of bitstream B can be achieved by enabling a splicing device, i.e., a device that concatenates portions of bitstream A and bitstream B, to find the AUs in bitstream A. Seamless splicing refers to the fact that the CPB removal time distance between the last AU of bitstream A and the first AU of bitstream B is the same as the CPB removal time of two consecutive AUs in bitstream A. Rather, non-seamless splicing means that decoding of the AUs is not achieved as early as desired because decoding (CPB removal) of the first AU of bitstream B must be delayed because the first AU of bitstream B cannot be transmitted to the decoder's CPB in the appropriate time (i.e., the last AU of bitstream A requires longer to be transmitted than desired). Current picture timing SEI messages in the VVC draft specification include a flag called delay_for_concatenation_ensured_flag which, when set to 1, indicates to a particular associated AU that splicing at this bitstream position will be seamless as long as the next AU (i.e., the beginning of bitstream B) has a BP SEI message with concatenation_flag equal to 1 and the selected InitCpbRemovalDelay is less than or equal to the value of max_initial_removal_delay_for_concatination indicated in bitstream A. Thus, for all of these AUs in bitstream A, delay_for_concatenation_ensured_flag represents a feasible option for seamless splicing.

[0160] The problem is that, given the above characteristics, not all of those access units also represent meaningful splicing points, and signaling the delay_for_concatenation_ensured_flag even for insignificant splicing points may lead to splicing by a poorly implemented splicing device at such suboptimal locations. Apart from that, the indication of the delay_for_concatenation_ensured_flag for all access units, even those that are not meaningful splicing points, incurs bitrate and processing overhead on the encoder side to correctly set the flag value even if it is unavailable to the splicing device. Insignificant splicing points in this context mean that even if the frame rate at the actual splicing points is maintained, some of these splicing points may omit parts of bitstream A, resulting in discontinuous playout of bitstream A before the splicing point, e.g., splicing in the middle of a hierarchical GOP may be omitted. An example is shown in Figure 5, where two bitstreams (A and B) are spliced ​​at two positions (I and II), with the first option (top) resulting in a constant frame rate throughout the example, while the second option (bottom) results in a reduced frame rate due to an incorrectly selected splicing point.

[0161] FIG. 5 shows two splicing point options, option I (top) maintaining the frame rate throughout the example, and option II (bottom) having a frame rate drop in the last GOP of bitstream A.

[0162] Therefore, the present invention reduces signaling overhead and avoids misleading splicing devices by selectively signaling the delay_for_concatenation_ensured_flag for a subset of access units.

[0163] In an embodiment where delay_for_concatenation_ensured_flag is signaled only for AUs of a particular temporal sublayer, the particular temporal sublayer is indicated in the associated BP SEI message.

[0164] In another embodiment where the delay_for_concatenation_ensured_flag is signaled only to a subset of all AUs during the buffering period, the AU subset is identified using information from the associated BP SEI message.

[0165] In one embodiment, the AU subset is indicated by signaling a divisor and, optionally, a remainder value. The delay_for_concatenation_ensured_flag in the PT SEI message for the buffering period is signaled only if the value of the associated AU divided by the divisor (e.g., cpb_removal_delay_minus1 or dpb_output_delay in the PT SEI message) is equal to the optional remainder, or, if it is not present, equal to zero.

[0166] Further, in one embodiment, if the output picture does not ensure the constant frame rate shown in the figure, the value of delay_for_concatenation_ensured_flag is set to 0. Note that the current specification currently focuses only on the decoding time and ensures that AUs with the flag set to 1 satisfy the condition (i.e., > threshold) that the difference between the last arrival time (the AU has reached the CPB completely) and the CPB removal delay satisfies the condition. Therefore, equidistance is achieved in the decoding times of consecutive AUs. The present invention here adds that when this flag is set to 1, the output times of consecutive AUs in bitstream A also have equidistant output times.

[0167] In the following, output layer set specific HRD is described.

[0168] According to one embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The apparatus processes the input bitstream according to an output layer set to obtain sub-bitstreams. If the output layer set does not include a predetermined layer (vps_layer_id[0]) of multiple layers present in the video data stream, the apparatus removes at least one non-scalable nested supplemental enhancement information message assigned to the predetermined layer (c]). If the output layer set includes the predetermined layer (vps_layer_id[0]), the apparatus is configured not to remove any non-scalable nested supplemental enhancement information message assigned to the predetermined layer (vps_layer_id[0]).

[0169] In one embodiment, if there is no output layer set consisting of all layers of the plurality of layers in the currently encoded video sequence of the video data stream, there may be no non-scalable nested supplemental enhancement information message, for example, for at least one of a buffering period payload, a picture timing payload, and a decoding unit information payload.

[0170] According to one embodiment, if there is no output layer set consisting of all layers of a plurality of layers in a currently encoded video sequence of a video data stream, there may be no non-scalable nested supplemental enhancement information message for, for example, at least one of a buffering period payload, a picture timing payload, a decoding unit information payload, and a sub-picture level information payload.

[0171] In one embodiment, if there is no output layer set consisting of all layers of a plurality of layers in a currently encoded video sequence of a video data stream, there may be no non-scalable nested supplemental enhancement information message for a payload type equal to a first value indicating a buffering period payload, and / or equal to a second value indicating a picture timing payload, or equal to a third value indicating a decoding unit information payload, or equal to a fourth value indicating a sub-picture level information payload, for example.

[0172] According to one embodiment, in the case of a non-scalable nested supplemental enhancement information message having a payload type equal to a first value, or equal to a second value, or equal to a third value, or equal to a fourth value, the non-scalable nested supplemental enhancement information message may be applied, if present, to all output layer sets consisting of, for example, all layers in the currently encoded video sequence within the entire bitstream.

[0173] In one embodiment, if there is no output layer set consisting of all layers of the plurality of layers in the currently encoded video sequence of the video data stream, there may be no non-scalable nested supplemental enhancement information message for a payload type, for example, equal to 0 indicating a buffering period payload, or equal to 1 indicating a picture timing payload, or equal to 130 indicating a decoding unit payload, or equal to 203 indicating a sub-picture level information payload.

[0174] According to one embodiment, for non-scalable nested supplemental enhancement information messages with payload type equal to 0, or equal to 1, or equal to 130, or equal to 203, the non-scalable nested supplemental enhancement information message is applied, if present, to all output layer sets consisting of all layers in the currently coded video sequence within the entire bitstream.

[0175] In one embodiment, if the output layer set does not include a specified layer (vps_layer_id[0]), the device may be configured to, for example, remove all nested supplemental enhancement information messages that are assigned the specified layer (vps_layer_id[0]).

[0176] Further, according to one embodiment, there is provided a video data stream having video encoded therein, the video data stream including a video parameter set, the video parameter set including a plurality of output layer sets, and if any of the plurality of output layer sets does not include all layers of a plurality of layers present in the video data stream, the video data stream does not include any non-scalable nested supplemental enhancement information messages for a hypothetical reference decoder.

[0177] In one embodiment, if there is no output layer set consisting of all layers of the plurality of layers in the currently encoded video sequence of the video data stream, there may be no non-scalable nested supplemental enhancement information message, for example, for at least one of a buffering period payload, a picture timing payload, and a decoding unit information payload.

[0178] According to one embodiment, if there is no output layer set consisting of all layers of a plurality of layers in a currently encoded video sequence of a video data stream, there may be no non-scalable nested supplemental enhancement information message for, for example, at least one of a buffering period payload, a picture timing payload, a decoding unit information payload, and a sub-picture level information payload.

[0179] In one embodiment, if there is no output layer set consisting of all layers of a plurality of layers in a currently encoded video sequence of a video data stream, there may be no non-scalable nested supplemental enhancement information message for a payload type equal to a first value indicating a buffering period payload, or equal to a second value indicating a picture timing payload, or equal to a third value indicating a decoding unit information payload, or equal to a fourth value indicating a sub-picture level information payload, for example.

[0180] According to one embodiment, in the case of a non-scalable nested supplemental enhancement information message having a payload type equal to a first value, or equal to a second value, or equal to a third value, or equal to a fourth value, the non-scalable nested supplemental enhancement information message may be applied, if present, to all output layer sets consisting of, for example, all layers in the currently encoded video sequence within the entire bitstream.

[0181] In one embodiment, if there is no output layer set consisting of all layers of the plurality of layers in the currently encoded video sequence of the video data stream, there may be no non-scalable nested supplemental enhancement information message for a payload type, for example, equal to 0 indicating a buffering period payload, or equal to 1 indicating a picture timing payload, or equal to 130 indicating a decoding unit payload, or equal to 203 indicating a sub-picture level information payload.

[0182] According to one embodiment, for non-scalable nested supplemental enhancement information messages with payload type equal to 0, or equal to 1, or equal to 130, or equal to 203, the non-scalable nested supplemental enhancement information message is applied, if present, to all output layer sets consisting of all layers in the currently coded video sequence within the entire bitstream.

[0183] In one embodiment, if the video data stream includes at least one non-scalable nested supplemental enhancement information message for a hypothetical reference decoder, at least one of the multiple output layer sets may include, for example, all layers of the multiple layers present in the video data stream.

[0184] According to one embodiment, all supplemental enhancement information messages of a hypothetical reference decoder are scalable nesting supplemental enhancement information messages since they are not associated with one output layer set of multiple output layer sets, which may, for example, include all layers of multiple layers present in a video data stream.

[0185] In one embodiment, all non-scalable nested supplemental enhancement information messages of a hypothetical reference decoder are associated with one output layer set of multiple output layer sets, which may, for example, include all layers of multiple layers present in the video data stream.

[0186] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes a video parameter set. Further, the video encoder generates the video data stream such that the video parameter set includes multiple output layer sets. If any of the multiple output layer sets does not include all layers of the multiple layers present in the video data stream, the video encoder generates the video data stream such that the video data stream does not include any non-scalable nested supplemental enhancement information messages for a hypothetical reference decoder.

[0187] In one embodiment, for example, if there is no output layer set consisting of all layers of a plurality of layers in a currently encoded video sequence of a video data stream, there may be no non-scalable nested supplemental enhancement information message for at least one of a buffering period payload, a picture timing payload, and a decoding unit information payload.

[0188] According to one embodiment, for example, if there is no output layer set consisting of all layers of a plurality of layers in a currently encoded video sequence of a video data stream, then there may be no non-scalable nested supplemental enhancement information messages for at least one of a buffering period payload, a picture timing payload, a decoding unit information payload, and a sub-picture level information payload.

[0189] In one embodiment, if there is no output layer set consisting of all layers of a plurality of layers in a currently encoded video sequence of a video data stream, there may be no non-scalable nested supplemental enhancement information message for a payload type equal to a first value indicating a buffering period payload, or equal to a second value indicating a picture timing payload, or equal to a third value indicating a decoding unit information payload, or equal to a fourth value indicating a sub-picture level information payload, for example.

[0190] According to one embodiment, in the case of a non-scalable nested supplemental enhancement information message having a payload type equal to a first value, or equal to a second value, or equal to a third value, or equal to a fourth value, the non-scalable nested supplemental enhancement information message may be applied, if present, to all output layer sets consisting of, for example, all layers in the currently encoded video sequence within the entire bitstream.

[0191] In one embodiment, if there is no output layer set consisting of all layers of the plurality of layers in the currently encoded video sequence of the video data stream, there is no non-scalable nested supplemental enhancement information message for a payload type equal to 0 indicating a buffering period payload, or equal to 1 indicating a picture timing payload, or equal to 130 indicating a decoding unit payload, or equal to 203 indicating a sub-picture level information payload.

[0192] According to one embodiment, for non-scalable nested supplemental enhancement information messages with payload type equal to 0, or equal to 1, or equal to 130, or equal to 203, the non-scalable nested supplemental enhancement information message is applied, if present, to all output layer sets consisting of all layers in the currently coded video sequence within the entire bitstream.

[0193] In one embodiment, if the video data stream may include at least one non-scalable nested supplemental enhancement information message, for example, of a hypothetical reference decoder, the video encoder may be configured to generate the video data stream such that at least one of the multiple output layer sets may include, for example, all layers of the multiple layers present in the video data stream.

[0194] According to one embodiment, the video encoder may be configured to generate a video data stream such that, for example, all supplemental enhancement information messages of a hypothetical reference decoder are scalable nesting supplemental enhancement information messages, since not all supplemental enhancement information messages are associated with one output layer set of a plurality of output layer sets, which may, for example, include all layers of a plurality of layers present in the video data stream.

[0195] In one embodiment, the video encoder may be configured to generate a video data stream such that, for example, all non-scalable nested supplemental enhancement information messages of a hypothetical reference decoder are associated with one output layer set of a plurality of output layer sets, which may, for example, include all layers of a plurality of layers present in the video data stream.

[0196] According to one embodiment, the device may be configured to, for example, decode an input bitstream to decode video.

[0197] The current VVC draft specification provides a means to carry HRD timing information (BP SEI message, PT SEI message, DUI SEI message) that applies only to sub-bitstreams, i.e., OLSs, in a nested format within the bitstream using scalable nesting SEI messages indicating the respective OLSs. Other HRD SEI messages are carried directly within the bitstream (as non-scalable nested SEI messages) and therefore need only apply to the 0th OLS, which is a single-layer sub-bitstream, as indicated in the VPS by the syntax element vps_layer_id[0]. However, as is typical for layered coding, removal of this layer or any other layer from the bitstream does not require a rewrite of the VPS, so there cannot even be a layer with nuh_layer_id equal to vps_layer_id[0].

[0198] This requirement poses the problem that when an OLS other than the 0th OLS is extracted, non-scalable nested SEI messages are not allowed, even if the extracted OLS contains a layer whose nuh_layer_id is equal to vps_layer_id[0]. Only after rewriting the VPS(vps_layer_id[0]) and the respective value into the NAL unit header of the respective layer can such non-scalable nested messages remain in the bitstream.

[0199] Below, two aspects of the present invention that solve the problem are introduced as follows. 1) As a first solution, the above complexity is kept, but the extraction process is adjusted in such a way that if the extracted OLS does not include a layer whose nuh_layer_id is equal to vps_layer_id[0], then only non-scalable nested SEI messages that apply to a layer whose nuh_layer_id is equal to vps_layer_id[0] are removed. This ensures that non-scalable nested SEI messages are not removed unnecessarily (or in other words, do not need to be unnecessarily duplicated for each OLS that includes a layer whose nuh_layer_id is equal to vps_layer_id[0]). 2) As a second solution, if there is no OLS defined in the VPS that corresponds to the complete bitstream (i.e., all layers therein), then no non-scalable nested SEI messages (BP, PT, DUI) can be present in the bitstream. Otherwise, if there is an OLS corresponding to all present layers in the bitstream, then each HRD SEI message can be present in the bitstream in a non-scalable nested manner. All other HRD SEI messages (applying to another OLS) must be present in a scalable nesting SEI message. Conversely, all non-scalable nested SEI messages correspond to the entire present bitstream.

[0200] For non-scalable nested supplemental enhancement information messages, if payloadType is equal to 0 (indicating buffering period, BP, content), 1 (indicating picture timing, PT, content), 130 (indicating decoding unit information, DUI, content), or 203 (indicating sub-picture level information, SLI, content), the non-scalable nested supplemental enhancement information message applies to all output layer sets, if present, consisting of all layers in the currently coded video sequence within the entire bitstream.

[0201] If there is no output layer set consisting of all layers in the currently coded video sequence in the entire bitstream, there shall be no non-scalable nested additive enhancement messages with payloadType equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI).

[0202] In the following, rewriting of the sublayer PTL into the DPS is described.

[0203] According to one embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes initial profile tier level information. Furthermore, the video data stream includes a first group of two or more layers. The apparatus processes the input bitstream according to an output layer set of a plurality of output layer sets and generates an output bitstream by removing at least one layer from the first group of two or more layers to obtain a second group of one or more layers, such that the output bitstream includes the second group of one or more layers but does not include the at least one layer removed from the first group. Furthermore, the apparatus removes from the initial profile tier level information those portions of the initial profile tier level information that are not associated with at least one layer of the second group of one or more layers.

[0204] According to one embodiment, the video data stream may include, for example, initial decoding parameter set profile tier level information as initial profile tier level information, and the device may be configured to, for example, remove from the initial decoding parameter set profile tier level information those portions of the initial decoding parameter set profile tier level information that are not associated with at least one layer of the second group of one or more layers.

[0205] Further, according to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein, the video data stream including scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets, and the apparatus processes the input bitstream according to an output layer set of the plurality of output layer sets to obtain an output bitstream, wherein the output bitstream includes the decoding parameter set information for the output layer set and the output bitstream does not include decoding parameter set information for any other output layer sets of the plurality of output layer sets.

[0206] In one embodiment, the video data stream may include, for example, a raw byte sequence payload, which may include, for example, decoding parameter set information for each output layer set of a plurality of output layer sets.

[0207] According to one embodiment, the video data stream may include, for example, a scalable nesting additional enhancement message, which may have, for example, decoding parameter set information for each output layer set of a plurality of output layer sets, which may have, for example, scalable nesting additional enhancement information.

[0208] In one embodiment, the video data stream may include a nesting type flag (nesting_type) having, for example, one or more bits.

[0209] According to one embodiment, when the nesting type flag (nesting_type) indicates a first value, this indicates that the scalable nesting supplemental extension message may include one or more decoding parameter sets, e.g., having decoding parameter set information, and that no further supplemental extension messages are nested within the scalable nesting supplemental extension message.

[0210] In one embodiment, when the nesting type flag (nesting_type) indicates the second value, this indicates that one or more supplemental enhancement messages nested within the scalable nesting supplemental enhancement message are applied to the output layer set.

[0211] According to one embodiment, when the nesting type flag (nesting_type) indicates a third value, this indicates that one or more additional enhancement messages nested within the scalable nesting additional enhancement message are applied to a subset of output layer sets.

[0212] In one embodiment, when the nesting type flag (nesting_type) indicates the fourth value, this indicates that the scalable nesting supplemental extension message may include, for example, one or more decoding parameter sets having decoding parameter set information.

[0213] Further, according to one embodiment, there is provided a video data stream having video encoded therein, the video data stream may include, for example, scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets.

[0214] In one embodiment, the video data stream may include, for example, a raw byte sequence payload, which may include, for example, decoding parameter set information for each output layer set of a plurality of output layer sets.

[0215] According to one embodiment, the video data stream may include, for example, a scalable nesting additional enhancement message, which may have, for example, decoding parameter set information for each output layer set of a plurality of output layer sets, which may have, for example, scalable nesting additional enhancement information.

[0216] In one embodiment, the video data stream may include a nesting type flag (nesting_type) having, for example, one or more bits.

[0217] According to one embodiment, when the nesting type flag (nesting_type) indicates a first value, this indicates that the scalable nesting supplemental extension message may include one or more decoding parameter sets, e.g., having decoding parameter set information, and that no further supplemental extension messages are nested within the scalable nesting supplemental extension message.

[0218] In one embodiment, when the nesting type flag (nesting_type) indicates the second value, this indicates that one or more supplemental enhancement messages nested within the scalable nesting supplemental enhancement message are applied to the output layer set.

[0219] According to one embodiment, when the nesting type flag (nesting_type) indicates a third value, this indicates that one or more additional enhancement messages nested within the scalable nesting additional enhancement message are applied to a subset of output layer sets, for example, one or more sub-pictures are assigned to the subset.

[0220] In one embodiment, when the nesting type flag (nesting_type) indicates the fourth value, this indicates that the scalable nesting supplemental extension message may include, for example, one or more decoding parameter sets having decoding parameter set information.

[0221] Further, according to an embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein, the video encoder generating the video data stream such that the video data stream includes scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets.

[0222] In one embodiment, a video encoder may be configured to, e.g., generate a video data stream, such that the video data stream may include, e.g., a raw byte sequence payload. The video encoder may be configured to, e.g., generate a video data stream, such that the raw byte sequence payload may include, e.g., decoding parameter set information for each output layer set of a plurality of output layer sets.

[0223] According to one embodiment, the video encoder may be configured, for example, to generate a video data stream such that the video data stream may have, for example, decoding parameter set information for each output layer set of a plurality of output layer sets, may have, for example, scalable nesting additional enhancement information, may include, for example, a scalable nesting additional enhancement message.

[0224] In one embodiment, the video encoder may be configured, for example, to generate the video data stream such that the video data stream may include a nesting type flag (nesting_type), for example having one or more bits.

[0225] According to one embodiment, the video encoder may be configured to generate a video data stream such that, for example, if the nesting type flag (nesting_type) indicates a first value, this indicates that the scalable nesting supplemental enhancement message may include one or more decoding parameter sets, for example having decoding parameter set information, and that no further supplemental enhancement messages are nested within the scalable nesting supplemental enhancement message.

[0226] In one embodiment, the video encoder may be configured to generate a video data stream such that, for example, if the nesting type flag (nesting_type) indicates a second value, this indicates that one or more additional enhancement messages nested within the scalable nesting additional enhancement message are applied to the output layer set.

[0227] According to one embodiment, the video encoder may be configured to generate a video data stream such that, for example, if the nesting type flag (nesting_type) indicates a third value, this indicates that one or more supplemental enhancement messages nested within the scalable nesting supplemental enhancement message are applied to a subset of output layer sets, for example, one or more sub-pictures are assigned to the subset.

[0228] In one embodiment, the video encoder may be configured to generate a video data stream such that, for example, if the nesting type flag (nesting_type) indicates a fourth value, this indicates that the scalable nesting supplemental extension message may include one or more decoding parameter sets, for example having decoding parameter set information.

[0229] Additionally, according to one embodiment, there is provided a video data stream having video encoded therein, the video data stream including an indication of a plurality of operating points, the video data stream including a first mapping for each of the plurality of operating points that assigns one or more profile tier levels of the plurality of profile tier levels to the operating point, and the video data stream including a plurality of second mappings, each of the plurality of mappings assigning one of the plurality of operating points to one output level set of the plurality of output level sets.

[0230] In one embodiment, the video data stream may include, for example, a decoding parameter set, which may include, for example, an indication of a number of operating points.

[0231] According to one embodiment, the video data stream may include, for example, a layer number indication indicating the number of layers in the video data stream, and / or the video data stream may include, for example, a sub-layer number indication indicating the number of sub-layers in the video data stream.

[0232] In one embodiment, the number of layers in a video data stream may be constant, for example, and / or the number of sub-layers in a video data stream may be constant, for example.

[0233] According to one embodiment, the video data stream may include scalable nesting supplemental enhancement information, for example, having decoding parameter set information for each output layer set of a plurality of output layer sets.

[0234] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes an indication of a plurality of operating points. Further, the video encoder generates the video data stream such that the video data stream includes a first mapping for each operating point of the plurality of operating points that assigns one or more profile tier levels of the plurality of profile tier levels to the operating point. Further, the video encoder generates the video data stream such that the video data stream includes a plurality of second mappings, each of the plurality of mappings assigning one of the plurality of operating points to one output level set of the plurality of output level sets.

[0235] In one embodiment, the video encoder may be configured to generate the video data stream, such that the video data stream may include, for example, a decoding parameter set. The video encoder may be configured to generate the video data stream, such that the decoding parameter set may include, for example, an indication of a plurality of operating points.

[0236] According to one embodiment, the video encoder may be configured, for example, to generate the video data stream such that the video data stream may include a layer number indication, for example, indicating the number of layers in the video data stream, and / or the video encoder may be configured, for example, to generate the video data stream such that the video data stream may include a sub-layer number indication, for example, indicating the number of sub-layers in the video data stream.

[0237] In one embodiment, the video encoder may be configured, for example, to generate the video data stream such that the number of layers in the video data stream may be, for example, constant, and / or the video encoder may be configured, for example, to generate the video data stream such that the number of sub-layers in the video data stream may be, for example, constant.

[0238] According to one embodiment, the video encoder may be configured, for example, to generate a video data stream such that the video data stream may include scalable nesting supplemental enhancement information having decoding parameter set information for each output layer set of a plurality of output layer sets.

[0239] Further, according to one embodiment, an apparatus is provided for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes an indication of a plurality of operating points. Furthermore, the video data stream includes a first mapping for each operating point of the plurality of operating points, mapping one or more profile tier levels of the plurality of profile tier levels to the operating point. Furthermore, the video data stream includes a plurality of second mappings, each of the plurality of mappings mapping one of the plurality of operating points to one output level set of the plurality of output level sets. The apparatus processes the input bitstream in accordance with an output layer set of the one of the plurality of output layer sets and in accordance with at least one of the second mappings, mapping one of the plurality of operating points to one output level set, and generating an output bitstream.

[0240] In one embodiment, the video data stream may include, for example, a decoding parameter set. The decoding parameter set may include, for example, an indication of a plurality of operating points. The device processes the decoding parameter set.

[0241] According to one embodiment, the video data stream may include, for example, a layer number indication indicating the number of layers in the video data stream, and / or the video data stream may include, for example, a sub-layer number indication indicating the number of sub-layers in the video data stream.

[0242] In one embodiment, the number of layers in a video data stream may be constant, for example, and / or the number of sub-layers in a video data stream may be constant, for example.

[0243] According to one embodiment, the video data stream may include, for example, scalable nesting supplemental enhancement information with decoding parameter set information for each output layer set of a plurality of output layer sets. An apparatus may, for example, be configured to process the scalable nesting supplemental enhancement information.

[0244] According to one embodiment, the device may be configured to, for example, decode an input bitstream to decode video.

[0245] Furthermore, according to one embodiment, there is provided a system for encoding video into a video data stream and for decoding the video. The system includes a video encoder as described above and an apparatus as described in claim 371. The video encoder encodes the video into the video data stream such that the video data stream has the video encoded therein. The apparatus receives the video data stream as an input bitstream. Furthermore, the apparatus decodes the input bitstream to decode the video.

[0246] The VVC draft specification includes a decoding parameter set (DPS) that describes bitstream constraints (PTLs). It is supported for use in capability exchange and negotiation, such as SDP in RTSP / SIP, or for stream selection in adaptive streaming scenarios. Its purpose is to indicate the maximum capabilities required for a given bitstream, so its range is greater than all parameter sets defined in previous codec generations (HEVC and AVC), which only had a coded video sequence (CVS) range. Here, the VVC draft specification describes CVC, i.e., bitstream concatenation constraints. The DPS mainly carries several profile tier-level information (PTLs), including all PTLs used in the bitstream; that is, all PTLs in this list must be supported for successful decoding of the corresponding bitstream. [Table 3]

[0247] The problem occurs when bitstream extraction or pruning is performed, which removes some layers. After such processing, the resulting bitstream no longer carries the full set of layers. For example, a version containing 3 layers (respective OLS) is extracted from a 6-layer bitstream. Therefore, in such cases, the DPS does not accurately describe the PTL of the content of the resulting bitstream after pruning, and therefore cannot further fulfill its purpose of capability negotiation.

[0248] Therefore, as part of this invention, three options are described on how to mitigate this problem and allow the use of DPS for capability negotiation.

[0249] Below, the DPS rewrite during the extraction process is described.

[0250] The PTL information in the DPS is adjusted during extraction in that only the PTL information corresponding to the remaining layers after extraction remains in the DPS, while other PTL information is removed.

[0251] In the following, sub-bitstream specific DPS nesting is described.

[0252] A scalable nesting SEI message carrying an OLS-specific DPS RBST is defined, and each DPS RBSP is extracted through a bitstream extraction process and written into a new DPS in the output bitstream.

[0253] In one embodiment, a new nesting SEI message is used to carry the parameter sets.

[0254] In another embodiment, a single scalable nesting SEI message is used to carry parameter sets or other SEI messages, in which case syntax elements are added to the scalable nesting SEI message that indicate what is being done inside. [Table 4]

[0255] In the above syntax, if a parameter set is included (nesting_type == 1), it is assumed that there are no other SEIs. In another embodiment, a parameter set and other SEIs can both be included in the nesting SEI message at the same time.

[0256] In another embodiment, nesting_type can be used for several purposes: - SEI messages nested within the nesting SEI are applied to OLS, An SEI message nested within a nesting SEI applies to a subset of the OLS, e.g., a subpicture within the OLS. There is a nested parameter set within the nesting SEI, Shows.

[0257] Below, the operating points within the DPS are described.

[0258] DPS rewriting during the extraction process and sub-bitstream-specific DPS nesting may work under certain circumstances, such as for constrained bitstreams. For example, DPS rewriting during the extraction process may only work if all VPSs of a bitstream are known in advance and the number of layers / sublayers does not change from CVS to CVS. In the case of sub-bitstream-specific DPS nesting, something similar happens, since in principle the OLSs (e.g., the number of them and IDs) can change from CVS to CVS.

[0259] In a further embodiment, operating points are defined. They may be defined within the DPS, and a PTL list is presented for each. The operating point index is added as input to the decoding process in addition to the OLS index to describe and select the appropriate operating point. The association of the operating point to the OLS is then signaled to the VPS.

[0260] The syntax elements of the DPS may change in number of sublayers for each CVS, and when an operation point is included, the number of sublayers is removed, resulting in the following changes: [Table 5]

[0261] In a further embodiment, num_sub_layers and the number of layers are kept constant in the bitstream, and an indication is added to the bitstream to indicate the mapping between the operating point and one or more sublayers.

[0262] In another embodiment, the scalable nesting SEI message defined in 4.2 is applied to the operating point and the mapping to the OLS is also provided in the bitstream.

[0263] Below, the MaxSubLayers in the Param set is described.

[0264] According to one embodiment, a video data stream having video encoded therein is provided. The video data stream includes a sequence parameter set and a decoding parameter set. The sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. The decoding parameter set includes a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. The first indication value is less than or equal to the second indication value.

[0265] In one embodiment, the video data stream may include, for example, a sequence parameter set and a decoding parameter set. The sequence parameter set may include, for example, a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. The decoding parameter set may include, for example, a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. The second indication value of the decoding parameter set is an upper bound that takes precedence over the first indication value of the sequence parameter set.

[0266] According to one embodiment, the constant value may be, for example, one.

[0267] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes a sequence parameter set and a decoding parameter set. The video encoder generates the video data stream such that the sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. Furthermore, the video encoder generates the video data stream such that the decoding parameter set includes a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. Furthermore, the video encoder generates the video data stream such that the first indication value is less than or equal to the second indication value.

[0268] Further, according to one embodiment, there is provided a video encoder for encoding video into a video data stream, such that the video data stream has video encoded therein. The video encoder generates the video data stream such that the video data stream includes a sequence parameter set and a decoding parameter set. Furthermore, the video encoder generates the video data stream such that the sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. Furthermore, the video encoder generates the video data stream such that the decoding parameter set includes a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. The second indication value of the decoding parameter set is an upper bound that takes precedence over the first indication value of the sequence parameter set.

[0269] In one embodiment, the constant value may be, for example, 1. The video encoder may be configured, for example, to generate the video data stream such that the sequence parameter set may include, for example, a first indication value indicating a maximum number of sublayers, or a maximum number of sublayers minus 1, according to information stored in the sequence parameter set. The video encoder may be configured, for example, to generate the video data stream such that the decoding parameter set may include, for example, a second indication value indicating a maximum number of sublayers, or a maximum number of sublayers minus 1, according to information stored in the decoding parameter set.

[0270] Further, according to one embodiment, there is provided an apparatus for receiving a video data stream as an input bitstream, the video data stream having video encoded therein. The video data stream includes a sequence parameter set and a decoding parameter set. The sequence parameter set includes a first indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the sequence parameter set. The decoding parameter set includes a second indication value indicating a maximum number of sublayers or a maximum number of sublayers minus a constant value according to information stored in the decoding parameter set. If the first indication value is greater than the second indication value, the apparatus processes the input bitstream to generate an output bitstream such that the output bitstream includes the second indication value as an indication for the maximum number of sublayers or for the maximum number of sublayers minus the constant value.

[0271] In one embodiment, the constant value may be, for example, 1. The sequence parameter set may include, for example, a first indication value indicating a maximum number of sublayers, or a maximum number of sublayers minus 1, according to information stored in the sequence parameter set. The decoding parameter set may include, for example, a second indication value indicating a maximum number of sublayers, or a maximum number of sublayers minus 1, according to information stored in the decoding parameter set. If the first indication value is, for example, greater than the second indication value, the apparatus may be configured to, for example, process the input bitstream to generate an output bitstream, such that the output bitstream may include, for example, the second indication value as an indication for the maximum number of sublayers minus 1.

[0272] When bitstream pruning, also called sublayer extraction, is performed and NAL units belonging to a certain Temporal ID are dropped from the bitstream, parameter sets are usually not changed, but some high-level parameter sets, such as DPS, may be rewritten.

[0273] This can lead to inconsistencies in the bitstream, making the decoding process unclear or more complex. For example, if the DPS is rewritten as described in the previous section and the actual number of sublayers is indicated as dps_max_sublayers_minus1 (excluding dropped Temporal IDs), but these values ​​in the SPS are not changed (which is not expected), then sps_max_sublayers_minus1 will have a higher value than that for the DPS. The decoder will not actually know which value to trust.

[0274] In one embodiment, the bitstream constraint is that sps_max_sublayers_minus1 must be <= dps_max_sublayers_minus1. Thus, some error resilience can be built in when faced with such "problems." In the case of such a constraint, when parsing a "bitstream" that violates this condition, it will be clear that an SPS with a higher value simply corresponds to a new bitstream (e.g., not found due to loss of an end-of-bitstream (EOB) NAL unit). This means that when bitstream pruning is performed, dps_max_sublayers_minus1 cannot change until sps_max_sublayers_minus1 also changes.

[0275] In another embodiment, no constraint is required and DPS signaling is used to set the operation point or output layer set of the bitstream sent to the decoder, i.e., if dps_max_sublayers_minus1 is less than sps_max_sublayers_minus1, the value signaled in DPS takes precedence over that in SPS, which indicates that the bitstream sent to the decoder has up to dps_max_sublayers_minus1.

[0276] If there is more than one layer, since the number of sublayers per layer can vary, dps_max_sublayers_minus1 applies to all output layers, and all non-output layers have either dps_max_sublayers_minus1 or their corresponding sps_max_sublayers_minus1 (if the latter is smaller than dps_max_sublayers_minus1).

[0277] While some aspects are described in terms of apparatus, it will be apparent that these aspects also represent descriptions of corresponding methods, where blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in terms of method steps also represent descriptions of corresponding blocks or items or features of corresponding apparatus. Some or all method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0278] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software, or at least partially in hardware or at least partially in software. This implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, having electronically readable control signals that may be stored thereon and that cooperate (or may be able to cooperate) with a programmable computer system to perform the respective methods. Thus, the digital storage medium may be computer-readable.

[0279] Some embodiments according to the invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0280] Generally, embodiments of the present invention may be implemented as a computer program product having program code operable to perform one of the methods when the computer program product runs on a computer, which may for example be stored on a machine-readable carrier.

[0281] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0282] In other words, an embodiment of the inventive methods is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0283] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer-readable medium) comprising, recorded on it, the computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0284] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals can be adapted to be transferred via a data communication connection, such as, for example, via the Internet.

[0285] A further embodiment comprises a processing means, such as for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0286] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0287] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0288] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0289] The devices described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0290] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0291] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore the intention to be limited only by the scope of the appended claims and not by the specific details presented by way of description and illustration of the embodiments herein. References

[0292] ISO / IEC,ITU-T.High efficiency video coding.ITU-T Recommendation H.265|ISO / IEC23008 10(HEVC),edition1,2013;edition2,2014.

Claims

1. encoding the plurality of access units into a video data stream including a first access unit and a second access unit; configured to encode a buffering period supplemental enhancement information (BP SEI) message specifying a temporal sub-layer into the video data stream; coding of the delay_for_concatenation_ensured_flag associated with the first access unit is performed only for access units associated with the temporal sublayer; a video encoder that, in response to a delay_for_concatenation_ensured_flag having a value of 1, determines that the first access unit is at a seamless splicing point with the second access unit in the video data stream when the second access unit has a BP SEI message with a concatenation_flag equal to 1 and a selected initCpbRemovalDelay that is less than or equal to a value of max_initial_removal_delay_for_concatenation.

2. 2. The video encoder of claim 1, wherein seamless splicing means that a coded picture buffer (CPB) deletion time distance between a first access unit associated with a delay_for_concatenation_ensured_flag having a value of 1 and the second access unit is the same as that of consecutive access units in the first access unit.

3. configured to encode a BP SEI message specifying the first access unit into the video data stream; The video encoder of claim 1 , wherein encoding of delay_for_concatenation_ensured_flag is performed only for the first access unit.

4. the BP SEI message specifying the first access unit includes a signaled divisor; 4. The video encoder of claim 3, wherein the first subset of access units includes all access units that have a dpb_output_delay that has a remainder of zero when divided by the signaled divisor or that has a dpb_output_delay that has a remainder of zero when divided by the signaled divisor.

5. the BP SEI message specifying the first access unit includes a signaled divisor and a signaled remainder; 4. The video encoder of claim 3, wherein the first subset of access units includes all of the access units having a cpb_removal_delay_minus1 that, when divided by the signaled divisor, has a remainder equal to the signaled remainder, or a dpb_output_delay that, when divided by the divisor, has a remainder equal to the signaled remainder.

6. encoding a plurality of access units from the video data stream, the access units including a first access unit and a second access unit; configured to encode, from the video data stream, a buffering period supplemental enhancement information (BP SEI) message specifying a temporal sub-layer; coding of the delay_for_concatenation_ensured_flag associated with the first access unit is performed only for access units associated with the temporal sublayer; and determining that the first access unit is at a seamless splicing point with the second access unit in the video data stream when the second access unit has a BP SEI message with a concatenation_flag equal to 1 and a selected initCpbRemovalDelay less than or equal to a value of max_initial_removal_delay_for_concatenation in response to a delay_for_concatenation_ensured_flag having a value of 1.

7. 7. The video decoder of claim 6, wherein seamless splicing means that a coded picture buffer (CPB) deletion time distance between a first access unit associated with a delay_for_concatenation_ensured_flag having a value of 1 and the second access unit is the same as that of consecutive access units in the first access unit.

8. configured to encode a BP SEI message specifying the first access unit from the video data stream; The video decoder of claim 6 , wherein coding of delay_for_concatenation_ensured_flag is performed only for the first access unit.

9. the BP SEI message specifying the first access unit includes a signaled divisor; 9. The video decoder of claim 8, wherein the first subset of access units includes all access units that have a dpb_output_delay that has a remainder of zero when divided by the signaled divisor or that has a dpb_output_delay that has a remainder of zero when divided by the signaled divisor.

10. the BP SEI message specifying the first access unit includes a signaled divisor and a signaled remainder; 9. The video decoder of claim 8, wherein the first subset of access units includes all of the access units having a cpb_removal_delay_minus1 that, when divided by the indicated divisor, has a remainder equal to the indicated remainder, or a dpb_output_delay that, when divided by the divisor, has a remainder equal to the indicated remainder.

11. decoding a plurality of access units from the video data stream, the access units including a first access unit and a second access unit; encoding, from the video data stream, a buffering period supplemental enhancement information (BP SEI) message specifying a temporal sub-layer into the video data stream; coding of the delay_for_concatenation_ensured_flag associated with the first access unit is performed only for access units associated with the temporal sublayer; and and determining, in response to a delay_for_concatenation_ensured_flag having a value of 1, that the first access unit is at a seamless splicing point with the second access unit in the video data stream when the second access unit has a BP SEI message with a concatenation_flag equal to 1 and a selected initCpbRemovalDelay that is less than or equal to a value of max_initial_removal_delay_for_concatenation.

12. 12. The video decoding method of claim 11, wherein seamless splicing means that a coded picture buffer (CPB) deletion time distance between a first access unit associated with a delay_for_concatenation_ensured_flag having a value of 1 and the second access unit is the same as that of consecutive access units in the first access unit.

13. decoding a BP SEI message specifying the first access unit from the video data stream; The video decoding method of claim 11 , wherein encoding of delay_for_concatenation_ensured_flag is performed only for the first access unit.

14. the BP SEI message specifying the first access unit includes a signaled divisor; 14. The video decoding method of claim 13, wherein the first subset of access units includes all access units that have a dpb_output_delay that has a remainder of zero when divided by the signaled divisor or that has a dpb_output_delay that has a remainder of zero when divided by the signaled divisor.

15. the BP SEI message specifying the first access unit includes a signaled divisor and a signaled remainder; 14. The video decoding method of claim 13, wherein the first subset of access units includes all of the access units having a cpb_removal_delay_minus1 that, when divided by the signaled divisor, has a remainder equal to the signaled remainder, or a dpb_output_delay that, when divided by the divisor, has a remainder equal to the signaled remainder.

16. encoding the plurality of access units into a video data stream including a first access unit and a second access unit; encoding a buffering period supplemental enhancement information (BP SEI) message specifying a temporal sub-layer into the video data stream; coding of the delay_for_concatenation_ensured_flag associated with the first access unit is performed only for access units associated with the temporal sublayer; and and determining that the first access unit is at a seamless splicing point with the second access unit in the video data stream when the second access unit has a BP SEI message with a concatenation_flag equal to 1 and a selected initCpbRemovalDelay that is less than or equal to a value of max_initial_removal_delay_for_concatenation in response to a delay_for_concatenation_ensured_flag having a value of 1.

Images