Highly scalable coding concept

Abstract

The disclosure describes a high-efficiency scalable coding concept. A video decoder, decoding a multi-layer video data stream encoded in a hierarchy of layers, decodes the multi-layer video data stream according to spatial segments obtained by subdividing pictures of the layers, including: an entropy decoder that entropy decodes data from the multi-layer video data stream to obtain residual data and coding parameters for the spatial segments; a predictor that generates a prediction signal related to a spatial segment based on at least the coding parameters; a combiner that combines the prediction signal and the residual data to reconstruct a portion of a picture, the subdivision of the pictures of the first and second layers being determined by decoding long-term and short-term syntax elements from the multi-layer video data stream, the long-term syntax elements having a first value indicating that the pictures of the second layer are subdivided in a manner that spatially aligns one spatial segment of a picture of the second layer with a corresponding spatial segment of a picture of the first layer, the short-term syntax elements including a second value indicating that at least one picture of the second layer is spatially aligned with a corresponding spatial segment of a picture of the first layer.

Description

[0001] The present application is a divisional application, the parent application of which has the application number 201811477939.1 and the filing date of January 4, 2014, and the title of “Efficient Scalable Coding Concept”. TECHNICAL FIELD

[0002] The present application relates to scalable coding concepts such as scalable video coding. BACKGROUND

[0003] In the art, scalable coding concepts are known. For example, in video coding, H.264 involves the SVC extension (Scalable Video Coding) allowing a base layer coded video data stream to be additionally augmented with extra enhancement layer data to improve the reconstructed quality of the base layer video in terms of different aspects such as spatial resolution, signal-to-noise ratio (SNR), etc. The recently finalized HEVC standard can also be extended by SVC profiles. HEVC differs from its predecessor H.264 in many aspects such as suitability for parallel decoding / coding and low-delay transmission, etc. As for the involved parallel coding / decoding, HEVC supports the WPP (Wavefront Parallel Processing) coding / decoding and the tile parallel processing concept. According to the WPP concept, each picture is split into sub-streams in a row-wise manner. The coding order within each sub-stream is directed from left to right. The sub-streams have a defined decoding order, i.e. from top sub-stream to bottom sub-stream. The entropy coding of the sub-streams is performed using probability adaptivity. The probability initialization is done independently for each sub-stream or the immediately preceding sub-stream is correspondingly reached from the left-hand edge of the preceding sub-stream at the end of the second CTB (Coding Tree Block) based on the initial adaptation state of the probabilities used at the time of entropy coding. Spatial prediction is not required to be restricted. That is, spatial prediction can cross the boundaries between intermediate consecutive sub-streams. In this way, the sub-streams can be coded / decoded in parallel with the current coding / decoding position forming a wavefront which proceeds in a diagonal manner from bottom left to top right, from left to right. According to the tile concept, a picture is segmented into tiles and spatial prediction across the boundary lines is suppressed in order to enable the coding / decoding of these tiles to be processed in parallel. Only in-loop filtering across tile boundary lines can be allowed. In order to support low-delay processing, the tile concept is extended in that it allows tiles to be switchable to re-initialize the entropy probabilities, to take the entropy probabilities saved in the sub-stream process before the processing, i.e. the sub-stream before the sub-stream the current tile starts to belong to, and to take the entropy probabilities which are continuously updated until the end of the immediately preceding tile. By this measure, the WPP and tile concepts are made more suitable for low-delay processing.

[0004] However, there are concepts under discussion which are more advantageous to have further improved scalable coding probabilities.

[0005] Therefore, it is an object of the present application to provide a concept which further improves scalable coding concepts.

[0006] This object is achieved by the subject matter of the independent claims pending. SUMMARY

[0007] The first aspect of the application relates to scalable video coding incorporating parallel processing concepts. Parallel processing concepts such as WPP and tile segmentation allow to decode pictures of a video in parallel with spatial segments into which the pictures are subdivided, e.g. in the form of sub-streams, tiles or slices. As spatial intra-picture prediction, inter-layer prediction limits the degree of parallelism of the decoding of layers that are dependent on each other by inter-layer prediction constraints. More precisely, spatial intra-layer prediction limits the degree of parallelism when decoding pictures of a single layer, which is addressed in different ways. For example, when using tiles as spatial segments, spatial intra-layer prediction is constrained by the fact that it cannot cross the boundary lines. In the case of WPP sub-streams, their parallel processing is performed in an interleaved way to produce a proper oblique processing wavefront. In the case of inter-layer prediction, the decoding of a dependent layer is performed according to collocated portions of a reference layer. Thus, the decoding of spatial segments of a dependent layer can be started first, at the time when the collocated portions of the reference layer have been processed / decoded. As in the case of inter-layer prediction of different views as different layers, or due to upsampling from a lower layer to a higher layer, the area of the "collocated portion" is enlarged in the case where "motion compensation" is allowed. That is, scalable decoding using inter-layer prediction is susceptible to make a video decoder that decodes in parallel the video stream derive from short-term syntax elements regarding these dependent layers the degree of parallelism of the parallel processing of the interdependent layers, the short-term syntax elements defining the subdivision of the pictures of these interdependent layers into their spatial segments. However, it is relatively cumbersome and computationally complex to do so consistently. In addition to that, when doing so, the video decoder is not able to properly schedule the decoding threads running in parallel to decode the multi-layer video stream. Thus, according to the first aspect of the application, the parallel decoding of interdependent layers of a multi-layer video stream is improved by introducing a long-term syntax element structure that guarantees that, when the long-term syntax element structure assumes a certain value, the video decoder subdivides the pictures of a dependent layer in such a way that the boundaries between the spatial segments of the pictures of a second layer overlap each boundary of the spatial segments of a first layer over a predetermined time period that is larger than the time interval of the short-term syntax elements. By this measure, the video decoder is able to rely on the fact that the multi-layer video stream is properly encoded so that the pictures of the interdependent layers are subdivided into spatial segments without neglecting the degree of parallelism that is feasible between these interdependent layers. More precisely, over the predetermined time period, by developing the constraint that the boundaries of the spatial segments in different layers are signaled to overlap each other, the decoder can schedule in advance the distribution of the spatial segments into the parallel processing threads of the video decoder. Nonetheless, the long-term syntax element structure allows to turn off this guarantee, thus allowing other application scenarios or high-end video decoders to perform the parallel processing scheduling based on the short-term only from the short-term syntax elements, i.e. without developing any guarantee regarding the relative positioning of the boundaries of the spatial segments of the interdependent layers. The long-term syntax element can also be used for the purpose of decisional opportunistic decoding.

[0008] Another aspect of the present application relates to scalability coding, according to which interdependent layers combine pictures of interdependent layers with parallel processing of interdependent layers undergo inter-layer prediction using upsampling from base layer to enhancement layer. In particular, this aspect relates to an interpolation method for performing upsampling from base layer to enhancement layer. Naturally, this interpolation method can make neighboring partitions of a base layer picture dependent on each other. That is, the interpolation method makes the interpolation results at the outer circumference of each portion of the upsampled base layer reference picture dependent on two pixels / pixels in the co-located partition of the base layer picture and the pixels of the neighboring partition. In other words, the area of the base layer picture that is used as a reference for inter-layer prediction of the co-located portion is predicted in the enhancement layer picture that is "painted black" or widened. By the way, the dependency caused by the interpolation method of inter-layer prediction has a negative impact on the degree of parallelism that can be achieved when parallel processing of interdependent layers. For example, according to a second aspect of the present application, a syntax element is introduced that informs the decoder that the interpolation is modified along the partitions of the base layer to not mix the pixels / pixels of neighboring partitions of the base layer picture, the partitions of the base layer picture and their upsampling pattern depending on the spatial segment of the enhancement layer picture or the spatial segment of the base layer and the enhancement layer. By introducing this syntax element, the encoder is able to switch between two modes: if the interpolation method is limited to make the portions of the base layer picture self-contained, i.e. the restriction is activated, the obtained degree of parallelism will be maximized when parallel decoding of interdependent layers, with a slight decrease in the quality of the interpolation along the edges of the partitions of the base layer picture, however, the degree of parallelism is reduced and the quality of the interpolation at the edges of the partitions is increased without restricting the interpolation method.

[0009] The third aspect of the present application relates to scalable video coding with parallel decoding of interdependent layers and attempts to alleviate the burden of the decoder to perform a parallel processing schedule by introducing a long-term syntax element structure which allows the decoder to determine the inter-layer offset or inter-layer delay for a predetermined time period which is larger than the time interval in which short-term syntax elements signal the size and positioning of spatial segments of pictures of interdependent layers and the spatial sampling resolution of these pictures. By introducing a long-term syntax element which signals the inter-layer offset, the video encoder is able to switch between two modes: according to a first mode, the encoder guarantees to the decoder a certain inter-layer offset corresponding to a certain degree of parallelism in the decoding of interdependent layers and sets the short-term syntax elements accordingly for the predetermined time period such that the actual inter-layer offset is equal to or even lower than the guaranteed one. According to another mode, the guarantee is not provided to the decoder, thus, the encoder is not required to set the short-term syntax elements to meet another requirement, such as, optionally, adapting the short-term syntax elements to the video content for the predetermined time period. Thus, when the count over the entire predetermined time period is obeyed, the first spatial segment of an enhancement layer picture to which the enhancement is co-located in time does not face any conflict when starting, at least with respect to the decoding of the first spatial segment of the enhancement layer picture for the predetermined time period, the explicitly signaled inter-layer offset in the data stream can be the basis for minimizing the count of the underlying base layer spatial segment which has to be decoded.

[0010] The fourth aspect of the present application concerns the signaling of the layers of the various NAL units to which a scalable video encoding and a multi-layer video data stream belong, as well as the positioning of these layers within the scalable space and the meaning of the scalable dimension across the scalable space. In order to facilitate the execution of tasks by intermediate network entities involved in the transmission of a multi-layer video data stream, this information is easily accessible to these intermediate network entities and can be easily managed. The fourth aspect of the present application, based on the inventors' insight, according to the present invention, typical application scenarios justify the expenditure of a type indicator field, the way the type indicator field concept, the layer indicator fields associated with the various layers within the NAL unit header, the type indicator field is interpreted as follows: if the type indicator field has a first status, mapping information in the generic information NAL unit maps the possible values of the layer indicator fields to operation points and the layer NAL units are associated with the operation points using the respective layer indicator fields and the mapping information. Likewise, the mapping between the various layers and the scalable constellation can be adapted to different degrees and a larger number of scalable spaces can be enabled, however, with the disadvantage of an increased total expenditure of management. If the type indicator field has a second status, the layer indicator fields are split into more than one part and the operation points associated with the respective NAL units are positioned by using the values of these parts as coordinates of a vector within the scalable space. By this measure, the mapping between the various layers and the scalable constellation allows a smaller number of scalable spaces to be enabled, however, with the advantage of a reduced total expenditure of management of the network entities. In both cases, the layer indicator fields can be identical, independent of the application request, however, the way the layer indicator fields direct the NAL units of the layers through the scalable space can be adapted to the current application and its specifics. The advantages of the adaptation outweigh the necessity of the additional expenditure of the type indicator field.

[0011] The fifth aspect of the present application concerns multi-layer video encoding, i.e. scalable video encoding, allowing different layers to use different codecs / standards. The possibility to allow successive layers to use different codecs / standards can support a delayed extension of already existing video environments, thus solving a multi-layer video data stream that is subsequently extended by further enhancement layers and using new and potentially better codecs / standards in view thereof. Network aggregates that are not able to understand the codecs / standards of some enhancement layers can still process the lower layers and feed back the multi-codec decoders through a transport layer decoder that, for each NAL unit of the multi-layer video data stream, identifies the same codec with which it is associated and hands over the NAL units of the multi-layer video data stream to a multi-standard multi-layer decoder accordingly

[0012] The sixth aspect of the present application relates to multi-layer video coding, which subdivides both pictures of a base layer and pictures of an enhancement layer into an array of blocks. In this case, the inter-layer offset can be effectively signaled by inserting a syntax element structure into the multi-layer video data stream, which indicates the inter-layer offset for decoding the pictures of the base layer and the enhancement layer in parallel in units of base layer blocks. That is, the sixth aspect of the present application is based on the finding that explicit transmission of a syntax element structure, which indicates the inter-layer offset between the base layer and the enhancement layer in units of base layer blocks, increases the amount of data to be transmitted only by mirroring, wherein the computational complexity of the decoder is greatly reduced compared to the case in which the decoder derives the inter-layer offset for decoding the pictures of the base layer and the enhancement layer in parallel based on other syntax elements, for example, other syntax elements showing the block size of base layer blocks and enhancement layer blocks, the sampling resolution of the pictures of the base layer and the pictures of the enhancement layer, etc. When the syntax element structure is implemented as a long-term syntax element structure, the sixth aspect is closely related to the third aspect, and thus the inter-layer offset indicates to the decoder a guarantee that also applies to a predetermined time period, which is larger than a time interval, within which short-term syntax elements in the multi-layer video data stream indicate these implicit necessities, otherwise referred to, to determine the inter-layer offset by combining these syntax elements in a relatively complex manner.

[0013] Naturally, all the above-mentioned aspects can be combined in a manner of two, three, four or all of them. BRIEF DESCRIPTION OF DRAWINGS

[0014] The preferred embodiments of the present application will be described below with reference to the drawings, in which:

[0015] FIG. 1 A video encoder is shown, which is used as a schematic embodiment for implementing any one of the multi-layer encoders further outlined below with reference to the following figures;

[0016] FIG. 2 A schematic block diagram of a video decoder is shown, which displays a video encoder mounted to FIG. 1

[0017] FIG. 3 A schematic diagram is shown, which illustrates a picture subdivided into sub-streams for WPP processing;

[0018] FIG. 4 A video decoder according to an embodiment is schematically shown, which employs the inter-layer alignment of spatial segments of the base layer and the enhancement layer according to the present embodiment for reducing the decoding process;

[0019] FIG. 5 ​A schematic diagram showing the subdivision of a picture into code blocks and slices, respectively, and slices being composed of integer multiples of code blocks and the decoding order defined between code blocks following the subdivision of the picture into slices;

[0020] FIG. 6 A syntax embodiment for implementing the embodiments in FIG. 4 ;

[0021] FIG. 7 A schematic diagram showing the different subdivision of a pair of base and enhancement layer pictures into slices;

[0022] FIG. 8 A further exemplary syntax applicable in connection with the embodiments in FIG. 4 ;

[0023] FIG. 9 A schematic diagram of a picture subdivided into slices and the application of an interpolation filter to perform upsampling for the sake of inter-layer prediction;

[0024] FIG. 10 A schematic block diagram of a multi-layer decoder configured to switch on or off upsampling interpolation separation in response to syntax elements in a multi-layer data stream;

[0025] FIG. 11 A schematic diagram of a pair of base and enhancement layer using inter-layer prediction from base to enhancement layer, thus upsampling from base layer sampling resolution to increased enhancement layer sampling resolution;

[0026] FIG. 12 A schematic diagram showing switchable upsampling interpolation separation according to FIG. 10 ;

[0027] FIG. 13 A schematic diagram showing a picture of a base layer overlapping a picture of an enhancement layer, and the picture of the base layer and the picture of the enhancement layer being subdivided into WPP sub-streams;

[0028] FIG. 14 An exemplary syntax for implementing the embodiments in FIG. 10 ;

[0029] FIG. 15A A schematic diagram showing a picture of a base layer spatially overlapping aligned with a picture of an enhancement layer, and the picture of the base layer and the picture of the enhancement layer being subdivided into different spatial segments;

[0030] FIG. 15B A schematic diagram showing a base layer picture overlapping an enhancement layer picture in FIG. 15A , but showing another possibility for selecting a partition along which to perform upsampling interpolation separation;

[0031] FIG. 16 A schematic block diagram of a video decoder according to an embodiment is shown, which video decoder according to the present embodiment is responsive to the long-term syntax element structure within a multi-layer video data stream to derive or not to derive a guarantee regarding the inter-layer offset between base layer and enhancement layer from the decoding thereof;

[0032] FIG. 17A A schematic diagram of a pair of base layer and enhancement layer pictures is shown, and the pair of base layer and enhancement layer pictures is subdivided into tiles to illustrate the inter-layer offset signaling by the long-term syntax element structure according to an embodiment of the present application; FIG. 16

[0033] FIG. 17B A schematic diagram of a pair of base layer and enhancement layer pictures is shown, and the pair of base layer and enhancement layer pictures is subdivided into sub-streams for WPP processing to illustrate an embodiment of the long-term syntax element structure according to an embodiment of the present application; FIG. 16

[0034] FIG. 17C A schematic diagram of a pair of base layer and enhancement layer pictures is shown, and the pair of base layer and enhancement layer pictures is subdivided into slices to illustrate an embodiment of the long-term syntax element structure according to an even further embodiment of the present application; FIG. 16

[0035] FIG. 18 A schematic diagram of a picture subdivided into sub-streams for WPP processing is shown, and additionally the wavefront results when decoding / encoding the picture in parallel using WPP according to an embodiment of the present application are indicated;

[0036] FIG. 19 A diagram showing the relationship between minimum inter-layer decoding offset and block size as well as the sampling resolution ratio between base layer and enhancement layer according to an embodiment of the present application is shown;

[0037] FIG. 20 An exemplary syntax for implementing the long-term syntax element structure signaling according to an embodiment of the present application is shown; FIG. 16

[0038] Another embodiment of a syntax for implementing an embodiment of the present application is shown; FIG. 21 FIG. 16 A syntax for implementing an embodiment of the present application is shown;

[0039] FIG. 22 A syntax for implementing an embodiment of the present application is shown;

[0040] FIG. 23 ​​​​A schematic block diagram of a network entity according to an embodiment is shown, by allowing inter-layer field interpolation switching within different layer indicator fields can reduce scalable coding according to the present embodiment;

[0041] FIG. 24 A schematic diagram showing the manner of switching in response to a type indicator field is shown;

[0042] FIG. 25 A schematic diagram showing the interpolation of switchable layer indicator fields according to an embodiment is shown in further detail;

[0043] FIG. 26 A schematic syntax for implementing switchable layer indicator field interpolation according to an embodiment is shown; FIG. 23

[0044] FIG. 27 A schematic diagram showing layer indicator field switching in relation to the syntax in FIG. 26

[0045] FIG. 28 A block diagram of a transport stream decoder configured to simplify the discard of enhancement layer NAL units for codecs other than the base layer codec is shown;

[0046] FIG. 29 A block diagram of a transport stream decoder connected to a single standard multi-layer decoder is shown, showing the behavior of the transport stream decoder according to an embodiment;

[0047] FIG. 30 A transport stream decoder connected to a multi-standard multi-layer decoder is shown, as well as the behavior of the transport stream decoder according to an embodiment;

[0048] FIG. 31 Another embodiment of a syntax for implementing switchable layer indicator field interpretation according to a further embodiment is shown;

[0049] FIG. 32 A schematic diagram showing a picture of any layer subdivided into blocks, the blocks indicating a further subdivision of the picture into spatial segments is shown;

[0050] FIG. 33 A schematic diagram showing a picture of any layer subdivided into blocks and slices is shown;

[0051] FIG. 34 A schematic diagram showing a picture subdivided into blocks and substreams is shown;

[0052] FIG. 35 ​​A schematic block diagram of a video decoder according to an embodiment is shown, the video decoder according to the present embodiment being configured to use a syntax element structure in a data stream to derive an inter-layer offset between pictures in an inter-prediction process in a unit of a block having a raster scan order defined therein;

[0053] FIG. 36 A schematic diagram showing possible modes of operation of a video decoder in FIG. 34 with respect to a syntax element structure within a data stream according to an embodiment is shown;

[0054] FIG. 37 A schematic diagram showing modes of operation of a video decoder in FIG. 35 in connection with a further embodiment, the inter-layer offset signaling according to the further embodiment being switchable between different explicit signaling types, i.e. signaling in different types of units;

[0055] FIG. 38 A schematic diagram showing modes of operation of a video decoder in FIG. 35 according to an even further embodiment, the inter-layer offset according to the even further embodiment being continuously measured during a parallel decoding process of base layer and enhancement layer pictures;

[0056] FIG. 39 A relationship between a rank of a particular block of a picture according to a raster scan decoding order on the one hand and a row index and a column index according to an embodiment on the other hand is shown;

[0057] FIG. 40 Different embodiments for a relatively regular subdivision of base layer pictures and enhancement layer pictures into blocks and the resulting from these different embodiments are shown;

[0058] FIG. 41 Embodiments for implementing a syntax of any of the embodiments in FIG. 35 to FIG. 40 are shown;

[0059] FIG. 42 Another syntax embodiment which can replace the syntax in FIG. 41 is shown;

[0060] FIG. 43 A syntax embodiment is shown, the embodiments in FIG. 16 and FIG. 35 according to which can be signaled in another part of a multi-layer data stream; and

[0061] FIG. 44 A schematic block diagram of a transport layer decoder connected to a multi-layer multi-standard decoder according to an embodiment is shown. DETAILED DESCRIPTION

[0062] First, as an overview, an embodiment of an encoder / decoder structure is proposed which is suitable for any of the subsequently proposed concepts.

[0063] FIG. 1 The overall structure of an encoder according to an embodiment is shown. The encoder 10 can be implemented to be able to operate in a multi-threaded manner or not only in a single-threaded manner. That is, the encoder 10 can for example be implemented to use multiple CPU cores. In other words, the encoder 10 can support parallel processing, but the encoder 10 does not necessarily have to support parallel processing. The produced bitstream can also be produced / decoded by a single-threaded encoder / decoder. However, the encoding concepts of the present application are able to support a parallel processing encoder, so that parallel processing can be applied effectively, without compromising compression efficiency. With respect to the parallel processing capability, reference is made to the following FIG. 2 The similar statement is valid for the decoder described.

[0064] The encoder 10 is a video encoder, but generally the encoder 10 can also be a picture encoder. A picture 12 of a video 14 is shown to be input to the encoder 10 at an input 16. The picture 12 shows a certain scene, i.e. the picture content. However, at the same time, the encoder 10 also receives at its input 16 another picture 15 which is related to both the picture 12 and the picture 15 belonging to different layers. For illustrative purposes only, the picture 12 is shown to belong to layer 0, while the picture 15 belongs to layer 1. FIG. 1 It is shown that layer 1 can involve a higher spatial resolution with respect to layer 0, i.e. the same scene can be shown with a higher number of picture samples, but this is only for illustrative purposes and alternatively, the picture 15 of layer 1 can have the same spatial resolution and can be different, e.g. can have a different spatial resolution in the direction of the field of view with respect to layer 0, i.e. the pictures 12 and 15 can be captured from different viewpoints.

[0065] The encoder 10 is a hybrid breed encoder, i.e. the pictures 12 and 15 are predicted by a predictor 18 and the prediction residual 20 obtained by a residual determiner 22 is subjected to a transform such as a DCT and a quantization by a transform / quantization module 24. The thus obtained transformed and quantized prediction residual 26 is subjected to an entropy coding in an entropy coder 28, such as an arithmetic coding or a variable length using e.g. context adaptivity. A decoder can use a reconstructible version of the residual, i.e. a dequantized and retransformed residual signal 30 is recovered by a retransform / requantization module 31 and recombined with a prediction signal 32 of the predictor 18 by a combiner 33, resulting in a reconstruction 34 of the pictures 12 and 15, respectively. However, the encoder 10 operates on a block basis. Therefore, the reconstruction signal 34 suffers from discontinuities at block boundaries and, thus, a filter 36 can be applied to the reconstruction signal 34 to result in a reference picture 38 of the pictures 12 and 15, respectively, on the basis of which the predictor 18 predicts the encoded pictures of the different layers successively. However, as indicated by the dashed lines in Fig. 1, the predictor 18 can also directly employ the reconstruction signal 34 without the filter 36 or an intermediate version in other prediction modes such as a spatial prediction mode. FIG. 1

[0066] The predictor 18 can select different prediction modes to predict a certain block of the picture 12. FIG. 1 One such block 39 of the picture 12 is exemplarily shown in Fig. 1. There can be a temporal prediction mode, according to which the picture 12 is partitioned into blocks 39 representing any block of the picture 12, which are predicted on the basis of a previously encoded picture such as the picture 12' of the same layer. There can also be a spatial prediction mode, according to which the blocks 39 are predicted on the basis of a previously encoded portion (neighboring block 39) of the same picture 12. FIG. 1 A block 41 of the picture 15 is also exemplarily shown in Fig. 1, representing any other block into which the picture 15 is partitioned. For the block 41, the predictor 18 can support the prediction modes just discussed, i.e. the temporal prediction mode and the spatial prediction mode. In addition, the predictor 18 can provide an inter-layer prediction mode, according to which the block 41 is predicted on the basis of a corresponding portion of the picture 12 of the lower layer. "Corresponding" in "corresponding portion" shall mean a spatial correspondence, i.e. a portion within the picture 12 that predicts the same scene portion as the block 41 shows in the picture 15.

[0067] Of course, the prediction by the predictor 18 can not be limited to picture samples. The prediction can also apply to any encoding parameter, i.e. prediction mode, motion vector for temporal prediction, disparity vector for multi-view prediction, etc. Then only the residual can be encoded in the bitstream 40.

[0068] ​The quantized residual data 26, the transform coefficients level and other residual data and the encoding parameters, including for example the prediction modes and prediction parameters for the individual blocks 39 of the picture 12 and for the individual blocks 41 of the picture 15 determined by the predictor 18, are compiled using a specific syntax and subjected to entropy coding by the entropy coder 28. The data stream 40 thus obtained output by the entropy coder 28 forms the bit stream 40 output by the encoder 10.

[0069] FIG. 2 A decoder adapted to the encoder in FIG. 1 , i.e. capable of decoding the bit stream 40, is shown. FIG. 2 The decoder in FIG. 1 The predictor 60 can alternatively or in addition use a filtered version of the reconstructed signal 68 or some intermediate version thereof. Likewise, the pictures of the different layers which are finally reproduced and output at the output 70 of the decoder 50 can be determined from the unfiltered version of the combined signal 68 or some filtered version thereof.

[0070] According to the slice concept, picture 12 is subdivided into slices 80 and picture 15 is subdivided into slices 82, respectively, and the prediction of blocks 39 within at least slice 80 and blocks 41 within slice 82 is restricted to use only data related to the same slice of the same picture 12, 15, respectively, as basis for spatial prediction. That is, the spatial prediction of blocks 39 is restricted to use previously encoded parts of the same slice, but temporal prediction modes are not restricted to rely on information of previously encoded pictures such as picture 12'. Likewise, the spatial prediction mode of blocks 41 is restricted to use only previously encoded data of the same slice, but temporal prediction modes and inter-layer prediction modes are not restricted. For illustrative purposes only, picture 15 and picture 12 are chosen to be subdivided into six slices, respectively. The subdivision into slices can be chosen individually in the bitstream 40 for pictures 12', 12 and 15, 15', respectively, and signaled. The number of slices per picture 12 and the number of slices per picture 15 can each be any one of one, two, three, four, six, etc., wherein the slice partitioning can be restricted to partitioning only regularly into rows and columns of slices. For the sake of completeness, it is noted that the way of independently encoding slices can not be restricted to intra prediction or spatial prediction, but can also include any prediction of coding parameters across slice boundaries, and the context selection in entropy coding can also be restricted to depend only on data of the same slice. Thus, a decoder is able to perform the just mentioned operations in parallel, i.e., in units of slices.

[0071] Alternatively or additionally, FIG. 1 and FIG. 2 The encoder and decoder in FIG. 3WPP substream 100 also represents a spatial partitioning of picture 12 and picture 15 into WPP substreams. In contrast to tiles and slices, WPP substreams do not impose restrictions on prediction and context selection across WPP substreams 100. WPP substreams 100 extend row-wise, such as across rows of LCUs (largest coding units) 101, i.e. the largest possible block of prediction coding mode that can be transmitted separately in the bitstream, and only a compromise with respect to entropy coding is made in order to enable support of parallel processing. Specifically, an order 102 is defined along the exemplary WPP substreams 100 leading from top to bottom, and for each WPP substream 100, except for the first WPP substream in order 102, the probability assessment of the sign Arabic letter, i.e. the entropy probability, is not completely reset but taken over or set equal to the probability resulting from the entropy encoding / decoding. Directly before a preceding WPP substream reaches its second LCU, as indicated by line 104, for each WPP substream located on the same side of picture 12 and picture 15, the decoder order is followed in LCU order or substream order, respectively, starting from the left-hand side as indicated by arrow 106 and leading to the other side in LCU row direction. Thus, by subjecting the sequence of WPP substreams of the same picture 12 and the same picture 15, respectively, to some coding delay between them, these WPP substreams 100 can be decoded / encoded in parallel, such that the parts of the respective pictures 12, 15 that are encoded / decoded in parallel, i.e. simultaneously, form some wave front 108 that moves in a left-to-right inclined manner over the picture.

[0072] Briefly, order 102 also defines a raster scan order among the LCUs, leading from top left LCU 101 to bottom right LCU row by row from top to bottom. Each WPP substream can correspond to one LCU row. Briefly, referring back to tiles, the latter can also be limited to align with LCU boundaries. Substreams can be segmented into one or more slices, without being constrained to LCU boundaries, as long as a boundary between two slices is involved that is within the substream. However, in case of a transition from one slice of a substream to the next slice of the substream, the entropy probability is taken over. If tiles, all tiles can be aggregated into one slice, or a tile can be segmented into one or more slices, without being constrained to LCU boundaries again, as long as a boundary between two slices is involved that is within the tile. If tiles, the order among the LCUs is changed, first, in tile order across the tile in raster scan order before proceeding to the next tile in tile order.

[0073] As described so far, picture 12 can be partitioned into tiles or WPP substreams, and likewise, picture 15 can be partitioned into tiles or WPP substreams. In theory, one can choose the WPP substream partitioning / concept for one of picture 12 and picture 15 and the tile partitioning / concept for the other of the two pictures. Alternatively, one can impose a restriction on the bitstream according to which the concept type (i.e., tile or WPP substream) must be the same for each layer. Another embodiment with respect to spatial segments includes slices. For transmission purposes, slices are used to divide the bitstream 40 into segments. Slices are packed into NAL units, which are the smallest entities for transmission. Each slice can be independently encoded / decoded. That is, as with context selection, etc., any prediction across slice boundaries is prohibited. In summary, these are three embodiments of spatial segments: slices, tiles, and WPP substreams. Furthermore, all three parallel concepts: tiles, WPP substreams, and slices can be used in combination, i.e., picture 12 or picture 15 can be divided into tiles, where each tile is divided into multiple WPP substreams. Furthermore, slices can be used to partition the bitstream into multiple NAL units, e.g., but not limited to, at tile or WPP boundaries. If tiles or WPP substreams are used and additionally slices are used to partition picture 12, 15 and the slice partitioning is misaligned with the other WPP / tile partitioning, then the spatial segment should be defined as the smallest independently decodable segment of picture 12, 15. Alternatively, one can impose a restriction on the bitstream with respect to the combination of concepts within a picture (12 or 15) and / or if one has to align the boundaries between differently used concepts.

[0074] Before discussing the above-described concepts of the present application, reference is again made to FIG. 1 and FIG. 2 It should be noted that the block structure of the encoder and decoder in FIG. 1 and FIG. 2 is only for illustrative purposes and the structure can also be different.

[0075] According to a first aspect, which is referred to as "alignment of tile boundaries", a long-term syntax element structure is used for signaling to ensure that, within a predetermined time period, such as a time period of a sequence of pictures, picture 15 of the second layer is subdivided such that the boundaries 84 between the spatial segments 82 of the picture of the second layer overlap with each boundary 86 between the spatial segments 80 of the picture of the first layer. Within a time interval shorter than the predetermined time period, such as in units of individual pictures, i.e., within an inter-picture distance interval, the decoder still determines periodically based on the short-term syntax elements of the multi-layer video data stream 40 how to actually subdivide picture 12 of the first layer into spatial segments 80 and how to actually subdivide picture 15 of the second layer into spatial segments 82, but the knowledge about the alignment already helps to plan the parallel processing workload distribution. For example, FIG. 1The solid line 84 in FIG. 8 represents an embodiment in which the slice boundary line 84 is spatially fully aligned with the slice boundary line 86 of layer 0. However, the above guarantee also allows the slice partitioning of layer 1 to be finer than the slice partitioning of layer 0, such that the slice partitioning of layer 1 further comprises additional slice boundary lines that do not spatially overlap with any of the slice boundary lines 86 of layer 0. In any case, the knowledge about the slice registration between layer 1 and layer 0 helps the decoder to distribute the available workload or processing power among the spatial segments that are processed in parallel at the same time. Without the long-term syntax element structure, the decoder would have to perform the workload distribution over shorter time intervals, i.e., per picture, wasting computer power that performs the workload distribution. The other aspect is "opportunistic decoding": a decoder with multiple CPU cores can exploit the knowledge about the parallelism of the layers to decide to try or not to try to decode a higher complexity layer, i.e., a higher spatial resolution or a higher number of layers. By exploiting all cores of the same decoder, it is possible to decode bitstreams that exceed the capabilities of a single core. This information is particularly useful if the profile and level indicators do not contain an indication about the minimum parallelism.

[0076] For a more clear understanding of the aspects just summarized, reference will now be made to the detailed description of the application: FIG. 4 , FIG. 4 A video decoder 600 is shown that can be implemented in accordance with the provisions of the references FIG. 2 . As described above, i.e., the decoder 600 is configured to decode a multi-layer video data stream into a scene that is encoded in a hierarchy of layers with inter-layer prediction from a first layer 0 to a second layer 1 as described above. The video decoder supports parallel decoding of the multi-layer video data stream in spatial segments into which a picture of the layer is subdivided, e.g., slices, WPP substreams, etc. In other words, the video decoder is capable of parallel decoding of the multi-layer video data stream, as such the video decoder 600 operates on the picture 12 of layer 0 and the picture 15 of layer 1 in units of spatial segments.

[0077] As outlined above, for example, the spatial segments can be tiles and the video decoder 600 is configured to decode the pictures 12 of layer 0 and the pictures 15 of layer 1 using intra-picture spatial prediction, the video decoder 600 discontinues the intra-picture spatial prediction for each tile at its tile boundary line. Such as, individually, for each temporal frame 604 related to each picture 12 and 15, i.e. for each pair of pictures 12 and 15 belonging to a certain temporal frame 604, it is signaled within the data stream 40 by means of a short-term syntax element 602 based on a short-term that the pictures 12 and the pictures 15 are subdivided into tiles, such as in time intervals. As outlined above, i.e. the subdivision of the pictures 12 and the pictures 15 into tiles can be restricted to a rectangular regular subdivision into rows and columns only. Thus, the short-term syntax element 602 sets the number of rows and the number of columns of the tile subdivision for each picture 12 and each picture 15 of both layers individually. When decoding the multi-layer video data stream 40 inbound, the video decoder 600 is configured to apply spatial prediction, potentially temporal prediction. Optionally, the video decoder 600 entropy decodes each tile individually. If probability adaptivity is used in the course of decoding each tile, the video decoder 600 initializes the entropy probabilities for each tile individually, so that the tiles are entropy decoded in parallel. In addition to spatial prediction and optionally temporal prediction, the video decoder 600 supports inter-layer prediction as soon as the decoding of a tile of the picture 15 of layer 1 is involved. As outlined above, the inter-layer prediction can involve different parameters of the picture 15 of layer 1 involved in the decoding: the inter-layer prediction can predict the prediction residual of layer 1, such as the transform coefficients, the prediction mode used in layer 1, the prediction parameters used in layer 1, the samples of the enhancement layer 1 and the picture, etc. For example, in case layer 0 and layer 1 involve different views of the same scene, by controlling the disparity vector prediction parameters of the inter-layer prediction, the inter-layer prediction predicts parts within the tile of the picture 15 of layer 1 based on the decoded parts of the picture 12 of layer 0 (directly co-located parts or parts spatially slightly deviating from the direct co-located position).

[0078] As used FIG. 4In response to the long-term syntax element structure of the data stream 40, the video decoder 600 processes the predetermined time period 608 following the long-term syntax element structure 606 to different extents, as indicated by the reference symbol 606, the predetermined time period 608 comprising a number of time intervals, i.e. a plurality of time frames 604, within which the short-term syntax elements 602 signal individually the subdivision of the pictures into tiles. It is noted that 608 can relate to the range (= time period) and the SPS change that causes a severe reinitialization in any way. As long as long-term features are mentioned here, the just mentioned notes are valid for all embodiments related to other aspects as well. Specifically, if the long-term syntax element structure 606 assumes a value outside the first possible set of values, the video decoder 600 interprets this situation as a guarantee that within the predetermined time period the picture 15 of layer 1 is subdivided such that the boundaries between the tiles of the picture 15 overlap with each boundary of the tiles of the picture 12 of layer 0. In this case, the video decoder 600 still detects the short-term syntax elements 602 to determine the subdivision of the picture 12 and the picture 15 into their tiles within the time intervals 602 of the predetermined time period 608, but by comparing each pair of time-aligned pictures 12 and 15, the video decoder 600 can rely on the fact and recognize that the boundaries of the base layer tiles of the picture 12 completely overlap with the boundaries of the enhancement layer tiles of the picture 15, i.e. the tile subdivision of the picture 15 locally corresponds to or represents a spatial refinement of the subdivision of the picture 12 into tiles. As mentioned above, by scheduling the parallel processing of the tiles of the picture 12 and the tiles of the picture 15 within the predetermined time period 608 accordingly in parallel, i.e. by decoding the time-aligned pair of pictures 12 and 15 in parallel, the video decoder 600 can exploit the advantage of this signaling, i.e. the long-term syntax element structure 606 assumes a value outside the first possible set of values. For example, if the long-term syntax element structure assumes a value outside the first possible set of values, the video decoder 600 can learn that for a certain picture 12 of layer 0 the first tile in the tile order of the picture 12 locally coincides with the respective tile of the time-aligned enhancement layer picture 15 or completely locally overlaps with the first tile in the tile order of the picture 15 of the enhancement layer picture 15. Thus, at least in the absence of inter-layer prediction without disparity / motion compensation, the video decoder 600 can start the decoding of the first tile of the enhancement layer picture 15 as soon as possible upon completion of the decoding of the first tile of the time-aligned base layer picture 12, since the just mentioned guarantee indicates to the video decoder 600 that the co-located portion of the base layer picture 12 required for the inter-layer prediction is available for the entire first tile of the enhancement layer picture 15. Thus, by the video decoder 600 it can be recognized / determined that the inter-layer offset or the degree of parallelism between the base layer picture 12 and the enhancement layer picture 15 equals one tile of the base layer picture 12.If the inter-layer prediction includes disparity vectors with a non-zero vertical component and / or disparity vectors with a horizontal component, the horizontal component shifts the corresponding portion within the base layer picture towards the right side, possibly with a slight increase of the offset, wherein the tile order between tiles can be directed in a row-wise raster scan order from the top left corner to the bottom right corner of picture 12, 15.

[0079] However, if the long-term syntax element structure assumes a value outside of the second set of possible values, which is significantly different from the first set of possible values, the video decoder 600 does not take advantage of any guarantees, but rather schedules and plans the parallel decoding of tiles of picture 12 and picture 15 based on the short-term schedule using the short-term syntax element 602 and, potentially, for at least some of the time-aligned pairs of picture 12 and picture 15, the parallel decoding of tiles of the base layer and the enhancement layer. In this case, however, the video decoder 600 determines the minimum inter-layer offset or the inter-layer spatial processing offset, i.e. the degree of parallelism between layer 0 and layer 1, in the parallel decoding between layer 0 and layer 1 based on the short-term (tricky procedure). For at least a subset of the set of possible values of the short-term syntax element, there are boundaries between spatial segments of the pictures of the second layer that do not overlap any of the boundaries of the spatial segments of the first layer. However, according to a further subset of the set of possible values of the short-term syntax element, there are boundaries between spatial segments of the pictures of the second layer that overlap each boundary of the spatial segments of the first layer. The latter subset is only used if the long-term syntax element indicates that the tile boundary lines between the base layer and the enhancement layer are aligned.

[0080] Additionally or alternatively, the video decoder 600 can use or employ the fact that the long-term syntax element structure assumes a value outside of the first set of possible values to perform a trial (i.e. to attempt to perform) a full decoding of layer 1 and to refrain from performing this trial if the long-term syntax element structure 606 assumes a value outside of the second set of possible values. In this case, especially for battery driven devices, valuable computational power is saved in case the result or success rate of decoding the enhancement layer 1 in time (i.e. in real-time) is speculative. It is worth mentioning that the refraining can also be based on the hierarchical level indicator as described below with respect to the fourth aspect.

[0081] Although the above is exemplarily described using tiles as spatial segments FIG. 4However, it is clear that the video decoder 600 can exploit the advantages of the long-term syntax element structure and the resulting guarantees issued thereby in connection with other spatial segments such as substreams or slices. In the former case, the video decoder 600 will decode the pictures 12 and 15 of the layer using intra-picture spatial prediction, and will decode spatial segments of the pictures of the first layer 12 in parallel, and support intra-picture spatial prediction across the boundaries of the spatial segments of the pictures of the first layer, and be subject to a decoding delay between the decoding of these spatial segments, i.e. substreams. As mentioned above, the substreams can correspond to horizontal slices of the respective pictures, i.e. vertically subdividing the respective pictures. In decoding each substream, the video decoder 600 can use a decoding order that typically leads from left to right, and the decoding order defined between the substreams of a picture can lead from top to bottom. According to the typical spatial prediction concept used, spatial prediction is performed from the top and left-hand side decoded portions of the current picture, subject to a certain decoding delay between the subsequent substreams immediately following, thereby allowing for parallel decoding of the substreams. The decoding delay can be measured, for example, within the unit of LCUs. This can be done in the pictures 12 of layer 0 as well as in the pictures 15 of layer 1. Thus, the parallelism in decoding the video data stream can comprise two kinds of parallelism individually within the pictures 12 and 15, but also parallel decoding of substreams of the pictures 12 and 15 belonging to different layers of one time frame 604. The parallelism can comprise an adaptivity of the entropy probabilities in the decoding process of the respective substream as far as optional entropy decoding of the substream is involved. The first substream in the substream order of each picture 12 or 15 can be subjected to a separate initialization of the entropy probabilities, independent of the other substreams. Any subsequent substream can be subjected to an initialization of the entropy probabilities according to the immediate adaptation of the entropy probabilities of the immediately preceding substream in the decoding order of the same picture, such as by adopting the entropy probabilities employed in the decoding process of the immediately preceding substream at a certain distance from the left-hand side of the respective preceding substream, such as after decoding two LCUs of the immediately preceding substream, etc.

[0082] Even in the case of wavefront parallel processing substreams, the video decoder 600 can exploit the advantages of the long-term syntax element structure 606: if a guarantee signal is issued by the syntax element structure 606, the video decoder 600 can rely on the fact that all boundaries between consecutive / adjacent substreams of a base layer picture 12 overlap with the respective boundaries between adjacent / consecutive substreams of a time-aligned enhancement layer picture 15 within a predetermined time period 608. That is, a base layer substream locally coincides with a respective enhancement layer substream of a time-aligned enhancement layer picture 15, or the base layer substream exactly corresponds to two or more substreams of a time-aligned enhancement layer picture. Thus, if the guarantee applies, the decoder 600 learns that after completing the decoding of a first substream of a base layer picture 12, the decoding of a first substream of a time-aligned enhancement layer picture 15 can be started as soon as possible.

[0083] As mentioned above, different from tile subdivision, the short-term syntax elements 602 can be chosen such that they define the position of substreams in the pictures 12 and 15 related to the subdivision of these pictures into some coding blocks, such as LCUs. Thus, a substream can be a collective term for one or more rows of coding blocks. As in the case of tile subdivision, the time interval 604 can be such that the short-term syntax elements 602 signal the subdivision of the pictures 12 and 15 into substreams on a per-picture basis, i.e. on a per-picture frame 604 basis. If the long-term syntax element structure 606 does not provide this guarantee, however, the video decoder 600 can attempt to decode substreams of different layers of a common event frame in parallel, but, in order to do so, the video decoder 600 needs to check the short-term syntax elements 602.

[0084] As in the case of using tiles as spatial segments, the video decoder 600 can perform a speculative trial of decoding the enhancement layer 1 according to the values assumed by the long-term syntax element structure 606.

[0085] It is noted that, as shown in FIG. 1 regarding the video encoder side, the insertion and setting of the long-term syntax element structure 606 into the data stream 40 and the decision whether or not to grant the decoder 600 the guarantee can be implemented accordingly by the video encoder. If granted, the coding probability constraints are imposed when the short-term syntax elements 602 are set to comply with the boundary alignment guarantee for the corresponding predetermined time period 608. If not granted, the encoder retains the freedom to set the short-term syntax elements 602 as it is used to do within the time period 608. When using tiles as spatial segments, the encoder complies with the following constraints: spatial prediction cannot cross the boundary line, and the optional entropy coding of the tiles of the pictures 12 and 15 is performed in a self-contained manner for each tile. For example, the entropy probabilities are reinitialized for a tile that is not interdependent with other tiles (for each tile). If a substream, the entropy probability initialization for a substream is performed anew for any first substream of the respective picture 12, 15, i.e. independently of any other substream, and, as far as any second and subsequent substream is concerned, the entropy probabilities are adopted in adaptation to the middle position of the immediately preceding substream. Spatial prediction is performed without any constraints with respect to the crossing of substreams.

[0086] By referring to FIG. 4 the description described in the following manner, i.e. in the manner described in the following, the alignment concept can be introduced into the HEVC standard as currently envisaged extension. So far, the description to be presented in the following should also be interpreted as being based on the description presented above with reference to FIG. 4 possible implementation details.

[0087] HEVC allows to divide the CTBs of a coded base layer picture into rectangular regions called slices through a grid of vertical and horizontal boundary lines and can be processed individually (except for in-loop filtering). At slice boundary lines, in-loop filtering can be turned off so that it is completely independent.

[0088] As at picture boundary lines, the parsing and prediction interdependencies are broken at slice boundary lines, where in-loop filters can cross the boundary lines, if configured accordingly, to reduce boundary line artifacts. Thus, the processing of an individual slice is not completely dependent on other slices within the picture or on a wide range of filtering configurations. The limitation imposed is that all CTBs of a slice shall belong to the same slice or all CTBs of a slice shall belong to the same slice. From FIG. 1 As can be seen from the above, slices force the CTB scan order to follow the slice order, i.e. all CTBs belonging to a first slice (e.g. the top-left slice) are traversed before proceeding to the CTBs belonging to a second slice (e.g. the top-right slice). The structure is limited by the number and size of CTBs within each slice row and each slice column constituting the grid within a picture. The structure can change on a per frame basis or remain constant within a coded video sequence.

[0089] FIG. 5 An exemplary division of CTBs within a picture into nine slices is shown. The thick black lines represent the boundary lines and the numbering represents the scan order of the CTBs, also reflecting the slice order.

[0090] As long as all slices covering the corresponding image region in the base layer bitstream are decoded, the enhancement layer slices of the HEVC extension can be decoded.

[0091] The following section describes constraints, signaling and decoding process modifications allowing easier access to base layer information using the concepts in FIG. 4

[0092] The simplest case regarding slice level parallelism is that the slice boundary lines in the base layer and the enhancement layer are aligned. For SNR scalability, this means that the boundary lines are located exactly at the same positions. For spatial scalability, this means that for each two enhancement layer samples belonging to the same slice, the corresponding base layer sample also belongs to the same slice and for each two base layer samples belonging to the same slice, the corresponding enhancement layer sample also belongs to the same slice.

[0093] HEVC is characterized by a short-term signaling corresponding to 602 in FIG. 4 based on column_width_minusl[i] and row_height_minusl[i] from [1] indicating the dimensions and structure of the slices within a picture on a per picture basis by using the following picture parameter set: FIG. 6 An exemplary syntax is shown.

[0094] ​Further features of HEVC are that there is a restriction signaling on certain settings of a HEVC coded video sequence, for example, indicating a fixed tile structure in a single-layer HEVC coded video sequence (cp. tiles_fixed_structure_flag in the VUI syntax given below). Further restrictions on tiles in a scalable coded video sequence are beneficial for decoder initialization and operation. In order to allow the decoder to start decoding the enhancement layer image region associated with an active layer tile after finishing the base layer tile, it is not necessary to enforce a complete alignment. It can be beneficial to allow more tile decoding in the enhancement layer than in the base layer, especially in the case of spatial scalability. For example, in the case of two spatial scalability factors, the enhancement layer image region contains four times the number of samples compared to the corresponding base layer image region. Therefore, it can be beneficial to allow four tile decoding in the enhancement layer for each base layer tile. See FIG. 7 , FIG. 7 An embodiment of aligned tile boundaries with spatial scalability is shown. All vertical boundaries in the base layer and the enhancement layer are aligned. In the enhancement layer, additional tiles (horizontal boundaries) are used to allow the same number of samples in parallel in each enhancement layer as when using base layer tile partitioning.

[0095] Therefore, we restrict the tile boundary alignment in the following way: only each base layer boundary has a corresponding boundary in the enhancement layer, but not vice versa. Precisely, this means that for each two enhancement layer samples belonging to the same tile, the corresponding base layer sample also belongs to the same tile.

[0096] The signaling 606 facilitates the initialization of a parallel decoder environment, otherwise the signaling 606 would have to gather the information by parsing a number of parameter sets. Moreover, for example in the case of bitstream restriction forms, FIG. 4 The concept in

[0097] If the tile boundaries of the base layer are a subset of the tile boundaries of the enhancement layer, the possible implementation allows to save the signaling of the base layer tile boundaries in the enhancement layer.

[0098] Precisely, the information about the tile alignment can be signaled in the bitstream which is easily accessible by the decoder.

[0099] In a specific implementation, the signaling can be achieved by using a flag in the VUI parameters of the enhancement layer SPS as given in FIG. 8

[0100] ​When present, tiles fixed structure flag equal to 1 indicates that each picture parameter set (i.e., active in the coded video sequence) has the same values for the syntax elements num tile columns minusl, num tile rows minusl, uniform spacing flag, column width minusl [i], row height minusl [i], and loop filter across tiles enabled flag. tiles fixed structure flag equal to 0 indicates that the tile syntax elements in different picture parameter sets can or can not have the same values. When the tiles fixed structure flag syntax element is not present, it is inferred to be equal to 0.

[0101] It should be noted that the signaling of tiles fixed structure flag equal to 1 guarantees to the decoder that each picture in the coded video sequence has the same number of tiles distributed in the same way, which can help with workload allocation in case of multi-threaded decoding.

[0102] tile boundaries aligned flag corresponds to structure 606 in FIG. 4 If tile boundaries aligned flag is equal to 1, it indicates that all tile boundaries of the corresponding base layer picture have corresponding tile boundaries in the given enhancement layer. tile boundaries aligned flag equal to 0 indicates that there is no restriction between the tile configuration of the corresponding base layer and the given enhancement layer.

[0103] It should be noted that the long-term syntax element structure can guarantee that the minimum number of spatial segments 82 into which a picture 15 of the second layer is subdivided (e.g., picture sequence) is n times the minimum number of spatial segments 80 into which a picture 12 of the first layer is subdivided, or that each spatial segment of picture 12 is exactly composed of n spatial segments of picture 15 that are time-aligned, and n depends on the value of the long-term syntax element structure. In FIG. 7In the case of n = 3, the decoder still determines on a periodic basis the actual subdivision of the pictures 12 of the first layer and 15 of the second layer into spatial segments 80 and 82, respectively, based on the short-term syntax elements of the multi-layer video data stream 40 within a time interval shorter than the predetermined time period, but, again, the decoder can exploit the guarantee to more efficiently perform the workload distribution. The other aspect is "opportunistic decoding": a decoder with multiple CPU cores can exploit the guarantee as an implication on the parallelism of the layers and decide accordingly to decode the layer with higher complexity, e.g. higher spatial resolution or higher number of layers. By utilizing all cores of the same decoder, bitstreams exceeding the capacity of a single core can be decoded. This information is particularly helpful if the profile and level indicators do not contain an indication on the minimum parallelism.

[0104] The second aspect discussed and proposed in the following relates to the concept of "restricted inter-layer up-sampling": if the multi-layer video is spatially scalable, the up-sampling filter 36 is adjusted using syntax elements in the bitstream, e.g. exemplary independent_tile_upsampling_idc. If the up-sampling filtering is performed in layer 0 across the spatial segment boundary line 86, the delay increase with respect to the encoding / decoding of the spatial segments 80 of layer 0 is met by the parallel decoding / encoding of the spatial segments 82 of layer 1 due to the combination of the up-sampling filtering and, thereby, the inter-dependency of the information of the adjacent spatial segments of layer 0 being used as prediction reference 38 for the inter-layer prediction of the blocks 41 of layer 1. See, e.g. FIG. 9Fig. 12 and Fig. 15 are shown in an overlapping fashion, and the two pictures are registered to each other according to spatial correspondence and are dimensionally consistent with each other, i.e., portions showing the same scene portion overlap each other. Fig. 12 and Fig. 15 are shown as being partitioned into 6 spatial segments (such as tiles) and 12 spatial segments, respectively. The filter kernel 200 is shown as moving over the top-left tile of Fig. 12 to obtain an upsampled pattern thereon, which is used as a basis for any block within the tile of Fig. 15 (spatially overlapping with the top-left tile) that is inter-predicted. In some intermediate instances, such as 202, the kernel 200 overlaps with neighboring tiles of Fig. 12. Thus, the sample value in the middle of the kernel 200 at the location 202 depends on samples of the top-left tile of Fig. 12 as well as samples of the tile to the right of the top-left tile of Fig. 12. If the upsampled pattern of Fig. 12 is used as a basis for inter-prediction, the inter-layer offset in terms of parallel processing of segments of a layer is increased. Thus, limiting can help increase the amount of parallelism between different layers and correspondingly reduce the overall encoding delay. Naturally, the syntax element can also be a long-term syntax element valid for a sequence of pictures. The limiting can be achieved in one of the following ways: filtering the overlapping portion of the kernel 200 at the overlapping location 202, e.g., with a center-tendency of the sample values being within the non-dashed portion of the kernel 200; extrapolating the non-dashed portion into the dashed portion using a linear function or other function; etc.

[0105] For the sake of clarity of the following aspects, reference is made to FIG. 10 , FIG. 10 A decoder 610 is shown that receives a spatially scalable bitstream 40 of pictures 12 in a layer 0 and pictures 15 in a layer 1 (the pictures are encoded into the spatially scalable bitstream 40). The decoder 610 is configured to decode the spatial layer of the spatial segments for at least one of the spatial layers. As mentioned above, the spatial segments can be tiles or slices or segments. Also, the decoder 610 can be configured to subject the spatial segments of the pictures 12 or 15 to parallel decoding. That is, the base layer pictures 12 can be subdivided into spatial segments, such as tiles and / or slices and / or segments, and / or the enhancement layer pictures 15 can be subdivided into tiles and / or slices and / or segments. For details regarding parallel processing, reference is made to the description in FIG. 4 , which has been transferred to FIG. 10decoder 610. That is, for example, when decoding the base layer picture 12, the decoder 610 uses spatial prediction and optionally temporal prediction if the base layer 12 and 15 are part of a layered video. If a tile, the spatial prediction is restricted to not cross tile boundary lines and the spatial prediction applies to entropy decoding (if used) that is performed completely separately for the tile. The spatial prediction applies to the enhancement layer picture 15 and additionally inter-layer prediction is supported. As already described above, the inter-layer prediction not only predicts the prediction parameters of the enhancement layer based on the corresponding prediction parameters used when decoding the base layer, but also predicts the prediction parameters of the enhancement layer based on a prediction derived from reconstructed samples of the base layer picture that is located in a co-located portion with respect to the portion of the enhancement layer picture 15 that is currently predicted using inter-layer prediction. However, since the bitstream 40 can be a spatially scalable bitstream, any co-located portion of the base layer picture 12 that forms the basis for the inter-layer prediction of the currently processed portion of the enhancement layer picture 15 can be up-sampled by the decoder 610 to account for the higher spatial resolution of the picture 15 with respect to the picture 12. See, for example, FIG. 11 , FIG. 11 A reference symbol 612 is used to indicate the currently predicted portion of the enhancement layer picture 15. A reference symbol 614 is used to indicate the co-located portion in the base layer picture 12. Due to the higher spatial resolution of the enhancement layer picture 15, the number of sample positions within the portion 612 (indicated by the dots) is shown to be higher than the samples within the portion 614 (also indicated by the dots). Thus, the decoder 610 up-samples the reconstructed version of the portion 614 of the base layer picture 12 using an interpolation method. As such, FIG. 10 The decoder 610 in FIG. 6 accordingly spatially scalable bitstream 40 in syntax elements 616.

[0106] In particular, reference is made to FIG. 12 The just mentioned responsiveness is explained in more detail. FIG. 12 The portion 614 within the base layer picture 12 (i.e., the reference portion of its non-up-sampled version) as well as its corresponding up-sampled version using a reference symbol 618 is shown. As just mentioned, the version 618 for later use in inter-layer prediction can be obtained from the base layer picture 12 by an interpolation method 620, for example, by copying the respective up-sampled sample values in the portion 618 into the portion 612 of the enhancement layer 15. However, the interpolation method 620 depends on the just mentioned syntax elements 616. The interpolation method 620 changes depending on the syntax elements 616 in a manner that is related to the area along the boundary between adjacent partitions of the base layer picture 12 and its up-sampled version, respectively. In particular, the partitions depend on the above described spatial segments into which at least one of the pictures 12 and 15 is subdivided. In FIG. 11In the example of FIG. 6, the partitioning within base layer picture 12 is illustrated using dashed lines 622. For example, as outlined in more detail below, partitioning 622 can correspond to a logical AND or a logical OR combination of the spatial overlap of the boundaries of the spatial segments of picture 12 and picture 15, or can coincide spatially with the partitioning defined by the spatial segments of enhancement layer picture 15. In any case, decoder 610 performs interpolation 620 according to syntax element 616, regardless of or taking into account partitioning 622. When taking into account partitioning 622, decoder 610 performs interpolation 620 such that all samples within up-sampling portion 618 originate from, or depend on, or are influenced by, only the samples of one of the partitions of partitioning 622, and not any of the other partitions of partitioning 622. For example, if partitioning 622 is a local AND combination or a local OR combination of the boundaries of the spatial segments of picture 12 and picture 15, then all samples of interpolation portion 618 originate from only one partition of base layer picture 12. However, if syntax element 616 directs decoder 610 to be insensitive to partitioning 622, then it can happen that different samples within interpolation portion 618 originate from adjacent partitions of partitioning 622.

[0107] For example, 612 illustrates the case where interpolation 620 is performed using filter kernel 200, and in order to obtain the FIG. 12 In the example of FIG. 6, the partitioning within base layer picture 12 is illustrated using dashed lines 622. For example, as outlined in more detail below, partitioning 622 can correspond to a logical AND or a logical OR combination of the spatial overlap of the boundaries of the spatial segments of picture 12 and picture 15, or can coincide spatially with the partitioning defined by the spatial segments of enhancement layer picture 15. In any case, decoder 610 performs interpolation 620 according to syntax element 616, regardless of or taking into account partitioning 622. When taking into account partitioning 622, decoder 610 performs interpolation 620 such that all samples within up-sampling portion 618 originate from, or depend on, or are influenced by, only the samples of one of the partitions of partitioning 622, and not any of the other partitions of partitioning 622. For example, if partitioning 622 is a local AND combination or a local OR combination of the boundaries of the spatial segments of picture 12 and picture 15, then all samples of interpolation portion 618 originate from only one partition of base layer picture 12. However, if syntax element 616 directs decoder 610 to be insensitive to partitioning 622, then it can happen that different samples within interpolation portion 618 originate from adjacent partitions of partitioning 622. FIG. 12 In the example of FIG. 6, the partitioning within base layer picture 12 is illustrated using dashed lines 622. For example, as outlined in more detail below, partitioning 622 can correspond to a logical AND or a logical OR combination of the spatial overlap of the boundaries of the spatial segments of picture 12 and picture 15, or can coincide spatially with the partitioning defined by the spatial segments of enhancement layer picture 15. In any case, decoder 610 performs interpolation 620 according to syntax element 616, regardless of or taking into account partitioning 622. When taking into account partitioning 622, decoder 610 performs interpolation 620 such that all samples within up-sampling portion 618 originate from, or depend on, or are influenced by, only the samples of one of the partitions of partitioning 622, and not any of the other partitions of partitioning 622. For example, if partitioning 622 is a local AND combination or a local OR combination of the boundaries of the spatial segments of picture 12 and picture 15, then all samples of interpolation portion 618 originate from only one partition of base layer picture 12. However, if syntax element 616 directs decoder 610 to be insensitive to partitioning 622, then it can happen that different samples within interpolation portion 618 originate from adjacent partitions of partitioning 622.

[0108] According to an embodiment of the application, the partitions 622 are selected such that they coincide with the spatial segmentation of the base layer picture, independently of the arbitrary spatial segmentation of the enhancement layer picture. Thus, the inter-layer offset between the base layer picture 12 and the enhancement layer picture 15 can be reduced by the decoder 610, because parts such as the part 614 of the base layer picture 12 do not require the decoder 610 to decode neighboring partitions / spatial segments before performing the inter-layer prediction of the part 612 of the enhancement layer picture 15.

[0109] Alternatively, the decoder 610 can be configured to determine that the partitions 622 coincide locally with the spatial segmentation of the picture 15. Yet another alternative, the decoder 610 can be configured to select the partitions 622 consisting only of the boundaries of the spatial segments of the picture 12 and the picture 15, which are spatially coinciding, i.e. in a local AND corresponding to the boundaries of the picture 12 and the picture 15, in other words, only these boundaries of the picture 15 are segmented into the form of spatial segments corresponding to the boundaries between the partitions of the partitions 622, i.e. spatially corresponding to the respective boundaries of the base layer picture 12 are segmented into spatial segments.

[0110] It is also possible that the syntax element 616 can guide the decoder 610 not only to ignore the partitions 622 in the interpolation 620, but also to distinguish between different ways of selecting the partitions 622, which will be outlined in more detail below. For example, see FIG. 9 wherein, exemplarily, tiles are used as spatial segments. For example, if the syntax element 616 signals to the decoder 610 to perform the interpolation 620 for the partitions 622 separately, the decoder 610 can use the boundaries of the base layer picture 12 as boundaries of the partitions 622, because the partitions 622 coincide with the fine segmentation of the enhancement layer picture 15 into tiles. Thus, in order to start decoding the second tile in the topmost tile row of the enhancement layer picture 15, the decoder 610 does not need to wait for the completion of the decoding of the second tile in the topmost row of the base layer picture 12, because the interpolation separation prohibits any mixing of the reconstructed samples of the first two tiles in the topmost row of the base layer picture 12. If the samples completely segment the enhancement layer picture 15 into tiles, in order to determine the partitions 622, also the interpolation is performed at the dashed line in FIG. 9 and the decoder 610 is able to start decoding the top-left tile of the enhancement layer picture 15 even earlier, i.e. the decoder 610 manages the decoding of the respective co-located sub-part of the first tile of the base layer picture 12 as soon as possible. In view of this, it should be noted that even when decoding the tiles, the decoder 610 can use a certain decoding order, for example, which contains a certain raster scan order leading from the top-left corner of the respective tile to the bottom-right corner of the respective tile in a row-wise manner.

[0111] That is, according to the second aspect, the encoder forming the bitstream 40 is able to select between two modes by means of the syntax element 616: if the syntax element 616 is set and inserted into the bitstream 40, the decoder 610 is directed to be insensitive to the partition 622, a better inter-layer prediction can be achieved due to the better interpolation, but the degree of parallelism available when decoding the picture 12 and the picture 15 in parallel is reduced, i.e. the minimum inter-layer decoding offset is increased. In the other mode, the syntax element 616 directs the decoder 610 to take the partition 622 into account when performing the interpolation 620 for the purpose of inter-layer prediction, and thus, the inter-layer prediction quality is reduced, favoring an increased degree of parallelism and a reduced minimum inter-layer decoding offset when decoding the picture 12 and the picture 15 in parallel, respectively.

[0112] Although the description of the second aspect of the present application mainly focuses on the tile subdivision or tile parallel processing concept, it is clear that the use of the syntax element 616 to control the interpolation 620 is also advantageous when using a WPP bitstream. See, for example, FIG. 13 , FIG. 13 The following case is shown: the base layer picture 12 is exemplarily subdivided into two bitstreams, wherein the co-located portions of the enhancement layer picture 15 are each subdivided into two bitstreams. If the interpolation separation is imposed in response to the syntax element 616, the decoder 610 can start decoding the first, i.e. the highest, bitstream of the enhancement layer picture 15, once the decoder 610 has decoded the first bitstream of the base layer picture 12, the response top-left portion of the first enhancement layer bitstream of the picture 15 is sufficiently far covered, and because the interpolation separation makes any inter-layer prediction independent from any reconstructed portion of the base layer bitstream of the picture 12 that spatially overlaps with the second enhancement layer bitstream, this is equally true even for those portions of the first enhancement layer bitstream of the picture 15 that overlap with the boundary of the second bitstream of the picture 15.

[0113] A detailed implementation of the switchable restricted inter-layer upsampling summarized above is provided in the following, it should be noted that, for example, as in the case of FIG. 4 , the syntax element 616 can signal or switch the restriction within each time frame if the picture 12 and the picture 15 are time-aligned paired pictures video. Also, again, it should be noted that the decoder according to the implementation of the present application can be consistent with the description and functionality provided above with reference to FIG. 4 and FIG. 10 . In view thereof, it should be noted that the signaling of the short-term syntax element and the position and location of the spatial segment of the picture 12 and the picture 15 provided above with reference to FIG. 4 should be considered to equally apply to the implementation described with reference to FIG. 10 to FIG. 13 . Finally, it should be noted that, if FIG. 10The second aspect is also advantageous in that the decoder in the decoder is a picture decoder that decodes pictures in layer 0 and layer 1. The temporal component is optional.

[0114] The implementation of restricted inter-layer upsampling in HEVC is done as follows. For spatial scalability, an enhancement layer picture is predicted using an upsampled base layer picture. In this process, the prediction value for each sample position of the enhancement layer is calculated using a number of sample values (typically in the horizontal and vertical direction) of the corresponding base layer image region. If samples from different base layer tiles are used, the enhancement layer tile cannot be decoded from the base layer tile information only covering the same image region as the enhancement layer tile. The signaling of restricted inter-layer upsampling as a bitstream restriction guarantees that the decoder complies with the signaled spatial partitioning in all parameter sets of the coded video sequence and thereby simplifies the initialization and operation of parallel inter-layer decoders.

[0115] FIG. 10 The concept in the decoder can be implemented as a mechanism that does not allow the use of neighboring sample information for upsampling that is not contained in the base layer tile corresponding to the enhancement layer tile. In the bitstream, it is signaled whether the decoder is allowed to use samples that are outside the corresponding image region of the base layer at all enhancement layer tile boundaries.

[0116] Alternatively, in the bitstream, it is signaled whether the decoder is allowed to use samples that are outside the corresponding image region of the base layer at the enhancement layer tile boundaries only for the enhancement layer tile boundaries that correspond to base layer tile boundaries.

[0117] In a specific implementation, the upsampling of the base layer at the tile boundary line is performed as if the base layer is located on a picture boundary where no neighboring samples are available.

[0118] In a specific implementation, the signaling is achieved by using a flag in the picture parameter set of the enhancement layer as given in FIG. 14

[0119] independent_tile_upsampling_idc corresponds to syntax element 612, independent_tile_upsampling_idc not equal to 0 restricts the upsample filter from crossing tile boundary lines. If independent_tile_upsampling_idc is equal to 2, any base layer samples that are outside the picture region corresponding to the enhancement layer tile are not available for upsampling. If independent_tile_upsampling_idc is equal to 1, this restriction only applies to the enhancement layer tile boundaries that are aligned with base layer tile boundaries. independent_tile_upsampling_idc equal to 0 does not imply this restriction.​

[0120] Finally, FIG. 15A Pictures 12 and 15 are shown exemplarily overlaid by spatially corresponding ways to illustrate FIG. 14 An embodiment of syntax element 612 in

[0121] As an alternative to the mode independent_tile_upsampling_idc = 2, or as an additional mode to modes such as independent_tile_upsampling_idc = 3, the upsampling filter can be restricted from crossing any tile boundary line, i.e., neither of the base layer nor of the enhancement layer. See line 404 in FIG. 15B

[0122] That is, as explained above with reference to FIG. 9 , according to this mode, the upsampling filter is processed at the boundaries 400, 402, or 404.

[0123] Before turning to the next aspect of the present application, it should be briefly noted that the interpolation 620 discussed above will be performed, for example, in the predictor 60 of FIG. 2 . Similarly at the encoder side (such as within the predictor 18), since the encoder performs the same prediction at the encoding side, the information 620 is performed according to the setting of the syntax element 616. For example, at the encoding side, the decision whether to set the syntax element can be performed according to the application scenario. For example, if the degree of latency is a high important factor, the syntax element can be set to restrict the inter-layer upsampling, and in other application scenarios, having a better prediction and increasing the compression ratio can be more important, so that it is more advisable to set the syntax element 612 to not restrict the inter-layer upsampling.

[0124] ​The minimum coding delay or offset between the encoding of the just mentioned spatial segments of consecutive layers is the subject of the next aspect and can be referred to as "layer decoding delay indication". The decoder is able to determine the minimum decoding delay or offset between the encoding of the spatial segments of picture 15 with respect to the spatial segments of picture 12 based on the short-term syntax elements, but according to the next concept, the inter-layer delay or offset is signaled in advance for a predetermined time period using the long-term syntax element structure. Again, this facilitates the workload distribution within the parallel decoding of the bitstream 40 by the decoder. As a measure for "delay" or "offset", the spatial segment can be used, i.e. the offset can be expressed within the unit of a spatial segment (tile, slice, or CTB row of WPP).

[0125] For a more detailed description of the following aspects, reference is made to the FIG. 4 FIG. 16 described above with reference to the FIG. 16 . Thus, FIG. 4 the same reference signs are used in FIG. 16 and the description presented above with reference to the FIG. 16 . It should also be mentioned that the video decoder 640 shown in FIG. 4 integrates the functionality described above with reference to the long-term syntax elements 606. At present, this aspect of the FIG. 16 application also uses the long-term syntax element structure, i.e. 642, which is also inserted into the bitstream 40 to refer to or relate to the predetermined time period 608. In other words, although the video decoder 640 can respond to both the syntax element structure 606 in FIG. 4 and the syntax element structure 642, however, only the latter functionality, which is further outlined below, is relevant for FIG. 16 ​The decoder 640 is of particular importance, where the functionality related to the syntax element structure 606 and the existence of the syntax element structure 606 in the bitstream 40 are optional for the video decoder 640. However, the description above with reference to the video decoder 600 also applies to the video decoder 640. That is, the video decoder 640 is capable of decoding multi-layer video data stream 40, i.e., using inter-layer predictions from the first layer (layer 0) to the second layer (layer 1) to encode the scene in the layer hierarchy. The video decoder 40 supports parallel decoding of multi-layer video data streams into spatial segments, i.e., partitioning the images of a layer into spatial segments by sequentially traversing the spatial segments in a temporally overlapping manner, with an inter-layer offset between the spatial segments of the first layer image traversed and the spatial segments of the second layer image traversed. It should be noted that this means that a spatial segment can be a slice, substream, or piece; however, even a mixture of the segment units just mentioned is possible. In fact, the definition of "spatial segment" can differ when combining the concept of a slice with the concept of a piece and / or the concept of a substream.

[0126] In any case, regarding images 12 and 15 of the common time frame 604, FIG. 16 The video decoder 640 is capable of decoding spatial segments of image 12 in parallel with spatial segments of image 15 on the other hand (i.e., by temporal overlap). Naturally, therefore, due to inter-layer prediction, the video decoder 640 needs to conform to a certain minimum decoding offset between the two layers, and the current decoding portion of enhancement layer 1 in image 15 must belong to the temporally aligned decoded portion of image 12 in layer 0.

[0127] in the case of FIG. 16 The video decoder 640 uses the long-term syntax element structure 642 to predetermine the interlayer offset for a predetermined time period 608.

[0128] Combination FIG. 16 In the implementation method described, the interlayer offset is related to one aspect. FIG. 12 The "distance" of the first spatial segment is a scalar measurement of the time-aligned image 15 on the other hand. Preferably, the spatial "distance" is measured. Moreover, for this to be meaningful, the inter-layer offset determined based on the long-term syntax element structure 642 should be valid for the entire decoding process of the first spatial segment of image 12. That is, all necessary reference portions of image 12 used for inter-layer prediction can be used for decoding the entire first spatial segment of image 15, provided that the first "inter-layer offset" spatial segment of the base layer image 12 has been decoded previously.

[0129] As described above, the "current decoding portion" within image 15 traverses image 15 in a specific predetermined manner; that is, if slice-parallel processing is used, it follows the slice order described above, and in the case of using the WPP concept of bitstream, it is done in the form of a tilted wavefront. The same applies to the spatial segments of the base layer image 12. Before the first spatial segment of image 15 undergoes initial decoding, the interlayer offset determines the traversed portion of image 12 that has already been processed.

[0130] For a more detailed description, please refer to [reference needed]. FIG. 17A and FIG. 17B . FIG. 17A The interlayer offsets related to the concept of slice, as determined from the long-term grammar element structure 642, are described in more detail. FIG. 17B The inter-layer offsets related to WPP, determined based on the Long Term Syntax Element Structure 642, are described in more detail below. This will be discussed in conjunction with... FIG. 17C The concept of inter-layer offset signaling using Long Syntax Element Structure 642 is shown to be unrestricted by the slice and / or WPP concepts used. More precisely, the interpretation of inter-layer offsets based on Long Syntax Element 642 is made feasible by defining that only the picture is subdivided into slices that are decodeable in a self-contained manner (i.e., entropy decoding and spatial intra-slice picture prediction are performed "in-slice" or entropy decoding and spatial intra-slice picture prediction are independent of adjacent slices).

[0131] FIG. 17A Two time-aligned images, 12 and 15, are shown, both subdivided into slices. From the description of the slice concept presented above, it becomes clear that, typically, there is an arbitrary fixed order within the slice of image 12 or image 15 that is being decoded. More precisely, the slices can be decoded in any order. However, combining... FIG. 16 In the implementation method described, at least relative to the sheet order of the base layer image 12, the sheet order 644 is defined as a row raster scan order guiding from the upper left sheet in a regular arrangement of sheets to the lower right sheet. According to... FIG. 17A In the implementation described, the inter-layer offset signal emitted by the long-term syntax element structure 642 indicates the number of slices that have been decoded according to the slice order 644 of the base layer image 12, allowing the decoder 640 to begin decoding the first slice of the enhancement layer image 15. To determine the "first slice" among the slices of the enhancement layer image 15, the first slice of the enhancement layer image 15 can be fixedly defined as the uppermost slice of the enhancement layer image 15. From the first slice to the enhancement layer image 15, the video decoder 640 can employ a slice order for traversing the enhancement layer slices of image 15 according to the slice subdivision of image 12. FIG. 17AIn some cases, such as when image 12 is subdivided into two rows and three columns, and image 15 is subdivided into four rows and two columns, it is advantageous for decoder 640 to select the order in which to traverse the enhancement layer slices, first traversing the left-hand slices of the first two rows, then traversing the right-hand slices of the first two rows, and then repeating the traversal for the lower rows of enhancement layer image 15, as indicated by arrow 646. However, according to alternative embodiments effective for all aspects described herein, the slice decoding order between enhancement layer slices of image 15 is fixed and independent of the subdivision of the base layer image. In summary, if only the signal layer coding offset is used as a trigger for starting / commencing decoding of the enhancement layer image, the recorder is unnecessary. FIG. 17A The dashed lines indicate the local location in image 12 corresponding to the first piece of the enhancement layer image 15. FIG. 17A It can become clear in the middle, in FIG. 17A In the exemplary case, the interlayer offset determined by the long-term syntax element structure 642 can be "2" because the first two slices of image 12 must be decoded before the decoder 640 begins decoding the first slice of enhancement layer image 15. In this case, the co-location positions required for interlayer prediction can only be used in the base layer image 12.

[0132] That is, in FIG. 17A In this case, the video decoder 640 will determine from the long syntax element structure 642 that the interlayer offset between the base layer of image 12 and the first layer of the enhancement layer of image 15 is two base layers: before the video decoder 640 starts decoding the first layer of the enhancement layer of image 15 in slice order 646, it has to wait for the first two base layers to be decoded in slice order 644.

[0133] FIG. 17B An exemplary case involves subdividing two time-aligned images 12 and 15 into substreams, namely, in FIG. 12 In the case of Figure 12, the subdivision is divided into two substreams, and in the case of Figure 15, it is divided into four substreams. For example, the substreams can be consistent with the rows and columns of the coded blocks that regularly subdivide Figures 12 and 15 in the manner described above, that is, in such a way that each substream corresponds to a row of such a coded block. In any case, as described above, due to WPP processing, a decoding order is defined between the substreams of Figure 12 and Figure 15, with decoding order 648 and decoding order 650 guided from top to bottom. FIG. 17AThe case is similar in the case of FIG. 6, the decoder 640 being configured to determine from the long-term syntax element structure 642 the number of leading sub-streams that have been decoded before starting decoding the first sub-stream of the picture 15. In this case, the long-term syntax element structure 642 will signal a layer offset of 1, since the full decoding of the first sub-stream of the base layer picture 12 is sufficient to provide the necessary basis for any inter-layer prediction of the first sub-stream of the enhancement layer picture 15.

[0134] FIG. 17C A pair of time-aligned pictures 12 and 15, both being subdivided into slices, is shown. Again, a slice order or decoding order is defined between the slices of the picture 12 and between the slices of the picture 15, respectively, the orders 652 and 654 leading from top to bottom. In the example case of FIG. 6, the picture 12 is a base layer picture and the picture 15 is an enhancement layer picture. The picture 12 is decoded first, the picture 15 being decoded after the picture 12 has been completely decoded. FIG. 17C In the exemplary case of FIG. 6, the boundaries between the slices within the picture 12 of one aspect and the slices within the picture 15 of the other aspect locally correspond to each other. Thus, due to the "fuzziness" introduced by the inter-layer prediction from the base layer picture 12 to the enhancement layer picture 15, the long-term syntax element structure 642 will signal an inter-layer prediction offset equal to 1 or equal to 2. In particular, for example, due to the upscaling of the corresponding collocated reference portion in the picture 12 for the inter-layer prediction portion of the picture 15, e.g. as discussed above with reference to FIG. 5, the first two slices of the picture 12 can have been decoded in the slice order 652 before the decoder 640 can start decoding the first slice of the enhancement layer picture 15, due to the disparity compensation vector or due to the up-sampling interpolation filter kernel. However, if the fuzziness option of the inter-layer prediction is turned off or not applied, the long-term syntax element structure can be set to equal 1 by the encoder, e.g. since the sampling resolution between the picture 12 and the picture 15 is equal to each other and the picture 12 and the picture 15 are related to the same field of view, so that no disparity compensation occurs, so that the decoder 640 can start decoding the first slice of the enhancement layer picture 15 as soon as possible, before the first slice of the base layer picture 12 is completely decoded. FIG. 9

[0135] Thus, FIG. 16 to FIG. 17C ​The description of the long-term syntax element structure 642 shows that it helps the encoder to support the decoder to schedule time-aligned pictures 12 and pictures 15 in parallel, i.e. by informing the decoder about the long-term syntax element structure regarding the inter-layer offset, which is valid for the whole predetermined time period 608 and is related to the number of spatial segments of the base layer picture 12, outside the first spatial segment of the time-aligned picture 15, the spatial segments of the base layer picture 12 have already been decoded. It should be noted that the video decoder 640 is able to signal the inter-layer offset by the long-term syntax element structure 642 (or even by itself) based on the detection / evaluation of the short-term syntax elements 602 and further syntax uniformity regarding potential options related to inter-layer prediction, which can switch on or off the ambiguity just outlined in the inter-layer prediction from the base layer to the enhancement layer. However, the video decoder 640 would have to detect a plurality of syntax elements to derive the same information provided by the long-term syntax element structure 642 and the video decoder 640 would only be able to derive the same information regarding the long predetermined time period 608 based on the short-term, but not in advance.

[0136] Similar to aspect 1 and aspect 2, possible ways to introduce the delay indication aspect into HEVC are described subsequently in the following.

[0137] First, it is described how WPP is currently implemented in HEVC with reference to FIG. 18 . This description also constitutes the basis for optional implementations of the WPP processing in any of the above-described embodiments.

[0138] In the base layer, the wavefront parallel processing allows for the parallel processing of the coding tree block (CTB) rows. The prediction dependencies across the CTB rows are not broken. With respect to the entropy coding, it can be seen from FIG. 18 that the WPP changes the CABAC dependencies for the top-left CTB in the respective upper layer CTB row. Once the entropy decoding of the corresponding top-right CTB is finished, the entropy coding of the CTBs in the subsequent row can be started.

[0139] In the enhancement layer, the decoding of a CTB can be started as soon as the CTB containing the corresponding image region is fully decoded and available.

[0140] FIG. 16 The decoding delay or offset in can just be a concept that can end in signaling that facilitates the initialization and operation of the decoder for a slice, WPP, or layered bitstream of slices that exploit parallelism.

[0141] When using spatial scalability, the decoding of an enhancement layer CTB can only start when the base layer CTB covering the corresponding image region is available. When using spatial scalability with WPP in parallel to layered bitstreams, the layers can differ in picture size, e.g. picture size scaling factors of 1.5 and 2 between layers are specified in the published call for proposals for scalability extensions to HEVC [1] in addition to the maximum CTB size, e.g. 16, 32, and 64 picture samples are supported in the main profile of HEVC. With respect to quality scalability, the picture size scaling factor is usually constant, however, the maximum CTB size between layers can still differ.

[0142] The ratio between the maximum CTB size and the picture size scaling factor of a layer influences the layer decoding delay, i.e. the first CTB row of an enhancement layer can be decoded with a preceding offset relative to the decoding of the base layer CTB row. FIG. 19 Exemplary parameter values for picture size scaling factors and CTB sizes as well as the introduced layer decoding delay in terms of CTB rows are reported for the ratio of CTBs covering corresponding image regions in two layers.

[0143] For quality scalability between layers, usually the picture size scaling factor between layers is equal to 1, while the maximum CTB size of the respective layers can still differ and influence the layer decoding delay.

[0144] The syntax element structure 642 provides a decoder hint in the bitstream, i.e. a signal about the layer decoding delay for an independent spatial or quality enhancement layer when parallelizing the WPP processing across spatial enhancement layers.

[0145] The implementation uses the picture size scaling factor and the maximum CTB size scaling factor between the respective layers to determine the signaled layer decoding delay.

[0146] Depending on the type of scalability between the independent base layer bitstream and the independent enhancement layer bitstream, the factors influencing the layer decoding delay can differ.

[0147] With respect to view scalability, the layers represent camera views and use the mechanism of inter-layer prediction to perform prediction between camera views from each perspective. This prediction utilizes a motion compensation mechanism to compensate for the different positions of the cameras in the camera setup. In this case, the layer decoding delay is further limited by the maximum or actual motion vector in the vertical direction compared to the case of spatial or quality scalability.

[0148] The syntax element structure 642 describes a decoder hint in the bitstream, i.e. a signal about the layer decoding delay for an independent camera view when parallelizing the WPP processing across multiple camera views.

[0149] Embodiments use a picture size scaling factor, a maximum CTB size scaling factor, and a maximum motion vector length between respective layers in the vertical direction to determine a signal layer decoding delay.

[0150] In the VUI syntax or SPS associated with the enhancement layer, or in the coding of the VPS extension syntax, when WPP is used, embodiments signal the layer decoding delay with respect to spatial segments (i.e. rows of CTBs) for spatial scalability, quality scalability, or multi-view scalability.

[0151] Tiles and slices

[0152] Parallel processing using other partitioning techniques similar to tiles or slices can also benefit from an implicit in the bitstream based on the division of the picture into spatial segments (i.e. tiles or slices) indicating the decoding delay. The enhancement layer decoding process requires information of the base layer (e.g. reconstructed picture data).

[0153] The syntax element structure 642 describes the decoder implication in the bitstream, i.e. signaling the layer decoding delay for tiles and / or slices.

[0154] Possible embodiments of the present invention use spatial segments as units to express the introduced layer processing delay depending on the type of parallel technique used in the coded video sequence.

[0155] FIG. 20 The syntax in the VUI parameter of the enhancement layer SPS gives an exemplary embodiment of the indication of min_spatial_segments_delay for the parallel tool WPP, tiles and slices (with respect to the embodiment of the syntax element structure 642).

[0156] min_spatial_segment_delay describes the decoding delay in spatial segments of the current layer relative to the corresponding base layer introduced by the encoding of the inter-dependency.

[0157] Depending on the value of min_spatial_segment_delay, the following applies:

[0158] If min_spatial_segment_delay is equal to 0, no signal is given about the minimum delay between the decoding of the layers

[0159] Otherwise, (min_spatial_segment_delay is not equal to 0), the bitstream conformance is specified to be true only if one of the following conditions is true:

[0160] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0 (i.e. neither tiles nor WPP are used in the video sequence), and all base layer resources for the decoding process of the first slice of the current layer in bit order are available when the first min_spatial_segment_delay slice of the base layer is completely decoded in bitstream order.

[0161] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0 (i.e. tiles are used in the video sequence), and all base layer resources for the decoding process of the first slice of the current layer in bitstream order are available when the first min_spatial_segment_delay slice covering the same picture area is completely decoded.

[0162] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1 (i.e. WPP is used in the coded video sequence), and all base layer resources for the decoding process of the first CTB row in the current layer are available when the first min_spatial_segment_delay CTB row of the base layer is completed.

[0163] Another exemplary embodiment is the indication of min_spatial_segments_delay in the VPS extension syntax as reported in [4] and extended as shown in FIG. 21

[0164] min_spatial_segment_delay describes the decoding delay in spatial segments of layer [i] relative to the corresponding base layer introduced by the encoding of the inter-dependencies.

[0165] Depending on the value of min_spatial_segment_delay, the following applies: If min_spatial_segment_delay is equal to 0, it signals a restriction on the minimum delay between the decoding of layers

[0166] Otherwise (min_spatial_segment_delay is not equal to 0), it is specified that only one of the following conditions is true for bitstream conformance: ​

[0167] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., neither tiles nor WPP are used in the video sequence), and all base layer resources for the decoding process of the first slice of the current layer in bit order are available when the first min_spatial_segment_delay slice of the base layer is completely decoded in bit order.

[0168] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., tiles are used in the video sequence), and all base layer resources for the decoding process of the first slice of the current layer in bit order are available when the first min_spatial_segment_delay slice covering the same picture area is completely decoded.

[0169] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1 (i.e., WPP is used in the coded video sequence), and all base layer resources for the decoding process of the first CTB row in the current layer are available when the first min_spatial_segment_delay CTB row of the base layer is completed.

[0170] The various prediction modes supported by the encoder and the decoder, as well as the restrictions imposed on the prediction modes, and the context derivation for entropy coding / decoding have been described above, so as to be able to support parallel processing concepts such as the tile and / or WPP concepts. It has also been mentioned above that the encoder and the decoder can operate on a block basis. For example, the prediction modes explained above are selected on a block basis, i.e. at a finer granularity than the picture itself. Before continuing with the description of another aspect of the present application, the relationship between the slices, tiles, WPP bitstream, and the blocks just mentioned will be explained.

[0171] FIG. 32A picture is shown: it can be a picture of layer 0 such as layer 12 or a picture of layer 1 such as picture 15. The picture is regularly subdivided into an array of blocks 90. Sometimes these blocks 90 are referred to as largest coding blocks (LCB), largest coding units (LCU), coding tree blocks (CTB), etc. Subdividing the picture into blocks 90 can form a kind of base or coarse granularity at which the aforementioned prediction and residual coding is performed, and this coarse granularity (i.e. the size of the blocks 90) can be signaled and set by the encoder separately for layer 0 and layer 1. For example, a multi-tree such as quad-tree subdivision can be used and signaled within the data stream to separately subdivide each block 90 into prediction blocks, residual blocks, and / or coding blocks. In particular, a coding block can be a leaf block of a recursive multi-tree subdivision of a block 90 and some prediction-related decisions such as prediction modes can be signaled at the granularity of the coding blocks, and a prediction block and a residual block can be separate leaf blocks of a recursive multi-tree subdivision within a coding block, for example, prediction parameters such as motion vectors (if temporal inter prediction) and disparity vectors (if inter-layer prediction) are coded at the granularity of the prediction blocks, and prediction residuals are coded at the granularity of the residual blocks.

[0172] A raster scan coding / decoding order 92 can be defined among the blocks 90. The coding / decoding order 92 limits the availability of neighboring portions for the purpose of spatial prediction: only the portions of a picture that precede the current position according to the coding / decoding order 92 are available for spatial prediction of the current position in terms of syntax elements that are currently being predicted. The coding / decoding order 92 traverses all blocks 90 of a picture within each layer, then continues to traverse the blocks of the next picture in the respective layer in picture coding / decoding order, which does not necessarily have to follow the temporal presentation order of the pictures. Within each block 90, the coding / decoding order 92 is refined into a scan among the smaller blocks such as coding blocks.

[0173] With respect to the just outlined blocks 90 and smaller blocks, each picture is further subdivided into one or more slices in the just mentioned coding / decoding order 92. FIG. 32 The exemplary slices 94a and 94b in FIGURE 1 cover the respective pictures without gaps, respectively. The boundaries or contact surfaces 96 between consecutive slices 94a and 94b in a picture can or can not be aligned with the boundaries of neighboring blocks 90. More precisely, FIG. 32 The consecutive slices 94a and 94b within one picture shown on the right-hand side in FIGURE 1 can border each other at the boundaries of smaller blocks such as coding blocks, i.e. the subdivided leaf blocks of a block 90.

[0174] Because slices 94a and 94b of the image can form a minimum unit, within which this portion of the data stream encoded from the image can be packaged into a packet, i.e., a NAL unit. For example, another possible property of slices described above is the restriction on slices determined by the prediction and entropy context across slice boundaries. Slices with this restriction can be called “normal” slices. In addition to normal slices, “dependent slices” can also exist, as summarized in more detail below.

[0175] If the concept of segmentation is applied to an image, the encoding / decoding order 92 defined between the array 90 of blocks can be changed. FIG. 33 This situation is illustrated in the example, where an image is exemplarily divided into four slices 82a to 82d. (See also...) FIG. 33 As shown, the slice itself is defined as regularly subdividing the image into blocks 90. That is, each of slices 82a to 82d consists of an n×m array of blocks 90, with n set individually for each row of slices and m set individually for each column of slices. Following the encoding / decoding order 92, before proceeding to the next slice 82b, etc., the blocks 90 in the first slice are first scanned according to the raster scan order, wherein slices 82a to 82d themselves are scanned according to the raster scan order.

[0176] Based on the WPP stream partitioning concept, the image in one or more rows of blocks 90 is subdivided into WPP substreams 98a to 98d according to the encoding / decoding order 92. For example... FIG. 34 As shown, for example, each WPP substream can cover a complete line of block 90.

[0177] However, the slice concept can also be combined with the WPP substream concept. In this case, for example, each WPP substream can cover one row of blocks 90 within each slice.

[0178] It is even possible to use image slice partitions and / or WPP substream partitions together. Regarding slices, each of the one or more slices into which the image is subdivided, according to the encoding / decoding order 92, can consist of only one complete slice, or more than one complete slice, or only a sub-part of a slice. Therefore, the slice constituting the smallest unit for parallelism can include normal slices on one hand and dependent slices on the other: normal slices impose the aforementioned constraints and entropy context on the prediction, while dependent slices do not. Dependent slices starting at image boundaries use the entropy context generated from the entropy decoding block 90 in the immediately preceding block 90, with the encoding / decoding order 92 unfolding sequentially in rows, and dependent slices starting elsewhere can use the entropy encoding context generated from entropy encoding / decoding the immediately preceding slice to its end. In this way, each of the WPP substreams 98a to 98b can consist of one or more dependent slices.

[0179] That is, the coding / decoding order 92 defined between the blocks 90 linearly leads from the first side (here, exemplarily, the left side) of the respective picture to the opposite side (exemplarily, the right side) and then steps in a downward / bottom direction to the next line of blocks 90. Thus, the coded / decoded parts of the current picture which are mainly arranged to the left and to the top of the current coding / decoding portion such as the current block 90 are available. Due to the prediction breaks and the entropy context derivation across the border lines, the slices of one picture can be processed in parallel. Even, the coding / decoding of the slices of one picture can be started at the same time. The restriction originates from the above mentioned in-loop filtering, in this case the restriction allows to cross the border lines. Further, the coding / decoding of the starting WPP sub-stream is performed in a top-to-bottom interleaved manner. The intra picture delay between consecutive WPP sub-streams is measured in blocks 90. It is two blocks 90.

[0180] However, even the coding / decoding of the picture 12 and the picture 15 is advantageously performed in parallel, i.e. time- sliced between the different layers. Obviously, with respect to the coding / decoding of the base layer, the coding / decoding of the dependent layer picture 15 has to be delayed to ensure that the "spatially corresponding" part of the base layer is already available. Even, these concepts are valid if no parallelism of the coding / decoding within either of the picture 12 and the picture 15 is used separately. Even, the coding / decoding of the picture 12 and the picture 15 can be performed in parallel if one slice is used to cover the entire picture 12 and the picture 15, respectively, and no slice and no WPP sub-stream processing is used. The below described signaling (i.e. the sixth aspect of the present application) is the possibility to express the decoding / coding delay of a layer even in this case, independent of whether slices or WPP processing is used for any picture of a layer or not.

[0181] From the above information about the minimum coding delay between the coding of consecutive layers it becomes clear that the decoder is able to determine the minimum decoding delay based on the short-term syntax elements. However, if the long-term syntax elements described above are used to signal in advance about the inter-layer time delay for a predetermined time period, the decoder can plan for the future to use the provided guarantees and can perform the workload distribution within the parallel decoding of the bitstream 40 more easily.

[0182] The below described aspect of the present application (i.e. the sixth aspect of the present application) is somehow related to aspect 3 in that explicit signaling of any inter-layer offset is concerned. However, with respect to the sixth aspect of the present application, the structure of the syntax elements explicitly signaling the inter-layer offset does not need to be based on long-term signaling in relation to short-term syntax elements from which the inter-layer offset can be derived otherwise. Rather, the sixth aspect of the present application adopts another finding that when describing FIG. 17A to FIG. 17CIt becomes clear that if the base layer pictures and the enhancement layer pictures are subdivided into blocks in accordance with the raster scan decoding order defined by the period, the layer offset between the base layer and the enhancement layer can be effectively and unambiguously signaled by measuring the layer offset within the unit of the base layer blocks. In connection with the aspects further described below, the base layer blocks within the unit in which the layer offset signal is unambiguously signaled are not limited to the spatial segments. More specifically, in view of this, other coding blocks can be used. Thus, when referring to FIG. 34 When describing embodiments relating to the sixth aspect of the present application, the reference FIG. 16 to FIG. 17C symbols used are largely used in common and the description made above with reference to the latter features shall apply equally to the embodiments further described below in order to avoid unnecessary repetition. In addition thereto, reference FIG. 32 and FIG. 33 is made to the description of Figures

[0183] Thus, FIG. 35 a video decoder 720 configured to receive a multi-layer video data stream 40 is shown, which encodes a scene in a hierarchy of layers with inter-layer prediction from a portion of a first layer to a collocated portion of a second layer, similar to the above described figure, exemplary for a representative embodiment of collocated layer 0 and layer 1. FIG. 35 Two temporally aligned pictures 12 and 15 of two layers are exemplary shown in Figure

[0184] The video data stream 40 comprises a syntax element structure 730 indicative of an inter-layer offset for decoding the pictures 12 and 15 in parallel by sequentially traversing the first blocks 722 and the second blocks 724 in a time overlapping manner and the inter-layer offset measured between the traversed first blocks 722 and the second blocks 724 in a unit of the base layer blocks 722. The video decoder 720 is configured to respond to the syntax element structure 730. In particular, the video decoder determines the inter-layer offset from the latter.

[0185] and FIG. 16In accordance with embodiments in

[0186] In further accordance with embodiments described above, the decoder 720 can use the slice offset signaled explicitly by the syntax element structure 730 as a measure of the offset in the traversal of the second block 724 with respect to the traversal of the first block 722 when decoding the pictures 12 and 15 in parallel, respectively. In other words, the video decoder 720 can be configured to derive a count value from the syntax element structure 730 and configured to calculate the decoded blocks 722 of the base layer picture 12 in a raster scan decoding order 726 while allowing the decoding of the sequence of blocks 724 of the enhancement layer picture 15 to start in the decoding order 728 only after the count of the decoded block 722 signaling the minimum count explicitly by the syntax element structure 730 has been reached. Thus, the video decoder 720 does not need to detect any high complexity and distributed parts of the video data stream 40 that would otherwise enable the video decoder 720 to calculate the actual minimum inter-layer offset between the start decoding block 722 on the one hand and the blocks 724 on the other hand by other means.

[0187] However, interestingly, in accordance with embodiments in FIG. 35 In accordance with embodiments in

[0188] In order to describe embodiments in FIG. 35 FIG. 36 In accordance with embodiments in FIG. 36 ​As shown, the video decoder 720 can use a counter 732 to count the number of decoded blocks 722 of the base layer image 12, starting from the first block 722 of image 12 according to the raster scan decoding order 726. The comparator 734 of the decoder 720 compares the steadily increasing count output by the counter 732 with the explicit signal value of the syntax element structure 730 obtained from the video data stream 40. If the count satisfies a predetermined relationship with the value indicated by the syntax element structure 730, such as once the count of the counter 732 reaches or equals the value indicated by the syntax element structure 730, the comparator is activated or causes the enhancement layer image 15 to begin decoding, i.e., causes the first block 724 of the enhancement layer image 15 to begin decoding according to the raster scan decoding order 728.

[0189] The syntax of grammar element structure 730, as described below in more detail, will be further explained below. FIG. 35 The syntax element structure and FIG. 16 The unification of syntactic element structure and its objectives is feasible. That is, as follows: Syntactic element structure 730 can have a set of possible values, i.e., a set of possible values. (See reference...) FIG. 16 As exemplarily mentioned, values ​​outside the non-explicit set of possible inter-layer offsets may cause the video decoder 720 to discard the values ​​of the syntax element structure 730 and not specify parallel decoding of images 12 and 15, or to determine arbitrary inter-layer offsets based on the short-term syntax element 602. If the syntax element structure 730 assumes values ​​outside the second set of possible values, this will cause the video decoder 720 to perform relative to... FIG. 36 The actions already summarized, for example, according to this action, the value of the syntax element structure 730 will explicitly signal the inter-layer offset within the cell of the base layer block 722. However, when assuming through the syntax element structure 730, there may be another subset of possible values ​​for the syntax element structure 730, causing the video decoder 720 to perform as described above. FIG. 16 The described action is to determine the interlayer offset between the base layer image 12 and the enhancement layer image 15 by interpreting the latter as the interlayer offset of the cells of the spatial segment consisting of blocks 722 and 724, which can but do not necessarily have to be integer multiples of each other.

[0190] refer to FIG. 37 It shows that FIG. 35 The implementation methods and FIG. 16 The possibilities for combining the implementation methods just mentioned. For example... FIG. 37As shown, the video decoder can inspect the syntax element structure 730 to determine whether the syntax element structure 730 has a value in a first subset 736, a second subset 738, or a third subset 740 of the set of possible values 742. Depending on the results of this investigation or check, the decoder 720 cannot derive any guarantees out of the syntax element structure 730 and cannot derive any explicit signaling of the slice offset from the syntax element structure, or performs a derivation of the inter-layer offset from the syntax element structure 730, i.e., either within the unit of a spatial segment or within the unit of a block. If it is the second subset 738, no derivation / any guarantees occur, if it is the subset 736, a derivation of the inter-layer offset within the unit of a spatial segment occurs, and if the syntax element 730 assumes a value outside the third subset 740, the inter-layer offset is derived within the unit of a block. In a particular syntax embodiment, which is further outlined below, the syntax element structure comprises two flags, namely ctb_delay_enabled_flag and min_spatial_segment_delay, wherein ctb_delay_enabled_flag = 0 and min_spatial_segment_delay ≠ 0 corresponds to the case of the subset 736, min_spatial_segment_delay = 0 corresponds to the second subset 738, and ctb_delay_enabled_flag = 1 and min_spatial_segment_delay ≠ 0 corresponds to the third subset 740.

[0191] Finally, with reference to FIG. 38 , it is shown that the decoder 720 can be configured to interpret any inter-layer offset signaled by the syntax element structure 730 not only as a decoded inter-layer offset with respect to the first block or spatial segment of the start enhancement layer picture 15, but also as a consecutive inter-layer offset, which results in a conflict-free parallel decoding of the pictures 12 and 15, respectively, when subject to the consecutive inter-layer offset. However, as FIG. 38As shown, the counter 732 still counts the number of decoded blocks 722 of the base layer picture 12, wherein an additional counter 744 likewise counts the decoded blocks 724 of the enhancement layer picture 15 in the decoding order 728, wherein a subtractor 746 forms the difference between the two counts, i.e. s and t-1, i.e. counts s-t+1. This difference value is compared by the comparator 734 with the inter-layer offset value derived from the syntax element structure 730, and as soon as the two values, i.e. the derived inter-layer offset and the interpolation between the counts, have a predetermined relationship, such as the difference value equals or exceeds the derived inter-layer offset, the block t is started to be decoded in the decoding order 728 between the enhancement layer blocks 724. In this way, a continuous check between the decoded blocks 722 of the base layer picture 12 on the one hand and the blocks 724 of the enhancement layer picture 15 on the other hand is provided.

[0192] It is obvious that the continuous check according to FIG. 38 can also be applied to spatial segments. More generally, the description of FIG. 38 and FIG. 36 has also been transferred to spatial segments and this statement also applies to the embodiments in FIG. 16 wherein the syntax element structure 642 can be used as the related syntax element structure shown in 730 in FIG. 36 and FIG. 38 . In other words, at least when using slices as spatial segments, there is also a raster scan decoding order defined between them, so that the changes discussed in FIG. 36 and FIG. 38 with respect to coding blocks can easily be transferred to the traversal of the slices and their decoding.

[0193] Briefly summarizing the sixth aspect and the embodiments described in connection therewith, the syntax element structure 730 can be inserted into the bitstream by the video encoder to provide an explicit indication to the decoder how to control the parallel decoding of base layer pictures and enhancement layer pictures with respect to each other. The inter-layer offset amount explicitly signaled by the syntax element structure can be signaled per slice or per slice or de-activated. If activated, the indication can be signaled within a unit of a block, such as a CTB, or within one of the more granular units of a block unit and a spatial segment. Due to the use of a raster scan order between base layer blocks on the one hand and enhancement layer blocks on the other hand, e.g. both being directed from the top left corner to the bottom right corner of each picture 12 / 15 by row-wise from top to bottom, the explicitly signaled inter-layer offset amount can be interpreted as a "trigger" for starting / commencing the decoding of the first block of the enhancement layer picture 15 only, or as a continuous "safe distance" between the currently decoded block of the base layer picture 12 on the one hand and the currently decoded block of the enhancement layer picture 15 on the other hand, i.e. for determining the trigger for decoding the respective blocks of the enhancement layer picture 15. The description set forth in connection with the sixth aspect can be transferred to the description and embodiments of the third aspect, in that, at least as far as the description of the slices as spatial segments is concerned, the interpretation and checking of the compliance of the signaled inter-layer offset amount can be done in the manner described in the embodiments of FIG. 36 and FIG. 38 The description set forth in connection with the sixth aspect can be transferred to the description and embodiments of the third aspect, in that, at least as far as the description of the slices as spatial segments is concerned, the interpretation and checking of the compliance of the signaled inter-layer offset amount can be done in the manner described in the embodiments of FIG. 36 and FIG. 38 The description set forth in connection with the sixth aspect can be transferred to the description and embodiments of the third aspect, in that, at least as far as the description of the slices as spatial segments is concerned, the interpretation and checking of the compliance of the signaled inter-layer offset amount can be done in the manner described in the embodiments of

[0194] Thus, the "delay" spatial segment can be used as a measure, i.e. the delay can be expressed within the unit of the spatial segment (slice, slice or CTB row), or the delay / offset can be measured within the unit of the block 90.

[0195] The High Efficiency Video Coding (HEVC) standard can be extended to comply with the following sixth aspect. Specifically, parallel decoding of individual layers (or views) is allowed if reference data is available. The minimum delay (specifically, the layer decoding delay) between the decoding of a base layer coding tree block (CTB) and the decoding of a corresponding independent enhancement layer CTB is determined by the granularity of the parallel tool, such as a slice. The wavefront, or slice, or motion compensation vector is applicable (e.g. in stereoscopic or multi-view video coding).

[0196] FIG. 20 A layer decoding delay indication is shown to be implemented by an enhancement layer sequence parameter set syntax (SequenceParameterSetSyntax) and a syntax element min_spatial_segment_delay.

[0197] min_spatial_segment_delay describes the decoding delay in spatial segments of the current layer relative to the corresponding base layer introduced by the encoding of the inter-dependencies.

[0198] The following can implement a mechanism in the HEVC high-level syntax that allows to express the layer decoding delay between an independent base layer and an independent enhancement layer independently relative, optionally, from the underlying parallel technology.

[0199] The signaling of the expression of the layer decoding delay (as signaled with the second syntax element) as a specific CTB address in the coded picture can be signaled using a flag of an exponent (e.g., ctb_delay_enabled_flag).

[0200] As can be seen from FIG. 39 and according to the following conditions, the CTB address in the raster scan order clearly defines the horizontal and vertical position within the picture for expressing the delay.

[0201] CTB coordinates = (CTB address % PicWidthInCTBs, CTB address / PicWidthInCTBs)

[0202] PicWidthInCTBs describes the width of the picture in CTB units.

[0203] FIG. 39 As shown below. The CTB address (e.g., 7) within the picture defines the horizontal CTB column and the vertical CTB row, e.g., the tuple (2, 1).

[0204] If the flag is enabled, the value of the other syntax element (cp.min_spatial_segment_delay) is interpreted as an offset to the CTB address of the collocated CTB in the base layer picture when decoding the CTB in the current independent layer.

[0205] As shown in FIG. 40 and the details in the following description process, the collocated CTB can be calculated according to the size of the CTB in the two respective layers and the width of the pictures of the two respective layers.

[0206] FIG. 40 The following shows various settings of the CTB size and picture size of the respective base layer and enhancement layer from left to right, independent of the picture aspect ratio. The bolded dashed boxes in the base layer picture mark the image area in the size of the enhancement layer CTB and its collocated image area in the CTB layout of the respective base layer.

[0207] In the enhancement layer Sequence Parameter Set Syntax and by FIG. 41An implementation of this optional indication of a CTB-based layer decoding delay is given in the semantics of the syntax element min_spatial_segment_delay.

[0208] The ctb_based_delay_enabled_flag equal to 1 indicates that the delay signaled by min_spatial_segment_delay is given in CTB units. The ctb_based_delay_enabled_flag indicates that min_spatial_segment_delay is not given in CTB units.

[0209] min_spatial_segment_delay describes the decoding delay in spatial segments of the current layer relative to the corresponding base layer introduced by the encoding of the inter-dependency.

[0210] Depending on the value of min_spatial_segment_delay, the following applies: If min_spatial_segment_delay is equal to 0, no signaling of any restriction on the minimum delay between the decoding of layers is signaled.

[0211] Otherwise (min_spatial_segment_delay is not equal to 0), and if ctb_based_delay_enabled_flag is equal to 1, the following conditions are specified to be true bitstream conformance:

[0212] • and CtbSizeY A , PicWidthInCtbsY A , and ctbAddrRs A are CtbSizeY and PicWidthInCtbsY of base layer A and CtbAddress of a Ctb in base layer A in raster scan order, CtbSizeY B , PicWidthInCtbsY B , and ctbAddrRs B are CtbSizeY and PicWidthInCtbsY of independent layer / view B and CtbAddress of a Ctb of independent layer B in raster scan order, and CtbScalingFactor BA , CtbRow BA (ctbAddrRs), and CtbCol BA (ctbAddrRs) are determined as follows:

[0213] CtbScalingFactorBA =(PicWidthInCtbsY A / PicWidthInCtbsY B )

[0214] CtbRow BA (ctbAddrRs)=Ceil((Floor(ctbAddrRs / PicWidthInCtbsY B )+1)*CtbScalingFactor BA )-1

[0215] CtbCol BA (ctbAddrRs)=Ceil(((ctbAddrRs%PicWidthInCtbsY B )+1)*CtbScalingFactor BA )–1

[0216] When using ctbAddrRs of the current enhancement layer / view B B When decoding CTB, if it has an equal value of PicWidthInCtbsY A *CtbRow BA (ctbAddrRs B )+CtbCol BA (ctbAddrRs B )+ctbAddrRs of min_spatial_segment_delay A When the base layer CTB is fully decoded, all necessary base layer resources are available.

[0217] Otherwise (min_spatial_segment_delay is not equal to 0 and ctb_based_delay_enabled is equal to 0), bitstream consistency is specified to be true only if one of the following conditions is true:

[0218] • In the set of picture parameters activated within the encoded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., neither slices nor WPP are used in the video sequence), and all base layer resources for decoding the first slice of the current layer in bitstream order are available when the first min_spatial_segment_delay slice of the base layer is fully decoded in bitstream order.

[0219] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0 (i.e. tiles are used in the video sequence), and all base layer resources used for the decoding process of the first tile of the current layer in bitstream order are available when the first min_spatial_segment_delay tile covering the same picture area is completely decoded.

[0220] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1 (i.e. WPP is used in the coded video sequence), and all base layer resources used for the decoding process of the first CTB row in the current layer are available when the first min_spatial_segment_delay CTB row of the base layer is completed.

[0221] Alternatively, as in the previous embodiment, it can be signaled that the inter-layer offset is a certain worst case delay of ctb_based_delay_enabled_flag, not the start-up delay of the first tile / tile / CTB row. The worst case delay gives the guarantee that during the decoding of a spatial segment of a dependent picture, all necessary corresponding base layer resources are available when the co-located spatial segment plus the signaled offset is completely decoded.

[0222] In FIG. 42 Embodiments for the syntax are shown in

[0223] min_spatial_segment_delay describes the decoding delay in spatial segment terms of the current layer relative to the corresponding base layer introduced by the encoding of the inter-dependencies.

[0224] The following applies depending on the value of min_spatial_segment_delay: if min_spatial_segment_delay is equal to 0, it signals that there is no restriction on the minimum delay between the decoding of the layers.

[0225] Otherwise (min_spatial_segment_delay is not equal to 0), it is specified that only one of the following conditions is true for bitstream conformance:

[0226] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0 (i.e. neither tiles nor WPP are used in the video sequence), and all base layer resources used for the decoding process of any slice segment A in the current layer in bitstream order are available when the first slice segment C (immediately following the (min_spatial_segment_delay - 1) slices in bitstream order) following the last slice segment B containing at least part of the same picture area as slice A in bitstream order is completely decoded.

[0227] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0 (i.e. tiles are used in the video sequence), and all base layer resources used for the decoding process of any tile A in the current layer in bitstream order are available when the first tile C (immediately following the (min_spatial_segment_delay - 1) tiles in bitstream order) following the last tile B containing at least part of the same picture area as tile A in bitstream order is completely decoded.

[0228] • In each picture parameter set activated within the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1 (i.e. WPP is used in the coded video sequence), and all base layer resources used for the decoding process of any CTB row A in the current layer are available when the first CTB row C (immediately following the (min_spatial_segment_delay - 1) CTB rows) following the last CTB row B of the base layer covering at least part of the same picture area as CTB row A in the enhancement layer is completely decoded.

[0229] Picture or min_spatial_segment_delay based signaling within a sub-coded video sequence is also possible. As given in FIG. 20 the scope of SEI messages is smaller than the coded video sequence in the time domain and the scope of SEI messages is limited by the position of the SEI message in the bitstream or by an index in terms of associated NAL units. In FIG. 43 an implementation is given in Layer_decoding_delay_SEI.

[0230] The syntax with respect to the previously described embodiments can be changed to reflect the scope of the SEI message and its syntax elements.

[0231] The explicit embodiments described above can be slightly modified. In the above described embodiments, the syntax element structure comprises a min_spatial_segment_delay and a ctb_based_delay_enabled_flag, and the min_spatial_segment_delay measures the inter-layer coding offset within the unit of spatial segment or CTB according to the ctb_based_delay_enabled_flag by one-dimensional or scalar manner following the spatial segment / CTB decoding order. However, since the number of CTBs of a base layer picture is usually equal to the number of spatial segments such as slices or bitstreams of the base layer picture, in slightly different embodiments, if the ctb_based_delay_enabled_flag indicates a CTB based inter-layer offset indication, the inter-layer offset cannot be determined based on the min_spatial_segment_delay only, but more precisely, in this case, the following syntax element is interpreted to indicate the positioning of the CTBs of the base layer picture in the horizontal dimension, and the full decoding of the base layer picture by the decoder can be used as a trigger to start the decoding of the enhancement layer picture. Naturally, alternatively, the min_spatial_segment_delay can be interpreted to indicate the CTBs of the base layer picture along the vertical dimension. According to the ctb_based_delay_enabled_flag, i.e. if it indicates a CTB based indication, further syntax elements are sent in the data stream acting on the positioning of the CTBs of the base layer picture in the other dimension of the just mentioned trigger.

[0232] Namely, the following syntax fragments for signaling can be used, i.e. can be used as syntax element structure:

[0233] The indices i and j can indicate the layer id of the base layer and the enhancement layer.

[0234] min_spatial_segment_offset_plus1[i][j] ue(v) if(min_spatial_segment_offset_plus1[i][j]>0){ ctu_based_offset_enabled_flag[i][j] u(1) if(ctu_based_offset_enabled_flag[i][j]) min_horizontal_ctu_offset_plus1[i][j] ue(v)

[0235] The semantics of the above syntax elements can be described as follows:

[0236] min_spatial_segment_offset_plus1[i][j] itself or jointly with min_horizontal_ctu_offset_plus1[i][j] indicates a spatial region in each picture of the j-th direct reference layer of the i-th layer (i.e. not used for inter-layer prediction for the decoding of any picture in the i-th layer). The value of min_spatial_segment_offset_plus1[i][j] shall be in the range of 0 to refPicWidthInCtbsY[i][j] * refPicHeightInCtbsY[i][j] inclusive. When not present, the value of min_spatial_segment_offset_plus1[i][j] is inferred to be equal to 0.

[0237] ctu_based_offset_enabled_flag[i][j] equal to 1, jointly indicated by min_spatial_segment_offset_plus1[i][j] and min_horizontal_ctu_offset_plus1[i][j], specifies a spatial region in each picture of the j-th direct reference layer of the i-th layer (i.e. not used for inter-layer prediction for the decoding of any picture in the i-th layer). ctu_based_offset_enabled_flag[i][j] equal to 0, indicated by min_spatial_segment_offset_plus1[i] only, specifies a spatial region in each picture of the j-th direct reference layer of the i-th layer (i.e. not used for inter-layer prediction for the decoding of any picture in the i-th layer) within the unit of slice segment or within the unit of CTU row. When not present, the value of ctu_based_offset_enabled_flag[i] is inferred to be equal to 0.

[0238] ctu_based_offset_enabled_flag[i][j] equal to 1, jointly indicated by min_spatial_segment_offset_plus1[i][j] and min_horizontal_ctu_offset_plus1[i][j], specifies a spatial region in each picture of the j-th direct reference layer of the i-th layer (i.e. not used for inter-layer prediction for the decoding of any picture in the i-th layer). The value of min_horizontal_ctu_offset_plus1[i][j] shall be in the range of 0 to refPicWidthInCtbsY[i][j] inclusive.

[0239] When ctu_based_offset_enabled_flag[ i ][ j ] is equal to 1, the variable minHorizontalCtbOffset[ i ][ j ] is derived as follows:

[0240] minHorizontalCtbOffset[ i ][ j ] = ( min_horizontal_ctu_offset_plus1[ i ][ j ] > 0 )? ( min_horizontal_ctu_offset_plus1[ i ][ j ] - 1 ) : ( refPicWidthInCtbsY[ i ][ j ] - 1 )

[0241] The variables curPicWidthInSamples L [ i ], curPicHeightInSamples L [ i ], curCtbLog2SizeY[ i ], curPicWidthInCtbsY[ i ], and curPicHeightInCtbsY[ i ] are set equal to PicWidthInSamples L , PicHeightInSamples L , CtbLog2SizeY, PicWidthInCtbsY, and PicHeightInCtbs, respectively, of the i-th layer.

[0242] The variables refPicWidthInSamples L [ i ][ j ], refPicHeightInSamples L [ i ][ j ], refCtbLog2SizeY[ i ][ j ], refPicWidthInCtbsY[ i ][ j ], and refPicHeightInCtbsY[ i ][ j ] are set equal to PicWidthInSamples L , PicHeightInSamplesL, CtbLog2SizeY, PicWidthInCtbsY, and PicHeightInCtbsY, respectively, of the j-th direct reference layer in the i-th layer.

[0243] curScaledRefLayerLeftOffset[ i ][ j ], curScaledRefLayerTopOffset[ i ][ j ], curScaledRefLayerRightOffset[ i ][ j ], and curScaledRefLayerBottomOffset[ i ][ j ] are set equal to scaled_ref_layer_left_offset[ j ] « 1, scaled_ref_layer_top_offset[ j ] « 1, scaled_ref_layer_right_offset[ j ] « 1, scaled_ref_layer_bottom_offset[ j ] « 1, respectively, of the j-th direct reference layer of the i-th layer.

[0244] colCtbAddr[ i ][ j ] specifies the raster scan address of the collocated CTU of a CTU, and the raster scan address is equal to ctbAddr in the picture of the j-th layer, the variable colCtbAddr[ i ][ j ] is derived as follows in the picture of the j-th direct reference layer of the i-th layer:

[0245] The variable ( xP, yP ) specifies the position of the top-left luma sample of a CTU, and the raster scan address is equal to ctbAddr related to the top-left luma sample in the picture of the i-th layer.

[0246] xP = ( ctbAddr % curPicWidthInCtbsY[ i ] ) « curCtbLog2SizeY yP = ( ctbAddr / curPicWidthInCtbsY[ i ] ) « curCtbLog2SizeY

[0247] The variables scaleFactorX[ i ][ j ] and scaleFactorY[ i ][ j ] are derived as follows:

[0248] curScaledRefLayerPicWidthInSamplesL[ i ][ j ] = curPicWidthInSamplesL[ i ] - curScaledRefLayerLeftOffset[ i ][ j ] - curScaledRefLayerRightOffset[ i ][ j ]

[0249] curScaledRefLayerPicHeightInSamplesL[i][j] = curPicHeightInSamplesL[i] - curScaledRefLayerTopOffset[i][j] - curScaledRefLayerBottomOffset[i][j]

[0250] scaleFactorX[i][j] = ((refPicWidthInSamplesL[i][j] << 16) + (curScaledRefLayerPicWidthInSamplesL[i][j] » 1)) / curScaledRefLayerPicWidthInSamplesL[i][j]

[0251] scaleFactorY[i][j] = ((refPicHeightInSamplesL[i][j] << 16) + (curScaledRefLayerPicHeightInSamplesL » 1)) / curScaledRefLayerPicHeightInSamplesL[i][j]

[0252] The variables (xCol[i][j], yCol[i][j]) specify the collocated luma sample location in the j-th direct reference layer picture of the luma sample location (xP, yP) in the i-th layer. The variables (xCol[i][j], yCol[i][j]) are derived as follows:

[0253] xCol[i][j] = Clip3(0, (refPicWidthInSamplesL[i][j] - 1), ((xP - curScaledRefLayerLeftOffset[i][j]) * scaleFactorX[i][j] + (1 << 15)) » 16)

[0254] yCol[i][j] = Clip3(0, (refPicHeightInSamplesL[i][j] - 1), ((yP - curScaledRefLayerTopOffset[i][j]) * scaleFactorY[i][j] + (1 << 15)) » 16)

[0255] The variable colCtbAddr[i][j] is derived as follows:

[0256] xColCtb[i][j] = xCol[i][j] » refCtbLog2SizeY[i][j]

[0257] yColCtb[i][j] = yCol[i][j] » refCtbLog2SizeY[i][j]

[0258] colCtbAddr[i][j] = xColCtb[i][j] + (yColCtb[i][j] * refPicWidthInCtbsY[i][j])

[0259] When min_spatial_segment_offset_plus1[i][j] is greater than 0, it is specified that bitstream conformance shall apply the following condition:

[0260] If ctu_based_offset_enabled_flag[i][j] is equal to 0, only one of the following is used:

[0261] • In each PSS of a picture referred to by the j-th direct reference layer of the i-th layer, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0, and the following applies:

[0262] • Let slice segment A be any slice segment of the picture of the i-th layer and let ctbAddr be the raster scan address of the last CTU in slice segment A. Let slice segment B be a slice segment (of the j-th direct reference layer of the i-th layer) that belongs to the same access unit as slice A and contains the CTU with the raster scan address colCtbAddr[i][j]. Let slice segment C be a slice segment that is within the same picture as slice segment B and immediately follows slice segment B in decoding order, and min_spatial_segment_offset_plus1[i] - 1 slice segments exist between slice segment B and this slice segment in decoding order. When slice segment C exists, the syntax elements of slice segment A are constrained such that any sample or syntax element value in slice segment C, or any slice segment that immediately follows C in decoding order, or the same picture is available for inter-layer prediction when decoding any sample of slice segment A.

[0263] • In each PPS of a picture referred to by the j-th direct reference layer of the i-th layer, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0, and the following applies:

[0264] • Let slice A be any slice in any picture picA of the i-th layer and let ctbAddr be the raster scan address of the last CTU in slice A. Let slice B be a slice in picture picB that belongs to the same access unit as picA and that is the j-th direct reference layer of the i-th layer and contains the CTU with the raster scan address colCtbAddr[i][j]. Let slice C be a slice that is also in picB and follows slice B in decoding order, and min_spatial_segment_offset_plusl[i] - 1 slices exist between slice B and this slice in decoding order. When slice segment C exists, the syntax elements of slice A are constrained such that any sample or syntax element value in slice C, or any slice following C in decoding order, or the same picture, can be used for inter-layer prediction in the decoding process of any sample within slice A.

[0265] • In each PPS assigned to pictures in the j-th direct reference layer of the i-th layer, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1, and the following applies:

[0266] • Let CTU row A be any CTU row in any picture picA of the i-th layer and let ctbAddr be the raster scan address of the last CTU in CTU row A. Let CTU row B be a CTU row in picture picB that belongs to the same access unit as picA and that is a CTU row of the j-th direct reference layer of the i-th layer and contains the CTU with the raster scan address colCtbAddr[i][j]. Let CTU row C be a CTU row that is also in picB and follows CTU row B in decoding order, and min_spatial_segment_offset_plusl[i] - 1 CTU rows exist between CTU row B and this CTU row in decoding order. When CTU row segment C exists, the syntax elements of CTU row A are constrained such that any sample or syntax element in CTU row C, or any row of the same picture following C in decoding order, can be used for inter-layer prediction in the decoding process of any sample within CTU row A.

[0267] Otherwise (ctu_based_offset_enabled_flag[i][j] is equal to 1), and the following applies:

[0268] • The variable refCtbAddr[i][j] is derived as follows:

[0269] xOffset[ i ][ j ] = ( ( xColCtb[ i ][ j ] + minHorizontalCtbOffset[ i ][ j ] ) > ( refPicWidthInCtbsY[ i ][ j ] - 1 ) )? ( refPicWidthInCtbsY[ i ][ j ] - 1 - xColCtb[ i ][ j ] ) : ( minHorizontalCtbOffset[ i ][ j ] )

[0270] yOffset[ i ][ j ] = ( min_spatial_segment_offset_plus1[ i ][ j ] - 1 ) * refPicWidthInCtbsY[ i ][ j ]

[0271] refCtbAddr[ i ][ j ] = colCtbAddr[ i ][ j ] + xOffset[ i ][ j ] + yOffset[ i ][ j ]

[0272] • makes CTUA a CTU in any picture picA of the i-th layer and makes ctbAddr the raster scan address ctbAddr of CTUA. makes CTUB a CTU in the same access unit as picA and of the j-th direct reference layer of the i-th layer in the picture and having a raster scan address greater than refCtbAddr[ i ][ j ]. When CTUB exists, the syntax elements of CTUA are constrained such that any sample or syntax element value in CTUB is used in the inter-layer prediction when decoding any sample within CTUA.

[0273] In other words, summarizing the just presented embodiments, the CTB based indication of the inter-layer offset can be accomplished by a flag in a switchable manner, the flag switching between the CTB based indication on the one hand and the spatial segment based indication of the inter-layer offset on the other hand, such that the alternative indication based on CTB can use syntax elements that are unconditionally sent, i.e. independent of the indication of the inter-layer offset based on CTB or based on spatial segments, i.e. if a signal is given for the spatial segment based indication, a syntax element is sent that indicates the inter-layer offset in the unit of the spatial segment, the syntax element being one component, such as a horizontal component or a vertical component, of the positioning of the "trigger CTB" in the base layer picture. Depending on the syntax element that switches between the CTB based indication of the inter-layer offset and the spatial segment based indication of the inter-layer offset, no further syntax element is sent. In particular, if a signal is given for the CTB based indication, a further syntax element is sent. In this case, the latter syntax element indicates the missing dimension of the positioning of the "trigger CTB", so that the decoder can use both syntax elements to identify the "trigger CTB" in the regular arrangement of the CTBs of the base layer picture in rows and columns and, once the CTB is decoded, the decoder can start decoding the enhancement layer picture. Nevertheless, by using one of the representative states of the first syntax element (i.e. min_spatial_segment_delay) the indication of the inter-layer offset can be switched completely. Since the predetermined decoding order is defined between the CTBs, in the case of the CTB based indication of the inter-layer offset, the decoder can still convert the horizontal component and the vertical component of the position of the trigger CTB into the number of CTBs of the base layer picture, without having to decode the base layer picture completely before starting the decoding of the first CTB of the enhancement layer picture, so that the decoder can use the FIG. 36 the CTB based indication of the inter-layer offset.

[0274] In the following, further aspects of the present application will be described in more detail. A fourth aspect relates to the problem that all network entities receiving the bitstream from the encoder should be able to easily distinguish the individual layers delivered in the multi-layer data stream. For example, an intermediate network entity can be interested in excluding certain information layers, such as layers regarding a sampling resolution that exceeds a certain resolution threshold, from further transmission. The following description provides an overview of the current situation with respect to the envisaged extensions of HEVC.

[0275] The video parameter set (VPS) of HEVC [1] provides high-level access to the coded bitstream and contains information that is important for processing of the bitstream in intermediate devices or end devices. The upcoming scalable and multiview extensions of HEVC will further benefit from a VPS extension that provides a syntax specified for scalable bitstreams. One of the main tasks of the VPS extension is to provide a uniform solution for the interpretation of the nuh_reserved_zero_6bits in the NAL unit header. The nuh_reserved_zero_6bits is intended to be repurposed as layer_id and used as a normal layer identifier in the context of scalable video coding. The layer_id syntax element in the NAL unit header is given in the following table and the NAL unit header as given in [1] and as shown in FIG. 2

[0276] Two common solutions are considered in the design process [5]. Finally, the solution is issued in the VPS profile that maps the value of a single identifier in the header of a NAL unit to potentially multiple scalable identifiers. Secondly, the solution is issued in the VPS extension that allocates individual bits (or blocks of bits) of a single identifier in the header of a NAL to specific scalable identifiers.

[0277] The design of the current VPS extension syntax reported in [4] uses the mapping solution, but, contains all syntax elements necessary for both solutions, i.e., two syntax elements indicate the type of scalability (cp.scalability_map) and the number of layers per scalability dimension (cp.dimension_id_len_minus1).

[0278] The mapping solution introduces further syntax elements to the VPS extension syntax, i.e., cp.scalability_id, which is the actual value of the scalable identifier that is coded into u(v), and, optionally, layer_id, if the encoder chooses to sparsely allocate the values of layer_id in a non-continuous fashion.

[0279] In many very similar scalable scenarios, e.g., two or three spatial layers, two or three views, etc., it is not necessary to use all 63 values of the 6-bit layer identifier in the NAL unit header. For these scalable scenarios, the solution that allocates individual bits of the layer identifier in the NAL unit header to specific scalability dimensions has two advantages compared to the mapping-based solution.

[0280] • The interpretation of the layer identifier value in the NAL unit header does not require an indirect or lookup approach in terms of scalability dimension identifiers.

[0281] ​• The VPS extension syntax elements needed for the mapping resolution approach do not need to be sent, and the VPS extension syntax elements are an important part of the VSP extension bits used for scalable signaling.

[0282] • The intermediate device does not need to store the mapping table for the mapping through the individual video bitstreams.

[0283] According to the concept of the fourth aspect described above, there can be an indication in the high level syntax of HEVC whether the mapping resolution approach or the partition resolution approach is used.

[0284] According to an embodiment, depending on the indication, a part of the mapping related syntax elements (cp.vps_nuh_layer_id_present_flag, layer_id_in_nuh[i] and dimension_id[i][j]) are sent or omitted, and the signaling of the syntax elements about the type of scalability (cp.scalability_mask) and the number of layers per scalability (cp.dimension_id_len_minus1) is made and the syntax elements are interpreted according to the indication, which is the information about the partitioning or mapping of the scalability identifier in the NAL unit header.

[0285] Reference FIG. 23 Embodiments are proposed which correspond to or employ the concept of the fourth aspect of the application. FIG. 23 A network entity is shown, which can be a video decoder of any of the kind already discussed above, or which can be an intermediate network entity generated between an encoder and a decoder. In general, the network entity is denoted using reference numeral 680. The network entity is configured to process a multi-layer video data stream 682, such as any of the data streams 40 described above. In case the network entity 680 is a video decoder, the processing will involve the decoding of the multi-layer video data stream 628. In case of an intermediate network entity, the processing can involve, for example, the forwarding of the video data stream.

[0286] The scene is encoded into a multi-layer video data stream in layers such that in each layer the scene is encoded into a different operating point of a scalability space extended by a scalability dimension, wherein the multi-layer video data stream is composed of first NAL units, each of the first NAL units being associated with one layer, and second NAL units interspersed within the first NAL units and presenting common information in the multi-layer video data stream. In other words, the first NAL units 684 can carry one or more slices of a picture of the video, and the "picture" corresponds to any of the layers of the multi-layer video data stream 682.

[0287] In the above described embodiments, only two layers have been discussed for ease of description. Naturally, the number of layers can be greater than two and even the classification of the information pertaining to any of the layers can differ from layer to layer. In addition to the first NAL unit 684, a NAL unit 686 is shown interspersed between the NAL units 684, but, in contrast to the first NAL units 684, its transmission can be performed over a separate channel. The second NAL unit presents common information in the multi-layered video data stream in a manner that will be set out in more detail below.

[0288] To describe the correlation between the first NAL units of one aspect and the arrangement of the layers of the data stream 682 of the other aspect in more detail, reference is made to FIG. 24 . FIG. 24 A first NAL unit 684 is shown that is representative of all first NAL units 684. The header 684 includes a layer indicator field 690. In addition to the header 688, the NAL unit 684 includes payload data 692 pertaining to slice data as discussed above with reference to other figures, i.e. data pertaining to video content that is encoded using inter-layer prediction. FIG. 24 Also shown is an arrangement of layers (i.e. 694). In particular, FIG. 24 The arrangement 694 of layers shown in Fig. 6 should represent all possible layers that can be represented and distinguished from each other by the layer indicator field 690 in the NAL units 684. That is, the correlation function between the arrangement 694 of one aspect and the possible values of the layer indicator field 690 should be assumed to be a one-to-one mapping. In FIG. 24 In Fig. 6, the individual layers in the arrangement 694 are exemplarily shown using small circles, each of which has a specific number marked thereon. Although these marked numbers suggest an order defined between the layers of the arrangement 694, it should be noted that the following discussion will reveal that the way the layers in the arrangement 694 are arranged or classified is not derived based on the layer indicator field 690 alone. Rather, in view of this, the network entity 680 needs to detect the type indicator field 696 in the interspersed second NAL units 686. This will be described later, however.

[0289] In other words, up to now, FIG. 24The individual elements of the arrangement 694 in the NAL unit 684 represent only one possible state of the layer indicator field 690 in the NAL unit 684. The layers in the arrangement 694 can be distinguished by the layer indicator field 690, but without additional information provided by the second NAL unit 686, the semantic meaning and order between the layers for the network entity 680 does not become clear, however, in fact, the layers of the arrangement 694 form nodes of a tree and the branches between the tree correspond to a particular scalability dimension or axis. For example, one layer is the base layer and corresponds to the root of the tree. Each branch connects two layers, i.e., two nodes of the tree, and describes how a particular layer contributes to another layer, i.e., an additional information classification is provided using inter-layer prediction, the information classification corresponds to a scalability dimension and can be, for example, an increase in spatial resolution, an increase in SNR, etc. For example, for the sake of simplicity, FIG. 24 A two-dimensional scalability space 698 is shown that is extended by two scalability dimensions 700 and 702 corresponding to spatial scalability and SNR scalability. FIG. 24 An exemplary tree of layers is shown in the extension of the layers 698. FIG. 24 It is also shown that not all possible layers of the arrangement 694 can be present in the data stream 682. For example, in the exemplary case of the arrangement 694, FIG. 22 In the exemplary case of the arrangement 694, in fact, only five layers are used in the data stream 682.

[0290] For example, FIG. 24 It is shown that the layer indicator field can have 6 bits, thereby distinguishing 2 6 = 64 possible states or possible layers of the arrangement 694. The mapping of these possible values or layers of the arrangement 694 to an operating point in the scalability space 698 can be adjusted via the second NAL unit 686 in the manner described in more detail below. FIG. 25 This mapping is denoted using reference symbol 704. The "operating point" shall denote the position of at least the actually present layer within the arrangement 694 in the scalability space 698. For example, the origin of the scalability space 698 can be associated with the base layer or the tree root, and each branch along either of the axes 700 and 702 of the scalability space 698 can have a fixed length of 1. Thus, a vector pointing to an operating point in the scalability space 698 can have integer-valued coordinates.

[0291] At present, a brief summary of the proposed description, the multi-layer video data stream 682 provides information about the video content or scene in a plurality of layers. The layers are arranged in a tree and each layer is connected to the tree via a branch. Starting from the base layer forming the root of the tree, the next layer following contributes to the reconstruction of the video content information about a certain information can be interpreted as a scalability dimension. Thus, each layer is either a tree root layer or is connected to the latter by a specific path of branches and requires NAL units 684 belonging to the layers arranged along this path to reconstruct the video content of the respective layer. Naturally, it is preferred if the mapping 704 is performed such that any "contributing" layer pointing to the distal end of the respective branch with respect to the tree root has a value of the layer indicator field 690, i.e. higher than the value of the layer indicator field of the layer located at the proximal end of the respective branch.

[0292] FIG. 25 The meaning of the type indicator field 696 is shown in more detail. FIG. 25 The layer indicator field 690 is shown as a fixed bit length field. In any case, the length of the field 690 is independent of the value of the type indicator field 696. However, if the type indicator field has a first state, the layer indicator field 690 is considered as a whole, i.e. all n bits thereof are jointly processed to distinguish between its possible values. Preferably, if the type indicator field 696 assumes the first state, an n-bit integer is derived from the layer indicator field 690 by the network entity 680. In the case that the type indicator field 696 assumes the first state, the network entity 680 performs a mapping 704 of the possible values of the m-bit field 690 to an operation point by using the mapping information conveyed within the second NAL unit 686. As FIG. 23 to FIG. 25As shown, for example, mapping information 708 includes a table of possible values ​​for each actually used layer indicator field 690, with vectors pointing to the associated operation points of the corresponding possible values. That is, if the type indicator field 696 assumes a first state, network entity 680 can derive mapping information 708 from the second NAL unit 686 and can perform a lookup of mapping information or label 708 for each layer indicator field 690 to find the associated vector to locate the corresponding / associated operation point in space 698. The number of dimensions p of the vectors associated with the possible values ​​in mapping information 708 can be a default setting or can be signaled within a data stream such as the second NAL unit 686. Subsequently, it is shown that the following signals can be emitted to convey information about mapping information 708: vps-_max_layers_minus1 determines the number M possible values ​​M of the M-bit field 690 actually used. num_dimensions_minus1 limits the dimensions. The latter two variables can be sent using the syntax element scalability_mask. Then, the table itself can be signaled via a pair of possible values ​​for a face (i.e., layer_id_in_nuh) and a p-dimensional vector (i.e., dimension_id[i][j]). Mapping 704 then directs to the p-dimensional vector, that is, a vector mapped via mapping information 708 to a corresponding vector 710, which points to the operation point within space 698 associated with the layer of the NAL cell having the layer indicator field 690.

[0293] However, if the type indicator field 696 assumes a second state, the mapping 704 is performed to a different degree. Specifically, in this case, the mapping is performed by dividing the layer indicator field 690 into more than one part, that is, by dividing the sequence of m-bit fields 690 into n consecutive subsequences of bits. Each part thus obtained is used as the coordinates x1…x1 of the n-dimensional vector 712. n Conversely, coordinates x1…x n This refers to the operation point associated with the layer within the scalable space 698 that includes the NAL unit containing the layer indicator field 690. For example, this is achieved by forming the first portion beyond the first (most important) m1 bits of field 690, the second portion beyond the subsequent m2 (less important) bits of field 690, and so on, up to m1+…+m… n =m, the nth part, divides the m bits of the layer indicator field 690 into n parts. The bits of each part can be directly treated as integer values.

[0294] In case the type indicator field assumes the second state, the dimension n can be a default setting or signaled via the data stream. If p is determined using the implementation described herein after n is derived based on the same syntax element, if the type indicator field 696 assumes the first state, i.e., based on scalability_mask. The bit length of the parts into which the layer indicator field 690 is subdivided (i.e., m1,..., m n ), but, again, this subdivision can be achieved by default (no explicit signaling).

[0295] It should be noted that the same syntax structure such as scalability_mask (independent of the type indicator field 696) that can be parsed can indicate the number of dimensions of the scalability space and the semantic meaning. However, the allowed states of this syntax element can be limited to a proper subset of the available possible states of the syntax element with respect to the mapping case, if the type indicator field 696 indicates a component-wise interpretation of the layer indicator field, e.g., because the maximum number of dimensions of the scalability space available in the mapping case (first state of the type indicator field) is higher than the maximum number available in the component-wise interpretation case (first state of the type indicator field). The encoder only complies with this limitation accordingly.

[0296] With respect to the implementation in FIG. 26 an exemplary use case for the implementation is as follows:

[0297] Massive multi-party conference

[0298] In the context of a massive session service over RTP, e.g., a network conference, the transmitted video between the parties is adapted in a multipoint control unit (MCU) that is aware of the parameter sets of the respective video bitstreams. Each party provides a thumb bitstream and two bitstreams with enhanced spatial resolution (e.g., 720p and 4K) for the talker. The MCU makes a decision about which streams to provide to which party. Thus, the easy parsing of the scalability parameters significantly relieves the burden of the MCU. The partition-based solution requires less computational and memory resources compared to the mapping-based solution of the scalability signaling.

[0299] Transmission system

[0300] In transport systems such as RTP or MPEG2-TS, the mapping of scalability related codec information to the corresponding elements can benefit from less complex bit preservation mechanisms such as partitioning compared to the mapping based solution. Transport systems are not likely to adopt scalability signaling in the form of mapping solutions but rather, more precisely, solve the mapping indirection and produce dedicated scalability identifiers for each scalability dimension, in turn explicitly signaling the scalability dimensions in a partitioning solution.

[0301] FIG. 27 The embodiment in [3] proposes a possible implementation as a syntax flag in the VPS extension syntax that allows switching between the mapping based solution and the partitioning based solution for scalability signaling in the extensions for HEVC based on the VPS extension syntax as reported in [4].

[0302] The dedicated_scalability_ids_flag having a value equal to 1 indicates that the bits of the layer_id field in the NAL unit header are split according to the values of dimension_id_len_minus1[ ] and the bits of the layer_id field belong to the corresponding scalability dimension signaled in the scalability_mask. The dedicated_scalability_ids_flag having a value equal to 1 indicates that no signal is emitted regarding the syntax elements vps_nuh_layer_id_present_flag, layer_id_in_nuh[ i ], and dimension_id[ i ][ j ]. The dedicated_scalability_ids_flag having a value equal to 1 indicates that the derivation of the variable describing the scalability identifier of the corresponding NAL unit uses only the corresponding associated bits of the scalability identifier in the NAL unit header (cp.layer_id), e.g. in the form:

[0303] DependencyId = layer_id && 0x07

[0304] QualityId = layer_id && 0x38

[0305] Given that layer_id is signaled in the NAL unit header, dedicated_scalability_ids_flag with value equal to 0 indicates that the syntax elements vps_nuh_layer_id_present_flag, layer_id_in_nuh[i], and dimension_id[i][j] are signaled and the bits of layer_id in the NAL unit header are not associated with a specific scalability dimension but are mapped to a scalability identifier in the VPS extension. dedicated_scalability_ids_flag with value equal to 0 indicates that the derivation of the variable describing the scalability identifier of the corresponding NAL unit uses the syntax element dimension_id[i][j], e.g. in the form of:

[0306] if (layer_id == layer_id_in_nuh[0]) {

[0307] DependencyId = dimension_id[0][0]

[0308] QualityId = dimension_id[0][1]

[0309] When dedicated_scalability_ids_flag is equal to 0, dimension_id_len_minus1[i] indicates the length of the bits of dimension_id[i][j]. When dedicated_scalability_ids_flag is equal to 1, dimension_id_len_minus1[i] indicates:

[0310] The number of bits of layer_id in the NAL unit header associated with the i-th scalability dimension as indicated by scalability_mask.

[0311] FIG. 31 The flowchart in Fig. 1 further illustrates possible embodiments of the present application. The scalability identifier is either derived directly from the bits of layer_id via a masked bit copy or signaled in the VPS via an association with a specific value of layer_id.

[0312] FIG. 25Another syntax embodiment is illustrated below. Here, the type indicator field is signaled via “splitting_flag”, and the layer indicator field is referred to as nuh_layer_id. Based on “splitting_flag”, the operation point of the corresponding NAL cell layer within the scalable space is derived from nuh_layer_id using either a mapping concept or a partitioning concept. A partitioning concept is exemplarily signaled via a splitting_flag equal to 1. The scalable identifier, i.e., the vector component of the scalable dimension of the scalable space, can then be derived from the nuh_layer_id syntax element in the NAL cell header via bitmask copying. FIG. 31 The corresponding bitmask for the i-th component (i-th scalability dimension) of vector 712 in the NAL unit is defined as follows. Specifically, a splitting_flag equal to 1 indicates the absence of a dimension_id[i][j] syntax element (i.e., no mapping information 708), and the binary identifier of the nuh_layer_id value in the NAL unit header is split into a series of NumScalabilityTypes, i.e., n segments, with a length x in bits according to the value of dimension_id_len_minus1[j]. 1…n Furthermore, the value of dimension_id[LayerIdxInVps[nuh_layer_id]][j] is inferred from the NumScalabilityType section of field 690, that is, the component x of vector 712. 1…n By issuing a series of flags, `scalability_mask_flag`, signals are sent regarding the syntactic meaning and number of scalable axes in the scalability space, thereby indicating to each of the predetermined number of scalable types (exemplarily fixed here) whether the corresponding scalability type belongs to any of the scalability dimensions of the scalability space 698. Specifically, network entity 680 can, according to... FIG. 31 The for loop in the code derives the scalability space from the sequence marked with scalability_mask_flag, that is, the syntactic meaning and number of scalability axes NumScalabilityType:

[0313] for(i=0,NumScalabilityTypes=0;i<16;i++){

[0314] scalability_mask_flag[i]

[0315] NumScalabilityTypes+=scalability_mask_flag[i]

[0316] }

[0317] where a scalability_mask_flag[i] equal to 1 indicates that the i-th scalability dimension is present and a scalability_mask_flag[i] equal to 0 indicates that the i-th scalability dimension is not present. Here, i = 1 can represent multi-view (view scalability), i = 2 can represent spatial / SNR scalability, and i = 0 can represent the addition of depth map information. Other scalability dimension types can also be present, and of course the just summarized embodiments are merely illustrative. Assuming that the length of nuh layer id is illustratively 6, the partitioning or mask copying can be done as follows:

[0318] The variable dimBitOffset[0] is set equal to 0, and for j in the range of 1 to NumScalabilityTypes - 1, inclusive, dimBitOffset[j] is derived as follows:

[0319]

[0320] The value of dimension_id_len_minus1[NumScalabilityTypes - 1] is inferred to be equal to 5 - dimBitOffset[NumScalabilityTypes - 1].

[0321] The value of dimBitOffset[NumScalabilityTypes] is set equal to 6.

[0322] Bitstream conformance is specified, namely, when NumScalabilityTypes is greater than 0, dimBitOffset[NumScalabilityTypes - 1] shall be less than 6.

[0323] For j from 0 to NumScalabilityTypes - 1, inclusive, dimension_id[i][j] is inferred to be equal to ((nuh_layer_id & ((1 « dimBitOffset[j + 1]) - 1)) » dimBitOffset[j]).

[0324] In summary, the following syntax elements are involved in associating individual first NAL units with their operation points within the scalability space:

[0325] 1) nuh layer id, i.e., layer indicator field 690

[0326] 2) the sequence of scalability_mask_flag, i.e. the information showing the number and meaning of the scalability axes 700, 702 of the display space 698, and the number p of components of the M vectors in the table 708, i.e. the sequence of layer_id_in_nuh[i] i the number n of layers

[0327] 3) the individual components x of the field 690 for each axis i dimension_id_len_minus1, i.e. the bit length (except for one, since the remaining one is supposed to include all remaining bits 706 of the field 690)

[0328] 4) Furthermore, according to an optional implementation in FIG. 31 vps_max_layers_minus1 is transmitted, which syntax element indicates the number of used / actual layers in some some of the possible values of the vps_max_layers_minus1 for the partition layer indicator field, i.e. the sequence of layer_id_in_nuh[i], thereby defining the order of the arrangement between the possible operation points.

[0329] If splitting_flag is equal to 0, the mapping concept is used. Thus, the mapping information 708 is transmitted using the following information:

[0330] 1) the sequence of scalability_mask_flag and the number p of components of the M vectors in the table 708, i.e. the information showing the number and meaning of the scalability axes 700, 702 of the display space 698

[0331] 2) the individual components x of the vectors dimension_id[i][j] in the table 708 j dimension_id_len_minus1, i.e. the bit length, i.e. each of the axes of the space 698

[0332] 3) optionally, layer_id_in_nuh[i] is used as an index for the list of the M vectors dimension_id[i][j]

[0333] 4) optionally, although not shown in FIG. 28 vps_max_layers_minus1 is transmitted, thereby indicating the number of possible i 2 dimension_id_len_minus1[i] layers M in some

[0334] Thus, if the splitting_flag is equal to 0, the vector is inherently derived by the partition without explicit signaling 710, i.e. without the signaling dimension_id[i][j] that is turned to be inferred.

[0335] Thus, according to a fourth aspect, the concept "switchable interpretation of NAL unit layer identifiers" of the bitstream 40 can comprise a NAL unit, namely a VPS NAL unit, which comprises a type indicator field 300 via which a switch between a mapping concept and a bit splitting concept can be made for interpreting the layer identifier field 302 in "normal" NAL units. Thus, the same bit positions of the field 302 are used in both modes, but a signal is sent out signaling the interpretation and the interpretation instruction information, namely the mapping information or the splitting and semantic information, when a bitstream change between the modes occurs. Although this makes an additional transmission of the type indicator field necessary, the concept leads together with the advantages of the mapping concept to a more efficient bitstream transmission and can root the need to employ the bit splitting concept as both concepts can be applicable to different multi-layer data from different degrees, such as depending on the number of layers.

[0336] A fifth aspect of the present application is about a multi-standard multi-layer video decoder interface. The concept set here describes an interface between a transport layer decoder and a scalable video decoder supporting different video coding standards in different layers (e.g. H.264 / AVC in the base layer and HEVC in the enhancement layer).

[0337] Scalable video bitstreams are composed of layers: a base layer containing a self-contained decoded video signal and one or more enhancement layers that can be decoded in combination with the base layer (and potentially other enhancement layers) and provide higher temporal resolution (temporal scalability), spatial resolution (spatial scalability), quality (SNR scalability), higher bit-depth (bit-depth scalability), video signals or other camera views (multi-view scalability)

[0338] Existing scalable video coding standards like H.264 / SVC define base and enhancement layers in the same standard. The base layer and the enhancement layers are designed in a way that the scalable bitstream has the same base format as a non-scalable bitstream. If the scalable bitstream is input into a non-scalable decoder, the packet types can still be found and unknown packets can be discarded.

[0339] HEVC is the first video coding standard that allows the use of a base layer of a different video coding standard (e.g. H.264 / AVC). The packet formats of the two standards are different, so a base layer decoder cannot understand the enhancement layer packets. On the other hand, an enhancement layer decoder can understand the enhancement layer packet format, but cannot understand the base layer packet format.

[0340] In audio / video systems, several audio streams are combined with video streams using a transport layer and metadata such as timing and stream type are provided.

[0341] In existing multi-layer transport layer decoders, the access units of the base layer and the enhancement layer are multiplexed into a single video data stream (e.g. Annex B Byte stream of H.264 / AVC). This video stream is input into a video decoder.

[0342] In all cases, if the base layer and the enhancement layer use different video coding standards, the base layer packets cannot be combined with the enhancement layer packets into a single bitstream.

[0343] According to embodiments of the fifth aspect, the transport layer decoder distinguishes between the following cases:

[0344] 1. The video decoder reading the output can only decode the base layer

[0345] 2. The video decoder reading the output can decode the base layer and the enhancement layer and the base layer and the enhancement layer are coded using the same video coding standard

[0346] 3. The video decoder reading the output can decode the base layer and the enhancement layer and the base layer and the enhancement layer are coded using different video coding standards

[0347] In case 1, the transport layer decoder uses the following behavior:

[0348] Only the packets containing the base layer are extracted from the transport layer and input into a single standard single layer video decoder in the format specified by the video coding standard.

[0349] The specific implementation is that the transport stream decoder extracts H.264 / AVC NAL units from the MPEG-2 transport stream by selecting only streams having an assigned stream type "AVC video stream or AVC video substream conforming to one or more profiles defined in Annex A of ITU-T Rec. H.264 | ISO / IEC 14496-10" and inputs the H.264 / AVC NAL units into an H.264 / AVC video decoder in the byte stream format defined in Annex B of the H.264 / AVC specification. NAL units belonging to streams whose stream type is not equal to the stream type "AVC video stream or AVC video substream conforming to one or more profiles defined in Annex A of ITU-T Rec. H.264 | ISO / IEC 14496-10" are discarded in the transport stream decoder. FIG. 29 An example of a specific implementation is given.

[0350] In case 2, the transport layer decoder uses the following behavior:

[0351] Packets from the base layer and the enhancement layer are extracted from the transport layer and input into a single-standard multi-layer video decoder in the format specified in the video coding standard.

[0352] The specific implementation can extract H.264 / AVC NAL units from the MPEG-2 transport stream by selecting a base layer stream having an assigned stream type "AVC video stream or AVC video substream conforming to one or more profiles defined in Annex A of ITU-T Rec. H.264 | ISO / IEC 14496-10" and also selecting one or more enhancement layers having an assigned stream type "SVC video substream of an AVC video stream conforming to one or more profiles defined in Annex G of ITU-T Rec. H.264 | ISO / IEC 14496-10". The NAL units of the different layers are multiplexed into the byte stream format defined in Annex B of the H.264 / AVC specification and input into an H.264 / AVC SVC video decoder. FIG. 30 An example of a specific implementation is given.

[0353] In case 3, the transport layer decoder uses the following behavior:

[0354] Packets from the base layer and the enhancement layer are extracted from the transport layer. These packets are processed in a multi-standard multi-layer video decoder by one of the ways described in the following sections.

[0355] Interface A

[0356] If the enhancement layer standard package format allows to carry the base layer packages, the base layer packages are encapsulated by the enhancement layer format. This means that a header is added to each base layer package that is understandable by the enhancement layer standard video decoder and allows the enhancement layer video decoder to identify the package as a base layer of a different video coding standard.

[0357] In the detailed embodiment of the application, H.264 / AVC NAL units are used as payload of HEVC NAL units, i.e. the HEVC NAL unit header is added in front of the H.264 / AVC NAL unit header. The payload is identified as a NAL unit of H.264 / AVC using a field in the HEVC NAL unit header (e.g. nal_unit_type). The bitstream can be input into a video decoder of the HEVC Annex B byte stream format. FIG. 44 An illustration of the detailed embodiment is given.

[0358] Interface B

[0359] The different layers of the scalable bitstream use different channels. The video coding standard is determined in the decoder by the channel selection.

[0360] In the detailed embodiment of the application, two layers are exchanged in two separate channels. The first channel is used for H.264 / AVC base layer packages (or Annex B byte stream) only, while the second channel is used for HEVC enhancement layer packages only.

[0361] Interface C

[0362] The metadata field indicating the type of the video coding standard is associated with each package that is passed from the transport stream decoder to the multi-standard multi-layer video decoder. In the same way other metadata can be signaled, e.g. similar to timing.

[0363] In the detailed embodiment, each base layer NAL unit is identified as a H.264 / AVC NAL unit by the associated metadata field and each enhancement layer NAL unit is identified as a HEVC NAL unit by the associated metadata field.

[0364] The concept of the fifth aspect can therefore be described as "multi-standard multi-layer video decoder interface", providing the possibility how to compose the bitstreams of different codecs.

[0365] According to the fifth aspect of the present application, the transport layer decoder can therefore be configured to compose the bitstream of the multi-standard multi-layer video decoder as described below with reference to FIG. 44 In summary, the reference sign 770 is used to generally indicate FIG. 44The transport layer decoder 770 is configured to pass the incoming multi-layer video data stream 40 through the scenes encoded in layers, so that they are decoded by a multi- standard multi-layer decoder 772 connected to the output interface of the transport layer decoder 770. The multi-layer video data stream 40 is composed of NAL units which have been outlined above with reference to the various other aspects and embodiments of the present application, the description of which can be transferred to the FIG. 44 embodiments. Each NAL unit is associated with one layer. Each layer is associated with a different codec, i.e. a different standard. For each layer, the NAL units associated with the respective layer are encoded using the same codec, i.e. one of the different standards.

[0366] For each NAL unit, the transport layer decoder 770 is configured to identify the same codec associated with it and to hand over the NAL unit of the multi-layer video data stream 40 to a multi-standard multi-layer decoder which uses inter-layer prediction between layers associated with different codecs to decode the multi-layer video data stream.

[0367] As outlined above, each NAL unit can be associated with one layer in the multi-layer video data stream 40 by a specific layer indicator field which has been outlined above with reference to the fourth aspect of the present application, some or all of the NAL units can carry content-related data, i.e. one or more slices. By collecting all NAL units on a specific set of layers, the decoder 772 decodes the video content or scene encoded into the data stream 40 with the amount of information granted by the layers of the set. With reference to the description of the fourth aspect of the present application, the options regarding layer dependency, more than one dimension of scalability, etc.

[0368] The multi-layer multi-standard decoder 772 is capable of handling different codecs / standards. Embodiments have been outlined above with respect to different standards, i.e. H.264, HEVC, however, other standards can also be mixed. The different codecs / standards are not limited to mixed codecs. Rather, different kinds of codecs can also be mixed. The inter-layer prediction used by the multi-layer multi-standard decoder 772 can involve prediction parameters used in different layers or can involve picture samples of each temporally aligned layer. This has been described above with reference to other aspects and embodiments.

[0369] The transport layer decoder 770 can be configured to perform a handover for NAL units belonging to layers of the encoder which the multi-layer multi-standard decoder 772 is only capable of handling. That is, the handover performed by the transport layer decoder 770 can depend on the identification of the transport layer decoder 770 with respect to the codec associated with each NAL unit. In particular, the transport layer decoder 770 can perform for each NAL unit:

[0370] • The layer associated with the currently detected NAL unit is identified, for example, by detecting a layer indicator field in the NAL unit header of the NAL unit.

[0371] • In one aspect, based on the association between the layers of the data stream 40 and the derivation of the same codec / standard based on the detection of the respective high level syntax of the data stream 40 by the transport layer decoder 770, the transport layer decoder 40 determines whether the currently detected NAL unit fulfills the following two criteria: the NAL unit layer belongs to a subset of layers which is forwarded to the decoder 772, the subset is determined by the operation point of the currently detected NAL unit layer in the scalability space, and an external instruction regarding which operation points within the scalability space are allowed to be forwarded to the multi-layer multi-standard decoder 772 and which operation points within the scalability space are not allowed to be forwarded to the multi-layer multi-standard decoder 772. Furthermore, the transport layer decoder 770 checks whether the currently detected NAL unit layer codec belongs to the set of codecs / standards which the multi-layer multi-standard decoder 772 is able to process.

[0372] • If the check shows that both criteria are fulfilled by the currently detected NAL unit, the transport layer decoder 770 forwards the current NAL unit to the decoder 772 for decoding.

[0373] For the transport layer decoder 770, there are different possibilities to determine the above mentioned association between the layers contained in the data stream 40 on the one hand and the codecs / standards based on the above mentioned association on the other hand. For example, as mentioned above with reference to "interface B", it is possible to use different channels to deliver the data stream 40 (i.e. NAL units of layers of one codec / standard in one channel) and NAL units of layers encoded according to another codec / standard according to another channel. By this measure, the transport layer decoder 770 is able to derive the just mentioned association between the layers on the one hand and the codecs / standards on the other hand by distinguishing the individual channels. For example, for each NAL unit of the data stream 40, the transport layer decoder 770 determines the channel the respective NAL unit arrived from in order to identify the codec / standard associated with the respective NAL unit or the layer of the respective NAL unit.

[0374] Furthermore or alternatively, the transport layer decoder 770 can forward the NAL units of layers belonging to different codecs / standards to the multi-layer multi-standard decoder 772 in a way that the NAL units of layers belonging to one codec / standard are transmitted to the decoder 772 by one channel and the NAL units of layers of a different codec / standard are forwarded to the multi-layer multi-standard decoder 772 by another channel.

[0375] The base transport layer can provide "different channels". Namely, for ease of understanding, the different channels are distinguished by ​ The different channel identifiers provided by the base transport layer not shown in the middle can enable the distinction between the different channels.

[0376] Another possibility to hand over the data stream 40 to the multi-layer multi- standard decoder 772 is that the transport layer decoder 770 encapsulates each NAL unit belonging to a layer not encoded using the predetermined codec using the NAL unit header of the predetermined codec and sets the NAL unit type indicator within the NAL unit header to indicate the status of the codec of the respective actual layer. For example, the base layer NAL units are H.264 NAL units of the base layer encapsulated using HEVC and the NAL unit header with the NAL unit type indicator set to indicate the status of H.264. Thus, the multi-layer multi-standard 772 will receive a data stream compliant with HEVC.

[0377] Naturally, as described with reference to interface C, alternatively, the transport layer decoder 770 can provide for each NAL unit of the inbound data stream 40 metadata indicating the codec associated with the layer associated with the respective NAL unit. Thus, the NAL units of the data stream 40 will be forwarded to the decoder 772 by way of extension thereof.

[0378] Using the just described alternative, it is easy to extend the content encoded into the data stream by further layers, however, using another encoder such as a new encoder will further encode the layers and no modification of the existing parts of the codec 40 is required. The multi-layer multi-standard decoder in turn is able to handle the new codec (i.e. the newly added one) and is able to handle the existing mixed data stream using the layers encoded with the new codec.

[0379] Thus, the above proposes a parallel / low delay video encoding concept with respect to HEVC scalable bitstreams.

[0380] The High Efficiency Video Coding (HEVC) standard [1] initially featured two dedicated parallel tools allowing parallel processing on the encoder side and on the decoder side: tiles and Wavefront Parallel Processing (WPP). These tools allow for intra-picture parallelism when compared to HEVC coded video featuring no intra-picture parallelism, targeting to improve processing time while minimizing coding efficiency loss.

[0381] In scalable [2] or multi-view [3] HEVC bitstreams, decoded base layer or base view pictures are used to predict enhancement layer or related view pictures. In the above description, the term "layer" can also cover the concept of view at the same time.

[0382] The above embodiments describe scalable video decoders capable of starting the decoding of an enhancement layer picture before the decoding of the associated base layer picture is completed. The picture area decoding is done based on the higher order parallel tools used in each layer. The base layer decoder and the enhancement layer decoder can operate in parallel to each other and also in parallel to the actual layer. The amount of parallelism within each layer between the base layer and the enhancement layer can be different. Furthermore, signaling is described that specifies the parallel decoding environment of a particular bitstream is properly set.

[0383] As a general note, it should be noted that the above embodiments describe decoders and accordingly the encoders are designed according to the respective aspects. It should be noted that as far as these aspects are concerned, all of them can have in common that the decoders and encoders support WPP and / or tile parallel processing and accordingly the details described in relation thereto should be considered to apply to any other aspect and its corresponding description as well to lead to new embodiments of these other aspects or to complement the description of embodiments of these other aspects, independent of the respective aspects described in relation to the use of the terms "part", "spatial segment" etc. (instead of tiles / substreams in relation to more generally denoted parallel processing segments of pictures) (the respective description is transferred to the respective aspect). The same applies to the details described in relation to the setting of coding / prediction parameters and description of possible ways of subdividing pictures: all aspects can be implemented to result in decoders / encoders that use the subdivision of LCUs / CTBs by tiles and / or substreams within the unit of LCUs / CTBs by determining the tiles. Thus, by any of these aspects, the LCUs / CTBs can be further subdivided into coding blocks by using the recursive multi-tree subdivision described above with reference to the aspects and subsets of their embodiments. Furthermore or alternatively, from the relationship between slices and substreams / tiles described with reference to these aspects, the embodiments of all aspects can employ the slice concept.

[0384] Although some aspects have been described in the context of an apparatus, it is clear that separate aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps can be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or electronic circuit. In some embodiments, some one or more of the most important method steps can be executed by such an apparatus.

[0385] It is envisaged that, according to a specific implementation, embodiments of the application can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy diskette, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium can be computer readable.

[0386] Some embodiments according to the application comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0387] Generally, embodiments of the present application can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code can for example be stored on a machine readable carrier.

[0388] A further embodiment comprises a computer having installed thereon the computer program according to the application.

[0389] A still further embodiment according to the application is a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program according to the application.

[0390] A further embodiment according to the application is a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program according to the application.

[0391] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals can for example be configured to be transferred via a data communication connection, for example via the Internet.

[0392] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

[0393] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0394] A further embodiment according to the application comprises an apparatus or a system configured to transfer (for example, by means of

[0395] In some embodiments, one or more or all of the functions of the methods described herein can be performed by a programmable logic device, for example a field programmable gate array. In some embodiments, the field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

[0396] The apparatus described herein can be implemented using a hardware apparatus, or using a computer, or using a combination of hardware and computer.

[0397] The methods described herein can be performed using a hardware apparatus, or using a computer, or using a combination of hardware and computer.

[0398] The above-described embodiments are merely meant to illustrate the principles of the application. It is appreciated that variations and modifications of the arrangements described herein, and details thereof, will be apparent to those skilled in the art, and it is the intention that the application not be limited by the details of the described embodiments, but by the scope of the claims that follow.

[0399] According to the application, the following can be included:

[0400] 1. A video decoder for decoding a multi-layer video data stream (40) into scenes encoded in a hierarchy of layers using inter-layer prediction from a first layer to a second layer, the video decoder supporting parallel decoding of the multi-layer video data stream in spatial segments (80) into which pictures (12, 15) of the layers are subdivided, wherein the decoder is configured to:

[0401] detect a long-term syntax element structure (606; e.g. tile_boundaries_aligned_flag) of the multi-layer video data stream to

[0402] interpret the long-term syntax element structure assuming a value outside a first set of possible values (e.g. tile_boundaries_aligned_flag = 1) as a guarantee that within a predetermined time period (608) pictures (15) of the second layer are subdivided such that boundaries between spatial segments of the pictures of the second layer overlap each boundary of spatial segments of pictures (12) of the first layer and within a time interval (604) shorter than the predetermined time period periodically determine from short-term syntax elements (602; e.g. column_width_minus1[i] and column_width_minus1[i]) of the multi-layer video data stream how the pictures of the first layer and the pictures of the second layer are subdivided into spatial segments; and

[0403] if the long-term syntax element structure assumes a value outside a second set of possible values (e.g. tile_boundaries_aligned_flag = 0), periodically determine from the short-term syntax elements of the multi-layer video data stream within a time interval shorter than the predetermined time period how the pictures of the layers are subdivided into spatial segments such that at least for a first possible value of the short-term syntax elements there are boundaries between spatial segments of the pictures of the second layer that do not overlap any of the boundaries of spatial segments of the first layer and at least for a second possible value of the short-term syntax elements there are boundaries between spatial segments of the pictures of the second layer that overlap each boundary of spatial segments of the first layer.

[0404] 2. The video decoder according to item 1, wherein the video decoder is configured to:

[0405] decode the pictures of the layers using intra-picture spatial prediction and the intra-picture spatial prediction for individual spatial segments is interrupted at the boundary lines of the respective spatial segment; or

[0406] decoding the pictures of the layers using the intra-picture spatial prediction by decoding spatial segments of the pictures of the first layer in parallel and supporting the intra-picture spatial prediction across boundary lines of the spatial segments of the pictures of the first layer and subject to a decoding delay between decodings of the spatial segments of the pictures of the first layer; and by decoding spatial segments of the pictures of the second layer in parallel and supporting the intra-picture spatial prediction across boundary lines of the spatial segments of the pictures of the second layer and subject to a decoding delay between decodings of the spatial segments of the pictures of the second layer.

[0407] 3. The video decoder according to clause 1 or 2, supporting slice-parallel decoding of the multi-layer video data stream in slices into which the pictures of the layers are subdivided, wherein the decoder is configured to:

[0408] interpreting the long-term syntax element assuming a value outside the first set of possible values as a guarantee that the subdivision of the pictures of the second layer into slices is periodic within a predetermined time period and based on the short-term syntax element within a time interval shorter than the predetermined time period; and

[0409] if the long-term syntax element assumes a value outside the second set of possible values, determining from the short-term syntax element of the multi-layer video data stream within a time interval shorter than the predetermined time period a subdivision of the pictures of the layers into slices such that, at least for a first possible value of the short-term syntax element, there is a boundary between slices of the pictures of the second layer that does not overlap any of the boundaries of the slices of the first layer, and, at least for a second possible value of the short-term syntax element, there is a boundary between slices of the pictures of the second layer that overlaps each of the boundaries of the slices of the first layer.

[0410] 4. The video decoder according to clause 3, wherein the video decoder is configured to:

[0411] decoding the pictures of the layers using the intra-picture spatial prediction, and the intra-picture spatial prediction for individual slices is interrupted at boundary lines of the respective slice.

[0412] 5. The video decoder according to clause 1 or 2, wherein the decoder is configured to:

[0413] interpreting the long-term syntax element structure assuming a value outside the first set of possible values as a guarantee that the pictures of the second layer are subdivided within a predetermined time period such that each spatial segment of the pictures of the first layer is exactly composed of n spatial segments of the pictures of the second layer, and n depends on the value of the long-term syntax element structure; and

[0414] if the long-term syntax element is set to a value outside the second set of possible values, periodically determining an inter-layer offset within the predetermined time period based on the short-term syntax elements of the multi-layer video data stream within a time interval shorter than the predetermined time period.

[0415] 6. The video decoder according to one of the items 1 or 2, wherein the decoder is configured to decide to start or not to start trying to decode the second layer of the multi-layer video data stream depending on the long-term syntax element structure whether a value outside the first possible value is assumed or not.

[0416] 7. The video decoder according to any one of the items 1 to 6, wherein the video decoder is a hybrid video decoder.

[0417] 8. A video encoder for encoding a scene into a multi-layer video data stream in a hierarchy of layers with inter-layer prediction from a first layer to a second layer such that the multi-layer video data stream can be decoded in parallel in spatial segments into which pictures of the layers are subdivided, wherein the encoder is configured to:

[0418] insert a long-term syntax element structure (606) and short-term syntax elements (602) into the multi-layer video data stream, the short-term syntax elements defining a subdivision of pictures of the first layer and pictures of the second layer into spatial segments within a time interval; and

[0419] switch between setting the long-term syntax element structure to a value between:

[0420] a value outside the first set of possible values and setting the short-term syntax elements to an appropriate subset outside the set of possible settings within a predetermined time period (608) larger than the time interval, the appropriate subset being selected such that the pictures of the second layer are subdivided within the predetermined time period such that boundaries between spatial segments of the pictures of the second layer overlap each boundary of the spatial segments of the first layer; or

[0421] a value outside the second set of possible values, and setting the short-term syntax element to any one of a set of possible settings during the predetermined time period, the set of possible settings including at least one setting and at least another setting, according to the at least one setting there are boundaries between spatial segments of the picture of the second layer that do not overlap any of the boundaries of the spatial segments of the first layer, and according to the at least another setting there are boundaries between spatial segments of the picture of the second layer that overlap each of the boundaries of the spatial segments of the first layer.

[0422] 9. The video encoder according to item 8, wherein the video encoder is configured to:

[0423] encode the pictures of the layer using intra-picture spatial prediction, and the intra-picture spatial prediction for each spatial segment is interrupted at the boundary line of the respective spatial segment; or

[0424] encode the pictures of the layer using the intra-picture spatial prediction across the boundary lines of the spatial segments of the first layer and initializing entropy context probabilities for entropy encoding the spatial segments of the first layer separately, or on the basis of the entropy context probabilities of a preceding subset of spatial segments of the first layer in the intermediate adaptation state according to an order between the subsets; and using the intra-picture spatial prediction across the boundary lines of the spatial segments of the second layer and initializing the entropy context probabilities for entropy encoding the spatial segments of the second layer separately, or on the basis of the entropy context probabilities of a preceding subset of spatial segments of the second layer in the intermediate adaptation state according to an order between the subsets, and using the intra-picture spatial prediction and entropy encoding with adaptation of the entropy context probabilities.

[0425] 10. The video encoder according to item 8 or 9, wherein the spatial segments are tiles, and the encoder is configured to:

[0426] if the long-term syntax element structure is set to:

[0427] a value outside the first set of possible values (tile_boundaries_aligned_flag = 1), setting the short-term syntax element to a proper subset of the set of possible settings during a predetermined time period that is larger than the time interval, the proper subset being selected such that the picture of the second layer that is subdivided into tiles is brought into conformity with the picture of the first layer that is subdivided into tiles, or the picture of the first layer that is subdivided into tiles is refined, during the predetermined time period;

[0428] or

[0429] a value outside the first set of possible values (tile_boundaries_aligned_flag = 0), the short-term syntax element is set to a suitable subset of the set of possible settings using a predetermined time period larger than the time interval, the suitable subset being selected such that within the predetermined time period each spatial segment of the pictures of the first layer is exactly composed of n spatial segments of the pictures of the second layer, and n depends on the value of the long-term syntax element structure.

[0430] 11. The video encoder according to item 10, wherein the video encoder is configured to encode the pictures of the layer with the intra-picture spatial prediction, and the intra-picture spatial prediction for each tile is interrupted at the boundary lines of the respective tile.

[0431] 12. The video encoder according to item 8 or 9, wherein the encoder is configured to:

[0432] if the long-term syntax element structure is set to:

[0433] a value outside the first set of possible values, the short-term syntax element is set to a suitable subset of the set of possible settings using a predetermined time period larger than the time interval, the suitable subset being selected such that within the predetermined time period each spatial segment of the pictures of the first layer is exactly composed of n spatial segments of the pictures of the second layer, and n depends on the value of the long-term syntax element structure.

[0434] 13. A decoder for decoding a spatially scalable bitstream (40) into pictures, the pictures being encoded in different spatial layers and for at least one of the spatial layers the pictures being encoded in a first spatial segment, wherein the decoder is configured to:

[0435] upsample the pictures (12) of the first spatial layer to obtain an upsampled reference picture and predict the pictures (15) of the second spatial layer using the upsampled reference picture, wherein the decoder is responsive to a syntax element (616; e.g. independent_tile_upsampling_idc) in the spatially scalable bitstream to insert (620) the pictures of the first spatial layer according to the syntax element,

[0436] such that any portion of the partition (622) of the upsampled reference picture that depends on the first spatial segment is independent of a portion of a picture of the first spatial layer covered by any of the other portions of the partition; or

[0437] such that any portion of the partition (622) of the upsampled reference picture depends on a portion of a picture of the first spatial layer covered by another portion of the partition that is spatially adjacent to the respective portion.

[0438] 14. The decoder of clause 13, wherein the decoder is configured to decode the different spatial layers in parallel.

[0439] 15. The decoder of clause 13 or 14, wherein the decoder is responsive to the syntax element (616) in the spatially scalable bitstream to insert (620) pictures of the first spatial layer in accordance with the syntax element,

[0440] such that any portion of the upsampled reference picture that is spatially covered by any of the first spatial segments is independent of a portion of a picture of the first spatial layer covered by another of the first spatial segments; or

[0441] such that any portion of the upsampled reference picture that is spatially covered by any of the first spatial segments depends on a portion of a picture of the first spatial segment covered by another of the first spatial segments that is spatially adjacent to the respective spatial segment.

[0442] 16. The decoder of any of clauses 13 to 15, wherein the spatially scalable bitstream has pictures of the second spatial layer encoded into the spatially scalable bitstream in the first spatial segments.

[0443] 17. The decoder of any of clauses 13 to 16, wherein the decoder is configured to perform decoding using intra-picture spatial prediction,

[0444] and the intra-picture spatial prediction for each first spatial segment is interrupted at a boundary line of the respective first spatial segment; or

[0445] and supports the intra-picture spatial prediction across the boundary line of the first spatial segments, entropy decodes each first spatial segment using an adaptivity of entropy context probabilities and initializes the entropy context probabilities of the first spatial segment independently of any other first spatial segment or in accordance with the entropy context probabilities of a preceding first spatial segment to adapt to an intermediate position of the preceding first spatial segment.

[0446] 18. The decoder of clause 13 or 14, wherein the spatially scalable bitstream has pictures in the first spatial segment encoded into the first spatial layer of the spatially scalable bitstream, wherein the spatially scalable bitstream has pictures in a second spatial segment encoded into the second spatial layer of the spatially scalable bitstream, wherein the decoder is responsive to the syntax element in the spatially scalable bitstream (606) to interpolate pictures of the first spatial layer according to the syntax element

[0447] (e.g., independent tile upsampling idc = 2) such that any portion of the upsampled reference picture spatially overlaid by any one of the second tiles is independent of portions of pictures of the first spatial layer spatially overlaid by another of the second tiles; or

[0448] (e.g., independent tile upsampling idc = 1) such that any portion of the upsampled reference picture spatially defined by a spatially co-located boundary line of the first and second tiles is independent of portions of pictures of the first spatial layer covered by another portion of the upsampled reference picture spatially defined by the spatially co-located boundary line of the first and second tiles; or

[0449] (e.g., independent tile upsampling idc = 0) such that any portion of the upsampled reference picture spatially overlaid by any one of the second tiles is dependent on portions of pictures of the first spatial layer covered by another second tile adjacent to the respective tile.

[0450] 19. The decoder of any of clauses 13 to 18, wherein the decoder is configured to fill in segments of a filter kernel used in interpolating pictures of the first spatial layer using a fallback rule, the filter kernel extending into any portion of a picture of the first spatial layer covered by any one of the other partitions of the partition, according to the fallback rule, the segments are filled in independent of the respective portion of the picture of the first spatial layer into which the filter kernel extends to achieve independence.

[0451] 20. The decoder of clause 19, wherein the decoder is configured to also use the fallback rule in filling in segments of the filter kernel extending outside boundary lines of pictures of the first spatial layer.

[0452] 21. The decoder according to any of the clauses 13 to 20, wherein the decoder is a video decoder and the decoder is configured to respond to the syntax element (606) in the spatially scalable bitstream on a per picture or per sequence of pictures basis.

[0453] 22. The decoder according to any of the clauses 13 to 21, wherein the spatially scalable bitstream has pictures of the first spatial layer encoded into the spatially scalable bitstream in the first spatial segment, wherein the spatially scalable bitstream has pictures of the second spatial layer encoded into the spatially scalable bitstream in a second spatial segment, wherein the boundaries of the partitions correspond to a logical AND of the boundaries of the first spatial segment and the second spatial segment or a spatial overlap of the boundaries of the second spatial segment, wherein the decoder responds to the syntax element (606) in the spatially scalable bitstream according to the syntax element to fill segments of filter kernels used in interpolation of pictures of the first spatial layer, the filter kernels using a fallback rule to extrapolate from one partition into a neighboring partition of the partition, the segments being filled according to the fallback rule independent of or using a respective portion of a picture of the first spatial layer into which the filter kernel extrapolates.

[0454] 23. The decoder according to any of the clauses 13 to 22, wherein the decoder is configured to decode the first layer and the second layer in parallel using an inter-layer offset dependent on the syntax element (606).

[0455] 24. The decoder according to clause 13 or 23, wherein the decoder is configured to place the boundaries of the partitions to correspond to a logical AND of the boundaries of the first spatial segment and the second spatial segment or a spatial overlap of the boundaries of the second spatial segment according to the syntax element.

[0456] 25. An encoder for encoding pictures into a spatially scalable bitstream in different spatial layers and encoding the pictures in a first spatial segment for at least one of the spatial layers, wherein the encoder is configured to:

[0457] upsample pictures of a first spatial layer to obtain upsampled reference pictures and predict pictures of a second spatial layer using the upsampled reference pictures, wherein the encoder is configured to set and insert a syntax element (606) into the spatially scalable bitstream according to the syntax element and interpolate pictures of the first spatial layer,

[0458] such that any portion of the partition of the upsampled reference picture that depends on the first spatial layer's picture portion covered by any of the first spatial segments is independent of the first spatial layer's picture portion covered by any other of the first spatial segments; or

[0459] such that any portion of the partition of the upsampled reference picture that depends on the first spatial layer's picture portion covered by another portion of the partition that is spatially adjacent to the respective portion.

[0460] 26. The decoder of clause 25, wherein the encoder is configured to set and insert the syntax element into the spatially scalable bitstream and interpolate the first spatial layer's pictures in dependence on the syntax element,

[0461] such that any portion of the upsampled reference picture that is spatially covered by any of the first spatial segments is independent of the first spatial layer's picture portion covered by any other of the first spatial segments; or

[0462] such that any portion of the upsampled reference picture that is spatially covered by any of the first spatial segments depends on the first spatial layer's picture portion covered by another one of the first spatial segments that is spatially adjacent to the respective first spatial segment.

[0463] 27. The encoder of clause 25 or 26, wherein the encoder is configured to encode the first spatial layer's pictures into the spatially scalable bitstream in the first spatial segments.

[0464] 28. The encoder of clause 27, wherein the encoder is configured to encode the first spatial layer's pictures using intra-picture spatial prediction, and the intra-picture spatial prediction for each first spatial segment is interrupted at the respective first spatial segment's boundary line.

[0465] 29. The encoder of clause 27 or 28, wherein the encoder is configured to encode the second spatial layer's pictures into the spatially scalable bitstream in second spatial segments, wherein the encoder is configured to set and insert the syntax element into the spatially scalable bitstream and interpolate the first spatial layer's pictures in dependence on the syntax element,

[0466] such that any portion of the upsampled reference picture that is spatially covered by any of the second spatial segments is independent of the first spatial layer's picture portion spatially covered by another one of the second spatial segments; or

[0467] such that any portion of the upsampled reference picture spatially defined by a spatially co-located boundary line of the first spatial segment and the second spatial segment is independent of a portion of the picture of the first spatial layer spatially defined by a spatially co-located boundary line of the first spatial segment and the second spatial segment that is covered by another partition of the upsampled reference picture; or

[0468] such that any portion of the upsampled reference picture spatially covered by any one of the second spatial segments is dependent on a portion of the picture of the first spatial layer covered by another second spatial segment adjacent to the respective spatial segment.

[0469] 30. The encoder of clause 29, wherein the encoder is configured to encode the pictures of the second spatial layer using intra-picture spatial prediction, and the intra-picture spatial prediction for individual second spatial segments is interrupted at the boundary lines of the respective second spatial segments.

[0470] 31. The encoder of any one of clauses 25 to 30, wherein the encoder is configured to use padding segments of a filter kernel in interpolating the pictures of the first spatial layer to achieve the independence from any portion of the pictures of the first spatial layer, the filter kernel reaching into any portion of the pictures of the first spatial layer using a fallback rule according to which the segments are padded independent of the respective portion of the pictures of the first spatial layer into which the filter kernel reaches.

[0471] 32. The encoder of clause 31, wherein the encoder is configured to use the fallback rule also in padding segments of the filter kernel that reach outside boundary lines of the pictures of the first spatial layer.

[0472] 33. The encoder of any one of clauses 25 to 32, wherein the encoder is a video encoder and the encoder is configured to set and insert the syntax elements into the spatially scalable bitstream on a per-picture or per-picture sequence basis.

[0473] 34. A video decoder for decoding a multi-layer video data stream into a scene encoded in a hierarchy of layers using inter-layer prediction from a first layer to a second layer, the video decoder exploiting an inter-layer delay between a traversal of spatial segments of a picture of the first layer relative to a traversal of spatial segments of a picture of the second layer to sequentially traverse spatial segments supported in spatial segments partitioning the pictures of the layers in a time-overlapping manner to decode the multi-layer video data stream in parallel, the video decoder being configured to:

[0474] detecting a long-term syntax element structure (642; e.g., min_spatial_segment_delay) of the multi-layer video data stream such that:

[0475] if the long-term syntax element structure (e.g., min_spatial_segment_delay ≠ 0) is set to a value of a first set of possible values, the value of the long-term syntax element determines a layer offset for a predetermined time period in advance and the size and positioning of spatial segments of pictures of the first layer and of pictures of the second layer and the spatial sampling resolution of pictures of the first layer and of pictures of the second layer are determined periodically within a time interval shorter than the predetermined time period based on the short-term syntax elements (602) of the multi-layer video data stream, respectively;

[0476] if the long-term syntax element is set to a value of a second set of possible values (e.g., min_spatial_segment_delay = 0) disjoint from the first set of possible values, the layer offset for the predetermined time period is determined periodically within a time interval shorter than the predetermined time period based on the short-term syntax elements of the multi-layer video data stream.

[0477] 35. The video decoder of clause 34, wherein the video decoder is configured to parallel decode the multi-layer video data stream into sub-streams of tiles of pictures of the layers partitioned by tiles and of rows of blocks of pictures of the layers regularly subdivided by tiles in a time-overlapping manner sequentially traversing wavefronts of sub-streams in parallel with an intra-picture sub-stream inter-delay between successive consecutive wavefronts of the same picture and a layer offset of a traversal of a sub-stream of pictures of the first layer relative to a traversal of a sub-stream of pictures of the second layer.

[0478] 36. The video decoder of clause 35, wherein the video decoder is configured to:

[0479] parallel decode the sub-streams and support the intra-picture spatial prediction across the boundary lines of the sub-streams.

[0480] 37. The video decoder of clause 34, wherein the video decoder is configured to decode the multi-layer video data stream into slices of pictures of the layers partitioned by slices and to traverse the slices in a slice order between the slices within each of the pictures of the first layer and of the second layer and to parallel decode successive consecutive slices of pictures of the first layer and of the second layer and consecutive slices of pictures of the layers partitioned by slices with a layer offset of a traversal of a slice of a picture of the first layer relative to a traversal of a slice of a picture of the second layer.

[0481] 38. The video decoder according to item 37, wherein the video decoder is configured to:

[0482] decode the pictures of the first layer and the pictures of the second layer using intra picture spatial prediction, and the intra picture spatial prediction for each slice is interrupted at the boundary line of the respective slice.

[0483] 39. The video decoder according to any one of items 34 to 38, wherein the video decoder is configured to use the value of the long-term syntax element when determining the inter-layer offset by using the value of the long-term syntax element as a measure for the inter-layer offset in the unit of a spatial segment of a picture of the first layer.

[0484] 40. The video decoder according to any one of items 34 to 39, wherein the video decoder is configured to use the value of the long-term syntax element when determining the inter-layer offset by using the value of the long-term syntax element as a number of spatial segments of a picture of the first layer by which a first spatial segment of a picture of the second layer is delayed with respect to starting the decoding and traversal of the picture of the first layer.

[0485] 41. A video encoder for encoding a scene encoded in a hierarchy of layers using inter-layer prediction from a first layer to a second layer into a multi-layer video data stream such that a parallel decoding of the multi-layer video data stream in spatial segments into which pictures of the layers are partitioned by sequentially traversing the spatial segments in a time-overlapping manner is enabled with an inter-layer offset between a traversal of spatial segments of the picture of the first layer and a traversal of spatial segments of the picture of the second layer, the video encoder is configured to:

[0486] set a long-term syntax element structure (min_spatial_segment_delay) and a short-term syntax element and insert the long-term syntax element structure and the short-term syntax element into the multi-layer video data stream, the short-term syntax element of the multi-layer video data stream periodically defining a size and positioning of spatial segments of the picture of the first layer and of the picture of the second layer within a time interval and a spatial sampling resolution of the picture of the first layer and of the picture of the second layer, respectively;

[0487] wherein the encoder is configured to switch between the following settings:

[0488] setting the long-term syntax element to a value in a first set of possible values (min_spatial_segment_delay ≠ 0) and signaling the inter-layer offset for a predetermined time period larger than the time interval and setting the short-term syntax element to an appropriate subset outside the set of possible settings for the predetermined time period, the appropriate subset being selected such that an actual inter-layer offset between traversals of spatial segments of pictures of the first layer with respect to traversals of spatial segments of pictures of the second layer, smaller than or equal to the inter-layer offset signaled by the long-term syntax element, is enabled by traversing spatial segments of pictures of the first layer and of the second layer in a time-overlapped manner in sequence, respectively, for decoding the multi-layer video data stream, the size and positioning of the spatial segments of pictures of the first layer and of the second layer and the spatial sampling resolution of the pictures of the first layer and of the second layer, respectively, supporting the decoding of the multi-layer video data stream by traversing spatial segments in a time-overlapped manner in sequence;

[0489] setting the long-term syntax element to a value in a second set of possible values disjoint from the first set of possible values (min_spatial_segment_delay = 0) and setting the short-term syntax element to any one of a set of possible settings for the predetermined time period, the set of possible settings including at least one setting and at least another setting, according to the at least one setting, an actual inter-layer offset between traversals of spatial segments of pictures of the first layer with respect to traversals of spatial segments of pictures of the second layer, smaller than or equal to the inter-layer offset signaled by the long-term syntax element, disabling the decoding of the multi-layer video data stream by traversing spatial segments of pictures of the first layer and of the second layer in a time-overlapped manner in sequence, respectively, and according to the at least another setting, an actual inter-layer offset between traversals of spatial segments of pictures of the first layer with respect to traversals of spatial segments of pictures of the second layer, smaller than or equal to the inter-layer offset signaled by the long-term syntax element, enabling the decoding of the multi-layer video data stream by traversing spatial segments of pictures of the first layer and of the second layer in a time-overlapped manner in sequence, respectively.

[0490] 42. The video encoder according to item 41, wherein the video encoder is configured to perform encoding such that the spatial segments are sub-streams of pictures of the layers partitioned and the sub-streams are constituted by lines of blocks into which pictures of the layers are regularly subdivided, in this way allowing the multi-layer video data stream to be decoded into the sub-streams in parallel using wavefront parallel processing by sequentially traversing the sub-streams in a time-overlapping manner with an intra-picture sub-stream delay between traversals of intermediate consecutive sub-streams of the same picture and the inter-layer offset between traversals of sub-streams of pictures of the first layer relative to sub-streams of pictures of the second layer.

[0491] 43. The video encoder according to item 42, wherein the video encoder is configured to use the intra-picture spatial prediction across boundary lines of the sub-streams and entropy context probabilities for entropy encoding the sub-streams individually or encoding the sub-streams using the intra-picture spatial prediction on the basis of the entropy context probabilities of a preceding sub-stream in an intermediate adaptation state according to the order between the sub-streams and with adapted entropy context probabilities.

[0492] 44. The video encoder according to item 41, wherein the video encoder is configured to perform encoding such that the spatial segments are tiles of pictures of the layers, in this way allowing the multi-layer video data stream to be decoded into the tiles by traversing the tiles in a tile order between the tiles within each of the pictures of the first and second layers and with the inter-layer offset between traversals of tiles of pictures of the first layer relative to traversals of tiles of the pictures of the second layer allowing a following consecutive tile of a picture of the first layer to be decoded in parallel with a following consecutive tile of a picture of the second layer.

[0493] 45. The video encoder according to item 44, wherein the video encoder is configured to encode pictures of the first layer with pictures of the second layer using intra-picture spatial prediction, the intra-picture spatial prediction for individual tiles being interrupted at boundary lines of the respective tile.

[0494] 46. The video encoder according to any one of items 41 to 45, wherein the video encoder is configured such that the value of the long-term syntax element defines a measure of the inter-layer offset in units of spatial segments of pictures of the first layer.

[0495] 47. The video decoder according to any of the items 41 to 46, wherein the video encoder is configured to set a value of the long-term syntax element to signal a number of spatial segments of a picture of the first layer by which a decoding of a first spatial segment of the picture of the second layer is delayed relative to a start of a decoding of the picture of the first layer and a traversal.

[0496] 48. A network entity for processing a multi-layer video data stream into a scene, the scene being encoded in layers such that in each layer, the scene is encoded in different operation points of a scalable space spanned by a scalable dimension, wherein the multi-layer video data stream is composed of first NAL units and second NAL units, each of the first NAL units being associated with one of the layers, and the second NAL units being interspersed within the first NAL units and presenting global information about the multi-layer video data stream, the network entity being configured to:

[0497] detect a type indicator field (609; e.g. dedicated_scalability_ids_flag) in the second NAL units;

[0498] if the type indicator field has a first state (e.g. dedicated_scalability_ids_flag = 0), read from the second NAL units mapping information (e.g. layer_id_in_nuh[i], dimension_id[i][j]) that maps possible values of a layer indicator field (e.g. layer_id) in the first NAL units to the operation points and associate the first NAL units with the operation points in the first NAL units by the layer indicator field and the mapping information;

[0499] if the type indicator field has a second state (dedicated_scalability_ids_flag = 1), associate the first NAL units with the operation points in the first NAL units by splitting the layer indicator field in the first NAL units into more than one part and locating the operation points of the first NAL units by using values of the parts as coordinates of a vector within the scalable space.

[0500] 49. The network entity according to item 48, wherein the network entity is configured to:

[0501] If the type indicator field has the second status (dedicated_scalability_ids_flag = 1), the first NAL unit is associated with the operation point by dividing the layer indicator field in the first NAL unit into more than one part according to a syntax element (dimension_id_len_minus1) in the second NAL unit, by using the value of the part as a coordinate of a vector within the scalable space to locate the operation point, and by semantically determining the scalable dimensions from another syntax element (scalability_mask) in the second NAL unit.

[0502] 50. The network entity of clause 48 or 49, wherein the network entity is configured to:

[0503] If the type indicator field has the first status (dedicated_scalability_ids_flag = 0), the number p and the semantic meaning of the scalable dimensions are determined from another syntax element (scalability_mask) in the second NAL unit and the possible values of the layer indicator field are associated with the operation point by reading a list of p-dimensional vectors from the second NAL unit (708).

[0504] 51. The network entity of clause 50, wherein the network entity is configured to skip reading the list from the second NAL unit if the type indicator field has the second status.

[0505] 52. The network entity of any of clauses 49 or 51, wherein the network entity is configured to read the another syntax element from the second NAL unit irrespective of the type indicator field having the first status or the second status; and such that the size of the layer indicator field is the same irrespective of the type indicator field having the first status or the second status.

[0506] 53. The network entity of any of clauses 48 or 52, wherein the network entity comprises a video decoder.

[0507] 54. A video encoder for encoding a scene into a multi-layer video data stream in layers such that in each layer, the scene is encoded in different operation points of a scalable space spanned by a scalable dimension, wherein the multi-layer video data stream is composed of first NAL units and second NAL units, each of the first NAL units being associated with one of the layers and the second NAL units being interspersed within the first NAL units and presenting global information about the multi-layer video data stream, the video encoder being configured to:

[0508] inserting a type indicator field into the second NAL units and switching between the following settings:

[0509] setting the type indicator field such that the type indicator field has a first status and mapping information of the mapping of possible values of a layer indicator field in a first NAL unit header to the operation points is inserted into the second NAL units and setting the layer indicator field in the first NAL units such that the operation points of the first NAL units are associated with the respective layer indicator field by the mapping information;

[0510] setting the type indicator field such that the type indicator field has a second status (dedicated_scalability_ids_flag = 1) and setting the layer indicator field in the first NAL units by splitting the layer indicator field in the first NAL units into more than one part and setting the more than one part such that the values of the part correspond to coordinates of a vector within the scalable space pointing to the operation points associated with the respective first NAL unit.

[0511] 55. The video encoder according to item 54, wherein the video encoder is configured to:

[0512] when setting the type indicator field such that the type indicator field has the second status, setting and inserting syntax elements into the second NAL units, the syntax elements defining the division of the type indicator field in the first NAL units into the more than one part, and setting and inserting further syntax elements into the second NAL units, the further syntax elements semantically defining the scalable dimension.

[0513] 56. A method of converting a multi-layer video data stream into a scene encoded in layers, such that in each layer, the scene is encoded in different operating points of a scalable space spanned by a scalable dimension, wherein the multi-layer video data stream is composed of first NAL units and second NAL units, each of the first NAL units being associated with one of the layers, and the second NAL units being interspersed within the first NAL units and presenting global information about the multi-layer video data stream, wherein a type indicator field (696; e.g. dedicated_scalability_ids_flag) is presented in the second NAL units according to the following conditions:

[0514] if the type indicator field has a first state (e.g. dedicated_scalability_ids_flag = 0), the mapping information in the second NAL units maps the possible values of a layer indicator field (e.g. layer_id) in the first NAL unit header to the operating points;

[0515] if the type indicator field has a second state (dedicated_scalability_ids_flag = 1), the layer indicator field in the first NAL units is split into more than one part, and the operating points of the first NAL units are defined by the values of this part as coordinates of a vector within the scalable space.

[0516] 57. A transport layer decoder for transforming a multi-layer video data stream into a scene encoded in layers to be decoded by a multi-standard multi-layer decoder, wherein the multi-layer video data stream is composed of NAL units, each of the NAL units being associated with one of the layers, wherein the layers are associated with different codecs, such that for each layer the NAL units associated with the respective layer are encoded using the codec associated with the respective layer, the transport layer decoder being configured to:

[0517] for each NAL unit, identify the same codec associated with the NAL unit, and hand over the NAL units of the multi-layer video data stream to the multi-standard multi-layer decoder, the multi-standard multi-layer decoder decoding the multi-layer video data stream using inter-layer prediction between layers associated with different codecs.

[0518] 58. The video decoder of item 57, further configured to:

[0519] NAL unit headers of a predetermined codec using a NAL unit type indicator set to indicate the state of the codec of the respective layer encapsulate NAL units of layers identified as being associated with any codec different from the predetermined codec.

[0520] 59. The video decoder according to any one of the items 57 or 58, further configured to:

[0521] performing the identification according to channels on which the NAL units respectively arrive.

[0522] 60. The video decoder according to any one of the items 57 or 59, further configured to perform a handover such that the NAL units associated with different codecs are handed over to the multi-standard multi-layer decoder on different channels.

[0523] 61. The video decoder according to any one of the items 57 or 60, further configured to:

[0524] providing each NAL unit with metadata indicating the codec associated with the layer associated with the respective NAL unit.

[0525] 62. A video decoder for decoding a multi-layer video data stream into a scene, the scene being encoded in a hierarchy of layers using inter-layer prediction from a portion of a first layer to a co-located portion of a second layer, wherein a picture of the first layer is subdivided into an array of first blocks and a picture of the second layer is subdivided into an array of second blocks, wherein a raster scan decoding order is defined in the first blocks and the second blocks, respectively, the video decoder being configured to:

[0526] determine an inter-layer offset for decoding a picture of the first layer in parallel with a picture of the second layer from a syntax element structure (e.g. ctb_delay_enabled_flag, min_spatial_segment_delay) of the multi-layer video data stream in a time-overlapping manner sequentially traversing the first blocks and the second blocks with a spatial traversal of the first blocks relative to the second blocks with the inter-layer offset measured in units of the first blocks.

[0527] 63. The video decoder according to item 62, wherein the syntax element structure is a long-term syntax element structure and the video decoder is configured to:

[0528] performing the determination in advance for a predetermined time period; and

[0529] determining the size and position of the first block of a picture of the first layer and the second block of a picture of the second layer and the spatial sampling resolution of the picture of the first layer and the picture of the second layer based on short-term syntax elements of the multi-layer video data stream in time intervals shorter than the predetermined time period, respectively.

[0530] 64. The video decoder of clause 63, wherein the video decoder supports parallel decoding of the multi-layer video data stream into pictures of the partitioned layers in a time- overlapping manner by sequentially traversing spatial segments with respect to the inter-layer offset of the traversal of spatial segments of the picture of the first layer relative to the traversal of spatial segments of the picture of the second layer, the video decoder configured to:

[0531] detecting the long-term syntax element structure of the multi-layer video data stream such that:

[0532] if the long-term syntax element structure is set to a value of a first set of possible values (e.g., ctb_delay_enabled_flag = 0, min_spatial_segment_delay ≠ 0), the value of the long-term syntax element structure pre-determines the inter-layer offset for a predetermined time period, the inter-layer offset being measured in units of spatial segments of the picture of the first layer, and determining the size and position of spatial segments of the picture of the first layer and spatial segments of the picture of the second layer and the spatial sampling resolution of the picture of the first layer and the picture of the second layer based on short-term syntax elements of the multi-layer video data stream in time intervals shorter than the predetermined time period, respectively;

[0533] if the long-term syntax element structure is set to a value of a second set of possible values disjoint from the first set of possible values (e.g., min_spatial_segment_delay = 0), determining the inter-layer offset for the predetermined time period based on the short-term syntax elements of the multi-layer video data stream in time intervals shorter than the predetermined time period; and

[0534] if the long-term syntax element is set to a value of a third set of possible values disjoint from the first and second sets of possible values (e.g., ctb_delay_enabled_flag = 1, min_spatial_segment_delay ≠ 0), performing determining the inter-layer offset in units of the first block and determining the size and position of the first block of a picture of the first layer and the second block of a picture of the second layer and the spatial sampling resolution of the picture of the first layer and the picture of the second layer, respectively.

[0535] 65. The video decoder according to any one of the items 63 or 64, wherein the video decoder utilizes an intra-picture substream inter delay between traversals of intermediate consecutive substreams of the same picture and a layer offset between traversals of substreams of pictures of the first layer relative to traversals of substreams of pictures of the second layer to sequentially traverse substreams in a time-overlapping manner to support parallel processing of wavefronts for parallel decoding of the multi-layer video data stream in substreams partitioned from the pictures of the layers and composed of the rows of the first and second blocks.

[0536] 66. The video decoder according to any one of the items 64 to 65, wherein the long-term syntax element structure comprises a unit flag (e.g., exemplarily, ctb_delay_enabled_flag) and a delay indicator (e.g., exemplarily, min_spatial_segment_delay), wherein, upon detecting the long-term syntax element structure, the video decoder is configured to:

[0537] detect the delay indicator to determine whether the delay indicator is set to zero;

[0538] if the delay indicator is set to zero, determine to set a value of the long-term syntax element structure to a value in the second set; and

[0539] if the delay indicator is set to a non-zero value, determine the value of the long-term syntax element structure using the non-zero value, and if the unit flag is zero, determine to set the value of the long-term syntax element structure to a value in the first set, and if the unit flag is one, determine to set the value of the long-term syntax element structure to a value in the third set.

[0540] 67. The video decoder according to any one of the items 65 to 66, the video decoder being configured to rely on the layer offset upon starting parallel decoding of the first and second layers.

[0541] 68. The video decoder according to any one of the items 62 to 67, the video decoder being configured to check whether spatial segments or coding blocks of the first layer of a number s are completely decoded, and the number s depends uniquely on the syntax element structure, and to postpone starting decoding of the second layer during decoding of the first layer except in case the check shows that at least s of the spatial segments or coding blocks of the first layer have been completely decoded.

[0542] 69. The video decoder according to any one of claims 62 to 68, configured to rely on the inter-layer offset when starting and fully processing parallel decoding of the first layer and the second layer.

[0543] 70. The video decoder according to any one of claims 62 to 69, configured to check whether a number s of spatial segments or coding blocks of the first layer are fully decoded, and the number s is uniquely dependent on the syntax element structure and a number t-1 of decoded spatial segments or coding blocks of the second layer, and to postpone starting decoding of a t-th spatial segment or coding block of the second layer during decoding of the first layer, in case the check shows that at least s are fully decoded among the spatial segments or coding blocks of the first layer.

[0544] 71. A method for decoding a multi-layer video data stream (40) into scenes encoded in a hierarchy of layers using inter-layer prediction from a first layer to a second layer, a video decoder supporting parallel decoding of the multi-layer video data stream in spatial segments (80) into which pictures (12, 15) of the layers are subdivided, wherein the method comprises:

[0545] detecting a long-term syntax element structure (606; e.g. tile_boundaries_aligned_flag) of the multi-layer video data stream to:

[0546] interpreting the long-term syntax element structure assuming a value outside a first set of possible values (e.g. tile_boundaries_aligned_flag = 1) as a guarantee that within a predetermined time period (608) pictures (15) of the second layer are subdivided such that boundaries between spatial segments of the pictures of the second layer overlap each boundary of spatial segments of pictures (12) of the first layer, and within a time interval (604) shorter than the predetermined time period periodically determine based on short-term syntax elements (602; e.g. column_width_minus1[i] and column_width_minus1[i]) of the multi-layer video data stream how the pictures of the first layer and the pictures of the second layer are subdivided into spatial segments; and

[0547] if the long-term syntax element structure assumes a value outside the first set of possible values (e.g., tile_boundaries_aligned_flag = 1), then the short-term syntax elements are periodically determined from the multi-layer video data stream within a predetermined time period (608) that is shorter than the time interval, such that at least for a first possible value of the short-term syntax element, there are boundaries between spatial segments of pictures of the second layer that do not overlap any of the boundaries of spatial segments of the first layer, and at least for a second possible value of the short-term syntax element, there are boundaries between spatial segments of pictures of the second layer that overlap each of the boundaries of spatial segments of the first layer.

[0548] 72. A method for encoding a scene in a hierarchy of layers into a multi-layer video data stream with inter-layer prediction from a first layer to a second layer such that the multi-layer video data stream can be decoded in parallel in spatial segments into which pictures of the layers are subdivided, wherein the method comprises:

[0549] inserting a long-term syntax element structure (606) and short-term syntax elements (602) into the multi-layer video data stream, the short-term syntax elements defining a subdivision of pictures of the first layer and pictures of the second layer into spatial segments within a time interval; and

[0550] switching between setting the long-term syntax element structure to a value between:

[0551] a value outside the first set of possible values and setting the short-term syntax elements to an appropriate subset of the set of possible settings within a predetermined time period (608) that is greater than the time interval, the appropriate subset being selected such that within the predetermined time period the pictures of the second layer are subdivided such that boundaries between spatial segments of pictures of the second layer overlap each of the boundaries of spatial segments of the first layer; or

[0552] a value outside the second set of possible values and setting the short-term syntax elements to any one of the set of possible settings within the predetermined time period, the set of possible settings including at least one setting according to which there are boundaries between spatial segments of pictures of the second layer that do not overlap any of the boundaries of spatial segments of the first layer and at least another setting according to which there are boundaries between spatial segments of pictures of the second layer that overlap each of the boundaries of spatial segments of the first layer.

[0553] 73. A method for decoding a spatially scalable bitstream (40) into pictures, the pictures being encoded in different spatial layers and, for at least one of the spatial layers, the pictures being encoded in a first spatial segment, wherein the method comprises:

[0554] upsampling a picture (12) of a first spatial layer to obtain an upsampled reference picture and predicting a picture (15) of a second spatial layer using the upsampled reference picture, wherein the method for decoding is responsive to a syntax element (616; e.g., independent_tile_upsampling_idc) in the spatially scalable bitstream to interpolate (620) the picture of the first spatial layer according to the syntax element,

[0555] such that any portion of a partition (622) of the upsampled reference picture that depends on the first spatial segment is independent of a portion of the picture of the first spatial layer covered by any of the other portions of the partition; or

[0556] such that any portion of the partition (622) of the upsampled reference picture depends on a portion of the picture of the first spatial layer covered by another portion of the partition that is spatially adjacent to the respective portion.

[0557] 74. A method for encoding pictures in different spatial layers into a spatially scalable bitstream and, for at least one of the spatial layers, the pictures being encoded in a first spatial segment, wherein the method comprises:

[0558] upsampling a picture of a first spatial layer to obtain an upsampled reference picture and predicting a picture of a second spatial layer using the upsampled reference picture, wherein the method comprises setting a syntax element syntax element and inserting the syntax element (606) into the spatially scalable bitstream and interpolating the picture of the first spatial layer according to the syntax element,

[0559] such that any portion of a partition of the upsampled reference picture that depends on the first spatial segment is independent of a portion of the picture of the first spatial layer covered by any of the other portions of the partition; or

[0560] such that any portion of the partition of the upsampled reference picture depends on a portion of the picture of the first spatial layer covered by another portion of the partition that is spatially adjacent to the respective portion.

[0561] 75. A method for decoding a multi-layer video data stream encoded with inter-layer prediction from a first layer to a second layer into a scene coded in a hierarchy of layers, a video decoder supporting inter-layer delay of traversal of spatial segments of pictures of the first layer relative to traversal of spatial segments of pictures of the second layer, supporting decoding the multi-layer video data stream in parallel into spatial segments into which pictures of the layers are subdivided by sequentially traversing spatial segments in a time overlapping manner, the method comprising:

[0562] detecting a long-term syntax element structure (642; e.g., min_spatial_segment_delay) of the multi-layer video data stream such that:

[0563] if the long-term syntax element structure (e.g., min_spatial_segment_delay ≠ 0) is set to a value of a first set of possible values, using the value of the long-term syntax element to predetermine inter-layer offsets for a predetermined time period and periodically determining the size and positioning of spatial segments of pictures of the first layer and of pictures of the second layer and the spatial sampling resolution of pictures of the first layer and of pictures of the second layer based on short-term syntax elements (602) of the multi-layer video data stream, respectively, for a time interval shorter than the predetermined time period;

[0564] if the long-term syntax element is set to a value of a second set of possible values (e.g., min_spatial_segment_delay = 0) disjoint from the first set of possible values, periodically determining inter-layer offsets for the predetermined time period based on the short-term syntax elements of the multi-layer video data stream for a time interval shorter than the predetermined time period.

[0565] 76. A method for encoding a scene into a multi-layer video data stream with inter-layer prediction from a first layer to a second layer in a hierarchy of layers enabling decoding the multi-layer video data stream in spatial segments into which pictures of the layers are subdivided by sequentially traversing spatial segments in a time overlapping manner with inter-layer offsets of traversal of spatial segments of pictures of the first layer relative to traversal of spatial segments of pictures of the second layer, the method comprising:

[0566] inserting and setting a long-term syntax element structure (min_spatial_segment_delay) and short-term syntax elements into the multi-layer video data stream, the short-term syntax elements of the multi-layer video data stream defining the size and positioning of spatial segments of pictures of the first layer and of pictures of the second layer and the spatial sampling resolution of pictures of the first layer and of pictures of the second layer, respectively, based on periodicity for a time interval;

[0567] wherein the method comprises switching between:

[0568] setting the long-term syntax element structure (min_spatial_segment_delay ≠ 0) to a value of a first set of possible values and signaling the inter-layer offset for a predetermined time period larger than the time interval and setting the short-term syntax elements to an appropriate subset of the set of possible settings for the predetermined time period, the appropriate subset being chosen such that, for the predetermined time period, the size and positioning of spatial segments of pictures of the first layer and of pictures of the second layer and the spatial sampling resolution of pictures of the first layer and of pictures of the second layer respectively enable the decoding of the multi-layer video data stream by sequentially traversing the spatial segments in a time-overlapping manner with an actual inter-layer offset smaller than or equal to the inter-layer offset signaled by the long-term syntax element using a traversal of spatial segments of pictures of the first layer relative to a traversal of spatial segments of pictures of the second layer;

[0569] setting the long-term syntax element to a value of a second set of possible values (min_spatial_segment_delay = 0) disjoint from the first set of possible values and setting the short-term syntax elements to any of the set of possible settings for the predetermined time period, the set of possible settings comprising at least one setting and at least another setting, according to the at least one setting, the size and positioning of spatial segments of pictures of the first layer and of pictures of the second layer and the spatial sampling resolution of pictures of the first layer and of pictures of the second layer respectively disable the decoding of the multi-layer video data stream by sequentially traversing the spatial segments in a time-overlapping manner with an actual inter-layer offset smaller than or equal to the inter-layer offset signaled by the long-term syntax element using a traversal of spatial segments of pictures of the first layer relative to a traversal of spatial segments of pictures of the second layer, and according to the at least another setting, the size and positioning of spatial segments of pictures of the first layer and of pictures of the second layer and the spatial sampling resolution of pictures of the first layer and of pictures of the second layer respectively enable the decoding of the multi-layer video data stream by sequentially traversing the spatial segments in a time-overlapping manner with an actual inter-layer offset smaller than or equal to the inter-layer offset signaled by the long-term syntax element using a traversal of spatial segments of pictures of the first layer relative to a traversal of spatial segments of pictures of the second layer.

[0570] 77. A method for processing a multi-layer video data stream into scenes encoded in layers, so that in each layer, the scenes are encoded in different operating points of a scalability space spanned by scalability dimensions, wherein the multi-layer video data stream is composed of first NAL units and second NAL units, each of the first NAL units being associated with one of the layers, and the second NAL units being interspersed within the first NAL units and presenting global information about the multi-layer video data stream, the method comprising:

[0571] detecting a type indicator field (696; e.g., dedicated_scalability_ids_flag) in the second NAL unit;

[0572] if the type indicator field has a first state (e.g., dedicated_scalability_ids_flag = 0), reading from the second NAL unit mapping information (e.g., layer_id_in_nuh[i], dimension_id[i][j]) that maps possible values of a layer indicator field (e.g., layer_id) in the first NAL units to the operating points and associating the first NAL units with the operating points within the first NAL units by the layer indicator field and the mapping information;

[0573] if the type indicator field has a second state (dedicated_scalability_ids_flag = 1), associating the first NAL units with the operating points by splitting the layer indicator field in the first NAL units into more than one part and locating the operating points of the first NAL units by using the values of the parts as coordinates of a vector within the scalability space.

[0574] 78. A method for encoding scenes into a multi-layer video data stream in layers, so that in each layer, the scenes are encoded in different operating points of a scalability space spanned by scalability dimensions, wherein the multi-layer video data stream is composed of first NAL units and second NAL units, each of the first NAL units being associated with one of the layers, and the second NAL units being interspersed within the first NAL units and presenting global information about the multi-layer video data stream, the method comprising:

[0575] inserting a type indicator field into the second NAL unit and switching between the following settings:

[0576] setting the type indicator field so that the type indicator field has a first state, and mapping the possible values of the layer indicator field in the first NAL unit header to the mapping information of the operation points, inserting the mapping information into the second NAL unit and setting the layer indicator field in the first NAL unit so that the operation points of the first NAL unit are associated by the mapping information with the indicator field of the respective layer;

[0577] setting the type indicator field so that the type indicator field has a second state (dedicated_scalability_ids_flag = 1), and setting the layer indicator field in the first NAL unit by splitting the layer indicator field in the first NAL unit into more than one part, and setting the more than one part so that the value of the part corresponds to a coordinate of a vector within the scalability space, thereby pointing to the operation point associated with the respective first NAL unit.

[0578] 79. A method for converting a multi-layer video data stream into a scene encoded in layers for decoding by a multi-standard multi-layer decoder, wherein the multi-layer video data stream is composed of NAL units, each of the NAL units being associated with one of the layers, wherein the layers are associated with different codecs so that for each layer, the NAL units associated with the respective layer are encoded using the codec associated with the respective layer, the method comprising:

[0579] identifying, for each NAL unit, the same codec associated with the NAL unit;

[0580] and

[0581] handing over the NAL units of the multi-layer video data stream to the multi- standard multi-layer decoder, the multi-standard multi-layer decoder using inter-layer prediction between layers associated with different codecs for decoding the multi-layer video data stream.

[0582] 80. A method for decoding a multi-layer video data stream into a scene, the scene being encoded in a hierarchy of layers using inter-layer prediction from a portion of a first layer to a collocated portion of a second layer, wherein a picture of the first layer is subdivided into an array of first blocks and a picture of the second layer is subdivided into an array of second blocks, wherein a raster scan decoding order is defined by the first blocks and the second blocks, respectively, the method comprising:

[0583] The inter-layer offset is determined from syntax element structures (e.g., ctb_delay_enabled_flag, min_spatial_segment_delay) of the multi-layer video data stream by sequentially traversing the first block and the second block in a time overlapping manner according to the inter-layer offset of a spatial traversal of the first block of the picture of the first layer measured within a unit of the first block relative to the second block of the picture of the second layer.

[0584] 81. A computer program having a program code for performing the method of any one of the items 71 to 80 when the computer program is run on a computer.

Claims

1. A video decoder for decoding a stream of multi-layer video data into a scene, the scene being encoded in a hierarchy of layers using inter-layer prediction from a portion of a first layer to a co-located portion of a second layer, wherein, subdividing the pictures of the first layer into an array of first blocks and subdividing the pictures of the second layer into an array of second blocks, wherein a raster scan decoding order is defined in the first blocks and the second blocks, respectively, the video decoder uses an inter-layer delay between a traversal of the first blocks of a picture of the first layer and a traversal of the second blocks of a picture of the second layer to sequentially traverse the first blocks and the second blocks in a time-overlapped manner to support parallel decoding of the multi-layer video data stream, and the video decoder is configured to: detect a long-term syntax element structure (642) of the multi-layer video data stream such that if the long-term syntax element structure is set to a value of a first set of possible values, a layer offset measured in the first blocks is predetermined for a predetermined time period using a value of the long-term syntax element, and a size and a position of the first blocks of a picture of the first layer and the second blocks of a picture of the second layer and a spatial sampling resolution of the pictures of the first layer and the pictures of the second layer are periodically determined based on short-term syntax elements (602) of the multi-layer video data stream for a time interval shorter than the predetermined time period, respectively.

2. The video decoder of claim 1, wherein, the video decoder is configured to use an intra-picture sub-stream inter-delay between a traversal of successive consecutive sub-streams of the same picture and an inter-layer offset of a traversal of a sub-stream of the pictures of the first layer with respect to a traversal of a sub-stream of the pictures of the second layer to parallel decode the multi-layer video data stream into sub-streams of tiles partitioning the pictures of the layers and blocks of rows regularly subdivided from the pictures of the layers by sequentially traversing wavefronts of the sub-streams in a time-overlapped manner.

3. The video decoder of claim 2, wherein, the video decoder is configured to: parallel decode the sub-streams and support the intra-picture spatial prediction across boundary lines of the sub-streams.

4. The video decoder of claim 1, wherein, the video decoder is configured to parallel decode the multi-layer video data stream into tiles partitioning the pictures of the layers and to sequentially traverse the tiles within each of the pictures of the first layer and the second layer in a tile order between the tiles, and to parallel decode successive consecutive tiles of the pictures of the first layer and the pictures of the second layer and consecutive tiles partitioning the pictures of the layers using an inter-layer offset of a traversal of the tiles of the pictures of the first layer with respect to a traversal of the tiles of the pictures of the second layer.

5. The video decoder of claim 4, wherein, the video decoder is configured to decode the pictures of the first layer and the pictures of the second layer using the intra-picture spatial prediction, and the intra-picture spatial prediction for respective tiles is interrupted at a boundary line of the respective tile.

6. The video decoder according to any one of claims 1 to 5, wherein, the video decoder is configured to use a value of the long-term syntax element when determining the inter-layer offset by using the value of the long-term syntax element structure as a number of the first blocks of a picture of the first layer, by which a decoding and traversal of the second blocks of a picture of the second layer is delayed with respect to starting the decoding and traversal of the picture of the first layer.

7. A video encoder for encoding a scene into a multi-layer video data stream in a hierarchy of layers using inter-layer prediction from a portion of a first layer to a collocated portion of a second layer, wherein, subdividing the pictures of the first layer into an array of first blocks and subdividing the pictures of the second layer into an array of second blocks, wherein a raster scan decoding order is defined in the first blocks and in the second blocks, respectively, the video encoder is configured to: setting a long-term syntax element structure and short-term syntax elements and inserting the long-term syntax element structure and the short-term syntax elements into the multi-layer video data stream, the short-term syntax elements of the multi-layer video data stream periodically defining, within a time interval, a size and a position of first blocks of pictures of the first layer and second blocks of pictures of the second layer and a spatial sampling resolution of the pictures of the first layer and of the pictures of the second layer, respectively, and, if the long-term syntax element structure is set to a value of a first set of possible values, the value signaling, over a predetermined time period which is larger than the time interval, an inter-layer delay between a traversal of the first blocks of pictures of the first layer and a traversal of the spatial second blocks of pictures of the second layer measured in units of the first blocks, signaling, in a time-overlapping manner, a layer- wise offset of a sequential traversal of the first blocks and of the second blocks for decoding the pictures of the first blocks and of the second blocks in parallel, and setting the short-term syntax elements to an appropriate subset outside of a set of possible settings for the predetermined time period, the appropriate subset being selected such that, within the predetermined time period, an actual inter-layer offset between the traversal of the first blocks of pictures of the first layer and the traversal of the second blocks of pictures of the second layer, which is smaller than or equal to the inter-layer offset signaled by the long-term syntax element structure, a size and a position of the first blocks of pictures of the first layer and of the second blocks of pictures of the second layer and a spatial sampling resolution of the pictures of the first layer and of the pictures of the second layer, respectively, can be supported for decoding the multi-layer video data stream by sequentially traversing the first blocks and the second blocks in a time-overlapping manner.

8. A method of decoding a multi-layer video data stream in a hierarchy of layers that is encoded for a layer-inter prediction scenario using from a portion of a first layer to a collocated portion of a second layer, wherein, subdividing the pictures of the first layer into an array of first blocks and subdividing the pictures of the second layer into an array of second blocks, wherein a raster scan decoding order is defined in the first blocks and in the second blocks, respectively, the method comprising decoding the multi-layer video data stream in parallel by sequentially traversing the first blocks and the second blocks in a time-overlapping manner with an inter-layer delay between a traversal of the first blocks of pictures of the first layer and a traversal of the second blocks of pictures of the second layer, and detecting a long-term syntax element structure of the multi-layer video data stream (642) such that: if the long-term syntax element structure is set to a value of a first set of possible values, using the value of the long-term syntax element structure to predetermine an inter-layer offset measured in units of the first blocks for a predetermined time period, and periodically determining, within a time interval which is shorter than the predetermined time period, a size and a position of first blocks of pictures of the first layer and of second blocks of pictures of the second layer and a spatial sampling resolution of the pictures of the first layer and of the pictures of the second layer, respectively, based on short-term syntax elements (602) of the multi-layer video data stream.

9. A method for encoding a scene into a multi-layer video data stream in a hierarchy of layers using inter-layer prediction from a portion of a first layer to a collocated portion of a second layer, wherein, The pictures of the first layer are subdivided into an array of first blocks and the pictures of the second layer are subdivided into an array of second blocks, wherein a raster scan decoding order is defined in the first blocks and in the second blocks, respectively, the method comprising: setting a long-term syntax element structure and short-term syntax elements and inserting the long-term syntax element structure and the short-term syntax elements into the multi-layer video data stream, the short-term syntax elements of the multi-layer video data stream periodically defining, within a time interval, the size and the positioning of first blocks of pictures of the first layer and second blocks of pictures of the second layer and the spatial sampling resolution of the pictures of the first layer and of the pictures of the second layer, respectively, and, if the long-term syntax element structure is set to a value of a first set of possible values, the value signaling, over a predetermined time period larger than the time interval, an inter-layer delay between traversals of the first blocks of pictures of the first layer and of the second blocks of pictures of the second layer measured in units of the first blocks in a time-overlapped manner sequentially traversing the first blocks and the second blocks for decoding the pictures of the first blocks and of the second blocks with an inter-layer offset, and setting the short-term syntax elements to an appropriate subset outside a set of possible settings for the predetermined time period, the appropriate subset being selected such that, for the predetermined time period, actual inter-layer offsets between traversals of the first blocks of pictures of the first layer and of the second blocks of pictures of the second layer smaller than or equal to the inter-layer offset signaled by the long-term syntax element structure, the size and the positioning of the first blocks of pictures of the first layer and of the second blocks of pictures of the second layer and the spatial sampling resolution of the pictures of the first layer and of the pictures of the second layer, respectively, can support decoding of the multi-layer video data stream by sequentially traversing the first blocks and the second blocks in a time-overlapped manner.

10. A method for storing a video of a scene, the method comprising: storing a multi-layer video data stream on a digital storage medium, the multi-layer video data stream having the scene encoded into the multi-layer video data stream using the method according to claim 9.

11. A method for transmitting data on a scene, the method comprising: transmitting a multi-layer video data stream having the scene encoded into the multi-layer video data stream, wherein the scene is encoded into the multi-layer video data stream using the method according to claim 9.

Images